Image authenticity determination method and device, equipment, medium and program product

Abstract

The invention provides an image authenticity determination method and device, equipment, a medium and a program product, and relates to the technical field of image processing. The method comprises the following steps: acquiring an original image; performing feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions; carrying out feature fusion processing on the image features extracted from different dimensions based on a self-adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; and inputting the multi-modal features into a trained image prediction model for prediction processing, and outputting a tampering probability value for tampering the original image. The embodiment of the invention is used for solving the defect of low image authenticity judgment accuracy in the prior art, effectively integrating the information of three dimensions of the spatial domain, the frequency domain and the statistical features of the original image, and predicting the tampering probability value of the original image on the basis of feature fusion, so as to determine the authenticity of the original image according to the probability value, thereby improving the authenticity judgment accuracy of the original image. And the authenticity discrimination accuracy is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image authenticity determination method, device, equipment, medium and program product. BACKGROUND

[0002] With the rapid development of digital image generation and editing technology, AI tools (such as deep learning models) can efficiently generate highly realistic fake images. Traditional image authenticity determination techniques mainly rely on single features, such as noise fingerprint, illumination consistency, or edge detection, which are low-level features. These methods perform poorly when facing cross-domain tampering or quality degradation. For example, although noise fingerprint detection methods are theoretically feasible, they are easily disturbed by image compression, filtering and other post-processing operations in practical applications, resulting in decreased detection accuracy. Illumination consistency detection methods can capture the illumination inconsistency in the splicing area, but they often fail for AI-generated images.

[0003] Therefore, how to effectively determine the authenticity of an image is a technical problem that needs to be solved at present. SUMMARY

[0004] The present application provides an image authenticity determination method, device, equipment, medium and program product to solve the low accuracy of image authenticity determination in the prior art. By effectively integrating the information of the three dimensions of the spatial domain, frequency domain and statistical features of the original image, and then predicting the probability value of the original image tampering based on feature fusion, the authenticity of the original image is determined according to the probability value, greatly improving the accuracy of authenticity determination.

[0005] In a first aspect, the present application provides an image authenticity determination method, comprising the following steps: obtaining an original image; performing feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions; performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; inputting the multi-modal features into a trained image prediction model for prediction processing, and outputting a tampering probability value of the original image; wherein the image prediction model is trained based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0006] Preferably, the image authenticity determination method provided by the application, the feature extraction processing of the original image from different dimensions to obtain the corresponding dimensional image features comprises: extracting the noise fingerprint of the original image, the noise fingerprint being generated based on the random noise in the original image; inputting the original image and the noise fingerprint into a composite scaling neural network for feature extraction to obtain spatial domain features of a spatial dimension; wherein the structure of the composite scaling neural network adopts a coordinate attention mechanism, uses a lightweight activation function, and removes the original classification head; the coordinate attention mechanism is used to split the spatial information of the image input into the composite scaling neural network into a height direction and a width direction, and model them respectively.

[0007] Preferably, the image authenticity determination method provided by the application, the feature extraction processing of the original image from different dimensions to obtain the corresponding dimensional image features further comprises: format conversion is performed on the original image, and a luminance channel image of the original image after format conversion is extracted; segmenting the luminance channel image to obtain a plurality of pixel blocks; performing discrete cosine transform processing on each pixel block to obtain a discrete transform coefficient matrix, and extracting high-frequency alternating current coefficients of a high-frequency region from the discrete transform coefficient matrix; performing convolution dimension reduction processing on the high-frequency alternating current coefficients to obtain dimension reduction features of each pixel block, and performing global average pooling processing on the dimension reduction features of each pixel block to obtain frequency domain features of a frequency domain dimension.

[0008] Preferably, the image authenticity determination method provided by the application, the feature extraction processing of the original image from different dimensions to obtain the corresponding dimensional image features further comprises: performing image grayscale processing on the original image, and performing normalization processing on the original image after image grayscale processing to obtain a normalized image; calculating the gradient amplitude and direction matrix of each pixel in the normalized image, dividing the normalized image into a cell unit of a preset pixel, and calculating the gradient direction histogram of each cell unit according to the gradient amplitude and direction matrix of each pixel; combining the cell units to obtain image blocks, and concatenating the gradient direction histograms of all cell units in each image block to form a feature vector corresponding to each image block; performing normalization processing on the feature vector of each image block to obtain the normalized features of each image block; The normalized features of all image blocks are extracted by traversing the normalized image through a sliding window, and the normalized features of all image blocks are concatenated to generate statistical features of a statistical dimension.

[0009] Preferably, according to the image authenticity determination method provided by the application, the image features extracted from different dimensions are subjected to feature fusion processing based on the adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights, which comprises: The spatial domain features are taken as query vectors, the frequency domain features are taken as key vectors, and the statistical features are taken as value vectors to calculate first attention weights; The frequency domain features are taken as query vectors, the spatial domain features are taken as key vectors, and the statistical features are taken as value vectors to calculate second attention weights; The statistical features are taken as query vectors, the spatial domain features are taken as key vectors, and the frequency domain features are taken as value vectors to calculate third attention weights; Based on the first attention weights, the second attention weights and the third attention weights, the spatial domain features, the frequency domain features and the statistical features are subjected to fusion calculation processing to obtain the multi-modal features.

[0010] Preferably, according to the image authenticity determination method provided by the application, whether the original image is a tampered image is determined, which comprises: In the case that the tampering probability value of the original image is greater than or equal to a preset threshold, the original image is determined to be a tampered image; In the case that the tampering probability value of the original image is less than a preset threshold, the original image is determined to be a real image; After the step of determining that the original image is a tampered image, the method comprises: The multi-modal features of the tampered image are subjected to up-sampling processing to generate a heat map of a tampered region.

[0011] In a second aspect, the application further provides an image authenticity determination device, which comprises the following modules: An acquisition module is configured to acquire an original image. A feature extraction module is configured to extract features of the original image from different dimensions to obtain image features corresponding to the dimensions. A feature fusion module is configured to fuse the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights. A prediction module is configured to input the multi-modal features into a trained image prediction model for prediction processing to output a tampering probability value of the original image. The image prediction model is obtained based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0012] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image authenticity determination method according to any one of the above aspects when executing the program.

[0013] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the image authenticity determination method according to any one of the above aspects.

[0014] In a fifth aspect, the present application also provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the image authenticity determination method according to any one of the above aspects.

[0015] The image authenticity determination method, device, equipment, medium and program product provided by the present application obtain an original image, perform feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions, perform feature fusion processing on the image features extracted from different dimensions based on a self-adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights, input the multi-modal features into a trained image prediction model for prediction processing, and output a tampering probability value of the original image. The image prediction model is obtained based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with. The present application solves the problem of low accuracy of image authenticity discrimination in the prior art, effectively integrates information in three dimensions of the spatial domain, the frequency domain and statistical features of the original image, then predicts a probability value of the original image tampering based on feature fusion, determines the authenticity of the original image according to the probability value, and greatly improves the accuracy of authenticity discrimination. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0017] Figure 1 is one of the flowcharts of the image authenticity determination method provided by the present application.

[0018] Figure 2 is another flowchart of the image authenticity determination method provided by the present application.

[0019] Figure 3 is a structural schematic diagram of the image authenticity determination device provided by the present application.

[0020] Figure 4 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] In the related art, at least the following technical problems exist: Deep learning methods have made significant progress in recent years, especially tamper detection models based on convolutional neural networks (CNN). These methods can learn complex image feature representations, but are vulnerable to adversarial attacks. Existing research has shown that deep learning models are vulnerable to adversarial samples, which can deceive the model to make incorrect classifications through subtle perturbations. In addition, large models, although accurate, have high computational costs and slow inference speeds, making it difficult to meet real-time detection needs.

[0023] The present application is described below in conjunction with Figures 1-4 An image authenticity determination method, device, equipment, medium, and program product are provided to solve the defect of low accuracy of image authenticity discrimination in the prior art. The information in three dimensions of the spatial domain, frequency domain, and statistical features of the original image is effectively integrated, and then the probability value of the original image tampering is predicted based on feature fusion to determine the authenticity of the original image according to the probability value, greatly improving the accuracy of authenticity discrimination.

[0024] Figure 1 is one of the flowcharts of the image authenticity determination method provided by the present application, as Figure 1 shown, the method can include but is not limited to steps S100 to S400: S100, obtaining an original image; S200, performing feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions; S300, performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; S400, inputting the multi-modal features into a trained image prediction model for prediction processing and outputting a tampering probability value of the original image; The image prediction model is trained based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0025] In step S100 of some embodiments, an original image is obtained.

[0026] The original image can be obtained from a monitoring video frame, a social media platform picture, a medical image, a scanned copy of a certificate, etc., to determine whether the original image is real or generated through tampering or artificial intelligence.

[0027] In step S200 of some embodiments, feature extraction processing is performed on the original image from different dimensions to obtain image features corresponding to the dimensions.

[0028] It can be understood that the original image is extracted from the spatial dimension, the frequency domain dimension, and the statistical dimension to obtain corresponding spatial domain features, frequency domain features, and statistical features. Specifically, it can be: extracting a noise fingerprint of the original image, the noise fingerprint being generated based on random noise in the original image; inputting the original image and the noise fingerprint into a composite scaling neural network for feature extraction to obtain spatial domain features in the spatial dimension; The structure of the composite scaling neural network is to use a coordinate attention mechanism, use a lightweight activation function, and remove the original classification head; the coordinate attention mechanism is used to split the spatial information of the image input into the composite scaling neural network into a height direction and a width direction, and model them respectively.

[0029] It can be understood that the present application uses a lightweight deep learning model (EfficientNet) to extract the spatial features of the original image, and at the same time, uses noise fingerprint (Noiseprint) technology to analyze the spatial distribution pattern of image noise. The specific implementation is as follows: The noise fingerprint of the original image is first extracted using the Noiseprint++ Extractor. The Noiseprint++ Extractor generates a unique fingerprint by analyzing the random noise in the original image, which has different responses to fake images and real images. The sensitivity of this fingerprint enables it to capture subtle traces of image tampering, thereby assisting other components in more accurate image forgery detection and positioning. The noise fingerprint and the original image are input into the EfficientNet (composite scaling neural network) for feature extraction to obtain spatial domain features with spatial dimensions In the embodiments of the present application, the composite scaling neural network replaces the original compression-excitation attention mechanism of EfficientNet with a coordinate attention mechanism to enhance the position sensitivity of the features.

[0030] The composite scaling neural network uses a lightweight activation function h-swish to further reduce computational complexity.

[0031] The composite scaling neural network removes the original classification head of EfficientNet, retains the backbone as a feature extractor, retains the output feature map of the last convolutional layer, and finally compresses it to a 1x1x512 vector through global average pooling to obtain the final spatial domain features.

[0032] Further, the noise fingerprint (Noise Fingerprint) is a random noise pattern introduced by the sensor or transmission channel in the image, which has device uniqueness (such as camera sensor fixed pattern noise).

[0033] High-frequency noise signals in the image are extracted through high-pass filtering or deep learning models (such as Noiseprint++ Extractor). The noise signals are converted into fixed-length feature vectors (such as through hash or statistical quantity calculation) to form noise fingerprints. Noise fingerprints have device dependency and can distinguish images generated by different devices, assisting in detecting fake content (such as deep fake or retouching tampering).

[0034] The original image and the noise fingerprint are input into the composite scaling neural network, and a 512-dimensional spatial domain feature vector is output.

[0035] The network structure characteristics of the composite scaling neural network are as follows: The role of the coordinate attention mechanism (Coordinate Attention) is to decompose the spatial information of the input image into height direction (row) and width direction (column), model them separately, and then fuse them to enhance the perception of local position relationships.

[0036] The specific execution steps of the coordinate attention mechanism are as follows: the input feature map is split into two independent channels: the row direction (height axis) and the column direction (width axis).

[0037] One-dimensional convolution and activation functions (such as h-swish) are performed on the row / column direction respectively to generate attention weights.

[0038] The attention weights of the two directions are multiplied to obtain a spatial attention map.

[0039] The input features are weighted using the attention map to enhance the features of key regions (such as edges and textures).

[0040] The beneficial effects of using the coordinate attention mechanism to replace the SE attention mechanism of EfficientNet mainly include two points: Unlike the SE attention mechanism, which only focuses on channel importance, the coordinate attention mechanism explicitly models spatial position information in the height and width directions, which helps the model perceive the geometric structure (such as edges and shapes) of objects and has high scene adaptability.

[0041] The parameter amount and computational amount of the coordinate attention mechanism are very small, which is beneficial to model lightweight and improve inference efficiency.

[0042] Lightweight activation function (h-swish): The h-swish activation function is a variant based on the swish activation function, aiming to reduce computational cost while maintaining similar performance. In this invention, the use of the h-swish activation function and other activation functions is the same.

[0043] Swish activation function: H-swish activation function: ReLU6 is a special variant of ReLU: This means that when the input x is less than -3, the output is 0; when x is greater than 3, the output is fixed at 6; and the value between -3 and 3 is linearly increased.

[0044] H-swish effectively reduces the computational burden by replacing complex nonlinear functions (such as Sigmoid) with simple linear segment functions, making the network more lightweight. In addition, h-swish has shown that it can match or even surpass traditional activation functions in many experiments, especially in deep neural networks.

[0045] Remove the classification head: delete the fully connected layer at the end of the network, keep the feature extraction backbone, and output a high-dimensional feature map.

[0046] Global Average Pooling (GAP): Apply GAP operation on the feature map output by the last convolutional layer of CSNN, such as a 7x7x512 size, to compress it into a 1x1x512 vector. It can reduce the dimension, preserve global features, and eliminate spatial position effects.

[0047] Final output: 512-dimensional spatial domain feature vector.

[0048] Format conversion is performed on the original image, and the luminance channel image of the original image after format conversion is extracted. The luminance channel image is segmented to obtain a plurality of pixel blocks. Discrete cosine transform is performed on each pixel block to obtain a discrete transform coefficient matrix, and high-frequency alternating current coefficients of a high-frequency region are extracted from the discrete transform coefficient matrix. The high-frequency alternating current coefficients are subjected to convolution dimension reduction processing to obtain the dimension reduction features of each pixel block, and the dimension reduction features of each pixel block are subjected to global average pooling processing to obtain frequency domain features of the frequency domain dimension.

[0049] It can be understood that the luminance channel image extraction step includes: format conversion: converting the original image from RGB color space to YCbCr or HSV color space, and extracting the luminance channel (such as Y channel). Luminance channel: represents the gray scale information of the image, reflects the light intensity distribution, and is independent of color.

[0050] Among them, color space conversion: converting an image from one color representation form (such as RGB) to another (such as YCbCr) to separate luminance and chrominance information.

[0051] Luminance channel (Y channel): In the YCbCr color space, the Y channel represents luminance, and Cb and Cr represent chrominance. The luminance channel is more sensitive to changes in light.

[0052] Execution steps: convert the RGB image to the YCbCr color space. Extract the Y channel as the luminance image.

[0053] Image segmentation and pixel block division steps include: dividing the luminance channel image into a plurality of fixed-size pixel blocks (such as 8x8 or 16x16).

[0054] Pixel block (patch): a continuous rectangular region in an image used for local feature extraction.

[0055] Divide the luminance image into non-overlapping pixel blocks of a fixed size (such as 8x8), and process each pixel block independently.

[0056] The step of performing discrete cosine transform (DCT) on each pixel block to obtain a DCT coefficient matrix and extracting high-frequency alternating current (AC) coefficients of a high-frequency region from the DCT coefficient matrix comprises: The step of performing discrete cosine transform (DCT) on each pixel block to obtain a DCT coefficient matrix and extracting high-frequency alternating current (AC) coefficients of a high-frequency region from the DCT coefficient matrix comprises:

[0057] Discrete cosine transform (DCT): converting a spatial domain signal to a frequency domain signal, commonly used in image compression (e.g., JPEG). Low-frequency coefficients represent smooth regions, and high-frequency coefficients represent edges and textures.

[0058] High-frequency alternating current (AC) coefficient: a coefficient of a non-direct current component after DCT transformation, corresponding to edges and details in an image.

[0059] The step of performing discrete cosine transform (DCT) on each pixel block to obtain a DCT coefficient matrix and extracting high-frequency alternating current (AC) coefficients of a high-frequency region from the DCT coefficient matrix comprises:

[0060] The step of performing discrete cosine transform (DCT) on each pixel block to obtain a DCT coefficient matrix and extracting high-frequency alternating current (AC) coefficients of a high-frequency region from the DCT coefficient matrix comprises:

[0061] Global average pooling (GAP): taking the average value of all positions of the feature map of each pixel block to generate a single feature value.

[0062] Convolutional dimension reduction: design a convolution kernel (e.g., 3x3 or 5x5) to perform convolution operation on the high-frequency AC coefficient matrix. Output a low-dimensional feature vector (e.g., 1x1xk) for each pixel block.

[0063] Global average pooling: taking the average value of the convolution feature vector of each pixel block to generate a scalar feature. Concatenate the scalar features of all pixel blocks to obtain a vector of global frequency domain features.

[0064] This embodiment reduces the impact of illumination changes on features by using the luminance channel and high-frequency coefficient extraction. Dividing the pixel blocks captures local textures, and global pooling retains overall features.

[0065] In some embodiments of the present application, the original image is converted in the frequency domain (using Discrete Cosine Transform - DCT), and the frequency domain features are extracted. There are significant differences between real images and AI-generated images in the frequency domain, especially in the high-frequency region information distribution. After the original image is processed by the DCT module, the frequency domain feature vector is extracted by the convolution network, and the specific implementation is as follows: First, the original image is converted into YCbCr format, and the luminance channel Y is extracted for subsequent processing; the Y channel image is segmented into 8x8 pixel blocks, and DCT transformation is performed on each block; 16 AC coefficients in the high-frequency region (u / v≥4) are extracted from the DCT coefficient matrix as features; for a two-dimensional data block of size MxN (such as an image sub-block), its DCT matrix can be represented as a coefficient matrix F(u,v) of the same size, where u=0,1,...,M-1 and v=0,1,...,N-1 represent the frequency index in the horizontal and vertical directions, respectively.

[0066] The mathematical expression of the DCT matrix is as follows: represents the original spatial domain data (such as image pixel values), is the corresponding DCT coefficient, M and N are the height and width of the data block, , and are normalization factors.

[0067] If , then ; otherwise, , and

[0068] The process of calculating the DCT coefficient matrix involves applying the above formula to each point , which usually requires a double summation operation for each frequency component u and v, and the amount of calculation is very large.

[0069] When discussing DCT, there are AC and DC coefficients: DC coefficient: this is the coefficient at F(0,0), which represents the average value or "direct current" part of the entire block. It is the average color intensity in the entire pixel block.

[0070] AC coefficient: all other coefficients except F(0,0) are AC coefficients. These coefficients describe the variation or detail information within the image block. Each AC coefficient corresponds to a specific spatial frequency pattern, and higher frequencies usually mean finer details or edge information.

[0071] ​Further, in the embodiments of the present application, multiprocessing.Pool is used to accelerate the calculation of block DCT. Using multiprocessing.Pool to accelerate the calculation of block DCT is an effective method, especially when processing a large amount of image data blocks. By parallelizing the DCT calculation process, the total processing time can be significantly reduced. multiprocessing.Pool is provided by an open source python package, which will not be described here. The feature dimensions obtained by each module (spatial domain feature extraction module, frequency domain feature extraction module and statistical feature extraction module) are different, so the dimensions need to be uniformly processed to the shape of 1x1x512, so that feature fusion can be performed; the extracted high-frequency coefficients are reduced in dimension by 1x1 convolution, and the 16-dimensional features of each block are compressed to 5-dimensional, and then converted to a 512-dimensional vector by global average pooling.

[0072] perform image grayscale processing on the original image, and perform normalization processing on the original image after image grayscale processing to obtain a normalized image; calculate the gradient amplitude and direction matrix of each pixel in the normalized image, divide the normalized image into cell units of a preset pixel, and calculate the gradient direction histogram of each cell unit according to the gradient amplitude and direction matrix of each pixel; combine the cell units to obtain image blocks, and concatenate the gradient direction histograms of all cell units in each image block to form a feature vector corresponding to each image block; perform normalization processing on the feature vector of each image block to obtain the normalized feature of each image block; extract the normalized features of all image blocks by traversing the normalized image through a sliding window, and concatenate the normalized features of all image blocks to generate statistical features of statistical dimensions.

[0073] It can be understood that the step of performing image grayscale processing on the original image and performing normalization processing on the original image after image grayscale processing to obtain a normalized image comprises: convert the original color image into a grayscale image (single channel), and normalize it (such as scaling to the range of [0, 1] or [0, 255]).

[0074] Grayscale: convert the RGB channel of a color image into a single intensity channel, eliminating color information interference.

[0075] Normalization: Unify pixel value range by linear transformation (e.g. divide by maximum pixel value), reduce the influence of illumination difference on features. The specific implementation steps are: the original image in RGB form can be converted into a gray image using the formula Gray = 0.299R + 0.587G + 0.114B.

[0076] Normalization: Normalized_Image = Gray / 255 (assuming the original gray range is [0, 255]).

[0077] The step of calculating the gradient magnitude and direction matrix of each pixel in the normalized image includes: calculating the gradient magnitude (intensity) and direction (angle) of each pixel, and dividing the direction into 9 intervals (20° per Bin, i.e. one interval).

[0078] It should be noted that the gradient magnitude: represents the degree of intensity change of the pixel, which is calculated by the Sobel operator. Gradient direction: represents the direction of intensity change, ranging from [0°, 180°) (unsigned direction). Bins: discretize the direction into multiple intervals for histogram statistics.

[0079] Calculate the horizontal gradient (Gx) and vertical gradient (Gy) using the Sobel operator, and the gradient magnitude: Magnitude = sqrt(Gx² + Gy²).

[0080] Gradient direction: Angle = arctan2(Gy, Gx) (180 / π), if the result is negative, add 180°. The direction is divided into 9 bins (e.g. 0°-20°, 20°-40°,..., 160°-180°).

[0081] Further, the normalized image is divided into a preset pixel cell unit, and the gradient direction histogram of each cell unit is calculated according to the gradient magnitude and direction matrix of each pixel, including: the image is divided into a cell unit of 8x8 pixels (Cell), and the gradient direction histogram is counted in each Cell.

[0082] Cell: a local area used to capture small-range texture features. Gradient direction histogram: statistics of the distribution of gradient direction in each Cell (9-dimensional vector).

[0083] Divide the normalized image into a grid of 8x8 pixel Cells. For each pixel in each Cell, accumulate the gradient magnitude as the histogram weight according to the bin to which the gradient direction belongs. Each Cell outputs a 9-dimensional histogram vector.

[0084] Further, the cell units are combined to obtain image blocks, and a gradient direction histogram of all the cell units in each image block is concatenated to form a feature vector corresponding to each image block, including: 2x2 Cells are combined into image blocks (Block), and their histograms are concatenated to form a 36-dimensional feature vector. It should be noted that the image block (Block): a larger area composed of multiple Cells, used to enhance the stability of the features. Feature vector concatenation: concatenating the histograms of multiple Cells into a long vector in order.

[0085] Sliding windows are used in units of 2x2 Cells to cover the entire image. The histograms of the 4 Cells (each 9-dimensional) in each Block are concatenated to obtain a 36-dimensional feature vector.

[0086] Further, the feature vector of each image block is normalized to obtain the normalized feature of each image block, i.e., L2 norm normalization, including: The 36-dimensional feature vector of each Block is L2 normalized to improve the robustness to light.

[0087] It should be noted that L2 norm normalization: the vector is divided by its module length, so that the vector length is 1.

[0088] The L2 norm of the Block feature vector is calculated: Norm = sqrt(Σv_i²). Normalization: v_i = v_i / Norm (if Norm is 0, the original value is retained).

[0089] Further, the normalized image is traversed by a sliding window to extract the normalized features of all image blocks, and the normalized features of all image blocks are concatenated to generate statistical features of statistical dimensions, including: All the normalized features of the Blocks are extracted and concatenated by traversing the image with a sliding window (step size is Cell size, i.e., 8 pixels).

[0090] It should be noted that the sliding window: the Block moves with a fixed step size to cover the entire image. Global HOG feature: a high-dimensional vector obtained by concatenating all Block features.

[0091] In the embodiment of the application, the sliding window starts from the top left corner and moves 8 pixels (Cell size) each time. The normalized feature vectors of each Block are concatenated in spatial order to form a global HOG feature.

[0092] Further, the embodiment of the present application uses 1x1 convolution to compress high-dimensional HOG features (such as thousands of dimensions) to 512 dimensions. 1x1 convolution: linearly combines input features through point convolution operation to realize dimension reduction or feature fusion. In the embodiment of the present application, the number of input channels is the dimension of the global HOG feature, and the number of output channels is 512 by designing a 1x1 convolution layer. Through convolution operation, high-dimensional features are mapped to a 512-dimensional low-dimensional space.

[0093] In step S300 of some embodiments, the image features extracted from different dimensions are subjected to feature fusion processing based on an adaptive weight fusion mechanism, to obtain multi-modal features with dynamically adjusted weights.

[0094] It can be understood that the attention module (such as SE-Block or CBAM) is used to dynamically weight the features of different dimensions. For example: If the image has obvious edge breaks, the weight of the spatial domain edge feature is increased; If high-frequency noise is detected, the confidence of the frequency domain feature is enhanced.

[0095] The weighted features are fused through concat or FPN (Feature Pyramid Network) to generate multi-modal features with unified representation.

[0096] Specifically, the spatial domain feature can be taken as a query vector, the frequency domain feature can be taken as a key vector, and the statistical feature can be taken as a value vector, to calculate a first attention weight; The frequency domain feature can be taken as a query vector, the spatial domain feature can be taken as a key vector, and the statistical feature can be taken as a value vector, to calculate a second attention weight; The statistical feature can be taken as a query vector, the spatial domain feature can be taken as a key vector, and the frequency domain feature can be taken as a value vector, to calculate a third attention weight; Based on the first attention weight, the second attention weight, and the third attention weight, the spatial domain feature, the frequency domain feature, and the statistical feature are subjected to fusion calculation processing to obtain the multi-modal feature.

[0097] It can be understood that the embodiment of this step dynamically adjusts the weights of different modal features through cross-attention mechanism, and realizes multi-modal feature fusion combined with learnable parameters (α, β, γ). According to the importance of input features (such as spatial edge, frequency domain noise, and statistical distribution), the weights are dynamically allocated to enhance the expression ability of key features. First of all, it should be pointed out that adaptive weight fusion: through attention mechanism or learnable parameters, the weights of features from different sources are automatically adjusted, rather than fixed weights.

[0098] Cross-attention mechanism: Different modalities of features alternate as Query, Key, Value to realize cross-modal feature interaction. Query / Key / Value (Q / K / V) vector: Query: "question" vector for matching other modalities of features. Key: "key" vector to be matched. Value: "value" vector to be weighted.

[0099] SE-Block: Squeeze-and-Excitation module, which generates channel weights through global pooling and fully connected layers. CBAM: Convolutional Block Attention Module, which combines channel attention and spatial attention. FPN (Feature Pyramid Network): Multi-scale feature fusion structure, used to integrate features at different levels.

[0100] Further, Spatial: 512-dimensional, capturing the spatial structure of the image (such as edges, textures). Frequency: 512-dimensional, extracting frequency information (such as high-frequency noise) through Fourier transform / wavelet transform. Statistics: 512-dimensional, representing global statistical properties (such as mean, variance, histogram). Initialize learnable parameters α = 0.5, β = 0.5, γ = 0.5 (later updated through training).

[0101] Further, based on spatial domain features, frequency domain features, and statistical features, three rounds of cross-attention calculations are performed to mine inter-modal correlations by alternating Q / K / V roles.

[0102] The spatial domain features are used as query vectors, the frequency domain features are used as key vectors, and the statistical features are used as value vectors to calculate the first attention weight, including: Attention 1 (spatial dominant): Query: spatial domain feature S. Key / Value: frequency domain feature F and statistical feature T.

[0103] Calculate the first attention weight: A1 = Softmax(Q1 [K1; K2]) Where Q1 = S, the spatial domain feature, K1 = F, the frequency domain feature, and K2 = T, the statistical feature.

[0104] Weighted sum value vector: V1 = A1[:,:F.size(1)] F + A1[:,F.size(1):] T.

[0105] Further, the frequency domain feature is taken as a query vector, the spatial domain feature is taken as a key vector, and the statistical feature is taken as a value vector, and a second attention weight is calculated, including: Attention 2 (frequency dominant): Query: frequency domain feature F, Key / Value: spatial domain feature S and statistical feature T.

[0106] The second attention weight is calculated: A2 = Softmax(Q2 [K2; K3]) Wherein, Q2=F, K2=S, K3=T.

[0107] The weighted sum value vector is calculated: V2 = A2[:,:S.size(1)] S + A2[:,S.size(1):] T.

[0108] Further, the statistical feature is taken as a query vector, the spatial domain feature is taken as a key vector, and the frequency domain feature is taken as a value vector, and a third attention weight is calculated, including: Attention 3 (statistical dominant): Query: statistical feature T, Key / Value: spatial domain feature S and frequency domain feature F.

[0109] The third attention weight is calculated: A3 = Softmax(Q3 [K3; K4]) Wherein, Q3=T, K3=S, K4=F.

[0110] The weighted sum value vector is calculated: V3 = A3[:,:S.size(1)] S + A3[:,S.size(1):] F.

[0111] Further, the adaptive weighted fusion step includes: The three rounds of attention results V1, V2, and V3 are weighted according to the following formula: Fusion = α·V1 + β·V 2 + γ·V3 Optimize α, β, and γ through back propagation, so that the fused feature is more suitable for the task (such as classification or heat map generation). The fused feature Fusion is input into the subsequent task (such as full connection layer classification or convolution heat map generation, which can be used for subsequent classification and heat map generation).

[0112] In some embodiments, the original features or FPN can also be concatenated to further fuse multi-scale information. The embodiment of the application can realize automatic focusing on key modalities (such as noise scene relying on frequency domain and blurred scene relying on spatial domain) through an attention mechanism.

[0113] In step S400 of some embodiments, the multi-modal features are input into a trained image prediction model for prediction processing, and a tampering probability value of tampering the original image is output. The image prediction model is trained based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered.

[0114] It can be understood that the real samples include natural images and artificially tampered images (such as DeepFake detection library).

[0115] The adversarial samples are synthetic images generated by a text-to-image model (such as DALL-E or Stable Diffusion) to describe “tampered images”, simulating advanced forgery attacks.

[0116] The combined classification cross-entropy loss (real / tampered classification) and adversarial loss (gradient penalty of adversarial samples) are used to optimize the robustness of the model.

[0117] The multi-modal features are input into a trained model (such as a lightweight ResNet or Transformer), and a tampering probability value (0-1) is output.

[0118] Threshold determination: if the probability is greater than 0.5, it is marked as tampered; otherwise, it is a real image.

[0119] In some embodiments of the application, in a deep learning-based multi-task learning architecture, the classification task is implemented through the following process: The multi-modal features (feature vectors fused after spatial domain, frequency domain, and statistical features) are input into a trained image prediction model, and a tampering probability value is output to determine whether the original image is tampered.

[0120] The image prediction model classifies the multi-modal features through a fully connected layer, and outputs a probability value of whether the image is tampered (such as 0.96 indicating 96% confidence that the image is not tampered).

[0121] The model uses Focal Loss as the loss function, dynamically adjusts the weights of difficult samples, and solves the class imbalance problem in tampering detection (such as the number of un-tampered samples being much larger than the number of tampered samples).

[0122] Adversarial training: Introduce adversarial image samples (generated based on text-to-image generation model) during model training, optimize model parameters through adversarial training, and improve the robustness of the model to complex tampering attacks.

[0123] In some embodiments, the multi-modal feature vector output by the feature fusion module is input into the image prediction model. ) input image prediction model.

[0124] The model structure of the image prediction model: fully connected layer + Sigmoid activation function. The fully connected layer is used to map the 512-dimensional features to a 1-dimensional output (tampering probability). Sigmoid is used to compress the output to the [0, 1] interval, representing the tampering probability.

[0125] The model training (including adversarial training) steps of the image prediction model include: Obtain training data, which includes at least image feature samples and adversarial image samples. The image feature sample is the multi-modal feature of the real un-tampered image. The adversarial image sample is the multi-modal feature of the fake image synthesized based on the text-to-image generation model.

[0126] Training process: 1. Data preparation: Real images: extract spatial, frequency, and statistical features, and label as "un-tampered". Adversarial samples: generate fake images through a text-to-image generation model (e.g., input the text "add noise"), extract multi-modal features, and label as "tampered".

[0127] 2. Loss calculation: Use Focal Loss to calculate the classification loss, as follows: FL(p_t) = -α(1-p_t)^γ log(p_t) Where p_t is the correct class probability (un-tampered or tampered) predicted by the model, α is the balance factor (which can be set as the inverse of the proportion of tampered samples), and γ is the adjustment of difficult sample weight (usually take 2).

[0128] 3. Adversarial training: Mix adversarial samples with real samples to train the model to distinguish between the two. Adversarial samples are dynamically updated through a generation model to simulate new attack methods.

[0129] Input the multi-modal feature of the image to be detected into the trained image prediction model, and output the tampering probability value (e.g., 0.96). The judgment rule is: if the probability value > threshold (e.g., 0.5), it is determined to be "tampered". If the probability value ≤ threshold, it is determined to be "un-tampered".

[0130] The adversarial training of the embodiment of the present application exposes the model to attacks by the generation model, improving the defense capability against unknown tampering methods.

[0131] In some embodiments of the present application, determining whether the original image is a tampered image comprises: In the case where the tampering probability value of the original image is greater than or equal to a preset threshold, determining that the original image is a tampered image; In the case where the tampering probability value of the original image is less than a preset threshold, determining that the original image is a real image; After the step of determining that the original image is a tampered image, the method comprises: Upsampling the multi-modal features of the tampered image to generate a heat map of the tampered region.

[0132] It can be understood that, in some embodiments of the present application, based on the U-Net upsampling path, the fused multi-modal features are upsampled layer by layer to restore the spatial resolution and generate a heat map of the tampered region. Through skip connection (Skip Connections) to fuse low-level detail features, combined with Dice Loss to optimize small target positioning, the final output is a pixel-level tampering probability distribution.

[0133] It should be noted that the meaning of upsampling (Upsampling) is to enlarge the low-resolution feature map to a high-resolution operation to restore spatial information. The meaning of transposed convolution (Transposed Convolution) is to derive the pixel position through convolution operation, and the upsampled parameters can be learned. The meaning of bilinear interpolation (Bilinear Interpolation) is to insert new pixels based on the weighted average of neighboring pixels, and the parameters cannot be learned.

[0134] The meaning of heat map (Heatmap) is a two-dimensional matrix, and each value represents the tampering probability (range [0, 1] or [0, 1] normalized) of the corresponding position.

[0135] The specific steps of upsampling the multi-modal features of the tampered image to generate a heat map of the tampered region can include but are not limited to the following steps: 1. The step of inputting the multi-modal features into the positioning prediction model is: The multi-modal features output in the feature fusion stage (such as 512-dimensional vectors or low-resolution feature maps, shape like [batch, 512, H, W]) are input into the positioning prediction model for tampering positioning prediction.

[0136] In some embodiments, if the multi-modal features are global vectors (without spatial information), they need to be reshaped (reshape) to two-dimensional feature maps (such as [batch, 512, 1, 1]→[batch, 512, H, W]).

[0137] 2. Up-sampling operation steps: Transposed convolution: learn up-sampling parameters using transposed convolution with stride 2 (e.g., ConvTranspose2d(512, 256, kernel_size=3, stride=2)). Bilinear interpolation: directly up-scale feature map size (e.g., from 32x32 to 64x64) without changing the number of channels.

[0138] Output: spatial resolution doubled (e.g., [batch, 256, 64, 64]), and the number of channels halved (symmetric to down-sampling path).

[0139] 3. Skip connection steps: Feature alignment: extract the corresponding scale feature map from the down-sampling path (e.g., the feature map from the 3rd layer of the encoder [batch, 256, 64, 64]).

[0140] Crop operation: if the up-sampled feature map size is larger than the encoder feature map (e.g., boundary expansion), crop to the consistent size.

[0141] Feature concatenation: concatenate the up-sampled feature and the encoder feature in the channel dimension (e.g., [batch, 256, 64, 64] + [batch, 256, 64, 64] → [batch, 512, 64, 64]).

[0142] 4. Convolution processing steps: apply two consecutive 3x3 convolutions + ReLU to the concatenated feature map to fuse the encoder and decoder features, restore spatial details, and reduce the number of channels.

[0143] 5. Repeat up-sampling and skip connection steps: loop execution: according to the symmetric structure of U-Net, repeat the above steps to gradually restore the original image resolution (e.g., from 64x64 → 128x128 → 256x256).

[0144] Final feature: obtain a feature map with the same size as the original image (e.g., [batch, 1, 256, 256]).

[0145] 6. Generate heat map steps: 1x1 convolution: map the number of channels to the target class number (e.g., binary classification output [batch, 1, 256, 256]).

[0146] Activation function: apply Sigmoid function to compress the value to [0, 1], representing the tampering probability.

[0147] Output: heat map (e.g., [batch, 1, 256, 256]), the larger the value, the higher the tampering probability.

[0148] The function of the Dice Loss loss function is to optimize the overlapping area of the predicted heat map and the real mask, and alleviate the difficulty of small target (such as a tiny tampering area) detection.

[0149] The formula of the Dice Loss loss function is as follows: In the formula, is the predicted value, that is, the heat map pixel, is the real label (a binary mask, the tampering area is 1, and the rest is 0).

[0150] The embodiment of the application restores the spatial resolution layer by layer through the upsampling path, combines the multi-scale features through the jump connection, and finally generates a pixel-level tampering probability heat map. The Dice Loss is optimized for small targets, and the detection accuracy of subtle tampering areas is improved In some application scenarios, such as modifying a person's photo using deep fake technology in fake news, how to distinguish true and false.

[0151] Through the embodiment provided by the application, the news picture to be detected is input, and the spatial domain edge feature (such as a face contour) and the frequency domain feature (such as a background compression trace) are extracted. After dynamically fusing the features, the model judges whether there is splicing or generative tampering.

[0152] Experimental results: accuracy > 92%, which can detect false pictures generated by FaceApp, GAN and other tools.

[0153] Figure 2 is the second flowchart of the image authenticity determination method provided by the application, the original image is obtained, and the original image is subjected to noise fingerprint extraction processing to obtain a noise fingerprint. The spatial domain feature is extracted through the spatial feature extraction module according to the noise fingerprint and the original image.

[0154] The frequency domain feature extraction module is used to extract the frequency domain feature of the original image, and the statistical feature extraction module is used to extract the statistical feature of the original image.

[0155] The spatial domain feature, the frequency domain feature and the statistical feature are fused through the adaptive weight fusion mechanism to obtain a multi-modal feature. The multi-modal feature is input into an image prediction model for prediction processing, and a tampering probability value of the original image is output.

[0156] The application can also input the multi-modal feature into a positioning prediction model to predict the tampering probability, and determine a tampering area heat map according to the tampering probability.

[0157] The application can achieve at least the following technical effects through the above embodiments: By integrating spatial domain, frequency domain and statistical features, more tampering means are covered, and the comprehensiveness of detection is improved.

[0158] Adaptive weight learning: using attention mechanism to dynamically adjust the weight of each feature source, improving the adaptability of the model to different image types.

[0159] Adversarial robustness: through adversarial training and feature compression technology, effectively identify adversarial samples, improve model security.

[0160] Multi-task output: simultaneously realize true and false classification and region positioning, provide visual basis for review, and improve artificial review efficiency.

[0161] The image authenticity determination device provided by the application will be described below. The image authenticity determination device described below can be referred to in conjunction with the image authenticity determination method described above.

[0162] As shown in Figure 3 The structure diagram of the image authenticity determination device provided by the application is shown. An image authenticity determination device comprises the following modules: The acquisition module 310 is used for acquiring an original image. The feature extraction module 320 is used for performing feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions. The feature fusion module 330 is used for performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights. The prediction module 340 is used for inputting the multi-modal features into a trained image prediction model for prediction processing and outputting a tampering probability value of tampering the original image. The image prediction model is trained based on image feature samples and adversarial image samples. The adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training. The tampering probability value is used to determine whether the original image is tampered.

[0163] Preferably, the image authenticity determination device provided by the application is specifically used for extracting noise fingerprints of the original image, and the noise fingerprints are generated based on random noise in the original image. The original image and the noise fingerprint are input into a composite scaling neural network for feature extraction to obtain spatial domain features of spatial dimensions. The structure of the composite scaling neural network is to adopt a coordinate attention mechanism, and a light-weight activation function is used and the original classification head is removed; the coordinate attention mechanism is used to split the spatial information of an image input into the composite scaling neural network into a height direction and a width direction, and model the height direction and the width direction respectively.

[0164] Preferably, the image authenticity determination apparatus provided by the present application is specifically used for format conversion on the original image, and extraction of a luminance channel image of the original image after format conversion; Segmentation processing is performed on the luminance channel image to obtain a plurality of pixel blocks; Discrete cosine transform processing is performed on each pixel block to obtain a discrete transform coefficient matrix, and high-frequency alternating current coefficients of a high-frequency region are extracted from the discrete transform coefficient matrix; Convolution dimension reduction processing is performed on the high-frequency alternating current coefficients to obtain a dimension reduction feature of each pixel block, and global average pooling processing is performed on the dimension reduction feature of each pixel block to obtain a frequency domain feature of a frequency domain dimension.

[0165] Preferably, the image authenticity determination apparatus provided by the present application is specifically used for image grayscale processing on the original image, and normalization processing on the original image after image grayscale processing to obtain a normalized image; A gradient amplitude and direction matrix of each pixel in the normalized image is calculated, the normalized image is segmented into preset pixel cell units, and a gradient direction histogram of each cell unit is calculated according to the gradient amplitude and direction matrix of each pixel; The cell units are combined to obtain image blocks, and the gradient direction histograms of all cell units in each image block are concatenated to form a feature vector corresponding to each image block; The feature vector of each image block is normalized to obtain a normalized feature of each image block; All normalized features of the image blocks are extracted by traversing the normalized image through a sliding window, and the normalized features of all image blocks are concatenated to generate a statistical feature of a statistical dimension.

[0166] Preferably, the image authenticity determination apparatus provided by the present application is specifically used for taking the spatial domain feature as a query vector, the frequency domain feature as a key vector, and the statistical feature as a value vector to calculate a first attention weight; The frequency domain feature is taken as a query vector, the spatial domain feature is taken as a key vector, and the statistical feature is taken as a value vector to calculate a second attention weight; The statistical feature is taken as a query vector, the spatial domain feature is taken as a key vector, and the frequency domain feature is taken as a value vector to calculate a third attention weight; Based on the first attention weight, the second attention weight and the third attention weight, the spatial domain feature, the frequency domain feature and the statistical feature are fused and calculated to obtain the multi-modal feature.

[0167] Preferably, the image authenticity determination device provided by the application is specifically used for determining that the original image is a tampered image when the tampering probability value of the original image is greater than or equal to a preset threshold value. In the case where the tampering probability value of the original image is less than the preset threshold value, the original image is determined to be a real image. Preferably, the image authenticity determination device provided by the application is specifically used for performing up-sampling processing on the multi-modal feature of the tampered image to generate a heat map of the tampered region.

[0168] The image authenticity determination method, device, equipment, medium and program product provided by the application obtain an original image, perform feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions, perform feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights, input the multi-modal features into a trained image prediction model for prediction processing, and output a tampering probability value of the original image, wherein the image prediction model is obtained by training based on image feature samples and adversarial image samples, the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training, and the tampering probability value is used to determine whether the original image is tampered with. To solve the defect of low accuracy of image authenticity discrimination in the prior art, the information of three dimensions of the spatial domain, the frequency domain and the statistical features of the original image is effectively integrated, and then the probability value of the original image tampering is predicted based on feature fusion, so that the authenticity of the original image is determined according to the probability value, and the accuracy of authenticity discrimination is greatly improved.

[0169] Figure 4 An example of an entity structure diagram of an electronic device is shown as Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can invoke the logic instructions in the memory 430 to execute an image authenticity determination method, which includes: obtaining an original image; performing feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions; performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; inputting the multi-modal features into a trained image prediction model for prediction processing to output a tampering probability value of the original image being tampered with; wherein the image prediction model is obtained by training based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0170] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0171] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable the computer to perform the image authenticity determination method provided by the above-mentioned methods, which comprises: obtaining an original image; performing feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions; performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; inputting the multi-modal features into a trained image prediction model for prediction processing to output a tampering probability value of the original image; wherein the image prediction model is obtained by training based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0172] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the image authenticity determination method provided by the above-mentioned methods, which comprises: obtaining an original image; performing feature extraction processing on the original image from different dimensions to obtain image features corresponding to the dimensions; performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multi-modal features with dynamically adjusted weights; inputting the multi-modal features into a trained image prediction model for prediction processing to output a tampering probability value of the original image; wherein the image prediction model is obtained by training based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model and are used to optimize the image prediction model parameters through adversarial training; and the tampering probability value is used to determine whether the original image is tampered with.

[0173] The above-mentioned apparatus embodiments are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0174] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0175] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining the authenticity of an image, characterized in that: include: Get the original image; Performing feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions; Performing feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multimodal features with dynamically adjusted weights; Inputting the multimodal features into a trained image prediction model for prediction processing, and outputting a tampering probability value of tampering with the original image; In which, the image prediction model is obtained by training based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model, and are used to optimize the image prediction model parameters through adversarial training; the tampering probability value is used to determine whether the original image is tampered with.

2. The image authenticity determination method according to claim 1, characterized in that: The performing feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions includes: Extracting a noise fingerprint of the original image, where the noise fingerprint is generated based on random noise in the original image; Inputting the original image and the noise fingerprint into a composite scaling neural network for feature extraction to obtain spatial domain features of spatial dimensions; Among them, the structure of the compound scaling neural network adopts a coordinate attention mechanism, uses a lightweight activation function, and removes the original classification head; the coordinate attention mechanism is used to split the spatial information of the image input into the compound scaling neural network into height direction and width direction, and model them separately in height direction and width direction.

3. The image authenticity determination method according to claim 2, characterized in that: The extracting features of the original image from different dimensions to obtain image features of corresponding dimensions further includes: Performing format conversion on the original image, and extracting a brightness channel image of the original image after the format conversion; Segmenting the brightness channel image to obtain a plurality of pixel blocks; Performing discrete cosine transform processing on each pixel block to obtain a discrete transform coefficient matrix, and extracting high-frequency AC coefficients in a high-frequency region from the discrete transform coefficient matrix; The high-frequency AC coefficients are subjected to convolution dimensionality reduction processing to obtain dimensionality reduction features of each pixel block, and the dimensionality reduction features of each pixel block are subjected to global average pooling processing to obtain frequency domain features of frequency domain dimension.

4. The image authenticity determination method according to claim 3, characterized in that: The extracting features of the original image from different dimensions to obtain image features of corresponding dimensions further includes: Performing image grayscale processing on the original image, and normalizing the grayscaled original image to obtain a normalized image; Calculating the gradient magnitude and direction matrix of each pixel in the normalized image, dividing the normalized image into cell units of preset pixels, and calculating the gradient direction histogram of each cell unit according to the gradient magnitude and direction matrix of each pixel; Combining the cell units to obtain image blocks, and concatenating the gradient direction histograms of all cell units in each image block to form a feature vector corresponding to each image block; Normalize the feature vector of each image block to obtain the normalized features of each image block; The normalized image is traversed through a sliding window to extract normalized features of all image blocks, and the normalized features of all image blocks are connected in series to generate statistical features of a statistical dimension.

5. The image authenticity determination method according to claim 4, characterized in that: The method of performing feature fusion processing on the image features extracted from different dimensions based on the adaptive weight fusion mechanism to obtain multimodal features with dynamically adjusted weights includes: The spatial domain feature is used as a query vector, the frequency domain feature is used as a key vector, and the statistical feature is used as a value vector to calculate a first attention weight; The frequency domain feature is used as a query vector, the spatial domain feature is used as a key vector, and the statistical feature is used as a value vector to calculate a second attention weight; The statistical feature is used as a query vector, the spatial domain feature is used as a key vector, and the frequency domain feature is used as a value vector to calculate a third attention weight; Based on the first attention weight, the second attention weight and the third attention weight, the spatial domain features, the frequency domain features and the statistical features are fused and calculated to obtain the multimodal features.

6. The image authenticity determination method according to claim 1, characterized in that: Determining whether the original image is a tampered image includes: If the tampering probability value of the original image is greater than or equal to a preset threshold, determining that the original image is a tampered image; When the tampering probability value of the original image is less than a preset threshold, determining that the original image is a genuine image; After the step of determining that the original image is a tampered image, the method includes: Upsampling is performed on the multimodal features of the tampered image to generate a heat map of the tampered area.

7. An image authenticity determination device, characterized in that: include: An acquisition module is used to acquire the original image; A feature extraction module is used to perform feature extraction processing on the original image from different dimensions to obtain image features of corresponding dimensions; A feature fusion module is used to perform feature fusion processing on the image features extracted from different dimensions based on an adaptive weight fusion mechanism to obtain multimodal features with dynamically adjusted weights; A prediction module, configured to input the multimodal features into a trained image prediction model for prediction processing, and output a tampering probability value of tampering with the original image; In which, the image prediction model is obtained by training based on image feature samples and adversarial image samples; the adversarial image samples are generated based on a text-to-image generation model, and are used to optimize the image prediction model parameters through adversarial training; the tampering probability value is used to determine whether the original image is tampered with.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the image authenticity determination method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image authenticity determination method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image authenticity determination method according to any one of claims 1 to 6 is implemented.

Images