Artifact enhancement fusion module for face image forgery detection

Abstract

The invention provides an artifact enhancement fusion module for face image forgery detection, and aims to solve the problems that the number of real samples in a face forgery detection scene is limited, artifact information is difficult to effectively fuse, and the generalization ability of a model is influenced. According to the method, a brand-new iron tower mixing mode and fusion operation with high-frequency artifacts are provided. Specifically, face positioning segmentation is carried out on a data set D to obtain a sample set S, and then face centers of samples in the sample set S are calculated and aligned; randomly selecting n samples as a group, and aligning a face center by using affine transformation in the group; a neural network capable of learning parameters is trained and fused with an enhanced sample EG obtained by an iron tower mixing module and an airspace high-frequency representation # imgabs0 # obtained by a high-frequency extraction module to finally obtain complementation artifact information FEG, and the complementation artifact information FEG is input into a discriminator to be trained. The module can be flexibly applied to each detection model to enhance the artifact perception generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of face image forgery detection, and solves the problem of limited number of real samples and difficulty in effectively fusing artifact information in face forgery detection, thereby affecting the generalization ability of the model, by enhancing data set features and fusing high-frequency artifacts. Background Art

[0002] In recent years, with the rapid development of deep learning technology, a series of revolutionary generative artificial intelligence models such as generative adversarial networks (GANs) and diffusion models have emerged, which have achieved remarkable achievements in the field of image generation. These advanced algorithms have greatly promoted the progress of image generation technology, and the quality of the generated images has reached an unprecedented level of realism.

[0003] However, with the popularity and abuse of generative artificial intelligence technology, a series of security issues have gradually surfaced. The emergence of deep fake content, especially the proliferation of face forgery technology, has made the spread of false information more difficult to detect, which not only poses a threat to personal privacy, but also poses a huge threat to the trust system of society. In this case, the research of face forgery detection technology as a countermeasure is imminent and has quickly become a hot topic in the industry.

[0004] In recent years, the industry has proposed a number of face forgery detection methods, but these methods generally have the problem of weak model generalization ability. The root cause of these problems is that the model is difficult to effectively capture the subtle artifacts left by forgeries, especially when processing cross-dataset samples. The model detection ability is significantly reduced, which limits its effectiveness and reliability in practical applications. In the current research on face forgery detection, most augmentation methods still use common technical means in other tasks, such as cutting, copying and pasting. However, these methods often ignore the unique features of face forgery. Therefore, relying solely on traditional augmentation methods may not fully explore and utilize the unique information in the field of face forgery. Summary of the invention

[0005] In view of the limitations and challenges of existing solutions, the present invention proposes an artifact enhancement and fusion module for face image forgery detection, which is aimed at the problem that the number of real samples in face forgery detection is limited and the artifact information is difficult to be effectively integrated. This method proposes a new iron tower hybrid mode and data set feature enhancement and high-frequency artifact fusion operations, which can be easily transplanted to any model; by segmenting the samples in the data set D, a face sample set S is formed, and the face center in the sample set S is calculated according to the feature points, and the n samples randomly sampled in the group are aligned by affine transformation; the same sample image first extracts high-frequency artifact information in the frequency domain, and then enhances the common forgery areas such as eyes, nose, and mouth in the spatial domain to obtain enhanced samples; the two are input into the neural network to train the learning parameter ω, so as to fully integrate and obtain the complementary artifact information for model training. Specifically, this method proposes a new iron tower hybrid mode and data set feature enhancement and high-frequency artifact fusion operations, combines high-frequency artifact information and cross-sample face features to fully integrate to obtain complementary artifact information FEG, and finally inputs it into the discriminator training to achieve the purpose of fully integrating information. This method better solves the problem of limited number of real samples and difficulty in effectively integrating artifact information in face forgery detection, and realizes the use of a dataset that fully integrates artifact information and real sample information to train the discriminator model to improve generalization ability.

[0006] An artifact enhancement fusion module for face image forgery detection of the present invention comprises the following steps:

[0007] S1: Locate the face of the samples in the dataset D to obtain the face sample set S; locate the face center of each image;

[0008] S2: Randomly sample n samples into a group, and use affine transformation to align the face center within the group to obtain the affine sample set G;

[0009] S3: GI of face sample images in each group i (x,y) converted to frequency domain GF i (u,v); then map it back to the spatial domain to get the high-frequency representation of the spatial domain

[0010] S4: target area mixing: in each of the n groups obtained by sampling, the three most common forgery replacement areas, namely eyes, nose and mouth, are mixed to obtain the enhanced sample set EG;

[0011] S5: Enhance the image EG and the corresponding high-frequency information GI high The input is input into the neural network to learn a parameter ω, and finally the two are fused to obtain the completed artifact information FEG which is input into the discriminator for training.

[0012] Furthermore, the specific steps of S1 are:

[0013] S11: For each sample D in a given dataset D i Input into the pre-trained deep neural network to detect and locate the face area, and the network outputs the corresponding coordinates and offsets:

[0014]

[0015] Where i∈m, m is the sample capacity of the original sample set D, DNN is a pre-trained neural network, are the x and y coordinates of the upper left corner of the face region anchor box, while w and h correspond to the width and height of the anchor box respectively;

[0016] S12: Cut the original sample d according to the output coordinates and adjust it to a fixed size of 224×224 to obtain the localized and segmented sample s. Repeat S11 for all samples in the dataset D to obtain the face sample set S:

[0017]

[0018] Where S is the sample set obtained after cutting;

[0019] S13: Input the sample set S obtained by segmentation in S12 into the face key point detection network to obtain each S i Key point information FLM i ;

[0020] S14: Calculate the center point of the eye; according to the obtained sample S i Key point information obtains the eye corner coordinates of the left and right eyes:

[0021]

[0022] Where i∈m, m is the sample capacity of the face sample set S, FLM i is the key point information obtained by the face key point detection network for the i-th sample image, FLM i (left_eye) represents the coordinate array of the left eye corner in the image, and FLM i (right_eye) represents the coordinate array of the right eye corner in the image, and then gets the horizontal and vertical coordinates according to the index, 0 represents the horizontal coordinate and 1 represents the vertical coordinate;

[0023] Calculate the average values ​​of the horizontal and vertical coordinates of the two eye corners as the eye distance center of the image:

[0024] (eye_center x , eye_center y )=((FLM i (left_eye)[0]+FLM i(right_eye)[0]) / 2, (FLM i (left_eye)[1]+FLM i (right_eye)[1]) / 2);

[0025] S15: Detect the nose tip; according to the obtained sample S i Key point information to obtain the coordinates of the nose tip:

[0026] (nose x , nose y )=(FLM i (nose)[0],FLM i (nose)[1]);

[0027] Among them FLM i (nose) represents the coordinate array of the nose tip in the image, and then the horizontal and vertical coordinates are obtained according to the index, 0 represents the horizontal coordinate and 1 represents the vertical coordinate;

[0028] S16: Calculate the face center; calculate the average value of the eye distance center and nose tip coordinates of each image sample obtained in S14 and S15 to obtain the face center of the sample:

[0029] (facial_center x , facial_center y )=((eye_center x +nose x ) / 2,(eye_center y +nose y ) / 2);

[0030] Where facial_center represents the coordinates of the face center of each picture, and the affine transformation of the image is calculated based on the face center.

[0031] Furthermore, the specific steps of S2 are:

[0032] S21: Randomly sample n samples from the face sample set S as a group S n ;

[0033] S22: Get an image of size 224×224 whose center point is located at:

[0034] (x target ,y target ) = (112, 112);

[0035] Calculate each sample image S separately n iThe offset between the face center and the target center:

[0036]

[0037] Where i∈n(n<m), n is the sampling sample size; Respectively represent the horizontal and vertical coordinates of the center point of the face of the i-th image in the n sampling samples;

[0038] S23: Construct affine transformation matrix:

[0039]

[0040] Among them, s is the scaling factor determined according to the model sample image size, Δx and Δy are the offsets of the i-th sample;

[0041] S24: Apply affine transformation to n sampled samples to obtain a set of face data samples with a fixed size, where the center of the image is the center of the face:

[0042]

[0043] Where G is the sample set after affine transformation of n samples, M i is the affine transformation matrix of the i-th sample, a·b is the multiplication of the a matrix and the b matrix, processing each pixel, applying the affine transformation matrix to the color channel of each pixel to obtain the transformed image.

[0044] Furthermore, the specific steps of S3 are:

[0045] S31: Use Fourier transform to transform each image in the affine sample set into the face sample image GI i (x, y) is converted to frequency domain GF i (u, v):

[0046]

[0047] in represents Fourier transform, (u, v) is the frequency domain coordinate, and i is the imaginary unit;

[0048] S32: In forgery detection, the frequency domain can well reflect the artifact details. By extracting high-frequency frequency domain information to represent the artifact details in the image, the original sample is enhanced:

[0049] FH high (u, v) = (1-FH low (u, v));

[0050] Among them, FH low(u, v) is a low-pass filter used to retain low-frequency information; a Gaussian filter is used as a low-pass filter in the model:

[0051]

[0052] Where D(u, v) calculates the distance to the center of the frequency domain; thus, the high-frequency component F high (u, v) is the artifact detail of the forged image in the frequency domain:

[0053] F high (u, v) = GF(u, v)⊙FH high (u, v);

[0054] The a⊙b operation represents the matrix dot multiplication operation;

[0055] S33: Convert high-frequency information back to the spatial domain; prepare for the next step of adding it to the spatial domain image and fusing the information of the two, using inverse Fourier transform:

[0056]

[0057] in stands for Inverse Fourier Transform; GI high (x, y) represents the representation of high-frequency information in the frequency domain in the spatial domain.

[0058] Furthermore, the specific steps of S4 are:

[0059] S41: The so-called iron tower mixing is to divide the face sample into four parts, each part contains the key areas of the face: left eye, right eye, nose and mouth; according to the distribution of forged data, different forgery methods tend to tamper with the key areas of mouth, nose and eyes, and the method of the present invention obtains enhanced samples mixed with different forgery methods by cross-replacing the three key areas, and at the same time masks the nose area in proportion to prevent the model from focusing too much on the central area and resulting in weak generalization ability;

[0060] According to the face center point calculated by S1, the rays connected to the midpoint of the upper border of the image, the coordinate point of the lower left corner of the image, and the coordinate point of the lower right corner of the image are introduced to divide the image into three parts, which are recorded as:

[0061] Operations={left_eye, rigth_eye, nose_mouth};

[0062] Which corresponds to the sample G i The three operations in can be defined as:

[0063]

[0064] Among them G i is the ith sample among the n samples that have undergone affine transformation, and lefe_eye, right_eye, and nose_mouth are the three most common areas for forgery replacement, namely, eye, nose, and mouth, respectively;

[0065] S42: In-group replacement; in a set of samples {G 1 , G 2 , …, G n} is divided into two parts according to the authenticity of the sample; the angles of the three partitions of the image are ρ left_eye , ρ right_eye , ρ nose_mouth , construct a two-dimensional coordinate system with the center of the image as the origin, and set the arc interval corresponding to the first quadrant to The arc interval corresponding to the second quadrant is The arc interval corresponding to the third quadrant is The arc interval corresponding to the fourth quadrant is According to the partitioning strategy, the arc intervals are:

[0066]

[0067] Where a ∪b represents the union of set a and set b;

[0068] For any pixel in the image, the angle between the ray connecting it with the center point of the image, i.e. the origin of the coordinate system, and the positive direction of the y-axis is the basis for determining which candidate region the current pixel is in:

[0069]

[0070] Among them G i (x, y) represents a sample image G in the affine sample set G i , and (x, y) represents the corresponding pixel coordinates in the two-dimensional rectangular coordinate system. Represents the sample image G i The arc matrix consists of the arcs corresponding to each pixel in the image; arctan2(y, x) represents the arc between the line from the center of the image to the pixel (x, y) and the positive direction of the x-axis, ranging from (-π, π] radians;

[0071] For the sample image G a And the sample image G b , randomly select one of the three regions to exchange and get two new samples:

[0072]

[0073]

[0074] Where Replace(G a , G b ,Operations i ) represents randomly extracting two samples a and b from the affine sample set G and performing a replacement operation in the i-th region to extract G a The content of the corresponding area of ​​the image is replaced by G b The pixel content of the corresponding area of ​​the image; the γ·θ operation means that it is γ itself when θ is true, otherwise it is 0; The operation represents determining whether the radian value of a certain coordinate in the radian matrix is ​​in the radian value interval corresponding to the i-th interval; Any pixel point (x, y) in the representative formula that belongs to the image G must satisfy the Replace operation to form a new sample image; In summary, for the above calculation of the new sample image G ab The formula can be parsed as a The pixel value and sample image G in the operation arc value interval are obtained by masking b The pixel values ​​of the non-operation arc value interval obtained by the mask in the middle are spliced ​​by element-by-element addition operation of the two calculation matrices to obtain a new sample image. For the new sample image G ab Similarly;

[0075] There are three strategies for label mixing of images:

[0076] True + True, True + False, False + False;

[0077] From this, we can generate a new mixed sample image with the label:

[0078] label ab =1-(1-lable a )(1-lable b );

[0079] The label value is 1 for false and 0 for true;

[0080] The new sample G obtained by replacement operation ab , G ba Add to the original affine sample set G and delete the original sample G from the set G a , G b , and obtain a new enhanced sample set G′:

[0081] G′=G∪{G ab , G ba}\{G a , G b};

[0082] A∪B represents the union operation of set A and set B, and A\B represents the difference operation of set A and set B, eliminating the same elements;

[0083] Repeat S42 operation k times, randomly ε times for each group, and obtain n enhanced samples each time. Finally, merge the ε results to obtain the enhanced sample set G for each group. ε′ :

[0084]

[0085] In order to obtain the enhanced sample set G ε′ Fully integrate face sample information without introducing a large number of redundant operations to reduce performance, defined in the model:

[0086] k=n*log 2 n;

[0087] ε=3;

[0088] where log 2 n represents the logarithm operation with base 2 on n, n is the sample capacity of the original affine sample set G, and ε is the number of times the tower mixing process is repeated;

[0089] S43: Enhanced masking; for the third operation area The key point information calculated by S1 is divided into two areas: nose and mouth. The segmentation strategy uses the midpoint between the center of the upper lip and the tip of the nose:

[0090]

[0091] Among them, FLM is the key point information set, Uplip_center is the center point of the upper lip, nose is the tip of the nose, and the coordinate array is obtained, and then the coordinate information is obtained according to 0 as the horizontal coordinate and 1 as the vertical coordinate; the midpoint of the two points is taken as the segmentation point:

[0092] (separation_point x , separation_point y )=((FLM i (Uplip_center)[0]+nose x ) / 2,(FLM i (Uplip_center)[1]+nose y ) / 2);

[0093] Draw a ray parallel to the x-axis through the midpoint and intersect the third operation area G i nose_mouth At two points, the original operation area is divided into:

[0094]

[0095] in Represents the i-th enhanced sample G ε′ i Operations = any (x, y) pixel point in nose_mouth; if (y < separation_point y ) means that when the y coordinate value of the pixel point is less than the coordinate value of the segmentation point, it is the nose area, otherwise it is the mouth area;

[0096] Traverse the enhanced sample set G ε′ For each sample, proportional enhancement and random masking operations are performed to obtain the final sample data set; the random enhancement ratio is set to μ∈[0,1), when 0≤μ<0.5, the nose area is replaced, when 0.5≤μ<0.7, no operation is performed, and when 0.7≤μ<1, the nose area is masked:

[0097]

[0098] Among them, Enhance(G ε′ a , G ε′ b , μ) represent the enhanced image a, enhanced image b and enhancement ratio μ respectively; different enhancement schemes are adopted according to different enhancement ratios;

[0099]

[0100] Where Replace(G ε′ a , G ε′ b , nose) represents the enhanced sample set G ε′ Randomly select two samples G from ε′ a and G ε′ b Perform a replacement operation in the nose part separated from the third operation area and extract G ε′ a The content of the corresponding area of ​​the image is replaced by G ε′ b The pixel content of the corresponding area of ​​the image; repeat the operation to extract G e′ b The content of the corresponding area of ​​the image is replaced by G ε′ a The pixel contents of the corresponding regions of the image constitute two new sample images; the γ·θ operation represents γ itself when the γ element is within the range of θ, otherwise it is 0;! G a ′noseRepresents the non-nose area in the image; In summary, the above formula divides the face nose area of ​​the enhanced image a and the non-face nose area of ​​the enhanced image b through element masking operation, and adds the two matrices element by element to obtain a new enhanced image, and vice versa;

[0101]

[0102] Among them, Blending (G ε′ i ) represents the i-th enhanced sample G e′ i Perform masking operation; for any pixel value, if it does not belong to the nose area, it remains unchanged, and the pixel value belonging to the nose area is set to 0, that is, the nose area is completely black;

[0103] For the original enhanced sample set {G ε′ 1 , G ε′ 2 , ..., G ε′ ε*n After the enhanced masking operation, the new enhanced sample set is recorded as EG:

[0104] EG=G ε′ ∪Enhence(G ε′ i , G ε′ i+1 , μ i )\{G ε′ i , G ε′ i+1}, i = o, 2, 4...(ε*n-2);

[0105] Where A∪B represents the union operation of set A and set B, A\B represents the difference operation of set A and set B, and the same elements are eliminated; the value of i starts from 0 and increases by 2 each time, ε*n represents the number of ε operations, and n image samples are obtained each time; G e′ i , G ε′ i+1 The two enhanced samples are input into the enhanced masking module to obtain new enhanced samples.

[0106] Furthermore, the specific steps of S5 are:

[0107] S51: Construct a feedforward neural network with mean square error loss MSE as the loss function, and use the fastest gradient descent as the optimization algorithm to back-propagate and update the fusion parameter ω:

[0108] ω=f NN (EG, GIhigh );

[0109] where f NN (.) represents the learning process of the neural network, and the fusion parameter ω is obtained to balance the contribution between the two; the fusion enhanced image EG and the corresponding spatial high-frequency representation GI high :

[0110] FEG=EG+ω·GI high ;

[0111] Where ω·GI high The operation represents the multiplication of the elements in the matrix with the weight coefficient ω, and FEG is the completion artifact information;

[0112] S52: Construct the network back propagation process using mean square error loss as the loss function:

[0113]

[0114] Where M is the number of pixels in each image in the enhanced sample set EG; minimize the loss function and back-propagate the update parameters ω through the fastest gradient descent method:

[0115]

[0116] where ω k is the parameter value at step k, α k is the learning rate of the kth step, and L(ω k ) is the loss function L in ω k The learning rate in the model can be updated according to the gradient of each step, making the model more and more sensitive to the parameter ω. The learning rate α of the kth step k :

[0117]

[0118] in Represents the gradient The transpose of Represents matrix multiplication;

[0119] S53: Train a feedforward neural network to fully fuse the enhanced image EG and the corresponding spatial high-frequency representation GI high The final completion artifact information FEG is input into the discriminator to train the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 A text flow chart of an artifact enhancement fusion module for face image forgery detection provided by the present invention;

[0121] Figure 2The algorithm model block diagram provided by the present invention;

[0122] Figure 3 A schematic diagram of the process flow of the iron tower hybrid module provided by the present invention;

[0123] Figure 4 A schematic diagram of the high frequency extraction process provided by the present invention. DETAILED DESCRIPTION

[0124] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described below in conjunction with examples and drawings.

[0125] As attached Figure 2 As shown, an artifact enhancement fusion module for face image forgery detection includes the following steps:

[0126] S1: For the dataset D which is a mixture of different datasets, locate the face for each sample and segment it to obtain the face sample set S. For the videos in the sample, take the interval key frame to capture the video screen and convert it into image processing; locate the face center of each face image in the face sample set S to prepare for the next affine transformation;

[0127] S2: randomly select n samples from the face sample set S as a group, construct an affine transformation matrix M according to the offset and scaling factor between the samples in the group and the target center, and unify the image size to a fixed size of 224×224 to speed up the calculation efficiency; align the face center of each sample in the group using affine transformation, introduce this step to reduce the degree of large segmentation of the tower hybrid module proposed in the present invention at the segmentation boundary, and finally merge each group of samples to obtain the affine sample set G;

[0128] S3: GI of face sample images in each group i (x, y) is converted to frequency domain GF i (u, v); then map back to the spatial domain to obtain the high-frequency representation of the spatial domain

[0129] S4: target area mixing: in each of the n groups obtained by sampling, the three most common forgery replacement areas, namely eyes, nose and mouth, are mixed to obtain the enhanced sample set EG;

[0130] S5: Enhance the image EG and the corresponding high-frequency information GI high The input is input into the neural network to learn a parameter ω, and finally the two are fused to obtain the completed artifact information FEG which is input into the discriminator for training.

[0131] As attached Figure 3 As shown, the high frequency extraction module proposed in the present invention comprises the following steps:

[0132] S31: Use Fourier transform to transform each image in the affine sample set into the face sample image GI i (x, y) is converted to frequency domain GF i (u, v):

[0133]

[0134] in represents Fourier transform, (u, v) is the frequency domain coordinate, and i is the imaginary unit;

[0135] S32: Extract high frequency domain information:

[0136] FH high (u, v) = (1-FH low (u, v));

[0137] Among them, FH low (u, v) is a low-pass filter used to retain low-frequency information; a Gaussian filter is used as a low-pass filter in the model:

[0138]

[0139] Where D(u, v) is the distance to the center of the frequency domain; from this we can get the high-frequency component F high (u, v):

[0140] F high (u, v) = GF(u, v)⊙FH high (u, v);

[0141] The a ⊙b operation represents a matrix dot multiplication operation;

[0142] S33: Convert high-frequency information back to the spatial domain; prepare for the next step of adding it to the spatial domain image and fusing the information of the two, using inverse Fourier transform:

[0143]

[0144] in stands for Inverse Fourier Transform; GI high (x, y) represents the representation of high-frequency information in the frequency domain in the spatial domain.

[0145] As attached Figure 4 As shown, the iron tower hybrid module proposed by the present invention comprises the following steps:

[0146] S41: Based on the face center point calculated in S1, introduce rays to the midpoint of the upper border of the image, the coordinate point of the lower left corner of the image, and the coordinate point of the lower right corner of the image, and divide the image into three parts, which are recorded as:

[0147] Operations={left_eye, rig th_eye, nose_mouth};

[0148] Which corresponds to the sample G i The three operations in can be defined as:

[0149]

[0150] Among them G i is the ith sample among the n samples that have undergone affine transformation, lefe_eye, right_eye, nose_mouth are the three most common areas for forgery replacement, namely, eye, nose and mouth;

[0151] S42: In-group replacement; in a set of samples {G 1 , G 2 , ..., G n} is divided into two parts according to the authenticity of the sample; the angles of the three partitions of the image are ρ left_eye , ρ right_eye , ρ nose_mouth , construct a two-dimensional coordinate system with the center of the image as the origin, and set the arc interval corresponding to the first quadrant to The arc interval corresponding to the second quadrant is The arc interval corresponding to the third quadrant is The arc interval corresponding to the fourth quadrant is According to the partitioning strategy, the arc intervals are:

[0152]

[0153] Where a ∪b represents the union of set a and set b;

[0154] For any pixel in the image, the angle between the ray connecting it with the center point of the image, i.e. the origin of the coordinate system, and the positive direction of the y-axis is the basis for determining which candidate region the current pixel is in:

[0155]

[0156] Among them G i (x, y) represents a sample image G in the affine sample set G i , and (x, y) represents the corresponding pixel coordinates in the two-dimensional rectangular coordinate system. Represents the sample image G i The arc matrix consists of the arcs corresponding to each pixel in the image; arctan2(y, x) represents the arc between the line from the center of the image to the pixel (x, y) and the positive direction of the x-axis, ranging from (-π, π] radians;

[0157] For the sample image G a And the sample image G b , randomly select one of the three regions to exchange and get two new samples:

[0158]

[0159]

[0160] Where Replace(G a , G b ,Operations i ) represents randomly extracting two samples a and b from the affine sample set G and performing a replacement operation in the i-th region to extract G a The content of the corresponding area of ​​the image is replaced by G b The pixel content of the corresponding area of ​​the image; the γ·θ operation means that it is γ itself when θ is true, otherwise it is 0; The operation represents determining whether the radian value of a certain coordinate in the radian matrix is ​​in the radian value interval corresponding to the i-th interval; Any pixel point (x, y) in the representative formula that belongs to the image G must satisfy the Replace operation to form a new sample image; In summary, for the above calculation of the new sample image G ab The formula can be parsed as a The pixel value and sample image G in the operation arc value interval are obtained by masking b The pixel values ​​of the non-operation arc value interval obtained by the mask in the middle are spliced ​​by element-by-element addition operation of the two calculation matrices to obtain a new sample image. For the new sample image G ab Similarly;

[0161] There are three strategies for label mixing of images:

[0162] True + True, True + False, False + False;

[0163] From this, we can generate a new mixed sample image with the label:

[0164] label ab =1-(1-lable a )(1-lable b );

[0165] The label value is 1 for false and 0 for true;

[0166] The new sample G obtained by replacement operation ab , G ba Add to the original affine sample set G and delete the original sample G from the set G a , Gb , and obtain a new enhanced sample set G′:

[0167] G′=G∪{G ab , G ba}\{G a , G b};

[0168] A∪B represents the union operation of set A and set B, and A\B represents the difference operation of set A and set B, eliminating the same elements;

[0169] Repeat S42 operation k times, randomly ε times for each group, and obtain n enhanced samples each time. Finally, merge the ε results to obtain the enhanced sample set G for each group. ε′ :

[0170] S43: Enhanced masking; for the third operation area The key point information calculated by S1 is divided into two areas: nose and mouth. The segmentation strategy uses the midpoint between the center of the upper lip and the tip of the nose:

[0171]

[0172] Among them, FLM is the key point information set, Uplip_center is the center point of the upper lip, nose is the tip of the nose, and the coordinate array is obtained, and then the coordinate information is obtained according to 0 as the horizontal coordinate and 1 as the vertical coordinate; the midpoint of the two points is taken as the segmentation point:

[0173] (separation_point x , separation_point y )=((FLM i (Uplip_center)[0]+nose x ) / 2,(FLM i (Uplip_center)[1]+nose y ) / 2);

[0174] Draw a ray parallel to the x-axis through the midpoint and intersect the third operation area G i nose_mouth At two points, the original operation area is divided into:

[0175]

[0176] in Represents the i-th enhanced sample G ε′ iOperations = any (x, y) pixel point in nose_mouth; if (y < separation_point y ) means that when the y coordinate value of the pixel point is less than the coordinate value of the segmentation point, it is the nose area, otherwise it is the mouth area;

[0177] Traverse the enhanced sample set G ε′ For each sample, proportional enhancement and random masking operations are performed to obtain the final sample data set; the random enhancement ratio is set to μ∈[0,1), when 0≤μ<0.5, the nose area is replaced, when 0.5≤μ<0.7, no operation is performed, and when 0.7≤μ<1, the nose area is masked:

[0178]

[0179] Among them, Enhance(G ε′ a , G ε′ b , μ) represent the enhanced image a, enhanced image b and enhancement ratio μ respectively; different enhancement schemes are adopted according to different enhancement ratios;

[0180]

[0181] Where Replace(G ε′ a , G ε′ b , nose) represents the enhanced sample set G ε′ Randomly select two samples G from ε′ a and G ε′ b Perform a replacement operation in the nose part separated from the third operation area and extract G ε′ a The content of the corresponding area of ​​the image is replaced by G ε′ b The pixel content of the corresponding area of ​​the image; repeat the operation to extract G e′ b The content of the corresponding area of ​​the image is replaced by G ε′ a The pixel contents of the corresponding regions of the image constitute two new sample images; the γ·θ operation represents γ itself when the γ element is within the range of θ, otherwise it is 0;! G a ′nose Represents the non-nose area in the image; In summary, the above formula divides the face nose area of ​​the enhanced image a and the non-face nose area of ​​the enhanced image b through element masking operation, and adds the two matrices element by element to obtain a new enhanced image, and vice versa;

[0182]

[0183] Among them, Blending (G ε′ i ) represents the i-th enhanced sample G ε′ i Perform masking operation; for any pixel value, if it does not belong to the nose area, it remains unchanged, and the pixel value belonging to the nose area is set to 0, that is, the nose area is completely black;

[0184] For the original enhanced sample set {G ε′ 1 , G ε′ 2 , ..., G ε′ ε*n After the enhanced masking operation, the new enhanced sample set is recorded as EG:

[0185] EG=G ε′ ∪Enhence(G ε′ i , G ε′ i+1 , μ i )\{G ε′ i , G ε′ i+1}, i = 0, 2, 4...(ε*n-2);

[0186] Where A∪B represents the union operation of set A and set B, A\B represents the difference operation of set A and set B, and the same elements are eliminated; the value of i starts from 0 and increases by 2 each time, ε*n represents the number of ε operations, and n image samples are obtained each time; G ε′ i , G ε′ i+1 The two enhanced samples are input into the enhanced masking module to obtain new enhanced samples.

Claims

1. An artifact enhancement fusion module for face image forgery detection, characterized in that: The following steps are involved: S1: Locate the face of the samples in the dataset D to obtain the face sample set S; locate the face center of each image; S2: Randomly sample n samples into a group, and use affine transformation to align the face center within the group to obtain the affine sample set G; S3: high frequency extraction; for each group of face sample images GI i (x, y) is converted to frequency domain GF i (u, v); extract the high-frequency part in the frequency domain and then map it back to the spatial domain middle; S4: Tower mixing: In each of the n groups obtained by sampling, the three most common areas of forgery replacement, namely eyes, nose and mouth, are mixed to obtain the enhanced sample set EG; S5: Enhance the image EG and the corresponding high-frequency information GI high The input is input into the neural network to learn a parameter ω, and finally the two are fused to obtain the completed artifact information FEG which is input into the discriminator for training.

2. The artifact enhancement fusion module for face image forgery detection according to claim 1 is characterized in that The specific method of step S1 is: S11: For each sample D in a given dataset D i Input into the pre-trained deep neural network to detect and locate the face area, and the network outputs the corresponding coordinates and offsets: Where i ∈ m, m is the sample capacity of the original sample set D, DNN is a pre-trained neural network, are the x and y coordinates of the upper left corner of the face region anchor box, while w and h correspond to the width and height of the anchor box respectively; S12: Cut the original sample dd according to the output coordinates and resize it to 224×224 to obtain the localized and segmented sample s. Repeat S11 for all samples in the dataset D to obtain the face sample set S: S13: Input the sample set S obtained by segmentation in S12 into the face key point detection network to obtain each S i Key point information FLM i ; S14: Calculate the center point of the eye; according to the obtained sample S i Key point information obtains the eye corner coordinates of the left and right eyes: Where i ∈ m, m is the sample capacity of the face sample set S, FLM i is the key point information obtained by the face key point detection network for the i-th sample image, FLM i (left_eye) represents the coordinate array of the left eye corner in the image, and FLM i (right_eye) represents the coordinate array of the right eye corner in the image, and then gets the horizontal and vertical coordinates according to the index, 0 represents the horizontal coordinate and 1 represents the vertical coordinate; Calculate the average values ​​of the horizontal and vertical coordinates of the two eye corners as the eye distance center of the image: (eye_center x ,eye_centery)= ((FLM i (left_eye)[0]+FLM i (right_eye)[0]) / 2,(FLM i (left_eye)[1]+FLM i (right_eye)[1]) / 2); S15: Detect the nose tip; according to the obtained sample S i Key point information to obtain the coordinates of the nose tip: (nose x ,nose y )=(FLM i (nose)[0],FLM i (nose)[1]); Among them FLM i (nose) represents the coordinate array of the nose tip in the image, and then the horizontal and vertical coordinates are obtained according to the index; S16: Calculate the face center; calculate the average value of the eye distance center and nose tip coordinates of each image sample obtained in S14 and S15 to obtain the face center of the sample: (facial_center x ,facial_center y )=((eye_center x +nose x ) / 2,(eye_center y +nose y ) / 2); Where facial_center represents the center coordinates of the face in each picture.

3. The artifact enhancement fusion module for face image forgery detection according to claim 1 is characterized in that The specific method of step S2 is: S21: Randomly sample n samples from the face sample set S as a group S n ; S22: Get an image of size 224×224 whose center point is located at: (x target ,y target )=(112,112); get the offset: S23: Construct affine transformation matrix: Where s is the scaling factor, Δx and Δy are the offsets of the i-th sample; S24: Apply affine transformation to n sampled samples to obtain a set of face data samples with a fixed size, where the center of the image is the center of the face: Where G is the sample set after affine transformation of n samples, M i is the affine transformation matrix of the i-th sample, a·b is the multiplication of the a matrix and the b matrix, processing each pixel, applying the affine transformation matrix to the color channel of each pixel to obtain the transformed image.

4. The artifact enhancement fusion module for face image forgery detection according to claim 1, characterized in that The specific method of step S3 is: S31: Use Fourier transform to transform each image in the affine sample set into the face sample image GI i (x, y) is converted to frequency domain GF i (u, v): in represents Fourier transform, (u, v) is the frequency domain coordinate, and i is the imaginary unit; S32: Extract high frequency domain information: FH high (u,v)=(1-FH loW (u,v)); Among them, FH low (u, v) is a low-pass filter used to retain low-frequency information; a Gaussian filter is used as a low-pass filter in the model: Where D(u, v) is the distance to the center of the frequency domain; from this we can get the high-frequency component F high (u, v): F high (u,v)=GF(u,v)⊙FH high (u,v); The a ⊙b operation represents a matrix dot multiplication operation; S33: Convert high-frequency information back to the spatial domain; prepare for the next step of adding it to the spatial domain image and fusing the information of the two, using inverse Fourier transform: in stands for Inverse Fourier Transform; GI high (x, y) represents the representation of high-frequency information in the frequency domain in the spatial domain.

5. The artifact enhancement fusion module for face image forgery detection according to claim 1, characterized in that The specific method of step S4 is: S41: Based on the face center point calculated in S1, introduce rays to the midpoint of the upper border of the image, the coordinate point of the lower left corner of the image, and the coordinate point of the lower right corner of the image, and divide the image into three parts, which are recorded as: Operations={left_eye, rig th_eye, nose_mouth}; Which corresponds to the sample G i The three operations in can be defined as: Among them G i is the ith sample among the n samples that have undergone affine transformation, lefe_eye, right_eye, nose_mouth are the three most common areas for forgery replacement, namely, eye, nose and mouth; S42: In-group replacement; in a set of sample sets {G1, G2, ..., G n } is divided into two parts according to the authenticity of the sample; the angles of the three partitions of the image are ρ left_eye , ρ right_eye , ρ nose_mouth , construct a two-dimensional coordinate system with the center of the image as the origin, and set the arc interval corresponding to the first quadrant to The arc interval corresponding to the second quadrant is The arc interval corresponding to the third quadrant is The arc interval corresponding to the fourth quadrant is According to the partitioning strategy, the arc intervals are: Where a ∪b represents the union of set a and set b; For any pixel in the image, the angle between the ray connecting it with the center point of the image, i.e. the origin of the coordinate system, and the positive direction of the y-axis is the basis for determining which candidate region the current pixel is in: Among them G i (x, y) represents a sample image G in the affine sample set G i , and (x, y) represents the corresponding pixel coordinates in the two-dimensional rectangular coordinate system. Represents the sample image G i The arc matrix consists of the arcs corresponding to each pixel in the image; arctan2(y, x) represents the arc between the line from the center of the image to the pixel (x, y) and the positive direction of the x-axis, ranging from (-π, π] radians; For the sample image G a And the sample image G b , randomly select one of the three regions to exchange and get two new samples: Where Replace(G a , G b ,Operations i ) represents randomly extracting two samples a and b from the affine sample set G and performing a replacement operation in the i-th region to extract G a The content of the corresponding area of ​​the image is replaced by G b The pixel content of the corresponding area of ​​the image; the γ·θ operation means that it is γ itself when θ is true, otherwise it is 0; The operation represents determining whether the radian value of a coordinate in the radian matrix is ​​in the radian value interval corresponding to the i-th interval; Any (x, y) in the representative formula belongs to the pixel point in the image G and needs to satisfy the Replace operation; In summary, for the above calculation of the new sample image Ga b The formula can be parsed as follows: a The pixel value and sample image G in the operation arc value interval are obtained by masking b The pixel values ​​of the non-operation arc value interval obtained by the mask in the middle are spliced ​​by element-by-element addition operation of the two calculation matrices to obtain a new sample image. For the new sample image G ab Similarly; There are three strategies for image label mixing: true + true, true + false, false + false; thus, we can get a new mixed sample image with the label: lablea b =1-(1-lable a )(1-lable b ), where label value is 1 for false and 0 for true; The new sample G obtained by replacement operation ab , G ba Add to the original affine sample set G and delete the original sample G from the set G a , G b , and obtain a new enhanced sample set G′: G′=G∪{G ab ,G ba }\{G a ,G b }; A∪B represents the union operation of set A and set B, and A\B represents the difference operation of set A and set B, eliminating the same elements; Repeat S42 operation k times, randomly ε times for each group, and obtain n enhanced samples each time. Finally, merge the ε results to obtain the enhanced sample set G for each group. ε′ : S43: Enhanced masking; for the third operation area Gε′ i nose_mouth The key point information calculated by S1 is divided into two areas: nose and mouth. The segmentation strategy uses the midpoint between the center of the upper lip and the tip of the nose: Among them, FLM is the key point information set, Uplip_center is the center point of the upper lip, nose is the tip of the nose, and the midpoint of the two points is taken as the segmentation point: (separation_pointx,separation_point y )= ((FLM i (Uplip_center)[0]+nose x ) / 2,(FLM i (Uplip_center)[1]+nose y ) / 2); Draw a ray parallel to the x-axis through the midpoint and intersect the third operation area G i nose_mouth At two points, the original operation area is divided into: Traverse the enhanced sample set G ε′ For each sample, proportional enhancement and random masking operations are performed to obtain the final sample data set; Set the random enhancement ratio to μ∈[0,1), when 0≤μ<0.5, use nose area replacement, when 0.5≤μ<0.7, do nothing, when 0.7≤μ<1, use nose area masking: The three parameters of Enhance represent the enhanced images a, b and enhancement ratio μ respectively; different enhancement schemes are adopted according to different enhancement ratios; {EG a ,EG b }=Replace(G ε′ a ,G ε′ b ,nose) ={G ε′ a ·(G ε′ a nose )+G ε′ b ·(!G ε′ a nose ),G ε′ a ·(!G ε′ b nose )+G ε′ b ·(G ε′ b nose )}; Where Replace represents the sample from the enhanced sample set G ε′ In the process, two samples are randomly taken out and replaced in the nose part separated from the third operation area. The γ·θ operation represents γ itself when the γ element is within the range of θ, otherwise it is 0;! G a ′nose Represents the non-nose area in the image; In summary, the above formula divides the face nose area of ​​the enhanced image a and the non-face nose area of ​​the enhanced image b through element masking operation, and adds the two matrices element by element to obtain a new enhanced image, and vice versa; EG i =Blending(G ε′ i )=G ε′ i ·(!G ε′ i nosnose )+0·(G ε′ i nose ); Among them, Blending (G ε′ i ) represents the i-th enhanced sample G ε′ i Perform masking operation; for any pixel value, if it does not belong to the nose area, it remains unchanged, and the pixel value belonging to the nose area is set to 0, that is, the nose area is completely black; For the original enhanced sample set {G ε′ 1, G ε′ 2, ..., G ε′ ε*n After the enhanced masking operation, the new enhanced sample set is recorded as EG: EG=G ε′ ∪Enhancement(G ε′ i ,G ε′ i+1 ,μ i )\{G ε′ i ,G ε′ i+1 },i=0,2,4...(ε*n-2)。 6. The artifact enhancement fusion module for face image forgery detection according to claim 1, characterized in that The specific method of step S5 is: S51: Construct a feedforward neural network with mean square error loss as the loss function, and use the fastest gradient descent back propagation as the optimization algorithm to update the fusion parameter ω: ω=f NN (EG, GI) high ); where f NN (.) represents the learning process of the neural network, and obtains the fusion parameter ω, the fusion enhanced image EG and the corresponding spatial high-frequency representation GI high : FEG=EG+ω·GI high : S52: Construct the network back propagation process using mean square error loss as the loss function: Where M is the number of pixels in each image in the enhanced sample set EG; minimize the loss function and update the parameter ω by the fastest gradient descent method: where ω k is the parameter value at step k, and L(ω k ) is the loss function L in ω k The gradient at step k; the learning rate α at step k k : in Represents the gradient The transpose of Represents matrix multiplication; S53: Train a feedforward neural network to fully fuse the enhanced image EG and the corresponding spatial high-frequency representation GI high The final completion artifact information FEG is input into the discriminator to train the model.

Images