Image tampering detection large model training method and electronic equipment

Abstract

The present application discloses a large model training method for image tampering detection and an electronic device. The large model for image tampering detection includes: a large language model, a visual encoder, a segmentation decoder, a visual text mapper, a classification layer and a positioning structure; in the pre-training stage, the parameters of the large language model, the visual encoder and the visual text mapper are frozen, and the parameters of the segmentation decoder, the classification layer and the positioning structure are trained by using sample data with tampering classification information and tampering positioning results; in the classification training stage, the parameters of the large language model and the visual encoder are frozen, the sample data is converted into text information with tampering classification results and tampering positioning features, and the parameters of the segmentation decoder, the visual text mapper, the classification layer and the positioning structure are trained; in the task training stage, the global structure of the large model for image tampering detection is unfrozen, and all parameters of the large model for image tampering detection are fine-tuned by using sample data with tampering classification results.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a large-scale model training method and electronic equipment for image tampering detection. Background Art

[0002] With the rapid development of computer vision and image processing technologies, people are increasingly demanding privacy and security protection for their faces. Facial tampering prevention technology is a popular application area in facial security. Although large language models are used in many scenarios, directly using large language models to identify facial tampering still suffers from low accuracy and poor interpretability.

[0003] How to accurately identify face tampering through a single-frame facial image is a technical problem that urgently needs to be solved in the facial security business. Summary of the Invention

[0004] The purpose of this application is to provide a training method and electronic equipment for a large model of image tampering detection, so as to solve the problem that the recognition method of face tampering in single-frame images is not accurate and has poor interpretability.

[0005] In a first aspect, the present application provides a method for training a large model for image tampering detection, wherein the large model for image tampering detection includes: a large language model, a visual encoder, a segmentation decoder, a visual text mapper, a classification layer, and a positioning structure;

[0006] In the pre-training phase, the parameters of the large language model, the visual encoder, and the visual text mapper are frozen, and the parameters of the segmentation decoder, the classification layer, and the localization structure are trained using sample data with tampered classification information and tampered localization results;

[0007] In the classification training phase, the parameters of the large language model and the visual encoder are frozen, the sample data is converted into text information with tampered classification results and tampered positioning features, and the parameters of the segmentation decoder, the visual text mapper, the classification layer, and the positioning structure are trained;

[0008] During the task training phase, the global structure of the image tampering detection model is unfrozen, and all parameters of the image tampering detection model are fine-tuned using sample data with tampering classification results.

[0009] Optionally, converting the sample data into text information with tampering classification results and tampering location features, and training parameters of the segmentation decoder, the visual text mapper, the classification layer, and the location structure, comprises:

[0010] The visual encoder converts the tampered face image in the sample data into a feature map with tampered area data, and the segmentation decoder converts the feature map into visual features;

[0011] converting the visual features into visual-text features by the visual-text mapper;

[0012] By means of the classification layer, the tampering classification information in the sample data is mapped into a tampering classification result through the prompt word template encoding;

[0013] Obtaining, through the positioning structure, a tampering positioning feature corresponding to the tampering positioning result in the sample data;

[0014] converting the tampering location feature into tampering location text by the visual text mapper;

[0015] Parameters of the segmentation decoder, the visual text mapper, the classification layer and the localization structure are trained based on the visual text features, the tampering classification results and the tampering localization text.

[0016] Optionally, before the pre-training stage, the method further includes:

[0017] Acquire an initial facial image, and replace the initial facial image based on a face-changing algorithm to obtain a first tampered facial image;

[0018] Calculating pixel differences between key points of the first tampered face image and key points of the original face image, wherein the pixel differences include at least one of: color histogram difference, structure difference, clarity difference, boundary difference, and texture difference;

[0019] For a first region in the first tampered facial image having excessively large pixel differences, replacing a corresponding region of the initial facial image with the first region to obtain a second tampered facial image and tampering information of the second tampered facial image, the tampering information comprising: tampering region data representing the tampered region of the second tampered facial image and tampering attribute information representing the pixel differences;

[0020] Based on the second tampered face image, the tampered information and the prompt word template, a VQA visual question answer sample with tampered information is constructed.

[0021] Optionally, the method further includes:

[0022] When the pixel difference is a structural difference, the structural difference is obtained by comparing brightness similarity, contrast similarity and structural similarity;

[0023] When the pixel difference is a definition difference, the definition difference is obtained by a convolution calculation based on a Labrass operator;

[0024] When the pixel difference is a boundary difference, determining whether there is a boundary difference at the boundary of the key area based on at least two indicators of gradient change, edge conversion, and frequency domain change at the boundary of the key area;

[0025] When the pixel difference is a texture difference, the texture difference is obtained based on a grayscale fairness matrix GLCM contrast.

[0026] Optionally, the positioning structure includes: a linear mapping, an activation function, and a segmentation linear mapping; the classification layer includes: a linear mapping, a pooling layer, a classification linear mapping, and an activation function;

[0027] In the pre-training stage, the parameters of the classification layer and the positioning structure are trained using sample data with tampered classification information and tampered positioning results, including:

[0028] The visual encoder converts the tampered face image in the sample data into a first feature map with tampered region data, and the segmentation decoder converts the first feature map into a tampered feature;

[0029] Converting the tampering feature into a segmentation feature through linear mapping and activation function, and converting the segmentation feature into a tampering positioning result through segmentation linear mapping and activation function;

[0030] Based on the tampered positioning result, adjusting the parameters of the positioning structure through a positioning loss function;

[0031] The tampering features are converted into tampering classification results through linear mapping, pooling layer, classification linear mapping and activation function, and the parameters of the classification layer are adjusted based on the tampering classification results through classification loss function.

[0032] Optionally, in the pre-training phase, freezing the language model, the visual encoder, and the visual text mapper, and training parameters of the classification layer and the localization structure using sample data with tampered classification information and tampered region data, includes:

[0033] In the pre-training stage, the initial face image is used as a positive sample, the second tampered face image is used as a hard sample, and the first tampered face image is used as a negative sample;

[0034] By contrasting the loss function, the classification layer and the positioning structure are trained based on the positive samples, the difficult positive samples and the negative samples.

[0035] The contrast loss function is implemented using the following formula:

[0036]

[0037] in, represents contrast loss, P represents the positive sample, N represents the negative sample, and H represents the difficult sample.

[0038] Optionally, the visual text mapper includes a first visual text mapper and a second visual text mapper, and in the classification training phase:

[0039] The tampered face image in the sample data is converted into a feature map with tampered region data through the visual encoder, the tampered region data in the feature map is converted into tampered features through a segmentation decoder, and the tampered features are converted into segmentation features through linear mapping and activation function;

[0040] converting the segmentation features into segmentation hints via a first visual-text mapper;

[0041] The feature map output by the visual encoder is converted into visual cues through the second visual-text mapper;

[0042] Converting the questions in the sample data into question prompts using the large language model;

[0043] Multiplying the tampering classification result and the text feature of the preset prompt word to obtain a classification prompt;

[0044] The segmentation features, the visual prompts, the question prompts and the classification prompts are respectively input into the large language model to perform tampering knowledge injection on the large language model.

[0045] Optionally, during the task training phase, the global structure of the image tampering detection model is unfrozen, and all parameters of the image tampering detection model are fine-tuned using sample data with tampering classification results, including:

[0046] For each linear mapping layer of the large language model:

[0047] Add a global LORA structure, multiple attribute expert LORA structures, and expert selection structure to each linear mapping layer;

[0048] According to the facial image attributes in the sample's tampering attribute information and the text features input into the current linear mapping layer, a corresponding attribute expert LORA structure is selected from the multiple attribute expert LORA structures through an expert selection structure;

[0049] Based on the text features input into the current linear mapping layer, low-rank features are generated through the global LORA structure and the selected attribute expert LORA structure. The low-rank features and the text features processed by the current linear mapping layer are fused in combination with probability weights to realize the reasoning of knowledge associated with facial image attributes.

[0050] In a second aspect, an embodiment of the present application provides an image tampering detection method, wherein a facial image to be identified and a tampering detection instruction are input into a large image tampering detection model, wherein the large image tampering detection model is trained using the above method;

[0051] The image tampering detection model determines whether the face image to be identified has been tampered with, and obtains a tampering detection result.

[0052] The tampering detection result includes: a tampering classification result, tampering positioning data and tampering attribute information, wherein the tampering classification result includes a fake label representing that the face image to be identified is a tampered image, and a real label representing that the face image to be identified is not a tampered image; the tampering positioning data represents the tampered area when the face image to be identified is a tampered image, and the tampering attribute information represents the pixel difference of the tampered area, wherein the pixel difference includes: at least one of: color histogram difference, structure difference, clarity difference, boundary difference, and texture difference.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one memory and at least one processor,

[0054] The at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement a training method for an image restoration model and / or an image restoration method.

[0055] Optionally, the electronic device further includes a camera and a display.

[0056] The camera is used to capture an initial face image;

[0057] The display is used to display the image tampering detection result output by the image tampering detection model.

[0058] The embodiment of the present application uses sample data with tampering classification information and tampering area data to train a large language model in stages, and converts the sample data into text with tampering positioning features. This can reduce the dependence of the large language model on the sample data, and achieve focus on different capabilities by freezing different modules of the large language model in stages, thereby obtaining more accurate recognition results. It does not need to rely on the correlation between the previous and next frames of the video, and can identify the tampering type and locate the tampering area through only a single frame image, thereby achieving accurate face tampering positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of the method for constructing a difficult sample provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the structure of a large model for image tampering detection provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the structure of a large image tampering detection model is provided for an embodiment of the present application;

[0062] Figure 4 A schematic structural diagram of a large model linear layer is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0063] The present application will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present application. Structural, methodological, or functional changes made by ordinary technicians in this field based on these embodiments are included in the scope of protection of the present application.

[0064] For tampered images, the present invention provides a method for training a large model for image tampering detection. The method can adopt the LLAMA framework, such as LLAMA, Mobile LLAMA, PHI, etc.

[0065] The large model for image tampering detection may include: a large language model, a visual encoder, a segmentation decoder, a visual text mapper, a classification layer, and a positioning structure.

[0066] In the pre-training phase, the parameters of the large language model, visual encoder, and visual text mapper are frozen, and the parameters of the segmentation decoder, classification layer, and localization structure are trained using sample data with tampered classification information and tampered localization results.

[0067] During the classification training phase, the parameters of the large language model and visual encoder are frozen, the sample data is converted into text information with tampered classification results and tampered localization features, and the parameters of the segmentation decoder, visual text mapper, classification layer, and localization structure are trained;

[0068] During the task training phase, the global structure of the large image tampering detection model is unfrozen, and all parameters of the large image tampering detection model are fine-tuned using sample data with tampering classification results.

[0069] The above sample data can be constructed as follows:

[0070] Obtaining an initial face image, and replacing the initial face image based on a face-changing algorithm to obtain a first tampered face image;

[0071] Calculating pixel differences between key points of the first tampered face image and key points of the original face image, wherein the pixel differences include at least one of: color histogram difference, structure difference, clarity difference, boundary difference, and texture difference;

[0072] For a first region in the first tampered facial image having excessively large pixel differences, replace the corresponding region of the original facial image with the first region to obtain a second tampered facial image and tampering information of the second tampered facial image, the tampering information including: tampering region data representing the tampered region of the second tampered facial image and tampering attribute information representing the pixel differences;

[0073] Based on the second tampered face image, tampered information and prompt word template, a VQA visual question answer sample with tampered information is constructed.

[0074] For example, the initial face image can be a face image taken by a camera, or a face image captured from a ID photo or the like. The face swap algorithm can use deepfakes, stable_diffusion, dalle, etc., without limitation.

[0075] For example, it can be achieved by Figure 1 The method shown constructs hard examples.

[0076] Step 101: Obtain an initial face image.

[0077] Step 102: Generate a first tampered face image.

[0078] The initial face image is processed using a face-changing algorithm to obtain a first tampered face image.

[0079] Step 103: Detect key points.

[0080] Specifically, step 103 includes detecting key points of the initial face image and the first tampered face image.

[0081] Step 104: Locate the current key area.

[0082] Specifically, step 104 can determine multiple key areas of the original face image and the tampered face image based on the key points detected in step 103. The key areas may include eye areas, mouth areas, nose areas, cheek areas, etc.

[0083] Step 105: Calculate pixel differences.

[0084] For each key area, a pixel difference between the key area of ​​the initial face image and the key area of ​​the first tampered face image is calculated.

[0085] For example, for the eye region, a pixel difference between the eye region of the initial face image and the eye region of the first tampered face image is calculated. The pixel difference may be a pixel mean difference.

[0086] Exemplarily, the pixel difference may include at least one of: color histogram difference, structure difference, definition difference, boundary difference, and texture difference.

[0087] Step 106: Determine whether the pixel difference in the current key area is greater than a threshold. If so, proceed to step 107; if not, return to step 102 to generate a new first tampered face image.

[0088] Specifically, step 106 includes determining whether a pixel difference between the current key area of ​​the initial facial image and the current key area of ​​the first tampered facial image is greater than a threshold.

[0089] Step 107: Replace the current key area to obtain a second tampered face.

[0090] Specifically, the current key area of ​​the first tampered face image may be used to replace the corresponding current key area of ​​the initial face image, to obtain a new second tampered face image as a difficult sample.

[0091] In an embodiment of the present application, the sample information may also include facial image attributes, such as light intensity, light consistency, clarity, visibility of facial features, skin integrity, etc., which are not limited here.

[0092] In an optional embodiment of the present application, step 106 may be implemented in at least one of the following five ways.

[0093] Method 1:

[0094] For the jth key area among the multiple key areas, the color histogram difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image is calculated using the following formula:

[0095] Formula (1)

[0096] in, Represents the pixel value in the jth key area of ​​the initial face image, Characterize the pixel values ​​in the jth key area of ​​the first tampered face image.

[0097] Assume that each face image has four key areas: mouth area, nose area, eye area and cheek area.

[0098] The following formula can be used to determine whether the mean of the color histogram difference of the jth key area is greater than the threshold :

[0099] Formula (2)

[0100] Represents the jth key area in the initial face image, Represents the sum of pixels in the jth key area in the initial face image.

[0101] If the mean of the color histogram difference of the jth key area in the above formula is greater than the threshold , then, use the jth key area of ​​the first tampered face image Replace the jth key area in the initial face image .

[0102] Optionally, in the embodiment of the present application, the average value of the color histogram difference in the jth key area is greater than the threshold When the jth key area is marked Add to the candidate set, after calculating the color histogram differences of all key areas, select a key area L from the candidate set pick , using L in the first tampered face image pick Replace L in the initial face image pick .

[0103] Method 2:

[0104] For the jth key region among the multiple key regions, calculating the structural difference between the jth key region of the original face image and the jth key region of the first tampered face image can be achieved by:

[0105] It is assumed that the structural similarity index SSIM is used to characterize the above structural differences. SSIM includes brightness similarity, contrast similarity and structural similarity.

[0106] Assume that the initial face image is represented by x and the first tampered face image is represented by y.

[0107] (1) Calculate the brightness similarity of x and y using formula (3) Formula (3),

[0108] in, represents the average brightness of the pixels in the jth key area in the initial face image, represents the average brightness of the jth key area in the first tampered face image, Represents a constant.

[0109] (2) Calculate the contrast similarity of x and y using formula (4) ,

[0110] Formula (4)

[0111] in, Characterizes the standard deviation of the jth key area in the initial face image, represents the standard deviation of the jth key area in the first tampered face image, Represents a constant.

[0112] (3) Calculate the structural similarity between x and y using formula (5)

[0113] Formula (5)

[0114] in, Characterizes the standard deviation of the jth key area in the initial face image, Characterizes the standard deviation of the jth key area in the first tampered face image, Characterizes the covariance of images x and y, Represents a constant.

[0115] (4) Calculate the SSIM structural similarity index of x and y using formula (6):

[0116] Formula (6)

[0117] Among them, x represents the initial face image, y represents the first tampered face image, wherein α, β, and γ are constants, and all three can be 1. l, c, and s represent weights used to balance the three types of similarity indexes, and l, c, and s can also be constants. For example, the sum of l, c, and s is 1.

[0118] In the above When the difference is greater than a threshold, it is determined that the structural difference of the j-th key area is greater than the threshold, and the j-th key area of ​​the initial face image is replaced with the j-th key area of ​​the first tampered face image.

[0119] Method 3:

[0120] For the jth key area among the multiple key areas, the clarity difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image is calculated as follows:

[0121] The blur degree of the j-th key area of ​​the first tampered face image is quantified by the Laplace operator as the above-mentioned clarity difference.

[0122] The Laplace operator can be used as follows The convolution kernel is represented as:

[0123] The convolution kernel can be used to calculate the second-order derivative of the j-th key region, thereby detecting the edges and details of the key region.

[0124] Convolve the Laplacian operator with the jth key area of ​​the initial face image to obtain the Laplacian response map lp1;

[0125] Perform a convolution operation on the Laplace operator and the j-th key area of ​​the first tampered face image to obtain a Laplace response map lp2;

[0126] The variance of lp1 and lp2 is calculated as the clarity difference between the jth key area of ​​the initial face image and the jth key area of ​​the first tampered face image.

[0127] Optionally, the embodiment of the present application may use the following formula (7) to calculate the mean of the Laplace response map lp1 of the j-th key area in the initial face image, and calculate the variance of the mean and the Laplace response map lp2 of the j-th key area of ​​the first tampered face image as the above-mentioned clarity difference.

[0128] Formula (7)

[0129] N represents the number of pixels in the key area, Characterize the Laplace response value of the i-th pixel in the j-th key area of ​​the first tampered face image, Characterizes the mean of the Laplace response map lp2 of the jth key area of ​​the first tampered face image.

[0130] If the variance between the Laplace response map lp2 of the jth key area in the first tampered face image and the mean value of the Laplace response map lp1 of the jth key area in the initial face image exceeds a set threshold, step 107 is executed.

[0131] Method 4: For the jth key area among the multiple key areas, the boundary difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image is calculated by the following method:

[0132] The boundary of the jth key region is evaluated by analyzing three indicators: gradient change, edge transition and frequency domain change. If at least two of these indicators exceed their respective thresholds, the key region is determined to have a boundary difference.

[0133] First, the Sobel operator can be used to calculate the gradient size of the inner boundary (the inner boundary of the first tampered image key area) and the outer boundary (the outer boundary of the initial face image key area) of the key area, and calculate the gradient change of the boundary of the jth key area:

[0134]

[0135]

[0136] in, Characterize the inner boundary of the critical area, Indicates the outer boundary of the critical area;

[0137] The mean gradient of the outer and inner boundaries can be calculated using the following two formulas:

[0138]

[0139]

[0140] in, Indicates the number of pixels on the outer boundary, The number of pixels representing the inner boundary.

[0141] Then, according to the gradient size of the inner and outer boundaries, the gradient discontinuity index is calculated :

[0142]

[0143] if If it is greater than the threshold, it is considered that there is a gradient discontinuity at the boundary of the key area.

[0144] When there is a gradient discontinuity at the boundary of the critical region, the blending artifact at the boundary can be calculated as follows:

[0145] Use the CANNY operator to detect inner and outer edges:

[0146]

[0147]

[0148] If the value is greater than the threshold, it is considered that edge artifacts exist.

[0149] The above scheme calculates the edge gradient difference through the Sobel operator and the edge density difference through the CANNY operator.

[0150] If the above If the value is greater than the threshold, DCT transform is performed on the inner and outer boundaries of the key area to detect whether the frequency of the boundary of the key area is abnormal:

[0151]

[0152] The boundary of the critical area can be divided into high-frequency and low-frequency parts, and the ratio of high-frequency to low-frequency can be calculated:

[0153]

[0154] if If the frequency is greater than the threshold, it is considered that there is a frequency anomaly at the boundary of the key area.

[0155] Method 5: For the jth key area among the multiple key areas, the texture difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image is calculated by the following method:

[0156] 1. Use the gray-level co-occurrence matrix (GLCM) to measure the texture clarity of the original face image and the first tampered face image. The GLCM is calculated as follows:

[0157] First, convert the two images from color images to grayscale images:

[0158]

[0159] Where I represents a color image, 、 、 Represent the red, green and blue pixel values ​​of the color image I respectively, Represents the grayscale value of the grayscale image.

[0160] The grayscale value of the grayscale image Mapping to a specified number of gray levels L (e.g. 256 levels), the mapping formula can be found in the following formula, rounded down:

[0161]

[0162] For the distance between the color image and the grayscale image in each direction, a gray-level co-occurrence matrix is ​​constructed. The directions can be 0°, 45°, 90°, and 135°, and the distance can be 1, 2, or 3 pixels. Assuming the size of the gray-level co-occurrence matrix is ​​L×L, then GLCM(i,j) represents the number of times a pixel with grayscale value i and a pixel with grayscale value j appear at the same time in the specified direction and distance.

[0163] Step 201: Initialize an L×L zero matrix:

[0164] Step 202: traverse each pixel in the image, and for each pixel (x, y), calculate its neighboring pixel (x', y') at the specified direction and distance;

[0165] Step 203: Compute the grayscale co-occurrence matrix of the grayscale values ​​of the pixel (x, y) and its neighboring pixel (x', y') Add 1;

[0166] Step 204: Repeat steps 201 and 202 until all pixels in the image are traversed.

[0167] For directions 0°, 45°, 90°, and 135°, the co-occurrence probability matrix of the current direction is calculated based on the gray level co-occurrence matrix of each pixel.

[0168] Formula (9)

[0169] in, The gray-level co-occurrence matrix (GLCM) representing the j-th key region of the image is the co-occurrence probability matrix in the current direction, i represents the pixel of the j-th key region of the image, and j represents the sequence number of the current key region.

[0170] For example, the contrast of the gray-level co-occurrence matrix (GLCM) of the j-th key region of the initial face image can be calculated by the following formula (9):

[0171] Formula (9)

[0172] in, Characterize the contrast of the gray-level co-occurrence matrix (GLCM) of the jth key region of the initial face image, The gray-level co-occurrence matrix (GLCM) of the j-th key region of the initial face image is represented, i represents the pixels of the j-th key region of the initial face image, and j represents the sequence number of the current key region.

[0173] The contrast of the gray level co-occurrence matrix (GLCM) of the jth key region of the first tampered face image can be calculated based on the following formula (10):

[0174] Formula (10)

[0175] in, Characterize the contrast of the gray-level co-occurrence matrix (GLCM) of the jth key region of the first tampered face image, The gray-level co-occurrence matrix (GLCM) represents the jth key region of the first tampered face image, i represents the pixels of the jth key region of the first tampered face image, j represents the sequence number of the current key region, and N represents the number of key regions in each image.

[0176] like and If the difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image is greater than a preset threshold, it is determined that the texture difference between the jth key area of ​​the original face image and the jth key area of ​​the first tampered face image exceeds the preset threshold. At this time, it can be determined that there is an abnormality in the texture of the jth key area of ​​the first tampered face.

[0177] In an embodiment of the present application, for a first region in a first tampered facial image with excessive pixel differences, the corresponding region of the original facial image can be replaced with the first region to obtain a second tampered facial image and tampering information of the second tampered facial image. The tampering information may include tampering region data and tampering attribute information. The tampering region data may include location information of the first region, and the tampering attribute information may include attributes of the pixel differences of the first region, such as color histogram differences, structural differences, clarity differences, boundary differences, texture differences, etc.

[0178] Based on the second tampered face image, tampered information and prompt word template, a VQA visual question answer sample with tampered information is constructed.

[0179] For example, refer to Figure 2 , the positioning structure in the embodiment of the present application may include: a linear mapping, an activation function and a segmentation linear mapping, and the classification layer in the embodiment of the present application may include: a linear mapping, a pooling layer, a classification linear mapping and an activation function;

[0180] In the pre-training phase, the parameters of the classification layer and positioning structure are trained using sample data with tampered classification information and tampered positioning results, including:

[0181] The tampered face image in the sample data is converted into a first feature map with tampered area data through a visual encoder, and the first feature map is converted into tampered features through a segmentation decoder;

[0182] The tampering features are converted into segmentation features through linear mapping and activation function, and the segmentation features are converted into tampering positioning results through segmentation linear mapping and activation function;

[0183] Based on the tampered positioning results, the parameters of the positioning structure are adjusted through the positioning loss function;

[0184] The tampering features are converted into tampering classification results through linear mapping, pooling layer, classification linear mapping and activation function. The parameters of the classification layer are adjusted based on the tampering classification results through the classification loss function.

[0185] The above prompt word template can be optimized in the following ways.

[0186] Reference Figure 3As shown in the figure, the tampered face image in the sample data is converted into a feature map with tampered area data through the visual encoder, the tampered area data in the feature map is converted into tampered features through the segmentation decoder, and the tampered features are converted into segmentation features through linear mapping and activation function;

[0187] The segmentation features are converted into segmentation hints through the first visual-text mapper;

[0188] The feature map output by the visual encoder is converted into visual cues through the second visual-text mapper;

[0189] Use a large language model to convert questions in sample data into question prompts;

[0190] Multiplying the tampering classification result and the text feature of the preset prompt word to obtain the classification prompt;

[0191] The segmentation features, visual prompts, question prompts and classification prompts are respectively input into the large language model to inject tampered knowledge into the large language model.

[0192] Since, in the embodiment of the present application, in addition to question prompts and visual prompts, segmentation features and classification prompts are also fed into the large language model, the large language model can better learn the tampering attributes in the image and improve the accuracy of tampering detection.

[0193] In the tampering localization branch, the tampering localization result can be an image composed of pixel values ​​0 and 1, and the value of each pixel represents the probability predicted by the image tampering detection model that the pixel belongs to a tampered face.

[0194] The loss function of the tampering localization result can be as follows, which is a pixel-by-pixel cross entropy loss function:

[0195]

[0196] Among them, N represents the number of pixels contained in the tampering positioning result. Represents the predicted value of each pixel belonging to a real face, Represents the predicted value of each pixel belonging to the tampered face, A label indicating whether the pixel is a tampered pixel.

[0197] Through the above loss function of the tampering localization result, the ability of the image tampering detection large model to predict the tampering localization result can be improved.

[0198] In the tampering classification branch, the value of the tampering classification result represents the probability of whether the image is a tampered face predicted by the image tampering detection model.

[0199] The loss function of the tampering classification result can be a pixel-by-pixel cross entropy loss function:

[0200]

[0201] Among them, N represents the number of pixels contained in the tampering positioning result. The predicted value that indicates that the face image belongs to a real face, The predicted value that indicates that the face image is a tampered face. A label indicating whether the face in the image is a tampered face.

[0202] Through the above loss function, the classification layer of the image tampering model can focus more on the tampering classification results, thereby improving the ability of the image tampering detection model to predict the tampering classification results.

[0203] During the pre-training phase, the initial facial image can be used as a positive sample, the second tampered facial image as a difficult sample (hard sample), and the first tampered facial image as a negative sample. Using a contrastive loss function, the classification layer and localization structure of the large image tampering detection model are trained based on these positive, difficult, and negative samples. This training process fine-tunes the parameters of the tampering module (classification layer and localization layer) of the large image tampering detection model, improving the model's ability to locate and classify tampering.

[0204] The above contrast loss calculation is implemented in the following way:

[0205]

[0206] in, denotes contrast loss, P represents the positive sample (real sample, i.e., the initial face image), N represents the first tampered face (negative sample), and H represents the second tampered face (hard sample). exp(⋅) denotes the exponential function, sim(⋅) denotes the similarity function, and log(⋅) denotes the logarithmic function.

[0207] This contrast loss function can shorten the feature distance between positive samples, shorten the feature distance between negative samples, and shorten the feature distance between negative samples and difficult samples while increasing the feature distance between positive samples and difficult samples.

[0208] During the classification training phase, the parameters of the large language model and the visual encoder are frozen. The visual encoder can be used to convert the tampered face images in the sample data into feature maps with tampered area data, and the segmentation decoder can be used to convert the feature maps into visual features.

[0209] Converting visual features into visual-text features through a visual-text mapper;

[0210] Through the classification layer, the tampering classification information in the sample data is mapped into the tampering classification result through the prompt word template encoding;

[0211] Obtaining, through the positioning structure, a tampering positioning feature corresponding to the tampering positioning result in the sample data;

[0212] The tampering location features are converted into tampering location text through the visual text mapper;

[0213] Based on the visual text features, tampering classification results and tampering localization text, the parameters of the segmentation decoder, visual text mapper, classification layer and localization structure are trained.

[0214] During the task training phase, the global structure of the image tampering detection model is unfrozen, and all parameters of the image tampering detection model are fine-tuned using sample data with tampering classification results. Its cross-entropy loss function can be used to improve the matching degree between real and fake text and sample labels, and increase the model's attention to tampered faces. The cross-entropy loss of the difference between real and fake text can be obtained using the following formula:

[0215]

[0216] Characterizes the tampered label of the sample, 0 represents the tampered label, 1 represents the true label, (a K-dimensional vector) represents the desired output “real / falsified” word prediction position, where K represents the vocabulary size in the visual-text mapper.

[0217] In an embodiment of the present application, the sample information may also include facial image attributes, such as light intensity, light consistency, clarity, visibility of facial features, skin integrity, etc., which are not limited here.

[0218] During the task training phase, the global structure of the large image tampering detection model is unfrozen, and a global LoRa structure, multiple attribute expert LoRa structures, and an expert selection structure are added to each linear mapping layer of the large language model (wherein the global LoRa structure and the attribute expert LoRa structure can include multiple low-rank matrices). All parameters of the large image tampering detection model are fine-tuned using sample data with tampering classification results. This can be achieved in the following way:

[0219] For each linear mapping layer of the large model, a global LORA structure, multiple attribute-specific expert LORA structures, and an expert selection structure are added. A global LORA structure, multiple attribute expert LORA structures, and an expert selection structure are added to each linear mapping layer; based on the facial image attributes in the sample and the text features input to the current linear mapping layer, the corresponding attribute expert LORA structure is selected from the multiple attribute expert LORA structures through the expert selection structure; based on the text features input to the current linear mapping layer, low-rank features are generated through the global LORA structure and the selected attribute expert LORA structure, and the low-rank features and the text features processed by the current linear mapping layer are fused in combination with probability weights to achieve reasoning of knowledge associated with facial image attributes.

[0220] Exemplarily, each attribute expert LORA structure includes two low-rank matrices A and B.

[0221] See also Figure 4 ,Each expert selection structure consists of a quality embedding layer, a text feature linear layer, and a selection linear layer.

[0222] For each sample data, based on the face image attributes and the input of the current linear mapping layer, a specific attribute expert LORA structure is selected through the expert selection structure. The selection of the attribute expert LORA structure can be achieved by:

[0223] Suppose nine image quality metrics (i.e., nine facial image attributes) are input into the quality embedding layer of the expert selection structure, and the input of the current linear mapping layer is fed into the text feature linear layer of the expert selection structure. The outputs of the quality embedding layer and the text feature linear layer are summed and input into the selection linear layer of the expert selection structure. The activation value obtained by applying a softmax function to the output of the selection linear layer is used as the probability of the attribute expert LoRa structure being selected. The attribute expert LoRa structure with the highest probability is selected.

[0224] Based on the text features input into the current linear mapping layer, low-rank features are generated through the global LORA structure and the selected attribute expert LORA structure. The low-rank features and the text features processed by the current linear mapping layer are fused with probability weights to achieve inference of knowledge related to facial image attributes. This can be achieved in the following ways:

[0225] According to the result probability p of the selected attribute expert LORA structure, the selected attribute expert LORA structure, the global LORA structure and the original linear mapping layer of the large language model are used to process the text features and perform weighted fusion output. The output formula can be as follows:

[0226]

[0227] Among them A n B n Respectively represent the low-rank matrices of the selected attribute expert LORA structure,

[0228] A g B g The low-rank matrix of the global LORA structure, linear represents the original linear mapping layer of the large language model, and x i Represents the input of the linear mapping layer, x i+1 Represents the output of the linear mapping layer.

[0229] See also Figure 4 As shown, through this structure, it is possible to reason about the associated knowledge of facial image attributes, thereby improving the learning ability of large language models for facial image attributes.

[0230] The embodiment of the present application trains a large language model in stages using sample data with tampering classification information and tampering area data, thereby reducing the dependence of the large language model on sample data. By freezing different modules of the large language model in stages, attention can be focused on different capabilities, thereby obtaining more accurate recognition results. It does not need to rely on the correlation between the previous and next frames of the video, and can identify the tampering type and locate the tampering area through only a single frame image, thereby achieving accurate facial tampering location.

[0231] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, comprising: at least one memory and at least one processor, wherein the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-mentioned image restoration model training method and / or image restoration method.

[0232] The above-mentioned electronic devices also include cameras and displays,

[0233] The camera is used to capture an initial face image;

[0234] The display is used to display the image tampering detection result output by the image tampering detection model.

[0235] The memory may be a random access memory, a read-only memory, a non-volatile memory, a programmable ROM, an erasable PROM, an electrically erasable memory, a flash memory, an optical memory, a register, and the like. The processor may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing a computer program stored in the memory, and the general-purpose processor may use data stored in the memory in the process of performing the steps and / or operations. The general-purpose processor may be a central processing unit, an ASIC, an FPGA, and the like. During implementation, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0236] Exemplarily, the input device includes but is not limited to at least one of a keyboard, a touch panel, a voice input device, and an image sensor.

[0237] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or solid-state drive (SSD).

[0238] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0239] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application are included in the scope of protection of this application.

Claims

1. A large-scale model training method for image tampering detection, characterized in that: include: The image tampering detection large model includes: a large language model, a prompt word embedding layer, a visual encoder, a segmentation decoder, a visual text mapper, a classification layer and a positioning structure; In the pre-training phase, the parameters of the large language model, the visual encoder, and the visual text mapper are frozen, and the parameters of the segmentation decoder, the classification layer, and the localization structure are trained using sample data with tampered classification information and tampered localization results; During the classification training phase, the parameters of the large language model and the visual encoder are frozen, the sample data is converted into text information with tampered classification results and tampered positioning features, and the parameters of the segmentation decoder, the visual text mapper, the classification layer, and the positioning structure are trained; the visual text mapper includes a first visual text mapper and a second visual text mapper, and the visual encoder is used to convert the tampered face image in the sample data into a feature map with tampered region data, the tampered region data in the feature map is converted into tampered features through the segmentation decoder, and the tampered features are converted into segmentation features through the linear mapping and activation function of the positioning structure; the tampered features are converted into tampered classification results through the classification layer; converting the segmentation features into segmentation hints via a first visual-text mapper; The feature map output by the visual encoder is converted into visual cues through the second visual-text mapper; Converting the questions in the sample data into question prompts through the prompt word embedding layer; Multiplying the tampering classification result and the text feature of the preset prompt word to obtain a classification prompt; Inputting the segmentation prompt, the visual prompt, the question prompt, and the classification prompt into the large language model respectively to perform tampering knowledge injection on the large language model; During the task training phase, the global structure of the image tampering detection model is unfrozen, and all parameters of the image tampering detection model are fine-tuned using sample data with tampering classification results.

2. The method according to claim 1, wherein Before the pre-training stage, the method further includes: Acquire an initial facial image, and replace the initial facial image based on a face-changing algorithm to obtain a first tampered facial image; Calculating pixel differences between key points of the first tampered face image and key points of the original face image, wherein the pixel differences include at least one of: color histogram difference, structure difference, clarity difference, boundary difference, and texture difference; For a first region in the first tampered facial image having excessively large pixel differences, replacing a corresponding region of the initial facial image with the first region to obtain a second tampered facial image and tampering information of the second tampered facial image, the tampering information comprising: tampering region data representing the tampered region of the second tampered facial image and tampering attribute information representing the pixel differences; Based on the second tampered face image, the tampered information and the prompt word template, a VQA visual question answer sample with tampered information is constructed.

3. The method according to claim 2, wherein The method further comprises: When the pixel difference is a structural difference, the structural difference is obtained by comparing brightness similarity, contrast similarity and structural similarity; When the pixel difference is a definition difference, the definition difference is obtained by a convolution calculation based on a Labrass operator; When the pixel difference is a texture difference, the texture difference is obtained based on a gray level co-occurrence matrix GLCM contrast.

4. The method according to claim 1, wherein The positioning structure includes: a linear map, an activation function and a segmentation linear map, and the classification layer includes: a linear map, a pooling layer, a classification linear map and an activation function; In the pre-training stage, the parameters of the classification layer and the positioning structure are trained using sample data with tampered classification information and tampered positioning results, including: The visual encoder converts the tampered face image in the sample data into a first feature map with tampered region data, and the segmentation decoder converts the first feature map into a tampered feature; Converting the tampering feature into a segmentation feature through linear mapping and activation function, and converting the segmentation feature into a tampering positioning result through segmentation linear mapping and activation function; Based on the tampered positioning result, adjusting the parameters of the positioning structure through a positioning loss function; The tampering features are converted into tampering classification results through linear mapping, pooling layer, classification linear mapping and activation function, and the parameters of the classification layer are adjusted based on the tampering classification results through classification loss function.

5. The method according to claim 2, wherein In the pre-training phase, the language model, the visual encoder, and the visual text mapper are frozen, and parameters of the classification layer and the localization structure are trained using sample data with tampered classification information and tampered region data, including: In the pre-training stage, the initial face image is used as a positive sample, the second tampered face image is used as a hard sample, and the first tampered face image is used as a negative sample; By contrasting the loss function, the classification layer and the positioning structure are trained based on the positive samples, the hard samples and the negative samples. The contrast loss function is implemented using the following formula: in, represents contrast loss, P represents the positive sample, N represents the negative sample, and H represents the difficult sample.

6. The method according to claim 1, wherein During the task training phase, the global structure of the image tampering detection model is unfrozen, and all parameters of the image tampering detection model are fine-tuned using sample data with tampering classification results, including: For each linear mapping layer of the large language model: Add a global LORA structure, multiple attribute expert LORA structures, and expert selection structure to each linear mapping layer; According to the facial image attributes in the sample and the text features input into the current linear mapping layer, a corresponding attribute expert LORA structure is selected from the multiple attribute expert LORA structures through the expert selection structure; Based on the text features input into the current linear mapping layer, low-rank features are generated through the global LORA structure and the selected attribute expert LORA structure. The low-rank features of the global LORA structure and the low-rank features generated by the selected attribute expert LORA structure are fused with the probability weights. The fused low-rank features are fused with the text features processed by the current linear mapping layer to realize the reasoning of knowledge associated with facial image attributes.

7. A method for detecting image tampering, characterized in that: Inputting the face image to be identified and the tampering detection instruction into a large image tampering detection model, wherein the large image tampering detection model is trained by the method according to any one of claims 1 to 6; The image tampering detection model determines whether the face image to be identified has been tampered with, and obtains a tampering detection result. The tampering detection result includes: a tampering classification result, a tampering positioning result and tampering attribute information, wherein the tampering classification result includes a fake label representing that the face image to be identified is a tampered image, and a real label representing that the face image to be identified is not a tampered image; the tampering positioning result represents the tampered area when the face image to be identified is a tampered image, and the tampering attribute information represents the pixel difference of the tampered area, wherein the pixel difference includes: at least one of: color histogram difference, structure difference, clarity difference, boundary difference, and texture difference.

8. An electronic device, characterized in that: include: at least one memory and at least one processor, The at least one memory stores an executable code, and the at least one processor is configured to execute the executable code in the at least one memory to implement the method according to any one of claims 1 to 7.

9. The electronic device according to claim 8, wherein Also includes a camera and a display, The camera is used to capture an initial face image; The display is used to display the tampering detection result output by the image tampering detection model.

Images