Face counterfeit segmentation method based on multi-condition input

Abstract

The invention relates to the technical field of artificial intelligence, and provides a face counterfeit segmentation method based on multi-condition input. The method comprises the following steps: extracting feature bottom noise in an image; and high-frequency information in the image is extracted. And constructing a multi-condition input segmentation neural network, and performing efficient fine tuning training on the multi-condition input segmentation neural network until the multi-condition input segmentation neural network converges. And the trained multi-condition input segmentation neural network model is utilized to carry out reasoning on the image, the segmentation neural network model finally outputs a binary segmentation image consistent with the original image in size, and a face counterfeit area in the image is detected and positioned. According to the method, feature bottom noise and high-frequency information in a forged image are extracted, the information serves as additional conditions and is input into a segmentation network together with an original image, a model is trained by means of a designed multi-condition input efficient fine tuning architecture, the model can predict a binary segmentation image, and the image segmentation efficiency is improved. Therefore, the face counterfeit area in the image is detected and accurately positioned.

Description

Technical Field

[0001] The invention relates to the field of artificial intelligence technology, and in particular to a forged face segmentation method based on multi-condition input. Background Art

[0002] In the current era of rapid technological advancement, image processing technology is developing at an unprecedented speed, bringing us endless convenience and surprises. However, at the same time, the advancement of this technology has also provided more advanced means for image forgery, making forged images more realistic and almost unbelievable. In the digital world, a seemingly ordinary picture may hide traces of careful forgery, and our naked eyes often find it difficult to distinguish the authenticity.

[0003] Faced with this challenge, how to accurately determine whether the image content has been forged has become a complex problem that needs to be solved urgently. In traditional image processing, when we face a synthesized image, there may be obvious stitching marks, or unnatural transitions and blurring between the edges of different image elements. These obvious flaws, like clues to solving a case in a detective novel, reveal the forgery of the image. In the stitched image, the inconsistencies in the shooting conditions, light sources, shadows, and perspective effects of different parts of the image often reveal the truth in the details. However, with the continuous advancement of technology, these traditional forgery clues are becoming increasingly difficult to detect.

[0004] Nowadays, with the rapid development of AI face-changing technology, image forgery has entered a new stage. With the help of advanced technologies such as deep learning, AI face-changing technology can seamlessly embed a person's facial features into another image, achieving a near-perfect fusion. The emergence of this technology undoubtedly provides new possibilities for image forgery, making forged images more realistic and difficult to identify.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a forged face segmentation method based on multi-condition input.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for face forgery segmentation based on multi-condition input, comprising:

[0009] In step 201, the characteristic background noise in the image is extracted by using the Noiseprint++ tool;

[0010] In step 202, the image is transformed in the frequency domain, and high-frequency information in the image is extracted using a high-pass filter.

[0011] In step 203, a multi-conditional input segmentation neural network is constructed, and the original image and the corresponding feature background noise and high-frequency information are input into the multi-conditional input segmentation neural network together. The multi-conditional input segmentation neural network is efficiently fine-tuned and trained using the supervisory information of the binary segmentation map of the forged face area until the multi-conditional input segmentation neural network converges.

[0012] In step 204, the trained multi-condition input segmentation neural network is saved, a forged face image is input, and the image is inferred using the trained multi-condition input segmentation neural network model. The segmentation neural network model finally outputs a binary segmentation map of the same size as the original image, and detects and locates the forged face area in the image.

[0013] Preferably, the performing frequency domain transformation on the image and extracting high-frequency information from the image using a high-pass filter specifically includes:

[0014] For the RGB three-channel image I, after passing through the high-pass filter, the 3-channel high-frequency information F with the same size as the original image I is obtained. H .

[0015] Preferably, the multi-conditional input segmentation neural network specifically includes a network backbone VIT, a multi-conditional input module and an image segmentation decoder.

[0016] Preferably, the network backbone VIT comprises a plurality of Transformer encoder modules;

[0017] Each Transformer encoder module contains a multi-head self-attention mechanism and a feed-forward neural network;

[0018] The image input to the network backbone VIT is divided into image blocks of fixed size, and the image blocks are linearly embedded and sent to each Transformer encoder module as an input sequence;

[0019] The Transformer encoder modules transfer features through residual connections, and an additional layer normalization operation is performed after each Transformer encoder module;

[0020] Finally, the network backbone VIT outputs feature maps.

[0021] Preferably, the multi-condition input module includes a plurality of independent embedding networks and a shared multi-layer perceptron;

[0022] Characteristic noise floor F N , original image I and high frequency information FH The features are extracted through their respective embedding networks, where each embedding network includes a shared downsampling MLP and independent MLP modules.

[0023] The features extracted by each embedding network are fused through the cross-attention mechanism. The features from the network backbone VIT are used as queries, and the feature background noise, image features and high-frequency information are added as keys and values ​​and input into the cross-attention module.

[0024] The fused features are normalized and then input into the shared upsampling MLP for further processing. Finally, they are accumulated layer by layer with the output of the upper layer to form the final multi-conditional input information to enhance the performance of the face forgery segmentation network.

[0025] Preferably, the shared downsampling MLP and the shared upsampling MLP are shared in each layer of the network backbone VIT, and are used to compress high-dimensional data features to low-dimensional;

[0026] In each layer of the network backbone VIT, an independent MLP module is used to further transform the data features in low-dimensional features to adapt to the specific weights of the current network layer.

[0027] Preferably, during the efficient fine-tuning training process, the model weights of the network backbone VIT are frozen, and the weights of the shared downsampling MLP, shared upsampling MLP, independent MLP modules and multi-conditional input modules are trained, so that the training parameters of the multi-conditional input segmentation neural network are controlled within 3% to ensure the efficiency of network training.

[0028] Preferably, the image segmentation decoder comprises two upsampling convolutional layers, two activation layers and a linear mapping layer;

[0029] The upsampling convolutional layer is used to gradually restore the spatial resolution of the feature map, ensuring that the final output matches the size of the original image;

[0030] Each upsampling convolution layer is followed by a nonlinear activation layer, and the activation function used in the nonlinear activation layer is a ReLU function or a Leaky ReLU function to introduce nonlinear features and enhance the expressiveness of the model;

[0031] Finally, the linear mapping layer further processes the upsampled features and maps the features to a binary space through a linear transformation, thereby generating a final binary segmentation map that distinguishes the forged part from the real part.

[0032] Preferably, each pixel point of the binary segmentation map is mapped to a binary space, wherein 1 indicates that it is detected as a forged area, and 0 indicates that it is detected as a real area.

[0033] Preferably, during the efficient fine-tuning training process, the binary segmentation map generated by the image segmentation decoder is also compared with the real binary segmentation map to calculate the value of the loss function; the loss function is selected from a cross entropy loss function or a Dice loss function, etc., which is used to measure the difference between the predicted binary segmentation map and the real label;

[0034] After calculating the value of the loss function, the back-propagation mechanism is used to pass the value of the loss function back to each layer of the multi-conditional input segmentation neural network, and the gradient descent algorithm is used to adjust the trainable parameter weights in the multi-conditional input segmentation neural network. This training process is repeated between the layers of the entire multi-conditional input segmentation neural network until the loss value of the model tends to stabilize, that is, the multi-conditional input segmentation neural network reaches a convergence state.

[0035] In a second aspect, the present invention further provides a face forgery segmentation device based on multi-condition input, which is used to implement the face forgery segmentation method based on multi-condition input described in the first aspect, and the device comprises:

[0036] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the forged face segmentation method based on multi-condition input as described in the first aspect.

[0037] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0038] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect to the second aspect and any one of them.

[0039] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0040] The present invention extracts characteristic background noise and high-frequency information from forged images, and inputs this information into the segmentation network together with the original image as additional conditions. The model is trained using a designed multi-condition input efficient fine-tuning architecture, so that the model can predict a binary segmentation map of the same size as the original image, thereby detecting and accurately locating the forged face area in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a flowchart of a method for forged face segmentation based on multi-condition input provided by an embodiment of the present invention;

[0043] Figure 2 It is a flowchart of a method for forged face segmentation based on multi-condition input provided by an embodiment of the present invention;

[0044] Figure 3 It is a flowchart of a method for forged face segmentation based on multi-condition input provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of a forged face segmentation method based on multi-condition input provided by an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of a forged face segmentation method based on multi-condition input provided by an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of the architecture of a face forgery segmentation device based on multi-condition input provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0050] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0051] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0052] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0053] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] Embodiment 1:

[0055] In order to detect and accurately locate the forged face area in the image and provide an effective solution for the automation of forged face identification, embodiment 1 of the present invention provides a forged face segmentation method based on multi-condition input, such as Figure 1 As shown, including:

[0056] In step 201, the characteristic background noise in the image is extracted by the Noiseprint++ tool; the image characteristic background noise is a series of subtle and unique traces left by lens distortion, sensor noise, color correction and sharpening in the camera, compression, transmission, editing and other operations outside the camera during the creation, transmission and storage of the digital image. These traces provide important clues for identifying forged images. When the image is manipulated, the image characteristic background noise may change significantly. After the image is cropped and the brightness is adjusted by image processing software, the original color distribution and detail structure may change; and when the image restoration technology is used to remove certain elements in the image, an unnatural transition may occur between the restoration area and the original area. This will make the characteristic background noise of the original area and the edited area of ​​the image discontinuous or inconsistent. The characteristic background noise of the entire image can be extracted by the Noiseprint++ tool. When the image is forged or tampered with, key clues can be obtained from the extracted characteristic background noise.

[0057] In step 202, the image is transformed in the frequency domain, and the high-frequency information in the image is extracted using a high-pass filter. The high-frequency information of the image is the information carried by those areas in the image that change rapidly and have large gradients. These high-frequency areas contain the key details and texture features of the image, and the high-frequency information is closely related to the image's edges, details, and texture features, and is an important clue for identifying forged images. The high-frequency information in the image is extracted through a high-pass filter. The high-pass filter filters out the low-frequency part (i.e., the smooth area) in the image, while retaining the high-frequency part (i.e., the edge and texture). In a forged picture, since some areas of the image have been tampered with and edited, these operations will to a certain extent destroy the high-frequency information in the edited area, modify the color distribution and texture features in the image, and thus cause the high-frequency information to change or produce anomalies. When an image is forged or tampered with, key clues can be obtained from the extracted high-frequency information. Given an original image I, I is an RGB three-channel image, after passing through a high-pass filter, a 3-channel high-frequency information F of the same size as the original image I is obtained. H .

[0058] In step 203, a multi-condition input segmentation neural network is constructed, and the original image and the corresponding feature background noise and high-frequency information are input into the multi-condition input segmentation neural network together, and the supervision information of the binary segmentation map of the face forgery area is used to efficiently fine-tune the multi-condition input segmentation neural network until the multi-condition input segmentation neural network converges. The multi-condition input segmentation neural network specifically includes a network backbone VIT, a multi-condition input module and an image segmentation decoder.

[0059] In step 204, the trained multi-condition input segmentation neural network is saved, a forged face image is input, and the image is inferred using the trained multi-condition input segmentation neural network model. The segmentation neural network model finally outputs a binary segmentation map of the same size as the original image, and detects and locates the forged face area in the image.

[0060] This embodiment extracts characteristic background noise and high-frequency information from the forged image, and inputs this information into the segmentation network together with the original image as additional conditions. The model is trained using a designed multi-condition input efficient fine-tuning architecture, so that the model can predict a binary segmentation map of the same size as the original image, thereby detecting and accurately locating the forged face area in the image.

[0061] The frequency domain transformation of the image and the extraction of high-frequency information in the image using a high-pass filter specifically include: for the RGB three-channel image I, after the high-pass filter, the three-channel high-frequency information F having the same size as the original image I is obtained. H .

[0062] In actual application scenarios, the network backbone VIT includes multiple Transformer encoder modules; each Transformer encoder module contains a multi-head self-attention mechanism and a feedforward neural network; the image input to the network backbone VIT is divided into image blocks of fixed size, and the image blocks are linearly embedded as input sequences into each Transformer encoder module; the Transformer encoder modules are connected through residual connections for feature transfer, and a layer normalization operation is added after each Transformer encoder module; finally, the network backbone VIT outputs a feature map. The feature map output by the network backbone VIT will be fused with the conditional information generated by the multi-conditional input module to enhance the overall performance of the segmentation network.

[0063] The multi-condition input module includes multiple independent embedding networks and a shared multi-layer perceptron; the feature noise floor F N , original image I and high frequency information F HThe features are extracted through their respective embedding networks; each embedding network includes a shared downsampling MLP and independent MLP modules; the features extracted by each embedding network are fused through a cross-attention mechanism; the features from the network backbone VIT are used as queries, and the feature background noise, image features and high-frequency information are input into the cross-attention module as keys and values ​​after an addition; the fused features are normalized and then input into the shared upsampling MLP for further processing, and finally accumulated layer by layer with the output of the upper layer to form the final multi-conditional input information to enhance the performance of the face forgery segmentation network.

[0064] Among them, the shared downsampling MLP and the shared upsampling MLP are shared in each layer of the network backbone VIT, which are used to compress high-dimensional data features to low-dimensional features; in each layer of the network backbone VIT, an independent MLP module is used to further transform the data features in the low-dimensional features to adapt to the specific weights of the current network layer.

[0065] In a preferred embodiment, Figure 2 As shown, the method also includes:

[0066] In step 301, during the efficient fine-tuning training process, the model weights of the network backbone VIT are frozen.

[0067] In step 302, the shared downsampling MLP, the shared upsampling MLP, the independent MLP modules and the weights of the multi-conditional input module are trained so that the training parameters of the multi-conditional input segmentation neural network are controlled within 3% to ensure the high efficiency of network training.

[0068] In practical application scenarios, the image segmentation decoder includes two upsampling convolutional layers, two activation layers and one linear mapping layer.

[0069] The upsampling convolution layer is used to gradually restore the spatial resolution of the feature map to ensure that the final output matches the size of the original image; each upsampling convolution layer is followed by a nonlinear activation layer, and the activation function used by the nonlinear activation layer is the ReLU function or the Leaky ReLU function to introduce nonlinear features and enhance the expressiveness of the model.

[0070] Finally, the linear mapping layer further processes the upsampled features and maps the features to a binary space through a linear transformation, thereby generating a final binary segmentation map that distinguishes the forged part from the real part. Each pixel of the binary segmentation map is mapped to a binary space, where 1 indicates that it is detected as a forged area and 0 indicates that it is detected as a real area. This precise calibration at the pixel level can capture the tiny details of the forged area, even the forged traces that are extremely difficult to detect. Then, through further analysis and processing of these areas, the model can accurately locate the forged content. The area where the "0" value is located in the binary segmentation map is marked as the original real part. These areas can be used to verify the integrity of the image to ensure that the non-forged area has not been tampered with or modified.

[0071] In an optional embodiment, if Figure 3 As shown, the method also includes:

[0072] In step 401, during the efficient fine-tuning training process, the binary segmentation map generated by the image segmentation decoder is also compared with the real binary segmentation map to calculate the value of the loss function; the loss function uses a cross entropy loss function or a Dice loss function, etc., to measure the difference between the predicted binary segmentation map and the real label; among them, the cross entropy loss measures the performance of the model by calculating the error between the predicted probability of each pixel and the real label, while the Dice loss pays more attention to the accuracy of the overlapping part in the segmentation result, which is particularly suitable for unbalanced category distribution.

[0073] In step 402, after calculating the value of the loss function, the back propagation mechanism is used to pass the value of the loss function back to each layer of the multi-conditional input segmentation neural network, and the gradient descent algorithm is used to adjust the trainable parameter weights in the multi-conditional input segmentation neural network. This training process is repeated between each layer of the multi-conditional input segmentation neural network until the loss value of the model tends to be stable, that is, the multi-conditional input segmentation neural network reaches a convergence state. At this point, the parameters of the model have been optimized to the optimal state, and the binary segmentation map of the input image can be accurately predicted, and the forged part and the real part can be reliably distinguished.

[0074] Embodiment 2:

[0075] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0076] The forged face segmentation method based on multi-condition input described in this embodiment is as follows: Figure 4 As shown, the following steps are included:

[0077] Step 1: Extract the characteristic noise in the image. Use the Noiseprint++ tool to extract the potential noise information in the image. The image characteristic noise is a series of subtle and unique traces left by lens distortion, sensor noise, color correction and sharpening in the camera, compression, transmission, editing and other operations during the creation, transmission and storage of digital images. These traces provide important clues for identifying forged images. When the image is manipulated, the image characteristic noise may change significantly. After using image processing software to crop the image, adjust the brightness and other operations, the original color distribution and detail structure may change; when using image restoration technology to remove certain elements in the image, an unnatural transition may occur between the repaired area and the original area. This will make the characteristic noise of the original area and the edited area of ​​the image discontinuous or inconsistent. The Noiseprint++ tool can be used to extract the characteristic noise of the entire image. When the image is forged or tampered with, key clues can be obtained from the extracted characteristic noise. Given the original image I, I is an RGB three-channel image. After the Noiseprint++ tool, a 1-channel characteristic noise F with the same size as the original image I is obtained. N .

[0078] Step 2: Extract high-frequency information from the image, transform the image in the frequency domain, and use a high-pass filter to extract the high-frequency information from the image. Image high-frequency information is the information carried by those areas in the image that change rapidly and have large gradients. These high-frequency areas contain the key details and texture features of the image, and the high-frequency information is closely related to the image's edges, details, and texture features, and is an important clue for identifying forged images. The high-frequency information in the image is extracted through a high-pass filter. The high-pass filter filters out the low-frequency part of the image (i.e., the smooth area) and retains the high-frequency part (i.e., the edge and texture). In forged pictures, since some areas of the image have been tampered with and edited, these operations will destroy the high-frequency information in the edited area to a certain extent, modify the color distribution and texture features in the image, and thus cause the high-frequency information to change or produce anomalies. When an image is forged or tampered with, key clues can be obtained from the extracted high-frequency information. Given an original image I, I is an RGB three-channel image, after passing through a high-pass filter, a 3-channel high-frequency information F of the same size as the original image I is obtained. H .

[0079] Step 3: Train the network and build a multi-condition input efficient fine-tuning segmentation neural network. Input the original image and the corresponding feature background noise and high-frequency information into the neural network together. Use the supervision information of the binary segmentation map of the forged face area to efficiently fine-tune the multi-condition input segmentation neural network until the neural network converges.

[0080] like Figure 5As shown in the figure, the multi-condition input efficient fine-tuning segmentation neural network model consists of three parts: the network backbone VIT, the multi-condition input module, and the image segmentation decoder. The network backbone VIT consists of several Transformer encoder modules, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The image input is first segmented into fixed-size image blocks, which are linearly embedded and fed into each encoder module as an input sequence. The encoder modules transfer features through residual connections, and an additional layer normalization operation is performed after each module. Finally, the feature map output by the VIT network will be fused with the conditional information generated by the multi-condition input module to enhance the overall performance of the segmentation network.

[0081] The multi-condition input module consists of multiple independent embedding networks and a shared multi-layer perceptron (MLP). Specifically, the feature noise F N , original image I and high frequency information F H The features are extracted through their respective embedding networks, each of which consists of a shared downsampling MLP and independent MLP modules. These extracted features are fused through a cross-attention mechanism, where the features from the network backbone VIT are used as queries (Q), and the feature noise, image features, and high-frequency information are added as keys (K) and values ​​(V) to the cross-attention module. The fused features are normalized (Add&Norm) and then input into the shared upsampling MLP for further processing. Finally, they are accumulated layer by layer with the output of the upper layer to form the final multi-conditional input information, which enhances the performance of the face forgery segmentation network.

[0082] The shared downsampling MLP and shared upsampling MLP are shared in each layer of the network backbone VIT, and are used to compress high-dimensional data features to low dimensions. In each layer, an independent MLP module is used to further transform the data features in low-dimensional features to adapt to the specific weights of the current network layer. The compression of features can ensure that the subsequent cross-attention mechanism is performed on low-dimensional data, which saves more computing resources during training. In the network, only the shared MLP, independent MLP modules, and QKV with multi-conditional inputs can be trained, while the original model weights of the network backbone VIT are frozen. This can ensure the high efficiency of network training, and the training parameters of the overall network are controlled within 3%.

[0083] The image segmentation decoder receives the output feature maps of the network backbone VIT and the multi-condition input module, and predicts a binary segmentation map with the same size as the original image. The design of the image segmentation decoder includes two upsampling convolution layers, two activation layers and a linear mapping layer. First, the upsampling convolution layer is used to gradually restore the spatial resolution of the feature map to ensure that the final output matches the size of the original image. Each upsampling convolution layer is followed by a nonlinear activation layer. The commonly used activation function can be ReLU (Rectified Linear Unit) or Leaky ReLU to introduce nonlinear features and enhance the expression ability of the model. Finally, the linear mapping layer further processes the upsampled features and maps the features to a binary space through a linear transformation, thereby generating a final binary segmentation map that clearly distinguishes the forged part from the real part. Each pixel in the binary segmentation map output by the linear mapping layer is mapped to a binary space, where "1" indicates that it is detected as a forged area and "0" indicates that it is detected as a real area. In the detection process of the forged part, the model determines the boundary and shape of the forged area by analyzing the location and distribution of the "1" value in the binary segmentation map. This pixel-level precision calibration can capture the tiny details of the forged area, even the traces of forgery that are extremely difficult to detect. Then, through further analysis and processing of these areas, the model can accurately locate the forged content. The areas where the "0" value is located in the binary segmentation map are marked as the original real parts. These areas can be used to verify the integrity of the image and ensure that the non-forged areas have not been tampered with or modified.

[0084] The binary segmentation map generated by the image segmentation decoder will be compared with the real binary segmentation map during the training process to calculate the loss function. Specifically, the loss function uses cross-entropy loss or Dice loss, etc., to measure the difference between the predicted binary segmentation map and the real label. Cross-entropy loss measures the performance of the model by calculating the error between the predicted probability of each pixel and the real label, while Dice loss focuses more on the accuracy of the overlapping part of the segmentation result, which is particularly suitable for unbalanced category distribution. After calculating the loss value, the model will use the backpropagation mechanism to pass the loss value back to each layer of the network. In this process, the gradient descent algorithm is used to adjust the trainable parameter weights in the network. This training process is repeated between each layer of the entire network until the loss value of the model tends to be stable, that is, the network reaches a convergence state. At this point, the parameters of the model have been optimized to the best state, which can accurately predict the binary segmentation map of the input image and reliably distinguish the forged part from the real part.

[0085] Step 4: Test the network. Save the trained multi-condition input segmentation neural network. When testing the network, input a fake face image and use the trained segmentation neural network model to infer the image. The network will eventually predict a binary segmentation map with the same size as the original image. Based on the results of the binary segmentation map, the fake face area in the image can be detected and accurately located.

[0086] This embodiment extracts characteristic background noise and high-frequency information from the forged image, and inputs this information into the segmentation network together with the original image as additional conditions. The model is trained using a designed multi-condition input efficient fine-tuning architecture, so that the model can predict a binary segmentation map of the same size as the original image, thereby detecting and accurately locating the forged face area in the image.

[0087] Embodiment 3:

[0088] like Figure 6 , is a schematic diagram of the structure of a face forgery segmentation device based on multi-condition input according to an embodiment of the present invention. The face forgery segmentation device based on multi-condition input according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0089] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0090] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the face forgery segmentation method based on multi-condition input in Example 1. The processor 21 executes the face forgery segmentation method based on multi-condition input by running the non-volatile software programs and instructions stored in the memory 22.

[0091] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0092] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the forged face segmentation method based on multi-condition input in the above-mentioned embodiment 1 is executed, for example.

[0093] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0094] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A face forgery segmentation method based on multi-condition input, characterized in that: include: The characteristic noise floor in the image is extracted by the Noiseprint++ tool; Perform frequency domain transformation on the image and use a high-pass filter to extract high-frequency information from the image; Construct a multi-condition input segmentation neural network, input the original image and the corresponding feature background noise and high-frequency information into the multi-condition input segmentation neural network, and use the supervision information of the binary segmentation map of the forged face area to efficiently fine-tune the multi-condition input segmentation neural network until the multi-condition input segmentation neural network converges; Save the trained multi-condition input segmentation neural network, input a forged face picture, use the trained multi-condition input segmentation neural network model to infer the picture, and the segmentation neural network model finally outputs a binary segmentation map of the same size as the original image to detect and locate the forged face area in the image.

2. The method for face forgery segmentation based on multi-condition input according to claim 1, characterized in that: The frequency domain transformation of the image and the extraction of high-frequency information from the image using a high-pass filter specifically include: For the RGB three-channel image I, after passing through the high-pass filter, the 3-channel high-frequency information F with the same size as the original image I is obtained. H .

3. The method for face forgery segmentation based on multi-condition input according to claim 1, characterized in that: The multi-condition input segmentation neural network specifically includes a network backbone VIT, a multi-condition input module and an image segmentation decoder.

4. The method for face forgery segmentation based on multi-condition input according to claim 3 is characterized in that: The network backbone VIT includes multiple Transformer encoder modules; Each Transformer encoder module contains a multi-head self-attention mechanism and a feed-forward neural network; The image input to the network backbone VIT is divided into image blocks of fixed size, and the image blocks are linearly embedded and sent to each Transformer encoder module as an input sequence; The Transformer encoder modules transfer features through residual connections, and an additional layer normalization operation is performed after each Transformer encoder module; Finally, the network backbone VIT outputs feature maps.

5. The method for face forgery segmentation based on multi-condition input according to claim 3 is characterized in that: The multi-condition input module includes multiple independent embedding networks and a shared multi-layer perceptron; Characteristic noise floor F N , original image I and high frequency information F H The features are extracted through their respective embedding networks, where each embedding network includes a shared downsampling MLP and independent MLP modules. The features extracted by each embedding network are fused through the cross-attention mechanism. The features from the network backbone VIT are used as queries, and the feature background noise, image features and high-frequency information are added as keys and values ​​and input into the cross-attention module. The fused features are normalized and then input into the shared upsampling MLP for further processing. Finally, they are accumulated layer by layer with the output of the upper layer to form the final multi-conditional input information to enhance the performance of the face forgery segmentation network.

6. The method for face forgery segmentation based on multi-condition input according to claim 5, characterized in that: The shared downsampling MLP and shared upsampling MLP are shared in each layer of the network backbone VIT to compress high-dimensional data features to low-dimensional ones; In each layer of the network backbone VIT, an independent MLP module is used to further transform the data features in low-dimensional features to adapt to the specific weights of the current network layer.

7. The method for face forgery segmentation based on multi-condition input according to claim 5, characterized in that: During the efficient fine-tuning training process, the model weights of the network backbone VIT are frozen, and the weights of the shared downsampling MLP, shared upsampling MLP, independent MLP modules and multi-conditional input modules are trained, so that the training parameters of the multi-conditional input segmentation neural network are controlled within 3% to ensure the efficiency of network training.

8. The method for face forgery segmentation based on multi-condition input according to claim 3, characterized in that: The image segmentation decoder includes two upsampling convolutional layers, two activation layers and a linear mapping layer; The upsampling convolutional layer is used to gradually restore the spatial resolution of the feature map, ensuring that the final output matches the size of the original image; Each upsampling convolution layer is followed by a nonlinear activation layer, and the activation function used in the nonlinear activation layer is a ReLU function or a Leaky ReLU function to introduce nonlinear features and enhance the expressiveness of the model; Finally, the linear mapping layer further processes the upsampled features and maps the features to a binary space through a linear transformation, thereby generating a final binary segmentation map that distinguishes the forged part from the real part.

9. The method for face forgery segmentation based on multi-condition input according to claim 8, characterized in that: Each pixel point of the binary segmentation image is mapped to a binary space, wherein 1 indicates that it is detected as a forged area, and 0 indicates that it is detected as a real area.

10. The method for face forgery segmentation based on multi-condition input according to claim 3, characterized in that: During the efficient fine-tuning training process, the binary segmentation map generated by the image segmentation decoder is also compared with the real binary segmentation map to calculate the value of the loss function; the loss function uses a cross entropy loss function or a Dice loss function, etc., to measure the difference between the predicted binary segmentation map and the real label; After calculating the value of the loss function, the back-propagation mechanism is used to pass the value of the loss function back to each layer of the multi-conditional input segmentation neural network, and the gradient descent algorithm is used to adjust the trainable parameter weights in the multi-conditional input segmentation neural network. This training process is repeated between the layers of the entire multi-conditional input segmentation neural network until the loss value of the model tends to stabilize, that is, the multi-conditional input segmentation neural network reaches a convergence state.

Images