Face forgery detection method and system based on multi-level discrimination

Abstract

The application provides a face forgery detection method and system based on multi-level discrimination, and belongs to the technical field of face recognition. The method comprises the following steps: inputting a to-be-detected face image into a visual encoder composed of a plurality of encoding layers connected in sequence to obtain a global visual feature vector corresponding to the to-be-detected face image; classifying the global visual feature vector to obtain a face forgery result corresponding to the to-be-detected face; the face forgery result comprises whether the face is forged and a forgery type; wherein in the visual encoder, the input vector of at least part of the encoding layers is obtained by fusing the output vector of the previous encoding layer and the visual cue vector corresponding to the previous encoding layer; the visual cue vector corresponding to each encoding layer is determined based on a visual cue pyramid; and each level of the visual cue pyramid is used to provide a visual cue vector of a different forgery type. The application can determine the forgery type of a face image while identifying the authenticity of the face image.

Description

Technical Field

[0001] This application belongs to the field of facial recognition technology, and more specifically, relates to a method and system for detecting facial forgery based on multi-level discrimination. Background Technology

[0002] With the rapid development of deep learning and generative models, technologies such as generative adversarial networks, diffusion models, and 3D reconstruction have driven the continuous iteration of face spoofing technology. Forgery forms such as face swapping, facial expression animation, and attribute editing are becoming increasingly realistic, posing serious challenges to fields such as information security and identity authentication. The importance of face spoofing detection technology is becoming increasingly prominent.

[0003] Currently, existing face forgery detection methods can classify and distinguish between genuine and fake face images. However, they can usually only distinguish between genuine and fake face images, and have certain shortcomings in determining the forgery type of face image. They cannot effectively distinguish the forgery category of face image, which limits the practical application effect of face forgery detection technology. Summary of the Invention

[0004] The purpose of this application is to provide a face forgery detection method and system based on multi-level discrimination, so as to realize the authenticity of face images and determine the forgery type of face images at the same time.

[0005] A first aspect of this application provides a face forgery detection method based on multi-level discrimination, comprising: The face image to be detected is input into a visual encoder consisting of multiple coding layers connected in sequence to obtain the global visual feature vector corresponding to the face image to be detected. The global visual feature vector is classified to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type; In the visual encoder, at least some of the input vectors of the encoding layers are obtained by fusing the output vector of the previous encoding layer with the visual cue vector corresponding to the previous encoding layer. The visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer. Each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types.

[0006] A second aspect of this application provides a face forgery detection system based on multi-level discrimination, comprising: The global visual vector module is used to input the face image to be detected into a visual encoder composed of multiple coding layers connected in sequence, and obtain the global visual feature vector corresponding to the face image to be detected. The face forgery detection module is used to classify global visual feature vectors to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type; In the visual encoder, at least some of the input vectors of the encoding layers are obtained by fusing the output vector of the previous encoding layer with the visual cue vector corresponding to the previous encoding layer. The visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer. Each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described face forgery detection method based on multi-level discrimination.

[0008] In a fourth aspect of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described face forgery detection method based on multi-level discrimination.

[0009] The beneficial effects of the face forgery detection method and system based on multi-level discrimination provided in this application are as follows: In this embodiment, the visual encoder consists of multiple sequentially connected coding layers, with at least some layers fusing the output of the previous layer with the corresponding visual cue vector. The visual cue pyramid contains forgery type cue vectors with different granularities of discrimination. The coding layers can retrieve matching cue vectors to assist in feature extraction, enabling the global visual feature vector extracted by the encoder to integrate multi-dimensional semantic information of the image and accurately capture the feature differences of different forgery types through multi-granularity cues, thus solving the problem that existing technologies cannot distinguish forgery types. Furthermore, this embodiment also classifies the global visual feature vector to simultaneously output whether it is forgery and the specific forgery type, providing a more comprehensive detection basis for fields such as information security and identity authentication, and meeting the practical needs of differentiated responses to different forgery types. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a face forgery detection method based on multi-level discrimination provided in an embodiment of this application; Figure 2 A flowchart illustrating another face forgery detection method based on multi-level discrimination provided in an embodiment of this application; Figure 3 A schematic diagram of a visual cue pyramid provided in one embodiment of this application; Figure 4 This is a structural block diagram of a face forgery detection system based on multi-level discrimination provided in an embodiment of this application; Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0014] Please refer to Figure 1 , Figure 1 The flowchart of a face forgery detection method based on multi-level discrimination provided in an embodiment of this application can be executed by an electronic device. The method may include: S101-S102.

[0015] S101: Input the face image to be detected into a visual encoder composed of multiple coding layers connected in sequence to obtain the global visual feature vector corresponding to the face image to be detected.

[0016] In this embodiment, the coding layer refers to the feature extraction module in the visual encoder. Each coding layer in the visual encoder is a serial structure connected in sequence. The coding layers at different positions are responsible for extracting different main features. Among them, the bottom coding layer is more inclined to extract local texture details in the input vector, the middle coding layer is more inclined to capture the geometric structure and local consistency in the input vector, and the high-level coding layer is more concerned with global semantics and identity consistency.

[0017] In this embodiment, the global visual feature vector refers to the high-bit feature vector output by the last layer of the visual encoder, which integrates key information of the low-level texture, mid-level structure and high-level semantics of the face image to be detected.

[0018] In this embodiment, in the visual encoder, the input vector of at least a portion of the coding layers is obtained by fusing the output vector of the preceding coding layer with the visual cue vector corresponding to the preceding coding layer; in one embodiment, such as Figure 2 As shown, Figure 2 The face image in the middle is the face image to be detected. BLOCK is the encoding layer. The input vectors of BLOCK2, BLOCK3, and BLOCK4 (not shown in the figure) are not the output vectors of their predecessor encoding layers, but rather vectors obtained by fusing the output vector of their predecessor encoding layers with the corresponding visual cue vectors (i.e., candidate cue vectors). In this embodiment, the fusion method can be concatenation.

[0019] In this embodiment, the visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer.

[0020] In one embodiment, each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types. For example, refer to Figure 3 The visual cue pyramid can include three levels: a true / false cue layer, a fake category cue layer, and a fake sub-category cue layer. The true / false cue layer stores visual cue vectors that characterize the authenticity of a face image. The fake category cue layer stores visual cue vectors that characterize the fake category attribute of a face image. The fake sub-category cue layer stores visual cue vectors that characterize the fake sub-category attribute of a face image. Each fake category attribute corresponds to at least one fake sub-category attribute.

[0021] In this embodiment, the first layer (true / false prompting layer) has the largest granularity in distinguishing forgery types and is only used to indicate the authenticity of the input vector; the second layer (forgery category prompting layer) has a smaller granularity in distinguishing forgery types compared to the first layer and is used to indicate the forgery category of the input vector, such as generation, manipulation, and adversarial types; the third layer (forgery sub-category prompting layer) has the smallest granularity in distinguishing forgery types and is used to indicate the forgery sub-category of the output vector, that is, to subdivide the forgery types, such as identity consistency, style transfer, attribute swapping, face swapping, pixelation, semantics, facial animation, attribute editing, local pixel attacks, and semantically driven anomalies, etc.

[0022] In this embodiment, combined with Figure 2 and Figure 3 The visual cue pyramid contains three levels that correspond one-to-one with the first three coding layers of the visual encoder: the true / false cue layer corresponds to the first coding layer, the fake major category cue layer corresponds to the second coding layer, and the fake minor category cue layer corresponds to the third coding layer.

[0023] In one embodiment, retrieving the visual cue vector matching the output vector of the coding layer from the corresponding layer includes: using the visual cue vector with the highest similarity to the output vector of the coding layer in the corresponding layer as the visual cue vector matching the output vector of the coding layer. In this embodiment, the similarity can be calculated based on cosine similarity or Euclidean distance, etc.

[0024] S102: Classify the global visual feature vector to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type.

[0025] In this embodiment, the face forgery result corresponding to the detected face can be obtained by classifying the global visual feature vector based on a multilayer perceptron. The calculation formula is as follows: ,in, This represents the output of the multilayer perceptron, which can be either real (real face) or fake (fake face). This represents the classification process of a multilayer perceptron. This represents the global visual feature vector.

[0026] As can be seen from the above, the bottom, middle, and top coding layers of the visual encoder in this embodiment each have their own functions. The bottom layer extracts texture details, the middle layer captures geometric structures, and the top layer integrates global semantics to achieve comprehensive hierarchical coverage of image features. Simultaneously, the first three coding layers correspond to the real / fake, forgery category, and forgery sub-category hierarchical prompts of the visual cue pyramid. By retrieving matching cue vectors through similarity retrieval, each coding layer's features incorporate corresponding granular forgery prior knowledge. The bottom layer integrates real / fake cue prompts to strengthen the distinction between real and fake, the middle layer integrates category prompts to clarify generation manipulation and other category directions, and the top layer integrates sub-category prompts to capture subdivided type features such as face swapping / attribute editing. After the above features are integrated into a global visual feature vector, classification can simultaneously output the real / fake result and the specific forgery type, filling the capability gap in existing technologies.

[0027] Considering that in the existing technology, after performing authenticity recognition on a face image, it is usually impossible to identify abnormal regions in the face image, based on this, in one embodiment of this application, the face forgery detection method based on multi-level discrimination further includes: fusing the visual cue vectors corresponding to each coding layer to obtain a fused cue vector; The forgery mask corresponding to the face forgery result is determined based on the global visual feature vector and the fused cue vector; the forgery mask is used to indicate abnormal regions in the face image to be detected.

[0028] In this embodiment, the visual cue vector corresponding to each coding layer refers to the visual cue vector retrieved from the corresponding level of the pyramid by each coding layer participating in cue fusion in the visual encoder, which matches the output features of that layer.

[0029] In this embodiment, the visual cue vectors corresponding to each coding layer can be concatenated to obtain a fused cue vector. The fused cue vector contains all prior information on true / false discrimination, forgery category, and forgery subtype, providing a comprehensive representation of forgery features.

[0030] In this embodiment, please refer to Figure 2 Following the connection order of multiple coding layers, the first to third coding layers can perform visual cue vector matching of the visual cue pyramid. In this embodiment, the visual cue vectors retrieved from the three levels are represented as follows: ,right The fusion and splicing process yields the fusion hint vector. .

[0031] In this embodiment, determining the forgery mask corresponding to the face forgery result based on the global visual feature vector and the fused cue vector includes: transforming the global visual feature vector and the fused cue vector based on a multilayer perceptron, and performing attention fusion on the transformed global visual feature vector and the transformed fused cue vector to obtain a spatial attention map; generating the forgery mask corresponding to the face forgery result based on the spatial attention map.

[0032] In one embodiment, the transformed global visual feature vector can be represented as: The transformed fusion cue vector is represented as ,in, Represents the global visual feature vector. This represents the spatial mapping process of a multilayer perceptron. This describes the process of classifying the types of multilayer perceptrons. , Representing a spatial attention map, This represents the attention mechanism. , This represents a pixel-level forgery region localization map, used to visualize forged evidence. In this embodiment, the forgery mask can be used to indicate abnormal regions in the face image to be detected, with accuracy down to the pixel level.

[0033] As can be seen from the above, in this embodiment, the true / false, forgery category, and forgery sub-category hint vectors matched by each coding layer are fused into a fused hint vector containing full-granularity forgery priors, which covers forgery feature information from true / false judgment to subdivided types; at the same time, it combines and integrates global visual feature vectors of image bottom texture and middle structure, and after being transformed by a multilayer perceptron, it uses an attention mechanism to match image spatial features and forgery knowledge features, generates a spatial attention map, and transforms it into a pixel-level forgery mask, thereby realizing the localization of abnormal regions in face images.

[0034] In one embodiment of this application, the visual cue pyramid is trained in the following manner: Obtain multiple training samples; each training sample contains: a training face image and corresponding labels for each forgery type. Each training face image is input into a visual encoder to obtain multiple training visual feature vectors corresponding to each training face image. For each training visual feature vector and each level of the initial visual cue pyramid, a target cue vector matching the training visual feature vector is retrieved from that level, and the target cue vector is fused with the training visual feature vector to obtain the level-enhanced features. The initial visual cue pyramid is trained based on a preset loss function until the loss function converges, thus obtaining the visual cue pyramid. The loss function is constructed based on the multi-task loss function and the hierarchical consistency constraint function. The multi-task loss function is constructed based on the hierarchical enhancement features of each level and the corresponding fake attribute labels, and is used to converge the initial visual cue vectors in different fake attribute dimensions in the initial visual cue pyramid. The hierarchical consistency constraint function is constructed based on the semantic distribution of the visual cue vectors of each level after projection transformation, and is used to constrain the visual cue vectors of different levels to maintain a preset semantic inclusion relationship.

[0035] In this embodiment, the training samples are represented as ,in This represents the i-th training face image. These represent the first-level true / false label, the second-level fake category label, and the third-level fake subcategory label of the i-th training face image, respectively. Multiple training visual feature vectors are obtained after feature extraction by the visual encoder. ,in This represents the feature extraction operation of the visual encoder. The visual cue pyramid contains the following three layers of cue libraries: , l∈{1,2,3}. Where The dimensional space representing the visual cue vector.

[0036] In this embodiment, each initial visual cue vector in the initial visual cue pyramid is a learnable parameter. During training, corresponding cues are selected based on labels and fused with visual features: ,in, This represents the hierarchical enhancement feature of the i-th training face image at the l-th level. This represents the target cue vector that matches the label of the i-th training sample in the l-th level.

[0037] In this embodiment, the multi-task loss function can be determined as follows: ,in , This represents the predicted label of the i-th training face image at layer l. This represents the true label of the i-th training face image at layer l. The predicted label is obtained by inputting the corresponding layer-enhanced features into the multilayer perceptron classifier.

[0038] In this embodiment, the hierarchical consistency constraint function can be determined as follows: Here, KL refers to the KL divergence, an index used to measure the similarity between two probability distributions. , These are all weighting coefficients, which can be set based on experience or preferences. This refers to the semantic distribution of the visual cue vector at layer l of the i-th training sample, where... , This represents the projection matrix of the l-th layer. Let represent the visual cue vector corresponding to the i-th training sample and the l-th layer. The predefined loss function can be expressed as: .

[0039] In this embodiment, the preset semantic inclusion relationship refers to the correspondence between each major category of forgery and the minor categories of forgery it contains. In this embodiment, the criterion for judging the convergence of the loss function can be that the difference between the loss functions of multiple consecutive iterations is less than a preset loss value. During training, all visual cue vectors participate in backpropagation. After training, the parameters and vectors in the visual cue pyramid are frozen for dynamic retrieval and cue enhancement during the inference phase.

[0040] As can be seen from the above, the embodiments of this application, by combining a multi-task loss function, not only highlight the core position of authenticity discrimination but also supervise the precise learning of forgery features at the corresponding granularity by each level of cue vector; the hierarchical consistency constraint in the embodiments of this application quantifies semantic distribution differences through KL divergence, forcing the forgery subclasses. Counterfeit categories The forged inclusion relationship avoids hierarchical semantic conflicts. During training, the hint vector participates in backpropagation throughout the entire process, fully absorbing the forged knowledge in the training data. After training, the parameters are frozen to ensure inference stability, and the optimal hint can be quickly matched during dynamic retrieval.

[0041] In one embodiment of this application, the number of multiple coding layers is n; According to the connection order of the n coding layers, the input of the first coding layer is the face image to be detected; the input vectors of the second to m-th coding layers are obtained by fusing the output vector of the previous coding layer with the visual cue vector corresponding to the previous coding layer. The inputs of the (m+1)th encoder layer to the nth encoder are the output vectors of the previous encoder layer; n and m are both positive integers, and m is less than n.

[0042] In this embodiment, m is set to 4, meaning that the output vectors of the first three encoding layers are matched with the corresponding levels in the visual cue pyramid to obtain visual cue vectors. Preferably, the encoding layers ranked earlier in the visual encoder are matched with the corresponding levels in the visual cue pyramid to obtain visual cue vectors. This allows subsequent encoding layers to extract features based on the matched visual cue vectors, improving the sensitivity of global features to forgery details.

[0043] Corresponding to the face forgery detection method based on multi-level discrimination in the above embodiments, Figure 4 This is a structural block diagram of a face forgery detection system based on multi-level discrimination, provided as an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 4The face forgery detection system 20 based on multi-level discrimination includes: a global visual vector module 21 and a face forgery detection module 22.

[0044] Among them, the global visual vector module 21 is used to input the face image to be detected into a visual encoder composed of multiple coding layers connected in sequence to obtain the global visual feature vector corresponding to the face image to be detected. The face forgery detection module 22 is used to classify the global visual feature vector to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type; In the visual encoder, at least some of the input vectors of the encoding layers are obtained by fusing the output vector of the previous encoding layer with the visual cue vector corresponding to the previous encoding layer. The visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer. Each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types.

[0045] In one embodiment of this application, the face forgery detection system 20 based on multi-level discrimination further includes: an abnormal region identification module, used to fuse the visual cue vectors corresponding to each coding layer to obtain a fused cue vector; The forgery mask corresponding to the face forgery result is determined based on the global visual feature vector and the fused cue vector; the forgery mask is used to indicate abnormal regions in the face image to be detected.

[0046] In one embodiment of this application, the face forgery detection system 20 based on multi-level discrimination further includes: a visual cue pyramid training module, used to acquire multiple training samples; each training sample contains: a training face image and various forgery type labels corresponding to the training face image; Each training face image is input into a visual encoder to obtain multiple training visual feature vectors corresponding to each training face image. For each training visual feature vector and each level of the initial visual cue pyramid, a target cue vector matching the training visual feature vector is retrieved from that level, and the target cue vector is fused with the training visual feature vector to obtain the level-enhanced features. The initial visual cue pyramid is trained based on a preset loss function until the loss function converges, thus obtaining the visual cue pyramid. The loss function is constructed based on the multi-task loss function and the hierarchical consistency constraint function. The multi-task loss function is constructed based on the hierarchical enhancement features of each level and the corresponding fake attribute labels, and is used to converge the initial visual cue vectors in different fake attribute dimensions in the initial visual cue pyramid. The hierarchical consistency constraint function is constructed based on the semantic distribution of the visual cue vectors of each level after projection transformation, and is used to constrain the visual cue vectors of different levels to maintain a preset semantic inclusion relationship.

[0047] In one embodiment of this application, the number of multiple coding layers is n; according to the connection order of the n coding layers, the input of the first coding layer is the face image to be detected; the input vectors of the second to the m-th coding layers are obtained by fusing the output vector of the previous coding layer with the visual cue vector corresponding to the previous coding layer; The inputs of the (m+1)th encoder layer to the nth encoder are the output vectors of the previous encoder layer; n and m are both positive integers, and m is less than n.

[0048] In one embodiment of this application, the visual cue pyramid contains at least three levels: a true / false cue layer, a fake major category cue layer, and a fake minor category cue layer; The system includes a true / false prompt layer that stores visual cue vectors representing the authenticity of face images; a forgery category prompt layer that stores visual cue vectors representing the forgery category of face images; a forgery sub-category prompt layer that stores visual cue vectors representing the forgery sub-category of face images; and each forgery category attribute corresponds to at least one forgery sub-category attribute.

[0049] In one embodiment of this application, the visual cue vector retrieved from the layer corresponding to the coding layer that matches the output vector of the coding layer includes: The visual cue vector with the highest similarity to the output vector of the corresponding encoding layer is taken as the visual cue vector that matches the output vector of the encoding layer.

[0050] In one embodiment of this application, multiple levels of the visual cue pyramid correspond one-to-one with a portion of the coding layer.

[0051] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the global visual vector module 21 and the face forgery detection module 22 are shown.

[0052] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0053] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0054] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0055] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the multi-level discrimination face forgery detection method provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0056] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0057] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0058] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0059] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0060] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0062] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0064] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A face forgery detection method based on multi-level discrimination, characterized in that, include: The face image to be detected is input into a visual encoder consisting of multiple coding layers connected in sequence to obtain the global visual feature vector corresponding to the face image to be detected. The global visual feature vector is classified to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type; In the visual encoder, at least some of the input vectors of the encoding layers are obtained by fusing the output vector of the previous encoding layer with the visual cue vector corresponding to the previous encoding layer; the visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer. Each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types.

2. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, Also includes: The visual cue vectors corresponding to each coding layer are fused to obtain the fused cue vector; The forgery mask corresponding to the face forgery result is determined based on the global visual feature vector and the fusion cue vector; the forgery mask is used to indicate abnormal regions in the face image to be detected.

3. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, The visual cue pyramid was trained in the following manner: Multiple training samples are obtained; each training sample contains: a training face image and various forgery type labels corresponding to the training face image; Each training face image is input into the visual encoder to obtain multiple training visual feature vectors corresponding to each training face image. For each training visual feature vector and each level of the initial visual cue pyramid, a target cue vector matching the training visual feature vector is retrieved from that level, and the target cue vector is fused with the training visual feature vector to obtain the level-enhanced feature. The initial visual cue pyramid is trained based on a preset loss function until the loss function converges, thus obtaining the visual cue pyramid. The loss function is constructed based on a multi-task loss function and a hierarchical consistency constraint function. The multi-task loss function is constructed based on the hierarchical enhancement features of each level and the corresponding fake attribute labels, and is used to converge the initial visual cue vectors in the initial visual cue pyramid at different fake attribute dimensions. The hierarchical consistency constraint function is constructed based on the semantic distribution of the visual cue vectors of each level after projection transformation, and is used to constrain the visual cue vectors of different levels to maintain a preset semantic inclusion relationship.

4. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, The number of the plurality of coding layers is n; According to the connection order of the n coding layers, the input of the first coding layer is the face image to be detected; the input vectors of the second to the m-th coding layers are obtained by fusing the output vector of the previous coding layer with the visual cue vector corresponding to the previous coding layer. The inputs of the (m+1)th encoder layer to the nth encoder are the output vectors of the previous encoder layer; n and m are both positive integers, and m is less than n.

5. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, The visual cue pyramid contains at least three levels: a true / false cue layer, a fake major category cue layer, and a fake minor category cue layer; The true / false prompt layer stores visual cue vectors used to characterize the true / false attributes of a face image; the forgery category prompt layer stores visual cue vectors used to characterize the forgery category attributes of a face image; the forgery sub-category prompt layer stores visual cue vectors used to characterize the forgery sub-category attributes of a face image; and each forgery category attribute corresponds to at least one forgery sub-category attribute.

6. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, The visual cue vector retrieved in the layer corresponding to the coding layer and matching the output vector of the coding layer includes: The visual cue vector with the highest similarity to the output vector of the corresponding encoding layer is taken as the visual cue vector that matches the output vector of the encoding layer.

7. The face forgery detection method based on multi-level discrimination as described in claim 1, characterized in that, The multiple levels of the visual cue pyramid correspond one-to-one with the partial coding layers.

8. A face forgery detection system based on multi-level discrimination, characterized in that, include: The global visual vector module is used to input the face image to be detected into a visual encoder composed of multiple coding layers connected in sequence to obtain the global visual feature vector corresponding to the face image to be detected. The face forgery detection module is used to classify the global visual feature vector to obtain the face forgery result corresponding to the face to be detected; the face forgery result includes whether it is forged and the forgery type; In the visual encoder, at least some of the input vectors of the encoding layers are obtained by fusing the output vector of the previous encoding layer with the visual cue vector corresponding to the previous encoding layer; the visual cue vector corresponding to each encoding layer is: after determining the level corresponding to the encoding layer in the multiple levels of the pre-trained visual cue pyramid, the visual cue vector that matches the output vector of the encoding layer is retrieved in the level corresponding to the encoding layer. Each level of the visual cue pyramid is used to provide visual cue vectors for different types of forgery, and different levels have different granularities for distinguishing forgery types.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images