Face forgery detection method, system, device and medium based on dual modal

Abstract

The application discloses a face forgery detection method and system based on a dual mode, equipment and a medium. The method comprises the following steps: inputting a face image into a first shallow convolutional neural network to extract RGB information and obtain an RGB feature; inputting a high-frequency image into a second shallow convolutional neural network to extract high-frequency information and obtain a high-frequency feature; wherein a first feature output by a first convolutional block in a current layer of the first shallow convolutional neural network and a second feature output by a second convolutional block in a corresponding layer of the second shallow convolutional neural network are interactively learned through a preset information interaction module to obtain a third feature, and the third feature is added to the first feature to serve as an input of a first convolutional block in a next layer of the first shallow convolutional neural network; and the RGB feature and the high-frequency feature are combined and input into a classifier to obtain the authenticity of an RGB face image. Through the two modes and the interactive learning of the two modes, the accuracy and the generalization performance of face forgery detection are improved.

Description

Technical Field

[0001] This invention relates to the field of face anti-spoofing technology, and in particular to a face forgery detection method, system, device and medium based on dual-modality. Background Technology

[0002] With advancements in facial spoofing technology, it has become increasingly difficult for the human eye to distinguish whether an image has been manipulated. Malicious users have abused this technology to launch attacks, causing significant harm to society. Therefore, developing an effective method for detecting facial spoofing is extremely important.

[0003] Current techniques attempt to mine artifacts in the feature space using CNNs. While achieving significant performance on high-quality images, performance degrades drastically when subjected to image compression or noise. Furthermore, most methods focus on in-database detection, meaning that forged images in both the training and test sets are synthesized using the same algorithm, neglecting generalization. Some methods utilize frequency domain information, but the information contained is insufficient and the relationships between different information types are not considered. Additionally, CNN-based methods are susceptible to the influence of image content features, resulting in the lack of learned discriminative features and limited generalization ability. These face forgery detection methods generally fail to achieve satisfactory performance, thus limiting their wider application. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a dual-modal face forgery detection method, system, device, and medium. It constructs a dual-stream network using RGB information and high-frequency information, considering the relationship between the two modalities to promote the learning of the convolutional neural network, improving intra-domain accuracy without sacrificing generalization. Furthermore, through an information interaction module, it enables deeper interactive learning between the two modalities, learning more comprehensive features, making it less susceptible to the influence of image content features, and further improving the accuracy of face forgery detection.

[0005] In a first aspect, embodiments of the present invention provide a bimodal face forgery detection method, the bimodal face forgery detection method comprising:

[0006] Acquire RGB face images and extract high-frequency images from the RGB face images;

[0007] The RGB face image is input into a preset first shallow convolutional neural network for RGB information extraction, resulting in RGB features output by the first shallow convolutional neural network. The high-frequency image is input into a preset second shallow convolutional neural network for high-frequency information extraction, resulting in high-frequency features output by the second shallow convolutional neural network. A preset information interaction module is used to interactively learn the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block of the second shallow convolutional neural network to obtain a third feature. The third feature is then added to the first feature and used as the input to the next first convolutional block in the first shallow convolutional neural network.

[0008] The RGB features and the high-frequency features are combined to obtain the combined features;

[0009] The merged features are input into a preset classifier to obtain the authenticity of the RGB face image output by the classifier.

[0010] The method according to embodiments of the present invention has at least the following beneficial effects:

[0011] High-frequency images are obtained by processing RGB images of faces. RGB and high-frequency information, along with the relationship between the two modalities, are used to promote the learning of the convolutional neural network, improving intra-domain accuracy without sacrificing generalization. First and second shallow convolutional neural networks are used to extract RGB and high-frequency features respectively, enhancing the ability to detect subtle forgery traces. An information interaction module is used to consider the interrelationship between the two modalities, enabling information exchange between the two streams to learn more comprehensive features. RGB and high-frequency features are then merged to obtain combined features, fusing RGB and high-frequency information to promote the learning of the entire network model. Finally, a classifier determines the authenticity of face images. Through sufficient learning, the classifier can quickly and accurately determine the authenticity of face images, improving the accuracy and efficiency of face forgery detection.

[0012] According to some embodiments of the present invention, extracting high-frequency images from the RGB face image includes:

[0013] The RGB face image is transformed to the frequency domain using DCT for each of the three RGB channels to obtain a spectral image.

[0014] The high-frequency information is obtained by filtering out low-frequency information from the spectrum image using a binary filter.

[0015] The high-frequency information is transformed back into the RGB color space using IDCT to obtain the high-frequency image.

[0016] According to some embodiments of the present invention, the method for constructing the information interaction module includes:

[0017] Set a filter kernel with the same dimension as the input features;

[0018] The filter kernel is connected to the convolutional blocks of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to form the information interaction module.

[0019] According to some embodiments of the present invention, the information interaction module is set starting from the second convolutional block of the first shallow convolutional neural network and the second shallow convolutional neural network, and each corresponding convolutional block of the first shallow convolutional neural network and the second shallow convolutional neural network is provided with one such information interaction module.

[0020] According to some embodiments of the present invention, the training method of the classifier includes:

[0021] End-to-end training is performed using binary cross-entropy loss, and the calculation formula for the training includes:

[0022]

[0023] Where N represents the number of training samples, and i represents the index of the training sample. Let y represent the predicted value of the i-th sample. i Let y represent the label of the i-th sample. If the sample category is false, then y i Set it to 1, otherwise set it to 0.

[0024] According to some embodiments of the present invention, both the first shallow convolutional neural network and the second shallow convolutional neural network use the Xception network; the Xception network only retains the input stream and the output stream.

[0025] According to some embodiments of the present invention, the training method for the filter kernel includes:

[0026] The filter kernel is trained using standard backpropagation and gradient descent.

[0027] Secondly, embodiments of the present invention provide a bimodal face forgery detection system, the bimodal face forgery detection system comprising:

[0028] An image acquisition module is used to acquire RGB face images and extract high-frequency images from the RGB face images;

[0029] An adaptive feature interaction module is used to input the RGB face image into a preset first shallow convolutional neural network for RGB information extraction, obtaining the RGB features output by the first shallow convolutional neural network; and to input the high-frequency image into a preset second shallow convolutional neural network for high-frequency information extraction, obtaining the high-frequency features output by the second shallow convolutional neural network. Specifically, a preset information interaction module interacts and learns the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block in the second shallow convolutional neural network to obtain a third feature. The third feature is then added to the first feature and used as the input to the next first convolutional block in the first shallow convolutional neural network.

[0030] The feature fusion module is used to merge the RGB features and the high-frequency features to obtain merged features;

[0031] The image authenticity determination module is used to input the merged features into a preset classifier to obtain the authenticity status of the RGB face image output by the classifier.

[0032] Thirdly, embodiments of the present invention provide an electronic device including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the bimodal face forgery detection method as described in the first aspect.

[0033] Fourthly, embodiments of the present invention provide a computer storage medium storing computer-executable instructions for causing a computer to perform the bimodal face forgery detection method as described in the first aspect.

[0034] It should be noted that the beneficial effects of the second to fourth aspects of the present invention compared with the prior art are the same as the beneficial effects of the bimodal face forgery detection method of the first aspect, and will not be described in detail here.

[0035] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0037] Figure 1This is a flowchart of a face forgery detection method based on dual-modality provided in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of extracting high-frequency images from RGB face images according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of a method for constructing an information interaction module according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of a face forgery detection method based on dual-modality provided in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of an information interaction module provided in an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram illustrating a specific implementation of a dual-modal face forgery detection method provided in an embodiment of the present invention;

[0043] Figure 7 This is a structural diagram of a dual-modal face forgery detection system provided in an embodiment of the present invention;

[0044] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0046] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0047] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0048] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0049] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are some embodiments of the present invention, not all embodiments.

[0050] Reference Figure 1 In some embodiments of the present invention, a face forgery detection method based on dual-modality is provided, comprising:

[0051] Step S100: Obtain an RGB face image and extract high-frequency images from the RGB face image.

[0052] Step S200: Input the RGB face image into a preset first shallow convolutional neural network to extract RGB information and obtain the RGB features output by the first shallow convolutional neural network. Input the high-frequency image into a preset second shallow convolutional neural network to extract high-frequency information and obtain the high-frequency features output by the second shallow convolutional neural network. In this process, through a preset information interaction module, the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block in the second shallow convolutional neural network are interactively learned to obtain a third feature. The third feature is then added to the first feature and used as the input to the first convolutional block of the next layer in the first shallow convolutional neural network.

[0053] Step S300: Merge RGB features and high-frequency features to obtain merged features.

[0054] Step S400: Input the merged features into a preset classifier to obtain the authenticity of the RGB face image output by the classifier.

[0055] First, in step S100, the RGB image of the face is processed to obtain a high-frequency image. RGB and high-frequency information are used, and the relationship between the two modalities is considered to promote the learning of the convolutional neural network, improving intra-domain accuracy without sacrificing generalization. In step S200, a first shallow convolutional neural network and a second shallow convolutional neural network are used to extract RGB and high-frequency features respectively, enhancing the ability to detect subtle forgery traces. Simultaneously, step S200 uses an information interaction module to consider the interrelationship between the two modalities, realizing information interaction between the two streams to learn more comprehensive features. Then, in step S300, the RGB and high-frequency features are merged to obtain merged features, fusing RGB and high-frequency information to promote the learning of the entire network model. Finally, in step S400, a classifier determines the authenticity of the face image. Through the classifier's thorough learning, the authenticity of the face image can be quickly and accurately determined, improving the accuracy and efficiency of face forgery detection.

[0056] Reference Figure 2 In some embodiments of the present invention, extracting high-frequency images from RGB face images includes:

[0057] Step S110: Use DCT to transform the RGB face image to the frequency domain for each of the three RGB channels to obtain a spectrum image.

[0058] Step S120: Filter the spectrum image through a binary filter to remove low-frequency information and obtain high-frequency information.

[0059] Step S130: Convert the high-frequency information back to the RGB color space using IDCT to obtain a high-frequency image.

[0060] Preferably, the size of the input RGB face image is 3×H×W, where 3 represents the three RGB channels, and H and W correspond to the height and width, respectively.

[0061] It should be noted that transforming high-frequency information back to the RGB color space using IDCT will preserve the shift invariance and local consistency of the frequency domain and RGB face images.

[0062] By transforming RGB face images into high-frequency images, more information that is difficult to notice in RGB face images can be extracted. At the same time, the shift invariance and local consistency of high-frequency images can ensure that there are no judgment errors caused by changes.

[0063] Reference Figure 3 In some embodiments of the present invention, the method for constructing the information interaction module includes:

[0064] Step S210: Set a filter kernel with the same dimension as the input features.

[0065] Step S220: Connect the filter kernel to the convolutional blocks of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to form an information interaction module.

[0066] It should be noted that the filter kernel is shared to project the feature maps of the two modalities into the same space. The feature vectors of the first and second shallow convolutional neural networks are filtered by dot product. The results are then merged across channels and passed to the softmax operator to obtain the weights for the two corresponding streams. Finally, the weights are multiplied by the features of the first and second shallow convolutional neural networks respectively to obtain the final fused feature (i.e., the third feature mentioned above).

[0067] By using filter kernels to fuse RGB and high-frequency information at different semantic layers of the first and second shallow convolutional neural networks, the first and second shallow convolutional neural networks can learn more deeply, enabling them to discover more feature information and improve the accuracy of face image anti-spoofing.

[0068] In some embodiments of the present invention, the information interaction module is set up starting from the second convolutional block of the first shallow convolutional neural network and the second shallow convolutional neural network, and each corresponding convolutional block of the first shallow convolutional neural network and the second shallow convolutional neural network is provided with an information interaction module.

[0069] It should be noted that the first and second shallow convolutional neural networks need to first perform a first convolution on the RGB face image and high-frequency image through the first convolutional block before inputting them into the second convolutional block. Then, an information interaction module is set in each convolutional block after the second convolutional block. The original RGB face image and high-frequency image are first preprocessed by convolution through the first convolutional block, but no information interaction module is added here to prevent misleading information. Then, an information interaction module is set in each convolutional block after the second convolutional block to ensure sufficient information interaction between the two modal layers of the first and second shallow convolutional neural networks, ensuring generalization ability and improving anti-spoofing accuracy.

[0070] In some embodiments of the present invention, the method for training the classifier includes:

[0071] End-to-end training is performed using binary cross-entropy loss. The training calculation formula includes:

[0072]

[0073] Where N represents the number of training samples, and i represents the index of the training sample. Let y represent the predicted value of the i-th sample. i Let y represent the label of the i-th sample. If the sample category is false, then y i Set it to 1, otherwise set it to 0.

[0074] End-to-end training using binary cross-entropy loss ensures high accuracy of the classifier, while also ensuring that merged feature inputs to the classifier can quickly obtain judgment results, thus improving the performance of the anti-counterfeiting method.

[0075] In some embodiments of the present invention, both the first shallow convolutional neural network and the second shallow convolutional neural network use the Xception network; the Xception network only retains the input stream and the output stream.

[0076] The Xception network simplifies the first and second shallow convolutional neural networks, making the convergence process faster and more accurate, thus greatly improving the performance of the first and second shallow convolutional neural networks.

[0077] In some embodiments of the present invention, the training method for the filter kernel includes:

[0078] The filter kernel is trained using standard backpropagation and gradient descent.

[0079] The backpropagation algorithm can be used to update the parameters in a neural network using a loss function, making the network output as close as possible to the true value. The backpropagation algorithm uses the chain rule to differentiate the parameters from the last loss function to each layer of the network, obtain the gradient, and then use the gradient descent algorithm to update the parameters, so that the output of the model's loss function is as small as possible, and the loss function is optimized to ensure the accuracy of face image anti-spoofing.

[0080] Reference Figure 4 and Figure 6 To facilitate understanding by those skilled in the art, a specific embodiment of the present invention provides a face forgery detection method based on dual-modality, comprising:

[0081] The first step is to implement this using the PyTorch deep learning framework. The FaceForensics++ dataset, which includes both high-quality and low-quality videos, is used. 720 videos are selected for training, and 140 videos are used for validation and testing. For testing generalization, 518 videos from the CelebDF dataset are used. After frame-splitting the videos, dlib technology is used to detect faces and extract face region images. The input images are then resized to 256×256 and fed into the constructed network.

[0082] The second step involves transforming the three channels of the input RGB face image to the frequency domain using DCT. Then, a binary filter is used to remove low-frequency information to obtain high-frequency information. Next, IDCT is used to inversely transform the high-frequency information back to the RGB color space to obtain the high-frequency image, which serves as the input to the high-frequency stream. Feature extraction is performed on the RGB and high-frequency streams respectively, using a simplified Xception network. The input and output streams of Xception are preserved, and the input stream is divided into four convolutional blocks. To effectively facilitate feature interaction between the two streams, a specially designed information interaction module is added between the second, third, and fourth convolutional blocks. (Refer to...) Figure 5 The features to be interacted are input into the information interaction module. First, both features are multiplied by a shared filter kernel, the size of which is the same as the input feature size. Then, the result is concatenated and passed to the softmax operator to obtain weights. These weights are then used to weight and sum the two interacting features to obtain a fused feature. Finally, the fused feature is added to the RGB stream. The entire process in the information interaction module only requires learning the filter kernel, which can be updated using standard backpropagation and gradient descent. At the end of the network, the features learned from the two branches are concatenated and then input into the classifier to determine whether the input image is real or fake.

[0083] During network training, binary cross-entropy loss is used to calculate the loss and backpropagation is performed to update the network weights. Training stops when a set condition is met, resulting in a trained network. The image to be detected is then input into the trained network to identify fake faces. The test results on the FaceForensics++ face fake detection dataset are shown in Table 1, where HQ represents high quality and LQ represents low quality.

[0084] Table 1

[0085]

[0086]

[0087] The generalization test results on the CelebDF dataset are shown in Table 2.

[0088] Table 2

[0089]

[0090] Reference Figure 7 An embodiment of the present invention also provides a dual-modal face forgery detection system, including an image acquisition module 1001, an adaptive feature interaction module 1002, a feature fusion module 1003, and an image authenticity judgment module 1004, wherein:

[0091] The image acquisition module 1001 is used to acquire RGB face images and extract high-frequency images from the RGB face images.

[0092] The adaptive feature interaction module 1002 is used to input RGB face images into a preset first shallow convolutional neural network for RGB information extraction, obtaining RGB features output by the first shallow convolutional neural network, and input high-frequency images into a preset second shallow convolutional neural network for high-frequency information extraction, obtaining high-frequency features output by the second shallow convolutional neural network. Specifically, through the preset information interaction module, the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block in the second shallow convolutional neural network are interactively learned to obtain a third feature. The third feature is then added to the first feature and used as the input to the next first convolutional block in the first shallow convolutional neural network.

[0093] The feature fusion module 1003 is used to merge RGB features and high-frequency features to obtain merged features.

[0094] The image authenticity judgment module 1004 is used to input the merged features into a preset classifier to obtain the authenticity status of the RGB face image output by the classifier.

[0095] It should be noted that since the bimodal face forgery detection system in this embodiment is based on the same inventive concept as the bimodal face forgery detection method described above, the corresponding content in the method embodiment is also applicable to this device embodiment, and will not be described in detail here.

[0096] refer to Figure 8 In another embodiment of the present invention, an electronic device 6000 is also provided, which can be any type of smart terminal, such as a mobile phone, tablet computer, personal computer, etc.

[0097] Specifically, the electronic device 6000 includes: one or more control processors 6001 and memory 6002. Figure 8 Taking a control processor 6001 and a memory 6002 as an example, the control processor 6001 and the memory 6002 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0098] The memory 6002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to an electronic device in an embodiment of the present invention.

[0099] The control processor 6001 executes various functional applications and data processing of a bimodal face forgery detection method by running non-transient software programs, instructions, and modules stored in the memory 6002, thereby implementing a bimodal face forgery detection method according to the above method embodiment.

[0100] The memory 6002 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created using a bimodal face forgery detection method, etc. Furthermore, the memory 6002 may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 6002 may optionally include memory remotely located relative to the control processor 6001, and these remote memories can be connected to the electronic device 6000 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] One or more modules are stored in memory 6002. When executed by one or more control processors 6001, a dual-modality face forgery detection method in the above method embodiments is executed, for example, the method described above. Figures 1 to 3 The method and steps.

[0102] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] It should be noted that since the electronic device in this embodiment is based on the same inventive concept as the aforementioned dual-modal face forgery detection method, the corresponding content in the method embodiment is also applicable to this device embodiment, and will not be described in detail here.

[0104] One embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions for performing: the bimodal face forgery detection method as described in the above embodiment.

[0105] It should be noted that since the computer-readable storage medium in this embodiment is based on the same inventive concept as the bimodal face forgery detection method described above, the corresponding content in the method embodiment is also applicable to this device embodiment, and will not be described in detail here.

[0106] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired data and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any data delivery medium.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A face forgery detection method based on dual-modality, characterized in that, The dual-modal face forgery detection method includes: Acquire an RGB face image and extract a high-frequency image from the RGB face image; the extraction of the high-frequency image from the RGB face image includes: transforming the RGB face image to the frequency domain using DCT for each of the three RGB channels to obtain a spectrum image; filtering out low-frequency information from the spectrum image using a binary filter to obtain high-frequency information; and transforming the high-frequency information back to the RGB color space using IDCT to obtain the high-frequency image; The RGB face image is input into a preset first shallow convolutional neural network for RGB information extraction, obtaining the RGB features output by the first shallow convolutional neural network. The high-frequency image is input into a preset second shallow convolutional neural network for high-frequency information extraction, obtaining the high-frequency features output by the second shallow convolutional neural network. A preset information interaction module is used to interactively learn the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block in the second shallow convolutional neural network to obtain a third feature. The third feature is then added to the first feature and used as the input to the next first convolutional block in the first shallow convolutional neural network. The information interaction module is set up starting from the second convolutional block of the first and second shallow convolutional neural networks, with one information interaction module set up for each corresponding convolutional block of the first and second shallow convolutional neural networks. The method for constructing the information interaction module includes: Set a filter kernel with the same dimension as the input features; The filter kernel is connected to the convolutional blocks of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to form the information interaction module. The filter kernel is used to project the feature maps of the two modalities into the same space, and filter the feature vectors of the two streams of the first shallow convolutional neural network and the second shallow convolutional neural network by dot product. Then, the results are merged on the channel and passed to the softmax operator to obtain the weights of the two streams. Finally, the weights are multiplied by the features of the two streams of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to obtain the final third feature. The RGB features and the high-frequency features are combined to obtain the combined features; The merged features are input into a preset classifier to obtain the authenticity of the RGB face image output by the classifier.

2. The face forgery detection method based on dual-modality according to claim 1, characterized in that, The training method for the classifier includes: End-to-end training is performed using binary cross-entropy loss, and the calculation formula for the training includes: in Represents the number of training samples. Indicates the index of the training sample. Indicates the first The predicted value for each sample, Indicates the first The label of each sample; if the sample category is false. Set it to 1, otherwise set it to 0.

3. The face forgery detection method based on dual-modality according to claim 1, characterized in that, Both the first shallow convolutional neural network and the second shallow convolutional neural network use the Xception network; the Xception network only retains the input stream and the output stream.

4. The face forgery detection method based on dual-modality according to claim 1, characterized in that, The training method for the filter kernel includes: The filter kernel is trained using standard backpropagation and gradient descent.

5. A face forgery detection system based on dual-modality, characterized in that, The dual-modal face forgery detection system includes: An image acquisition module is used to acquire an RGB face image and extract a high-frequency image from the RGB face image. The extraction of the high-frequency image from the RGB face image includes: transforming the RGB face image to the frequency domain using DCT for each of the three RGB channels to obtain a spectrum image; filtering out low-frequency information from the spectrum image using a binary filter to obtain high-frequency information; and transforming the high-frequency information back to the RGB color space using IDCT to obtain the high-frequency image. An adaptive feature interaction module is used to input the RGB face image into a preset first shallow convolutional neural network for RGB information extraction, obtaining the RGB features output by the first shallow convolutional neural network; and to input the high-frequency image into a preset second shallow convolutional neural network for high-frequency information extraction, obtaining the high-frequency features output by the second shallow convolutional neural network. Specifically, a preset information interaction module interacts with the first feature output by the first convolutional block of the current layer in the first shallow convolutional neural network and the second feature output by the corresponding second convolutional block in the second shallow convolutional neural network to obtain a third feature. The third feature is then added to the first feature and used as the input to the next first convolutional block in the first shallow convolutional neural network. The information interaction module is set up starting from the second convolutional block of the first and second shallow convolutional neural networks, with one information interaction module set up for each corresponding convolutional block of the first and second shallow convolutional neural networks. The method for constructing the information interaction module includes: Set a filter kernel with the same dimension as the input features; The filter kernel is connected to the convolutional blocks of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to form the information interaction module. The filter kernel is used to project the feature maps of the two modalities into the same space, and filter the feature vectors of the two streams of the first shallow convolutional neural network and the second shallow convolutional neural network by dot product. Then, the results are merged on the channel and passed to the softmax operator to obtain the weights of the two streams. Finally, the weights are multiplied by the features of the two streams of the first shallow convolutional neural network and the second shallow convolutional neural network respectively to obtain the final third feature. The feature fusion module is used to merge the RGB features and the high-frequency features to obtain merged features; The image authenticity determination module is used to input the merged features into a preset classifier to obtain the authenticity status of the RGB face image output by the classifier.

6. An electronic device, characterized in that: It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the bimodal face forgery detection method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the bimodal face forgery detection method as described in any one of claims 1 to 4.

Images