Training method of face image counterfeit detection model, face image counterfeit detection method and device

Abstract

The present disclosure relates to a training method for a facial image counterfeit detection model, a facial image counterfeit detection method and a device, including: extracting multiple types of specified features from a training sample image; splicing multiple types of specified features; inputting the specified spliced ​​features into multiple feature extraction modules respectively; fusing the model extracted features output by each feature extraction module; inputting the fused features into a fully connected layer to predict the authenticity of the training sample image; calculating the loss based on the true category label, the prediction result and the fused features; and adjusting the parameters of each feature extraction module according to the loss. In this way, by designing targeted specified features from different angles in advance, it is possible to improve the comprehensiveness and richness of feature extraction. In addition, the robustness and diversity of feature representation can be improved by combining the advantages of different feature extraction modules, so that complex and changeable counterfeit detection scenarios can be better coped with.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a training method for a facial image counterfeit detection model, a facial image counterfeit detection method and a facial image counterfeit detection device. Background Art

[0002] Fake face technology mainly refers to the use of deep learning generative models (such as GANs, VAEs, etc.) to create realistic fake face images by generating, replacing, migrating or modifying facial features. In order to deal with various frauds carried out by criminals using fake face technology, corresponding fake face identification algorithms have emerged. Fake face identification algorithms can refer to algorithms that use certain technical means to identify fake face images and distinguish them from real face images that have not been tampered with.

[0003] In the related art, image authentication mainly relies on a single feature or a single model, which has great limitations. This is because different model structures have their own advantages and disadvantages. A certain type of model may only be able to identify fake face images produced in a certain way. When faced with fake face images produced by other methods, this type of model may be difficult to accurately identify the authenticity; similarly, if a single feature is used for image authentication, it may only be effective for a certain type of fake face image, but may not be applicable to other types of forged images. It can be seen that in the face of various complex and changeable forged images, the authentication methods in the related art are difficult to achieve full coverage, that is, the authentication accuracy and robustness are poor. Summary of the invention

[0004] The present disclosure provides a training method for a facial image authentication model, a facial image authentication method and a device, so as to at least solve the problem that the authentication method in the above-mentioned related technology is difficult to achieve full coverage, that is, the authentication accuracy and robustness are poor.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a facial image counterfeit detection model is provided, comprising: extracting multiple types of specified features from a training sample image, wherein the multiple types are pre-set multiple types; splicing the multiple types of specified features to obtain specified splicing features; inputting the specified splicing features into multiple feature extraction modules included in the facial image counterfeit detection model respectively to obtain model extraction features output by each feature extraction module; using a feature fusion module included in the facial image counterfeit detection model to fuse the model extraction features output by each feature extraction module to obtain fusion features; inputting the fusion features into a fully connected layer included in the facial image counterfeit detection model to predict the authenticity of the training sample image to obtain a prediction result; calculating the loss based on the true category label corresponding to the training sample image, the prediction result and the fusion feature; and training the facial image counterfeit detection model by adjusting the parameters of each feature extraction module according to the loss.

[0006] Optionally, the loss is calculated based on the true category label corresponding to the training sample image, the prediction result and the fusion feature, including: calculating the cross entropy loss based on the true category label corresponding to the training sample image and the prediction result; calculating the diversity loss based on the fusion feature; and performing weighted summation of the cross entropy loss and the diversity loss to obtain the loss.

[0007] Optionally, the diversity loss is calculated by the following formula:

[0008]

[0009] in is the i-th feature extraction module among the multiple feature extraction modules, is the first of the multiple feature extraction modules feature extraction module, is the k-th dimension feature value extracted by the i-th feature extraction module from the multi-dimensional feature values ​​contained in the fusion feature, The multidimensional feature value contained in the fusion feature is the first The k-th dimension feature value extracted by the feature extraction module, The maximum similarity allowed by the preset.

[0010] Optionally, the multiple types of specified features include: texture features of the entire image, local noise distribution features, and information features of the original image; wherein the texture features of the entire image are obtained in the following manner: converting the training sample image from RGB channels to YCrCb channels, and generating a grayscale co-occurrence matrix corresponding to each channel in the Y channel, Cr channel, and Cb channel as the texture features of the entire image; the local noise distribution features are obtained in the following manner: using a preset sliding window to intercept multiple local image areas in the training sample image; calculating the texture complexity corresponding to each local image area in the multiple local image areas; determining a target local image area with the minimum texture complexity corresponding to the multiple local image areas; filtering the target local image area to obtain the local noise distribution features; the information features of the original image are obtained in the following manner: scaling the training sample image to a preset size; performing RGB image channel separation on the training sample image scaled to the preset size to obtain the information features of the original image.

[0011] Optionally, the multiple feature extraction modules include: a SwinTransform module, a ConvNeXt module, and an Xception module.

[0012] According to a second aspect of an embodiment of the present disclosure, a method for identifying facial images is provided, comprising: obtaining a target image to be identified; extracting multiple types of specified features in the target image, wherein the multiple types are pre-set types; splicing the multiple types of specified features to obtain specified splicing features; respectively inputting the specified splicing features into multiple feature extraction modules included in a facial image identification model trained according to the training method of the present disclosure to obtain model extraction features output by each feature extraction module; using a feature fusion module included in the facial image identification model to fuse multiple model extraction features output by the multiple feature extraction modules one by one to obtain fusion features; inputting the fusion features into a fully connected layer included in the facial image identification model to predict the authenticity of the target image to be identified.

[0013] According to a third aspect of an embodiment of the present disclosure, a training device for a face image counterfeit detection model is provided, comprising: a designated training feature extraction module, configured to extract designated features of multiple types in a training sample image, wherein the multiple types are pre-set types; a training feature splicing module, configured to splice the designated features of the multiple types to obtain designated splicing features; a training input module, configured to input the designated splicing features into multiple feature extraction modules included in the face image counterfeit detection model, respectively, to obtain model extraction features output by each feature extraction module; a training feature fusion module, configured to use the face image counterfeit detection model to extract designated features of multiple types in a training sample image, wherein the multiple types are pre-set types; a training feature splicing module, configured to splice the designated features of the multiple types to obtain designated splicing features; a training input module, configured to input the designated splicing features into multiple feature extraction modules included in the face image counterfeit detection model, respectively, to obtain model extraction features output by each feature extraction module; and a training feature fusion module, configured to use the face image counterfeit detection model to extract designated features of multiple types in a training sample image. The training feature fusion module included in the image authentication model fuses the model extraction features output by each feature extraction module to obtain a fused feature; the image authenticity prediction module is configured to input the fused feature into the fully connected layer included in the face image authentication model to predict the authenticity of the training sample image to obtain a prediction result; the loss calculation module is configured to calculate the loss based on the true category label corresponding to the training sample image, the prediction result and the fused feature; the parameter adjustment module is configured to train the face image authentication model by adjusting the parameters of each feature extraction module according to the loss.

[0014] Optionally, the loss calculation module is configured to: calculate the cross entropy loss based on the true category label corresponding to the training sample image and the prediction result; calculate the diversity loss based on the fusion feature; and perform weighted summation of the cross entropy loss and the diversity loss to obtain the loss.

[0015] Optionally, the diversity loss is calculated by the following formula:

[0016]

[0017] in is the i-th feature extraction module among the multiple feature extraction modules, is the first of the multiple feature extraction modules feature extraction module, is the k-th dimension feature value extracted by the i-th feature extraction module from the multi-dimensional feature values ​​contained in the fusion feature, The multidimensional feature value contained in the fusion feature is the first The k-th dimension feature value extracted by the feature extraction module, The maximum similarity allowed by the preset.

[0018] Optionally, the multiple types of specified features include: texture features of the entire image, local noise distribution features, and information features of the original image; wherein the texture features of the entire image are obtained in the following manner: converting the training sample image from RGB channels to YCrCb channels, and generating a grayscale co-occurrence matrix corresponding to each channel in the Y channel, Cr channel, and Cb channel as the texture features of the entire image; the local noise distribution features are obtained in the following manner: using a preset sliding window to intercept multiple local image areas in the training sample image; calculating the texture complexity corresponding to each local image area in the multiple local image areas; determining a target local image area with the minimum texture complexity corresponding to the multiple local image areas; filtering the target local image area to obtain the local noise distribution features; the information features of the original image are obtained in the following manner: scaling the training sample image to a preset size; performing RGB image channel separation on the training sample image scaled to the preset size to obtain the information features of the original image.

[0019] Optionally, the multiple feature extraction modules include: a SwinTransform module, a ConvNeXt module, and an Xception module.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a facial image authentication device is provided, comprising: an image acquisition module, configured to acquire a target image to be authenticated; a designated feature extraction module, configured to extract designated features of multiple types in the target image, wherein the multiple types are pre-set types; a feature splicing module, configured to splice the designated features of the multiple types to obtain designated splicing features; an input module, configured to input the designated splicing features respectively into multiple feature extraction modules included in a facial image authentication model trained according to the training method of the present disclosure, to obtain model extraction features output by each feature extraction module; a feature fusion module, configured to fuse multiple model extraction features output by the multiple feature extraction modules one by one using the feature fusion module included in the facial image authentication model, to obtain fusion features; and an authenticity prediction module, configured to input the fusion features into a fully connected layer included in the facial image authentication model to predict the authenticity of the target image to be authenticated.

[0021] According to the fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a training method for a facial image authentication model or a facial image authentication method according to the present disclosure.

[0022] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of a facial image authentication model or the facial image authentication method according to the present disclosure.

[0023] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the training method of a facial image authentication model or the facial image authentication method according to the present disclosure.

[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0025] In the present disclosure, targeted designated features can be designed in advance from different angles, so that image features can be captured from multiple perspectives, thereby improving the comprehensiveness and richness of feature extraction. In addition, the extracted image features of these multiple perspectives can be spliced, thereby ensuring that most image synthesis methods are covered. In addition, on the basis of the above-mentioned artificially designed splicing features, the learning capabilities of multiple different feature extraction modules contained in the face image counterfeit detection model can be used to extract corresponding high-level features, and the high-level features extracted by different feature extraction modules can be fused. In this way, the advantages of different feature extraction modules can be combined, thereby improving the robustness and diversity of feature representation, so as to better cope with complex and changeable counterfeit detection scenes. It can be seen that compared with the method of relying on a single feature or a single model for image counterfeiting in the related art, the face image counterfeiting detection method provided by the present disclosure can achieve full coverage of various counterfeit detection scenes through a diverse combination of features and feature extraction modules, and can significantly improve the accuracy and robustness of image counterfeiting detection.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0028] Figure 1 is a schematic diagram showing an architecture of an image authentication process according to an exemplary embodiment of the present disclosure;

[0029] Figure 2 is a flowchart illustrating a method for training a face image counterfeit detection model according to an exemplary embodiment of the present disclosure;

[0030] Figure 3is a flow chart showing a method for identifying counterfeit facial images according to an exemplary embodiment of the present disclosure;

[0031] Figure 4 is a block diagram showing a training device for a face image counterfeit identification model according to an exemplary embodiment of the present disclosure;

[0032] Figure 5 is a block diagram showing a facial image counterfeit identification device according to an exemplary embodiment of the present disclosure;

[0033] Figure 6 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] With the development of science and technology, the technology of forging faces has been continuously updated and iterated. The forging methods have evolved from simple copying and pasting to partial face replacement and tampering based on VAEs technology, and then to the whole image generation based on GANs or SD technology. The facial images synthesized by the above methods are highly realistic, making it difficult for users to distinguish the authenticity with the naked eye. In related technologies, in the face of complex and changeable forged face images, image authentication is usually based on specific domain features and single model learning. Its authentication ability is difficult to cover all kinds of forgery scenarios. The specific reasons are as follows:

[0038] 1. Domain-specific features

[0039] In related technologies, three types of image features are usually used for image authentication: artificially designed features, features based on the spatial domain, and features based on the frequency domain. Specifically:

[0040] a. Artificially designed features are only effective for image forgery detection in simple scenarios, and have poor adaptability when faced with complex and diverse image forgery methods.

[0041] b. Features based on the spatial domain can directly analyze the spatial information of the image (e.g., texture, edge, etc.), and can quickly discover unnatural details in the forged image (e.g., blurred edges, distortion, etc.). However, it is easily affected by image quality (e.g., resolution, noise).

[0042] c. Frequency-domain-based features can capture forgery traces that are difficult for the human eye to detect (e.g., high-frequency noise, compression artifacts, etc.). In addition, they are more sensitive to certain subtle tampering and are not affected by spatial changes such as illumination and color. However, they are highly dependent on image preprocessing, and frequency-domain features can be easily circumvented by forgery techniques.

[0043] 2. Single model dependency

[0044] In the related art, image forgery detection mainly relies on a single classification model, which has great limitations. This is because different model structures have their own strengths and weaknesses, resulting in each type of model structure only showing good detection capabilities in specific image forgery scenarios.

[0045] For example, VGG has a simple structure, deep layers and small convolution kernels, which is suitable for capturing shallow detail features and performs well in high-resolution image classification tasks. However, it is not sensitive enough to complex textures and subtle forged features (for example, high-frequency detail changes) and is easily confused by fine forged images.

[0046] Regarding ResNet: It uses residual connections to alleviate the gradient vanishing problem, can build deeper networks, is suitable for high-resolution image processing, and performs well in forgery detection. However, its sensitivity to extremely small details (such as edge processing and texture fine-tuning) is low, resulting in its ability to detect detailed forgeries and complex textures being slightly insufficient.

[0047] Models for texture analysis: They perform well in detecting some types of forgeries (e.g., images generated by GANs), but perform poorly in dealing with other types of forgeries (e.g., simple copy / paste) because texture features cannot effectively capture the subtle differences in cutting and splicing.

[0048] For noise pattern-based models: They perform well in source device identification (for example, the device that captured the image can be identified by analyzing the sensor noise characteristics), but their effectiveness is limited when dealing with post-edited forged images. This is because post-compression, filtering, and other editing operations can significantly change the noise characteristics of the image, making it difficult to identify image forgeries.

[0049] For geometric feature-based models: They can effectively detect certain forgeries, such as image stitching and geometric deformation, but they are not very effective in dealing with non-geometric feature-related forgeries (e.g., color adjustment, slight local modifications).

[0050] 3. Diversity of methods for forging images and monotony of methods for detecting forgeries

[0051] When generating fake images, GAN is centered on adversarial training, and is good at generating highly realistic details, especially in texture, edges, and light and shadow effects; while SD, with its step-by-step denoising generation process, not only performs well in local details, but also achieves highly realistic effects in complex scenes and diverse layouts. These advances make fake images more realistic in terms of pixel-level details, structure, and content diversity, greatly increasing the difficulty of detection.

[0052] In addition, the image authentication methods in related technologies are also relatively limited, and can be mainly divided into: acquisition-based methods, encoding-based methods, and editing-based methods. Specifically:

[0053] 1. Acquisition-based methods: mainly rely on the traces left by the image during the capture process to perform image authentication. For example, the traces left by the image during the capture process can be: sensor noise (Sensor Noise), photo response non-uniformity (Photo Response Non-Uniformity, PRNU).

[0054] 2. Coding-based method: mainly detects the traces left by the image during the compression and encoding process to perform image authentication. For example, the traces left by the image during the compression and encoding process can be: JPEG compression block effect and quantization noise.

[0055] 3. Editing-based method: mainly focuses on the traces left by the image during the editing process to conduct image authentication. For example, the traces left by the image during the editing process can be: cutting, copying, pasting, and image splicing.

[0056] Furthermore, although the images generated based on GAN technology are visually very realistic, the statistical characteristics of their pixels will change. Therefore, the related art also proposes to use the co-occurrence matrix to capture the texture information in the image and the spatial relationship between pixels, and to identify the fake images generated by GAN by detecting the deviation of the statistical characteristics of natural images; in addition, the SD-based generation model usually focuses on generating areas with rich textures to make the image more realistic, thereby ignoring the implicit noise caused by camera capture in simple areas. Therefore, the related art also proposes a noise pattern feature based on a single simple area to identify fake images.

[0057] However, both of the above methods are based on a single feature for image authentication, which may be effective for specific types of fake images (for example, whole image generation based on SD or GAN technology), but may not be applicable to other types of fake images (for example, traditional partial face replacement, simple copy and paste, partial face tampering or editing), that is, they have poor adaptability and insufficient robustness.

[0058] In order to solve the above-mentioned problems existing in the related art, the training method of the face image counterfeit detection model, the face image counterfeit detection method and the device provided by the present disclosure can design targeted specified features from different angles in advance, so as to ensure that the image features are captured from multiple perspectives, thereby improving the comprehensiveness and richness of feature extraction. In addition, the image features extracted from these multiple perspectives can be spliced, thereby ensuring that most of the image synthesis methods are covered. In addition, on the basis of the above-mentioned artificially designed splicing features, the learning ability of multiple different feature extraction modules contained in the face image counterfeit detection model can be used to extract corresponding high-level features, and the high-level features extracted by different feature extraction modules can be fused. In this way, the advantages of different feature extraction modules can be integrated, thereby improving the robustness and diversity of feature representation, so as to better cope with complex and changeable counterfeit detection scenes. It can be seen that compared with the method of relying on a single feature or a single model for image counterfeit detection in the related art, the face image counterfeit detection method provided by the present disclosure can achieve full coverage of various counterfeit detection scenes through the diverse combination of features and feature extraction modules, and can significantly improve the accuracy and robustness of image counterfeit detection.

[0059] Figure 1 FIG. 1 is a schematic diagram showing an image authentication process according to an exemplary embodiment of the present disclosure. Figure 1 The image authentication process in the present disclosure may mainly include "specified type feature extraction", "specified type feature splicing", "model feature extraction", "model feature fusion" and "image authenticity prediction". Specifically:

[0060] "Specified type features" can be manually preset features that need to be extracted from the image. Exemplary, they can include but are not limited to: "texture features of the entire image", "local noise distribution features", "information features of the original image"; "Specified type feature stitching" mainly refers to the channel stitching of multiple types of specified features extracted from the image, and then the specified stitching features can be obtained.

[0061] "Model feature extraction" mainly refers to the use of each of the multiple different feature extraction modules included in the face image authentication model to extract the model extraction features corresponding to the feature extraction module from the aforementioned specified splicing features. Exemplarily, "multiple different feature extraction modules" may include but are not limited to: "SwinTransform module", "ConvNeXt module", "Xception module". Moreover, the model extraction features output by the "SwinTransform module" may be "f_s", the model extraction features output by the "ConvNeXt module" may be "f_c", and the model extraction features output by the "Xception module" may be "f_x".

[0062] "Model feature fusion" mainly refers to the fusion of the model extraction features output by each feature extraction module in the aforementioned multiple feature extraction modules, thereby obtaining fusion features; "Image authenticity prediction" mainly refers to the use of the fully connected layer (FC_layer) contained in the face image authentication model to predict the authenticity (real / fake) of the image.

[0063] It should be noted that the facial image forgery detection method provided by the present invention can be applied to various common types of synthetic image forgery detection (Image Forgery Detection), which can include but are not limited to: whole images generated using AIGC technology (for example, scene images or facial images), partial face-changing images, image copying, image pasting, image splicing, etc.

[0064] Figure 2 Detailed description is a flowchart illustrating a method for training a facial image counterfeit identification model according to an exemplary embodiment of the present disclosure.

[0065] Reference Figure 2 In step 201, multiple types of specified features can be extracted from the training sample image, where these multiple types can be multiple pre-set types.

[0066] It should be noted that, for image forgery methods such as image copying, pasting, partial face editing, tampering, and GAN or SD series-based synthetic images, the present disclosure can pre-specify multiple types of specified features that need to be extracted from the image in a targeted manner, thereby making feature extraction more reasonable and accurate.

[0067] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple types of specified features may include: "texture features of the entire image", "local noise distribution features", and "information features of the original image", wherein "information features of the original image" may refer to information features such as the geometric shape or color of entities within the image.

[0068] The "texture features of the entire image" can be obtained in the following ways:

[0069] The training sample image is converted from RGB channels to YCrCb channels, and then the three channels of Y, Cr, and Cb can be separated. Then, the grayscale co-occurrence matrix corresponding to each channel in the Y channel, Cr channel, and Cb channel can be generated as the texture feature of the above-mentioned whole image. Exemplarily, the 256-level grayscale co-occurrence matrix of each channel image can be calculated separately, and then three 256*256 matrix feature maps can be obtained. It should be noted that the "texture feature of the whole image" is mainly set for the identification of forged images synthesized based on GAN technology.

[0070] The “local noise distribution characteristics” can be obtained by:

[0071] A preset sliding window can be used to intercept multiple local image areas in the training sample image. Exemplarily, a sliding window with a size of 32*32 and a step size of 32 can be set, and the sliding window can be used to intercept multiple local image areas with a size of 32*32 in the training sample image. Then, the texture complexity corresponding to each local image area in the multiple local image areas can be calculated. Next, the target local image area with the smallest texture complexity in the multiple local image areas can be determined, that is, the target local image area with the simplest texture in the multiple local image areas can be determined. Then, filtering can be performed on the target local image area, and then a local noise distribution feature can be obtained. Exemplarily, the above-mentioned target local image area can be scaled to a size of 256*256, and then the target local image area scaled to a size of 256*256 can be filtered using a (Spatial Rich Model, SRM) filter to obtain a noise feature map with a size of 256*256. It should be noted that the "local noise distribution feature" is mainly set for the identification of forged images synthesized by SD technology.

[0072] The “information features of the original image” can be obtained in the following ways:

[0073] The training sample image is scaled to a preset size. Then, the RGB image channel separation can be performed on the training sample image scaled to the preset size to obtain the information features of the original image. Exemplarily, the input original RGB training sample image can be scaled to 256*256 size, and then the RGB image channel separation can be performed on the training sample image scaled to 256*256 size, so that three 256*256 feature maps can be obtained. It should be noted that the "information features of the original image" are mainly set to identify forged images such as local face image editing or tampering, face image copying, etc.

[0074] In step 202, the above-mentioned multiple types of specified features can be spliced ​​to obtain specified spliced ​​features. That is, the above-mentioned three types of specified features can be channel-spliced ​​to obtain a spliced ​​feature map: [7, 256, 256].

[0075] In this disclosure, after fully considering various image forgery scenarios such as copying, pasting, partial face replacement, and whole image generation in reality, various types of specified features are designed in a targeted manner from a multi-domain perspective, and multi-domain adapter fusion (Multi-domain Fusion) is set for various types of specified features to perform image forgery detection. By fusing various types of specified features, including but not limited to: image texture features, noise patterns, pixel statistical characteristics, geometric shapes, and color information, it is possible to cover common image synthesis methods, thereby ensuring a more comprehensive capture of the diversity of image information, thereby improving the accuracy and robustness of image forgery detection.

[0076] In step 203, the aforementioned designated splicing features may be respectively input into a plurality of feature extraction modules included in the face image counterfeit detection model to obtain model extraction features output by each feature extraction module.

[0077] It should be noted that in the present disclosure, taking into account the limitations of single model learning, on the basis of manually designed feature splicing, it is also designed to utilize the learning capabilities of different feature extraction modules to extract high-level features in different directions in the image and fuse them, so as to enhance the expressive power and authentication ability of the face image authentication model, thereby improving the authentication precision and accuracy of the face image authentication model.

[0078] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple feature extraction modules may include but are not limited to: "SwinTransform module", "ConvNeXt module", and "Xception module".

[0079] For the "SwinTransform module", return to the reference Figure 1, the spliced ​​feature map [batch_size, 7, 256, 256] obtained by splicing multiple types of specified features can be input into the SwinTransform module, and then its own hierarchical structure and window attention mechanism can be used to extract high-level features of the image, and the dimension of the high-level features can be [batch_size, 1024]. Then, the high-level features can be subjected to dimensionality reduction processing, and the feature f_s with a fixed size dimension of 512 can be obtained, that is, [batch_size, 512]. Exemplarily, the dimensionality reduction processing module can be composed of two fully connected layers, and its dimension can be 1024->512->512.

[0080] For "ConvNeXt module", return to reference Figure 1 , the concatenated feature map [batch_size, 7, 256, 256] obtained by concatenating multiple types of specified features can be input into the ConvNeXt module, and then the high-level features of the image can be extracted using its own convolutional neural network architecture, and the dimension of the high-level features can be: [batch_siz, 768]. Then, the high-level features can also be subjected to dimensionality reduction processing, and the feature f_c with a fixed size dimension of 512 can be obtained, that is, [batch_size, 512]. Exemplarily, the dimensionality reduction processing module can be composed of two fully connected layers, and its dimension can be 768->512->512.

[0081] For the "Xception module", return the reference Figure 1 , the spliced ​​feature map [batch_size, 7, 256, 256] obtained by splicing multiple types of specified features can be input into the Xception module, and its depth-separable convolution structure can be used to extract high-level features of the image, and the dimension of the high-level features can be [batch_size, 2048]. Then, the high-level features can be subjected to dimensionality reduction processing, and the feature f_x with a fixed size dimension of 512 can be obtained, that is, [batch_size, 512]. Exemplarily, the dimensionality reduction processing module can be composed of two fully connected layers, and its dimension can be 2048->1024->512.

[0082] Thus, in order to overcome the limitations of a single model, the present disclosure proposes a multi-model ensemble and fused image authentication method, that is, multiple different feature extraction modules can be used to extract image features. Multiple different feature extraction modules may include but are not limited to: a SwinTransforme module based on a Transformer architecture, a ConvNeXt module based on a convolutional neural network architecture, an Xception module based on deep separable convolution, and the like. By combining the characteristics of each module architecture, it is possible to extract high-level features in different directions, thereby enhancing the authentication capability of the face image authentication model, thereby achieving more efficient and accurate image authentication. In addition, by combining the advantages of each module architecture, it is also possible to improve the adaptability and robustness of the face image authentication model in various authentication scenarios.

[0083] In step 204, the feature fusion module included in the face image counterfeit detection model can be used to fuse the model extraction features output by each feature extraction module to obtain a fusion feature. That is, the feature fusion module included in the face image counterfeit detection model can be used to fuse the model extraction features f_s output by the SwinTransform module, the model extraction features f_c output by the ConvNeXt module, and the model extraction features f_x output by the Xception module to obtain a fusion feature: f = [batch_size, 3, 512].

[0084] In step 205, the above fusion feature can be input into the fully connected layer (FC_layer) included in the face image authentication model to predict the authenticity of the training sample image and obtain a prediction result. That is, the above fusion feature: f = [batch_size, 3, 512] can be input into the last few fully connected layers (512->512->256->2) included in the face image authentication model for classification, and then the final image authentication result (Real / Fake) can be obtained.

[0085] In step 206, the loss may be calculated based on the real category label corresponding to the training sample image, the prediction result (Real / Fake), and the fusion feature f = [batch_size, 3, 512].

[0086] According to an exemplary embodiment of the present disclosure, the cross entropy loss can be calculated based on the real category label and the prediction result (Real / Fake) corresponding to the training sample image. ; You can also calculate the diversity loss based on the fusion feature f = [batch_size, 3, 512] Then, we can use the cross entropy loss and diversity loss Perform weighted summation to obtain the above loss.

[0087] Exemplarily, the above loss can be calculated by the following formula:

[0088]

[0089] in, For the above losses, is the cross entropy loss, is the diversity loss, is the cross entropy loss The corresponding weights, Loss of diversity Corresponding weights. For example, you can set .

[0090] According to an exemplary embodiment of the present disclosure, in order to ensure the classification accuracy of the face image counterfeit detection model for the input image, that is, to correctly distinguish between real images and counterfeit images, the present disclosure introduces a cross entropy loss function , which can be expressed by the following formula:

[0091]

[0092] in, Represents the cross entropy loss function value, which is used to measure the model prediction results With the true label The difference between them, N represents the total number of samples, and means that the loss is the average of all samples. represents the true label of the i-th sample, and =1 means the true category is positive. =0 means the true category is negative. and They are used to measure the degree of match between the probability of the model predicting a positive or negative class and the true label. is the probability that the model predicts the category of the i-th sample image as the positive class, which can be calculated by the Sigmoid function:

[0093]

[0094] in, is the raw output (logits) of the model.

[0095] According to an exemplary embodiment of the present disclosure, in order to enhance the diversity of features, the present disclosure also defines a diversity loss function: , which can be calculated by the following formula:

[0096]

[0097] Among them, i and j are used to represent the index of the feature extraction module (for example, SwinTransformer, ConvNeXt, Xception), that is, It can be the i-th feature extraction module among multiple feature extraction modules, It can be used for the first of multiple feature extraction modules feature extraction modules, k is used to represent the specific dimension of the feature vector (the k-th eigenvalue contained in the 512-dimensional feature vector), is the k-th dimension feature value extracted by the i-th feature extraction module from the multi-dimensional feature values ​​contained in the aforementioned fusion feature f=[batch_size, 3, 512], The multidimensional feature values ​​contained in the fusion feature f=[batch_size, 3, 512] are The k-th dimension feature value extracted by the feature extraction module, Indicates channel and The Euclidean distance between is a hyperparameter, which means the maximum similarity allowed by default, and, The value range of can generally be [0.2, 0.6].

[0098] The above diversity loss function The calculation process can be as follows:

[0099] 1. Traverse all feature extraction module pairs: i≠j, specifically:

[0100] (a), i and j represent two different feature extraction modules;

[0101] (b) Traverse all feature extraction module pairs where i≠j and calculate the feature similarity between them.

[0102] 2. Calculate the Euclidean distance between feature vectors:

[0103]

[0104] 3. Compare the calculated Euclidean distance with the threshold For comparison:

[0105] If the Euclidean distance between eigenvectors is less than , then the similarity between them is considered to be within the allowable range and the loss is 0; if the Euclidean distance between the feature vectors is greater than , the excess can be calculated as loss.

[0106] 4. Accumulate the losses of all feature extraction module pairs to get the total diversity loss .

[0107] Thus, by minimizing the diversity loss , it can ensure that the feature vectors extracted by each feature extraction module (for example, SwinTransformer, ConvNeXt, Xception) are diverse and avoid redundancy. That is, different feature extraction modules can be forced to learn different features, so that the feature vectors learned by different feature extraction modules do not overlap with each other, and then the features can be represented more diversely, thereby improving the generalization and robustness of the face image forgery detection model.

[0108] In step 207, the parameters of each feature extraction module may be adjusted according to the calculated loss to train the face image counterfeit detection model.

[0109] The present disclosure proposes a method for identifying forged facial images based on multimodal fusion, multi-domain feature extraction, and multi-model integration. Targeted features can be designed and spliced ​​from different angles, and different learning capabilities of different feature extraction modules can be used to extract more advanced representation features and fuse them. In addition, the present disclosure also introduces a diversity loss function to ensure the diversity of learning features, which can ensure a comprehensive response to the challenges brought by increasingly complex deep fake technology. Compared with the method of using a single feature and a single model for image authentication in related technologies, the facial image authentication method provided by the present disclosure has the following advantages:

[0110] 1. Multi-domain feature fusion: Designing targeted features from different angles and splicing these features can ensure that image features are captured from multiple perspectives, making up for the shortcomings of a single feature and improving the comprehensiveness and richness of feature extraction.

[0111] 2. Integrated learning of multiple feature extraction modules: Utilize the different learning capabilities of different feature extraction modules (e.g., SwinTransformer, ConvNeXt, and Xception) to extract more advanced representation features. These feature extraction modules can capture deeper and more subtle image information through their own architectural characteristics, thereby enhancing the accuracy of image authentication.

[0112] 3. Feature extraction module fusion: By fusing the high-level features extracted by multiple feature extraction modules, the advantages of different feature extraction modules can be combined, thereby improving the robustness and diversity of feature representation. This fusion strategy not only improves the overall performance of the face image authentication model, but also better copes with complex and changeable image authentication scenarios.

[0113] 4. Diversity loss function: By introducing a diversity loss function, the diversity of learning features can be guaranteed. That is, by encouraging different feature extraction modules to learn complementary features, feature redundancy can be avoided, thereby improving the generalization and identification capabilities of the face image authentication model. That is, this mechanism can ensure that the face image authentication model can effectively perform image authentication in different scenarios.

[0114] Figure 3 FIG. 4 is a flowchart illustrating a method for identifying a fake face image according to an exemplary embodiment of the present disclosure.

[0115] Reference Figure 3 In step 301, a target image to be authenticated may be obtained.

[0116] In step 302, multiple types of specified features in the target image may be extracted, wherein the multiple types may be multiple pre-set types.

[0117] It should be noted that, for image forgery methods such as image copying, pasting, partial face editing, tampering, and GAN or SD series-based synthetic images, the present disclosure can pre-specify multiple types of specified features that need to be extracted from the image in a targeted manner, thereby making feature extraction more reasonable and accurate.

[0118] In step 303, the aforementioned multiple types of designated features may be spliced ​​to obtain designated spliced ​​features, that is, the aforementioned multiple types of designated features may be channel-spliced ​​to obtain a spliced ​​feature map.

[0119] In step 304, the above-mentioned designated splicing features can be respectively input into a plurality of feature extraction modules included in the face image counterfeit detection model trained according to the above-mentioned training method, and then the model extraction features output by each feature extraction module can be obtained.

[0120] In step 305, a feature fusion module included in the facial image authentication model may be used to fuse the multiple model extraction features outputted one by one by the aforementioned multiple feature extraction modules to obtain a fused feature.

[0121] It should be noted that in the present disclosure, taking into account the limitations of single model learning, on the basis of manually designed feature splicing, it is also designed to utilize the learning capabilities of different feature extraction modules to extract high-level features in different directions in the image and fuse them, so as to enhance the expressive power and authentication ability of the face image authentication model, thereby improving the authentication precision and accuracy of the face image authentication model.

[0122] In step 306, the aforementioned fusion features can be input into the fully connected layers included in the face image authentication model to predict the authenticity of the target image to be authenticated. That is, the aforementioned fusion features can be input into the last few fully connected layers (512->512->256->2) included in the face image authentication model for classification, and then the final image authentication result (Real / Fake) can be obtained.

[0123] Figure 4 4 is a block diagram showing a training apparatus 400 for a face image counterfeit identification model according to an exemplary embodiment of the present disclosure.

[0124] Reference Figure 4 The training device 400 of the face image counterfeit detection model may include a designated training feature extraction module 401, a training feature splicing module 402, a training input module 403, a training feature fusion module 404, an image authenticity prediction module 405, a loss calculation module 406 and a parameter adjustment module 407.

[0125] The designated training feature extraction module 401 can extract designated features of multiple types from the training sample image, wherein the multiple types can be preset multiple types.

[0126] It should be noted that, for image forgery methods such as image copying, pasting, partial face editing, tampering, and GAN or SD series synthetic images, the present disclosure can pre-specify multiple types of specified features that need to be extracted from the image in a targeted manner, thereby making feature extraction more reasonable and accurate.

[0127] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple types of specified features may include: "texture features of the entire image", "local noise distribution features", and "information features of the original image", wherein "information features of the original image" may refer to information features such as the geometric shape or color of entities within the image.

[0128] The "texture features of the entire image" can be obtained in the following ways:

[0129] The training sample image is converted from RGB channels to YCrCb channels, and then the three channels of Y, Cr, and Cb can be separated. Then, the grayscale co-occurrence matrix corresponding to each channel in the Y channel, Cr channel, and Cb channel can be generated as the texture feature of the above-mentioned whole image. Exemplarily, the 256-level grayscale co-occurrence matrix of each channel image can be calculated separately, and then three 256*256 matrix feature maps can be obtained. It should be noted that the "texture feature of the whole image" is mainly set for the identification of forged images synthesized based on GAN technology.

[0130] The “local noise distribution characteristics” can be obtained by:

[0131] A preset sliding window can be used to intercept multiple local image areas in the training sample image. Exemplarily, a sliding window with a size of 32*32 and a step size of 32 can be set, and the sliding window can be used to intercept multiple local image areas with a size of 32*32 in the training sample image. Then, the texture complexity corresponding to each local image area in the multiple local image areas can be calculated. Next, the target local image area with the smallest texture complexity in the multiple local image areas can be determined, that is, the target local image area with the simplest texture in the multiple local image areas can be determined. Then, filtering can be performed on the target local image area, and then a local noise distribution feature can be obtained. Exemplarily, the above-mentioned target local image area can be scaled to a size of 256*256, and then the target local image area scaled to a size of 256*256 can be filtered using a (Spatial Rich Model, SRM) filter to obtain a noise feature map with a size of 256*256. It should be noted that the "local noise distribution feature" is mainly set for the identification of forged images synthesized by SD technology.

[0132] The “information features of the original image” can be obtained in the following ways:

[0133] The training sample image is scaled to a preset size. Then, the RGB image channel separation can be performed on the training sample image scaled to the preset size to obtain the information features of the original image. Exemplarily, the input original RGB training sample image can be scaled to 256*256 size, and then the RGB image channel separation can be performed on the training sample image scaled to 256*256 size, so that three 256*256 feature maps can be obtained. It should be noted that the "information features of the original image" are mainly set to identify forged images such as local face image editing or tampering, face image copying, etc.

[0134] The training feature splicing module 402 can splice the above-mentioned multiple types of specified features to obtain a specified splicing feature, that is, the above-mentioned three types of specified features can be channel-spliced ​​to obtain a spliced ​​feature map: [7, 256, 256].

[0135] The training input module 403 can input the aforementioned specified splicing features into multiple feature extraction modules included in the face image counterfeit detection model respectively to obtain the model extraction features output by each feature extraction module.

[0136] It should be noted that in the present disclosure, taking into account the limitations of single model learning, on the basis of manually designed feature splicing, it is also designed to utilize the learning capabilities of different feature extraction modules to extract high-level features in different directions in the image and fuse them, so as to enhance the expressive power and authentication ability of the face image authentication model, thereby improving the authentication precision and accuracy of the face image authentication model.

[0137] According to an exemplary embodiment of the present disclosure, the above-mentioned multiple feature extraction modules may include but are not limited to: "SwinTransform module", "ConvNeXt module", and "Xception module".

[0138] Thus, in order to overcome the limitations of a single model, the present disclosure proposes a multi-model ensemble and fused image authentication method, that is, multiple different feature extraction modules can be used to extract image features. Multiple different feature extraction modules may include but are not limited to: a SwinTransforme module based on a Transformer architecture, a ConvNeXt module based on a convolutional neural network architecture, an Xception module based on deep separable convolution, and the like. By combining the characteristics of each module architecture, it is possible to extract high-level features in different directions, thereby enhancing the authentication capability of the face image authentication model, thereby achieving more efficient and accurate image authentication. In addition, by combining the advantages of each module architecture, it is also possible to improve the adaptability and robustness of the face image authentication model in various authentication scenarios.

[0139] The training feature fusion module 404 can use the training feature fusion module included in the face image counterfeit detection model to fuse the model extraction features output by each feature extraction module to obtain a fusion feature. That is, the training feature fusion module included in the face image counterfeit detection model can be used to fuse the model extraction features f_s output by the SwinTransform module, the model extraction features f_c output by the ConvNeXt module, and the model extraction features f_x output by the Xception module, and then the fusion feature can be obtained: f = [batch_size, 3, 512].

[0140] The image authenticity prediction module 405 can input the above fusion feature into the fully connected layer (FC_layer) included in the face image authentication model to predict the authenticity of the training sample image and obtain a prediction result. That is, the above fusion feature f = [batch_size, 3, 512] can be input into the last few fully connected layers included in the face image authentication model for classification, and then the final image authentication result (Real / Fake) can be obtained.

[0141] The loss calculation module 406 can calculate the loss based on the real category label corresponding to the training sample image, the prediction result (Real / Fake) and the fusion feature f = [batch_size, 3, 512].

[0142] According to an exemplary embodiment of the present disclosure, the loss calculation module 406 can calculate the cross entropy loss based on the real category label and the prediction result (Real / Fake) corresponding to the training sample image. The loss calculation module 406 can also calculate the diversity loss based on the fusion feature f = [batch_size, 3, 512] Then, the loss calculation module 406 can calculate the cross entropy loss and diversity loss Perform weighted summation to obtain the above loss.

[0143] Exemplarily, the above loss can be calculated by the following formula:

[0144]

[0145] in, For the above losses, is the cross entropy loss, is the diversity loss, is the cross entropy loss The corresponding weights, Loss of diversity Corresponding weights. For example, you can set .

[0146] According to an exemplary embodiment of the present disclosure, in order to enhance the diversity of features, the present disclosure also defines a diversity loss function: , which can be calculated by the following formula:

[0147]

[0148] Among them, i and j are used to represent the index of the feature extraction module (for example, SwinTransformer, ConvNeXt, Xception), that is, It can be the i-th feature extraction module among multiple feature extraction modules, It can be used for the first of multiple feature extraction modules feature extraction modules, k is used to represent the specific dimension of the feature vector (the k-th eigenvalue contained in the 512-dimensional feature vector), is the k-th dimension feature value extracted by the i-th feature extraction module from the multi-dimensional feature values ​​contained in the aforementioned fusion feature f=[batch_size, 3, 512], The multidimensional feature values ​​contained in the fusion feature f=[batch_size, 3, 512] are The k-th dimension feature value extracted by the feature extraction module, Indicates channel and The Euclidean distance between is a hyperparameter, which means the maximum similarity allowed by default, and, The value range of can generally be [0.2, 0.6].

[0149] Thus, by minimizing the diversity loss , it can ensure that the feature vectors extracted by each feature extraction module (for example, SwinTransformer, ConvNeXt, Xception) are diverse and avoid redundancy. That is, different feature extraction modules can be forced to learn different features, so that the feature vectors learned by different feature extraction modules do not overlap with each other, and then the features can be represented more diversely, thereby improving the generalization and robustness of the face image forgery detection model.

[0150] The parameter adjustment module 407 can adjust the parameters of each feature extraction module according to the calculated loss to train the face image counterfeit detection model.

[0151] Figure 5 is a block diagram showing a facial image authentication apparatus 500 according to an exemplary embodiment of the present disclosure.

[0152] Reference Figure 5 The facial image authentication device 500 may include an image acquisition module 501, a specified feature extraction module 502, a feature splicing module 503, an input module 504, a feature fusion module 505 and a true-false prediction module 506.

[0153] The image acquisition module 501 can acquire a target image to be authenticated.

[0154] The designated feature extraction module 502 may extract designated features of multiple types in the target image, wherein the multiple types may be pre-set multiple types.

[0155] It should be noted that, for image forgery methods such as image copying, pasting, partial face editing, tampering, and GAN or SD series-based synthetic images, the present disclosure can pre-specify multiple types of specified features that need to be extracted from the image in a targeted manner, thereby making feature extraction more reasonable and accurate.

[0156] The feature stitching module 503 can stitch the aforementioned multiple types of designated features to obtain designated stitching features, that is, can perform channel stitching on the aforementioned multiple types of designated features to obtain a stitched feature map.

[0157] The input module 504 can input the above-mentioned specified splicing features into multiple feature extraction modules included in the face image counterfeit detection model trained according to the above-mentioned training method, and then obtain the model extraction features output by each feature extraction module.

[0158] The feature fusion module 505 can utilize the feature fusion module included in the face image counterfeit detection model to fuse the multiple model extraction features output by the aforementioned multiple feature extraction modules one by one to obtain fused features.

[0159] It should be noted that in the present disclosure, taking into account the limitations of single model learning, on the basis of manually designed feature splicing, it is also designed to utilize the learning capabilities of different feature extraction modules to extract high-level features in different directions in the image and fuse them, so as to enhance the expressive power and authentication ability of the face image authentication model, thereby improving the authentication precision and accuracy of the face image authentication model.

[0160] The authenticity prediction module 506 can input the aforementioned fusion features into the fully connected layers included in the face image authentication model to predict the authenticity of the target image to be authenticated. That is, the aforementioned fusion features can be input into the last few fully connected layers (512->512->256->2) included in the face image authentication model for classification, and then the final image authentication result (Real / Fake) can be obtained.

[0161] Figure 6 is a block diagram illustrating an electronic device 600 according to an exemplary embodiment of the present disclosure.

[0162] Reference Figure 6 The electronic device 600 includes at least one memory 601 and at least one processor 602, wherein the at least one memory 601 stores instructions, and when the instructions are executed by the at least one processor 602, a training method for a facial image authentication model or a facial image authentication method according to an exemplary embodiment of the present disclosure is executed.

[0163] As an example, the electronic device 600 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 600 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 600 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0164] In the electronic device 600, the processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0165] The processor 602 may execute instructions or codes stored in the memory 601, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0166] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 601 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 601 and the processor 602 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 602 can read files stored in the memory.

[0167] In addition, the electronic device 600 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 600 may be connected to each other via a bus and / or a network.

[0168] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned training method of the facial image authentication model or the facial image authentication method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0169] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, which, when executed by a processor, implements the training method of the facial image counterfeit identification model or the facial image counterfeit identification method according to the present disclosure.

[0170] According to the training method, method and device of the face image authentication model disclosed in the present invention, targeted designated features can be designed in advance from different angles, so that image features can be captured from multiple perspectives, thereby improving the comprehensiveness and richness of feature extraction. In addition, the extracted image features of these multiple perspectives can be spliced, thereby ensuring that most image synthesis methods are covered. In addition, on the basis of the above-mentioned artificially designed splicing features, the learning ability of multiple different feature extraction modules included in the face image authentication model can be used to extract corresponding high-level features, and the high-level features extracted by different feature extraction modules can be fused. In this way, the advantages of different feature extraction modules can be combined, thereby improving the robustness and diversity of feature representation, so as to better cope with complex and changeable authentication scenes. It can be seen that compared with the method of relying on a single feature or a single model for image authentication in the related art, the face image authentication method provided by the present invention can achieve full coverage of various authentication scenes through a diverse combination of features and feature extraction modules, and can significantly improve the accuracy and robustness of image authentication.

[0171] According to the exemplary embodiments of the present disclosure, after fully considering various image forgery scenarios such as copying, pasting, partial face replacement, and whole image generation in reality, various types of specified features are designed in a targeted manner from a multi-domain perspective, and multi-domain adapter fusion (Multi-domainFusion) is set for various types of specified features to perform image forgery detection. By fusing various types of specified features, including but not limited to: image texture features, noise patterns, pixel statistical characteristics, geometric shapes, and color information, it is possible to cover common image synthesis methods, thereby ensuring a more comprehensive capture of the diversity of image information, thereby improving the accuracy and robustness of image forgery detection.

[0172] According to an exemplary embodiment of the present disclosure, in order to overcome the limitations of a single model, the present disclosure proposes a multi-model ensemble and fused image authentication method, that is, multiple different feature extraction modules can be used to extract image features. Multiple different feature extraction modules may include but are not limited to: a Swin Transforme module based on a Transformer architecture, a ConvNeXt module based on a convolutional neural network architecture, an Xception module based on deep separable convolution, and the like. By combining the characteristics of each module architecture, it is possible to extract high-level features in different directions, thereby enhancing the authentication capability of the face image authentication model, thereby achieving more efficient and accurate image authentication. In addition, by combining the advantages of each module architecture, the adaptability and robustness of the face image authentication model in various authentication scenarios can also be improved.

[0173] According to an exemplary embodiment of the present disclosure, by minimizing the diversity loss , it can ensure that the feature vectors extracted by each feature extraction module (for example, SwinTransformer, ConvNeXt, Xception) are diverse and avoid redundancy. That is, different feature extraction modules can be forced to learn different features, so that the feature vectors learned by different feature extraction modules do not overlap with each other, and then the features can be represented more diversely, thereby improving the generalization and robustness of the face image forgery detection model.

[0174] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0175] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a face image counterfeit detection model, characterized in that: include: Extracting multiple types of specified features from the training sample image, wherein the multiple types are multiple types that are preset; Splicing the multiple types of specified features to obtain specified splicing features; Inputting the specified splicing features into a plurality of feature extraction modules included in the face image counterfeit detection model respectively, and obtaining model extraction features output by each feature extraction module; Using the feature fusion module included in the face image counterfeit detection model, the model extraction features output by each feature extraction module are fused to obtain fused features; Inputting the fusion feature into the fully connected layer included in the face image authentication model to predict the authenticity of the training sample image and obtain a prediction result; Calculating the loss based on the true category label corresponding to the training sample image, the prediction result and the fusion feature; By adjusting the parameters of each feature extraction module according to the loss, the face image counterfeit detection model is trained; The multiple types of specified features include: Texture features of the entire image, local noise distribution features, and information features of the original image; The multiple feature extraction modules include: SwinTransform module, ConvNeXt module, Xception module.

2. The training method according to claim 1, characterized in that: The calculating the loss based on the real category label corresponding to the training sample image, the prediction result and the fusion feature includes: Calculating a cross entropy loss based on the true category label corresponding to the training sample image and the prediction result; Based on the fusion features, calculating diversity loss; A weighted sum is performed on the cross entropy loss and the diversity loss to obtain the loss.

3. The training method according to claim 2, characterized in that: The diversity loss is calculated by the following formula: in is the i-th feature extraction module among the multiple feature extraction modules, is the first of the multiple feature extraction modules feature extraction module, is the k-th dimension feature value extracted by the i-th feature extraction module from the multi-dimensional feature values ​​contained in the fusion feature, The multidimensional feature value contained in the fusion feature is the first The k-th dimension feature value extracted by the feature extraction module, The maximum similarity allowed by the preset.

4. The training method according to claim 1, characterized in that: The texture features of the whole image are obtained by: Convert the training sample image from RGB channels to YCrCb channels, and generate a gray level co-occurrence matrix corresponding to each channel in the Y channel, Cr channel, and Cb channel as the texture feature of the whole image; The local noise distribution characteristics are obtained in the following manner: Using a preset sliding window to intercept multiple local image areas in the training sample image; calculating the texture complexity corresponding to each local image area in the multiple local image areas; determining a target local image area with the minimum texture complexity corresponding to the multiple local image areas; filtering the target local image area to obtain the local noise distribution characteristics; The information features of the original image are obtained in the following way: The training sample image is scaled to a preset size; and RGB image channels are separated for the training sample image scaled to the preset size to obtain information features of the original image.

5. A method for detecting fake faces, characterized in that: include: Acquire a target image to be authenticated; Extracting multiple types of specified features from the target image, wherein the multiple types are multiple preset types; Splicing the multiple types of specified features to obtain specified splicing features; Inputting the designated splicing features into a plurality of feature extraction modules included in the face image counterfeit detection model trained by the training method according to any one of claims 1 to 4, respectively, to obtain model extraction features output by each feature extraction module; Using the feature fusion module included in the face image counterfeit detection model, the multiple model extraction features output by the multiple feature extraction modules are fused to obtain fused features; Inputting the fused features into the fully connected layer included in the face image authentication model to predict the authenticity of the target image to be authenticated; The multiple types of specified features include: Texture features of the entire image, local noise distribution features, and information features of the original image; The multiple feature extraction modules include: SwinTransform module, ConvNeXt module, Xception module.

6. A training device for a face image counterfeit detection model, characterized in that: include: A designated training feature extraction module is configured to extract designated features of multiple types from a training sample image, wherein the multiple types are preset types; A training feature splicing module is configured to splice the specified features of the multiple types to obtain a specified splicing feature; A training input module is configured to input the specified splicing features into a plurality of feature extraction modules included in the face image counterfeit detection model respectively, and obtain model extraction features output by each feature extraction module; A training feature fusion module is configured to use the feature fusion module included in the face image counterfeit detection model to fuse the model extraction features output by each feature extraction module to obtain a fusion feature; An image authenticity prediction module is configured to input the fusion feature into the fully connected layer included in the face image authentication model to predict the authenticity of the training sample image and obtain a prediction result; A loss calculation module, configured to calculate the loss based on the true category label corresponding to the training sample image, the prediction result and the fusion feature; A parameter adjustment module, configured to train the face image counterfeit detection model by adjusting the parameters of each feature extraction module according to the loss; The multiple types of specified features include: Texture features of the entire image, local noise distribution features, and information features of the original image; The multiple feature extraction modules include: SwinTransform module, ConvNeXt module, Xception module.

7. A facial image authentication device, characterized in that: include: An image acquisition module is configured to acquire a target image to be authenticated; A specified feature extraction module is configured to extract a plurality of types of specified features in the target image, wherein the plurality of types are a plurality of preset types; A feature splicing module is configured to splice the specified features of the multiple types to obtain a specified splicing feature; An input module, configured to input the specified splicing features into a plurality of feature extraction modules included in the face image counterfeit detection model trained according to any one of the training methods of claims 1 to 4, respectively, to obtain model extraction features output by each feature extraction module; A feature fusion module is configured to use the feature fusion module included in the face image counterfeit detection model to fuse the multiple model extraction features output by the multiple feature extraction modules one by one to obtain a fusion feature; an authenticity prediction module configured to input the fusion feature into a fully connected layer included in the face image authentication model to predict the authenticity of the target image to be authenticated; The multiple types of specified features include: Texture features of the entire image, local noise distribution features, and information features of the original image; The multiple feature extraction modules include: SwinTransform module, ConvNeXt module, Xception module.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the facial image authentication model as described in any one of claims 1 to 4, or to implement the facial image authentication method as described in claim 5.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method for a facial image authentication model as described in any one of claims 1 to 4, or to execute the facial image authentication method as described in claim 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a facial image authentication model as claimed in any one of claims 1 to 4 is implemented, or the method for authentication of facial images as claimed in claim 5 is implemented.

Images