Face image detection method and device based on multi-scale multi-level feature fusion

Abstract

The embodiment of the application provides a face image detection method and device based on multi-scale multi-level feature fusion. The method first enlarges a face image to be detected to obtain a high-resolution image; two high-level feature maps are obtained after full convolution of two different resolution images on two branches in a feature extraction network, and a plurality of low-level feature maps corresponding to the two branches are extracted after a plurality of convolutions in the two branches; then different scale information of the two low-level feature maps is fully learned to obtain a fused low-level feature map; finally, the fused low-level feature map and the two high-level feature maps are subjected to hierarchical feature fusion, so that the feature information extracted by the low-level feature is not lost, and can be fused with the extracted high-level feature, more general features in face forgery are extracted and utilized, the generalization ability of the model for face detection is improved, and the accuracy of judging face image forgery is improved; finally, whether the face is forged is judged based on the fused feature map.

Description

A face image detection method and device based on multi-scale and multi-level feature fusion Technical Field

[0001] This application relates to the field of face forgery detection and processing technology, and in particular to a face image detection method and apparatus based on multi-scale, multi-level feature fusion. Background Technology

[0002] Currently, deep learning technology is often used to detect face forgery. Although deep learning-based face forgery detection has achieved good results on a single dataset, its performance across different datasets is unsatisfactory, and it suffers from poor generalization performance, resulting in poor accuracy in face forgery detection. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] The main objective of this disclosure is to propose a face image detection method and apparatus based on multi-scale, multi-level feature fusion, which can improve the model's generalization ability for face detection and thus improve the accuracy of judging face image forgery.

[0005] To achieve the above objectives, a first aspect of this disclosure proposes a face image detection method based on multi-scale, multi-level feature fusion, wherein the face image detection method based on multi-scale, multi-level feature fusion includes:

[0006] Acquire the image of the face to be detected;

[0007] The image of the face to be detected is magnified to obtain a high-resolution image;

[0008] The image of the face to be detected is input into the first branch network of the preset feature extraction network for multiple rounds of convolution to obtain the first high-level feature map output by the first branch network. The high-resolution image is input into the second branch network of the feature extraction network for multiple rounds of convolution to obtain the second high-level feature map output by the second branch network.

[0009] The first low-level feature map obtained after several rounds of convolution in the first branch network and the second low-level feature map obtained after several rounds of convolution in the second branch network are input into a preset first scale fusion network for multi-scale information fusion to obtain the third low-level feature map output by the first scale fusion network.

[0010] The first high-level feature map, the second high-level feature map, and the third low-level feature map are input into a preset hierarchical feature fusion network for hierarchical feature fusion to obtain the final fused feature map.

[0011] The authenticity of the face image to be detected is determined based on the final fused feature map.

[0012] In some embodiments, the hierarchical feature fusion network further includes a second-scale fusion network, wherein the hierarchical feature fusion network performs hierarchical feature fusion on the first high-level feature map, the second high-level feature map, and the third low-level feature map in the following manner to obtain a final fused feature map:

[0013] The first high-level feature map and the second high-level feature map are input into the second scale fusion network for multi-scale information fusion to obtain the third high-level feature map output by the second scale fusion network.

[0014] The number of channels and scale of the third low-level feature map are adjusted so that the number of channels and scale of the adjusted third low-level feature map are consistent with those of the third high-level feature map.

[0015] The third high-level feature map and the adjusted third low-level feature map are added element-wise to obtain the final fused feature map.

[0016] In some embodiments, the first scale fusion network performs multi-scale information fusion on the first low-level feature map and the second low-level feature map to obtain a third low-level feature map in the following manner:

[0017] The first low-level feature map is processed by CBR to obtain the first intermediate feature map.

[0018] The second low-level feature map is processed by CBR to obtain the second intermediate feature map. The second intermediate feature map is then processed by max pooling and average pooling respectively, and the feature elements are added together to obtain the third intermediate feature map. The third intermediate feature map is then processed by CBR to obtain the fourth intermediate feature map.

[0019] By splicing the first intermediate feature map and the fourth intermediate feature map, a fifth intermediate feature map is obtained;

[0020] After performing multiple consecutive CBR processes and a single convolution on the fifth intermediate feature map, a sixth intermediate feature map is obtained.

[0021] The sixth intermediate feature map is element-wise multiplied with the first low-level feature map and the second low-level feature map respectively to obtain the seventh intermediate feature map and the eighth intermediate feature map respectively;

[0022] The third low-level feature map is obtained by adding the elements of the seventh intermediate feature map and the eighth intermediate feature map.

[0023] In some embodiments, the second scale fusion network and the first scale fusion network have the same network structure.

[0024] In some embodiments, adjusting the number of channels and scale of the third low-level feature map includes:

[0025] The third low-level feature map is adjusted for the number of channels using a dilated spatial convolutional pooling pyramid, and its scale is adjusted using an average pooling layer.

[0026] In some embodiments, determining the authenticity of the detected face image based on the final fused feature map includes:

[0027] The final fused feature map is converted into a one-dimensional vector;

[0028] The one-dimensional vector is input into a fully connected layer for prediction, and the authenticity of the face image to be detected is determined by the fully connected layer.

[0029] In some embodiments, the step of magnifying the face image to be detected includes:

[0030] The face image to be detected is magnified using bilinear interpolation.

[0031] To achieve the above objectives, a second aspect of this disclosure provides a face image detection device based on multi-scale, multi-level feature fusion, the face image detection device comprising:

[0032] A face image acquisition unit is used to acquire the face image to be detected;

[0033] A face image magnification unit is used to magnify the face image to be detected to obtain a high-resolution image;

[0034] The advanced feature extraction unit is used to input the face image to be detected into the first branch network of the preset feature extraction network for multiple rounds of convolution to obtain the first advanced feature map output by the first branch network, and to input the high-resolution image into the second branch network of the feature extraction network for multiple rounds of convolution to obtain the second advanced feature map output by the second branch network.

[0035] The multi-scale feature fusion unit is used to input the first low-level feature map obtained after several rounds of convolution in the first branch network and the second low-level feature map obtained after several rounds of convolution in the second branch network into a preset first-scale fusion network for multi-scale information fusion to obtain the third low-level feature map output by the first-scale fusion network.

[0036] A multi-level feature fusion unit is used to input the first high-level feature map, the second high-level feature map and the third low-level feature map into a preset multi-level feature fusion network for multi-level feature fusion to obtain the final fused feature map.

[0037] The authenticity determination unit is used to determine the authenticity of the face image to be detected based on the final fused feature map.

[0038] To achieve the above objectives, a third aspect of the present disclosure provides an electronic device including at least one memory;

[0039] At least one processor;

[0040] At least one computer program;

[0041] The computer program is stored in the memory, and the processor executes the at least one computer program to achieve:

[0042] The face image detection method based on multi-scale, multi-level feature fusion as described in any of the first aspect embodiments.

[0043] To achieve the above objectives, a fourth aspect of this disclosure also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform:

[0044] A face image detection method based on multi-scale, multi-level feature fusion as described in any of the first aspect embodiments.

[0045] The first aspect of this application provides a face image detection method based on multi-scale, multi-level feature fusion. This method utilizes the fact that images with different scaling ratios typically contain specific information. First, the face image to be detected is enlarged to obtain a high-resolution image. Then, two images of different resolutions are input into a feature extraction network to form a two-stream network. The two images undergo multiple convolutions on two branches to obtain two high-level feature maps corresponding to the two images. Simultaneously, two low-level feature maps corresponding to the two images after several convolutions on the two branches are extracted. A preset first-scale fusion network fully learns the diverse information of images at different scales to obtain a fused low-level feature map. Finally, the fused low-level feature map and the two high-level feature maps are subjected to hierarchical feature fusion. This ensures that the feature information extracted from the low-level features is not lost while being fused with the extracted high-level features, extracting and utilizing more general features in face forgery, and improving the model's generalization ability for face detection. Finally, the fused feature map is used to determine whether a face is forged. This method takes the fact that images at different scaling ratios typically contain specific information as a starting point, and integrates differentiated information at different scales, which can improve the model's generalization ability for face detection, thereby improving the accuracy of judging face image forgery.

[0046] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 is a flowchart illustrating a face image detection method based on multi-scale, multi-level feature fusion according to an embodiment of this application;

[0049] Figure 2 is a schematic diagram of the specific process of step S104 in Figure 1;

[0050] Figure 3 is a schematic diagram of the specific process of step S105 in Figure 1;

[0051] Figure 4 is a schematic diagram of the specific process of step S106 in Figure 1;

[0052] Figure 5 is a schematic diagram of the network structure of a feature extraction network provided in an embodiment of this application;

[0053] Figure 6 is a schematic diagram of the network structure of MSIM provided in an embodiment of this application;

[0054] Figure 7 is a schematic diagram of the network structure of MLFM provided in an embodiment of this application;

[0055] Figure 8 is a schematic diagram of the network structure of a feature extraction network provided in another embodiment of this application;

[0056] Figure 9 is a schematic diagram of the structure of a face image detection device based on multi-scale, multi-level feature fusion according to an embodiment of this application;

[0057] Figure 10 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] Please refer to Figure 1. Figure 1 illustrates a face image detection method based on multi-scale, multi-level feature fusion according to an embodiment of this application. It should be understood that the face image detection method based on multi-scale, multi-level feature fusion according to this embodiment includes, but is not limited to, steps S101, S102, S103, S104, S105, and S106. Steps S101 to S106 will be described in detail below with reference to Figure 1:

[0062] Step S101: Obtain the face image to be detected.

[0063] This application does not specifically limit the type or source of the face image to be detected.

[0064] Step S102: Enlarge the face image to be detected to obtain a high-resolution image.

[0065] In some embodiments of this application, bilinear interpolation is used to enlarge the face image to be detected, resulting in a high-resolution image. Referring to Figure 5, for example, a 256*256 face image to be detected is converted into a 299*299 high-resolution image.

[0066] Step S103: Input the face image to be detected into the first branch network of the preset feature extraction network for multiple rounds of convolution to obtain the first high-level feature map output by the first branch network. Input the high-resolution image into the second branch network of the feature extraction network for multiple rounds of convolution to obtain the second high-level feature map output by the second branch network.

[0067] Referring to Figure 5, which is a schematic diagram of the feature extraction network structure of this application, the feature extraction network includes two branch networks (both with identical structures). Each branch network includes multiple convolutional units (detailed explanation is shown in Figure 8 below). One branch is used to perform multiple rounds of convolution on the face image to be detected, and the other branch is used to perform multiple rounds of convolution on the high-resolution image. The two branches output the first high-level feature map of the face image to be detected and the second high-level feature map of the high-resolution image, respectively. It should be noted that in Figure 5, "MSIM" is used to represent the scale fusion network, and "MLFM" is used to represent the hierarchical feature fusion network. Among them, MSIM in Figure 5 is the first scale fusion network, and the subsequent second scale fusion network is located in the hierarchical feature fusion network.

[0068] Step S104: Input the first low-level feature map obtained after several rounds of convolution in the first branch network and the second low-level feature map obtained after several rounds of convolution in the second branch network into a preset first scale fusion network for multi-scale information fusion to obtain the third low-level feature map output by the first scale fusion network.

[0069] It should be noted that, since the first and second high-level feature maps have been processed by all convolutional units in the branch network, the high-level feature maps have richer scale information than the low-level feature maps.

[0070] Referring to Figures 2 and 6, in the two branch networks of the feature extraction network, the feature maps output by the several convolutional units of the two branch networks are first obtained, namely, the first low-level feature map obtained after several rounds of convolution in the first branch network and the second low-level feature map obtained after several rounds of convolution in the second branch network; then, the first low-level feature map and the second low-level feature map are fused using multi-scale information. Specifically, the fusion includes the following steps S1041 to S1046:

[0071] Step S1041: Perform CBR processing on the first low-level feature map to obtain the first intermediate feature map. CBR stands for Conv-BN-ReLU, which is common knowledge in the field and will not be elaborated here.

[0072] Step S1042: Perform CBR processing on the second low-level feature map to obtain the second intermediate feature map. Then, perform max pooling and average pooling on the second intermediate feature map and sum the feature elements to obtain the third intermediate feature map. Finally, perform CBR processing on the third intermediate feature map to obtain the fourth intermediate feature map. Since the second low-level feature map is a feature map extracted from the high-resolution image by the feature extraction network, max pooling and average pooling are used for downsampling to preserve as many fake features as possible in the high-resolution features.

[0073] Step S1043: Concatenate the first intermediate feature map and the fourth intermediate feature map to obtain the fifth intermediate feature map. Perform a concatenation operation between the two intermediate features along the channel dimension.

[0074] Step S1044: After performing multiple consecutive CBR processes and a single convolution on the fifth intermediate feature map, a sixth intermediate feature map is obtained. In some embodiments, after performing three consecutive CBR processes and a single convolution on the fifth intermediate feature map, an attention map is obtained.

[0075] Step S1045: Multiply the sixth intermediate feature map element-wise with the first low-level feature map and the second low-level feature map respectively to obtain the seventh intermediate feature map and the eighth intermediate feature map respectively.

[0076] Step S1046: Add the elements of the seventh intermediate feature map and the eighth intermediate feature map to obtain the third low-level feature map.

[0077] Step S105: Input the first high-level feature map, the second high-level feature map, and the third low-level feature map into the preset hierarchical feature fusion network for hierarchical feature fusion to obtain the final fused feature map.

[0078] Referring to Figures 3 and 7, in some embodiments of this application, the hierarchical feature fusion network further includes a second-scale fusion network. The hierarchical feature fusion network performs hierarchical feature fusion on the first high-level feature map, the second high-level feature map, and the third low-level feature map to obtain the final fused feature map, specifically including steps S1051 to S1053:

[0079] Step S1051: Input the first high-level feature map and the second high-level feature map into the second scale fusion network for multi-scale information fusion to obtain the third high-level feature map output by the second scale fusion network.

[0080] Similar to steps S1041 to S1046, steps S1041 to S1046 involve multi-scale information fusion of two low-level feature maps. Here, step S1051 involves multi-scale information fusion of two high-level feature maps. Since the second-scale fusion network has the same network structure as the first-scale fusion network, its process will not be described in detail here.

[0081] Step S1052: Adjust the number of channels and scale of the third low-level feature map so that the number of channels and scale of the adjusted third low-level feature map are consistent with those of the third high-level feature map.

[0082] In some embodiments of this application, adjusting the number of channels and scale of the third low-level feature map in step S1052 includes: Since directly using average pooling operations can easily overlook subtle forgery traces, which is not conducive to the model's capture of minor forgery traces, in order to obtain a larger receptive field, capture global contextual information, and ensure that subtle forgery traces are not lost, this step first adjusts the number of channels of the third low-level feature map to be consistent with the third high-level feature map through dilated spatial convolutional pooling pyramid, and then adjusts the scale to be consistent with the third high-level feature map through an average pooling layer.

[0083] Step S1053: Add the third high-level feature map and the adjusted third low-level feature map element by element to obtain the final fused feature map.

[0084] Step S106: Determine the authenticity of the face image to be detected based on the final fused feature map. In some embodiments of this application, step S106 includes steps S1061 and S1062:

[0085] Step S1061: Convert the final fused feature map into a one-dimensional vector.

[0086] Step S1062: Input the one-dimensional vector into the fully connected layer for prediction to obtain the authenticity of the face image to be detected as determined by the fully connected layer.

[0087] This method leverages the fact that images at different scaling ratios typically contain specific information. First, the image of the face to be detected is enlarged to obtain a high-resolution image. Then, both images are input into a feature extraction network to form a two-stream network. The two images undergo multiple convolutions on two separate branches to obtain two high-level feature maps corresponding to each image. Simultaneously, two low-level feature maps are extracted from each image after several convolutions on each branch. A pre-defined first-scale fusion network fully learns the diverse information of the images at different scales to obtain a fused low-level feature map. Finally, a second-scale fusion network learns the scale information of the two high-level feature maps, performing hierarchical feature fusion between the fused low-level and high-level feature maps. This ensures that the extracted feature information from the low-level features is not lost while fusing with the extracted high-level features, extracting and utilizing more general features in face forgery, thus improving the model's generalization ability for face detection. Finally, the fused feature map is used to determine whether a face is forged. This method improves the model's generalization ability for face detection, thereby increasing the accuracy of predicting face image forgery.

[0088] Referring to Figures 6 to 8, one embodiment of this application provides a face image detection method based on multi-scale, multi-level feature fusion. This method includes the following steps:

[0089] The face image to be detected is enlarged using bilinear interpolation to obtain a high-resolution image. Then, the two images are input separately into a feature extraction network to form a two-stream network. After multiple convolutions, the first high-level feature map is obtained. and Next and Two different operations will be performed: one is to input the data into the first-scale fusion network (MSIM) for information fusion to obtain the third low-level feature map; the other is to continue convolution to obtain two high-level feature maps. Then, the third low-level feature map and these two high-level feature maps are input into the hierarchical feature fusion network (MLFM) to obtain the final fused feature map. Finally, the final fused feature map is flattened into a one-dimensional vector, which is then input into a fully connected layer for prediction to determine whether the detected face image is a forgery. It should be noted that the English explanations of the sub-modules in Figures 6 to 8 are common domain-specific identifiers (such as MaxPooling, Conv, etc.), and their Chinese meanings will not be elaborated here. It is also worth noting that the Entry flow, Middle flow, and Exit flow in Figure 8 represent three stages, and the specific structure of each stage has been shown and will not be repeated here.

[0090] To fully integrate and utilize information from different scales, this embodiment designs a first-scale fusion network. Features of high-resolution images... First, a CBR is applied. Then, to preserve as many fake features as possible in the high-resolution features, max pooling and average pooling are used for downsampling. The features obtained from the two pooling methods are then element-wise summed. Finally, CBR is applied to the resulting features to obtain the feature map. Features of the face image to be detected Only one CBR is needed to obtain the feature map. .Will and After concatenating the channels, three CBR operations are performed, followed by one convolution and a softmax activation function, ultimately resulting in an attention map. ,Will , respectively with Element-wise multiplication followed by element-wise addition yields a low-level feature map that incorporates information from different resolutions. .

[0091] To fully utilize the low-level and high-level features extracted by the model, a hierarchical feature fusion network is designed, including a second-scale fusion network with the same structure as the first-scale fusion network described above. First, the high-level features of the high-resolution image and the original image are fused through the second-scale fusion network to obtain a high-level feature map that integrates information from different resolutions. Because directly using average pooling operations can easily overlook subtle forgery traces, which is detrimental to the model's ability to capture minor forgery traces, the aforementioned low-level feature maps are used to obtain a larger receptive field, capture global contextual information, and ensure that subtle forgery traces are not lost. First, adjust the number of channels using a hollow spatial convolution pooling pyramid to match the previously obtained number. Consistent with the scale, it is then scaled up to match the high-level feature map using an average pooling layer. Consistent. Finally, and The feature map is obtained by adding elements together. This achieves the fusion of low-level and high-level features. Finally, the feature maps are... The image is flattened into a one-dimensional vector (2048 values), and then fed into a fully connected layer for prediction to determine whether the image is a fake.

[0092] This application proposes to enlarge images to obtain high-resolution images through bilinear interpolation, and then carefully designs MSIM to utilize the diverse information of images at different scales. In addition, MLFM is designed to ensure that the feature information extracted from low-level features is not lost, while being able to be well integrated with the extracted high-level features, thereby better extracting and utilizing more general features in face forgery, improving the model's generalization ability for face detection, and thus improving the model's accuracy in judging face images.

[0093] The following is a set of experimental examples:

[0094] (1) Experimental environment of the embodiment;

[0095] The model will be trained on four NVIDIA GeForce GTX TITAN X GPU platforms with 12GB of memory.

[0096] Two datasets were used in the experiment: Faceforensics++ (FF++) and CelebDF. The FF++ dataset, proposed by Rossler et al. in "Faceforensics++: Learning to detect manipulated facial images," includes four forgery methods: Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). CelebDF, proposed by Li et al. in "Celeb-df: A large-scale challenging dataset for deepfake forensics," was used as the training set in this embodiment, and then tested on the CelebDF dataset.

[0097] (2) Experimental results of the embodiments;

[0098] The table below compares the generalization performance with other state-of-the-art methods on the Celeb-DF dataset, using Area Under Curve (AUC) as the benchmark. The best results are indicated in bold. These models were trained on the FF++ dataset and tested on CelebDF. This embodiment outperforms other methods on CelebDF and performs well on FF++. This demonstrates the effectiveness and superior generalization ability of the method.

[0099]

[0100] Referring to an embodiment of this application (Figure 9), a face image detection device based on multi-scale, multi-level feature fusion is provided. The face image detection device based on multi-scale, multi-level feature fusion includes: a face image acquisition unit 1001, a face image magnification unit 1002, a high-level feature extraction unit 1003, a multi-scale feature fusion unit 1004, a multi-level feature fusion unit 1005, and a authenticity judgment unit 1006. Specifically:

[0101] The face image acquisition unit 1001 is used to acquire the face image to be detected.

[0102] The face image magnification unit 1002 is used to magnify the face image to be detected to obtain a high-resolution image.

[0103] The advanced feature extraction unit 1003 is used to input the face image to be detected into the first branch network of the preset feature extraction network for multiple rounds of convolution to obtain the first advanced feature map output by the first branch network, and input the high-resolution image into the second branch network of the feature extraction network for multiple rounds of convolution to obtain the second advanced feature map output by the second branch network.

[0104] The multi-scale feature fusion unit 1004 is used to input the first low-level feature map obtained after several rounds of convolution in the first branch network and the second low-level feature map obtained after several rounds of convolution in the second branch network into a preset first-scale fusion network for multi-scale information fusion, so as to obtain the third low-level feature map output by the first-scale fusion network.

[0105] The multi-level feature fusion unit 1005 is used to input the first high-level feature map, the second high-level feature map and the third low-level feature map into the preset multi-level feature fusion network for multi-level feature fusion to obtain the final fused feature map.

[0106] The authenticity judgment unit 1006 is used to judge the authenticity of the face image to be detected based on the final fused feature map.

[0107] It should be noted that this embodiment and the above method embodiment are based on the same inventive concept. Therefore, the relevant content of the above method embodiment is also applicable to this device embodiment, and will not be repeated here.

[0108] This device utilizes the fact that images at different scaling ratios typically contain specific information. First, the image of the face to be detected is enlarged to obtain a high-resolution image. Then, both images are input into a feature extraction network to form a two-stream network. The two images undergo multiple convolutions on two separate branches to obtain two high-level feature maps corresponding to each image. Simultaneously, two low-level feature maps are extracted from each image after several convolutions on each branch. A pre-defined first-scale fusion network fully learns the diverse information of images at different scales to obtain a fused low-level feature map. Finally, the fused low-level feature map and the two high-level feature maps undergo hierarchical feature fusion. This ensures that the feature information extracted from the low-level features is not lost while fusing with the extracted high-level features, extracting and utilizing more general features in face forgery detection, thus improving the model's generalization ability for face detection. Finally, the fused feature map is used to determine whether a face is forged. This device can improve the model's generalization ability for face detection, thereby improving the accuracy of judging face image forgery.

[0109] This application also provides an electronic device, which includes:

[0110] At least one memory;

[0111] At least one processor;

[0112] At least one program;

[0113] The program is stored in memory, and the processor executes at least one program to implement the face image detection method based on multi-scale and multi-level feature fusion described above in this disclosure.

[0114] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0115] The electronic device in this application embodiment is used to execute the above-described face image detection method based on multi-scale, multi-level feature fusion.

[0116] The electronic device of the present application embodiment will be described in detail below with reference to FIG10.

[0117] Figure 10 illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0118] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0119] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the face image detection method based on multi-scale, multi-level feature fusion of the embodiments of this disclosure.

[0120] The input / output interface 1800 is used to implement information input and output.

[0121] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0122] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0123] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0124] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described face image detection method based on multi-scale, multi-level feature fusion.

[0125] The storage medium of this application embodiment is used to execute the above-described face image detection method based on multi-scale, multi-level feature fusion. Taking into account that images with different scaling ratios typically contain specific information, the face image to be detected is first enlarged to obtain a high-resolution image. Then, the two images are input into a feature extraction network to form a two-stream network. The two images undergo multiple convolutions on two branches to obtain two high-level feature maps corresponding to the two images. Simultaneously, two low-level feature maps corresponding to the two images after several convolutions on the two branches are extracted. A preset first-scale fusion network is used to fully learn the diverse information of images at different scales to obtain a fused low-level feature map. Finally, the fused low-level feature map and the two high-level feature maps are subjected to hierarchical feature fusion. This ensures that the feature information extracted from the low-level features is not lost while being fused with the extracted high-level features, extracting and utilizing more general features in face forgery, and improving the model's generalization ability for face detection. Finally, the fused feature map is used to determine whether a face is forged, which improves the model's generalization ability for face detection and thus improves the accuracy of judging face image forgery.

[0126] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0128] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0131] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0132] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A face image detection method based on multi-scale, multi-level feature fusion, characterized in that, The face image detection method based on multi-scale, multi-level feature fusion includes: acquiring a face image to be detected; enlarging the face image to be detected to obtain a high-resolution image; inputting the face image to be detected into a first branch network of a preset feature extraction network for multiple rounds of convolution to obtain a first high-level feature map output by the first branch network; inputting the high-resolution image into a second branch network of the feature extraction network for multiple rounds of convolution to obtain a second high-level feature map output by the second branch network; and inputting the first low-level feature map obtained after multiple rounds of convolution in the first branch network and the second low-level feature map obtained after multiple rounds of convolution in the second branch network into a preset first-scale fusion network for further processing. Multi-scale information fusion is performed to obtain a third low-level feature map output by the first-scale fusion network. The first-scale fusion network performs multi-scale information fusion on the first low-level feature map and the second low-level feature map in the following manner to obtain the third low-level feature map: the first low-level feature map is processed by CBR to obtain a first intermediate feature map; the second low-level feature map is processed by CBR to obtain a second intermediate feature map; the second intermediate feature map is processed by max pooling and average pooling respectively, and then the feature elements are added to obtain a third intermediate feature map; the third intermediate feature map is processed by CBR to obtain a fourth intermediate feature map; the first intermediate feature map and the fourth intermediate feature map are concatenated to obtain a fifth intermediate feature map. Figure 1 shows the process of performing multiple consecutive CBR processing steps on the fifth intermediate feature map, followed by a single convolution to obtain the sixth intermediate feature map. The sixth intermediate feature map is then element-wise multiplied with the first low-level feature map and the second low-level feature map to obtain the seventh and eighth intermediate feature maps, respectively. The seventh and eighth intermediate feature maps are then element-wise added to obtain the third low-level feature map. The first high-level feature map, the second high-level feature map, and the third low-level feature map are input into a preset hierarchical feature fusion network for hierarchical feature fusion to obtain the final fused feature map. The hierarchical feature fusion network further includes a second-scale fusion network, wherein the hierarchical feature fusion network operates as follows: The method involves hierarchical feature fusion of the first high-level feature map, the second high-level feature map, and the third low-level feature map to obtain a final fused feature map: the first high-level feature map and the second high-level feature map are input into the second scale fusion network for multi-scale information fusion to obtain the third high-level feature map output by the second scale fusion network; the number of channels and scale of the third low-level feature map are adjusted so that the number of channels and scale of the adjusted third low-level feature map are consistent with those of the third high-level feature map; the third high-level feature map and the adjusted third low-level feature map are element-wise added to obtain the final fused feature map; and the authenticity of the face image to be detected is determined based on the final fused feature map.

2. The face image detection method based on multi-scale, multi-level feature fusion according to claim 1, characterized in that, The second-scale fusion network has the same network structure as the first-scale fusion network.

3. The face image detection method based on multi-scale, multi-level feature fusion according to claim 1, characterized in that, The adjustment of the number of channels and scale of the third low-level feature map includes: adjusting the number of channels of the third low-level feature map using a dilated spatial convolutional pooling pyramid, and adjusting the scale of the third low-level feature map using an average pooling layer.

4. The face image detection method based on multi-scale, multi-level feature fusion according to claim 1, characterized in that, The step of determining the authenticity of the face image to be detected based on the final fused feature map includes: converting the final fused feature map into a one-dimensional vector; inputting the one-dimensional vector into a fully connected layer for prediction, and obtaining the authenticity of the face image to be detected as determined by the fully connected layer.

5. The face image detection method based on multi-scale, multi-level feature fusion according to claim 1, characterized in that, The step of enlarging the face image to be detected includes: enlarging the face image to be detected using bilinear interpolation.

6. A face image detection device based on multi-scale, multi-level feature fusion, characterized in that, The face image detection device based on multi-scale, multi-level feature fusion includes: a face image acquisition unit for acquiring a face image to be detected; a face image magnification unit for magnifying the face image to be detected to obtain a high-resolution image; a high-level feature extraction unit for inputting the face image to be detected into a first branch network of a preset feature extraction network for multiple rounds of convolution to obtain a first high-level feature map output by the first branch network, and inputting the high-resolution image into a second branch network of the feature extraction network for multiple rounds of convolution to obtain a second high-level feature map output by the second branch network; and a multi-scale feature fusion unit for fusing the first low-level feature image obtained after multiple rounds of convolution in the first branch network. The feature map and the second low-level feature map obtained after multiple rounds of convolution in the second branch network are input into a preset first-scale fusion network for multi-scale information fusion to obtain the third low-level feature map output by the first-scale fusion network. The first-scale fusion network performs multi-scale information fusion on the first low-level feature map and the second low-level feature map in the following manner to obtain the third low-level feature map: the first low-level feature map is processed by CBR to obtain a first intermediate feature map; the second low-level feature map is processed by CBR to obtain a second intermediate feature map; the second intermediate feature map is processed by max pooling and average pooling respectively, and then the feature elements are added to obtain a third intermediate feature map; the third intermediate feature map is... The feature map undergoes CBR processing to obtain a fourth intermediate feature map; the first intermediate feature map and the fourth intermediate feature map are concatenated to obtain a fifth intermediate feature map; the fifth intermediate feature map undergoes multiple consecutive CBR processing steps, followed by a single convolution to obtain a sixth intermediate feature map; the sixth intermediate feature map is element-wise multiplied with the first low-level feature map and the second low-level feature map respectively to obtain a seventh intermediate feature map and an eighth intermediate feature map; the seventh intermediate feature map and the eighth intermediate feature map are element-wise added to obtain the third low-level feature map; a multi-level feature fusion unit is used to input the first high-level feature map, the second high-level feature map, and the third low-level feature map into a preset layer. The hierarchical feature fusion network performs hierarchical feature fusion to obtain a final fused feature map. The hierarchical feature fusion network further includes a second-scale fusion network. The hierarchical feature fusion network performs hierarchical feature fusion on the first high-level feature map, the second high-level feature map, and the third low-level feature map in the following manner to obtain the final fused feature map: the first high-level feature map and the second high-level feature map are input into the second-scale fusion network for multi-scale information fusion to obtain the third high-level feature map output by the second-scale fusion network; the number of channels and scale of the third low-level feature map are adjusted to make the number of channels and scale of the adjusted third low-level feature map consistent with those of the third high-level feature map.The third high-level feature map and the adjusted third low-level feature map are element-wise added to obtain the final fused feature map; the authenticity judgment unit is used to judge the authenticity of the face image to be detected based on the final fused feature map.

7. An electronic device, characterized in that, include: At least one memory; At least one processor; at least one computer program; said computer program is stored in said memory, and the processor executes said at least one computer program to implement: the face image detection method based on multi-scale multi-level feature fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform: the face image detection method based on multi-scale, multi-level feature fusion as described in any one of claims 1 to 5.

Images