Method, device, equipment and medium for identifying forged faces based on audio and image

Abstract

The present invention provides a method, apparatus, device, and medium for identifying forged faces based on audio and image, relating to the technical field of forged face identification. The method comprises extracting image data and audio data from video data of a face to be identified to construct a global image; inputting the global image into a trained forged face identification model to obtain a forged face identification result. The forged face identification model comprises: a global feature encoder for extracting global feature vectors of the image data and audio data in the global image; a global region encoder for constructing a head region feature set, a facial region feature set, and a lip close-up feature set; a region perception module for constructing a weight set; a feature fusion module for generating fused features; and a multi-layer perceptron classifier for outputting a forged face identification result based on the fused features. The present invention performs forged face identification from both audio and image perspectives, improving recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of face forgery recognition, and in particular to a method, device, equipment and medium for face forgery recognition based on audio and image. Background Art

[0002] Facial recognition is a biometric technology that automatically identifies or verifies a person's identity by analyzing and comparing digital facial images or video frames. With the development of this technology, facial recognition has become widely used in security surveillance, access control, mobile device unlocking, and other fields. In various fields, facial forgery technology has emerged to deceive or train facial recognition technology. Facial forgery uses various techniques to create false facial images or videos.

[0003] In related technologies, in order to identify whether a face is forged, facial images are mainly analyzed and identified. However, since facial forgery technology can generate very realistic facial images, these forged contents can be visually almost indistinguishable from the real ones, making it difficult to accurately distinguish them through image analysis alone. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the accuracy of face forgery recognition.

[0005] To solve the above problems, the present invention provides a method, device, equipment and medium for identifying forged faces based on audio and image.

[0006] In a first aspect, the present invention provides a method for identifying forged faces based on audio and images, comprising:

[0007] Extract image data and audio data from the face video data to be recognized and construct a global image;

[0008] The global image is input into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes:

[0009] A global feature encoder, configured to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism;

[0010] a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set;

[0011] A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set;

[0012] a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature;

[0013] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0014] Optionally, extracting image data and audio data from the face video data to be recognized and constructing a global image includes:

[0015] intercepting a preset number of frames of original face images from the face video data to be recognized, and cropping the original face images to generate a face image without background;

[0016] Extracting audio data corresponding to the preset number of frames from the face video data to be recognized, and plotting the audio data into a Mel-spectrogram;

[0017] The face image and the Mel-level spectrogram are concatenated to form the global image.

[0018] Optionally, cropping the original facial image to generate a background-free facial image includes:

[0019] Based on the face cropping model of dlib, the original face image is cropped according to a preset cropping side length to generate the background-free face image.

[0020] Optionally, the trained face forgery recognition model is obtained by training an initial face forgery recognition model based on a binary classification loss function.

[0021] Optionally, the binary classification loss function includes:

[0022] ;

[0023] in, is the binary classification loss function value, y is the true label, is the true label of the i-th one, To predict the results, is the i-th prediction result, and N is the number of samples.

[0024] Optionally, the trained face forgery recognition model is obtained by training the initial face forgery recognition model based on the binary classification loss function and the region-aware loss function, and the region-aware loss function is used to update the parameters of the fully connected layer of the region-aware module.

[0025] Optionally, the region-aware loss function includes:

[0026] ;

[0027] Among them, loss2 is the value of the region-aware loss function, L is the length of the weight vector, T is the batch size, and K is the hyperparameter. is the maximum weight of the i-th sample under the j-th cropping ratio, is the edge weight of the i-th sample at the j-th cropping ratio.

[0028] In a second aspect, the present invention provides an audio and image-based face forgery recognition device, which applies the audio and image-based face forgery recognition method as described in the first aspect, and the audio and image-based face forgery recognition device comprises:

[0029] An extraction module is used to extract image data and audio data from the face video data to be recognized and construct a global image;

[0030] A result module is used to input the global image into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes:

[0031] A global feature encoder, configured to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism;

[0032] a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set;

[0033] A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set;

[0034] a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature;

[0035] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0036] In a third aspect, the present invention provides an electronic device comprising a memory and a processor;

[0037] The memory is used to store computer programs;

[0038] The processor is configured to implement the method for identifying forged faces based on audio and images as described in the first aspect when executing the computer program.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying forged faces based on audio and images as described in the first aspect is implemented.

[0040] The beneficial effects of the audio and image-based face forgery recognition method, device, equipment and medium of the present invention are:

[0041] By extracting image data and audio data from the face video data to be identified, a global image is constructed to facilitate unified modeling of images and audio, achieve effective fusion of multimodal information, improve the consistency and operability of input, and input the global image into the trained face forgery recognition model. The global feature encoder is used to extract the global feature vectors of the image data and the audio data in the global image according to the self-attention mechanism, which can capture the cross-modal association and overall structural information between the image and audio, enhance the global perception ability of forgery behavior, and thus identify face forgery through image and audio dimensions, improve the recognition accuracy, and then use the global region encoder to crop the image into head, face and lip regions, respectively, to extract the head, face and lip regions. Local features are spliced with global features to form a multi-scale feature representation with fine-grained semantics, so that the model can not only perform consistency analysis on the whole, but also pay attention to key local tampering traces and perform detailed analysis of local features; further, the regional perception module dynamically generates the weights of each regional feature through the fully connected layer and the Sigmoid function, so that the model can adaptively adjust the focus according to the input content, thereby improving the robustness to different forgery methods; then, the feature fusion module fuses the features of each region according to the weighted fusion to generate more representative fusion features and reduce redundant information interference; finally, the multi-layer perceptron classifier with strong nonlinear discrimination ability and good practicality performs analysis and prediction to output accurate face forgery recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of a trained face forgery recognition model provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a global image provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a head region image provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a facial region image provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of a close-up image of lips provided in an embodiment of the present invention;

[0047] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0049] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0050] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0052] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0053] In response to the problems existing in the above-mentioned related technologies, this embodiment provides a method, device, equipment and medium for identifying forged faces based on audio and images.

[0054] An embodiment of the present invention provides a method for identifying forged faces based on audio and images, comprising:

[0055] Extract image data and audio data from the face video data to be recognized and construct a global image.

[0056] Specifically, multimedia processing tools (such as FFmpeg) or programming libraries (such as OpenCV, PyDub) can be used to complete the demultiplexing → decoding → output process, thereby extracting image data from the face video data to be identified, and extracting an audio file in wav format from the face video data to be identified.

[0057] The global image is input into a trained face forgery recognition model to obtain a face forgery recognition result.

[0058] Specifically, the global image is Figure 2 As shown, Figure 2 The upper half represents the audio data, and the lower half represents the image data. The global image is input into the trained face forgery recognition model to obtain the face forgery recognition result. The trained face forgery recognition model can be trained by the initial recognition model, and the initial recognition model can be constructed by a neural network.

[0059] Among them, Figure 1 As shown, the face forgery recognition model includes:

[0060] A global feature encoder is used to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism.

[0061] Specifically, the global feature encoder, built using a self-attention mechanism, can extract temporal dependencies between image and audio data in a global image. By focusing on the correspondence between lip movements and audio frequencies, it can identify inconsistencies and abnormal patterns in forged videos. The global feature encoder comprises convolutional layers and a self-attention encoder. After the convolutional layers extract preliminary features, the self-attention encoder captures and encodes long-term cross-modal feature associations to produce a global feature vector. For example, the self-attention encoder can employ the CLIP ViT-L / 14 image encoder.

[0062] A global region encoder is used to crop the image data in the global image into a head region image, a face region image and a lip close-up image, and extract the head region local feature vector of the head region image, the face region local feature vector of the face region image and the lip close-up local feature vector of the lip close-up image according to a residual neural network, and respectively splice the head region local feature vector, the face region local feature vector and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set and a lip close-up feature set.

[0063] Specifically, the global area encoder first crops the head image corresponding to the head of the image data in the global image through the cropping unit. The size of the head image can be 224*224, that is, 224 pixels multiplied by 224 pixels. The cropping unit crops the head image according to the regional relationship. The regional relationship includes that the head area position can be [0:223,0:223], the face area position can be [28:195,28:195], and the lip close-up area position can be [71+35:152+35,71:152]. The cropping unit divides the head image according to the regional relationship to obtain the corresponding head area primary image, face area primary image and lip close-up primary image, and then crops the head area primary image with a cropping ratio of 1.00x, the face area primary image with a cropping ratio of 0.65x, and the lip close-up primary image with a cropping ratio of 0.37x, and then adjusts the size of the head area primary image, the face area primary image and the lip close-up primary image to 224*224, and obtains the following Figure 3 The head area image shown, Figure 4 The facial region image shown and Figure 5 The lip close-up image shown is used to extract the head region local feature vector of the head region image, the face region local feature vector of the face region image, and the lip close-up local feature vector of the lip close-up image according to the residual neural network, and the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector are respectively spliced with the global feature vector by the splicing unit to construct the head region feature set, the face region feature set, and the lip close-up feature set. Among them, the head region image, the face region image, and the lip close-up image all include images of a preset number of frames. The preset number of frames can be set according to actual conditions, for example, 5 frames, that is, the head region image, the face region image, and the lip close-up image all include corresponding 5 images.

[0064] For example, because the lips are at the bottom of the picture, an additional parameter is added to adjust the center position to place the lips at the center of the lip close-up image. Since the facial structures of different people are not much different and the area ratio is often very fixed, the area can be cropped using fixed parameters.

[0065] The region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a Sigmoid function, and to construct a weight set.

[0066] Specifically, the regional perception module extracts the head feature weights of the head region feature set, the facial feature weights of the face region feature set, and the lip feature weights of the lip close-up feature set through the fully connected layer and the Sigmoid function, and constructs a weight set. The parameters of the fully connected layer can learn and assign weights according to the contribution of different regional features in forgery identification, so as to achieve the purpose of highlighting the expressiveness of key regional features, weakening redundant or noise information, and effectively improving the overall detection performance. Then, normalized by the Sigmoid function, the head feature weights, facial feature weights, and lip feature weights can be obtained, and the head feature weights, facial feature weights, and lip feature weights are constructed into a weight set.

[0067] The feature fusion module is used to weightedly fuse the features in the head region feature set, the facial region feature set and the lip close-up feature set according to the weight set to generate a fused feature.

[0068] Specifically, the feature fusion module adopts a weighted fusion formula based on the weight set to weightedly fuse the features in the head region feature set, the facial region feature set, and the lip close-up feature set to generate fusion features. The weighted fusion formula includes

[0069] ;

[0070] Wherein, F is the fusion feature, is the i-th feature under the j-th cropping ratio in the head region feature set, facial region feature set, and lip close-up feature set, is the feature weight corresponding to the i-th feature at the j-th cropping ratio. For example, there are three cropping ratios, namely 1.00x, 0.65x, and 0.37x. The number of features i corresponds to the preset number of frames. When the preset number of frames is 5, there are 5 features at each cropping ratio.

[0071] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0072] Specifically, the multi-layer perceptron classifier is a multi-layer perceptron structure. The fused features are first input into a fully connected layer to obtain a scalar, which is then subjected to sigmoid processing to obtain the face forgery recognition result.

[0073] In this embodiment, a global image is constructed by extracting image data and audio data from the face video data to be identified, which facilitates unified modeling of images and audio, realizes effective fusion of multimodal information, improves the consistency and operability of input, and inputs the global image into a trained face forgery recognition model. The global feature encoder extracts the global feature vectors of the image data and the audio data in the global image according to the self-attention mechanism, which can capture the cross-modal association and overall structural information between the image and audio, enhance the global perception ability of forgery behavior, and thus identify face forgery through image and audio dimensions, improve the recognition accuracy, and then use the global region encoder to crop the image into head, face and lip regions, and divide them into It extracts local features separately and splices them with global features to form a multi-scale feature representation with fine-grained semantics, so that the model can not only perform consistency analysis on the whole, but also pay attention to key local tampering traces and perform detailed analysis of local features; further, the regional perception module dynamically generates the weights of each regional feature through the fully connected layer and the Sigmoid function, so that the model can adaptively adjust the focus according to the input content, thereby improving the robustness to different forgery methods; after that, the feature fusion module fuses the features of each region according to the weighted fusion to generate more representative fusion features and reduce redundant information interference; finally, the multi-layer perceptron classifier with strong nonlinear discrimination ability and good practicality performs analysis and prediction to output accurate face forgery recognition results.

[0074] Optionally, extracting image data and audio data from the face video data to be recognized and constructing a global image includes:

[0075] intercepting a preset number of frames of original face images from the face video data to be recognized, and cropping the original face images to generate a face image without background;

[0076] Extracting audio data corresponding to the preset number of frames from the face video data to be recognized, and plotting the audio data into a Mel-spectrogram;

[0077] The face image and the Mel-level spectrogram are concatenated to form the global image.

[0078] Specifically, intercept the original face image of a preset number of frames in the face video data to be identified, crop the original face image, remove irrelevant background, and generate a background-free face image with only the face. The preset frame number can be set according to actual conditions, for example, 5 frames; extract the audio data of the time corresponding to the preset frame number in the face video data to be identified, the audio data format can be wav, and draw the audio data into a Mel spectrogram. The Mel spectrogram size is 500*2500, and the size of a single face image is 500*500. When the preset frame number is 5 frames, 5 face images are obtained, and 5 face images of 500*500 size and a Mel spectrogram of 500*2500 size are spliced to form a composite image of 1000*2500 size. The composite image is dimensionally adjusted and resized to form the following Figure 2 The global image shown, where Figure 2 The upper part represents the Mel spectrogram, and the lower part represents 5 face images.

[0079] Optionally, cropping the original facial image to generate a background-free facial image includes:

[0080] Based on the face cropping model of dlib, the original face image is cropped according to a preset cropping side length to generate the background-free face image.

[0081] Specifically, dlib's face cropping model is used to detect frontal faces in images. The default cropping edge length is set based on actual conditions. For example, increasing the cropping area edge length to 1.3 times that of the standard dlib face cropping model allows for more comprehensive facial information, avoiding information loss caused by cropping parts of the face image.

[0082] Optionally, the trained face forgery recognition model is obtained by training an initial face forgery recognition model based on a binary classification loss function.

[0083] Specifically, the trained face forgery recognition model is obtained by training the initial face forgery recognition model based on a binary classification loss function. The binary classification loss function can solve the problem that conventional loss functions, such as the binary cross-entropy loss function, make the model unable to fit the data. When training the initial face forgery recognition model, the global feature encoder crops the global image into a mel-level spectrogram and a facial image, and shuffles the order before performing feature extraction to prevent the model from overfitting specific data. In addition, the global image composed of the mel-level spectrogram and the facial image is then input into the face forgery recognition model and cropped into the mel-level spectrogram and facial image by the global feature encoder, which facilitates data storage and retrieval.

[0084] Optionally, the binary classification loss function includes:

[0085] ;

[0086] in, is the binary classification loss function value, y is the true label, is the true label of the i-th one, To predict the results, is the i-th prediction result, and N is the number of samples.

[0087] Optionally, the trained face forgery recognition model is obtained by training the initial face forgery recognition model based on the binary classification loss function and the region-aware loss function, and the region-aware loss function is used to update the parameters of the fully connected layer of the region-aware module.

[0088] Specifically, the trained face forgery recognition model is obtained by training the initial face forgery recognition model based on a binary loss function and a region-aware loss function. The sum of the loss results of the binary loss function and the region-aware loss function is used as the training basis for backpropagation and gradient calculation to update the model parameters. The weight set obtained by the region-aware module through the fully connected layer and the sigmoid function also includes a maximum weight and an edge weight. The maximum weight is the maximum of all items, and the edge weight is the first item. The maximum weight and edge weight are used to determine the result of the region-aware loss function, which then updates the parameters of the fully connected layer of the region-aware module based on the result of the region-aware loss function.

[0089] Optionally, the region-aware loss function includes:

[0090] ;

[0091] Among them, loss2 is the value of the region-aware loss function, L is the length of the weight vector, T is the batch size, and K is a hyperparameter that can adjust the steepness of the loss change and is used to update the parameters of the fully connected layer. is the maximum weight of the i-th sample under the j-th cropping ratio, is the edge weight of the i-th sample at the j-th cropping ratio.

[0092] An embodiment of the present invention provides an audio and image-based face forgery recognition device, which applies the above-mentioned audio and image-based face forgery recognition method. The audio and image-based face forgery recognition device includes:

[0093] An extraction module is used to extract image data and audio data from the face video data to be recognized and construct a global image;

[0094] A result module is used to input the global image into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes:

[0095] A global feature encoder, configured to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism;

[0096] a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set;

[0097] A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set;

[0098] a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature;

[0099] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0100] like Figure 6 As shown, an electronic device 600 provided by an embodiment of the present invention includes a memory 610 and a processor 620; the memory 610 is used to store a computer program; the processor 620 is used to implement the above-mentioned audio and image-based face forgery recognition method when executing the computer program.

[0101] In other words, an electronic device 600 includes a memory 610 and a processor 620 coupled to the memory 610; the memory 610 is configured to store a computer program; and the processor 620 is configured to perform the following operations when executing the computer program:

[0102] Extract image data and audio data from the face video data to be recognized and construct a global image;

[0103] The global image is input into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes:

[0104] A global feature encoder, configured to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism;

[0105] a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set;

[0106] A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set;

[0107] a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature;

[0108] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0109] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the forged face recognition method based on audio and image is implemented as described above.

[0110] In other words, a non-volatile computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following operations:

[0111] Extract image data and audio data from the face video data to be recognized and construct a global image;

[0112] The global image is input into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes:

[0113] A global feature encoder, configured to extract global feature vectors of the image data and the audio data in the global image based on a self-attention mechanism;

[0114] a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set;

[0115] A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set;

[0116] a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature;

[0117] A multi-layer perceptron classifier is used to output the face forgery recognition result based on the fusion features.

[0118] An electronic device 600 that can serve as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device 600 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 600 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] Electronic device 600 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. An input / output (I / O) interface is also connected to the bus.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM). In this application, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network elements. Some or all of these units can be selected based on actual needs to achieve the objectives of the embodiments of the present invention. Furthermore, the functional units in the various embodiments of the present invention can be integrated into a single processing unit, each unit can exist physically separately, or two or more units can be integrated into a single unit. These integrated units can be implemented in either hardware or software functional units.

[0121] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for identifying forged faces based on audio and images, characterized in that: include: Extract image data and audio data from the face video data to be recognized and construct a global image; The global image is input into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes: a global feature encoder configured to extract, based on a self-attention mechanism, a global feature vector of the image data and the audio data in the global image, wherein the global feature vector represents a correspondence between lip movements and audio frequencies, including temporal dependency information between the image data and the audio data, and is configured to identify inconsistencies and abnormal patterns in the facial video data; a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set; A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set; a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature; A multi-layer perceptron classifier, configured to output the face forgery recognition result based on the fusion features; The step of extracting image data and audio data from the face video data to be recognized and constructing a global image includes: intercepting a preset number of frames of original face images from the face video data to be recognized, and cropping the original face images to generate a face image without background; Extracting audio data corresponding to the preset number of frames from the face video data to be recognized, and plotting the audio data into a Mel-spectrogram; The face image and the Mel-level spectrogram are concatenated to form the global image.

2. The method for identifying forged faces based on audio and images according to claim 1, characterized in that: The step of cropping the original face image to generate a background-free face image includes: Based on the face cropping model of dlib, the original face image is cropped according to a preset cropping side length to generate the background-free face image.

3. The method for identifying forged faces based on audio and images according to claim 1, characterized in that: The trained face forgery recognition model is obtained by training the initial face forgery recognition model based on a binary classification loss function.

4. The method for identifying forged faces based on audio and image according to claim 3, characterized in that: The binary classification loss function includes: ； in, is the binary classification loss function value, y is the true label, is the true label of the i-th one, To predict the results, is the i-th prediction result, and N is the number of samples.

5. The method for identifying forged faces based on audio and images according to claim 3, characterized in that: The trained face forgery recognition model is obtained by training the initial face forgery recognition model based on the binary classification loss function and the region-aware loss function, and the region-aware loss function is used to update the parameters of the fully connected layer of the region-aware module.

6. The method for identifying forged faces based on audio and images according to claim 5, characterized in that: The region-aware loss function includes: ； Among them, loss2 is the value of the region-aware loss function, L is the length of the weight vector, T is the batch size, and K is the hyperparameter. is the maximum weight of the i-th sample under the j-th cropping ratio, is the edge weight of the i-th sample at the j-th cropping ratio.

7. A face forgery recognition device based on audio and image, characterized in that: The method for identifying forged faces based on audio and images according to any one of claims 1 to 6 is applied, wherein the device for identifying forged faces based on audio and images comprises: An extraction module is used to extract image data and audio data from the face video data to be recognized and construct a global image; A result module is used to input the global image into a trained face forgery recognition model to obtain a face forgery recognition result, wherein the face forgery recognition model includes: a global feature encoder configured to extract, based on a self-attention mechanism, a global feature vector of the image data and the audio data in the global image, wherein the global feature vector represents a correspondence between lip movements and audio frequencies, including temporal dependency information between the image data and the audio data, and is configured to identify inconsistencies and abnormal patterns in the facial video data; a global region encoder, configured to crop the image data in the global image into a head region image, a face region image, and a lip close-up image; extract, based on a residual neural network, a head region local feature vector of the head region image, a face region local feature vector of the face region image, and a lip close-up local feature vector of the lip close-up image; and concatenate the head region local feature vector, the face region local feature vector, and the lip close-up local feature vector with the global feature vector to construct a head region feature set, a face region feature set, and a lip close-up feature set; A region perception module is used to extract the head feature weights of the head region feature set, the facial feature weights of the facial region feature set, and the lip feature weights of the lip close-up feature set through a fully connected layer and a sigmoid function, and to construct a weight set; a feature fusion module, configured to weightedly fuse features in the head region feature set, the facial region feature set, and the lip close-up feature set according to the weight set to generate a fused feature; A multi-layer perceptron classifier, configured to output the face forgery recognition result based on the fusion features; The step of extracting image data and audio data from the face video data to be recognized and constructing a global image includes: intercepting a preset number of frames of original face images from the face video data to be recognized, and cropping the original face images to generate a face image without background; Extracting audio data corresponding to the preset number of frames from the face video data to be recognized, and plotting the audio data into a Mel-spectrogram; The face image and the Mel-level spectrogram are concatenated to form the global image.

8. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for identifying forged faces based on audio and images according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for identifying forged faces based on audio and images according to any one of claims 1 to 6 is implemented.

Images