Face authenticity detection method and device, electronic equipment and storage medium
By training the multimodal feature extraction model and the face authenticity detection model, combining face and voice information to generate virtual sample features, the problem of low accuracy of face authenticity detection in face recognition is solved, and the detection accuracy and generalization ability of the model are improved.
Patent Information
- Application Number
- CN202411649578.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art is difficult to accurately detect the authenticity of faces during the face recognition process, resulting in security risks such as identity theft.
By obtaining video sample data containing real face tags, training a multimodal feature extraction model and a face authenticity detection model, combining face image information and voice audio information, virtual sample features are generated to improve model generalization capabilities.
It improves the accuracy of face authenticity detection, enhances the generalization ability of the model in the real environment, and reduces the risk of identity theft.
Smart Images
Figure CN119516593A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of face recognition and artificial intelligence technology, and in particular to a face authenticity detection method and device, an electronic device and a storage medium. Background Art
[0002] Facial recognition is a technology that uses facial feature information to identify a person. For example, in the financial field, facial recognition can be performed on image data containing user facial information provided by the user when purchasing insurance to ensure the safety of the user's property.
[0003] At present, in the face recognition process, usually only the user identity represented by the face can be identified, while it is difficult to detect the authenticity of the face. It is easy to make errors in face authenticity detection, which may cause user identity theft and endanger the user's property safety. Therefore, how to improve the accuracy of face authenticity detection has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a face mask detection method and device, an electronic device and a storage medium, aiming to improve the accuracy of face authenticity detection.
[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for detecting authenticity of a face, the method comprising:
[0006] Acquire video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face;
[0007] Based on the video sample data, a preset initial multimodal feature extraction model is trained to obtain a target multimodal feature extraction model;
[0008] Based on the target multimodal feature extraction model, feature extraction is performed on the video sample data to obtain multimodal real sample features;
[0009] Based on the target multimodal feature extraction model and the multimodal real sample feature, generating a first multimodal virtual sample feature;
[0010] Based on the first multimodal virtual sample features and the multimodal real sample features, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model;
[0011] Based on the target multimodal feature extraction model and the target face authenticity detection model, face detection is performed on the target video data containing the target face to obtain a target face label, wherein the sample face includes the target face, and the target face label includes the real face label and the non-real face label.
[0012] In some embodiments, the step of training a preset initial multimodal feature extraction model based on the video sample data to obtain a target multimodal feature extraction model includes:
[0013] Extracting features from the video sample data to obtain a sample multimodal feature vector;
[0014] Based on the multimodal feature vector, the video sample data is labeled to obtain labeled video data;
[0015] Based on the initial multimodal feature extraction model, feature extraction is performed on the annotated video data to obtain a predicted multimodal feature vector;
[0016] Based on the sample multimodal feature vector and the predicted multimodal feature vector, parameters of the initial multimodal feature extraction model are adjusted to obtain the target multimodal feature extraction model.
[0017] In some embodiments, extracting features from the video sample data to obtain a sample multimodal feature vector includes:
[0018] Extracting face data from the video sample data to obtain face data;
[0019] Extracting audio data from the video sample data to obtain voice data;
[0020] Performing feature extraction on the face data to obtain a face feature vector;
[0021] Extracting features from the speech data to obtain an audio feature vector;
[0022] Feature fusion is performed on the facial feature vector and the audio feature vector to obtain the sample multimodal feature vector.
[0023] In some embodiments, the initial face authenticity detection model includes an initial sample generation network and an initial face discrimination network, and the method of training the preset initial face authenticity detection model based on the first multimodal virtual sample features and the multimodal real sample features to obtain a target face authenticity detection model includes:
[0024] Based on the initial sample generation network, performing feature transformation on the multimodal real sample features to obtain second multimodal virtual sample features;
[0025] Labeling the first multimodal virtual sample feature to obtain a first feature label, and labeling the second multimodal virtual sample feature to obtain a second feature label;
[0026] Based on the initial face recognition network, predicting feature labels for the first multimodal virtual sample features, the multimodal real sample features, and the second multimodal virtual sample features to obtain predicted feature labels;
[0027] Based on the real face label, the first feature label, the second feature label and the predicted feature label, the parameters of the initial face authenticity detection model are adjusted to obtain the target face authenticity detection model.
[0028] In some embodiments, the step of adjusting parameters of the initial face authenticity detection model based on the real face label, the first feature label, the second feature label, and the predicted feature label to obtain the target face authenticity detection model includes:
[0029] Calculating face recognition loss data of the initial face recognition network based on the real face label, the first feature label, the second feature label and the predicted feature label;
[0030] Based on the face recognition loss data, adjusting the parameters of the initial face recognition network to obtain a target face recognition network;
[0031] Predicting feature labels for the second multimodal virtual sample features based on the target face discrimination network to obtain a target predicted label;
[0032] Based on the target prediction label and the second feature label, calculating the sample generation loss data of the initial sample generation network;
[0033] Based on the sample generation loss data, adjusting parameters of the initial sample generation network to obtain a target sample generation network;
[0034] Based on the target sample generation network and the target face discrimination network, the target face authenticity detection model is determined.
[0035] In some embodiments, the multimodal real sample features include target facial features and target audio features, and the step of generating the first multimodal virtual sample features based on the target multimodal feature extraction model and the multimodal real sample features includes:
[0036] Performing feature transformation on the target facial features based on the target multimodal feature extraction model to obtain virtual facial features;
[0037] The virtual face feature is fused with the target audio feature to obtain the first multimodal virtual sample feature.
[0038] In some embodiments, performing face detection on target video data containing a target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label includes:
[0039] Based on the target multimodal feature extraction model, multimodal feature extraction is performed on the target video data to obtain target multimodal features;
[0040] Based on the target face authenticity detection model, feature detection is performed on the target multimodal features to obtain the target face label.
[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a face authenticity detection device, the device comprising:
[0042] A face video acquisition module, used to acquire video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face;
[0043] A first model training module is used to perform model training on a preset initial multimodal feature extraction model based on the video sample data to obtain a target multimodal feature extraction model;
[0044] A video feature extraction module, used to extract features from the video sample data based on a target multimodal feature extraction model to obtain multimodal real sample features;
[0045] A virtual feature generation module, which generates a first multimodal virtual sample feature based on the target multimodal feature extraction model and the multimodal real sample feature;
[0046] A second model training module, used to perform model training on a preset initial face authenticity detection model based on the first multimodal virtual sample features and the multimodal real sample features to obtain a target face authenticity detection model;
[0047] A face authenticity detection module is used to perform face detection on target video data containing a target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label, wherein the sample face includes the target face, and the target face label includes the real face label and the non-real face label.
[0048] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0050] The face authenticity detection method and device, electronic device and storage medium proposed in the present application perform model training on an initial multimodal feature extraction model through video sample data to obtain a target multimodal feature extraction model, and then use the target multimodal feature extraction model to perform feature extraction on the video sample data to obtain multimodal real sample features, which can combine face image information with voice audio information, thereby improving the accuracy of face authenticity detection. Secondly, based on the target multimodal feature extraction model and the multimodal real sample features, a first multimodal virtual sample feature is generated to provide negative sample data for the model, thereby improving the generalization ability of the model in a real environment. Finally, based on the first multimodal virtual sample feature and the multimodal real sample feature, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model, which can improve the accuracy of the model, thereby improving the accuracy of face authenticity detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 is a flow chart of a face authenticity detection method provided by an embodiment of the present application;
[0052] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.
[0053] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.
[0054] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.
[0055] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.
[0056] Figure 6 yes Figure 5 Flow chart of step S504 in FIG.
[0057] Figure 7 yes Figure 1Flow chart of step S106 in FIG.
[0058] Figure 8 It is a structural schematic diagram of a face authenticity detection device provided in an embodiment of the present application;
[0059] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0063] First, some nouns involved in this application are analyzed:
[0064] Generative Adversarial Models: Generative adversarial models are a deep learning framework consisting of two neural networks: a generator and a discriminator. The generator attempts to generate fake samples that are similar to real data, while the discriminator is responsible for distinguishing between real samples and generated samples. The two networks are trained through an adversarial process, where the generator continuously improves to deceive the discriminator, while the discriminator improves its recognition ability, ultimately enabling the generator to generate high-quality, credible samples. Generative adversarial models have performed well in areas such as image generation, video generation, and speech synthesis.
[0065] Multimodality: Multimodality refers to combining and processing data from different sources or types to obtain more comprehensive information and understanding. By simultaneously analyzing text, images, audio, and other forms of data, multimodal technology can better capture and represent complex phenomena and is widely used in many fields such as computer vision, natural language processing, and human-computer interaction. Multimodal learning aims to utilize the mutual complementarity between modalities to improve the performance and generalization ability of the model, thereby achieving a more intelligent system.
[0066] Facial recognition is a technology that uses facial feature information to identify a person. For example, in the financial field, facial recognition can be performed on image data containing user facial information provided by the user when purchasing insurance to ensure the safety of the user's property.
[0067] At present, in the face recognition process, usually only the user identity represented by the face can be identified, while it is difficult to detect the authenticity of the face. It is easy to make errors in face authenticity detection, which may cause user identity theft and endanger the user's property safety. Therefore, how to improve the accuracy of face authenticity detection has become a technical problem that needs to be solved urgently.
[0068] Based on this, the embodiments of the present application provide a method and device for face authenticity detection, an electronic device and a storage medium, aiming to improve the accuracy of face authenticity detection.
[0069] The face authenticity detection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the face authenticity detection method in the embodiments of the present application is described.
[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0071] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0072] The face authenticity detection method provided in the embodiment of the present application relates to the field of face recognition and artificial intelligence technology. The face authenticity detection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the face authenticity detection method, etc., but is not limited to the above forms.
[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0074] Figure 1 is an optional flowchart of the face authenticity detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.
[0075] Step S101, obtaining video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face;
[0076] Step S102, based on the video sample data, performing model training on the preset initial multimodal feature extraction model to obtain a target multimodal feature extraction model;
[0077] Step S103, based on the target multimodal feature extraction model, feature extraction is performed on the video sample data to obtain multimodal real sample features;
[0078] Step S104, generating a first multimodal virtual sample feature based on the target multimodal feature extraction model and the multimodal real sample feature;
[0079] Step S105, based on the first multimodal virtual sample features and the multimodal real sample features, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model;
[0080] Step S106, performing face detection on the target video data containing the target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label, wherein the sample face includes the target face, and the target face label includes a real face label and a non-real face label.
[0081] In the steps S101 to S106 shown in the embodiment of the present application, video sample data is obtained by acquiring video data containing real face labels, wherein the video sample data also includes sample faces, and the real face labels are used to indicate that the sample faces are real faces. Further, based on the video sample data, a preset initial multimodal feature extraction model is trained to obtain a target multimodal feature extraction model, and then the target multimodal feature extraction model is used to extract features from the video sample data to obtain multimodal real sample features. Secondly, based on the target multimodal features, Extract the model and multimodal real sample features to generate a first multimodal virtual sample feature, then train the preset initial face authenticity detection model based on the first multimodal virtual sample feature and the multimodal real sample feature to obtain a target face authenticity detection model, and finally, perform face detection on the preset target video data containing the target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label, wherein the target face is at least one of the sample faces, and the target face label includes a real face label and a non-real face label. Therefore, the present application performs model training on the initial multimodal feature extraction model through video sample data to obtain a target multimodal feature extraction model, and then uses the target multimodal feature extraction model to perform feature extraction on the video sample data to obtain multimodal real sample features, which can combine facial image information with voice audio information, thereby improving the accuracy of face authenticity detection. Secondly, according to the target multimodal feature extraction model and the multimodal real sample features, a first multimodal virtual sample feature is generated to provide negative sample data for the model, thereby improving the generalization ability of the model in a real environment. Finally, according to the first multimodal virtual sample feature and the multimodal real sample feature, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model, which can improve the accuracy of the model, thereby improving the accuracy of face authenticity detection.
[0082] In step S101 of some embodiments, the video sample data refers to audio and video data including audio and video. The sample face refers to face data that has been detected as a real face.
[0083] The present application can obtain video sample data by marking the video data of the user obtained before the current time. Specifically, video data containing a face shot by the user under the camera is obtained, wherein the video data includes audio information of the user reciting preset text information and the user's face video data. Furthermore, the video data can be marked with data of a real face label to obtain video sample data.
[0084] It should be noted that, since the above video data can be shot by users under manual supervision, each video data can be labeled with a real face label.
[0085] In step S102 of some embodiments, by extracting features from video sample data, a sample multimodal feature vector for feature comparison can be obtained, and then the video sample data is input into a pre-built initial multimodal feature extraction model to obtain a predicted multimodal feature vector output by the initial multimodal feature extraction model. Furthermore, according to the sample multimodal feature vector and the predicted multimodal feature vector, the parameters of the initial multimodal feature extraction model are adjusted to obtain a target multimodal feature extraction model.
[0086] For details, see Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S204:
[0087] Step S201, extracting features from video sample data to obtain a sample multimodal feature vector;
[0088] Step S202, annotating the video sample data based on the multimodal feature vector to obtain annotated video data;
[0089] Step S203, based on the initial multimodal feature extraction model, feature extraction is performed on the annotated video data to obtain a predicted multimodal feature vector;
[0090] Step S204: Based on the sample multimodal feature vector and the predicted multimodal feature vector, the parameters of the initial multimodal feature extraction model are adjusted to obtain a target multimodal feature extraction model.
[0091] In step S201 of some embodiments, face data and voice data are extracted from the video sample data, and then features are extracted from the face data and voice data respectively to obtain a face feature vector and an audio feature vector, and the face feature vector and the audio feature vector are fused to obtain a sample multimodal feature vector.
[0092] For details, see Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S305:
[0093] Step S301, extracting face data from the video sample data to obtain face data;
[0094] Step S302, extracting audio data from the video sample data to obtain voice data;
[0095] Step S303, extracting features from the face data to obtain a face feature vector;
[0096] Step S304, extracting features from the speech data to obtain an audio feature vector;
[0097] Step S305, performing feature fusion on the facial feature vector and the audio feature vector to obtain a sample multimodal feature vector.
[0098] In step S301 and step S302 of some embodiments, face data can be obtained by performing frame-by-frame face extraction on video data in the video sample data, and then voice data can be obtained by performing frame-by-frame audio extraction on audio data in the video sample data.
[0099] In steps S303 and S304 of some embodiments, a convolutional neural network can be used to extract image information from facial data to obtain a facial feature vector representing facial features, and then a recursive neural network can be used to extract acoustic information from audio data to obtain an audio feature vector representing audio features.
[0100] It should be noted that this application does not limit the model for extracting facial features and audio features.
[0101] In step S305 of some embodiments, by fusing the audio feature vector with the facial feature vector, the model can not only detect the authenticity of the face to be identified based on the facial features, but also detect the authenticity of the face to be identified based on the audio features. For example, the pronunciation of each word in the audio features is associated with the changes in facial details in the facial features, so that the facial detail features of the user when speaking certain words can be determined, thereby improving the accuracy of facial authenticity detection.
[0102] In steps S301 to S305 shown in the present embodiment, face data is extracted from video sample data to obtain face data, audio data is extracted from video sample data to obtain voice data, feature extraction is performed on face data to obtain face feature vectors, feature extraction is performed on voice data to obtain audio feature vectors, and feature fusion is performed on face feature vectors and audio feature vectors to obtain sample multimodal feature vectors, which can increase the path of face authenticity detection and improve the accuracy of face authenticity detection.
[0103] In step S202 of some embodiments, in order to implement supervised training of the initial multimodal feature extraction model, the sample multimodal feature vector may be used as annotation information of the video sample data, so that the annotated video data can be obtained.
[0104] In step S203 of some embodiments, the initial multimodal feature extraction model can synchronously perform audio feature extraction and video feature extraction on the input video sample data through a pre-built neural network architecture, and obtain predicted audio features and predicted video features. The predicted audio features and predicted video features are then feature-fused by weighted summation or the like, and a predicted multimodal feature vector is obtained.
[0105] In step S204 of some embodiments, the gap between the predicted multimodal feature vector and the sample multimodal feature vector is calculated by using the feature extraction loss function pre-set in the initial multimodal feature extraction model to obtain feature extraction loss data. Based on the feature extraction loss data, the initial multimodal feature extraction model is back-propagated and parameter adjustment is performed, thereby achieving model training of the initial multimodal feature extraction model and obtaining the target multimodal feature extraction model.
[0106] In steps S201 to S204 shown in the present embodiment, by performing feature extraction on the video sample data to obtain a sample multimodal feature vector, and then performing video annotation on the video sample data based on the multimodal feature vector to obtain annotated video data, data processing of the video sample data can be achieved, so that the video sample data can be input into the initial multimodal feature extraction model as training data. Furthermore, according to the initial multimodal feature extraction model, feature extraction is performed on the annotated video data to obtain a predicted multimodal feature vector, and then according to the sample multimodal feature vector and the predicted multimodal feature vector, the parameters of the initial multimodal feature extraction model are adjusted to obtain a target multimodal feature extraction model, which can improve the accuracy of the model in extracting multimodal fusion features, thereby realizing the combination of multimodal information and improving the accuracy of face authenticity detection.
[0107] In step S103 of some embodiments, after obtaining the trained target multimodal feature extraction model, the target multimodal feature extraction model can be used to extract features from video sample data to obtain multimodal real sample features. Multimodal real sample features can also be obtained by finding the features of the video sample data extracted after the last parameter adjustment during the training process of the target multimodal feature extraction model.
[0108] It is important to note that the difference between the multimodal true sample features and the above sample multimodal feature vector is smaller than the loss threshold pre-set in the target multimodal feature extraction model.
[0109] In step S104 of some embodiments, the target multimodal feature extraction model can be used to perform feature transformation on the target facial features in the multimodal real sample features to obtain non-real facial features, i.e., virtual facial features, and then the virtual facial features are feature fused with the target audio features in the multimodal real sample features to generate non-real multimodal features, i.e., the first multimodal virtual sample features.
[0110] For details, see Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:
[0111] Step S401, performing feature transformation on target facial features based on a target multimodal feature extraction model to obtain virtual facial features;
[0112] Step S402: Fusing the virtual face feature with the target audio feature to obtain a first multimodal virtual sample feature.
[0113] In step S401 and step S402 of some embodiments, the above-mentioned target multimodal feature extraction can be used to add noise, change posture, adjust lighting, etc. to the target facial features, so as to obtain non-real facial features, that is, virtual facial features. In addition, the present application can also interchange the target facial features of different users to obtain virtual facial features, and then perform feature fusion of the changed virtual facial features with the target audio features to obtain the first multimodal virtual sample features.
[0114] In steps S401 to S402 shown in the present embodiment, the target facial features are transformed according to the target multimodal feature extraction model to obtain virtual facial features, and then the virtual facial features are fused with the target audio features to obtain the first multimodal virtual sample features, which can realize the generation of negative sample data when the model is trained, thereby improving the accuracy of the model.
[0115] In step S105 of some embodiments, the initial face authenticity detection model can be a generative adversarial model, which includes an initial sample generation network and an initial face discrimination network, wherein the role of the initial sample generation network in the initial face authenticity detection model is the same as that of the generation network in the generative adversarial model, and the role of the initial face discrimination network in the initial face authenticity detection model is the same as that of the adversarial network in the generative adversarial model.
[0116] The present application can utilize the above-mentioned initial sample generation network to regenerate non-real multimodal features to obtain second multimodal virtual sample features, and then label the first multimodal virtual sample features and the second multimodal virtual sample features respectively to obtain first feature labels and second feature labels, wherein the first feature label is used to indicate that the first multimodal virtual sample features are non-real features, and the second feature label is used to indicate that the second multimodal virtual sample features are non-real features. Furthermore, the above-mentioned initial face discrimination network is used to predict the authenticity of the first multimodal virtual sample features, the multimodal real sample features and the second multimodal virtual sample features, thereby obtaining predicted labels, and then according to the real face labels, the first feature labels, the second feature labels and the predicted feature labels, the parameters of the initial face authenticity detection model are adjusted to obtain the target face authenticity detection model.
[0117] For details, see Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S504:
[0118] Step S501, based on the initial sample generation network, performing feature transformation on the multimodal real sample features to obtain second multimodal virtual sample features;
[0119] Step S502, labeling the first multimodal virtual sample feature to obtain a first feature label, and labeling the second multimodal virtual sample feature to obtain a second feature label;
[0120] Step S503, based on the initial face recognition network, predicting feature labels for the first multimodal virtual sample features, the multimodal real sample features and the second multimodal virtual sample features to obtain predicted feature labels;
[0121] Step S504, based on the real face label, the first feature label, the second feature label and the predicted feature label, the parameters of the initial face authenticity detection model are adjusted to obtain the target face authenticity detection model.
[0122] In step S501 of some embodiments, in order to increase the negative sample data for face authenticity detection and improve the accuracy of the initial face authenticity detection model, the present application can also use the initial sample generation network in the initial face authenticity detection model to change the multimodal real sample features, thereby obtaining a second multimodal virtual sample features.
[0123] In step S502 of some embodiments, the first feature label and the second feature label may be obtained by labeling the first multimodal virtual sample feature and the second multimodal virtual sample feature with non-real feature labels.
[0124] In step S503 of some embodiments, the initial face discrimination network in the initial face authenticity detection model can be used to predict the authenticity of the first multimodal virtual sample features, the multimodal real sample features and the second multimodal virtual sample features, and a predicted label for each feature, i.e., a predicted feature label, can be obtained.
[0125] In step S504 of some embodiments, based on the similarities and differences between the real face label, the first feature label, the second feature label and the predicted feature label, the face discrimination loss data of the initial face discrimination network can be calculated, and then based on the face discrimination loss data, the initial face discrimination network is adjusted to obtain the target face discrimination network. Furthermore, when the parameters of the target face discrimination network remain unchanged, the second multimodal virtual sample features are input into the target face discrimination network again, and the sample generation loss data of the initial sample generation network is calculated based on the target predicted label generated by the target face discrimination network. Finally, based on the sample generation loss data, the initial sample generation network is adjusted to obtain the target sample generation network. The target sample generation network and the target face discrimination network are combined to obtain the target face authenticity detection model.
[0126] For details, see Figure 6 In some embodiments, step S504 may include but is not limited to steps S601 to S606:
[0127] Step S601, calculating face recognition loss data of an initial face recognition network based on the real face label, the first feature label, the second feature label and the predicted feature label;
[0128] Step S602, adjusting parameters of the initial face recognition network based on the face recognition loss data to obtain a target face recognition network;
[0129] Step S603, predicting feature labels of the second multimodal virtual sample features based on the target face recognition network to obtain target predicted labels;
[0130] Step S604, calculating sample generation loss data of the initial sample generation network based on the target prediction label and the second feature label;
[0131] Step S605, adjusting the parameters of the initial sample generation network based on the sample generation loss data to obtain a target sample generation network;
[0132] Step S606, determining a target face authenticity detection model based on the target sample generation network and the target face discrimination network.
[0133] In step S601 of some embodiments, the loss data of the initial face recognition network, i.e., the face recognition loss data, can be obtained by determining whether the predicted feature label of each first multimodal virtual sample feature, multimodal real sample feature and second multimodal virtual sample feature is the same as the corresponding first feature label, real face label and second feature label of the first multimodal virtual sample feature, multimodal real sample feature and second multimodal virtual sample feature.
[0134] In step S602 of some embodiments, when the face recognition loss data is greater than or equal to a pre-set first loss threshold, it indicates that the label prediction effect of the initial face recognition network is not up to standard. Therefore, it is necessary to adjust the parameters in the initial face recognition network to improve the label prediction ability of the initial face recognition network so as to obtain the target face recognition network.
[0135] In step S603 of some embodiments, by fixing the parameters in the target face recognition network, the initial sample generation network transmits the second multimodal virtual sample features to the target face recognition network. The target face recognition network can predict the feature labels of the second multimodal virtual sample features according to the current parameters to obtain the target predicted labels.
[0136] In step S604 of some embodiments, the difference between the target prediction label and the second feature label can be calculated using a loss function pre-set in the initial sample generation network, thereby obtaining sample generation loss data indicating the sample generation effect of the initial sample generation network.
[0137] In step S605 of some embodiments, when the sample generation loss data is greater than or equal to a pre-set second loss threshold, it indicates that the sample generation effect of the initial sample generation network is not up to standard. Therefore, it is necessary to adjust the parameters in the initial sample generation network to improve the sample generation ability of the initial sample generation network so as to obtain the target sample generation network.
[0138] In step S606 of some embodiments, the target sample generation network and the target face discrimination network are combined to obtain a trained target face authenticity detection model.
[0139] In steps S601 to S606 shown in the present embodiment, face discrimination loss data of the initial face discrimination network is calculated according to the real face label, the first feature label, the second feature label and the predicted feature label. According to the face discrimination loss data, the parameters of the initial face discrimination network are adjusted to obtain the target face discrimination network. The target face discrimination network then predicts the feature labels of the second multimodal virtual sample features to obtain the target predicted labels. According to the target predicted labels and the second feature labels, the sample generation loss data of the initial sample generation network is calculated. According to the sample generation loss data, the parameters of the initial sample generation network are adjusted to obtain the target sample generation network. This can achieve confrontation between the initial sample generation network and the initial face discrimination network, forcing the performance of the initial sample generation network and the initial face discrimination network to be improved, thereby improving the accuracy of the target face authenticity detection model and thus improving the accuracy of face authenticity detection.
[0140] In steps S501 to S504 shown in the present embodiment, according to the initial sample generation network, feature transformation is performed on the multimodal real sample features to obtain the second multimodal virtual sample features, and then the first multimodal virtual sample features are labeled to obtain the first feature labels, and the second multimodal virtual sample features are labeled to obtain the second feature labels, thereby completing the construction of the training data of the initial face discrimination network. Furthermore, according to the initial face discrimination network, feature label prediction is performed on the first multimodal virtual sample features, the multimodal real sample features and the second multimodal virtual sample features to obtain predicted feature labels, and according to the real face labels, the first feature labels, the second feature labels and the predicted feature labels, the parameters of the initial face authenticity detection model are adjusted to obtain the target face authenticity detection model, so that the initial face authenticity detection model can be trained using a small amount of video sample data, and the accuracy of the target face authenticity detection model is improved, thereby improving the accuracy of face authenticity detection.
[0141] In step S106 of some embodiments, after the target video data containing the target face is obtained, the target video data can be input into the above-mentioned trained target multimodal feature extraction model and target face authenticity detection model, thereby realizing face authenticity detection of the target video data.
[0142] For details, see Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S702:
[0143] Step S701, based on the target multimodal feature extraction model, multimodal feature extraction is performed on the target video data to obtain target multimodal features;
[0144] Step S702: Perform feature detection on the target multimodal features based on the target face authenticity detection model to obtain a target face label.
[0145] In step S701 of some embodiments, based on the target multimodal feature extraction model, multimodal feature extraction is performed on the target video data to obtain the target multimodal feature. This method is similar to the above-mentioned method of performing feature extraction on the video sample data based on the target multimodal feature extraction model to obtain multimodal real sample features, so it is not repeated here.
[0146] In step S702 of some embodiments, the target multimodal features are input into a target face authenticity detection model, and a target face discrimination network in the target face authenticity detection model can predict feature labels for the target multimodal features, thereby generating a target face label.
[0147] In steps S701 to S702 shown in this embodiment, multimodal feature extraction is performed on the target video data according to the target multimodal feature extraction model to obtain target multimodal features, and then feature detection is performed on the target multimodal features according to the target face authenticity detection model to obtain a target face label. This can realize face authenticity detection of any video data containing face data, thereby improving the accuracy of face authenticity detection.
[0148] The present application performs model training on an initial multimodal feature extraction model through video sample data to obtain a target multimodal feature extraction model, and then uses the target multimodal feature extraction model to perform feature extraction on the video sample data to obtain multimodal real sample features, which can combine facial image information with voice audio information, thereby improving the accuracy of face authenticity detection. Secondly, according to the target multimodal feature extraction model and the multimodal real sample features, a first multimodal virtual sample feature is generated to provide negative sample data for the model, thereby improving the generalization ability of the model in a real environment. Finally, according to the first multimodal virtual sample feature and the multimodal real sample feature, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model, which can improve the accuracy of the model, thereby improving the accuracy of face authenticity detection.
[0149] See also Figure 8 The embodiment of the present application also provides a face authenticity detection device, which can implement the above face authenticity detection method, and the device includes:
[0150] The face video acquisition module 801 is used to acquire video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face;
[0151] A first model training module 802 is used to perform model training on a preset initial multimodal feature extraction model based on video sample data to obtain a target multimodal feature extraction model;
[0152] The video feature extraction module 803 is used to extract features from the video sample data based on the target multimodal feature extraction model to obtain multimodal real sample features;
[0153] A virtual feature generation module 804 generates a first multimodal virtual sample feature based on a target multimodal feature extraction model and multimodal real sample features;
[0154] The second model training module 805 is used to perform model training on a preset initial face authenticity detection model based on the first multimodal virtual sample features and the multimodal real sample features to obtain a target face authenticity detection model;
[0155] The face authenticity detection module 806 is used to perform face detection on the target video data containing the target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain the target face label, wherein the sample face includes the target face, and the target face label includes a real face label and a non-real face label.
[0156] The specific implementation of the face authenticity detection device is basically the same as the specific implementation of the above-mentioned face authenticity detection method, and will not be repeated here.
[0157] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face authenticity detection method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.
[0158] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0159] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0160] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the face authenticity detection method of the embodiment of this application;
[0161] Input / output interface 903, used to implement information input and output;
[0162] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0163] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0164] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0165] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned face authenticity detection method is implemented.
[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0167] The face authenticity detection method, face authenticity detection device, electronic device and storage medium provided in the embodiments of the present application obtain video data containing real face labels to obtain video sample data, wherein the video sample data also includes sample faces, and the real face labels are used to indicate that the sample faces are real faces. Furthermore, based on the video sample data, a preset initial multimodal feature extraction model is trained to obtain a target multimodal feature extraction model, and then the target multimodal feature extraction model is used to extract features from the video sample data to obtain multimodal real sample features. Secondly, based on the target multimodal feature extraction model, a preset initial multimodal feature extraction model is trained to obtain a target multimodal feature extraction model. The method comprises the following steps: first, performing face detection on a preset target video data containing a target face according to the target multimodal feature extraction model and the target face authenticity detection model, and obtaining a target face label, wherein the target face is at least one of the sample faces, and the target face label is a real face label or a non-real face label, and the non-real face label is used to indicate that the sample face is a non-real face.
[0168] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0169] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0170] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0171] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0173] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0175] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for detecting the authenticity of a face, characterized in that: The method comprises: Acquire video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face; Based on the video sample data, a preset initial multimodal feature extraction model is trained to obtain a target multimodal feature extraction model; Based on the target multimodal feature extraction model, feature extraction is performed on the video sample data to obtain multimodal real sample features; Based on the target multimodal feature extraction model and the multimodal real sample feature, generating a first multimodal virtual sample feature; Based on the first multimodal virtual sample features and the multimodal real sample features, a preset initial face authenticity detection model is trained to obtain a target face authenticity detection model; Based on the target multimodal feature extraction model and the target face authenticity detection model, face detection is performed on the target video data containing the target face to obtain a target face label, wherein the sample face includes the target face, and the target face label includes the real face label and the non-real face label.
2. The method according to claim 1, characterized in that The method of training a preset initial multimodal feature extraction model based on the video sample data to obtain a target multimodal feature extraction model includes: Extracting features from the video sample data to obtain a sample multimodal feature vector; Based on the multimodal feature vector, the video sample data is labeled to obtain labeled video data; Based on the initial multimodal feature extraction model, feature extraction is performed on the annotated video data to obtain a predicted multimodal feature vector; Based on the sample multimodal feature vector and the predicted multimodal feature vector, parameters of the initial multimodal feature extraction model are adjusted to obtain the target multimodal feature extraction model.
3. The method according to claim 2, characterized in that The extracting features of the video sample data to obtain a sample multimodal feature vector includes: Extracting face data from the video sample data to obtain face data; Extracting audio data from the video sample data to obtain voice data; Performing feature extraction on the face data to obtain a face feature vector; Extracting features from the speech data to obtain an audio feature vector; Feature fusion is performed on the facial feature vector and the audio feature vector to obtain the sample multimodal feature vector.
4. The method according to claim 1, characterized in that: The initial face authenticity detection model includes an initial sample generation network and an initial face discrimination network. Based on the first multimodal virtual sample features and the multimodal real sample features, the preset initial face authenticity detection model is trained to obtain a target face authenticity detection model, including: Based on the initial sample generation network, performing feature transformation on the multimodal real sample features to obtain second multimodal virtual sample features; Labeling the first multimodal virtual sample feature to obtain a first feature label, and labeling the second multimodal virtual sample feature to obtain a second feature label; Based on the initial face recognition network, predicting feature labels for the first multimodal virtual sample features, the multimodal real sample features, and the second multimodal virtual sample features to obtain predicted feature labels; Based on the real face label, the first feature label, the second feature label and the predicted feature label, the parameters of the initial face authenticity detection model are adjusted to obtain the target face authenticity detection model.
5. The method according to claim 4, characterized in that The step of adjusting parameters of the initial face authenticity detection model based on the real face label, the first feature label, the second feature label and the predicted feature label to obtain the target face authenticity detection model includes: Calculating face recognition loss data of the initial face recognition network based on the real face label, the first feature label, the second feature label and the predicted feature label; Based on the face recognition loss data, adjusting the parameters of the initial face recognition network to obtain a target face recognition network; Predicting feature labels for the second multimodal virtual sample features based on the target face discrimination network to obtain a target predicted label; Based on the target prediction label and the second feature label, calculating the sample generation loss data of the initial sample generation network; Based on the sample generation loss data, adjusting parameters of the initial sample generation network to obtain a target sample generation network; Based on the target sample generation network and the target face discrimination network, the target face authenticity detection model is determined.
6. The method according to any one of claims 1 to 5, characterized in that: The multimodal real sample features include target facial features and target audio features, and the step of generating a first multimodal virtual sample feature based on the target multimodal feature extraction model and the multimodal real sample features includes: Performing feature transformation on the target facial features based on the target multimodal feature extraction model to obtain virtual facial features; The virtual face feature is fused with the target audio feature to obtain the first multimodal virtual sample feature.
7. The method according to any one of claims 1 to 5, characterized in that: The method of performing face detection on target video data containing a target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label includes: Based on the target multimodal feature extraction model, multimodal feature extraction is performed on the target video data to obtain target multimodal features; Based on the target face authenticity detection model, feature detection is performed on the target multimodal features to obtain the target face label.
8. A face authenticity detection device, characterized in that: The device comprises: A face video acquisition module, used to acquire video sample data containing a real face label, wherein the video sample data contains a sample face, and the real face label is used to indicate that the sample face is a real face; A first model training module is used to perform model training on a preset initial multimodal feature extraction model based on the video sample data to obtain a target multimodal feature extraction model; A video feature extraction module, used to extract features from the video sample data based on a target multimodal feature extraction model to obtain multimodal real sample features; A virtual feature generation module, which generates a first multimodal virtual sample feature based on the target multimodal feature extraction model and the multimodal real sample feature; A second model training module, used to perform model training on a preset initial face authenticity detection model based on the first multimodal virtual sample features and the multimodal real sample features to obtain a target face authenticity detection model; A face authenticity detection module is used to perform face detection on target video data containing a target face based on the target multimodal feature extraction model and the target face authenticity detection model to obtain a target face label, wherein the sample face includes the target face, and the target face label includes the real face label and the non-real face label.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the face authenticity detection method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the face authenticity detection method according to any one of claims 1 to 7 is implemented.