Face video classification method and training method of face video classification model

Abstract

The embodiment of the specification discloses a face video classification method, a training method and device of a face video classification model, a storage medium and an electronic device, which comprises: performing feature extraction on a plurality of video frames of a face verification video to obtain manual features and deep features of each video frame; performing classification based on the manual features of each video frame to obtain a first classification result of each video frame; performing classification based on the deep features of each video frame to obtain a second classification result of each video frame, wherein the first classification result and the second classification result are used to represent whether each video frame is a defective frame; determining at least one defective frame from the plurality of video frames based on the manual features, the deep features, the first classification result and the second classification result of each video frame; and determining whether the face verification video is a tampered video based on the at least one defective frame.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer technology, and particularly relates to a face video classification method, a face video classification model training method, device, storage medium and electronic equipment. BACKGROUND

[0002] With the development of computer technology, face recognition technology has been widely used in recent years. For example, a face recognition system using face recognition technology is widely deployed on a payment platform, and a user can quickly complete payment through the face recognition system.

[0003] However, while bringing convenience to people's production and life, the face recognition system is also under the test of various attacks. For example, the Deepfakes technology derived from deep learning can highly accurately tamper with face information in videos and images, which has a huge impact on the security of the face recognition system. Therefore, how to identify face verification videos tampered by Deepfakes technology is a research hotspot. SUMMARY

[0004] The present specification provides a face video classification method, a face video classification model training method, device, storage medium and electronic equipment, which can identify face verification videos tampered by Deepfakes technology and improve the security of face recognition.

[0005] In one aspect, the present specification provides a face video classification method, comprising:

[0006] performing feature extraction on a plurality of video frames of a face verification video to obtain handcrafted features and deep features of each video frame;

[0007] performing classification based on the handcrafted features of each video frame to obtain a first classification result of each video frame; performing classification based on the deep features of each video frame to obtain a second classification result of each video frame, wherein the first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects;

[0008] determining at least one defective frame from the plurality of video frames based on the handcrafted features, the deep features, the first classification result and the second classification result of each video frame;

[0009] determining whether the face verification video is a tampered video based on the at least one defective frame.

[0010] In one aspect, the present specification provides a face video classification model training method, comprising:

[0011] input a sample face verification video into a face video classification model;

[0012] extract features of a plurality of video frames of the sample face verification video through the face video classification model to obtain hand-crafted features and deep features of each of the video frames;

[0013] classify each of the video frames based on the hand-crafted features of each of the video frames through the face video classification model to obtain a first classification result of each of the video frames, and classify each of the video frames based on the deep features of each of the video frames to obtain a second classification result of each of the video frames, the first classification result and the second classification result being used to indicate whether each of the video frames is a defective frame, the defective frame being a video frame that may have a tampering defect;

[0014] determine at least one defective frame from the plurality of video frames based on the hand-crafted features, the deep features, the first classification result, and the second classification result of each of the video frames through the face video classification model;

[0015] determine a predicted type of the sample face verification video based on the at least one defective frame through the face video classification model, the predicted type being used to indicate whether the sample face verification video is a tampered video;

[0016] train the face video classification model based on first difference information between the predicted type of the sample face verification video and a labeled type of the sample face verification video.

[0017] In one aspect, an embodiment of the present specification provides a face video classification device, comprising:

[0018] a first feature extraction module configured to extract features of a plurality of video frames of a face verification video to obtain hand-crafted features and deep features of each of the video frames;

[0019] a first video frame classification module configured to classify each of the video frames based on the hand-crafted features of each of the video frames to obtain a first classification result of each of the video frames, and classify each of the video frames based on the deep features of each of the video frames to obtain a second classification result of each of the video frames, the first classification result and the second classification result being used to indicate whether each of the video frames is a defective frame, the defective frame being a video frame that may have a tampering defect;

[0020] a first defective frame determination module configured to determine at least one defective frame from the plurality of video frames based on the hand-crafted features, the deep features, the first classification result, and the second classification result of each of the video frames;

[0021] a first video classification module configured to determine whether the face verification video is a tampered video based on the at least one defective frame.

[0022] In a possible implementation, the first feature extraction module is configured to input the plurality of video frames into a hand-crafted feature extractor, perform hand-crafted feature extraction on the plurality of video frames by using the hand-crafted feature extractor, and obtain hand-crafted features of each of the video frames; and input the plurality of video frames into a deep feature extractor, perform deep feature extraction on the plurality of video frames by using the deep feature extractor, and obtain deep features of each of the video frames.

[0023] In a possible implementation, the first feature extraction module is configured to perform at least one of the following:

[0024] determine a first area proportion of a face region in each of the video frames in the corresponding video frame;

[0025] determine a pixel value variance of a pixel point in the face region in each of the video frames;

[0026] determine a second area proportion of an edge of the face region in each of the video frames in the corresponding face region.

[0027] In a possible implementation, the first video frame classification module is configured to, for any video frame in the plurality of video frames, determine a first classification result of the video frame as a defective frame in a case where at least one of the following conditions is met: the first area proportion corresponding to the video frame is less than a first area threshold, the pixel value variance corresponding to the video frame is less than a variance threshold, and the second area proportion corresponding to the video frame is less than a second area threshold.

[0028] In a possible implementation, the first video frame classification module is configured to, for any video frame in the plurality of video frames, determine a first classification result of the video frame as a non-defective frame in a case where any one of the following conditions is met: the first area proportion corresponding to the video frame is greater than or equal to the first area threshold, the pixel value variance corresponding to the video frame is greater than or equal to the variance threshold, and the second area proportion corresponding to the video frame is greater than or equal to the second area threshold.

[0029] In a possible implementation, the first feature extraction module is configured to, for any video frame in the plurality of video frames, perform any one of convolution, full connection, and attention encoding on the video frame to obtain deep features of the video frame.

[0030] In a possible implementation, the first video frame classification module is configured to perform defect frame prediction based on the deep features of each video frame to obtain a first classification value of each video frame, the first classification value being used to represent a possibility that the corresponding video frame is a defect frame; perform tampering frame prediction based on the deep features of each video frame to obtain a second classification value of each video frame, the first classification value being used to represent a possibility that the corresponding video frame is a tampering frame; and determine a second classification result of each video frame based on the first classification value and the second classification value of each video frame.

[0031] In a possible implementation, the first video frame classification module is configured to fuse the first classification value and the second classification value of each video frame to obtain a third classification value of each video frame; and determine a second classification result of each video frame based on a size relationship between the third classification value and a classification value threshold of each video frame.

[0032] In a possible implementation, the first defect frame determination module is configured to fuse the manual features and the deep features of each video frame to obtain fusion features of each video frame; fuse the first classification result and the second classification result of each video frame to obtain a fusion classification result of each video frame; and determine the at least one defect frame from the plurality of video frames based on the fusion features and the fusion classification result of each video frame.

[0033] In a possible implementation, the first defect frame determination module is configured to splice the fusion features and the fusion classification result of each video frame to obtain target classification features of each video frame; perform full connection and normalization on the target classification features of each video frame to obtain a target classification value of each video frame; and determine a video frame with a target classification value greater than a classification value threshold in the plurality of video frames as a defect frame.

[0034] In a possible implementation, the first video classification module is configured to perform feature extraction on the at least one defect frame to obtain image features of the at least one defect frame; perform classification based on the image features of the at least one defect frame to determine a type of the at least one defect frame; in a case where any defect frame in the at least one defect frame is a tampering frame, determine that the face verification video is a tampered video; and in a case where none of the at least one defect frame is a tampering frame, determine that the face verification video is not a tampered video.

[0035] In an aspect, an embodiment of the present specification provides a training device of a face video classification model, comprising:

[0036] An input module configured to input a sample face verification video into the face video classification model.

[0037] a second feature extraction module configured to perform feature extraction on the plurality of video frames of the sample face verification video by using the face video classification model, to obtain hand-crafted features and deep features of each of the video frames;

[0038] a second video frame classification module configured to perform classification on each of the video frames based on the hand-crafted features by using the face video classification model, to obtain a first classification result of each of the video frames; and perform classification on each of the video frames based on the deep features, to obtain a second classification result of each of the video frames, wherein the first classification result and the second classification result are used to indicate whether each of the video frames is a defective frame, and the defective frame is a video frame that may have tampering defects;

[0039] a second defective frame determination module configured to determine at least one defective frame from the plurality of video frames based on the hand-crafted features, the deep features, the first classification result and the second classification result of each of the video frames by using the face video classification model;

[0040] a second video classification module configured to determine a predicted type of the sample face verification video based on the at least one defective frame by using the face video classification model, wherein the predicted type is used to indicate whether the sample face verification video is a tampered video;

[0041] a training module configured to train the face video classification model based on first difference information between the predicted type of the sample face verification video and a labeled type of the sample face verification video.

[0042] In a possible implementation, the second video classification module is configured to, for any defective frame in the at least one defective frame, input the defective frame into a compression parameter generator, generate a compression parameter of the defective frame by using the compression parameter generator, compress the defective frame based on a compression method corresponding to the compression parameter generator by using the compression parameter of the defective frame, to obtain a compressed defective frame, and input the defective frame and the compressed defective frame into a classifier, and output the predicted type of the sample face verification video by using the classifier.

[0043] In a possible implementation, the training module is configured to perform adversarial training on the compression parameter generator and the classifier based on the first difference information.

[0044] In a possible implementation, the second video frame classification module is configured to: perform defect frame prediction based on the deep features of each video frame to obtain a first classification value of each video frame, the first classification value being used to represent a possibility that the corresponding video frame is a defect frame; perform tampering frame prediction based on the deep features of each video frame to obtain a second classification value of each video frame, the first classification value being used to represent a possibility that the corresponding video frame is a tampering frame; and determine a second classification result of each video frame based on the first classification value and the second classification value of each video frame.

[0045] In a possible implementation, the training module is further configured to train the face video classification model based on second difference information between the first classification value of each video frame and a first annotation type of each video frame, and third difference information between the second classification value of each video frame and a second annotation type of each video frame, wherein the first annotation type is used to represent whether the corresponding video frame is a defect frame, and the second annotation type is used to represent whether the corresponding video frame is a tampering frame.

[0046] In a possible implementation, the training module is further configured to train the face video classification model based on fourth difference information between the at least one defect frame and at least one annotated defect frame in the plurality of video frames.

[0047] In an aspect, an embodiment of the present specification provides a computer storage medium, which stores a plurality of instructions, the instructions being suitable for being loaded and executed by a processor to perform the method described above.

[0048] In an aspect, an embodiment of the present specification provides an electronic device, which comprises a processor and a memory, wherein the memory stores a computer program, the computer program being suitable for being loaded and executed by the processor to perform the method described above.

[0049] In an aspect, an embodiment of the present specification provides a computer program product containing instructions, which, when executed on a computer or a processor, cause the computer or the processor to perform the method described above.

[0050] The technical solution provided by the embodiments of the present specification extracts features of a plurality of video frames of a face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and the deep features of each video frame to obtain a first classification result and a second classification result of each video frame. At least one defective frame is determined from the plurality of video frames based on the manual features, the deep features, the first classification result, and the second classification result of each video frame. Whether the face verification video is a tampered video, i.e., whether the face verification video is a video processed by deepfakes technology, is determined based on the at least one defective frame, thereby improving the security of face recognition. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present specification, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 A schematic diagram of an implementation environment of a face video classification method provided by the embodiments of the present specification;

[0053] Figure 2 A flowchart of a face video classification method provided by the embodiments of the present specification;

[0054] Figure 3 A flowchart of another face video classification method provided by the embodiments of the present specification;

[0055] Figure 4 A structural schematic diagram of a manual classification sub-model provided by the embodiments of the present specification;

[0056] Figure 5 A structural schematic diagram of a deep classification sub-model provided by the embodiments of the present specification;

[0057] Figure 6 A structural schematic diagram of a fusion classification sub-model provided by the embodiments of the present specification;

[0058] Figure 7 A flowchart of a training method of a face video classification model provided by the embodiments of the present specification;

[0059] Figure 8 A structural schematic diagram of a face video classification model provided by the embodiments of the present specification;

[0060] Figure 9A structural schematic diagram of a face video classification device provided by an embodiment of the present specification;

[0061] Figure 10 A structural schematic diagram of a face video classification model training device provided by an embodiment of the present specification;

[0062] Figure 11 A structural schematic diagram of an electronic device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0063] To make the features and advantages of the present specification more obvious and easy to understand, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present specification, but not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present specification.

[0064] First, the terms related to one or more embodiments of the present specification are explained.

[0065] Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0066] Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its application covers various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0067] Biometrics: biometric technology is a combination of computer and high-tech means such as optics, acoustics, biosensors and biostatistics principles, which uses inherent physiological characteristics of human body (such as fingerprints, face, iris, etc.) and behavior characteristics (such as handwriting, voice, gait, etc.) to identify personal identity.

[0068] Face recognition: face recognition is a biometric technology based on face feature information for identity recognition. Use camera or camera to collect image or video stream containing face, and automatically detect and track face in image, and then perform a series of related technologies for face recognition of detected face, which is also called portrait recognition, face recognition.

[0069] Defect frame: In the embodiments of the present specification, it refers to a frame containing certain characteristics (i.e. defects) that cause the deepfake generation effect to be poor. Typical defects include but are not limited to face occlusion, extreme lighting conditions, rapid motion, etc.

[0070] Deepfakes: refers to a technology that uses algorithms to tamper / generate face verification videos, for example, replacing the face of A in the video with the face of B, which brings great risk to biometric authentication and content security.

[0071] Deepfakes detection: refers to a technology that uses deep learning, machine learning, etc. to detect whether the video contains Deepfakes tampering.

[0072] Hand-crafted features: mainly refers to features that can be obtained based on the image itself through simple operations. For example, color, texture, shape, gradient, etc. belong to low-level features, and they are also the basis for intermediate feature operations. Feature: relatively simple, no need to learn and train, only need simple calculation and statistics.

[0073] Deep features: usually design neural network models to dig deeper and more abstract features of images.

[0074] Normalization: map the number series with different value ranges to the (0, 1) interval, which is convenient for data processing. In some cases, the normalized numerical value can be directly implemented as a probability.

[0075] Dropout: is a method for optimizing artificial neural networks with deep structure. During the learning process, by randomly setting some weights or outputs of the hidden layer to zero, the interdependence between nodes is reduced, thereby realizing the regularization of the neural network and reducing its structural risk. For example, during model training, there is a vector (1, 2, 3, 4), after inputting the vector into the dropout layer, the dropout layer can randomly convert a number in the vector (1, 2, 3, 4) to 0, such as converting 2 to 0, then the vector becomes (1, 0, 3, 4).

[0076] Learning rate: used to control the learning progress of the model. Learning rate can guide the model to adjust network weights using the gradient of the loss function in the gradient descent method. If the learning rate is too large, the loss function may directly pass through the global optimum, which is manifested as a large loss; if the learning rate is too small, the change speed of the loss function is slow, which greatly increases the convergence complexity of the network, and it is easy to be trapped in local minimum or saddle point.

[0077] Embedded Coding: Embedded coding represents a corresponding relationship in mathematics, that is, mapping data on X space to Y space through a function F, where the function F is a single function, and the mapping result is structure preservation. The single function means that the mapped data is uniquely corresponding to the pre-mapped data, and the structure preservation means that the size relationship of the pre-mapped data is the same as that of the post-mapped data. For example, there are data X1 and X2 before mapping, and Y1 corresponding to X1 and Y2 corresponding to X2 after mapping. If the data X1 > X2 before mapping, then the data Y1 > Y2 after mapping correspondingly. For words, it is to map words to another space, which is convenient for subsequent machine learning and processing.

[0078] Attention weight: can represent the importance of certain data in the training or prediction process. The importance represents the size of the influence of the input data on the output data. The data with high importance has a higher value of the corresponding attention weight, and the data with low importance has a lower value of the corresponding attention weight. In different scenarios, the importance of data is not the same, and the process of training attention weight of the model is also the process of determining the importance of data.

[0079] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present specification are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the face video involved in the embodiments of the present specification is obtained under sufficient authorization.

[0080] Next, the implementation environment of the technical solutions provided by the embodiments of the present specification is introduced.

[0081] Figure 1 is an implementation environment schematic diagram of a face video classification method provided by the embodiments of the present specification, referring to Figure 1 The implementation environment includes a terminal 110 and a server 120.

[0082] The terminal 110 is connected to the server 120 through a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, etc., but is not limited thereto. The terminal 110 is installed and runs an application program supporting face recognition.

[0083] The server 120 is a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and big data and artificial intelligence platform. The server 120 provides background services for the application program running on the terminal 110, for example, the server 120 provides corresponding services for the application program running on the terminal 110, in the embodiment of the present specification, the server 120 provides background services for the application program running on the terminal supporting face recognition, that is, classifies the face verification video before face recognition to determine whether the face verification video is a tampered video.

[0084] Those skilled in the art can know that the number of the terminal 110 and the server 120 described above can be more or less. For example, the terminal 110 and the server 120 described above are only one, or the terminal 110 and the server 120 described above are dozens or hundreds, or more, at this time, the implementation environment described above also includes other terminals and servers, and the number of terminals and the type of equipment are not limited in the embodiment of the present specification.

[0085] After introducing the implementation environment of the embodiment of the present specification, the application scenario of the embodiment of the present specification will be introduced in combination with the implementation environment described above. In the following description, the terminal is the terminal 110 in the implementation environment described above, and the server is the server 120 in the implementation environment described above. The technical scheme provided by the embodiment of the present specification can be applied in various application scenarios of face recognition system, for example, applied in various payment applications providing face payment function, or applied in various payment devices providing face payment function, or applied in various vending machines with face payment function, or applied in various access control devices with face recognition, which is not limited in the embodiment of the present specification.

[0086] In the technical solution provided by the embodiments of the present specification, the technical solution is applied in various payment applications providing face payment functions, for example, when using the face payment function provided by the payment application, the terminal collects a face verification video of a target object, sends the face verification video to the server, the server receives the face verification video, the target object is a user using the face payment function, and the face verification video is used to verify the identity of the target object. The server extracts features of a plurality of video frames of the face verification video to obtain manual features and deep features of each video frame. The manual features are also referred to as statistical features, and the deep features are also referred to as model features. The server classifies based on the manual features of each video frame to obtain a first classification result of each video frame. The server classifies based on the deep features of each video frame to obtain a second classification result of each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defective frame, that is, a video frame that may have tampering defects. For any video frame in the plurality of video frames, the first classification result and the second classification result of the video frame are classification results obtained based on different types of features. The server determines at least one defective video frame from the plurality of video frames based on the manual features, the deep features, the first classification result, and the second classification result of each video frame, and concentrates processing on the at least one defective video frame. The server determines whether the face verification video is a tampered video based on the at least one defective frame, that is, determines whether the face verification video is a video tampered by Deepfakes, improves the security of face recognition, and accordingly, classifying the face verification video by using the technical solution provided by the embodiments of the present specification is Deepfakes detection of the face verification video.

[0087] The above is an example of the technical solution provided by the embodiments of the present specification applied in various payment applications providing face payment functions. In the above other application scenarios, the face verification video can be classified by the above method, and the specific process is not repeated here.

[0088] After introducing the implementation environment and application scenario of the embodiments of the present specification, the technical solution provided by the embodiments of the present specification is introduced, referring to Figure 2 The execution subject is the server, and the method includes the following steps.

[0089] 202. The server extracts features of a plurality of video frames of the face verification video to obtain manual features and deep features of each video frame.

[0090] The face verification video is a video used for identity verification, that is, a video used for face recognition. The face verification video includes a plurality of video frames. The feature extraction process performed by the server on the plurality of video frames is also a process of abstractly representing the plurality of video frames, facilitating subsequent processing of the plurality of video frames.

[0091] 204. The server classifies each video frame based on the manual feature of each video frame to obtain a first classification result of each video frame.

[0092] The classification based on the manual feature of each video frame is also a classification of each video frame according to a rule in which the manual feature of each video frame conforms to the rule, and the rule is set by a technician according to actual conditions, and embodiments of the present specification are not limited thereto. The classification of the video frame is to determine whether the video frame is a defective frame, and the first classification result is the result of the classification of the video frame by the server using the manual feature.

[0093] 206. The server classifies each video frame based on the deep feature of each video frame to obtain a second classification result of each video frame, and the first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects.

[0094] The classification based on the deep feature of each video frame is also a classification of the video frame using machine learning technology, and the second classification result is the result of the classification of the video frame by the server using the deep feature. The first classification result and the second classification result are both used to indicate whether the video frame is a defective frame, and the first classification result and the second classification result may be the same or different.

[0095] 208. The server determines at least one defective frame from the plurality of video frames based on the manual feature, the deep feature, the first classification result, and the second classification result of each video frame.

[0096] The server determines at least one defective frame from the plurality of video frames based on the manual feature, the deep feature, the first classification result, and the second classification result of each video frame, that is, the classification of the video frame is realized by comprehensively using the manual feature and the deep feature, and the accuracy of the at least one defective frame obtained is higher.

[0097] 210. The server determines whether the face verification video is a tampered video based on the at least one defective frame.

[0098] The at least one defective frame is determined from the plurality of video frames, so that the processing range in video classification is reduced, attention can be focused on the defective frame with higher discrimination in the classification process, and the influence of the non-defective frame on the classification result is avoided, the generalization ability of the method is improved, and the video classification accuracy is improved.

[0099] The technical solution provided by the embodiments of the present specification extracts features from a plurality of video frames of a face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and deep features of each video frame to obtain first classification results and second classification results of each video frame. At least one defective frame is determined from the plurality of video frames based on the manual features, deep features, first classification results, and second classification results of each video frame. Whether the face verification video is a tampered video is determined based on the at least one defective frame, that is, whether the face verification video is a video processed by deepfakes technology, thereby improving the security of face recognition.

[0100] The above steps 202-210 are a brief introduction to the technical solution provided by the embodiments of the present specification. In order to more clearly illustrate the technical solution provided by the embodiments of the present specification, the technical solution provided by the embodiments of the present specification will be described below in combination with some examples, referring to Figure 3 The method comprises the following steps.

[0101] 302. The server obtains a face verification video of a target object.

[0102] The face verification video is a face video of the target object, and the face verification video is used for using a face recognition service. The face recognition service is used for identity verification, and the target object is a user using the face recognition service. In some embodiments, the face verification video is collected by the target object when using the face recognition service. The collection and use of the face verification video are fully authorized by the target object. In addition, the server does not store the face verification video of the target object. After completing video classification and face recognition based on the face verification video, the server deletes the face verification video or anonymizes the face verification video to eliminate the association between the face verification video and the target object, thereby protecting the privacy of the target object.

[0103] In one possible implementation, the server obtains the face verification video of the target object through a terminal.

[0104] The terminal is a terminal running a payment application, or a payment device providing a face payment function, or a vending machine with a face payment function, or an access control device with face recognition, etc., which is not limited in the embodiments of the present specification.

[0105] In this implementation, the server can quickly obtain the face verification video of the target object from the terminal, and subsequently perform identity verification on the target object based on the face verification video.

[0106] For example, the server obtains an initial verification video of a target object uploaded by a terminal, and the initial verification video is used for identity verification. The server performs face detection on the initial verification video to determine whether the initial verification video includes a face. In the case where the initial verification video includes a face, the server determines the initial verification video as the face verification video of the target object. In the case where the initial verification video does not include a face, the server sends prompt information to the terminal, and the prompt information is used to prompt the target object to re-shoot the initial verification video. Through this implementation, the server can perform face recognition on the initial verification video uploaded by the terminal, and further process based on the initial verification video in the case where the initial verification video includes a face, thereby avoiding wasting the operation resources of the server. Of course, in some embodiments, the terminal can also determine whether the initial verification video includes a face, and the initial verification video is uploaded to the server only in the case where the initial verification video includes a face, and the initial verification video is the face verification video of the target object. In the case where the initial verification video does not include a face, the terminal displays prompt information.

[0107] For example, in response to a face recognition operation, the terminal displays a face recognition interface, and the face recognition interface is used to prompt the target object to start face recognition. After a target time period, the terminal collects an initial verification video of the target object, and the target time period is set by a technician according to actual conditions, which is not limited in the embodiments of the present specification. The terminal sends the collected initial verification video of the target object to the server, and the server obtains the initial verification video of the target object. The server inputs the initial verification video into a face recognition model to identify the initial verification video through the face recognition model, and determines whether the initial verification video includes a face, wherein the face recognition model can be any structure model, which is not limited in the embodiments of the present specification. In the case where the initial verification video includes a face, the server determines the initial verification video as the face verification video of the target object.

[0108] In a possible implementation, the server obtains the face verification video of the target object from an object video database that stores face verification videos of a plurality of objects. In some embodiments, for any object in the plurality of objects, after the terminal used by the object collects the face verification video of the object, the face verification video of the object is uploaded to the object video database, the server obtains the face verification video of the object from the object video database, and subsequent operations related to face recognition are performed based on the face verification video of the object. Of course, after the server processes the face verification videos in the object video database, the face verification videos in the object video database are deleted, so as to protect the privacy of the object and avoid misuse of the face verification videos.

[0109] In this implementation, by setting the object video database as a transit of the face verification video, the success rate of face recognition and video classification is improved in the case that a large number of concurrent face recognition and video classification tasks cannot be processed in time by the server, resulting in data loss.

[0110] In some embodiments, after the server obtains the face verification video, the server can also score the quality of the face verification video to obtain a video quality score of the face verification video. In the case that the video quality score of the face verification video is greater than or equal to a video quality score threshold, the server performs subsequent video classification based on the face verification video. In the case that the video quality score of the face verification video is less than the video quality score threshold, the server re-obtains the face verification video of the target object. The video quality score is used to represent the quality of the face verification video, and the higher the video quality score, the better the quality of the face verification video. The lower the video quality score, the worse the quality of the face verification video. The video quality score threshold is set by a technician according to actual conditions, and embodiments of the present specification do not limit this.

[0111] Through this implementation, the server can score the quality of the face verification video before performing video classification based on the face verification video, and filter the face verification video based on the video quality score of the face verification video, so as to ensure that the video classification is performed based on a face verification video with high quality, and the accuracy of the video classification is ensured.

[0112] 304. The server performs manual feature extraction on the plurality of video frames of the face verification video to obtain manual features of the video frames.

[0113] The manual features are unsupervised features that can be obtained without training, and the manual features are also referred to as statistical features or traditional features.

[0114] In a possible implementation, the server inputs the plurality of video frames into a manual feature extractor, extracts manual features of the plurality of video frames through the manual feature extractor, and obtains manual features of each video frame.

[0115] The manual feature extractor belongs to a face video classification model, the face video classification model is used for classifying face videos, the manual feature extractor is used for extracting manual features, and a type of the manual features extracted by the manual feature extractor is set by a technician according to an actual situation, and embodiments of the specification are not limited in this regard. The manual feature extractor integrates a plurality of manual feature extraction methods, and directly uses a corresponding method to extract manual features, without training.

[0116] Through the above implementation, the server can quickly extract manual features of video frames through the manual feature extractor, and the extraction efficiency of the manual features is relatively high.

[0117] It should be noted that the manual features have a plurality of types, and the above implementation is described below by taking extraction of different types of manual features as an example.

[0118] Example 1: The server inputs the plurality of video frames into a manual feature extractor, determines a first area ratio of a face region in each video frame in the corresponding video frame through the manual feature extractor.

[0119] In a possible implementation, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection on the video frame through the manual feature extractor, and obtains a face region in the video frame. The server determines a first area ratio of the face region in the video frame through the manual feature extractor, for example, the server divides the number of pixel points in the face region by the number of all pixel points in the video frame to obtain the first area ratio of the face region in the video frame.

[0120] The first area ratio of the face region in the video frame can represent the size of the face in the video frame, the higher the first area ratio, the larger the face in the video frame; the lower the first area ratio, the smaller the face in the video frame.

[0121] In this implementation, the server can perform face detection on the video frame through the manual feature extractor to obtain the face region in the video frame, and the first area ratio of the face region in the video frame is a type of manual features.

[0122] In a possible implementation, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection and area calculation on the video frame through the manual feature extractor, and obtains a first area ratio of a face region in the video frame in the video frame.

[0123] In some embodiments, the manual feature extractor is a multi-task processing model through which face detection and area calculation can be simultaneously performed on the video frame. For example, the manual feature extractor is a multi-task cascaded convolutional network (MTCNN).

[0124] In this implementation, the first area ratio can be obtained through single processing of the manual feature extractor, and the determination efficiency of the first area ratio is relatively high.

[0125] Example 2: The server inputs the plurality of video frames into the manual feature extractor, and determines, through the manual feature extractor, a pixel value variance of a pixel point in a face region in each video frame.

[0126] In a possible implementation, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection on the video frame through the manual feature extractor, and obtains a face region in the video frame. The server determines, through the manual feature extractor, a pixel value variance of a pixel point in the face region.

[0127] The pixel value variance of the pixel point in the face region is used to reflect the illumination or blurring in the face region. The smaller the pixel value variance, the stronger or more blurred the illumination in the face region; the larger the pixel value variance, the weaker or clearer the illumination in the face region.

[0128] In this implementation, the server can determine the face region in the video frame through the manual feature extractor, and further determine the pixel value variance of the pixel point in the face region, which is a kind of manual feature.

[0129] For example, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection on the video frame through the manual feature extractor, and obtains a face region in the video frame. The server determines, through the manual feature extractor, an average pixel value of a pixel point in the face region. The server determines, through the manual feature extractor, a pixel value variance of the pixel point in the face region based on the average pixel value and a pixel value of the pixel point in the face region.

[0130] In example 3, the server inputs the plurality of video frames into the manual feature extractor, and determines, by the manual feature extractor, a second area proportion of edges of a face region in each video frame in the corresponding face region.

[0131] In a possible implementation, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection on the video frame by the manual feature extractor, and obtains a face region in the video frame. The server performs edge detection on the face region by the manual feature extractor, and obtains edges of the face region. The server determines, by the manual feature extractor, the second area proportion of the edges in the face region.

[0132] In this implementation, the server can perform face detection on a video frame by the manual feature extractor to obtain a face region in the video frame. The edges of the face region are obtained by performing edge detection on the face region by the manual feature extractor, and the second area proportion of the edges in the corresponding face region is determined, which is a manual feature.

[0133] For example, for any video frame in the plurality of video frames, the server inputs the video frame into the manual feature extractor, performs face detection on the video frame by the manual feature extractor, and obtains a face region in the video frame. The server performs edge detection on the face region by the manual feature extractor, and obtains edges of the face region. The server divides the number of pixel points in the edges by the number of all pixel points in the face region by the manual feature extractor, and obtains the second area proportion of the edges in the face region.

[0134] It should be noted that the above is an example of the server extracting the first area proportion, the pixel value variance, and the second area proportion by the manual feature extractor respectively, and in other possible implementations, the server can extract any two or all of the first area proportion, the pixel value variance, and the second area proportion by the manual feature extractor at the same time, which is not limited in the embodiments of the present disclosure.

[0135] In addition, since the extraction of the above three manual features all depends on the face region in the video frame, in the process of extracting any two of the above manual features at the same time, only one face detection needs to be performed, without repeated execution, thereby saving the operation resources and improving the efficiency of manual feature extraction.

[0136] In addition, the above example is an example of performing manual feature extraction on any video frame in the plurality of video frames, and the method of performing manual feature extraction on other video frames in the plurality of video frames belongs to the same inventive concept as the above description, and the implementation process is not described again.

[0137] 306、The server extracts deep features of the plurality of video frames of the face verification video to obtain deep features of each video frame.

[0138] The deep features are supervised features that need to be trained to obtain. The deep features are also referred to as model features. The purpose of feature extraction of the video frame is to abstract the video frame to facilitate subsequent video classification.

[0139] In a possible implementation, the server inputs the plurality of video frames into a deep feature extractor, and extracts deep features of the plurality of video frames by the deep feature extractor to obtain deep features of each video frame.

[0140] The deep feature extractor belongs to a face video classification model, and is configured to extract deep features.

[0141] For example, for any video frame in the plurality of video frames, the server inputs the video frame into the deep feature extractor, and performs any one of convolution, full connection, and attention encoding on the video frame by the deep feature extractor to obtain deep features of the video frame.

[0142] To make the above implementation clearer, the following examples are used to further illustrate the feature extraction manner in the above examples.

[0143] Example 1: For any video frame in the plurality of video frames, the server performs convolution on the video frame by a deep feature extractor of the face video classification model to output deep features of the video frame.

[0144] In a possible implementation, for any video frame in the plurality of video frames, the server performs sliding on the video frame by a convolution layer of the deep feature extractor using at least one convolution kernel, and performs convolution operation on a covered part in the sliding process to obtain deep features of the video frame. For example, the server performs sliding on the video frame by the convolution layer of the deep feature extractor using a plurality of convolution kernels, and performs convolution operation on a covered part in the sliding process to obtain a plurality of convolution features respectively corresponding to the plurality of convolution kernels. The server fuses the plurality of convolution features to obtain deep features of the video frame. In some embodiments, the number of the plurality of convolution kernels is an integer multiple of the number of color channels of the video frame.

[0145] In the above implementation, the server can extract deep features of the video frame by convolution operation. Since the convolution operation is fast, the server can quickly extract features of the video frame.

[0146] It should be noted that the above is an example of the server extracting features of a video frame through the deep feature extractor, and the manner of extracting features of other video frames in the plurality of video frames by the server belongs to the same inventive concept as the manner of extracting features of the video frame, which will not be repeated here. In addition, the above is an example of the server extracting features of a single video frame, and in other possible implementations, the server can also extract features of multiple video frames simultaneously, which is not limited by the embodiments of the present application.

[0147] Example 2, for any video frame in the plurality of video frames, the server extracts features of the video frame through the deep feature extractor of the face video classification model.

[0148] In a possible implementation, for any video frame in the plurality of video frames, the server multiplies the video frame by at least one fully connected matrix through the fully connected layer of the deep feature extractor to obtain the deep features of the video frame. For example, the server multiplies the video frame by a plurality of fully connected matrices through the fully connected layer of the deep feature extractor and then performs pooling to obtain the deep features of the video frame. By performing pooling, the dimensionality of the features can be reduced, improving the efficiency of subsequent adversarial attack detection.

[0149] In the above implementation, the server can extract deep features of a video frame through full connection. Since full connection is fast, the server can quickly extract features of a video frame.

[0150] It should be noted that the above is an example of the server extracting features of a video frame through the deep feature extractor, and the manner of extracting features of other video frames in the plurality of video frames by the server belongs to the same inventive concept as the manner of extracting features of the video frame, which will not be repeated here. In addition, the above is an example of the server extracting features of a single video frame, and in other possible implementations, the server can also extract features of multiple video frames simultaneously, which is not limited by the embodiments of the present application.

[0151] Example 3, for any video frame in the plurality of video frames, the server encodes the video frame through the deep feature extractor of the face video classification model. Attention, output the deep features of the video frame.

[0152] In a possible implementation, for any video frame in the plurality of video frames, the server divides the video frame into a plurality of parts. The server embeds encodes the plurality of parts through the attention encoding layer of the deep feature extractor to obtain a plurality of embedding features of the plurality of parts, one embedding feature corresponding to one part. The server determines attention weights between each two parts in the plurality of parts based on the plurality of embedding features of the plurality of parts through the attention encoding layer of the deep feature extractor. The server determines a plurality of attention features of the plurality of parts based on the plurality of embedding features and the attention weights between each two parts in the plurality of parts through the attention encoding layer of the deep feature extractor. The server fuses the plurality of attention features of the plurality of parts through the attention encoding layer of the deep feature extractor to obtain the deep feature of the video frame.

[0153] In this implementation, the server encodes the video frame based on the attention mechanism to obtain the deep feature of the video frame, and the attention mechanism can make full use of the association between the parts of the video frame, and the obtained deep feature can more accurately reflect the characteristics of the video frame.

[0154] For example, for any video frame in the plurality of video frames, the server divides the video frame into a plurality of parts, the plurality of parts have the same size, each part is an image region of the video frame, and the plurality of parts have no overlapping positions. The server embeds encodes the plurality of parts through the attention encoding layer of the deep feature extractor to obtain a plurality of embedding features of the plurality of parts, one embedding feature corresponding to one part. The server respectively multiplies the plurality of embedding features with a query parameter matrix, a key parameter matrix, and a value parameter matrix through the attention encoding layer of the deep feature extractor to obtain a query matrix, a key matrix, and a value matrix of each part. The server determines attention weights between each two parts in the plurality of parts based on the query matrix and the key matrix of each two parts through the attention encoding layer of the deep feature extractor. The server fuses the value matrices of each two parts based on the attention weights between the two parts through the attention encoding layer of the deep feature extractor to obtain an attention feature between the two parts. The server fuses the plurality of attention features of the plurality of parts through the attention encoding layer of the deep feature extractor to obtain the deep feature of the video frame.

[0155] It should be noted that the server can extract the deep feature of the video frame through any of the above manners, and the embodiments of the present specification do not limit this.

[0156] Further, the above examples are described by taking the depth feature extraction of any video frame in the plurality of video frames as an example, the method of depth feature extraction of other video frames in the plurality of video frames belongs to the same inventive concept as described above, and the implementation process will not be described again.

[0157] In addition, the above steps 304 and 306 can be executed in sequence or synchronously, and the embodiments of the present specification do not limit this.

[0158] 308、The server classifies based on the manual features of each video frame to obtain a first classification result of each video frame.

[0159] The classification based on the manual features of each video frame is to classify each video frame according to the rule that the manual features of each video frame conform to, and the rule is set by the technical personnel according to the actual situation, and the embodiments of the present specification do not limit this. The classification of the video frame is to determine whether the video frame is a defective frame, and the first classification result is the result of the classification of the video frame by the server using the manual features.

[0160] In a possible implementation, for any video frame in the plurality of video frames, in the case where at least one of the following conditions exists: the first area ratio corresponding to the video frame is less than a first area threshold, the pixel value variance corresponding to the video frame is less than a variance threshold, and the second area ratio corresponding to the video frame is less than a second area threshold, the server determines the first classification result of the video frame as a defective frame.

[0161] The first area corresponding to the video frame refers to the first area ratio of the face region in the video frame, the pixel value variance corresponding to the video frame refers to the pixel value variance of the pixel points in the face region in the video frame, and the second area ratio corresponding to the video frame refers to the second area ratio of the edge of the face region in the face region. The first area threshold, the variance threshold, and the second area threshold are set by the technical personnel according to the actual situation, and the embodiments of the present specification do not limit this. The first area ratio corresponding to the video frame being less than the first area threshold indicates that the face region in the video frame is too small; the pixel value variance corresponding to the video frame being less than the variance threshold indicates that the face region in the video frame has extreme light or is relatively blurred; and the second area ratio corresponding to the video frame being less than the second area threshold indicates that the face region in the video frame has obvious motion. The defective frame is a video frame that may have tampering defects, for example, the video frame can be determined as a defective frame when at least one of the following conditions exists in the face region in the video frame: face occlusion, extreme light, and rapid motion.

[0162] In a case where any one of the first area ratio corresponding to the video frame is greater than or equal to the first area threshold, the pixel value variance corresponding to the video frame is greater than or equal to the variance threshold, and the second area ratio corresponding to the video frame is greater than or equal to the second area threshold exists, the server determines the first classification result of the video frame as a non-defect frame.

[0163] In this implementation, the server can realize classification of the video frame based on the manual features. The classification is realized based on rules, and the rule-based classification can classify some obvious defect frames. Subsequently, combined with the rule classification result, a more accurate classification effect can be achieved.

[0164] The following will be described in combination with Figure 4 The steps 304 and 308 are described below.

[0165] Referring to Figure 4 In some embodiments, the server extracts manual features of each video frame by the manual feature extractor 401. The server classifies each video frame based on the manual features of each video frame by the manual feature classifier 402, and outputs a first classification result of each video frame. The manual feature extractor 401 and the manual feature classifier 402 belong to the manual classification sub-model 400, and the manual classification sub-model 400 belongs to the face video classification model.

[0166] 310. The server classifies based on the depth features of each video frame to obtain a second classification result of each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defect frame, and the defect frame is a video frame that may have tampering defects.

[0167] The classification based on the depth features of each video frame is to classify the video frame by using machine learning technology. The second classification result is a result of classifying the video frame by the server using the depth features. The first classification result and the second classification result are both used to indicate whether the video frame is a defect frame. The first classification result and the second classification result may be the same or different.

[0168] In a possible implementation, the server performs defect frame prediction based on the depth features of each video frame to obtain a first classification value of each video frame. The first classification value is used to indicate the possibility that the corresponding video frame is a defect frame. The server performs tampering frame prediction based on the depth features of each video frame to obtain a second classification value of each video frame. The first classification value is used to indicate the possibility that the corresponding video frame is a tampering frame. The server determines the second classification result of each video frame based on the first classification value and the second classification value of each video frame.

[0169] The defect frame is a video frame that may have tampering defects, and the tampered frame is a video frame that may be tampered, such as a video frame tampered by a Deepfakes technology.

[0170] In this implementation, the server can combine the defect frame prediction and the tampered frame prediction to realize the final defect frame prediction, and the accuracy of the defect frame prediction is higher.

[0171] In order to more clearly illustrate the above implementation, the following will be divided into three parts to illustrate the above implementation.

[0172] In the first part, the server performs defect frame prediction based on the deep features of each video frame to obtain a first classification value of each video frame.

[0173] In a possible implementation, the server inputs each video frame into a defect frame predictor, and performs full connection and normalization on the deep features of each video frame through the defect frame predictor to output a first classification value of each video frame.

[0174] The defect frame predictor is a classifier, and the first classification value is used to represent the possibility of the corresponding video frame being a defect frame. For example, the first classification value is a probability, and the higher the first classification value of a video frame, the higher the possibility of the video frame being a defect frame; the lower the first classification value of the video frame, the lower the possibility of the video frame being a defect frame. In some embodiments, the defect frame predictor is obtained through pre-training and joint training. The defect frame predictor is pre-trained by using a plurality of sample video frames and a plurality of sample video frame labels. The plurality of sample video frames include defect frames and non-defect frames, and the sample video frame label is used to represent the type of the corresponding sample video frame, that is, the Ground Truth in pre-training. The joint training is a fine-tuning in the process of training the face video classification model.

[0175] In the second part, the server performs tampered frame prediction based on the deep features of each video frame to obtain a second classification value of each video frame.

[0176] In a possible implementation, the server inputs each video frame into a tampered frame predictor, and performs full connection and normalization on the deep features of each video frame through the tampered frame predictor to output a second classification value of each video frame.

[0177] The tampered frame predictor is a classifier, and the second classification value is used to represent the possibility of the corresponding video frame being a tampered frame. For example, the second classification value is a probability, and the higher the second classification value of a video frame, the higher the possibility of the video frame being a tampered frame; and the lower the second classification value of the video frame, the lower the possibility of the video frame being a tampered frame. In some embodiments, the tampered frame predictor is obtained through pre-training and joint training. The pre-training of the tampered frame predictor is performed by using a plurality of sample video frames and a plurality of sample video frame labels. The plurality of sample video frames include tampered frames and non-tampered frames, and the sample video frame label is used to represent the type of the corresponding sample video frame, that is, the Ground Truth in pre-training. The joint training is fine-tuning performed in the process of training the face video classification model.

[0178] The third part, the server determines the second classification result of each video frame based on the first classification value and the second classification value of each video frame.

[0179] In a possible implementation, the server fuses the first classification value and the second classification value of each video frame to obtain a third classification value of each video frame. The server determines the second classification result of each video frame based on the size relationship between the third classification value of each video frame and the classification threshold value.

[0180] The third classification value is used to represent the possibility of the corresponding video frame being a defective frame. The classification threshold value is set by a technician according to actual conditions, and the embodiments of the present specification are not limited in this regard.

[0181] In this implementation, the server can fuse the first classification value and the second classification value of each video frame, that is, fuse the defective frame prediction result and the tampered frame prediction result to obtain a third classification value for the final classification of the video frame, and the classification result of the video frame based on the third classification value is more accurate.

[0182] For example, for any video frame in the plurality of video frames, the server multiplies the first classification value and the second classification value of the video frame to obtain a third classification value of the video frame. In the case where the third classification value of the video frame is greater than or equal to the classification threshold value, the server determines the second classification result of the video frame as a defective frame; and in the case where the third classification value of the video frame is less than the classification threshold value, the server determines the second classification result of the video frame as a non-defective frame.

[0183] The above steps 306 and 310 will be described below. Figure 5 The above steps 306 and 310 will be described below.

[0184] Referring to Figure 5In some embodiments, the server performs deep feature extraction on each video frame by the deep feature extractor 501 to obtain deep features of each video frame. The server performs defect frame prediction based on the deep features of each video frame by the defect frame predictor 502 to obtain a first classification value of each video frame. The server performs tampering frame prediction based on the deep features of each video frame by the tampering frame predictor 503 to obtain a second classification value of each video frame. The deep feature extractor 501, the defect frame predictor 502, and the tampering frame predictor 503 belong to the deep classification sub-model 500, and the deep classification sub-model 500 belongs to the face video classification model.

[0185] In addition, as with steps 304 and 306 described above, steps 308 and 310 can be executed in sequence or synchronously, and the embodiments of the present specification do not limit this.

[0186] 312、The server fuses the hand-crafted features and the deep features of each video frame to obtain fused features of each video frame.

[0187] The hand-crafted features and the deep features are features obtained by two different methods and have strong complementarity. Fusing the hand-crafted features and the deep features can effectively fuse the advantages of the two feature extraction methods and form a better decision.

[0188] In a possible implementation, the server adds the hand-crafted features and the deep features of each video frame to obtain fused features of each video frame.

[0189] In this implementation, by adding the hand-crafted features and the deep features of each video frame, the relevant information of the hand-crafted features and the deep features can be completely retained, and the accuracy of subsequent classification based on the fused features is relatively high.

[0190] For example, the server performs dimension transformation on the hand-crafted features and the deep features of each video frame to make the dimensions of the hand-crafted features and the deep features of each video frame the same. The server adds the hand-crafted features and the deep features of each video frame after dimension transformation to obtain fused features of each video frame.

[0191] In a possible implementation, the server splices the hand-crafted features and the deep features of each video frame to obtain spliced features. The server performs any one of convolution and full connection on the spliced features to obtain fused features of each video frame.

[0192] In this implementation, the server splices the hand-crafted features and the deep features of each video frame to obtain spliced features, and further processes the spliced features to further improve the expression ability of the spliced features, and finally obtains fused features of each video frame.

[0193] In some embodiments, the server fuses the hand-crafted features and the deep features of each video frame by a feature fusioner of a fusion classification sub-model belonging to the face video classification model.

[0194] 314、The server fuses the first classification result and the second classification result of each video frame to obtain a fusion classification result of each video frame.

[0195] In a possible implementation, the server splices the first classification result and the second classification result of each video frame to obtain a fusion classification result of each video frame.

[0196] In this implementation, the server splices the first classification result and the second classification result of each video frame to obtain the fusion classification result, and the efficiency of obtaining the fusion classification result is high.

[0197] In a possible implementation, the server fuses the first classification result and the second classification result of each video frame by weighted fusion to obtain a fusion classification result of each video frame.

[0198] The weight of the weighted fusion is set by a technician according to actual conditions, and the embodiments of the present specification are not limited in this regard.

[0199] In this implementation, the server can fuse the first classification result and the second classification result by weighted fusion, so that the fusion classification result has stronger pertinence.

[0200] In some embodiments, the server fuses the first classification result and the second classification result of each video frame by a result fusioner of a fusion classification sub-model.

[0201] 316、The server determines the at least one defective frame from the plurality of video frames based on the fusion feature and the fusion classification result of each video frame.

[0202] In a possible implementation, the server fuses the fusion feature and the fusion classification result of each video frame to obtain a target classification feature of each video frame. The server fully connects and normalizes the target classification feature of each video frame to obtain a target classification value of each video frame. The server determines a video frame with a target classification value greater than a classification value threshold in the plurality of video frames as a defective frame, thereby obtaining at least one defective frame.

[0203] In this implementation, the server can determine a target classification feature based on the fusion feature and the fusion classification result, and perform final classification on the video frame based on the target classification feature, so that the classification accuracy is high.

[0204] For example, the server splices or adds the fusion features and the fusion classification results of the video frames to obtain target classification features of the video frames. The server performs full connection and normalization on the target classification features of the video frames to obtain target classification values of the video frames. The server determines a video frame with a target classification value greater than a classification value threshold in the multiple video frames as a defect frame, thereby obtaining at least one defect frame.

[0205] In some embodiments, the server classifies the video frames based on the target classification features through a fusion classifier of the fusion classification sub-model to obtain at least one defect frame.

[0206] The above steps 312, 314, and 316 will be described below. Figure 6 The above steps 312, 314, and 316 will be described below.

[0207] Referring to Figure 6 In some embodiments, the server fuses the manual features and the deep features of the video frames through the feature fusioner 601 to obtain fusion features of the video frames. The server fuses the first classification results and the second classification results of the video frames through the result fusioner 602 to obtain fusion classification results of the video frames. The server determines the at least one defect frame from the multiple video frames based on the fusion features and the fusion classification results of the video frames through the fusion classifier 603. The feature fusioner 601, the result fusioner 602, and the fusion classifier 603 all belong to the fusion classification sub-model 600, and the fusion classification sub-model 600 belongs to the face video classification model.

[0208] 318、The server determines whether the face verification video is a tampered video based on the at least one defect frame.

[0209] The determination of whether the face verification video is a tampered video is equivalent to the determination of whether the face verification video is a video processed by deepfakes technology.

[0210] In a possible implementation, the server extracts features from the at least one defect frame to obtain image features of the at least one defect frame. The server classifies based on the image features of the at least one defect frame to determine the type of the at least one defect frame. In a case where any defect frame in the at least one defect frame is a tampered frame, the server determines that the face verification video is a tampered video. In a case where none of the at least one defect frame is a tampered frame, the server determines that the face verification video is not a tampered video.

[0211] In this implementation, the server can classify the at least one defective frame, implement classification of the face verification video based on a classification result of classifying the at least one defective frame, and only focus on the defective frame when classifying the face verification video, thereby avoiding adverse effects of the non-defective frame on the classification result and improving the accuracy of classifying the face verification video.

[0212] For example, for any defective frame in the at least one defective frame, the server inputs the defective frame into a defective frame classification sub-model of the face video classification model, extracts features of the defective frame through the defective frame classification sub-model, and obtains image features of the defective frame. The server determines the type of the defective frame by full connection and normalization of the image features of the defective frame through the classification sub-model, and the type is a tampered frame or a non-tampered frame. For example, the server outputs a classification value of the defective frame by full connection and normalization of the image features of the defective frame through the classification sub-model. In a case where the classification value is greater than or equal to a classification value threshold, the server determines the defective frame as a tampered frame; and in a case where the classification value is less than the classification value threshold, the server determines the defective frame as a non-tampered frame.

[0213] Correspondingly, in a case where any defective frame in the at least one defective frame is a tampered frame, the server determines that the face verification video is a tampered video. In a case where none of the at least one defective frame is a tampered frame, the server determines that the face verification video is not a tampered video.

[0214] The defective frame classification sub-model is obtained by adversarial training with a compression parameter generator, the compression parameter generator is used to generate different compression parameters during training, and the input of the defective frame classification sub-model during training is a video frame compressed based on the compression parameters. Through adversarial training, the defective frame classification sub-model improves the generalization ability for compressed defective frames, thereby enhancing the accuracy of classifying defective frames. In this case, the defective frame classification sub-model is also referred to as a compression-robust defective frame classification sub-model.

[0215] In a case where the server determines that the face verification video is a tampered video, the server does not perform subsequent face recognition based on the face verification video; and in a case where the server determines that the face verification video is not a tampered video, the server performs subsequent face recognition based on the face verification video, thereby ensuring the security of face recognition.

[0216] All the optional technical solutions described above can be combined to form optional embodiments of the present specification, which will not be described here.

[0217] The technical solution provided by the embodiments of the present specification extracts features of a plurality of video frames of a face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and deep features of each video frame to obtain first classification results and second classification results of each video frame. At least one defective frame is determined from the plurality of video frames based on the manual features, deep features, first classification results, and second classification results of each video frame. Whether the face verification video is a tampered video, i.e., whether the face verification video is a video processed by deepfakes technology, is determined based on the at least one defective frame, thereby improving the security of face recognition.

[0218] In order to more clearly illustrate the technical solutions provided by the embodiments of the present specification, the training method of the face video classification model will be described below. Taking a server as an example for the execution subject of training the face video classification model, referring to Figure 7 , the method includes the following steps. The training of the model includes a plurality of iteration processes. One iteration process will be described below, and other iteration processes belong to the same inventive concept.

[0219] 702、The server inputs a sample face verification video into the face video classification model.

[0220] The sample face verification video is a face verification video used for model training. The collection and use of the plurality of sample face verification videos are both subject to sufficient authorization of the corresponding object of the sample face verification video. Referring to Figure 8 , the face video classification model 800 includes a manual classification sub-model 801, a deep classification sub-model 802, a fusion classification sub-model 803, and a defective frame classification sub-model 804. The manual classification sub-model 801 is used to extract manual features of a video frame and classify the video frame based on the manual features. The deep classification sub-model 802 is used to extract deep features of the video frame and classify the video frame based on the deep features. The fusion classification sub-model 803 is used to fuse the manual features and deep features of the video frame, and fuse the classification results based on the manual features and the classification results based on the deep features, to finally determine whether the video frame is a defective frame. The defective frame classification sub-model 804 is used to classify the defective frame to determine whether the defective frame is a tampered frame.

[0221] 704、The server extracts features of a plurality of video frames of the sample face verification video through the face video classification model to obtain manual features and deep features of each video frame.

[0222] The manner in which the server extracts features of the plurality of video frames through the face video classification model belongs to the same inventive concept as the manners described in steps 304 and 306, and the implementation process will not be described again.

[0223] 706、The server classifies each video frame based on the hand-crafted features of each video frame through the face video classification model to obtain a first classification result of each video frame.

[0224] The method of classifying each video frame based on the hand-crafted features of each video frame to obtain a first classification result of each video frame belongs to the same inventive concept as the method described in step 308, and the implementation process will not be described again.

[0225] 708、The server classifies each video frame based on the deep features of each video frame through the face video classification model to obtain a second classification result of each video frame, and the first classification result and the second classification result are used to represent whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects.

[0226] In one possible implementation, the server performs defective frame prediction based on the deep features of each video frame through the face video classification model to obtain a first classification value of each video frame, and the first classification value is used to represent the possibility of the corresponding video frame being a defective frame. The server performs tampering frame prediction based on the deep features of each video frame through the face video classification model to obtain a second classification value of each video frame, and the first classification value is used to represent the possibility of the corresponding video frame being a tampering frame. The server determines a second classification result of each video frame based on the first classification value and the second classification value of each video frame through the face video classification model.

[0227] The method of determining the second classification result of each video frame in the above implementation belongs to the same inventive concept as the description in step 310, and the implementation process will not be described again.

[0228] In some embodiments, the server trains the face video classification model based on second difference information between the first classification value of each video frame and the first annotation type of each video frame, and third difference information between the second classification value of each video frame and the second annotation type of each video frame. Wherein, the first annotation type is used to represent whether the corresponding video frame is a defective frame, and the second annotation type is used to represent whether the corresponding video frame is a tampering frame.

[0229] The face video classification model is trained based on the second difference information and the third difference information to make the second difference information and the third difference information as small as possible, so that the face video classification model can accurately identify defective frames and tampering frames. In the case where the face video classification model includes a deep classification sub-model 802, that is, the server trains the deep classification sub-model 802 based on the second difference information and the third difference information, and the loss function of the training is as follows formula (1).

[0230] Loss total =Loss qx-cls +Loss df-cls +Loss cls-consistency (1)

[0231] wherein, Loss total is the total loss for training the deep classification sub-model 802, Loss qx-cls is the defect frame prediction loss corresponding to the second difference information, Loss df-cls is the tampered frame prediction loss corresponding to the third difference information, Loss cls-consistency is the consistency loss of the defect frame prediction loss and the tampered frame prediction loss.

[0232] 710、The server determines at least one defect frame from the plurality of video frames based on the manual features, the deep features, the first classification result and the second classification result of each video frame through the face video classification model.

[0233] wherein, the method for determining at least one defect frame in the step 710 belongs to the same inventive concept as the description in the step 316, and the implementation process will not be repeated.

[0234] In some embodiments, the server trains the face video classification model based on fourth difference information between the at least one defect frame and at least one labeled defect frame in the plurality of video frames. The face video classification model is trained based on the fourth difference information so that the fourth difference information is as small as possible.

[0235] In the case that the face video classification model includes the fusion classification sub-model 803, that is, the server trains the fusion classification sub-model 803 based on the fourth difference information. Referring to Figure 6 , the fusion classification sub-model 803 includes the feature fusioner 601, the result fusioner 602 and the fusion classifier 603. The above process is that the server trains the feature fusioner 601, the result fusioner 602 and the fusion classifier 603 based on the fourth difference information. The loss function of the training is shown in the following formula (2).

[0236] Loss total =Loss cls +Loss cls-consis (2)

[0237] wherein, Loss total is the total loss for training the fusion classification sub-model 803, Loss cls is the classification loss corresponding to the fourth difference information, Loss cls-consisThe consistency loss is classified by the feature fusioner 601, the result fusioner 602 and the classifier 603.

[0238] 712、The server determines, by the face video classification model, a predicted type of the sample face verification video based on the at least one defect frame, the predicted type being used to represent whether the sample face verification video is a tampered video.

[0239] In a possible implementation, for any defect frame in the at least one defect frame, the server inputs the defect frame into a compression parameter generator, and generates a compression parameter of the defect frame by the compression parameter generator. The server compresses the defect frame by using the defect frame compression parameter based on a compression method corresponding to the compression parameter generator, to obtain a compressed defect frame. The server inputs the defect frame and the compressed defect frame into a classifier, and outputs, by the classifier, the predicted type of the sample face verification video.

[0240] The defect frame is compressed before being input into the classifier for classification, which can train the generalization ability of the classifier for video compression and improve the accuracy of classification.

[0241] 714、The server trains the face video classification model based on first difference information between the predicted type of the sample face verification video and a labeled type of the sample face verification video.

[0242] In a possible implementation, the server performs adversarial training on the compression parameter generator and the classifier based on the first difference information.

[0243] The adversarial training on the compression parameter generator and the classifier is to form an antagonism between the compression parameter generator and the classifier, so as to improve the abilities of the compression parameter generator and the classifier. That is, the training target of the compression parameter generator is to maximize the classification error rate of the classifier by the generated compression parameter, and the training target of the classifier is to minimize the classification error rate of itself. The first difference information can reflect the accuracy of classification of the classifier. The loss function of training is shown in the following formula (3).

[0244] Loss total =Loss adv +Loss cls +Loss consistency (3)

[0245] Loss total is the total loss of the adversarial training on the compression parameter generator and the classifier, Loss adv is the adversarial loss of the compression parameter generator and the classifier, and Loss clsFor the classification loss of the classifier, corresponding to the first difference information, Loss consistency For the consistency loss of the classifier based on the defect frames before and after compression.

[0246] All the optional technical solutions described above can be combined to form optional embodiments of the present specification, which will not be described one by one.

[0247] Through the technical solutions provided by the embodiments of the present specification, the server extracts features from multiple video frames of a sample face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and deep features of each video frame to obtain first classification results and second classification results of each video frame. At least one defect frame is determined from the multiple video frames based on the manual features, deep features, first classification results and second classification results of each video frame. The predicted type of the face verification video is determined based on the at least one defect frame. The face video classification model is trained based on the difference information between the predicted type and the labeled type, so that the face video classification model has the ability to identify tampered videos. Using the face video classification model to classify the face verification video before face recognition can improve the security of face recognition.

[0248] Figure 9 is a structural schematic diagram of a face video classification device provided by an embodiment of the present specification, referring to Figure 9 The device comprises a first feature extraction module 901, a first video frame classification module 902, a first defect frame determination module 903 and a first video classification module 904.

[0249] The first feature extraction module 901 is configured to extract features from multiple video frames of a face verification video to obtain manual features and deep features of each video frame.

[0250] The first video frame classification module 902 is configured to classify each video frame based on the manual features of each video frame to obtain first classification results of each video frame. Each video frame is classified based on the deep features of each video frame to obtain second classification results of each video frame. The first classification results and the second classification results are used to indicate whether each video frame is a defect frame. The defect frame is a video frame that may have tampering defects.

[0251] The first defect frame determination module 903 is configured to determine at least one defect frame from the multiple video frames based on the manual features, deep features, first classification results and second classification results of each video frame.

[0252] The first video classification module 904 is configured to determine whether the face verification video is a tampered video based on the at least one defect frame.

[0253] In a possible implementation, the first feature extraction module 901 is configured to input the plurality of video frames into a hand-crafted feature extractor, perform hand-crafted feature extraction on the plurality of video frames by the hand-crafted feature extractor, and obtain hand-crafted features of each of the video frames; and input the plurality of video frames into a deep feature extractor, perform deep feature extraction on the plurality of video frames by the deep feature extractor, and obtain deep features of each of the video frames.

[0254] In a possible implementation, the first feature extraction module 901 is configured to perform at least one of the following:

[0255] determine a first area ratio of a face region in each of the video frames in the corresponding video frame.

[0256] determine a pixel value variance of a pixel point in the face region in each of the video frames.

[0257] determine a second area ratio of an edge of the face region in each of the video frames in the corresponding face region.

[0258] In a possible implementation, the first video frame classification module 902 is configured to, for any video frame in the plurality of video frames, determine, in a case where at least one of the following conditions is met: the first area ratio corresponding to the video frame is less than a first area threshold, the pixel value variance corresponding to the video frame is less than a variance threshold, and the second area ratio corresponding to the video frame is less than a second area threshold, the first classification result of the video frame as a defective frame.

[0259] In a case where any one of the following conditions is met: the first area ratio corresponding to the video frame is greater than or equal to the first area threshold, the pixel value variance corresponding to the video frame is greater than or equal to the variance threshold, and the second area ratio corresponding to the video frame is greater than or equal to the second area threshold, determine the first classification result of the video frame as a non-defective frame.

[0260] In a possible implementation, the first feature extraction module 901 is configured to, for any video frame in the plurality of video frames, perform any one of convolution, full connection, and attention encoding on the video frame to obtain the deep features of the video frame.

[0261] In a possible implementation, the first video frame classification module 902 is configured to perform defective frame prediction based on the deep features of each of the video frames to obtain a first classification value of each of the video frames, where the first classification value is used to represent a possibility that the corresponding video frame is a defective frame; perform tampered frame prediction based on the deep features of each of the video frames to obtain a second classification value of each of the video frames, where the first classification value is used to represent a possibility that the corresponding video frame is a tampered frame; and determine a second classification result of each of the video frames based on the first classification value and the second classification value of each of the video frames.

[0262] In a possible implementation, the first video frame classification module 902 is configured to fuse the first classification value and the second classification value of each video frame to obtain a third classification value of each video frame. The second classification result of each video frame is determined based on a size relationship between the third classification value of each video frame and a classification value threshold.

[0263] In a possible implementation, the first defect frame determination module 903 is configured to fuse the manual feature and the deep feature of each video frame to obtain a fusion feature of each video frame. The first classification result and the second classification result of each video frame are fused to obtain a fusion classification result of each video frame. The at least one defect frame is determined from the plurality of video frames based on the fusion feature and the fusion classification result of each video frame.

[0264] In a possible implementation, the first defect frame determination module 903 is configured to splice the fusion feature and the fusion classification result of each video frame to obtain a target classification feature of each video frame. The target classification feature of each video frame is fully connected and normalized to obtain a target classification value of each video frame. The video frame with the target classification value greater than the classification value threshold in the plurality of video frames is determined as a defect frame.

[0265] In a possible implementation, the first video classification module 904 is configured to perform feature extraction on the at least one defect frame to obtain an image feature of the at least one defect frame. The type of the at least one defect frame is determined based on the image feature of the at least one defect frame. In a case where any defect frame in the at least one defect frame is a tampered frame, the face verification video is determined as a tampered video. In a case where none of the at least one defect frame is a tampered frame, the face verification video is determined as a non-tampered video.

[0266] It should be noted that the face video classification apparatus provided in the above embodiments is only taken as an example in classification of a face video, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the face video classification apparatus and the face video classification method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0267] The technical solution provided by the embodiments of the present specification extracts features from multiple video frames of a face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and deep features of each video frame to obtain first classification results and second classification results of each video frame. At least one defective frame is determined from the multiple video frames based on the manual features, deep features, first classification results, and second classification results of each video frame. Whether the face verification video is a tampered video, i.e., whether the face verification video is a video processed by deepfakes technology, is determined based on the at least one defective frame, thereby improving the security of face recognition.

[0268] Figure 10 is a structural schematic diagram of a training device of a face video classification model provided by the embodiments of the present specification, referring to Figure 10 The device comprises an input module 1001, a second feature extraction module 1002, a second video frame classification module 1003, a second defective frame determination module 1004, a second video classification module 1005, and a training module 1006.

[0269] The input module 1001 is configured to input a sample face verification video into a face video classification model.

[0270] The second feature extraction module 1002 is configured to extract features from multiple video frames of the sample face verification video by the face video classification model to obtain manual features and deep features of each video frame.

[0271] The second video frame classification module 1003 is configured to classify each video frame based on the manual features of each video frame by the face video classification model to obtain first classification results of each video frame. Each video frame is classified based on the deep features of each video frame to obtain second classification results of each video frame. The first classification results and the second classification results are used to indicate whether each video frame is a defective frame. The defective frame is a video frame that may have tampering defects.

[0272] The second defective frame determination module 1004 is configured to determine at least one defective frame from the multiple video frames based on the manual features, deep features, first classification results, and second classification results of each video frame by the face video classification model.

[0273] The second video classification module 1005 is configured to determine a predicted type of the sample face verification video based on the at least one defective frame by the face video classification model. The predicted type is used to indicate whether the sample face verification video is a tampered video.

[0274] The training module 1006 is configured to train the face video classification model based on first difference information between the predicted type of the sample face verification video and the labeled type of the sample face verification video.

[0275] In a possible implementation, the second video classification module 1005 is configured to, for any defective frame in the at least one defective frame, input the defective frame into a compression parameter generator, and generate a compression parameter of the defective frame by the compression parameter generator. The defective frame is compressed based on the compression method corresponding to the compression parameter generator to obtain a compressed defective frame. The defective frame and the compressed defective frame are input into a classifier, and the classifier outputs the predicted type of the sample face verification video.

[0276] In a possible implementation, the training module 1006 is configured to perform adversarial training on the compression parameter generator and the classifier based on the first difference information.

[0277] In a possible implementation, the second video frame classification module 1003 is configured to perform defective frame prediction based on the deep feature of each video frame to obtain a first classification value of each video frame, where the first classification value is used to indicate a possibility that the corresponding video frame is a defective frame. Tampered frame prediction is performed based on the deep feature of each video frame to obtain a second classification value of each video frame, where the second classification value is used to indicate a possibility that the corresponding video frame is a tampered frame. A second classification result of each video frame is determined based on the first classification value and the second classification value of each video frame.

[0278] In a possible implementation, the training module 1006 is further configured to train the face video classification model based on second difference information between the first classification value of each video frame and a first labeled type of each video frame, and third difference information between the second classification value of each video frame and a second labeled type of each video frame. The first labeled type is used to indicate whether the corresponding video frame is a defective frame, and the second labeled type is used to indicate whether the corresponding video frame is a tampered frame.

[0279] In a possible implementation, the training module 1006 is further configured to train the face video classification model based on fourth difference information between the at least one defective frame and at least one labeled defective frame in the plurality of video frames.

[0280] It should be noted that the training device of the face video classification model provided in the above embodiment only divides the above functions into different functional modules for example when training the face video classification model. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the face video classification device and the face video classification method provided in the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0281] Through the technical solutions provided by the embodiments of the present specification, the server extracts features from multiple video frames of a sample face verification video to obtain manual features and deep features of each video frame. Each video frame is classified based on the manual features and deep features of each video frame to obtain first classification results and second classification results of each video frame. At least one defective frame is determined from the multiple video frames based on the manual features, deep features, first classification results and second classification results of each video frame. The predicted type of the face verification video is determined based on the at least one defective frame. The face video classification model is trained based on the difference between the predicted type and the labeled type, so that the face video classification model has the ability to identify tampered videos. Using the face video classification model to classify the face verification video before face recognition can improve the security of face recognition.

[0282] The embodiments of the present specification also provide a computer storage medium, which can store a plurality of program instructions. The program instructions are suitable for being loaded and executed by a processor to implement the schemes described in the above method embodiments, which will not be repeated here.

[0283] The embodiments of the present specification also provide a computer program product, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the schemes described in the above method embodiments, which will not be repeated here.

[0284] Please refer to Figure 11 which shows a structural schematic diagram of an electronic device provided by an example embodiment of the present specification. The electronic device can be provided as a server or a terminal. The electronic device in the present specification can include one or more of the following components: a processor 1110, a memory 1120, an input device 1130, an output device 1140 and a bus 1160. The processor 1110, the memory 1120, the input device 1130 and the output device 1140 can be connected through the bus 1160.

[0285] The processor 1110 can include one or more processing cores. The processor 1110 connects various parts within the entire electronic device by various interfaces and lines, performs various functions of the electronic device 1100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1120, and calling data stored in the memory 1120. Alternatively, the processor 1110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 1110 can integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1110, but can be realized by a separate communication chip.

[0286] The memory 1120 can include a random access memory (RAM) and can also include a read-only memory (ROM). Alternatively, the memory 1120 includes a non-transitory computer-readable storage medium. The memory 1120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the methods described below, etc., and the operating system can be an Android system, an IOS system developed by Apple Inc., a system developed based on the Android system, an IOS system developed based on the IOS system, or other systems.

[0287] In order to enable the operating system to distinguish the specific application scenarios of the third-party application, it is necessary to open up the data communication between the third-party application and the operating system, so that the operating system can obtain the current scenario information of the third-party application at any time, and then perform targeted system resource adaptation based on the current scenario.

[0288] The input device 1130 is configured to receive input instruction or data, and the input device 1130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 1140 is configured to output instruction or data, and the output device 1140 includes, but is not limited to, a display device, a speaker, and the like. In an example, the input device 1130 and the output device 1140 can be combined, and the input device 1130 and the output device 1140 are a touch display screen.

[0289] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-mentioned drawings does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the drawings, or combine certain components, or different component arrangements. For example, the electronic device also includes radio frequency circuitry, an input unit, a sensor, audio circuitry, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and the like, which are not described here.

[0290] In Figure 11 In the electronic device shown in the above-mentioned drawings, the processor 1110 can be configured to invoke an application program for video classification stored in the memory 1120, and execute the method described in the above-mentioned method embodiments.

[0291] The above is a schematic scheme of an electronic device of an embodiment of the present specification. It should be noted that the technical scheme of the electronic device belongs to the same concept as the technical schemes of the face video classification method and the training method of the face video classification model described above, and the details of the technical scheme of the electronic device which are not described in detail can be referred to the description of the technical scheme of the face video classification method.

[0292] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. The storage medium of the computer program can be a magnetic disc, an optical disc, a read-only memory, or a random access memory, etc.

[0293] The above is only an optional embodiment of the present specification, and does not limit the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present specification shall be included in the protection scope of the present specification.

[0294] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the following claims. In some cases, the acts or steps recited in the claims can be performed in a different order than those in the embodiments and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A face video classification method, comprising: Feature extraction is performed on multiple video frames of the face verification video to obtain the hand-crafted features and depth features of each video frame. The hand-crafted features are unsupervised features, and the depth features are supervised features. Classification is performed based on the manual features of each video frame to obtain a first classification result for each video frame; classification is performed based on the depth features of each video frame to obtain a second classification result for each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects. The handcrafted features and depth features of each video frame are fused to obtain the fused features of each video frame; The first classification result and the second classification result of each video frame are fused to obtain the fused classification result of each video frame; Based on the fusion features and fusion classification results of each video frame, at least one defective frame is determined from the plurality of video frames; Feature extraction is performed on the at least one defective frame to obtain the image features of the at least one defective frame; Classify the at least one defective frame based on its image features to determine the type of the at least one defective frame; If any of the at least one defective frame is a tampered frame, the face verification video is determined to be a tampered video. If none of the at least one defective frame is a tampered frame, then the face verification video is determined not to be a tampered video.

2. The method according to claim 1, wherein feature extraction of multiple video frames of the face verification video to obtain the handcrafted features and depth features of each video frame includes: The multiple video frames are input into a manual feature extractor, and the manual feature extractor performs manual feature extraction on the multiple video frames to obtain the manual features of each video frame. The multiple video frames are input into a depth feature extractor, which extracts depth features from the multiple video frames to obtain the depth features of each video frame.

3. The method according to claim 2, wherein the manual feature extraction of the plurality of video frames to obtain the manual features of each video frame includes at least one of the following: Determine the first area ratio of the face region in each video frame; Determine the variance of pixel values ​​of pixels within the face region in each of the video frames; Determine the second area ratio of the edge of the face region in each of the video frames within the corresponding face region.

4. The method according to claim 3, wherein classifying based on the manual features of each video frame to obtain a first classification result for each video frame includes: For any video frame among the plurality of video frames, if at least one of the following exists: the first area proportion of the video frame is less than the first area threshold, the corresponding pixel value variance is less than the variance threshold, and the corresponding second area proportion is less than the second area threshold, the first classification result of the video frame is determined to be a defective frame. If any one of the following exists: the first area proportion corresponding to the video frame is greater than or equal to the first area threshold, the corresponding pixel value variance is greater than or equal to the variance threshold, and the corresponding second area proportion is greater than or equal to the second area threshold, the first classification result of the video frame is determined to be a non-defect frame.

5. The method according to claim 2, wherein extracting depth features from the plurality of video frames to obtain the depth features of each video frame includes: For any one of the plurality of video frames, perform any one of convolution, fully connected, and attention encoding on the video frame to obtain the depth features of the video frame.

6. The method according to claim 1, wherein classifying based on the depth features of each video frame to obtain a second classification result for each video frame includes: Defect frame prediction is performed based on the depth features of each video frame to obtain a first classification value for each video frame. The first classification value is used to indicate the probability that the corresponding video frame is a defect frame. Based on the depth features of each video frame, tampering frame prediction is performed to obtain a second classification value for each video frame. The second classification value is used to indicate the probability that the corresponding video frame is a tampered frame. Based on the first classification value and the second classification value of each video frame, the second classification result of each video frame is determined.

7. The method according to claim 6, wherein determining the second classification result of each video frame based on the first classification value and the second classification value of each video frame includes: The first and second classification values ​​of each video frame are fused to obtain the third classification value of each video frame. Based on the relationship between the third classification value and the classification value threshold of each video frame, the second classification result of each video frame is determined.

8. The method according to claim 1, wherein determining at least one defective frame from the plurality of video frames based on the fusion features and fusion classification results of each of the video frames comprises: The fusion features and fusion classification results of each video frame are concatenated to obtain the target classification features of each video frame. The target classification features of each video frame are fully connected and normalized to obtain the target classification value of each video frame. Video frames whose target classification value is greater than the classification value threshold among the multiple video frames are identified as defective frames.

9. A training method for a face video classification model, comprising: Input the sample face verification video into the face video classification model; Using the face video classification model, feature extraction is performed on multiple video frames of the sample face verification video to obtain the hand-crafted features and deep features of each video frame. The hand-crafted features are unsupervised features, and the deep features are supervised features. The face video classification model is used to classify each video frame based on the manual features of each video frame to obtain a first classification result for each video frame; and to classify each video frame based on the depth features of each video frame to obtain a second classification result for each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects. The face video classification model is used to fuse the handcrafted features and depth features of each video frame to obtain the fused features of each video frame; the first classification result and the second classification result of each video frame are fused to obtain the fused classification result of each video frame; based on the fused features and fused classification results of each video frame, at least one defective frame is determined from the plurality of video frames. Using the face video classification model, feature extraction is performed on the at least one defective frame to obtain the image features of the at least one defective frame; based on the image features of the at least one defective frame, classification is performed to determine the type of the at least one defective frame; if any of the at least one defective frame is a tampered frame, the predicted type of the sample face verification video is determined to be a tampered video; if none of the at least one defective frame is a tampered frame, the predicted type of the sample face verification video is determined to be a non-tampered video. The face video classification model is trained based on the first difference information between the predicted type of the sample face verification video and the labeled type of the sample face verification video.

10. The method according to claim 9, wherein determining the prediction type of the sample face verification video based on the at least one defective frame comprises: For any defective frame in the at least one defective frame, the defective frame is input into the compression parameter generator, and the compression parameter generator generates the compression parameters of the defective frame. Based on the compression method corresponding to the compression parameter generator, the defect frame is compressed using the defect frame compression parameters to obtain a compressed defect frame; The defective frame and the compressed defective frame are input into a classifier, which outputs the predicted type of the sample face verification video.

11. The method according to claim 10, wherein training the face video classification model based on the first difference information between the predicted type of the sample face verification video and the labeled type of the sample face verification video comprises: Based on the first difference information, the compressed parameter generator and the classifier are subjected to adversarial training.

12. The method according to claim 9, wherein classifying based on the depth features of each video frame to obtain a second classification result for each video frame includes: Defect frame prediction is performed based on the depth features of each video frame to obtain a first classification value for each video frame. The first classification value is used to indicate the probability that the corresponding video frame is a defect frame. Based on the depth features of each video frame, tampering frame prediction is performed to obtain a second classification value for each video frame. The second classification value is used to indicate the probability that the corresponding video frame is a tampered frame. Based on the first classification value and the second classification value of each video frame, the second classification result of each video frame is determined.

13. The method according to claim 12, further comprising: The face video classification model is trained based on the second difference information between the first classification value and the first annotation type of each video frame, and the third difference information between the second classification value and the second annotation type of each video frame. The first annotation type is used to indicate whether the corresponding video frame is a defective frame, and the second annotation type is used to indicate whether the corresponding video frame is a tampered frame.

14. The method according to claim 9, further comprising: The face video classification model is trained based on the fourth difference information between the at least one defective frame and at least one labeled defective frame among the plurality of video frames.

15. A face video classification device, comprising: The first feature extraction module is used to extract features from multiple video frames of the face verification video to obtain the manual features and depth features of each video frame. The first video frame classification module is used to classify each video frame based on its manual features to obtain a first classification result for each video frame; and to classify each video frame based on its depth features to obtain a second classification result for each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects. The first defect frame determination module is used to fuse the manual features and depth features of each video frame to obtain the fused features of each video frame; fuse the first classification result and the second classification result of each video frame to obtain the fused classification result of each video frame; and determine at least one defect frame from the plurality of video frames based on the fused features and fused classification results of each video frame. The first video classification module is used to extract features from the at least one defective frame to obtain the image features of the at least one defective frame; classify the at least one defective frame based on the image features to determine the type of the at least one defective frame; determine that the face verification video is a tampered video if any of the at least one defective frame is a tampered frame; and determine that the face verification video is not a tampered video if none of the at least one defective frame is a tampered frame.

16. A training device for a face video classification model, comprising: The input module is used to input sample face verification videos into the face video classification model; The second feature extraction module is used to extract features from multiple video frames of the sample face verification video using the face video classification model, and obtain the hand-crafted features and depth features of each video frame. The hand-crafted features are unsupervised features, and the depth features are supervised features. The second video frame classification module is used to classify each video frame based on the hand-crafted features of each video frame using the face video classification model to obtain a first classification result for each video frame; and to classify each video frame based on the depth features of each video frame to obtain a second classification result for each video frame. The first classification result and the second classification result are used to indicate whether each video frame is a defective frame, and the defective frame is a video frame that may have tampering defects. The second defect frame determination module is used to fuse the manual features and depth features of each video frame to obtain the fused features of each video frame; fuse the first classification result and the second classification result of each video frame to obtain the fused classification result of each video frame; and determine at least one defect frame from the plurality of video frames based on the fused features and fused classification results of each video frame. The second video classification module is used to extract features from the at least one defective frame using the face video classification model to obtain image features of the at least one defective frame; classify the at least one defective frame based on its image features to determine its type; if any of the at least one defective frame is a tampered frame, determine that the predicted type of the sample face verification video is a tampered video; if none of the at least one defective frame is a tampered frame, determine that the predicted type of the sample face verification video is a non-tampered video. The training module is used to train the face video classification model based on the first difference information between the predicted type of the sample face verification video and the labeled type of the sample face verification video.

17. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method as claimed in any one of claims 1 to 14.

18. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 14.

19. A computer program product comprising instructions that, when run on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 1 to 14.

Images