Video-based false face recognition method and device, equipment and storage medium

Abstract

The invention provides a video-based false face recognition method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the steps that on one hand, face feature extraction, time sequence information extraction and attention processing are carried out on an original face image frame, and a face time sequence weighted vector matrix can be obtained; on the other hand, frequency feature extraction and frequency domain image reconstruction are carried out on the original face image frame, and a face frequency domain image matrix can be obtained. And based on the face time sequence weighted vector matrix and the face frequency domain image matrix, target face fusion features can be obtained through fusion. The target face fusion feature comprises change information and associated information of the face in time sequence and space, and importance information between different original image frames. And finally, performing false face discrimination on the original face video based on the target face fusion feature. According to the invention, false face video recognition can be accurately carried out, and the recognition efficiency and applicability can be balanced.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a video-based false face recognition method and apparatus, device, and storage medium. Background Art

[0002] Fake face videos refer to the use of deep learning and other technologies to replace or transfer one person's facial features into another person's video, thereby generating fake video content. In the field of financial technology, there may be many potential risks of applying fake face videos. For example, identity verification of financial institutions: Financial institutions such as banks, insurance companies, securities companies, etc. need to verify the identity of customers to achieve anti-fraud and prevent theft.

[0003] There are some fake face recognition technologies, but they cannot balance recognition efficiency and applicability. For example, fake face recognition technologies that usually recognize single face images cannot be well applied to face videos. There are also some fake face recognition technologies that recognize face videos. Since they recognize face videos, the recognition process usually requires a lot of time and resources, resulting in low recognition efficiency. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a video-based false face recognition method and apparatus, equipment, and storage medium that can accurately perform false face video recognition and balance recognition efficiency and applicability.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a video-based false face recognition method, the method comprising:

[0006] Extracting an original face image frame containing a face from an original face video containing a face;

[0007] Extracting facial features from the original facial image frame to obtain a facial feature vector;

[0008] Extracting time series information from each of the facial feature vectors to obtain a facial time series information vector;

[0009] Performing attention processing on each of the face time sequence information vectors to obtain a face time sequence weighted vector matrix;

[0010] Extracting frequency features of the original face image frame to obtain frequency feature coefficients;

[0011] Perform frequency domain image reconstruction according to the frequency characteristic coefficients to obtain a frequency domain image matrix of the face;

[0012] Performing feature fusion on the face time-series weighted vector matrix and the face frequency-domain image matrix to obtain target face fusion features;

[0013] The original face video is subjected to false face discrimination based on the target face fusion feature.

[0014] In some embodiments, the performing false face discrimination on the original face video based on the target face fusion feature includes:

[0015] Extracting latent features from the target face fusion features to obtain latent discriminant features of the face;

[0016] Performing dimensionality reduction processing on the potential discriminant features of the face to obtain dimensionality reduction discriminant features of the face;

[0017] Performing linear processing on the face dimension reduction discriminant features to obtain face linear discriminant features;

[0018] The face linear discriminant features are processed nonlinearly to obtain a face category; wherein the face category indicates whether the face in the original face video is a false face or a non-false face.

[0019] In some embodiments, after performing nonlinear processing on the linear discriminant features of the face to obtain the face category, the method further includes:

[0020] If the face category indicates that the face in the original face video is a fake face, each of the face feature vectors is subjected to feature fusion with the target face fusion feature to obtain a fake face image feature;

[0021] The false face image features are classified into face forgery types using a preset face forgery type classification model to obtain a target face forgery type.

[0022] In some embodiments, performing attention processing on each of the face time sequence information vectors to obtain a face time sequence weighted vector matrix includes:

[0023] Performing self-attention processing on each of the face time series information vectors to obtain a face time series weighted vector;

[0024] Vector splicing is performed according to each of the face temporal weighted vectors to obtain the face temporal weighted vector matrix.

[0025] In some embodiments, reconstructing the frequency domain image according to the frequency characteristic coefficients to obtain a frequency domain image matrix of the face includes:

[0026] splicing at least one of the frequency characteristic coefficients in the same preset frequency band to obtain a frequency characteristic channel;

[0027] Channel merging is performed according to each of the frequency feature channels to obtain the face frequency domain image matrix.

[0028] In some embodiments, extracting time series information from each of the facial feature vectors to obtain a facial time series information vector includes:

[0029] Taking the face feature vector corresponding to the first original face image frame as a state input feature vector;

[0030] Performing time series feature fusion on the state input feature vector of the first frame and the face feature vector of the second frame through a time series recurrent neural network to obtain a time series feature fusion vector, and updating the time series feature fusion vector to the state input feature vector; wherein the first frame and the second frame correspond to different original face image frames, and the time corresponding to the first frame is earlier than the time corresponding to the second frame;

[0031] The face temporal information vector is determined according to the temporal feature fusion vector corresponding to the last original face image frame.

[0032] In some embodiments, performing time series feature fusion on the state input feature vector of the first frame and the face feature vector of the second frame through a time series recurrent neural network to obtain a time series feature fusion vector includes:

[0033] Performing feature updates on the state input feature vector of the first frame and the face feature vector of the second frame by using the time series recurrent neural network to obtain a face update feature vector;

[0034] Resetting the state input feature vector of the first frame and the face feature vector of the second frame by using the time series recurrent neural network to obtain a face reset feature vector;

[0035] Performing feature mixing on the face reset feature vector, the state input feature vector of the first frame, and the face feature vector of the second frame through the time series recurrent neural network to obtain a face candidate hidden state vector;

[0036] The face update feature vector, the state input feature vector of the first frame and the face candidate hidden state vector are subjected to feature selection fusion through the temporal recurrent neural network to obtain the temporal feature fusion vector.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a video-based false face recognition device, the device comprising:

[0038] A face image extraction module, used for extracting an original face image frame containing a face from an original face video containing a face;

[0039] A face feature extraction module, used to extract face features from the original face image frame to obtain a face feature vector;

[0040] A time series information extraction module, used to extract time series information from each of the facial feature vectors to obtain a facial time series information vector;

[0041] An attention processing module, used for performing attention processing on each of the face time sequence information vectors to obtain a face time sequence weighted vector matrix;

[0042] A frequency feature extraction module, used to extract frequency features from the original face image frame to obtain frequency feature coefficients;

[0043] A frequency domain image reconstruction module, used to perform frequency domain image reconstruction according to the frequency characteristic coefficients to obtain a frequency domain image matrix of a face;

[0044] A face feature fusion module, used for fusing the face time series weighted vector matrix and the face frequency domain image matrix to obtain a target face fusion feature;

[0045] The false face identification module is used to perform false face identification on the original face video based on the target face fusion feature.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the video-based false face recognition method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the video-based false face recognition method described in the first aspect above.

[0048] The video-based false face recognition method, device, equipment, and storage medium proposed in the present application can obtain a face time-series weighted vector matrix by extracting face features, extracting time-series information, and processing attention for the original face image frame on the one hand. On the other hand, frequency feature extraction and frequency domain image reconstruction are performed on the original face image frame to obtain a face frequency domain image matrix. Then, the face time-series weighted vector matrix and the face frequency domain image matrix are feature fused to obtain the target face fusion feature. In this way, the target face fusion feature contains the temporal and spatial change information, correlation information, and importance information between different original image frames of the face in the face video. Finally, the original face video is distinguished from false faces based on the target face fusion feature, which has higher accuracy and applicability. In the above recognition process, each original image frame in the face video is processed separately, and the processing difficulty is low, so the recognition efficiency is improved on the basis of being able to improve applicability.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a video-based false face recognition method provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step 103 in;

[0052] Figure 3 yes Figure 2 Flowchart of step 202 in FIG.

[0053] Figure 4 yes Figure 1 Flowchart of step 104 in;

[0054] Figure 5 yes Figure 1 Flowchart of step 106 in;

[0055] Figure 6 yes Figure 1 Flowchart of step 108 in FIG.

[0056] Figure 7 It is a module structure block diagram of a video-based false face recognition device provided in an embodiment of the present application;

[0057] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0064] In the field of financial technology, there are many potential risks of using fake face videos. For example, in the identity authentication scenario of financial institutions, financial institutions such as banks, insurance companies, securities companies, etc. need to authenticate customers to prevent fraud and theft. Traditional verification methods such as face recognition may be cracked by fake face videos. For another example, in the evidence collection scenario of financial supervision, financial regulatory authorities need to investigate and collect evidence of illegal and irregular behaviors in the financial market. These behaviors may involve video evidence. These video evidence may be forged or tampered with. For another example, in the information dissemination scenario of financial media, financial media need to report and analyze the dynamics and information of the financial market. These reports and analyses may contain video content, such as news, comments, interviews, etc. These video contents may be forged or tampered with.

[0065] Based on the above problems, the embodiments of the present application provide a video-based false face recognition method and device, electronic device, and storage medium. The embodiments of the present application aim to effectively recognize false face videos of different types and qualities by capturing the temporal and spatial changes and associations of faces in face videos, and being able to adaptively focus on the importance of different image frames, and have strong generalization and robustness. At the same time, it can improve recognition efficiency and effect, reduce computational complexity and parameter quantity, and save storage space and computing resources.

[0066] The video-based false face recognition method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video-based false face recognition method, but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] The embodiments of the present application provide a video-based false face recognition method and a video-based false face recognition device, an electronic device, and a computer-readable storage medium, which are specifically illustrated by the following embodiments. First, the video-based false face recognition method in the embodiments of the present application is described.

[0069] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0070] Reference Figure 1 , Figure 1 It is an optional flowchart of the video-based false face recognition method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 108.

[0071] Step 101, extracting an original face image frame containing a face from an original face video containing a face;

[0072] Step 102, extracting facial features from the original facial image frame to obtain a facial feature vector;

[0073] Step 103, extracting time series information from each face feature vector to obtain a face time series information vector;

[0074] Step 104, performing attention processing on each face time series information vector to obtain a face time series weighted vector matrix;

[0075] Step 105, extracting frequency features from the original face image frame to obtain frequency feature coefficients;

[0076] Step 106, reconstructing the frequency domain image according to the frequency characteristic coefficients to obtain a frequency domain image matrix of the face;

[0077] Step 107, performing feature fusion on the face time series weighted vector matrix and the face frequency domain image matrix to obtain the target face fusion feature;

[0078] Step 108, performing false face discrimination on the original face video based on the target face fusion features.

[0079] In the steps 101 to 108 shown in the embodiment of the present application, on the one hand, the present application performs facial feature extraction, temporal information extraction and attention processing on the original face image frame, and a face temporal weighted vector matrix can be obtained. On the other hand, the present application performs frequency feature extraction and frequency domain image reconstruction on the original face image frame, and a face frequency domain image matrix can be obtained. Then, the face temporal weighted vector matrix and the face frequency domain image matrix are feature fused to obtain the target face fusion feature. In this way, the target face fusion feature contains the temporal and spatial change information, correlation information, and importance information between different original image frames of the face in the face video. Finally, the original face video is distinguished from false faces based on the target face fusion feature, with higher accuracy and applicability. In the above recognition process, each original image frame in the face video is processed separately, and the processing difficulty is low, so the recognition efficiency is improved on the basis of being able to improve applicability.

[0080] In step 101 of some embodiments, an original face image frame containing a face is extracted from an original face video containing a face. The original face video refers to a video content containing a face. The original face video usually includes at least one image frame. Some image frames contain faces, and some image frames do not contain faces. The original face image frame refers to an image frame containing a face in the original face video.

[0081] In one embodiment, the original face video may be obtained in the following manner:

[0082] (1) Read the face video from the video database to obtain the original face video.

[0083] (2) Start the camera to capture the target face video and obtain the original face video.

[0084] (3) An online video recognition channel is deployed in the target fake face recognition application in advance, and the face video uploaded by the user is received through the online video recognition channel to obtain the original face video.

[0085] In one embodiment, step 101 may include:

[0086] Extract frames from the original face video containing the face to obtain image frames;

[0087] Perform face detection on each image frame to obtain the face detection category;

[0088] If the face detection category indicates that the image frame contains a face, the image frame is determined to be an original face image frame.

[0089] The benefit of the above embodiment is that image frames that do not contain faces can be filtered out, thereby improving the recognition efficiency of false faces.

[0090] In step 102 of some embodiments, facial features are extracted from the original facial image frame to obtain a facial feature vector. The facial feature vector is used to represent the feature information of the face contained in the original facial image frame. The facial features can be extracted from the original facial image frame using a facial feature extraction model to obtain a facial feature vector. The facial feature extraction model is a neural network model, such as a residual neural network model (ResNet).

[0091] In one example, for each original face image frame, the MTCNN algorithm is first used to detect all face regions in the original face image frame, and alignment, cropping and scaling operations are performed to make the size and position of each face region consistent; then the ResNet-50 model is used to extract features from each face region and obtain a 2048-dimensional feature vector. Multi-task convolutional neural network (MTCNN). MTCNN can simultaneously complete tasks such as face detection, face alignment and face calibration with high accuracy and speed.

[0092] In step 103 of some embodiments, time sequence information is extracted from each face feature vector to obtain a face time sequence information vector. The face time sequence information vector can represent the time sequence information of the original face image frame corresponding to the face feature vector in the original face video.

[0093] In one embodiment, referring to Figure 2 , step 103 may include:

[0094] Step 201, taking the face feature vector corresponding to the first original face image frame as the state input feature vector;

[0095] Step 202, performing time series feature fusion on the state input feature vector of the first frame and the face feature vector of the second frame through a time series recurrent neural network to obtain a time series feature fusion vector, and updating the time series feature fusion vector to the state input feature vector; wherein the first frame and the second frame correspond to different original face image frames, and the time corresponding to the first frame is earlier than the time corresponding to the second frame;

[0096] Step 203, determining a face temporal information vector according to the temporal feature fusion vector corresponding to the last original face image frame.

[0097] The benefit of the above-mentioned embodiment of step 201 to step 203 is that the temporal correlation between the original facial image frames can be captured.

[0098] A temporal recurrent neural network is an artificial neural network architecture used to process sequence data, which can maintain and utilize the temporal dependencies in the input data sequence. For example, RNN, LSTM, GRU, etc. RNN stands for Recurrent Neura lNetwork. LSTM stands for Long Short-Term Memory. GRU stands for Gated Recurrent Unit.

[0099] The state input feature vector represents the vector of the current state of the temporal recurrent neural network, which is used to transfer information and memory. The face feature vector represents the vector of the current frame of the temporal recurrent neural network, which is used to transfer additional information and guide the behavior of the network. The temporal recurrent neural network performs temporal feature fusion based on the state input feature vector of the first frame and the face feature vector of the second frame to obtain a temporal feature fusion vector of the second frame, and the temporal feature fusion vector is updated to the state input feature vector of the second frame. The temporal recurrent neural network can perform temporal feature fusion based on the state input feature vector of the second frame and the face feature vector of the third frame to obtain a temporal feature fusion vector of the third frame. The time corresponding to the second frame is earlier than the time corresponding to the third frame. The first frame, the second frame and the third frame are consecutive original face image frames.

[0100] In one embodiment, referring to Figure 3 , step 202 may include:

[0101] Step 301, updating the state input feature vector of the first frame and the face feature vector of the second frame by using a time series recurrent neural network to obtain a face update feature vector;

[0102] Step 302, resetting the state input feature vector of the first frame and the face feature vector of the second frame by using a time series recurrent neural network to obtain a face resetting feature vector;

[0103] Step 303, performing feature mixing on the face reset feature vector, the state input feature vector of the first frame, and the face feature vector of the second frame through a temporal recurrent neural network to obtain a face candidate hidden state vector;

[0104] Step 304, performing feature selection fusion on the face update feature vector, the state input feature vector of the first frame, and the face candidate hidden state vector through a temporal recurrent neural network to obtain a temporal feature fusion vector.

[0105] In step 301, the temporal recurrent neural network includes an update gating subunit. The state input feature vector of the first frame and the face feature vector of the second frame can be updated by the update gating subunit to obtain a face update feature vector. The update gating subunit determines the extent to which the state input feature vector of the previous moment is brought into the current state. In one embodiment, the state input feature vector of the first frame and the face feature vector of the second frame are vector-concatenated to obtain a first concatenated vector; a preset first weight matrix is ​​multiplied with the first concatenated vector to obtain a first multiplied vector; and the first multiplied vector is nonlinearly processed using a preset first activation function to obtain a face update feature vector. The first activation function can be a sigmoid function.

[0106] In step 302, the temporal recurrent neural network includes a reset gating subunit. The state input feature vector of the first frame and the face feature vector of the second frame are reset by resetting the gating subunit to obtain a face reset feature vector. The reset gating subunit determines the degree of influence of the state input feature vector of the previous moment on the current face feature vector. In one embodiment, the state input feature vector of the first frame and the face feature vector of the second frame are vector-concatenated to obtain a first concatenated vector; the preset second weight matrix is ​​multiplied with the first concatenated vector to obtain a second multiplied vector; the second multiplied vector is nonlinearly processed using a preset first activation function to obtain a face reset feature vector. The first weight matrix is ​​different from the second weight matrix.

[0107] In step 303, the temporal recurrent neural network includes a feature mixing subunit. The feature mixing subunit performs feature mixing on the face reset feature vector, the state input feature vector of the first frame, and the face feature vector of the second frame to obtain a face candidate hidden state vector. In one embodiment, the face reset feature vector is multiplied with the state input feature vector of the first frame to obtain a third multiplication vector; the third multiplication vector is spliced ​​with the face feature vector of the second frame to obtain a second splicing vector; the preset third weight matrix is ​​multiplied with the second splicing vector to obtain a fourth multiplication vector; the fourth multiplication vector is nonlinearly processed using a preset second activation function to obtain a face candidate hidden state vector. The second activation function may be a tanh function.

[0108] In step 304, the temporal recurrent neural network includes a feature selection subunit. The feature selection subunit performs feature selection fusion on the face update feature vector, the state input feature vector of the first frame, and the face candidate hidden state vector to obtain a temporal feature fusion vector. In one embodiment, the face update feature vector is subtracted from 1 to obtain a first weight vector; the first weight vector is multiplied by the state input feature vector of the first frame to obtain a fifth multiplication vector; the face update feature vector is multiplied by the face candidate hidden state vector to obtain a sixth multiplication vector; the fifth multiplication vector is added to the sixth multiplication vector to obtain a temporal feature fusion vector.

[0109] The benefit of the above-mentioned embodiment of step 301 to step 304 is that a temporal feature fusion vector can be obtained by using a temporal recurrent network with high accuracy.

[0110] In one embodiment, step 203 may include: determining the temporal feature fusion vector corresponding to the last original face image frame as the face temporal information vector, or standardizing or normalizing the temporal feature fusion vector corresponding to the last original face image frame to obtain the face temporal information vector.

[0111] In step 104 of some embodiments, attention processing is performed on each face time series information vector to obtain a face time series weighted vector matrix. Attention processing is a feature extraction method based on the attention mechanism. The attention mechanism is a neural network mechanism that imitates human attention and is used to improve the efficiency and effectiveness of the model when processing a large amount of information. It dynamically focuses on the important parts of the input data, thereby enhancing the memory and processing capabilities of key information.

[0112] In one embodiment, referring to Figure 4 , step 104 may include:

[0113] Step 401, performing self-attention processing on each face time series information vector to obtain a face time series weighted vector;

[0114] Step 402: perform vector concatenation based on each face temporal weighted vector to obtain a face temporal weighted vector matrix.

[0115] Specifically, self-attention processing is a feature extraction method based on the self-attention mechanism. The self-attention mechanism means that each element in the input sequence can interact with other elements to calculate the relationship between features. The face time series weighted vector matrix is ​​a two-dimensional matrix.

[0116] The benefit of the above-mentioned embodiment of step 401 to step 402 is that self-attention processing is introduced to perform feature extraction, thereby improving the ability to capture temporal information.

[0117] In step 105 of some embodiments, frequency feature extraction is performed on the original face image frame to obtain frequency feature coefficients. Frequency feature extraction is the process of converting the original face image frame from the time domain to the frequency domain. Frequency feature extraction can be performed in the following ways: (1) Discrete Fourier Transform (DFT) is used to convert discrete signals from the time domain to the frequency domain to help identify the frequency components in the signal. (2) Fourier Transform: used to analyze periodic signals and display the amplitude and phase of the signal at different frequencies. The frequency feature coefficients may include DFT coefficients, frequency amplitude, and frequency phase.

[0118] In step 106 of some embodiments, frequency domain image reconstruction is performed according to the frequency characteristic coefficients to obtain a frequency domain image matrix of the face. Frequency domain image reconstruction is the process of inversely transforming the frequency domain information of the image back to the time domain.

[0119] In one embodiment, referring to Figure 5 , step 106 may include:

[0120] Step 501, splicing at least one frequency characteristic coefficient in the same preset frequency band to obtain a frequency characteristic channel;

[0121] Step 502: Merge channels according to the frequency feature channels to obtain a frequency domain image matrix of the face.

[0122] Specifically, there are multiple different preset frequency bands. The preset frequency band specifically refers to a frequency interval. For example, the face area is extracted from the original face image frame, and the size of the face area is corrected to 8X8. 2DDCT is performed on each 8X8 face area to obtain the DCT coefficient, which is similar to the jpeg compression process. Then, according to the position of the face area in the original face image frame, the DCT coefficients of the same preset frequency band in all face areas are spliced ​​into one channel to obtain a frequency feature channel. Finally, all frequency feature channels are merged together to obtain a face frequency domain image matrix.

[0123] The benefit of the embodiment of steps 501 to 502 above is that the face frequency domain image matrix obtained after processing retains both the information of each frequency band and the structural information of the original face image frame, which means that features can be directly extracted from the face frequency domain image matrix using an existing neural network.

[0124] In step 107 of some embodiments, the face temporal weighted vector matrix and the face frequency domain image matrix are feature fused to obtain the target face fusion feature. Specifically, the face temporal weighted vector matrix and the face frequency domain image matrix can be matrix multiplied to obtain the target face fusion feature.

[0125] In step 108 of some embodiments, false face discrimination is performed on the original face video based on the target face fusion feature. False face discrimination refers to discriminating whether the face in the original face video corresponding to the target face fusion feature is a false face or a non-false face.

[0126] In one embodiment, referring to Figure 6 , step 108 may include:

[0127] Step 601, extracting latent features from the target face fusion features to obtain latent discriminant features of the face;

[0128] Step 602, performing dimensionality reduction processing on the potential discriminant features of the face to obtain the reduced dimensionality discriminant features of the face;

[0129] Step 603, performing linear processing on the face dimension reduction discriminant features to obtain face linear discriminant features;

[0130] Step 604: Perform nonlinear processing on the linear discriminant features of the face to obtain the face category.

[0131] In step 601, the target face fusion feature can be extracted by the convolution layer to obtain the potential discriminant feature of the face. The convolution layer can specifically include a first convolution sublayer, a second convolution sublayer and a third convolution sublayer. The convolution kernel size of the first convolution sublayer, the convolution kernel size of the second convolution sublayer and the convolution kernel size of the third convolution sublayer decrease in sequence. In this way, richer feature information can be extracted.

[0132] In step 602, the potential discriminant features of the face can be reduced in dimension by a maximum pooling layer to obtain the reduced-dimensional discriminant features of the face. The maximum pooling layer may include a first maximum pooling sublayer and a second maximum pooling sublayer. The partition size of the first maximum pooling sublayer is greater than or equal to the partition size of the second maximum pooling sublayer. In this way, not only can the dimension of the feature be reduced, but also important feature information can be retained.

[0133] In step 603, the face dimension reduction discriminant features may be linearly processed by a linear layer to obtain the face linear discriminant features. The linear layer may include a first linear sublayer and a second linear sublayer. In this way, the accuracy of the linear processing is ensured.

[0134] In step 604, the linear discriminant features of the face may be processed nonlinearly by a nonlinear layer to obtain a face category. The face category indicates whether the face in the original face video is a false face or a non-false face. The nonlinear layer may be composed of a softmax function.

[0135] The benefit of the above-mentioned embodiment of step 601 to step 604 is that it has a high accuracy while being able to realize false face discrimination.

[0136] In one embodiment, after step 604, the video-based false face recognition method provided by this embodiment may further include:

[0137] If the face category indicates that the face in the original face video is a fake face, each face feature vector is fused with the target face fusion feature to obtain the fake face image feature;

[0138] The face forgery type classification model is used to classify the fake face image features into face forgery types to obtain the target face forgery type.

[0139] If the face category indicates that the face in the original face video is a fake face, it means that the original face video is a video content with fake faces, but it is not known which faces in the original face image frames are fake faces, nor how the fake faces are forged. Since the target fused face feature is obtained based on the feature fusion of each original face image frame, the feature information representation capability of a single original face image frame is relatively weak, so this embodiment further performs feature fusion on the face feature vector and the target face fusion feature to obtain the fake face image feature. Then a face forgery type classification model is provided to classify the face forgery type of the fake face image feature to obtain the target face forgery type. The target face forgery type may include compression, cropping, replacement, mixing, etc. Compression refers to compressing a real face to obtain a new face. Cropping specifically refers to cropping a real face to obtain a new face. Replacement specifically refers to replacing the facial features of one face with the facial features of another face to obtain a new face. Mixing specifically refers to mixing the facial features of one face with the facial features of another face to obtain a new face. The face forgery type classification model is a neural network model, such as a convolutional neural network.

[0140] In one embodiment, the above-mentioned face forgery type classification model is trained in the following manner: obtaining sample false face images and sample face forgery types, the sample face forgery types indicating the forgery type corresponding to the sample false face images; adjusting the parameters of the initial forgery type classification model according to the sample false face images and the sample face forgery types to obtain the face forgery type classification model.

[0141] In summary of the above embodiments, the present application provides a technology for extracting the spatiotemporal features of faces in videos using a temporal recurrent neural network combined with a self-attention mechanism, and performing false face discrimination. The present application can make full use of the advantages of a temporal recurrent neural network and a self-attention mechanism, that is, it can capture the temporal and spatial changes and associations of faces in videos, and can adaptively focus on the importance of different frames. In this way, the recognition accuracy and interpretability can be improved, and the generalization ability and robustness for different forgery types and noise interference can be enhanced. The present application can effectively identify forged face videos of different types and qualities, including high quality, low quality, compression, cropping, etc., with strong generalization ability and robustness. At the same time, the recognition efficiency and effect can be improved, the computational complexity and parameter amount can be reduced, and storage space and computing resources can be saved.

[0142] See also Figure 7 The embodiment of the present application also provides a video-based false face recognition device, which can implement the above-mentioned video-based false face recognition method. Figure 7The module structure block diagram of the video-based false face recognition device provided in an embodiment of the present application includes: a face image extraction module 701, a face feature extraction module 702, a timing information extraction module 703, an attention processing module 704, a frequency feature extraction module 705, a frequency domain image reconstruction module 706, a face feature fusion module 707 and a false face discrimination module 708. Among them, the face image extraction module 701 is used to extract the original face image frame containing the face from the original face video containing the face; the face feature extraction module 702 is used to extract the face features of the original face image frame to obtain the face feature vector; the timing information extraction module 703 is used to extract the timing information of each face feature vector to obtain the face timing information vector; the attention processing module 704 is used to perform attention processing on each face timing information vector to obtain the face timing weighted vector matrix; the frequency feature extraction module 705 is used to extract the frequency features of the original face image frame to obtain the frequency feature coefficient; the frequency domain image reconstruction module 706 is used to reconstruct the frequency domain image according to the frequency feature coefficient to obtain the face frequency domain image matrix; the face feature fusion module 707 is used to fuse the face timing weighted vector matrix and the face frequency domain image matrix to obtain the target face fusion feature; the false face discrimination module 708 is used to discriminate the false face of the original face video based on the target face fusion feature.

[0143] In one embodiment, the fake face discrimination module 708 is specifically used to: extract latent features from the target face fusion features to obtain latent discriminant features of the face; perform dimensionality reduction processing on the latent discriminant features of the face to obtain reduced dimensionality discriminant features of the face; perform linear processing on the reduced dimensionality discriminant features of the face to obtain linear discriminant features of the face; perform nonlinear processing on the linear discriminant features of the face to obtain a face category; wherein the face category indicates whether the face in the original face video is a fake face or a non-false face.

[0144] In one embodiment, after performing nonlinear processing on the linear discriminant features of the face to obtain the face category, the video-based false face recognition device also includes: a forged type classification module, which is used to: if the face category indicates that the face in the original face video is a false face, then perform feature fusion on each face feature vector and the target face fusion feature to obtain a false face image feature; and classify the false face image features into a face forged type using a preset face forged type classification model to obtain a target face forged type.

[0145] It should be noted that the specific implementation of the video-based false face recognition device is basically the same as the specific implementation of the above-mentioned video-based false face recognition method, and will not be repeated here.

[0146] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video-based false face recognition method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0147] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0148] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0149] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the video-based false face recognition method of the embodiment of this application;

[0150] Input / output interface 803, used to implement information input and output;

[0151] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0152] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0153] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0154] An embodiment of the present application also provides a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned video-based false face recognition method.

[0155] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0157] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0158] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0160] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0166] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video-based false face recognition method, characterized in that: The method comprises: Extracting an original face image frame containing a face from an original face video containing a face; Extracting facial features from the original facial image frame to obtain a facial feature vector; Extracting time series information from each of the facial feature vectors to obtain a facial time series information vector; Performing attention processing on each of the face time sequence information vectors to obtain a face time sequence weighted vector matrix; Extracting frequency features of the original face image frame to obtain frequency feature coefficients; Perform frequency domain image reconstruction according to the frequency characteristic coefficients to obtain a frequency domain image matrix of the face; Performing feature fusion on the face time-series weighted vector matrix and the face frequency-domain image matrix to obtain target face fusion features; The original face video is subjected to false face discrimination based on the target face fusion feature.

2. The method according to claim 1, characterized in that The performing false face discrimination on the original face video based on the target face fusion feature includes: Extracting latent features from the target face fusion features to obtain latent discriminant features of the face; Performing dimensionality reduction processing on the potential discriminant features of the face to obtain dimensionality reduction discriminant features of the face; Performing linear processing on the face dimension reduction discriminant features to obtain face linear discriminant features; The face linear discriminant features are processed nonlinearly to obtain a face category; wherein the face category indicates whether the face in the original face video is a false face or a non-false face.

3. The method according to claim 2, characterized in that After performing nonlinear processing on the linear discriminant features of the face to obtain the face category, the method further includes: If the face category indicates that the face in the original face video is a fake face, each of the face feature vectors is subjected to feature fusion with the target face fusion feature to obtain a fake face image feature; The false face image features are classified into face forgery types using a preset face forgery type classification model to obtain a target face forgery type.

4. The method according to any one of claims 1 to 3, characterized in that: The attention processing is performed on each of the face time series information vectors to obtain a face time series weighted vector matrix, including: Performing self-attention processing on each of the face time series information vectors to obtain a face time series weighted vector; Vector splicing is performed according to each of the face temporal weighted vectors to obtain the face temporal weighted vector matrix.

5. The method according to any one of claims 1 to 3, characterized in that: The frequency domain image reconstruction is performed according to the frequency characteristic coefficients to obtain a frequency domain image matrix of a face, including: splicing at least one of the frequency characteristic coefficients in the same preset frequency band to obtain a frequency characteristic channel; Channel merging is performed according to each of the frequency feature channels to obtain the face frequency domain image matrix.

6. The method according to any one of claims 1 to 3, characterized in that: The step of extracting time series information from each of the face feature vectors to obtain a face time series information vector comprises: Taking the face feature vector corresponding to the first original face image frame as a state input feature vector; Performing time series feature fusion on the state input feature vector of the first frame and the face feature vector of the second frame through a time series recurrent neural network to obtain a time series feature fusion vector, and updating the time series feature fusion vector to the state input feature vector; wherein the first frame and the second frame correspond to different original face image frames, and the time corresponding to the first frame is earlier than the time corresponding to the second frame; The face temporal information vector is determined according to the temporal feature fusion vector corresponding to the last original face image frame.

7. The method according to claim 6, characterized in that The step of performing time series feature fusion on the state input feature vector of the first frame and the face feature vector of the second frame through a time series recurrent neural network to obtain a time series feature fusion vector includes: Performing feature updates on the state input feature vector of the first frame and the face feature vector of the second frame by using the time series recurrent neural network to obtain a face update feature vector; Resetting the state input feature vector of the first frame and the face feature vector of the second frame by using the time series recurrent neural network to obtain a face reset feature vector; Performing feature mixing on the face reset feature vector, the state input feature vector of the first frame, and the face feature vector of the second frame through the time series recurrent neural network to obtain a face candidate hidden state vector; The face update feature vector, the state input feature vector of the first frame and the face candidate hidden state vector are subjected to feature selection fusion through the temporal recurrent neural network to obtain the temporal feature fusion vector.

8. A video-based false face recognition device, characterized in that: The device comprises: A face image extraction module, used for extracting an original face image frame containing a face from an original face video containing a face; A face feature extraction module, used to extract face features from the original face image frame to obtain a face feature vector; A time series information extraction module, used to extract time series information from each of the facial feature vectors to obtain a facial time series information vector; An attention processing module, used for performing attention processing on each of the face time sequence information vectors to obtain a face time sequence weighted vector matrix; A frequency feature extraction module, used to extract frequency features from the original face image frame to obtain frequency feature coefficients; A frequency domain image reconstruction module, used to perform frequency domain image reconstruction according to the frequency characteristic coefficients to obtain a frequency domain image matrix of a face; A face feature fusion module, used for fusing the face time series weighted vector matrix and the face frequency domain image matrix to obtain a target face fusion feature; The false face identification module is used to perform false face identification on the original face video based on the target face fusion feature.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images