Face recognition model training method, face recognition method and related equipment

Abstract

The invention provides a training method of a face recognition model, a face recognition method and related equipment, and is used for solving the problem that the recognition accuracy of the face recognition model obtained through training is relatively low. The method at least comprises the following steps: respectively detecting a face region of a sample character in each video frame contained in a selected sample video; extracting human face change features of the sample figure from each obtained human face region; the face change feature is used for describing the face change condition of the sample character among the video frames from the angle of biological signal feature, the angle of face visual feature and the angle of associated feature; and predicting the synthesis degree of the sample character based on the obtained face change characteristics, and performing parameter adjustment on the face recognition model based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video. And in combination with biological signal training of the sample character, the recognition accuracy of the target face recognition model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a face recognition model, a face recognition method, and related equipment. Background Technology

[0002] With the continuous development of technology, deepfake technology has become increasingly sophisticated, capable of synthesizing or altering multimedia content to generate increasingly realistic fake videos. When applied to fields such as film special effects production, virtual reality, or augmented reality, it can achieve stunning visual effects. However, in some areas, fake videos can be used to trigger abnormal operations, resulting in serious adverse effects.

[0003] Therefore, more and more devices can use facial recognition models to perform facial recognition on facial regions in videos to identify fake videos containing fake faces, thereby identifying potential false information.

[0004] Under relevant technologies, the method for training a face recognition model is usually as follows: during each iteration of training based on a sample image set, the face recognition model extracts the image quality features of the sample images, and adjusts the model parameters of the face recognition model based on the difference between the obtained image quality features and the preset reference quality features of each forged image, so as to achieve the purpose of training the face recognition model and obtain the trained target face recognition model.

[0005] However, this target face recognition model trained based on image quality features can only identify low-quality forged images from the perspective of image quality. As technology advances, the image quality of forged images is getting higher and higher. It is impossible to accurately identify high-quality forged images based solely on image quality. It is easy to identify forged images as real images, which can lead to serious adverse consequences.

[0006] It is evident that the methods used to train face recognition models under these technologies result in face recognition models with low accuracy. Summary of the Invention

[0007] This application provides a training method for a face recognition model, a face recognition method, and related equipment to solve the problem of low recognition accuracy of the trained face recognition model.

[0008] Firstly, a method for training a face recognition model is provided, including:

[0009] Obtain a sample video set; each sample video is associated with a synthetic label indicating whether it contains a synthetic face;

[0010] Based on the sample video set, the face recognition model to be trained undergoes multiple rounds of iterative training, outputting the trained target face recognition model; wherein each round of iterative training includes:

[0011] The facial regions of the sample individuals are detected in each video frame of the selected sample video.

[0012] From each obtained face region, facial change features of the sample person are extracted; the facial change features are used to describe the facial changes of the sample person between each video frame from multiple feature perspectives; the multiple feature perspectives include: biosignal feature perspective, facial visual feature perspective, and correlation feature perspective; the facial changes described by the correlation feature perspective represent the correlation changes between the biosignals and facial vision of the sample person;

[0013] The degree of synthesis of the sample person is predicted based on the obtained facial change features, and the facial recognition model is tuned based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

[0014] Secondly, a facial recognition method is provided, including:

[0015] Obtain the video to be recognized;

[0016] The facial region of the target person is detected in each video frame contained in the video to be identified;

[0017] Using a trained target face recognition model, facial change features of the target person are extracted from each obtained face region, and the obtained facial change features are identified to obtain a synthetic result of the target person; wherein, the synthetic result represents whether the target person is a synthetic face; the target face recognition model is trained using the method described in the first aspect.

[0018] Thirdly, a training device for a face recognition device is provided, comprising:

[0019] Acquisition module: used to acquire sample video sets; each sample video is associated with a synthetic label indicating whether it contains a synthetic face;

[0020] Processing module: used to perform multiple rounds of iterative training on the face recognition model to be trained based on the sample video set, and output the trained target face recognition model; wherein, each round of iterative training includes:

[0021] The processing module is specifically used to: detect the face region of the sample person in each video frame contained in the selected sample video;

[0022] The processing module is specifically used to: extract facial change features of the sample person from each obtained face region; the facial change features are used to describe the facial changes of the sample person between each video frame from multiple feature perspectives; the multiple feature perspectives include: biosignal feature perspective, facial visual feature perspective, and correlation feature perspective; the facial changes described by the correlation feature perspective represent the correlation changes between the biosignals and facial vision of the sample person;

[0023] The processing module is specifically used to: predict the degree of synthesis of the sample person based on the obtained facial change features, and adjust the parameters of the face recognition model based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

[0024] Optionally, the processing module is specifically used for:

[0025] According to the preset number of frames, obtain multiple consecutive video frames from the selected sample video;

[0026] Each obtained video frame contains an image region containing a face;

[0027] Based on the obtained image regions, the face regions containing the faces that belong to the sample person are determined.

[0028] Optionally, the processing module is specifically used for:

[0029] For each obtained image region, the following operations are performed: key point detection is performed on the image region to obtain a facial key point dataset; wherein, the facial key point dataset is used to describe the facial features and facial pose of the face contained in the image region;

[0030] Based on the data similarity between the facial key point datasets of each image region, multiple image regions with a data similarity of not less than a similarity threshold are selected from each image region.

[0031] Based on the multiple image regions, the facial regions containing the included faces that belong to the sample person are determined.

[0032] Optionally, the processing module is specifically used for:

[0033] Select one image region from the plurality of image regions as a reference region;

[0034] Based on the data differences between the facial key point datasets of every two image regions in the plurality of image regions, an image mapping strategy between the plurality of image regions and the reference region is determined respectively;

[0035] Based on the obtained image mapping strategies, image mapping transformations are performed on the corresponding image regions to obtain the corresponding face regions.

[0036] Optionally, the processing module is specifically used for:

[0037] From each obtained face region, the biosignal features of the sample person are extracted; wherein, the biosignal features represent the correlation between changes in facial blood flow and changes in facial light intensity of the sample person;

[0038] From each face region, extract the visual features of the face reflected by the pixel values ​​contained in the corresponding face region;

[0039] Based on the obtained facial visual features and the biosignal features, at least one round of feature fusion is performed to determine the facial change features of the sample person.

[0040] Optionally, the processing module is specifically used for:

[0041] For each obtained face region, the following operations are performed: based on multiple preset multi-channel convolution kernels, multiple rounds of convolution operations are performed on the face region to obtain initial local features; wherein, the kernel size of the multi-channel convolution kernel is smaller than the region size of the face region;

[0042] Based on the obtained initial local features, the biosignal features of the sample individuals are determined.

[0043] Optionally, the processing module is specifically used for:

[0044] According to the preset tubular size, each initial local feature is divided into tubular features to obtain multiple tubular features; wherein, each tubular feature contains multiple local sub-features, and each local sub-feature contained in the same tubular feature comes from different initial local features, and each local sub-feature has the same sub-feature position in the corresponding initial local feature.

[0045] Semantic features are extracted from multiple tubular features to obtain the corresponding semantic features;

[0046] Based on the obtained semantic features, the biosignal features of the sample individuals are determined.

[0047] Optionally, the initial local features include multiple single-channel sub-features corresponding to each color channel;

[0048] The processing module is specifically used for:

[0049] For each obtained initial local feature, the following operations are performed: based on the single-channel convolution kernels corresponding to each of the preset multiple color channels, convolution operations are performed on the corresponding single-channel sub-features in the initial local features, and the obtained convolution results are fused to obtain comprehensive local features; wherein, the convolution kernel size of the single-channel convolution kernel is smaller than the feature size of the initial local features;

[0050] The biosignal characteristics of the sample individuals are determined based on the obtained comprehensive local features.

[0051] Optionally, the processing module is specifically used for:

[0052] For each round of feature fusion, perform the following operations:

[0053] Based on the element similarity between the feature elements contained in the intermediate visual features, the element similarity between the feature elements contained in the intermediate signal features, and the feature similarity between the intermediate visual features and the intermediate signal features, feature fusion is performed on the intermediate visual features and the intermediate signal features to determine multi-head attention features; wherein, in the first round of feature fusion, the intermediate visual features include the facial visual features, and the intermediate signal features are the biosignal features; in other rounds of feature fusion, the intermediate visual features are the facial visual change features obtained after the previous round of feature fusion, and the intermediate signal features are the biosignal change features obtained after the previous round of feature fusion;

[0054] Biometric identification is performed on the multi-head attention features to obtain biosignal change features;

[0055] Visual feature recognition is performed on the multi-head attention features to obtain facial visual change features;

[0056] The facial change features are determined based on the biosignal change features and facial visual change features obtained from the last round of feature fusion in the at least one round of feature fusion.

[0057] Optionally, the processing module is specifically used for:

[0058] Obtain the number of training iterations in the multi-round training;

[0059] When the number of training iterations is determined to be within a first range, the first model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video; wherein, the first model parameters include: model parameters used when performing visual feature recognition using the face recognition model;

[0060] When the number of training iterations is determined to be within a second range, the second model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video; wherein, the second model parameters include: model parameters used when performing biometric recognition using the face recognition model; the minimum value within the first range is less than the minimum value within the second range.

[0061] Optionally, when the sample video contains a non-synthetic human face, the sample video is also associated with a biosignal sample of the non-synthetic human face;

[0062] The processing module is specifically used for:

[0063] Predict the biosignals of the sample individuals based on the obtained biosignal features;

[0064] The face recognition model is tuned based on the difference between the obtained degree of synthesis and the synthetic label associated with the sample video, as well as the difference between the biosignal and the biosignal sample.

[0065] Fourthly, a facial recognition device is provided, comprising:

[0066] Acquisition module: Used to acquire the video to be recognized;

[0067] Processing module: used to detect the face region of the target person in each video frame contained in the video to be identified;

[0068] The processing module is further configured to: use a trained target face recognition model to extract the facial change features of the target person from each obtained face region, and identify the obtained facial change features to obtain a synthetic result of the target person; wherein the synthetic result indicates whether the target person is a synthetic face; the target face recognition model is trained using the method described in the first aspect.

[0069] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described in the first aspect or the second aspect.

[0070] Sixthly, a computer device is provided, comprising:

[0071] Memory, used to store program instructions;

[0072] A processor is configured to invoke program instructions stored in the memory and execute the method as described in the first aspect or the second aspect according to the obtained program instructions.

[0073] A seventh aspect provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in the first aspect or the second aspect.

[0074] In this embodiment, the face region of the sample person is detected in the sample video, thereby obtaining multiple face regions of the sample person in the sample video. Through the multiple face regions of the sample person, the face recognition model can analyze the face changes of the sample person in the sample video, thereby improving the accuracy of the face recognition model in identifying whether a face is a synthetic face.

[0075] Furthermore, the facial change features of the sample person extracted from each obtained face region represent changes in biosignals, changes in facial vision, and changes in the correlation between biosignals and facial vision. Thus, by combining the biosignals of the sample person, it is possible to identify whether the face of the sample person in the sample video is a synthetic face, rather than simply identifying synthetic faces based on image quality or a single image pixel value. This can improve the recognition accuracy of the trained target face recognition model.

[0076] Furthermore, facial variation features that can represent multiple modalities can be extracted from each facial region alone. This eliminates the need to first input data from multiple modalities and then set up feature extraction modules for each modality to extract features from the corresponding input data. This avoids the use of a dual-tower feature extraction process, reduces the number of model parameters, avoids the problem of inconsistent feature spaces between multimodal feature extraction modules, and improves the efficiency and accuracy of model training. Attached Figure Description

[0077] Figure 1A This is a schematic diagram illustrating an application field of the training method for the face recognition model provided in the embodiments of this application;

[0078] Figure 1B This is one application scenario of the training method for the face recognition model provided in the embodiments of this application;

[0079] Figure 2 A schematic flowchart illustrating a method for training a face recognition model provided in an embodiment of this application;

[0080] Figure 3A A schematic diagram illustrating the principle of a face recognition model training method provided in this application embodiment;

[0081] Figure 3B A schematic diagram of the principle of the face recognition model training method provided in the embodiments of this application. Figure 2 ;

[0082] Figure 3C A schematic diagram three illustrating the principle of a face recognition model training method provided in this application embodiment;

[0083] Figure 4A A schematic diagram four illustrating the principle of a face recognition model training method provided in this application embodiment;

[0084] Figure 4B A schematic diagram five illustrating the principle of a face recognition model training method provided in an embodiment of this application;

[0085] Figure 4C A schematic diagram of the principle of the face recognition model training method provided in the embodiments of this application. Figure 6 ;

[0086] Figure 4D A schematic diagram of the principle of the face recognition model training method provided in the embodiments of this application. Figure 7 ;

[0087] Figure 5A Eight is a schematic diagram illustrating the principle of a face recognition model training method provided in this application embodiment;

[0088] Figure 5B A schematic diagram of the principle of the face recognition model training method provided in the embodiments of this application. Figure 9 ;

[0089] Figure 6 A schematic flowchart of a face recognition method provided in an embodiment of this application;

[0090] Figure 7 A schematic diagram ten illustrating the principle of a face recognition model training method provided in an embodiment of this application;

[0091] Figure 8A A schematic diagram of a training device for a face recognition model provided in an embodiment of this application;

[0092] Figure 8B A schematic diagram of the structure of a face recognition device provided in an embodiment of this application;

[0093] Figure 9 This is another structural schematic diagram of a training device for a face recognition model or a face recognition device provided in the embodiments of this application. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0095] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0096] (1) Multimodal:

[0097] In the field of machine learning, multimodal learning refers to learning tasks and techniques that involve two or more different types of data modalities. These data modalities can include, but are not limited to, text, images, audio, video, and biosignals. Multimodal learning aims to improve the performance of models by combining different types of input data.

[0098] (2) Transformer structure:

[0099] The transformer architecture can include an encoder-decoder architecture, where the encoder is responsible for converting the input sequence into an intermediate representation, and the decoder is responsible for using the representation generated by the encoder to generate the output sequence.

[0100] The transformer architecture can also include a self-attention mechanism layer, where multi-head attention allows the model to learn information from different representation subspaces; positional encoding is used to preserve the positional information of the input sequence.

[0101] The transformer architecture can also include a feedforward neural network, which can be placed after each self-attention mechanism layer for further feature processing.

[0102] The transformer structure can also include residual connections, placed before and after each sub-layer, which helps to alleviate the vanishing gradient problem and accelerate the training process.

[0103] The transformer architecture can also include layer normalization applied to the output of each sub-layer to help stabilize the training process.

[0104] (3) Feedforward Neural Network (FFN):

[0105] A feedforward neural network is a basic neural network structure in which information flows unidirectionally, from the input layer through several hidden layers to the output layer. In this network structure, each neuron is only connected to the neurons in the layer preceding it, with no feedback connections.

[0106] (4) Deepfake:

[0107] Deepfake is a type of fake image or video generated using artificial intelligence technology, typically used to replace or synthesize facial expressions of a person.

[0108] It should be noted that the embodiments of this application involve operations such as obtaining sample video sets and other data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0109] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0110] The following is a brief introduction to the training method of the face recognition model and the application fields of the face recognition method provided in the embodiments of this application.

[0111] With the continuous development of technology, deepfake technology has become increasingly sophisticated, capable of synthesizing or altering multimedia content to generate increasingly realistic fake videos. When applied to fields such as film special effects production, virtual reality, or augmented reality, it can achieve stunning visual effects. However, in some areas, fake videos can be used to trigger abnormal operations, resulting in serious adverse effects.

[0112] Please refer to Figure 1A This is a schematic diagram illustrating a field of creating a synthetic face based on a real human face. The device can model a real human face and generate a corresponding synthetic face. Thus, the synthetic face can simulate a real human face to achieve dynamic expressions and actions; for example, the synthetic face can simulate a real human face saying things that the real human face has never said.

[0113] To prevent fake videos from being used to trigger abnormal operations, more and more devices can use facial recognition models to perform facial recognition on the facial regions in videos, in order to identify fake videos containing fake faces and thus identify potential false information.

[0114] Under related technologies, there are various methods for training face recognition models. One method is to use a face recognition model to extract image quality features of sample images during each iteration of training based on a sample image set. The model parameters of the face recognition model are adjusted based on the difference between the obtained image quality features and the preset reference quality features of each forged image, so as to achieve the purpose of training the face recognition model and obtain the trained target face recognition model.

[0115] Another approach is to use a face recognition model to extract pixel features from sample images during each iteration of training based on a massive set of sample images. The obtained pixel features are then identified to obtain the recognition results. The model parameters of the face recognition model are adjusted based on the difference between the recognition results and the synthetic labels of the sample images, in order to train the face recognition model and obtain the trained target face recognition model.

[0116] However, this target face recognition model trained based on image quality features can only identify low-quality forged images from the perspective of image quality. As technology advances, the image quality of forged images is getting higher and higher. It is impossible to accurately identify high-quality forged images based solely on image quality. It is easy to identify forged images as real images, which can lead to serious adverse consequences.

[0117] This method of training a face recognition model based on pixel features requires the construction of a sample image set containing a massive number of labeled sample images in order to learn the pixel features of the synthetic face from the sample image set containing the synthetic face. However, the synthetic face obtained by different synthesis techniques exhibits different features, which makes the recognition accuracy of the trained target face recognition model unstable.

[0118] It is evident that the methods used to train face recognition models under these technologies result in face recognition models with low accuracy.

[0119] To address the issue of low recognition accuracy in trained face recognition models, this application proposes a training method for face recognition models. This method involves acquiring a sample video set, with each sample video associated with a synthetic label indicating whether it contains a synthetic face. Based on the sample video set, the face recognition model to be trained undergoes multiple rounds of iterative training, outputting a trained target face recognition model.

[0120] Each round of iterative training includes:

[0121] In each video frame of the selected sample video, the facial regions of the sample person are detected. Facial change features are extracted from each obtained facial region. These features describe the facial changes of the sample person across video frames from multiple perspectives, including biosignal features, facial visual features, and correlation features. The correlation feature perspective describes the changes in facial changes that characterize the correlation between the sample person's biosignals and facial visual features. Based on the obtained facial change features, the synthesis degree of the sample person is predicted, and the facial recognition model is tuned based on the difference between the obtained synthesis degree and the synthesis labels associated with the sample video.

[0122] In this embodiment, the face region of the sample person is detected in the sample video, thereby obtaining multiple face regions of the sample person in the sample video. Through the multiple face regions of the sample person, the face recognition model can analyze the face changes of the sample person in the sample video, thereby improving the accuracy of the face recognition model in identifying whether a face is a synthetic face.

[0123] Furthermore, the facial change features of the sample person extracted from each obtained face region represent changes in biosignals, changes in facial vision, and changes in the correlation between biosignals and facial vision. Thus, by combining the biosignals of the sample person, it is possible to identify whether the face of the sample person in the sample video is a synthetic face, rather than simply identifying and synthesizing faces based on image quality or a single image pixel value. This can improve the recognition accuracy of the trained target face recognition model.

[0124] Furthermore, facial variation features that can represent multiple modalities can be extracted from each facial region alone. This eliminates the need to first input data from multiple modalities and then set up feature extraction modules for each modality to extract features from the corresponding input data. This avoids the use of a dual-tower feature extraction process, reduces the number of model parameters, avoids the problem of inconsistent feature spaces between multimodal feature extraction modules, and improves the efficiency and accuracy of model training.

[0125] The following describes the application scenarios of the training method for the face recognition model provided in this application.

[0126] Please refer to Figure 1BThis is a schematic diagram illustrating an application scenario of the face recognition model training method provided in this application. The application scenario includes a client 101 and a server 102. The client 101 and the server 102 can communicate with each other. The communication method can be wired, such as through a network cable or serial cable; or wireless, such as through Bluetooth or Wi-Fi. No specific limitation is imposed.

[0127] Client 101 generally refers to devices capable of capturing video and presenting corresponding composite results, such as terminal devices, third-party applications accessible by terminal devices, or web pages accessible by terminal devices. Terminal devices include, but are not limited to, mobile phones, computers, smart medical devices, smart home appliances, vehicle terminals, or aircraft. Server 102 generally refers to devices capable of training and using facial recognition models, such as terminal devices or servers. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers. Both client 101 and server 102 can utilize cloud computing to reduce the consumption of local computing resources; similarly, they can also utilize cloud storage to reduce the consumption of local storage resources.

[0128] As one embodiment, the client 101 and the server 102 can be the same device or different devices, and there is no specific limitation.

[0129] The following is based on Figure 1B This document provides a detailed description of the training method for the face recognition model provided in the embodiments of this application. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a method for training a face recognition model provided in an embodiment of this application.

[0130] S201, Obtain the sample video set.

[0131] The sample video set contains multiple sample videos, each associated with a synthetic label indicating whether it contains a synthetic face. For example, if a sample video contains a sample person, the synthetic label indicates whether the sample person's face is a fictional face obtained through synthesis or other methods, or whether the sample person's face is a real face obtained through filming or other methods.

[0132] The sample video may contain only one sample person or multiple sample people. When the sample video contains multiple sample people, each sample person is associated with a synthetic tag. In this embodiment, the example of a sample video containing one sample person will be used for illustration.

[0133] Sample video sets can be obtained from online resources or received from other devices. For example, sample video sets may include video data and their synthetic labels obtained from various datasets such as FaceForensics++ (FF), Celeb-DF, and UADFV.

[0134] The FaceForensics++ (FF) dataset contains 1000 real videos and 4000 synthetic videos. These synthetic videos were generated using different synthesis strategies, such as FaceSwap, Face2Face, Deepfakes, and Neural Textures. Constructing a sample video set using synthetic videos obtained through various strategies can enhance the generalization ability of the trained target face recognition model, avoid overfitting, and improve the recognition accuracy of the model to some extent.

[0135] The Celeb-DF dataset contains 590 real celebrity videos and 5639 deepfake versions. The UADFV dataset contains 48 real videos and 48 synthetic videos generated by the FakeAPP application.

[0136] By constructing a sample video set based on video data from multiple datasets, the trained target face recognition model can have a higher generalization ability, avoid overfitting, and improve the recognition accuracy of the trained target face recognition model to a certain extent.

[0137] S202, based on a sample video set, performs multiple rounds of iterative training on the face recognition model to be trained, and outputs the trained target face recognition model.

[0138] After obtaining the sample video set, the face recognition model to be trained can be iteratively trained multiple times based on the sample video set. If the training objective is not reached, the model parameters of the face recognition model are adjusted, and the next round of iterative training begins. When the training objective is reached, the trained target face recognition model is output based on the current model parameters. The training objective can be set based on the number of training iterations of the face recognition model, or it can be determined based on the training loss of the face recognition model, etc., without any specific restrictions.

[0139] The following description uses one round of iterative training as an example. Other rounds of iterative training are similar and will not be repeated here. Please refer to S2021 to S2023.

[0140] S2021, detect the face region of the sample person in each video frame contained in the selected sample video.

[0141] The sample video may contain only one video clip containing the sample person, or it may contain multiple video clips containing the sample person. When the sample video contains only one video clip of the sample person, the face region of the sample person can be detected from all the video frames contained in that video clip; alternatively, the face region of the sample person can be detected from only a portion of the video frames contained in that video clip, etc., without any specific restrictions.

[0142] When a sample video contains multiple video clips of a sample person, the face region of the sample person can be detected for each of the multiple video clips, or at least one video clip can be selected from the multiple video clips to detect the face region of the sample person, etc. There are no specific restrictions.

[0143] For example, when the duration of a video clip exceeds a preset duration threshold, multiple video frames can be selected from the video clip according to the preset duration threshold, and the facial regions of the sample people can be detected separately. This avoids the problem of a large amount of data to be processed due to the excessive length of the video clip, which would reduce the efficiency of model training.

[0144] For example, when the number of sample videos in the sample video set exceeds a preset threshold, multiple video frames can be selected from the video clips according to a preset number of video frames to detect the face region of the sample person respectively; alternatively, one video clip can be selected from multiple video clips to detect the face region of the sample person respectively.

[0145] For example, when the video quality of the sample video is higher than a preset quality threshold, multiple video frames can be selected from the video clip according to the first number of video frames to detect the face region of the sample person separately; alternatively, one video clip can be selected from multiple video clips to detect the face region of the sample person separately. Similarly, when the video quality of the sample video is lower than the preset quality threshold, multiple video frames can be selected from the video clip according to the second number of video frames to detect the face region of the sample person separately; alternatively, at least two video clips can be selected from multiple video clips to detect the face region of the sample person separately. The first number of video frames is less than the second number of video frames.

[0146] By selectively adjusting the number of video frames that need to be processed for each sample video, the training efficiency of the face recognition model can be dynamically adjusted, thereby balancing training accuracy and training efficiency.

[0147] Detecting whether a video frame contains a face, whether the contained face belongs to the sample person, and determining the face region where the contained face is located can be done using a trained face detection module, or by using edge detection technology to detect whether a face is contained and the face region. By matching image features with a pre-stored reference face, it can be determined whether the face belongs to the sample person, etc. There are no specific restrictions.

[0148] As one embodiment, multiple consecutive video frames can be acquired from a sample video according to a preset number of frames. Each acquired video frame is then checked to determine whether it contains a face, and the image region containing the face. If more than a certain number of video frames do not contain a face, multiple consecutive video frames can be acquired again, and each newly acquired video frame is checked to determine whether it contains a face, and the image region containing the face.

[0149] An image region can be represented by a bounding box (such as a bounding box) that encloses the area where the face is located in the corresponding video frame. That is, an image region can be a region marked in the corresponding video frame by a bounding box; an image region can also be a region obtained by cropping the area where the face is located in the corresponding video frame, etc. There are no restrictions on the specifics.

[0150] After obtaining the image regions contained in each of several consecutive video frames, the face regions containing the faces can be determined as belonging to the sample person based on the obtained image regions. If the number of face regions belonging to the sample person is not greater than the region number threshold, then several consecutive video frames can be reacquired, and each reacquired video frame can be checked to see if it contains a face, the image regions containing the face, and the face regions containing the face belonging to the sample person selected from the reacquired image regions.

[0151] Therefore, multiple facial regions of a sample person can be obtained from multiple consecutive video frames containing the sample person. These multiple facial regions obtained from consecutive video frames can reflect changes in the face, thus helping to obtain the biometric signals exhibited in the facial regions and improve the recognition accuracy of the face recognition model.

[0152] Please refer to Figure 3AWhen acquiring multiple video frames, a sliding window approach can be used. Taking a T-frame window as an example, the window selects T consecutive video frames from the sample video each time, such as 5 video frames. The T video frames are then input into the trained face detection module in the order they appear in the sample video. The face detection module detects whether each input video frame contains a face, the person identifier of the person to whom the face belongs (e.g., R1), and the coordinates of the image region of the face in the corresponding video frame (e.g., (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0153] The sample person can be the person who occupies the longest time in the sample video, the person who occupies the longest time in the current video segment, or the person who occupies the longest time in the current video frame within the sliding window, etc. There are no specific restrictions.

[0154] If the person identifier for the sample person is R2, then it can be determined whether the person to whom the face in each video frame contains a face is the sample person based on the person identifier. Thus, the face region of each video frame can be determined. The corresponding face region can be obtained by cropping the corresponding video frames according to their coordinate representations.

[0155] If the currently acquired T-frame video frames do not contain human faces, or the number of video frames containing human faces is small, then the window can be slid with a certain step size, such as sliding the window according to a non-repeating detection strategy, to reacquire T-frame video frames.

[0156] As one embodiment, when selecting facial regions of a sample person from various image regions, the selection can be made by comparing the similarity of faces between the obtained image regions. For example, if many image regions contain similar faces, then those facial regions can be selected. If the number of similar facial regions is small, such as five image regions containing dissimilar faces or only two similar faces, then multiple video frames need to be selected again for re-detection. Therefore, it is not necessary to pre-set a sample person; instead, face recognition can be performed on each person in the sample video to determine whether it is a synthetic face, thus ensuring the accuracy of the face recognition model.

[0157] For each obtained image region, the following operations are performed: Keypoint detection is performed on the image region to obtain a facial keypoint dataset. The facial keypoint dataset is used to describe the facial features and pose of the faces contained in the image regions.

[0158] Based on the data similarity between the facial landmark datasets of each image region, multiple image regions with a data similarity of not less than a similarity threshold are selected from each image region. Based on these multiple image regions, each facial region containing faces belonging to the sample person is determined.

[0159] Please refer to Figure 3B It includes a first image region, a second image region, and a third image region; the first image region corresponds to the first facial key point dataset, the second image region corresponds to the second facial key point dataset, and the third image region corresponds to the third facial key point dataset.

[0160] The data similarity between the first and second facial landmark datasets is less than a similarity threshold, and the data similarity between the first and third facial landmark datasets is also less than a similarity threshold; however, the data similarity between the second and third facial landmark datasets is greater than a similarity threshold. Therefore, the second and third image regions can be considered as two face regions, namely the first face region and the second face region, respectively.

[0161] As one embodiment, when determining the face region, multiple selected image regions can be registered. For example, the image region can be cropped so that the tip of the nose of the face in the image region is located at the center point of the cropped face region; or the image region can be enlarged so that the face in the image region occupies two-thirds of the enlarged face region, and the eyes of the face are located on both sides of the central axis of the enlarged face region; or the pose of the face in the image region can be adjusted so that the face is fully frontal, etc., and there are no specific limitations.

[0162] Registration ensures that faces in a face image have a uniform standard, thus preventing the negative impact of the face's position in the face region on the face recognition process during subsequent feature extraction. This reduces the learning difficulty of the face recognition model, improves training efficiency, and enhances the recognition accuracy of the trained target face recognition model.

[0163] One image region can be selected from multiple image regions as a reference region. Based on the data differences between the facial keypoint datasets of every two image regions, image mapping strategies between the multiple image regions and the reference region are determined. Based on the obtained image mapping strategies, image mapping transformations are performed on the corresponding image regions to obtain the corresponding face regions.

[0164] Taking two facial landmark datasets from two image regions as an example, a keypoint descriptor is determined for each facial landmark in each dataset. This descriptor describes the facial features within the neighborhood of the corresponding keypoint. Based on the obtained descriptors, the data similarity between any two facial landmarks in the two datasets is determined. Two facial landmarks with a similarity greater than a preset threshold are considered matched pairs. Based on each pair, a transformation matrix, such as an affine transformation matrix or a perspective transformation matrix, is determined as the image mapping strategy. Each transformation matrix is ​​multiplied by the pixel matrix of the corresponding image region to obtain the corresponding face region.

[0165] Please refer to Figure 3C From the sample video, three video frames are extracted through a three-frame window. The face detection module detects the image regions containing faces in each video frame and marks them with rectangular boxes.

[0166] For each pair of consecutive video frames, a facial landmark dataset is determined. Based on each pair of facial landmark datasets, an image mapping strategy for the corresponding image region can be determined. According to the image mapping strategy, the corresponding image region is transformed to obtain the corresponding face region. This ensures that the positions of the faces in each obtained face region are roughly the same, and the size and position of the faces in each face region are relatively uniform. Each face region is then cropped from each video frame for subsequent processing.

[0167] S2022, extract the facial variation features of the sample person from each obtained face region.

[0168] Facial change features are used to describe the facial changes of a sample person across video frames from multiple feature perspectives. These multiple feature perspectives include biosignal feature perspectives, facial visual feature perspectives, and correlation feature perspectives. The correlation feature perspective describes the facial changes that characterize the correlation between the sample person's biosignals and facial visual features.

[0169] Facial vision includes features such as facial contours, facial smoothness, and hairstyle that can be perceived visually.

[0170] Biosignals are physiological signals extracted from the facial region that can reflect the body's internal physiological state. Examples of biosignals include remote photoplethysmographic pulse wave signals (RPPGs), which reflect changes in blood flow caused by heartbeats; facial expressions or micro-expressions, such as changes in facial muscles; eye states, such as blinking frequency and pupil size; skin temperature, which reflects blood circulation; and Galvanic Skin Response (GSR), which reflects autonomic nervous system activity, such as emotional fluctuations. This application uses RPPGs as an example for illustration; other biosignal types will not be elaborated upon here.

[0171] There may be a correlation between facial vision and biosignals. For example, when facial expressions such as frowning or opening the mouth wide appear, biosignals may also reflect signals such as an increased heart rate; similarly, when facial expressions such as smiling appear, biosignal features may also reflect signals such as a stable heart rate.

[0172] Taking RPPG signals as an example, the correlation between biosignals and facial vision can be as follows: when the RPPG signal is a high-frequency, large-amplitude waveform, facial vision displays exaggerated expressions; conversely, when the RPPG signal is a low-frequency, small-amplitude waveform, facial vision displays no expression. Therefore, changes in the correlation between biosignals and facial vision can be seen as synchronous changes between variations in the RPPG signal waveform and variations in facial expression.

[0173] Therefore, the changes of a face in multiple consecutive video frames can be described from multiple feature perspectives, including biosignal features, facial visual features, and related features. This allows for the acquisition of more accurate facial change features, thereby improving the learning efficiency of the face recognition model. On the one hand, it can reduce the number of sample videos required during the learning process, thus reducing the training difficulty. On the other hand, it can improve the recognition accuracy of the target face recognition model obtained through training.

[0174] As one embodiment, when extracting facial change features, on the one hand, biosignal features of the sample person are extracted from each obtained facial region. Biosignal features characterize the correlation between changes in facial blood flow and facial light intensity, reflecting the sample person's heartbeat, i.e., pulse rate, etc. On the other hand, the visual features of the face reflected by the pixel values ​​contained in each facial region are extracted from each region. Therefore, at least one round of feature fusion is performed based on the obtained visual facial features and biosignal features to determine the facial change features of the sample person.

[0175] Facial visual features and biosignal features can be extracted using the same feature extraction network structure, or different feature extraction network structures can be used. When using different feature extraction network structures, facial visual features can be extracted using feature extraction network structures obtained from network resources, without any specific restrictions. This application will provide a detailed description of the extraction process for biosignal features.

[0176] Since the biosignal features and facial visual features extracted by the face recognition model are both extracted from single-modality input data such as the face region, the face recognition model does not need to set up corresponding modality-adapted feature extraction networks for multimodal input data. For example, it does not need to set up a feature extraction network for the face region and another feature extraction network for the RPPG signal. That is, it avoids the use of multi-tower feature extraction networks. Instead, the feature extraction process of multimodal data can be realized by using a single-tower feature extraction network that only adapts to a single modality.

[0177] Since it is not necessary to set up a special network structure for feature extraction for each modality, it not only simplifies the model structure of the face recognition model and reduces the number of model parameters, but also avoids the inconsistency of feature spaces of features extracted by network structures dedicated to each modality. This simplifies the feature space alignment process required when feature spaces are inconsistent, reduces the training difficulty of the face recognition model, and improves the training efficiency of the face recognition model.

[0178] Please refer to Figure 4A After obtaining three face regions, the biosignal features of the sample person can be extracted from each face region. Through these biosignal features, the changes in the biosignal of the sample person within the time range of the video frames corresponding to these three face regions can be identified.

[0179] Simultaneously, from each face region, the visual features of the face reflected by the pixel values ​​contained within that face region are extracted, including visual features in multiple aspects such as face pixel values, face edges, facial feature positions, and muscle texture. Therefore, by obtaining the visual features of each face region, the visual changes of the sample person's face within the time range of the video frames corresponding to these three face regions can be reflected.

[0180] The obtained biosignal features can be identified using sequence embeddings, and the obtained facial visual features can be stitched together according to the order of the corresponding video frames and identified using image embeddings.

[0181] Therefore, at least one round of feature fusion can be performed based on sequence embedding and image embedding to obtain facial variation features. Feature fusion allows the face recognition model to learn more abstract features that are difficult to discern visually in each facial region, thereby improving the model's ability to recognize synthetic faces and increasing accuracy. The more rounds of feature fusion, the more abstract the features learned by the face recognition model. The number of feature fusion rounds can be set according to the complexity of the face to be recognized in the actual use case. For example, a larger number of feature fusion rounds can be set for micro-expression recognition scenarios, while a smaller number can be set for exaggerated expression recognition scenarios; there are no specific limitations.

[0182] The process of extracting biosignal features from the human face region will be described in detail below.

[0183] For each obtained face region, the following operations are performed: Multiple rounds of convolution operations are conducted on the face region based on preset multi-channel convolution kernels to obtain initial local features. Based on the obtained initial local features, the biometric features of the sample person are determined. The multi-channel convolution kernel contains multiple channels corresponding to the face region. For example, if the face region contains R, G, and B channels, then the multi-channel convolution kernel also contains R, G, and B channels. The kernel elements in each channel can be the same or different; the specific values ​​can be learned through the training process of the face recognition model.

[0184] The kernel size of a multi-channel convolutional kernel is smaller than the size of the face region. A multi-channel convolutional kernel can be applied to a single face region or multiple consecutive face regions, thus capturing the temporal features of each face region and improving the accuracy of the initial local features obtained. Therefore, multi-channel convolutional kernels can extract coarse local spatiotemporal features from the face region, which helps the face recognition model converge quickly during training.

[0185] Please refer to Figure 4B Taking a three-layer convolutional layer as an example, this paper introduces the extraction of initial local features for a face region. The kernel sizes of the multi-channel convolutional kernels in each convolutional layer are (1×5×5), (3×3×3), and (3×3×3), respectively. Each convolutional layer can be followed by a batch normalization (BN) layer, an activation function layer (ReLU), and a max pooling layer (MaxPool).

[0186] For example, if the size of the face region is (3×H×W), when passing through the first convolutional layer, the first convolution kernel (1×5×5) is used to perform convolution operations on the three channels contained in the face region to obtain the first convolution result.

[0187] By padding with zeros, the size of the first convolution result remains (3×H×W). Then, it passes through the first normalization layer and the first activation function layer in sequence, and finally through the first pooling layer. Max pooling with a stride of 2 is performed using a pooling kernel with a size of (3×2×2) to obtain the size of the first pooling output as (3×H / 2×W / 2).

[0188] In the second convolutional layer, a second convolution kernel (3×3×3) is used to perform a convolution operation on the first pooling output to obtain the second convolution result. By padding with zeros, the size of the second convolution result is still (3×H / 2×W / 2). Then, it passes through the second normalization layer and the second activation function layer in sequence. Finally, a max pooling process with a pooling kernel size of (3×2×2) and a stride of 2 is performed to obtain the size of the second pooling output as (3×H / 4×W / 4).

[0189] When passing through the third convolutional layer, a third convolution kernel (3×3×3) is used to perform a convolution operation on the output of the second pooling layer to obtain the third convolution result. By padding with zeros, the size of the third convolution result is still (3×H / 2×W / 2). Then, it passes through the third normalization layer and the third activation function layer in sequence. Finally, a max pooling process with a pooling kernel size of (3×2×2) and a stride of 2 is performed to obtain the initial local feature size of (3×H / 8×W / 8).

[0190] As one example, in order to reduce the amount of computational data, the obtained initial local features can be divided into multiple tubular features in a tubular form. Each tubular feature is represented by its semantic features to obtain the biosignal features of the sample person. Converting a large number of initial local features into a small number of semantic features can reduce the amount of biosignal feature data, improve the efficiency of computation based on biosignal features, and thus improve the training efficiency of training the face recognition model.

[0191] According to a preset tubular size, each initial local feature is divided into tubular features, resulting in multiple tubular features. Each tubular feature contains multiple local sub-features. The local sub-features contained in the same tubular feature originate from different initial local features, and each local sub-feature has the same sub-feature position in its corresponding initial local feature. Semantic features are extracted from each of the multiple tubular features to obtain corresponding semantic features. Based on the obtained semantic features, the biosignal features of the sample individuals are determined.

[0192] Please refer to Figure 4CThe preset tubular dimensions are (Ts×Hs×Ws), where Ts represents the feature length, such as the number of initial local features; Hs represents the feature height; and Ws represents the feature width. If the size of the feature sequence formed by each initial local feature is (T×H×W), then the feature sequence formed by each initial local feature can be divided into multiple tubular features.

[0193] The corresponding tubular feature can be represented by the semantic features of each tubular feature; alternatively, the mapping relationship between the semantic features of each tubular feature and the tubular identifier can be stored, so that the corresponding tubular feature can be represented by the tubular identifier. Each tubular marker independently captures the features of a specific spatiotemporal segment in multiple consecutive video frames, avoiding redundant information processing and improving computational efficiency and feature independence.

[0194] As one embodiment, after obtaining each initial local feature, convolution operation can be performed on the initial local features in each channel first, and then the operation results obtained from multiple channels can be fused. In this way, the features conveyed by each channel can be captured independently, avoiding interference between multiple channels. Finally, the features of multiple channels are fused, which can capture the relationship between the features of multiple channels, which helps to further refine local inconsistencies and noise features, and at the same time provides rich location information for subsequent feature fusion.

[0195] If the initial local features contain single-channel sub-features corresponding to multiple color channels, then for each obtained initial local feature, the following operations are performed: based on the preset single-channel convolution kernels corresponding to multiple color channels, convolution operations are performed on the corresponding single-channel sub-features in the initial local features, and the obtained convolution results are fused to obtain comprehensive local features. Thus, the biosignal features of the sample person can be determined based on the obtained comprehensive local features. The kernel size of the single-channel convolution kernel is smaller than the feature size of the initial local features. The single-channel convolution kernel can be a convolution kernel for one initial local feature or a convolution kernel for multiple consecutive initial local features, thereby capturing the temporal features of each initial local feature and improving the accuracy of the obtained comprehensive local features.

[0196] Please refer to Figure 4D After obtaining the semantic features that characterize the tubular features, taking a single semantic feature as an example, a first linear transformation layer can be used to perform a linear transformation on the semantic feature. This can reduce the dimensionality of high-dimensional data or increase the dimensionality of low-dimensional data, mapping the feature to other spaces, without any specific restrictions. This yields the result of the first linear transformation.

[0197] Then, using single-channel convolution kernels corresponding to multiple color channels, depth-separable 3D convolution operations are performed on the first linear transformation result. Taking a convolution kernel size of (3×3) as an example, the operation results corresponding to multiple channels are obtained. Then, a (1×1) convolution kernel is used to fuse the operation results corresponding to multiple channels. A second linear transformation layer is used to perform a linear transformation on the fused result to obtain the corresponding comprehensive local features.

[0198] As one embodiment, when determining the facial change features of the sample person by performing at least one round of feature fusion based on the obtained facial visual features and the biosignal features, the following description uses one round of feature fusion as an example. Other rounds of feature fusion are similar and will not be repeated here.

[0199] In the current first round of feature fusion, each face visual feature is used as an intermediate visual feature, and the biosignal feature is used as an intermediate signal feature. In the current round of feature fusion other than the first round, the face visual change features obtained after the previous round of feature fusion are used as intermediate visual features, and the biosignal change features obtained after the previous round of feature fusion are used as intermediate signal features.

[0200] Based on the element similarity between feature elements contained in intermediate visual features, the element similarity between feature elements contained in intermediate signal features, and the feature similarity between each intermediate visual feature and intermediate signal feature, feature fusion is performed on intermediate visual features and intermediate signal features to determine multi-head attention features. Biometric recognition is then performed on the multi-head attention features to obtain biometric signal change features. Finally, visual feature recognition is performed on the multi-head attention features to obtain facial visual change features.

[0201] Based on the biosignal change features and facial visual change features obtained from the last round of feature fusion in at least one round of feature fusion, facial change features are determined.

[0202] The element similarity between feature elements in intermediate visual features, the element similarity between feature elements in intermediate signal features, and the feature similarity between intermediate visual features and intermediate signal features can be calculated using a multi-head self-attention (MSA) mechanism, which can be implemented using a transformer structure. Intermediate visual features and intermediate signal features are each assigned a feature vector for each time step, creating corresponding feature vector sequences. The self-attention calculation includes: for each time step, calculating the query, key, and value vectors. The query and key vectors are multiplied by a dot product to obtain attention weights, which are then weighted and summed to obtain the output. The input data is divided into multiple heads, each performing the above self-attention calculation independently. The outputs of all heads are then combined to form the final output. This multi-head self-attention mechanism captures the dependencies between different time steps in the sequence data, helping the model better understand the dynamic changes of faces in the video.

[0203] S2023, based on the obtained facial change features, predict the degree of synthesis of the sample person, and based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video, adjust the parameters of the face recognition model.

[0204] After obtaining facial change features, the degree of synthesis of the sample person can be predicted based on the obtained facial change features, and the facial recognition model can be tuned based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

[0205] For example, based on the obtained facial change features, the probability that the face of a sample person is a synthetic face can be predicted. The degree of synthesis can be determined based on the predicted probability. The higher the probability, the more likely the face is to be a synthetic face, and the higher the degree of synthesis. The lower the probability, the less likely the face is to be a synthetic face, and the lower the degree of synthesis.

[0206] If the synthetic label associated with the sample video indicates that the face in the sample task is a synthetic face, then the degree of synthesis can be considered to be 100%; if the synthetic label associated with the sample video indicates that the face in the sample task is not a synthetic face, then the degree of synthesis can be considered to be 0%.

[0207] Therefore, by observing the difference between the obtained synthesis degree and the synthesis label associated with the sample video, it is possible to determine when the training objective has not been achieved and to adjust the parameters of the face recognition model; until the training objective is achieved, the trained target face recognition model is output.

[0208] After obtaining facial change features, biosignals can be predicted based on the changes in biosignals. The facial recognition model can then be tuned based on the differences between the predicted biosignals and the biosignal samples of the individuals associated with the sample video.

[0209] It can also predict face images based on the visual changes in the face, and adjust the parameters of the face recognition model based on the differences between the predicted face images and the image samples of the sample people associated with the sample videos.

[0210] Furthermore, multiple differences can be combined to tune the face recognition model, including the differences between the degree of synthesis and the synthetic labels associated with the sample video, the differences between the predicted biosignals and the biosignal samples of the sample person associated with the sample video, and the differences between the predicted face images and the image samples of the sample person associated with the sample video.

[0211] As an example, if it is necessary to adjust the model parameters of the face recognition model during each round of training iterations, only some of the parameters can be adjusted to achieve targeted adjustments and improve adjustment efficiency.

[0212] Obtain the number of training iterations for statistical analysis. Once the number of training iterations is within a certain range, adjust the first model parameters in the face recognition model based on the difference between the obtained synthetic degree and the synthetic labels associated with the sample videos. The first model parameters include the model parameters used when performing visual feature recognition using the face recognition model. The first model parameters may also include other model parameters in the face recognition model besides those used for biometric recognition.

[0213] When the number of training iterations is determined to be within the second range, the second model parameters in the face recognition model are adjusted based on the difference between the obtained syntheticity and the synthetic labels associated with the sample videos. The second model parameters include the model parameters used when performing biometric recognition using the face recognition model. The second model parameters may also include other model parameters in the face recognition model besides those used when performing visual feature recognition.

[0214] The first number range and the second number range are two non-overlapping numerical intervals. The first number range and the second number range can each be a continuous numerical interval, or they can each contain multiple continuous numerical intervals, etc. There are no specific restrictions.

[0215] Each value in the first range can be less than each value in the second range, or each value in the first range can be greater than each value in the second range.

[0216] When the first and second number ranges each contain multiple consecutive numerical intervals, the minimum value within the first number range is less than the minimum value within the second number range. For example, the first number range might include [0, 10], [20, 30], and [40, 50], while the second number range might include (10, 20), (30, 40), and (50, 60). Alternatively, the minimum value within the second number range might be less than the minimum value within the first number range; no specific restriction applies. This allows for alternating training of the first and second model parameters, facilitating the full utilization of the information contained in each modality and improving the accuracy of synthetic face recognition.

[0217] Please refer to Figure 5A Taking a feature fusion process as an example, each face visual feature is converted into a face visual feature sequence, and the biosignal features are converted into a biosignal feature sequence. A multi-head self-attention mechanism is used to perform self-attention calculation to obtain multi-head attention features.

[0218] The RPPG feature expert feedforward neural network (PPG-FFN) is used to perform biometric recognition on multi-head attention features to obtain biometric signal change features; the image feature expert feedforward neural network (I-FFN) is used to perform visual feature recognition on multi-head attention features to obtain facial visual change features.

[0219] Therefore, if the training objective is not achieved, in the first stage (when the number of training iterations is less than a preset threshold), adjustments can be made only to the first model parameters involved in the image feature expert feedforward neural network. In the second stage (when the number of training iterations is not less than the preset threshold), adjustments can be made only to the second model parameters involved in the RPPG feature expert feedforward neural network. Thus, the face recognition model focuses on learning key features of different modalities at different stages, and can gradually learn and integrate features from both modalities, improving the training stability and efficiency of the face recognition model.

[0220] As one example, if the sample video contains non-synthetic faces and is also associated with biosignal samples of non-synthetic faces, then the biosignal of the person in the sample is predicted based on the obtained biosignal features. The face recognition model is then tuned based on the difference between the obtained syntheticity and the synthetic label associated with the sample video, as well as the differences between biosignals and biosignal samples. Tuning the face recognition model by referencing multiple datasets improves tuning accuracy, training efficiency, and avoids problems such as overfitting.

[0221] Please refer to Figure 5BAfter obtaining multiple face regions, the resulting face region sequence can be input into a shallow backbone (STEM) layer. In the shallow backbone, multi-channel convolutional kernels are used to perform convolution operations on the multiple face regions, obtaining an initial local feature sequence containing multiple initial local features. For example, the size of the face region sequence is (3×T×H×W), where 3 represents 3 channels, T represents the number of region elements in the face region sequence, H represents the height of each face region, and W represents the width of each face region. After processing by the shallow backbone layer, the size of the obtained initial local feature sequence is (3×T×H / 8×W / 8).

[0222] The initial local feature sequence is divided into a tubular label map composed of semantic features of multiple tubular features using a preset tubular size (Ts×Hs×Ws), with a size of (3×T'×H'×W').

[0223] The obtained tubular label map is input into a multi-head self-attention computation layer to obtain a self-attention feature sequence. This self-attention feature sequence is then input into a spatio-temporal feed-forward (ST-FF) neural network to obtain biosignal features. The ST-FF neural network consists of two linear transformation layers, and a depth-separable 3D convolutional layer between these two linear transformation layers. Following this 3D convolutional layer, a normalization layer and a ReLU activation function layer can also be applied.

[0224] By inputting the obtained biosignal features into the prediction layer, the biosignals of the sample person can be predicted. By comparing the differences between the obtained biosignal features and the biosignal samples of the sample person associated with the sample video, it can be determined whether the face recognition model needs to be tuned.

[0225] The following continues based on Figure 1B This document provides a detailed description of the face recognition method provided in the embodiments of this application. Please refer to [link / reference]. Figure 6 This is a flowchart illustrating a face recognition method provided in an embodiment of this application.

[0226] S601, acquire the video to be recognized.

[0227] The video to be identified can be obtained from real-time transmission from the shooting device, or sent from other devices, or obtained by shooting the content played on the display screen, etc. There are no specific restrictions.

[0228] S602 detects the face region of the target person in each video frame contained in the video to be identified.

[0229] The video frames included in the video to be identified can be all the video frames in the video to be identified, or multiple consecutive video frames in the video to be identified, etc. For details, please refer to the introduction in the training process above, which will not be repeated here.

[0230] S603 uses a trained target face recognition model to extract the facial change features of the target person from each obtained face region, and identifies the obtained facial change features to obtain the synthesized result of the target person.

[0231] The process of extracting facial change features of the target person from each obtained face region using a target face recognition model, and the process of recognizing the obtained facial change features to obtain the synthetic result of the target person, can be referred to the previous introduction on the training process of the face recognition model, and will not be repeated here.

[0232] The synthesized result characterizes whether the target person is a synthesized face. The target face recognition model is trained using the methods described in claims 1-11.

[0233] The following section uses RPPG signals as an example to illustrate the training method and face recognition method of the face recognition model provided in this application.

[0234] Please refer to Figure 7 The face recognition model includes a face detection network, an RPPG signal feature extraction network, an image feature extraction network, a feature fusion network, an RPPG feature expert feedforward neural network, an image feature expert feedforward neural network, and a prediction network.

[0235] A face detection network is used to obtain multiple consecutive video frames from the sample video. Each of these video frames contains the face of the sample person. Taking three video frames as an example, the face region containing the face of the sample person can be detected from each video frame. The image region containing the face can be extracted from the video frame to remove the interference factors of the background region.

[0236] After obtaining three image regions, a face detection network can be used to register the faces in these three image regions. Based on the set of facial key points contained in each image region, an image mapping strategy is calculated. Thus, based on the image mapping strategy, image mapping transformation can be performed on each image region to obtain the corresponding face region, so that the relative positions of the faces in each face region are roughly the same.

[0237] An RPPG signal feature extraction network is used to extract biosignal features from each face region, forming a biosignal feature sequence. An image feature extraction network is used to extract the facial visual features reflected by the pixel values ​​of each face region, forming a facial visual feature sequence.

[0238] The feature fusion network, RPPG feature expert feedforward neural network, and image feature expert feedforward neural network are used to perform at least one round of feature fusion on the obtained biosignal feature sequence and facial visual feature sequence. For example, N rounds of feature fusion are performed to obtain facial change features.

[0239] A predictive network is used to identify facial variation features and predict the degree of synthesis of the sample person. When the number of training iterations is within the range corresponding to the first stage, the first model parameters involved in the image feature expert feedforward neural network are adjusted based on the difference between the degree of synthesis and the synthesis label. When the number of training iterations is within the range corresponding to the second stage, the second model parameters involved in the RPPG signal feature extraction network are adjusted based on the difference between the degree of synthesis and the synthesis label.

[0240] Enter the next round of iteration training until the training objective is achieved, and then output the trained target face recognition model.

[0241] During the training phase, the sample video set consists of 70% of the FaceForensics++ dataset, including 700 real videos and 4*700 deepfake videos generated by four different generative models. During the testing phase, the test video set consists of the remaining 30% of the FaceForensics++ dataset, including 300 real videos and 4*300 deepfake videos, also generated by four different generative models. An out-of-domain testing phase may also be included, with the out-of-domain test video set including UADFV, used to test unseen generators. The out-of-domain testing phase allows for the evaluation of the face recognition model's generalization ability on new data and unknown generators.

[0242] The process of performing face recognition using the target face recognition model is similar to the training process described above, and will not be repeated here.

[0243] In this embodiment, the RPPG feature expert feedforward neural network and the image feature expert feedforward neural network perform feature recognition on the RPPG signal and the visual content of the face region, respectively. This allows the face recognition model to process features of two modalities simultaneously in the same feature space, achieving deep feature interaction and fusion. Compared to traditional multi-tower methods for processing multimodal data, the single-tower structure in this embodiment simplifies the face recognition model architecture and reduces the number of model parameters. Furthermore, by using both the RPPG feature expert feedforward neural network and the image feature expert feedforward neural network, the model's ability to capture complex features is improved. This not only enhances the face recognition model's performance in detecting synthetic faces in deepfake videos but also strengthens its generalization ability across different datasets and unknown generators.

[0244] Furthermore, by alternately freezing the model parameters of the RPPG feature expert feedforward neural network and the image feature expert feedforward neural network without adjusting them, the mutual interference in the feature learning process of different modalities can be effectively reduced. This allows the face recognition model to learn the features of each modality more stably, and improves the face recognition model's comprehensive understanding and analysis of facial visual features and RPPG signal features. As a result, higher accuracy and robustness are achieved in deepfake video detection tasks.

[0245] Furthermore, when extracting biosignal features, a shallow trunk structure is adopted, which can quickly extract key local spatiotemporal features. The tubular labeling map can effectively reduce computational costs. At the same time, the introduction of a spatiotemporal feedforward neural network can refine features and enhance the expression of location information, which can effectively improve the efficiency and accuracy of the face recognition model in biosignal detection.

[0246] Based on the same inventive concept, embodiments of this application provide a training device for a face recognition model, capable of realizing the functions corresponding to the aforementioned face recognition model training method. Please refer to... Figure 8A The device includes an acquisition module 81 and a processing module 82, wherein:

[0247] Acquisition Module 81: Used to acquire a set of sample videos; each sample video is associated with a synthetic label indicating whether it contains a synthetic face;

[0248] Processing module 82: Used to perform multiple rounds of iterative training on the face recognition model to be trained based on the sample video set, and output the trained target face recognition model; wherein each round of iterative training includes:

[0249] The processing module 82 is specifically used to: detect the face region of the sample person in each video frame contained in the selected sample video;

[0250] The processing module 82 is specifically used to: extract facial change features of the sample person from each obtained face region; the facial change features are used to describe the facial changes of the sample person between video frames from multiple feature perspectives; the multiple feature perspectives include: biosignal feature perspective, facial visual feature perspective and correlation feature perspective; the facial changes described by the correlation feature perspective represent the correlation changes between the biosignals and facial vision of the sample person.

[0251] The processing module 82 is specifically used to: predict the degree of synthesis of the sample person based on the obtained facial change features, and adjust the parameters of the face recognition model based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

[0252] In one possible embodiment, the processing module 82 is specifically used for:

[0253] According to the preset number of frames, obtain multiple consecutive video frames from the selected sample video;

[0254] Each obtained video frame contains an image region containing a face;

[0255] Based on the obtained image regions, the face regions containing the faces that belong to the sample person are determined.

[0256] In one possible embodiment, the processing module 82 is specifically used for:

[0257] For each obtained image region, the following operations are performed: key point detection is performed on the image region to obtain a face key point dataset; wherein, the face key point dataset is used to describe the facial features and facial pose of the face contained in the image region;

[0258] Based on the data similarity between the facial landmark datasets of each image region, multiple image regions with a data similarity of not less than a similarity threshold are selected from each image region.

[0259] Based on multiple image regions, the facial regions containing the included faces that belong to the sample person are determined.

[0260] In one possible embodiment, the processing module 82 is specifically used for:

[0261] Select one image region from multiple image regions as the reference region;

[0262] Based on the data differences between the facial key point datasets of every two image regions in multiple image regions, the image mapping strategy between multiple image regions and the reference region is determined respectively.

[0263] Based on the obtained image mapping strategies, image mapping transformations are performed on the corresponding image regions to obtain the corresponding face regions.

[0264] In one possible embodiment, the processing module 82 is specifically used for:

[0265] From the obtained facial regions, the biosignal features of the sample individuals are extracted; among them, the biosignal features represent the correlation between changes in facial blood flow and changes in facial light intensity of the sample individuals;

[0266] Extract the visual features of the face reflected by the pixel values ​​contained in each face region from each face region;

[0267] Based on the obtained facial visual features and biosignal features, at least one round of feature fusion is performed to determine the facial variation features of the sample individuals.

[0268] In one possible embodiment, the processing module 82 is specifically used for:

[0269] For each obtained face region, the following operations are performed: Based on multiple preset multi-channel convolution kernels, multiple rounds of convolution operations are performed on the face region to obtain initial local features; wherein, the kernel size of the multi-channel convolution kernel is smaller than the region size of the face region;

[0270] Based on the obtained initial local features, the biosignal characteristics of the sample individuals are determined.

[0271] In one possible embodiment, the processing module 82 is specifically used for:

[0272] According to the preset tubular size, each initial local feature is divided into tubular features to obtain multiple tubular features; each tubular feature contains multiple local sub-features, and each local sub-feature contained in the same tubular feature comes from different initial local features, and each local sub-feature has the same sub-feature position in the corresponding initial local feature.

[0273] Semantic features are extracted from multiple tubular features to obtain the corresponding semantic features;

[0274] Based on the obtained semantic features, the biosignal features of the sample individuals are determined.

[0275] In one possible embodiment, the initial local features include multiple single-channel sub-features corresponding to each color channel;

[0276] The processing module 82 is specifically used for:

[0277] For each of the obtained initial local features, the following operations are performed respectively: based on the single-channel convolution kernels corresponding to each of the preset multiple color channels, the corresponding single-channel sub-features in the initial local features are convolved, and the obtained convolution results are fused to obtain the comprehensive local features; wherein, the kernel size of the single-channel convolution kernel is smaller than the feature size of the initial local features.

[0278] The biosignal characteristics of the sample individuals are determined based on the obtained comprehensive local features.

[0279] In one possible embodiment, the processing module 82 is specifically used for:

[0280] For each round of feature fusion, perform the following operations:

[0281] Based on the element similarity between the feature elements contained in the intermediate visual features, the element similarity between the feature elements contained in the intermediate signal features, and the feature similarity between each intermediate visual feature and the intermediate signal features, feature fusion is performed on the intermediate visual features and intermediate signal features to determine multi-head attention features. In the first round of feature fusion, the intermediate visual features include each face visual feature, and the intermediate signal features are bio-signal features. In other rounds of feature fusion, the intermediate visual features are the face visual change features obtained after the previous round of feature fusion, and the intermediate signal features are the bio-signal change features obtained after the previous round of feature fusion.

[0282] Biometric identification of multi-head attention features to obtain biosignal change characteristics;

[0283] Visual feature recognition is performed on multi-head attention features to obtain facial visual change features;

[0284] Based on the biosignal change features and facial visual change features obtained from the last round of feature fusion in at least one round of feature fusion, facial change features are determined.

[0285] In one possible embodiment, the processing module 82 is specifically used for:

[0286] Obtain the number of training iterations in the multi-round training;

[0287] When the number of training iterations is determined to be within the first range, the first model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis labels associated with the sample videos; wherein, the first model parameters include: the model parameters used when performing visual feature recognition using the face recognition model;

[0288] When the number of training iterations is determined to be within the second range, the second model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video; wherein, the second model parameters include: the model parameters used when performing biometric recognition using the face recognition model; the minimum value within the first range is less than the minimum value within the second range.

[0289] In one possible embodiment, when the sample video contains non-synthetic human faces, the sample video is also associated with biosignal samples of non-synthetic human faces.

[0290] The processing module 82 is specifically used for:

[0291] Predict the biosignals of sample individuals based on the obtained biosignal features;

[0292] Based on the differences between the obtained synthetic degree and the synthetic labels associated with the sample videos, as well as the differences between biological signals and biological signal samples, the parameters of the face recognition model are tuned.

[0293] Based on the same inventive concept, this application provides a face recognition device capable of realizing the functions corresponding to the aforementioned face recognition method. Please refer to... Figure 8B The device includes an acquisition module 801 and a processing module 802, wherein:

[0294] Acquisition module 801: Used to acquire the video to be recognized;

[0295] Processing module 802: used to detect the face region of the target person in each video frame contained in the video to be identified;

[0296] The processing module 802 is further configured to: use a trained target face recognition model to extract the facial change features of the target person from each obtained face region, and identify the obtained facial change features to obtain the synthetic result of the target person; wherein, the synthetic result represents whether the target person is a synthetic face; the target face recognition model is trained using the method as described in claims 1 to 11.

[0297] Please refer to Figure 9 This is a computer device 900 provided in the embodiments of this application. The computer device 900 can, for example, be... Figure 1B The client 101 or server 102 in the system. The current and historical versions of the data storage program and the application software corresponding to the data storage program can be installed on a computer device 900, which includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, which includes a display panel 941 for displaying a user-interactive interface, etc.

[0298] In one possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0299] The processor 980 is used to read a computer program and then execute the methods defined by the computer program. For example, the processor 980 reads a data storage program or file, thereby running the data storage program on the computer device 900 and displaying the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided in the embodiments of this application.

[0300] The memory 920 generally includes main memory and secondary storage. Main memory can be random access memory (RAM), read-only memory (ROM), and cache, etc. Secondary storage can be a hard disk, optical disk, USB flash drive, floppy disk, or magnetic tape drive, etc. The memory 920 is used to store computer programs and other data. The computer programs include applications corresponding to each client, and other data may include data generated after the operating system or applications are run, including system data (e.g., operating system configuration parameters) and user data. In this embodiment, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods described in the preceding figures.

[0301] The aforementioned display unit 940 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and to generate signal inputs related to user settings and function control of the computer device 900. Specifically, in this embodiment, the display unit 940 may include a display panel 941. The display panel 941, for example, is a touch screen, which can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.

[0302] In one possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the player's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 980. It can also receive and execute commands from the processor 980.

[0303] The display panel 941 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may also include an input unit 930. The input unit 930 may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0304] In addition to the above, the computer device 900 may also include a power supply 990 for powering other modules, an audio circuit 960, a near-field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an accelerometer, a light sensor, and a pressure sensor. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, for example, the computer device 900 can use the microphone 962 to collect the user's voice and perform corresponding operations.

[0305] As one embodiment, the number of processors 980 can be one or more, and the processors 980 and the memory 920 can be coupled together or relatively independent.

[0306] As one example, Figure 9 The processor 980 in the middle can be used to implement, for example Figure 8A The functions of the acquisition module 81 and the processing module 82 are described; it can also be used to implement, for example... Figure 8B The functions of the acquisition module 801 and the processing module 802 in the process.

[0307] As one example, Figure 9 The processor 980 in the text can be used to implement the functions of the server or terminal devices discussed above.

[0308] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by a computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of software products, for example, through computer program products. These computer program products are stored in a storage medium and include computer programs used to cause a computer device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0310] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for training a face recognition model, characterized in that, include: Obtain a sample video set; each sample video is associated with a synthetic label indicating whether it contains a synthetic face; Based on the sample video set, the face recognition model to be trained undergoes multiple rounds of iterative training, outputting the trained target face recognition model; wherein each round of iterative training includes: The facial regions of the sample individuals are detected in each video frame of the selected sample video. From each obtained face region, facial change features of the sample person are extracted; the facial change features are used to describe the facial changes of the sample person between each video frame from multiple feature perspectives; the multiple feature perspectives include: biosignal feature perspective, facial visual feature perspective, and correlation feature perspective; the facial changes described by the correlation feature perspective represent the correlation changes between the biosignals and facial vision of the sample person; The degree of synthesis of the sample person is predicted based on the obtained facial change features, and the facial recognition model is tuned based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

2. The method according to claim 1, characterized in that, The step of detecting the facial regions of the sample individuals in each video frame of the selected sample video includes: According to the preset number of frames, obtain multiple consecutive video frames from the selected sample video; Each obtained video frame contains an image region containing a face; Based on the obtained image regions, the face regions containing the faces that belong to the sample person are determined.

3. The method according to claim 2, characterized in that, The step of determining which face regions, based on the obtained image regions, belong to the sample person includes: For each obtained image region, the following operations are performed: key point detection is performed on the image region to obtain a facial key point dataset; wherein, the facial key point dataset is used to describe the facial features and facial pose of the face contained in the image region; Based on the data similarity between the facial key point datasets of each image region, multiple image regions with a data similarity of not less than a similarity threshold are selected from each image region. Based on the multiple image regions, the facial regions containing the included faces that belong to the sample person are determined.

4. The method according to claim 3, characterized in that, The step of determining which facial regions, based on the multiple image regions, belong to the sample person includes: Select one image region from the plurality of image regions as a reference region; Based on the data differences between the facial key point datasets of every two image regions in the plurality of image regions, an image mapping strategy between the plurality of image regions and the reference region is determined respectively; Based on the obtained image mapping strategies, image mapping transformations are performed on the corresponding image regions to obtain the corresponding face regions.

5. The method according to any one of claims 1 to 4, characterized in that, The step of extracting facial variation features of the sample person from each obtained facial region includes: From each obtained face region, the biosignal features of the sample person are extracted; wherein, the biosignal features represent the correlation between changes in facial blood flow and changes in facial light intensity of the sample person; From each face region, extract the visual features of the face reflected by the pixel values ​​contained in the corresponding face region; Based on the obtained facial visual features and the biosignal features, at least one round of feature fusion is performed to determine the facial change features of the sample person.

6. The method according to claim 5, characterized in that, The step of extracting biometric features from each obtained facial region of the sample person includes: For each obtained face region, the following operations are performed: based on multiple preset multi-channel convolution kernels, multiple rounds of convolution operations are performed on the face region to obtain initial local features; wherein, the kernel size of the multi-channel convolution kernel is smaller than the region size of the face region; Based on the obtained initial local features, the biosignal features of the sample individuals are determined.

7. The method according to claim 6, characterized in that, The determination of the biosignal features of the sample person based on the obtained initial local features includes: According to the preset tubular size, each initial local feature is divided into tubular features to obtain multiple tubular features; wherein, each tubular feature contains multiple local sub-features, and each local sub-feature contained in the same tubular feature comes from different initial local features, and each local sub-feature has the same sub-feature position in the corresponding initial local feature. Semantic features are extracted from multiple tubular features to obtain the corresponding semantic features; Based on the obtained semantic features, the biosignal features of the sample individuals are determined.

8. The method according to claim 6, characterized in that, The initial local features include multiple single-channel sub-features corresponding to each color channel; The process of determining the biosignal features of the sample person based on the obtained initial local features includes: For each obtained initial local feature, the following operations are performed: based on the single-channel convolution kernels corresponding to each of the preset multiple color channels, convolution operations are performed on the corresponding single-channel sub-features in the initial local features, and the obtained convolution results are fused to obtain comprehensive local features; wherein, the convolution kernel size of the single-channel convolution kernel is smaller than the feature size of the initial local features; The biosignal characteristics of the sample individuals are determined based on the obtained comprehensive local features.

9. The method according to claim 5, characterized in that, The process of performing at least one round of feature fusion based on the obtained facial visual features and the biosignal features to determine the facial variation features of the sample individuals includes: For each round of feature fusion, perform the following operations: Based on the element similarity between the feature elements contained in the intermediate visual features, the element similarity between the feature elements contained in the intermediate signal features, and the feature similarity between the intermediate visual features and the intermediate signal features, feature fusion is performed on the intermediate visual features and the intermediate signal features to determine multi-head attention features; wherein, in the first round of feature fusion, the intermediate visual features include the facial visual features, and the intermediate signal features are the biosignal features; in other rounds of feature fusion, the intermediate visual features are the facial visual change features obtained after the previous round of feature fusion, and the intermediate signal features are the biosignal change features obtained after the previous round of feature fusion; Biometric identification is performed on the multi-head attention features to obtain biosignal change features; Visual feature recognition is performed on the multi-head attention features to obtain facial visual change features; The facial change features are determined based on the biosignal change features and facial visual change features obtained from the last round of feature fusion in the at least one round of feature fusion.

10. The method according to claim 9, characterized in that, The parameter tuning of the face recognition model based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video includes: Obtain the number of training iterations in the multi-round training; When the number of training iterations is determined to be within a first range, the first model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video; wherein, the first model parameters include: model parameters used when performing visual feature recognition using the face recognition model; When the number of training iterations is determined to be within a second range, the second model parameters in the face recognition model are adjusted based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video; wherein, the second model parameters include: model parameters used when performing biometric recognition using the face recognition model; the minimum value within the first range is less than the minimum value within the second range.

11. The method according to any one of claims 1 to 4, characterized in that, When the sample video contains a non-synthetic human face, the sample video is also associated with a biosignal sample of the non-synthetic human face. The parameter tuning of the face recognition model based on the difference between the obtained synthesis degree and the synthesis label associated with the sample video includes: Predict the biosignals of the sample individuals based on the obtained biosignal features; The face recognition model is tuned based on the difference between the obtained degree of synthesis and the synthetic label associated with the sample video, as well as the difference between the biosignal and the biosignal sample.

12. A face recognition method, characterized in that, include: Obtain the video to be recognized; The facial region of the target person is detected in each video frame contained in the video to be identified; Using a trained target face recognition model, facial change features of the target person are extracted from each obtained face region, and the obtained facial change features are identified to obtain a synthetic result of the target person; wherein, the synthetic result indicates whether the target person is a synthetic face; the target face recognition model is trained using the method described in any one of claims 1 to 11.

13. A training device for a face recognition model, characterized in that, include: Acquisition module: used to acquire sample video sets; each sample video is associated with a synthetic label indicating whether it contains a synthetic face; Processing module: used to perform multiple rounds of iterative training on the face recognition model to be trained based on the sample video set, and output the trained target face recognition model; wherein, each round of iterative training includes: The processing module is specifically used to: detect the face region of the sample person in each video frame contained in the selected sample video; The processing module is specifically used to: extract facial change features of the sample person from each obtained face region; the facial change features are used to describe the facial changes of the sample person between each video frame from multiple feature perspectives; the multiple feature perspectives include: biosignal feature perspective, facial visual feature perspective, and correlation feature perspective; the facial changes described by the correlation feature perspective represent the correlation changes between the biosignals and facial vision of the sample person; The processing module is specifically used to: predict the degree of synthesis of the sample person based on the obtained facial change features, and adjust the parameters of the face recognition model based on the difference between the obtained degree of synthesis and the synthesis label associated with the sample video.

14. A face recognition device, characterized in that, include: Acquisition module: Used to acquire the video to be recognized; Processing module: used to detect the face region of the target person in each video frame contained in the video to be identified; The processing module is further configured to: use a trained target face recognition model to extract the facial change features of the target person from each obtained face region, and identify the obtained facial change features to obtain a synthetic result of the target person; wherein, the synthetic result indicates whether the target person is a synthetic face; the target face recognition model is trained using the method described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 12.

16. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 12 according to the obtained program instructions.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 12.

Images