Counterfeit video identification method and device, equipment and medium

Abstract

The embodiment of the invention provides a forged video identification method and device, equipment and a medium. The method comprises the following steps: acquiring a to-be-identified video, and determining a multi-modal feature and a key point feature of the to-be-identified video; wherein the multi-modal features represent fusion features, synchronized with face texture and audio vision and related to physiological signals, in each frame of the to-be-recognized video, and the key point features represent spatial position change conditions of face key parts in continuous frames in the to-be-recognized video; determining an identification result according to the multi-modal features and the key point features; wherein the identification result is used for evaluating the possibility that the to-be-identified video is a forged video. The method is used for improving the precision of identifying the forged video.

Description

Technical Field

[0001] This application relates to the field of video recognition technology, and in particular to a method, apparatus, device and medium for identifying forged videos. Background Technology

[0002] With the explosive growth of applications such as short videos, live streaming, and virtual meetings, the technical barrier for deepfake technology has rapidly decreased, making the need to identify whether a video is fake increasingly urgent.

[0003] In existing technologies, deepfake face videos are detected by segmenting the video into frames, extracting facial image features from the video frames based on face markers, and then processing these facial image features using gradient descent. However, this approach has limitations: it only considers facial image features, and the facial image feature processing steps are too simple, resulting in poor accuracy in identifying fake videos.

[0004] Therefore, there is an urgent need for a solution that can improve the accuracy of identifying fake videos. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for identifying forged videos, thereby improving the accuracy of identifying forged videos.

[0006] In a first aspect, embodiments of this application provide a method for identifying forged videos, including:

[0007] The video to be identified is acquired, and its multimodal features and key point features are determined. The multimodal features represent the fusion features related to facial texture, audio-visual synchronization, and physiological signals in each frame of the video to be identified, while the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames.

[0008] The identification result is determined based on multimodal features and key point features; the identification result is used to evaluate the likelihood that the video to be identified is a fake video.

[0009] In one possible implementation, determining the multimodal features of the video to be identified includes:

[0010] The facial texture features, audio-visual synchronization features, and biometric features of the video to be identified are determined. Among them, facial texture features represent the image texture of the facial region, audio-visual synchronization features represent the synchronization between audio and mouth movements, and biometric features represent the matching between heart rate and facial expression.

[0011] Based on facial texture features, audio-visual synchronization features, and biometric features, the multimodal features of the video to be identified are determined.

[0012] In one possible implementation, facial texture features, audio-visual synchronization features, and biometric features of the video to be identified are determined, including:

[0013] The video to be identified is input into a preset first network layer, a second network layer, and a third network layer, respectively, to obtain the facial texture features output by the first network layer, the audio-visual synchronization features output by the second network layer, and the biometric features output by the third network layer. The first network layer includes a convolutional neural network, the second network layer includes a preset dynamic time warping algorithm, and the third network layer includes a remote photoplethysmography algorithm.

[0014] In one possible implementation, multimodal features of the video to be identified are determined based on facial texture features, audio-visual synchronization features, and biometric features, including:

[0015] Based on facial texture features, audio-visual synchronization features, and biometric features, and using a pre-defined long short-term memory network, the first weight corresponding to facial texture features, the second weight corresponding to audio-visual synchronization features, and the third weight corresponding to biometric features are determined.

[0016] The facial texture features and the first weight, the audio-visual synchronization features and the second weight, as well as the biometric features and the third weight, are weighted to obtain the multimodal features of the video to be identified.

[0017] In one possible implementation, the recognition result is determined based on multimodal features and key point features, including:

[0018] Based on multimodal features, temporal features are determined, and based on keypoint features, motion anomaly features are determined; wherein, temporal features characterize the dynamic pattern of multimodal features in the time dimension, and motion anomaly features characterize the non-coherent behavior of keypoint features between adjacent frames.

[0019] The identification result is determined based on temporal characteristics and abnormal motion characteristics.

[0020] In one possible implementation, determining the temporal features based on multimodal features includes:

[0021] The multimodal features are arranged in chronological order of each frame to obtain the feature sequence;

[0022] The feature sequence is input into a preset temporal convolutional network to obtain the output temporal features.

[0023] In one possible implementation, determining motion anomaly features based on key point features includes:

[0024] Based on key point features, the optical flow field information and the corresponding residual values ​​between adjacent segments of the video to be identified are determined; whereby the optical flow field information represents the vector field of the movement of key facial parts in adjacent segments between adjacent frames, and the number of frames in adjacent segments is the same.

[0025] If the residual value corresponding to the optical flow field information is greater than or equal to the preset residual threshold, then the adjacent segment to which the optical flow field information belongs is identified as a motion anomaly feature.

[0026] In one possible implementation, the identification result is determined based on temporal characteristics and motion anomaly characteristics, including:

[0027] The temporal features and motion anomaly features are fused together to obtain the features to be classified;

[0028] The features to be classified are input into a preset classifier to obtain the output recognition result.

[0029] In one possible implementation, the preset classifier includes a support vector machine module and a multilayer perceptron module; the features to be classified are input into the preset classifier to obtain the output recognition result, including:

[0030] The features to be classified are input into the support vector machine module to obtain the classification results and confidence scores of each frame of the video to be identified, which constitute the first classification sequence.

[0031] Furthermore, the features to be classified are input into the multilayer perceptron module to obtain the classification results and confidence scores of each frame of the video to be identified, which constitute the second classification sequence; wherein, the classification result is either a fake result or a real result, and the confidence score represents the degree of certainty of the support vector machine module and the multilayer perceptron module in judging the classification result;

[0032] The identification result is determined based on the first classification sequence and the second classification sequence.

[0033] In one possible implementation, the first classification sequence includes a first classification result and a first confidence level of the first classification result for each frame, and the second classification sequence includes a second classification result and a second confidence level of the second classification result for each frame; determining the identification result based on the first classification sequence and the second classification sequence includes:

[0034] For the same frame in the first classification sequence and the second classification sequence, if the first classification result and the second classification result of the frame are consistent, then the first classification result or the second classification result is determined as the target classification result, and the target confidence of the target classification result is determined according to the first confidence and the second confidence.

[0035] If the first classification result and the second classification result of the frame are inconsistent, the maximum value between the first confidence and the second confidence is determined as the target confidence, and the classification result to which the target confidence belongs is determined as the target classification result;

[0036] The identification results are obtained by weighting the classification results of each target and the target confidence scores of the classification results.

[0037] Secondly, embodiments of this application provide a forged video identification device, comprising:

[0038] The determination module is used to acquire the video to be identified and determine the multimodal features and key point features of the video to be identified. Among them, the multimodal features represent the fusion features related to facial texture, audio-visual synchronization and physiological signals in each frame of the video to be identified, and the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames.

[0039] The recognition module is used to determine the recognition result based on multimodal features and key point features; the recognition result is used to evaluate the likelihood that the video to be recognized is a fake video.

[0040] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0041] The memory stores the instructions that the computer executes;

[0042] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0044] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0045] The forged video recognition method, apparatus, device, and medium provided in this application determine the multimodal features and key point features of the video to be recognized, and then determine the recognition result based on the multimodal features and key point features. The multimodal features characterize the fusion features related to facial texture, audio-visual synchronization, and physiological signals in each frame of the video to be recognized. Compared with the facial image features of existing technologies, multimodal features can improve the accuracy and breadth of feature extraction. Furthermore, combining multimodal features and key point features can further improve the accuracy of the recognition result. The method of this application aims to improve the accuracy of forged video recognition. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating a method for identifying forged videos provided in this application. Figure 1 ;

[0048] Figure 2 A flowchart illustrating a method for identifying forged videos provided in this application. Figure 2 ;

[0049] Figure 3 A schematic diagram of a forged video recognition device provided in this application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0054] With the explosive growth of applications such as short videos, live streaming, and virtual meetings, the technical threshold for deepfakes has rapidly decreased, making the need to identify whether a video is fake increasingly urgent.

[0055] In existing technologies, deepfake face videos are detected by segmenting the video into frames, extracting facial image features from the video frames based on face markers, and then processing these facial image features using gradient descent. However, this approach has limitations: it only considers facial image features, and the facial image feature processing steps are too simple, resulting in poor accuracy in identifying fake videos.

[0056] Therefore, there is an urgent need for a solution that can improve the accuracy of identifying fake videos.

[0057] The forged video recognition method, apparatus, device, and medium provided in this application determine the multimodal features and key point features of the video to be recognized, and then determine the recognition result based on the multimodal features and key point features. The multimodal features characterize the fusion features related to facial texture, audio-visual synchronization, and physiological signals in each frame of the video to be recognized. Compared with the facial image features of existing technologies, multimodal features can improve the accuracy and breadth of feature extraction. Furthermore, combining multimodal features and key point features can further improve the accuracy of the recognition result. The method of this application aims to improve the accuracy of forged video recognition.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] Figure 1 A flowchart illustrating a method for identifying forged videos provided in this application. Figure 1 The execution subject of this method can be a server, host, or other device, such as... Figure 1 As shown, the method includes:

[0060] S101. Acquire the video to be identified and determine the multimodal features and key point features of the video to be identified; wherein, the multimodal features represent the fusion features related to facial texture, audio-visual synchronization and physiological signals in each frame of the video to be identified, and the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames.

[0061] S102. Determine the recognition result based on multimodal features and key point features; wherein, the recognition result is used to evaluate the possibility that the video to be recognized is a fake video.

[0062] In step S101, the video to be identified can refer to a video sequence of any length containing a face region, obtained by decoding through a mobile terminal camera, a network streaming media interface, or a local storage file. For example, the video to be identified can include, but is not limited to, short videos, live streams, surveillance recordings, and video conference recordings.

[0063] Multimodal features can refer to a joint feature vector formed by splicing or weighting the facial visual texture vector (also known as facial texture features), the audio and mouth movement synchronization error scalar (also known as audio-visual synchronization features), and the remote photoplethysmography pulse wave physiological signal vector (also known as biometric features) at the same timestamp or the same frame of a video. It is used to simultaneously reflect facial appearance, audio-visual consistency, and heart rate variability.

[0064] For example, for each frame of the video to be identified, a lightweight convolutional neural network (CNN) such as MobileNetV3 is used to extract facial texture features, the synchronization error between the Mel frequency cepstral coefficients (MFCC) of the audio and the key points of the mouth is calculated to obtain audio-visual synchronization features, and biometric features are extracted by remote photoplethysmography (rPPG).

[0065] Key point features can refer to the two-dimensional coordinate offset of key facial features in consecutive frames, used to characterize the instantaneous and short-term motion trajectory of key facial features.

[0066] For example, key facial features can be key features of the eyes, nose, mouth, eyebrows, etc., and the number of key facial features can be, for example, 68.

[0067] Key point features can be determined by using face key point detection tools such as Dlib (which can refer to an open-source cross-platform machine learning and computer vision library) to extract key facial features from face image frames, resulting in a two-dimensional coordinate vector.

[0068] In step S102, the identification result can be a scalar. The identification result is used to evaluate the probability that the video to be identified is a fake video. For example, if the identification result is within [0, 0.5], it means that the video to be identified is not a fake video. If the identification result is within (0.5, 1], it means that the video to be identified is a fake video.

[0069] The recognition result is determined based on multimodal features and key point features. This can be achieved by fusing the multimodal features and key point features, inputting them into a preset classifier, and then outputting the recognition result.

[0070] The forged video recognition method provided in this application, compared with the facial image features of existing technologies, can improve the accuracy and breadth of feature extraction through multimodal features. Furthermore, combining multimodal features and key point features can further improve the accuracy of the recognition results. The method of this application aims to improve the accuracy of forged video recognition.

[0071] Figure 2 A flowchart illustrating a method for identifying forged videos provided in this application. Figure 2 The execution entity of this method is the server, host, or other device, such as Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the method for identifying forged videos is described in detail. The method includes:

[0072] S201. Obtain the video to be identified.

[0073] S202. Determine the facial texture features, audio-visual synchronization features, and biometric features of the video to be identified; among them, facial texture features represent the image texture of the facial region, audio-visual synchronization features represent the synchronization between audio and mouth movements, and biometric features represent the matching between heart rate and facial expression.

[0074] Among them, facial texture features can be represented as facial visual texture vectors, used to characterize the image texture of the facial region; audio-visual synchronization features can be represented as audio and mouth movement synchronization error scalars, used to characterize the synchronization between audio and mouth movement; biometric features can be represented as remote photoplethysmography pulse wave physiological signal vectors, used to characterize the matching between heart rate and facial expression.

[0075] In one alternative implementation, step S202 may include:

[0076] The video to be identified is input into a preset first network layer, a second network layer, and a third network layer, respectively, to obtain the facial texture features output by the first network layer, the audio-visual synchronization features output by the second network layer, and the biometric features output by the third network layer. The first network layer includes a convolutional neural network, the second network layer includes a preset dynamic time warping algorithm, and the third network layer includes a remote photoplethysmography algorithm.

[0077] The first network layer includes a convolutional neural network. For example, in the first network layer, the facial landmark detection tool Dlib can be used to extract multiple (e.g., 68) facial landmarks from each frame of the video to be recognized, and the facial landmarks are mapped (e.g., the coordinates of the facial landmarks are mapped to a normalized coordinate system of a preset size, such as 224×224) to obtain a 136-dimensional coordinate vector. At the same time, the lightweight convolutional neural network MobileNetV3 can be used to extract the texture features of each frame of the video to be recognized (its dimension is, for example, 128-dimensional). Furthermore, the coordinate vectors and texture features of each frame are concatenated to obtain the facial texture features (its dimension is, for example, 264-dimensional).

[0078] The second network layer includes a pre-defined Dynamic Time Warping (DTW) algorithm. For example, in this second layer, the audio of each frame in the video to be recognized is processed through frame segmentation, windowing, pre-emphasis, Fast Fourier Transform, and Mel filter filtering to determine the MFCC (Multiple Facial Convergence Corner) features of the audio in each frame. Furthermore, multiple mouth keypoints in each frame are extracted using the facial landmark detection tool Dlib, and these keypoints are mapped to obtain their coordinate vectors, i.e., mouth key features. Further, based on the MFCC features and mouth key features of each frame, the audio-visual synchronization features are determined using the DTW algorithm. For example, the audio-visual synchronization features satisfy the following:

[0079] ;

[0080] In the above formula, Indicates audio-visual synchronization characteristics. This indicates the total number of frames in the video to be identified. This represents the DTW algorithm. This represents the MFCC feature of the i-th frame of the video to be identified. This represents the key mouth features in the i-th frame of the video to be identified.

[0081] The principle of the DTW algorithm is to find an alignment path that minimizes the cumulative distance among all possible alignment methods of two sequences in order to measure the similarity between the sequences. In this application, the two sequences used for the DTW algorithm can be a sequence of MFCC features of the video to be identified according to the number of frames and a sequence of mouth key features of the video to be identified according to the number of frames.

[0082] The third network layer includes the remote photoplethysmography (rPPG) algorithm. For example, in the third network layer, the heart rate variability features of each frame in the video to be identified can be extracted based on the rPPG algorithm. For example, within 10 consecutive frames, the one-dimensional blood volume pulse signal sequence obtained by performing color separation and bandpass filtering on the G channel pixels of the region of interest (ROI) of the forehead and cheeks is used to reflect the fluctuation of heart rate over time. Then, the heart rate variability features are input into a three-dimensional (3D) convolutional neural network to obtain the output biometric features (the dimension of which is, for example, 64 dimensions).

[0083] The rPPG algorithm works by capturing changes in the reflected light from the skin of a face in a video to be identified. After preprocessing (e.g., filtering), it uses wavelet transform, deep learning, and other methods to separate features such as heart rate and respiratory rhythm from the changes in the reflected light of the skin to obtain heart rate variability features. Heart rate variability features characterize the matching between heart rate and facial expression. It can be understood that the more mismatched the heart rate and facial expression indicated by the heart rate variability features of the video to be identified are, the more likely it is to be a fake video.

[0084] 3D convolutional neural networks are mainly used in the field of spatiotemporal sequence data processing. Specifically, heart rate variability features are input into a 3D convolutional neural network to obtain output biometric features. For example, based on the height and width of the ROI of the forehead and cheeks of a human face, the heart rate variability features are mapped into a spatiotemporal tensor. Then, after processing by, for example, three convolutional layers in a 3D convolutional neural network, the output result is compressed into, for example, a 64-dimensional vector through global average pooling, which is the biometric feature.

[0085] The beneficial effect of this setup is that by inputting the video to be identified into the preset first, second, and third network layers respectively, the facial texture features, audio-visual synchronization features, and biometric features of the video to be identified are determined. This comprehensively considers factors such as facial appearance, audio-visual synchronization, and heart rate and expression matching as criteria for judging whether the video to be identified is a forged video, which can significantly improve the accuracy and robustness of forged video identification and effectively reduce the risk of false alarms faced by considering only the features of a single modality.

[0086] S203. Based on facial texture features, audio-visual synchronization features, and biometric features, determine the multimodal features of the video to be identified.

[0087] In one alternative implementation, step S203 may include:

[0088] S2031. Based on facial texture features, audio-visual synchronization features, and biometric features, and using a pre-defined long short-term memory network, determine the first weight corresponding to facial texture features, the second weight corresponding to audio-visual synchronization features, and the third weight corresponding to biometric features.

[0089] S2032. The facial texture features and the first weight, the audio-visual synchronization features and the second weight, as well as the biological features and the third weight, are weighted to obtain the multimodal features of the video to be identified.

[0090] Among them, the preset Long Short-Term Memory (LSTM) network can refer to a type of recurrent neural network. LSTM networks can be used to learn the importance distribution of facial texture features, audio-visual synchronization features, and biometric features over time.

[0091] Specifically, facial texture features, audio-visual synchronization features, and biometric features are concatenated (e.g., horizontally) to obtain concatenated features. These concatenated features are then input into an LSTM network, allowing the LSTM network to learn the dependencies between these three features over time. This process aims to uncover the potential importance of different modal features in detecting fake videos. Furthermore, the output of the LSTM network can be normalized using the softmax function to obtain a combination of weight coefficients. This combination includes a first weight corresponding to facial texture features, a second weight corresponding to audio-visual synchronization features, and a third weight corresponding to biometric features. The values ​​of the first, second, and third weights are all between 0 and 1, and the sum of the first, second, and third weights is 1.

[0092] For example, the combination of weighting coefficients satisfies:

[0093] ;

[0094] in, This represents the combination of weighting coefficients. Indicates the first weight. Indicates the second weight. Indicates the third weight. This represents the softmax function. Indicates an LSTM network. Represents facial texture features. Indicates audio-visual synchronization characteristics. Indicates biological characteristics.

[0095] Furthermore, by weighting the facial texture features with the first weight, the audio-visual synchronization features with the second weight, and the biometric features with the third weight, the multimodal features of the video to be identified can be obtained. These multimodal features can satisfy the following:

[0096] ;

[0097] in, Represents multimodal features, Indicates the first weight. Indicates the second weight. Indicates the third weight. Represents facial texture features. Indicates audio-visual synchronization characteristics. Indicates biological characteristics.

[0098] For example, if, for a given video to be identified, the LSTM network finds that facial texture features and audio-visual synchronization features play a crucial role in determining whether the video is forged, then the first weight... Second weight The value will be relatively large, for example When determining multimodal features, facial texture features and audio-visual synchronization features are given higher weights to occupy a more important position in multimodal features.

[0099] The beneficial effect of this setup is that by introducing an LSTM network, the first weight corresponding to facial texture features, the second weight corresponding to audio-visual synchronization features, and the third weight corresponding to biometric features are determined. This differs from the simple feature concatenation method, ensuring that the contribution of facial texture features, audio-visual synchronization features, and biometric features can be automatically adjusted according to the characteristics of different videos to be identified. This allows for more accurate capture of anomalies in forged videos under different modal features, thereby improving the accuracy of forged video identification.

[0100] S204. Based on the multimodal features, determine the temporal features; whereby the temporal features characterize the dynamic patterns of the multimodal features in the time dimension.

[0101] Temporal features can characterize the dynamic patterns of multimodal features over time. In other words, temporal features can be used to capture the evolution trends of facial textures, audio-visual synchronization, and physiological signals in videos to be identified over long time scales.

[0102] Specifically, temporal features can refer to multimodal features that take into account the time dimension. It can be understood that by capturing the dynamic patterns of multimodal features in the time dimension, temporal anomalies in the video to be identified can be better revealed, thereby improving the accuracy and robustness of identifying fake videos.

[0103] Based on multimodal features, temporal features can be determined by using bidirectional long short-term memory networks and / or extended temporal convolutional networks to determine temporal features.

[0104] In one alternative implementation, step S204 may include:

[0105] S2041. Arrange the multimodal features according to the chronological order of each frame to obtain the feature sequence.

[0106] S2042. Input the feature sequence into the preset temporal convolutional network to obtain the output temporal features.

[0107] Among them, the feature sequence can refer to a set of multimodal features at the frame level stacked in time order, and the preset temporal convolutional network (TCN) can refer to a temporal modeling network based on causal dilated convolution, which is used to expand the receptive field without increasing the number of parameters, capture long-distance temporal dependencies in the feature sequence, and input temporal features of the same length as the feature sequence.

[0108] For example, the time series characteristics can satisfy:

[0109] , ;

[0110] in, This represents the output feature of the TCN at time step t, and the output feature of the TCN at each time step, i.e., the time sequence feature. Indicates the expansion rate in TCN. This represents the number of convolutional layers in the TCN (e.g., 3 layers). This indicates the number of weights in the convolutional kernel of the TCN. This indicates that in the feature sequence input to the TCN, pushing back from time step t... The multimodal features at each time step are understandable. This embodies the "expansion" characteristic of dilated convolution, enabling TCN to span a certain number of time steps during convolution and capture longer-term temporal dependencies. For example, when d=2 and i=1, it will consider features from time step t−2 in the feature sequence. This represents the i-th weight in the convolution kernel of TCN.

[0111] It is understandable that temporal features are calculated by dividing the feature sequence and then performing a weighted summation. This reflects the comprehensive result after considering multiple time steps, and can better reveal temporal anomalies in the video to be identified, thereby improving the accuracy and robustness of identifying forged videos.

[0112] In one optional implementation, before inputting the feature sequence into a preset temporal convolutional network, the feature sequence can also be input into a preset bidirectional LSTM to obtain the output target feature sequence, so that the target feature sequence can be input into the preset temporal convolutional network to obtain the output temporal features.

[0113] Bidirectional LSTM (Bi-LSTM, Bidirectional Long Short-Term Memory Network) can refer to a special type of LSTM, a recurrent network composed of a forward LSTM and a backward LSTM connected in parallel, used to process the forward and reverse information of a sequence simultaneously, that is, to capture the long-distance dependencies of the feature sequence in both the historical and future directions.

[0114] The advantage of this setup is that it first uses bidirectional LSTM to capture the long-distance dependencies of feature sequences in both historical and future directions to obtain the target feature sequence. Then, it uses TCN to further expand the receptive field and refine the target feature sequence, so as to be able to perceive both local mutations in multimodal features and integrate the long-term evolution trend of multimodal features, thereby more accurately identifying whether the video to be identified is a fake video.

[0115] S205. Based on the key point features, determine the motion anomaly features; whereby the motion anomaly features characterize the non-coherent performance of the key point features between adjacent frames.

[0116] Motion anomaly features can characterize the non-coherent behavior of key point features between adjacent frames. In other words, motion anomaly features can be used to identify non-coherent, jittery, and other abnormal behaviors of the movement of key facial features in a video to be identified within a short time window.

[0117] Specifically, motion anomaly features can refer to key point features that take into account changes in the optical flow field. It can be understood that by determining motion anomaly features, we can better capture local motion abrupt changes, micro-jitter, and other situations caused by defects in the synthesis algorithm in the video to be identified, thereby providing a more granular basis for subsequent identification results and improving the accuracy and robustness of identifying the video to be identified.

[0118] For example, determining motion anomaly features based on key point features can be achieved by determining residual values ​​based on key point features and a preset optical flow algorithm. If the residual values ​​meet preset conditions, the video frame corresponding to the key point features is determined as a motion anomaly feature.

[0119] The residual value can satisfy:

[0120] ;

[0121] In the above formula, Represents the residual value. Let represent the optical flow field of the t-th frame in the video to be identified. The optical flow field describes the vector field of the motion of objects (i.e., key facial features) in the video between adjacent frames. It can record the motion direction and velocity of each pixel between adjacent frames, reflecting the true motion of objects in the video. The optical flow field can be determined by a preset optical flow algorithm, such as the dense optical flow Farneback algorithm or the Total Variation-L1 (TV-L1) optical flow algorithm. The motion optical flow field is obtained by taking the median value of the optical flow field pixel by pixel (i.e., the median value of the optical flow field of the 11 frames before and after the t-th frame as the center, which is used as the motion baseline. 2 indicates that the residual norm indicated by the residual value is 2.

[0122] It is understandable that optical flow algorithms are used to calculate the motion vector field of key point regions in adjacent frames (i.e., ), and uses the median optical flow field centered on the current frame (i.e., frame t) as the motion baseline (i.e. ), by calculating the residual value between the current optical flow field and the motion baseline (i.e. If the residual value exceeds the preset residual threshold, the frame is marked as abnormal, and the motion abnormal area indicated by the frame is identified as a motion abnormal feature.

[0123] In an optional implementation, step S205 may include:

[0124] S2051. Based on the key point features, determine the optical flow field information and the corresponding residual value between adjacent segments of the video to be identified; wherein, the optical flow field information represents the vector field of the movement of key facial parts in adjacent segments between adjacent frames, and the number of frames in adjacent segments is the same.

[0125] S2052. If the residual value corresponding to the optical flow field information is greater than or equal to the preset residual threshold, then the adjacent segment to which the optical flow field information belongs is determined as a motion anomaly feature.

[0126] Among them, optical flow field information can refer to the vector field that characterizes the movement of key facial features in adjacent segments between adjacent frames. The number of frames in adjacent segments is the same, for example, 5 frames each.

[0127] The residual value corresponding to the optical flow field information can satisfy:

[0128] ;

[0129] In the above formula, The residual value corresponding to the optical flow field information, This represents the optical flow field of the t-th frame in the video to be identified, i.e., the optical flow field information. The value represents the moving optical flow field obtained by taking the midpoint of the optical flow field pixel by pixel in the 11 frames before and after frame t, which serves as the motion baseline. That is, with frame t as the center, the first 5 frames are a segment and the last 5 frames are a segment, representing adjacent segments. 2 indicates that the residual norm indicated by the residual value is 2.

[0130] For example, if the residual value (e.g., 2.15) exceeds a preset residual threshold (e.g., 1.5), the frame is marked as abnormal, and the motion abnormal region indicated by the frame is identified as a motion abnormal feature.

[0131] The beneficial effect of this setup is that by considering the optical flow field to determine motion anomaly features, it can accurately capture instantaneous motion changes of key facial features in adjacent segments, thereby eliminating periodic interference caused by normal head movements or speaking. This provides a more reliable basis for the subsequent classifier to output recognition results, improving the recognition accuracy and robustness of the video to be recognized.

[0132] S206. Determine the identification result based on temporal characteristics and motion anomaly characteristics.

[0133] In an optional implementation, step S206 may include:

[0134] S2061. The temporal features and motion anomaly features are fused to obtain the features to be classified.

[0135] S2062. Input the features to be classified into the preset classifier to obtain the output recognition result.

[0136] The fusion processing can include, but is not limited to, channel splicing, weighted summation, attention weighting, bilinear pooling, or cross-attention, etc. The specific form of the fusion processing is not restricted here.

[0137] For example, temporal features and motion anomaly features are concatenated frame by frame in the time dimension to obtain the features to be classified.

[0138] It is understandable that by fusing temporal features and motion anomaly features, the resulting features to be classified contain both long-term multimodal evolution patterns and short-term local motion mutation patterns. This can provide subsequent classifiers with both global temporal context and local anomaly information, thereby significantly improving the recognition accuracy and robustness of the video to be identified.

[0139] The preset classifier can refer to a machine learning or deep learning discrimination model used to distinguish between true and false features after fusion. For example, the preset classifier can include, but is not limited to, Support Vector Machine (SVM), Multilayer Perceptron (MLP), Random Forest, eXtreme Gradient Boosting (XGBoost), Convolutional Neural Network, etc.

[0140] In an optional implementation, the preset classifier includes a support vector machine module and a multilayer perceptron module; step S2062 may include:

[0141] The features to be classified are input into the support vector machine module to obtain the classification results and confidence scores of each frame of the video to be identified, which is the first classification sequence. The features to be classified are input into the multilayer perceptron module to obtain the classification results and confidence scores of each frame of the video to be identified, which is the second classification sequence. The classification results are either fake or real, and the confidence scores represent the degree of certainty that the support vector machine module and the multilayer perceptron module have in judging the classification results. The identification result is determined based on the first classification sequence and the second classification sequence.

[0142] The Support Vector Machine (SVM) module contains a Support Vector Machine (SVM), and the Multilayer Perceptron (MPP) module contains a Multilayer Perceptron (MPP). The SVM is used to output the first classification sequence, and the MPP is used to output the second classification sequence.

[0143] The classification result can refer to the label of each frame of the video to be identified by the support vector machine module and the multilayer perceptron module as a fake result or a real result. For example, 1 represents a fake result and 0 represents a real result.

[0144] The confidence level of the classification result can refer to the posterior probability or decision distance of the classification result obtained by the support vector machine or multilayer perceptron, that is, the degree of confidence of the support vector machine module and the multilayer perceptron module in judging the classification result.

[0145] For example, the confidence level of the classification result output by the Support Vector Machine (SVM) is the decision distance (e.g., 0.75), and the confidence level of the classification result output by the Multilayer Perceptron (MLP) is the posterior probability (e.g., 0.88). Here, the posterior probability can refer to the probability value of the corresponding category (e.g., a fake result) after normalization by the Softmax function in the output of the MLP, and the decision distance can refer to the geometric distance from the input sample (e.g., a frame) to the target plane in the SVM, which is converted into a confidence level falling within the range of 0 to 1 after normalization.

[0146] It is understandable that by combining the first classification sequence and the second classification sequence to determine the recognition result, the risk of bias or failure of a single classifier can be effectively reduced, and the robustness of the preset classifier in the decision classification of a single frame of the video to be recognized and the recognition accuracy of the video as a whole can be improved.

[0147] In one optional implementation, the first classification sequence includes a first classification result and a first confidence level of the first classification result for each frame, and the second classification sequence includes a second classification result and a second confidence level of the second classification result for each frame; determining the identification result based on the first classification sequence and the second classification sequence may include:

[0148] For the same frame in the first classification sequence and the second classification sequence, if the first classification result and the second classification result of the frame are consistent, then the first classification result or the second classification result is determined as the target classification result, and the target confidence of the target classification result is determined according to the first confidence and the second confidence.

[0149] If the first classification result and the second classification result of the frame are inconsistent, the maximum value between the first confidence and the second confidence is determined as the target confidence, and the classification result to which the target confidence belongs is determined as the target classification result;

[0150] The identification results are obtained by weighting the classification results of each target and the target confidence scores of the classification results.

[0151] For example, for the same frame in the first classification sequence and the second classification sequence, if the first classification result and the second classification result of the frame are the same, for example, if both the first classification result and the second classification result are fake results, then the target classification result is a fake result; if both the first classification result and the second classification result are real results, then the target classification result is a real result. For example, the target confidence level of the target classification result can be the average between the first confidence level and the second confidence level.

[0152] For example, if the first classification result and the second classification result of the frame are inconsistent, for example, the first classification result is a fake result with a first confidence level of 0.9, and the second classification result is a real result with a second confidence level of 0.5, then the maximum value of the first confidence level and the second confidence level, i.e., 0.9, is determined as the target confidence level, and the classification result to which the target confidence level belongs, i.e., the first classification result, is determined as the target classification result, i.e., the fake result.

[0153] Furthermore, by weighting the target classification results and the target confidence scores of each frame, the recognition results can be obtained.

[0154] In one possible implementation, the recognition result can satisfy:

[0155] ;

[0156] In the above formula, Indicates the recognition result. This indicates the total number of frames in the video to be identified. This represents the target confidence level of the i-th frame. This represents the target classification result for the i-th frame.

[0157] For example, the value of the forged result is 1, and the value of the real result is 0. Taking the target classification results and target confidence scores of the three frames as follows: forged result (target confidence score 0.9), real result (target confidence score 0.85), and forged result (target confidence score 0.8), the recognition result satisfies... If the recognition result is greater than the preset recognition threshold (e.g., 0.5), it means that the conclusion of the recognition result of the video to be recognized is that the video to be recognized is a fake video.

[0158] The advantage of this setting is that by weighting the target classification results and the target confidence scores of each target classification result, the recognition result can be obtained. This fully utilizes the target classification results and target confidence scores of each frame, reduces noise interference, and improves the accuracy of the preset classifier in recognizing the video to be recognized.

[0159] The forged video recognition method provided in this application, compared with the facial image features of existing technologies, can improve the accuracy and breadth of feature extraction through multimodal features. Furthermore, combining multimodal features and key point features can further improve the accuracy of the recognition results. The method of this application aims to improve the accuracy of forged video recognition.

[0160] Figure 3 A schematic diagram of a forged video recognition device provided in this application is shown below. Figure 3 As shown, the forged video recognition device 30 provided in this embodiment includes: a determination module 301 and a recognition module 302.

[0161] The determination module 301 is used to acquire the video to be identified and determine the multimodal features and key point features of the video to be identified; wherein, the multimodal features represent the fusion features related to facial texture, audio-visual synchronization and physiological signals in each frame of the video to be identified, and the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames.

[0162] The recognition module 302 is used to determine the recognition result based on multimodal features and key point features; wherein, the recognition result is used to evaluate the possibility that the video to be recognized is a fake video.

[0163] In an optional example, the determination module 301 is further configured to determine the facial texture features, audio-visual synchronization features, and biometric features of the video to be identified; wherein, the facial texture features represent the image texture of the facial region, the audio-visual synchronization features represent the synchronization between audio and mouth movements, and the biometric features represent the matching between heart rate and facial expression.

[0164] Based on facial texture features, audio-visual synchronization features, and biometric features, the multimodal features of the video to be identified are determined.

[0165] In an optional example, the determining module 301 is further configured to input the video to be recognized into a preset first network layer, a second network layer, and a third network layer respectively, to obtain the facial texture features output by the first network layer, the audio-visual synchronization features output by the second network layer, and the biometric features output by the third network layer; wherein the first network layer includes a convolutional neural network, the second network layer includes a preset dynamic time warping algorithm, and the third network layer includes a remote photoplethysmography algorithm.

[0166] In an optional example, the determining module 301 is further configured to determine, based on a preset long short-term memory network, the first weight corresponding to the facial texture features, the second weight corresponding to the audio-visual synchronization features, and the third weight corresponding to the biometric features, the facial texture features, the audio-visual synchronization features, and the biometric features.

[0167] The facial texture features and the first weight, the audio-visual synchronization features and the second weight, as well as the biometric features and the third weight, are weighted to obtain the multimodal features of the video to be identified.

[0168] In an optional example, the recognition module 302 is further configured to determine temporal features based on multimodal features, and to determine motion anomaly features based on keypoint features; wherein, the temporal features characterize the dynamic pattern of multimodal features in the time dimension, and the motion anomaly features characterize the non-coherent behavior of keypoint features between adjacent frames.

[0169] The identification result is determined based on temporal characteristics and abnormal motion characteristics.

[0170] In an optional example, the recognition module 302 is also used to arrange the multimodal features in chronological order of each frame to obtain a feature sequence;

[0171] The feature sequence is input into a preset temporal convolutional network to obtain the output temporal features.

[0172] In an optional example, the recognition module 302 is further configured to determine the optical flow field information and the corresponding residual value between adjacent segments of the video to be recognized based on the key point features; wherein, the optical flow field information characterizes the vector field of the movement of key facial parts in adjacent segments between adjacent frames, and the number of frames in adjacent segments is the same.

[0173] If the residual value corresponding to the optical flow field information is greater than or equal to the preset residual threshold, then the adjacent segment to which the optical flow field information belongs is identified as a motion anomaly feature.

[0174] In an optional example, the recognition module 302 is also used to fuse temporal features and motion anomaly features to obtain features to be classified;

[0175] The features to be classified are input into a preset classifier to obtain the output recognition result.

[0176] In an optional example, the preset classifier includes a support vector machine module and a multilayer perceptron module; the recognition module 302 is further used to input the features to be classified into the support vector machine module to obtain the classification results and confidence scores of each frame of the video to be recognized, which constitute the first classification sequence.

[0177] Furthermore, the features to be classified are input into the multilayer perceptron module to obtain the classification results and confidence scores of each frame of the video to be identified, which constitute the second classification sequence; wherein, the classification result is either a fake result or a real result, and the confidence score represents the degree of certainty of the support vector machine module and the multilayer perceptron module in judging the classification result;

[0178] The identification result is determined based on the first classification sequence and the second classification sequence.

[0179] In an optional example, the first classification sequence includes the first classification result and the first confidence level of the first classification result for each frame, and the second classification sequence includes the second classification result and the second confidence level of the second classification result for each frame; the identification module 302 is further configured to, for the same frame in the first classification sequence and the second classification sequence, if the first classification result and the second classification result of the frame are consistent, determine the first classification result or the second classification result as the target classification result, and determine the target confidence level of the target classification result based on the first confidence level and the second confidence level;

[0180] If the first classification result and the second classification result of the frame are inconsistent, the maximum value between the first confidence and the second confidence is determined as the target confidence, and the classification result to which the target confidence belongs is determined as the target classification result;

[0181] The identification results are obtained by weighting the classification results of each target and the target confidence scores of the classification results.

[0182] The fake video recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0183] Figure 4 A schematic diagram of the structure of an electronic device provided in this application, such as... Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0184] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0185] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0186] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0187] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0188] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0189] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0190] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0191] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0192] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0193] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0194] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0196] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0198] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for identifying forged videos, characterized in that, include: The video to be identified is acquired, and the multimodal features and key point features of the video to be identified are determined. The multimodal features represent the fusion features related to facial texture, audio-visual synchronization, and physiological signals in each frame of the video to be identified, and the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames. The identification result is determined based on the multimodal features and the key point features; wherein the identification result is used to evaluate the probability that the video to be identified is a fake video.

2. The method according to claim 1, characterized in that, Determining the multimodal features of the video to be identified includes: The facial texture features, audio-visual synchronization features, and biometric features of the video to be identified are determined; wherein, the facial texture features represent the image texture of the facial region, the audio-visual synchronization features represent the synchronization between audio and mouth movements, and the biometric features represent the matching between heart rate and facial expression; The multimodal features of the video to be identified are determined based on the facial texture features, the audio-visual synchronization features, and the biometric features.

3. The method according to claim 2, characterized in that, Determining the facial texture features, audio-visual synchronization features, and biometric features of the video to be identified includes: The video to be identified is input into a preset first network layer, a second network layer, and a third network layer, respectively, to obtain the facial texture features output by the first network layer, the audio-visual synchronization features output by the second network layer, and the biometric features output by the third network layer; wherein, the first network layer includes a convolutional neural network, the second network layer includes a preset dynamic time warping algorithm, and the third network layer includes a remote photoplethysmography algorithm.

4. The method according to claim 2, characterized in that, Based on the facial texture features, the audio-visual synchronization features, and the biometric features, the multimodal features of the video to be identified are determined, including: Based on the facial texture features, the audio-visual synchronization features, and the biometric features, and using a preset long short-term memory network, a first weight corresponding to the facial texture features, a second weight corresponding to the audio-visual synchronization features, and a third weight corresponding to the biometric features are determined. The facial texture features and the first weight, the audio-visual synchronization features and the second weight, and the biometric features and the third weight are weighted to obtain the multimodal features of the video to be identified.

5. The method according to any one of claims 1-4, characterized in that, Based on the multimodal features and the key point features, the recognition result is determined, including: Based on the multimodal features, temporal features are determined, and based on the keypoint features, motion anomaly features are determined; wherein, the temporal features characterize the dynamic pattern of the multimodal features in the time dimension, and the motion anomaly features characterize the non-coherent behavior of the keypoint features between adjacent frames. The identification result is determined based on the temporal features and the motion anomaly features.

6. The method according to claim 5, characterized in that, Based on the multimodal features, temporal features are determined, including: The multimodal features are arranged in chronological order according to the time sequence of each frame to obtain a feature sequence; The feature sequence is input into a preset temporal convolutional network to obtain the output temporal features.

7. The method according to claim 5, characterized in that, Based on the key point features, motion anomaly features are determined, including: Based on the key point features, optical flow field information between adjacent segments of the video to be identified and the corresponding residual value of the optical flow field information are determined; wherein, the optical flow field information represents the vector field of the movement of key facial parts in adjacent segments between adjacent frames, and the number of frames in the adjacent segments is the same; If the residual value corresponding to the optical flow field information is greater than or equal to a preset residual threshold, then the adjacent segment to which the optical flow field information belongs is determined as the motion anomaly feature.

8. The method according to claim 5, characterized in that, Based on the temporal features and the motion anomaly features, the identification result is determined, including: The temporal features and the motion anomaly features are fused together to obtain the features to be classified. The features to be classified are input into a preset classifier to obtain the output recognition result.

9. The method according to claim 8, characterized in that, The preset classifier includes a support vector machine module and a multilayer perceptron module; the features to be classified are input into the preset classifier to obtain the output recognition result, including: The features to be classified are input into the support vector machine module to obtain the classification results of each frame of the video to be identified and the confidence scores of the classification results, which constitute the first classification sequence. Furthermore, the features to be classified are input into the multilayer perceptron module to obtain the classification results of each frame of the video to be identified and the confidence level of the classification results, which constitutes the second classification sequence; wherein, the classification results are either fake results or real results, and the confidence level represents the degree of certainty of the support vector machine module and the multilayer perceptron module in judging the classification results; The identification result is determined based on the first classification sequence and the second classification sequence.

10. The method according to claim 9, characterized in that, The first classification sequence includes the first classification result of each frame and the first confidence level of the first classification result; the second classification sequence includes the second classification result of each frame and the second confidence level of the second classification result. Determining the identification result based on the first classification sequence and the second classification sequence includes: For the same frame in the first classification sequence and the second classification sequence, if the first classification result and the second classification result of the frame are consistent, then the first classification result or the second classification result is determined as the target classification result, and the target confidence of the target classification result is determined according to the first confidence and the second confidence. If the first classification result and the second classification result of the frame are inconsistent, the maximum value between the first confidence and the second confidence is determined as the target confidence, and the classification result to which the target confidence belongs is determined as the target classification result; The identification result is obtained by weighting the target classification results and the target confidence scores of the target classification results.

11. A device for identifying forged videos, characterized in that, include: The determination module is used to acquire the video to be identified and determine the multimodal features and key point features of the video to be identified; wherein, the multimodal features represent the fusion features related to facial texture, audio-visual synchronization and physiological signals in each frame of the video to be identified, and the key point features represent the spatial position changes of key facial parts in the video to be identified in consecutive frames. The identification module is used to determine the identification result based on the multimodal features and the key point features; wherein the identification result is used to evaluate the probability that the video to be identified is a fake video.

12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

Images