Deepfake content detection method and electronic apparatus

Abstract

A deepfake content detection method and electronic apparatus are provided. The method includes the following steps. A video including a first image frame is obtained. A deep learning model is used to identify the authenticity of the first image frame to obtain an authenticity evaluation value output by the deep learning model. A plurality of facial feature points of the first image frame are identified to determine at least one facial feature information according to the plurality of facial feature points. Based on the authenticity evaluation value and at least one facial feature information, the video is determined to be real content or fake content.

Description

[Technical Field]

[0001] This invention relates to an artificial intelligence technology, and more particularly to an electronic device and a method for detecting deepfake content. [Previous Technology]

[0002] With the rapid development of AI technology, AI-based deception has become increasingly sophisticated and widespread. The rise of deepfake technology has brought significant challenges to media, security, and social networks. Deepfake technology can generate realistic fake videos, images, and sounds through AI, making false information more deceptive and posing a potential threat to security. Current deepfake detection technologies mainly rely on feature analysis of images and videos, such as optical flow analysis and image statistical feature analysis. However, with the advancement of technologies such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), the realism of deepfake content has been greatly improved, making it difficult for traditional methods based on static images or simple dynamic analysis to accurately distinguish between real and fake content. In addition, existing technologies cannot effectively handle subtle changes in micro-expressions and natural movements, and are also difficult to deal with complex deepfake content (such as those combining face replacement, voice forgery, and background synthesis). Furthermore, current deepfake detection technologies utilizing AI largely rely on the powerful computing capabilities of cloud resources. Users need to upload videos or images to servers for analysis, which may lead to privacy and data security issues, especially for the detection of sensitive or personal content. Currently, existing deepfake detection technologies still have significant shortcomings in terms of accuracy, privacy protection, computing resource requirements, and adaptability. [Summary of the Invention]

[0003] In view of the above, the present invention proposes a method and electronic device for detecting deepfake content, which can solve the above-mentioned technical problems.

[0004] This invention provides a method for detecting deepfake content, comprising the following steps: Acquiring a video including a first image frame; Using a deep learning model to identify the authenticity of the first image frame and obtaining an authenticity evaluation value output by the deep learning model; Identifying multiple facial feature points of the first image frame to determine at least one facial feature based on the multiple facial feature points; Determining whether the video is real content or fake content based on the authenticity evaluation value and the at least one facial feature information.

[0005] This embodiment of the invention provides an electronic device including a storage device and a processor. The storage device records a plurality of instructions. The processor is coupled to the storage device and configured to execute the aforementioned instructions to perform the following operations: Acquire a video including a first image frame. Use a deep learning model to identify the authenticity of the first image frame and obtain an authenticity evaluation value output by the deep learning model. Identify a plurality of facial feature points of the first image frame to determine at least one facial feature based on the plurality of facial feature points. Determine whether the video is real content or fake content based on the authenticity evaluation value and the at least one facial feature information.

[0006] Based on the above, in the embodiments of the present invention, a trained deep learning model can be used to identify the authenticity of the first image frame, thereby obtaining the authenticity evaluation value of the first image frame output by the deep learning model. Furthermore, by identifying multiple facial feature points in the first image frame, facial feature information of the face appearing in the first image frame can be obtained. By simultaneously referencing the authenticity evaluation value predicted by the deep learning model and the facial feature information, it is possible to more accurately determine whether the video is real or fake content. Therefore, the accuracy of identifying deepfake content can be significantly improved.

Implementation Method

[0007] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered to be the same or similar components when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of the methods and apparatus within the scope of the present invention's patent application.

[0008] FIG1 is a block diagram of an electronic device according to an embodiment of the present invention. Referring to FIG1, the electronic device 100 includes a display 110, a storage device 120, a transceiver 130, a transceiver 140, and a processor 150. The electronic device 100 may be, for example, a smartphone, a tablet computer, a laptop computer, or an all-in-one computer, etc., and the present invention is not limited thereto.

[0009] The display 110 provides image display functionality, such as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or other types of displays, which are not limited by the present invention. In some embodiments, the display 110 may display video.

[0010] Storage device 120 is used to temporarily or permanently store data, such as images, instructions, program code, software modules, etc. Specifically, storage device 120 may include volatile storage circuitry. Volatile storage circuitry is used to store data in a volatile manner. For example, volatile storage circuitry may include random access memory (RAM) or similar volatile storage media. Alternatively, storage device 120 may include non-volatile storage circuitry. Non-volatile storage circuitry is used to store data in a non-volatile manner. For example, non-volatile storage circuitry may include read-only memory (ROM), solid-state drive (SSD), and / or traditional hard disk drive (HDD) or similar non-volatile storage media. The number of storage devices 120 may be one or more, and the present invention is not limited thereto.

[0011] Transceiver 130 can transmit and receive signals wirelessly or via a wired connection. The transceiver can also perform operations such as low-noise amplification, impedance matching, mixing, up- or down-frequency conversion, filtering, amplification, and similar operations. Electronic device 100 can receive and transmit data via transceiver 130, such as receiving streaming frames of a video stream. In some embodiments, electronic device 100 may also include an antenna (not shown) for receiving radio frequency signals.

[0012] The speaker 140 provides sound playback functionality. The speaker 140 converts electronic signals into sound, which is emitted through the vibration of the diaphragm of a speaker unit. The speaker 140 can be a dynamic speaker, a flat panel speaker, or other types of speaker; this invention is not limited thereto. In some embodiments, the display 110 can play audio data from a video.

[0013] Processor 150 is coupled to display 110, storage device 120, transceiver 130, and speaker 140, and may be a general-purpose processor, special-purpose processor, conventional processor, digital signal processor, multiple microprocessors, one or more microprocessors incorporating digital signal processor cores, controller, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), any other type of integrated circuit, state machine, processor based on Advanced Reduced Instruction Set Machine (ARM), and the like. Processor 150 may access and execute instructions or program code recorded in storage device 120 to implement the deepfake content detection method in embodiments of the present invention.

[0014] In some embodiments, processor 150 may include a Neural Network Processing Unit (NPU). A neural network processor is a hardware accelerator specifically designed to handle artificial intelligence (AI) workloads, performing neural network computations. The network processor can accelerate inference operations of machine learning and deep learning models, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformers, etc. Network processor 150 has a large number of low-power processing cores, thus enabling highly efficient and low-power neural network computations. In some embodiments, the neural network processor can handle the inference and training tasks of the neural network.

[0015] FIG2 is a flowchart of a deepfake content detection method according to an embodiment of the present invention, and the method flow of FIG2 can be implemented by the various components of the electronic device 100 of FIG1. ​​Please refer to FIG1 and FIG2 simultaneously. The steps of the deepfake content detection method of this embodiment will be described below with reference to the various components of the electronic device 100 in FIG1.

[0016] In step S210, the processor 150 acquires a video including the first image frame. In different embodiments, this video may be a video stored locally on the electronic device 100 or a video stored on a cloud server. The video consists of a series of static multiple image frames, and the first image frame is one of the aforementioned multiple image frames. The frame rate (FPS) of the video may be, for example, 24 FPS, 30 FPS, or 60 FPS, etc. It should be emphasized that the first image frame in this disclosure may represent any frame of the video, and is not used to represent the first frame of the video.

[0017] In some embodiments, the processor 150 may execute an application and acquire a video played by the application. The application may be, for example, a browser, a social media application, or a video conferencing application, etc., that can receive and play video streams. When the application plays a video, the processor 150 may acquire multiple image frames of the video through screenshots, application programming interface (API) interfaces, or the operating system's video sharing mechanism.

[0018] In step S220, the processor 150 uses a deep learning model to identify the authenticity of the first image frame and obtains an authenticity evaluation value output by the deep learning model. Further, the processor 150 may use a trained deep learning model to identify the authenticity of the first image frame. In some embodiments, the deep learning model may output a classification result, which may include multiple classification categories and corresponding classification probabilities. The classification categories may include a real content category and a fake content category. The authenticity evaluation value may be the classification probability corresponding to a certain classification category, such as the classification probability corresponding to the fake content category.

[0019] In some embodiments, the deep learning model may include a feature extraction network, a recursive neural network, and a classification neural network sequentially connected. The feature extraction network is used to extract image features and may use a convolutional neural network (CNN) architecture (e.g., ResNeXt-50) to extract spatial features of the image to generate image feature vectors for subsequent analysis. The recursive neural network is used to analyze the temporal series changes of the image feature vectors to learn the temporal correlations across frames, for example, by using a Long Short-Term Memory (LSTM) network to capture the change patterns between consecutive image frames. The classification neural network is used to classify the output of the recursive neural network, for example, by using a fully connected layer (FC) plus a softmax layer to map the final output of the recursive neural network to different classification categories.

[0020] Specifically, FIG3 is a schematic diagram of a deep learning model identifying the authenticity of a first image frame according to an embodiment of the present invention. Referring to FIG3, the processor 130 can acquire a video v30. The video v30 includes a series of consecutive image frames (e.g., image frame F31). The processor 130 can repeat a similar process for each sample frame in the video v30 to predict the authenticity evaluation value of each sample frame. However, for clarity, the following description will use image frame F31 (i.e., the first image frame) as an example.

[0021] First, the processor 130 may perform image preprocessing on image frame F31 (i.e., the first image frame) to obtain a standardized face image Fs31. In some embodiments, the processor 130 may perform face detection on image frame F31 (i.e., the first image frame) to extract face image blocks from image frame F31. The processor 130 may perform pixel normalization processing. The processor 130 may adjust the image size of image frame F31 or the extracted face image blocks to generate a standardized face image Fs31 that conforms to the model input size.

[0022] Next, the processor 130 can input the standardized facial image Fs31 into the deep learning model M30. The deep learning model M30 includes a feature extraction network M31, a recursive neural network M32 and a classification neural network M33 connected in sequence.

[0023] In the example of Figure 3, the processor 130 can input a standardized facial image Fs31 into the feature extraction network M31 of the deep learning model M30 to obtain an image feature vector Fv31 output by the feature extraction network M31. The feature extraction network M31 may include multiple convolutional layers (e.g., convolutional layer CL1 and convolutional layer CLL), and the feature extraction network M31 may be, for example, the first few convolutional layers of the ResNeXt-50 model. The processor 130 can feed the standardized facial image Fs31 into the convolutional layer CL1 of the feature extraction network M31, and after multiple convolutional operations, the final convolutional layer CLL outputs the image feature vector Fv31. For example, the image feature vector Fv31 may be a 2048-dimensional feature vector.

[0024] Next, the processor 130 feeds the image feature vector Fv31 of image frame F31 into the recursive neural network M32 of the deep learning model M30, and obtains a state output Fv32 from the recursive neural network M32. The recursive neural network M32 is, for example, a Long Short-Term Memory (LSTM) model. The recursive neural network M32 is responsible for processing the image feature vector Fv31 from the feature extraction network M31, and further learning the temporal information in the image frame sequence. The LSTM model has long short-term memory capabilities, can accumulate and analyze change patterns across image frames, and adjust its internal state through forget gates, input gates, and output gates to ensure that key information is retained. The state output Fv32 of the LSTM model contains temporal dependent features of the entire image sequence, and can capture subtle changes between consecutive image frames, such as mouth movements, blink frequency changes, or other possible abnormal temporal features in video v30. By leveraging the temporal analysis capabilities of LSTM, the deep learning model M30 can integrate information from multiple frames of video, enhancing the accuracy of video authenticity assessment.

[0025] Subsequently, the processor 130 feeds the state output Fv32 into the classification neural network M33 to output a realism evaluation value for the image frame F31. Specifically, the classification neural network M33 may include a fully connected layer 331, an average pooling layer 332, and a softmax layer 333. The fully connected layer 331 receives the state output Fv32 from the recursive neural network M32 and converts it into a low-dimensional feature representation to accommodate classification computation. For example, if the state output Fv32 is a 2048-dimensional feature vector, the fully connected layer can map it to a smaller dimension (such as 256 or 128 dimensions) to reduce computational cost and emphasize key features.

[0026] The average pooling layer 332 performs feature compression on the output of the fully connected layer 331, extracting the main features and reducing redundant information through the average pooling operation. The Softmax layer 333 is responsible for converting the output of the previous layer into a probability distribution, ensuring that the sum of the probabilities of all categories is 1. For example, if the classification task is binary classification (i.e., real content and fake content), the Softmax layer outputs two classification probabilities, corresponding to "real content" and "fake content" respectively.

[0027] In some embodiments, the authenticity evaluation value of image frame F31 may be the classification probability corresponding to the classification category "fake content". When the authenticity evaluation value of image frame F31 is higher than a threshold value (e.g., 0.5), it means that the deep learning model M30 identifies image frame F31 as fake content. When the authenticity evaluation value of image frame F31 is lower than a threshold value (e.g., 0.5), it means that the deep learning model M30 identifies image frame F31 as real content.

[0028] It should be noted that the training data set for the deep learning model M30 can be collected from multiple publicly available deepfake datasets, such as the Deepfake Detection Challenge (DFDC) dataset or the Celeb-DF dataset, etc. The processor 150 can train the deep learning model based on the deepfake videos in the training data set and their corresponding real classification labels, thereby establishing the trained deep learning model M30.

[0029] Returning to Figure 2, in step S230, the processor 150 identifies multiple facial feature points in the first image frame to determine at least one facial feature based on the multiple facial feature points. The facial feature information may include mouth feature information, eye feature information, and other facial feature information. Mouth feature information may include mouth size, mouth movements, mouth width-to-height ratio, lip synchronization information, or opening frequency, etc. Eye feature information may include eye size, eye movements, blinking information, or eye width-to-height ratio, etc.

[0030] For example, FIG4 is a schematic diagram of multiple facial landmarks according to an embodiment of the present invention. Referring to FIG4, the processor 150 can perform facial landmark detection on a first image frame or its standardized face image F41 to obtain multiple facial landmarks. These facial landmarks may include multiple right eye landmarks Fp41, multiple left eye landmarks Fp42, and multiple mouth landmarks Fp43.

[0031] In some embodiments, the processor 150 may determine numerical eye feature information based on the right eye feature point Fp41. The processor 150 may determine other numerical eye feature information based on the left eye feature point Fp42. The processor 150 may determine numerical mouth feature information based on the mouth feature point Fp43.

[0032] In step S240, the processor 150 determines whether the video is real or fake based on the authenticity evaluation value and information from at least one facial feature. In other words, the processor 150 can comprehensively consider the recognition results of the reference deep learning model and facial feature information to determine whether the video is fake content generated by the deepfake model.

[0033] In some embodiments, the processor 150 may determine whether a video is real or fake content by performing a lookup table based on a realism assessment value and at least one facial feature information. In some embodiments, the processor 150 may compare the realism assessment value with a first threshold value and compare the facial feature information with a second threshold value, and determine whether the video is real or fake content by voting based on the comparison results. For example, when the realism assessment value and multiple facial feature information indicate that the video is fake content, the processor 150 may determine that the video is fake content. When the realism assessment value and the first facial feature information indicate that the video is fake content, but the second facial feature information indicates that the video is real content, the processor 150 may determine that the video is fake content. When the realism assessment value and multiple facial feature information indicate that the video is real content, the processor 150 may determine that the video is real content. Alternatively, in some embodiments, the processor 150 may use a trained classification model to determine whether the video is real or fake content based on a realism assessment value and at least one facial feature.

[0034] FIG5 is a flowchart of a deepfake content detection method according to an embodiment of the present invention, and the method flowchart in FIG5 can be implemented by the various components of the electronic device 100 in FIG1. ​​Please refer to FIG1 and FIG5 at the same time. The steps of the deepfake content detection method of this embodiment will be described below with reference to the various components of the electronic device 100 in FIG1.

[0035] In step S510, the processor 150 acquires a video including the first image frame. In step S520, the processor 150 uses a deep learning model to identify the authenticity of the first image frame and obtains the authenticity evaluation value output by the deep learning model. Detailed implementation details of the above steps can be found in the foregoing embodiments and will not be repeated here.

[0036] In step S530, the processor 150 identifies multiple facial feature points of the first image frame to determine at least one facial feature based on the multiple facial feature points. In this embodiment, step S530 can be implemented as steps S531 to S534.

[0037] In step S531, the processor 150 identifies multiple eye feature points in the first image frame. In step S532, the processor 150 determines eye feature information from at least one set of facial feature information based on the eye feature points. The eye feature information includes the eye aspect ratio (EAR) or blink information. The processor 150 can obtain the width and height of the eye based on the eye feature points and further calculate the eye aspect ratio. Based on the eye aspect ratio of the left eye, the processor 150 can determine the opening and closing state of the left eye, i.e., the closed eye state or the open eye state. When the eye aspect ratio of the left eye is less than a threshold value, the processor 150 can determine that the left eye is in a closed eye state. When the eye aspect ratio of the left eye is greater than a threshold value, the processor 150 can determine that the left eye is in an open eye state. Based on the opening and closing states of the left and right eyes in multiple consecutive image frames, the processor 150 can determine blink information such as the number of blinks, blink frequency, or blink interval. The processor 150 can determine the width-to-height ratio or blink information of the left eye based on the left eye feature points of the first image frame, and determine the width-to-height ratio or blink information of the right eye based on the right eye feature points of the first image frame.

[0038] In step S533, the processor 150 identifies multiple mouth feature points in the first image frame. In step S534, the processor 150 determines at least one mouth feature information from the facial feature information based on the mouth feature points. The mouth feature information includes the mouth aspect ratio (MAR) or lip synchronization information. The processor 150 can obtain the width and height of the mouth based on the mouth feature points and further calculate the mouth aspect ratio. Based on the mouth aspect ratio, the processor 150 can determine the opening and closing state of the mouth, that is, the closed state or the open state. When the mouth aspect ratio is less than a threshold value, the processor 150 can determine that the mouth is in a closed state. When the mouth aspect ratio is greater than a threshold value, the processor 150 can determine that the mouth is in an open state. The processor 150 can determine whether the lips move with speech based on the changes in the mouth aspect ratio in multiple consecutive image frames, thereby obtaining lip synchronization information. Theoretically, when the mouth is in an open state, the volume of human speech should be high. Conversely, when the mouth is closed, the volume of human speech should be low. Lip synchronization information can be the number of times or the frequency at which the opening and closing of the mouth matches the volume of human speech at the same point in time.

[0039] In step S540, the processor 150 determines whether the video is real or fake content based on the authenticity assessment value and at least one facial feature information. In this embodiment, step S540 can be implemented as steps S541 to S542.

[0040] In step S541, the processor 150 inputs the authenticity assessment value and information about at least one facial feature into a classification model. The classification model can be a Support Vector Machine (SVM) model, a neural network model, a random forest model, or other machine learning models suitable for classification tasks. The processor 150 can integrate the authenticity assessment value and numerical information about at least one facial feature into an input feature vector, and input this input feature vector into the classification model so that the classification model can determine the authenticity of the video.

[0041] In step S542, the processor 150 determines whether the video is real or fake content based on the classification result of the classification model. Furthermore, the classification model can classify the input feature vector as real or fake content using different probability thresholds or decision rules. It should be noted that the training of the classification model used in step S541 and the deep learning model used in step S520 are independent of each other. The processor 150 can train the classification model using the output values ​​of the trained deep learning model after completing its own deep learning model training.

[0042] Specifically, Figure 6 is a schematic diagram of detecting deepfake content based on a realism assessment value and facial feature information according to an embodiment of the present invention. Referring to Figure 6, the processor 150 can input a standardized facial image F61 of a certain image frame into a deep learning model M61 to obtain a realism assessment value E1. Furthermore, the processor 150 can perform eye feature extraction 611 and mouth feature extraction 612 on the standardized facial image F61 to obtain eye feature information EF6 and mouth feature information MF6. Then, the processor 150 can combine the realism assessment value E1, eye feature information EF6, and mouth feature information MF6 into an input feature vector including multiple numerical elements, and input the realism assessment value E1, eye feature information EF6, and mouth feature information MF6 into a classification model M62. In this example, the classification model M62 can output a classification result R61, which includes the classification probabilities of multiple classification categories. The processor 150 can determine whether the video is real content or fake content based on these classification probabilities.

[0043] It should be noted that through the training and application of the classification model M62, it is not necessary to set threshold values ​​for the diverse realism evaluation values ​​E1, eye feature information EF6 and mouth feature information MF6 respectively for judgment. Therefore, it is possible to avoid the situation of inaccurate judgment due to improper threshold setting.

[0044] In step S550, in response to the determination that the video is fake content, the processor 150 provides a warning. In some embodiments, when the processor 150 determines that the video is fake content, the processor 150 may provide a warning to the viewer through the display 110 to prevent the viewer from being deceived by false information. The warning may be a prompt window or a notification message, etc.

[0045] FIG7 is a flowchart of a deepfake content detection method according to an embodiment of the present invention, and the method flow of FIG7 can be implemented by the various components of the electronic device 100 of FIG1. ​​Please refer to FIG1 and FIG7 simultaneously. The steps of the deepfake content detection method of this embodiment will be described below with reference to the various components of the electronic device 100 in FIG1.

[0046] In step S710, processor 150 acquires a video including the first image frame. In step S720, processor 150 uses a deep learning model to identify the authenticity of the first image frame and obtains an authenticity evaluation value output by the deep learning model. In step S730, processor 150 identifies multiple facial feature points of the first image frame to determine at least one facial feature based on the multiple facial feature points. In step S740, processor 150 determines whether the video is real content or fake content based on the authenticity evaluation value and the at least one facial feature information.

[0047] In step S750, the processor 150 extracts multiple audio features from the audio data of the video. Furthermore, AI-generated sounds often have overly uniform pitch and lack the natural variations found in human voices. AI-generated sounds may have unnatural rhythms or smoothness, which can be identified through temporal feature analysis. Therefore, in this embodiment, the processor 150 can further identify whether the video is multimedia content generated by a deepfake model based on the video's audio data. The processor 150 can analyze the frequency characteristics, temporal characteristics, and statistical characteristics of the video's audio data to determine the video's authenticity based on these audio features.

[0048] In some embodiments, audio features include Mel-frequency cepstral coefficients (MFCC), spectral contrast, formant distribution, and pitch features. Spectral contrast is the contrast between spectral peaks and valleys. Formant distribution is the frequency distribution of formants in the audio data. Pitch features refer to the fundamental frequency of the sound, that is, the most basic frequency component of the sound wave.

[0049] In terms of Mel-frequency cepstral coefficients, deepfake content is smoother, has a consistent pattern, and exhibits fewer anomalous changes compared to genuine content. In terms of spectral contrast, deepfake content has lower contrast compared to genuine content. In terms of formant distribution, deepfake content exhibits inconsistent formant transitions and unnatural transitions between phonemes compared to genuine content. In terms of pitch characteristics, deepfake content exhibits monotonous or repetitive pitches and a lack of subtle variations compared to genuine content.

[0050] In step S760, the processor 150 combines multiple audio features into an integrated audio feature. The processor 150 can combine multiple numerically quantified audio features into an integrated audio feature. For example, the processor 150 can combine 13 MFCC coefficients, spectral contrast, the frequencies of 3 formants, and 1 pitch feature into an integrated audio feature vector including 18 feature values.

[0051] In step S770, the processor 150 uses a classification model to determine whether the video is real or fake content based on the integrated audio features. Furthermore, this classification model can classify the integrated audio features as real or fake content using different probability thresholds or decision rules. The aforementioned classification model can be a support vector machine (SVM) model, a neural network model, a random forest model, or other machine learning models applicable to classification tasks.

[0052] Specifically, Figure 8 is a schematic diagram of detecting deepfake content based on audio features according to an embodiment of the present invention. Referring to Figure 8, the processor 150 can perform MFCC extraction 811, spectrum comparison extraction 812, formant extraction 813, and pitch extraction 814 on the audio data Au81 of the video, thereby obtaining MFCC coefficients AF1, spectrum comparison AF2, formant distribution AF3, and pitch features AF4. The MFCC coefficients AF1, spectrum comparison AF2, formant distribution AF3, and pitch features AF4 will be integrated and input into the classification model M81. In this example, the classification model M81 can output a classification result R81, which includes the classification probability of multiple classification categories. The processor 150 can determine whether the video is real content or fake content based on these classification probabilities.

[0053] Finally, in step S780, in response to the determination that the video is fake content, the processor 150 provides a warning. For example, Figures 9A and 9B are schematic diagrams of providing a warning according to an embodiment of the present invention.

[0054] Referring to Figure 9A, when the electronic device 100 executes a social media application and this social media application plays video V91, the processor 150 can instantly determine whether video V91 is real or fake content based on the continuous video frames and audio data of video V91. When it is determined that video V91 is fake content, the processor 150 can display a notification 911 on the user interface U91 via the display 110. In this way, the user can be immediately alerted and is less likely to be deceived by false information.

[0055] Referring to Figure 9B, when a user uses electronic device 100 to watch a video V92 shared by another person, display 110 shows a window W91 for playing the video V92. Processor 150 can instantly determine whether the video V92 is real or fake based on the continuous video frames and audio data of the video V92. When the video V92 is determined to be fake, processor 150 can display a notification window W92 on display 110 in window W91. In this way, the user can be immediately alerted and is less likely to be deceived by false information.

[0056] In summary, in this embodiment of the invention, by simultaneously referencing the authenticity assessment value predicted by the deep learning model and facial feature information, it is possible to more accurately determine whether a video is real or fake content. Based on this, the accuracy of identifying deepfake content can be significantly improved, effectively preventing the spread of misinformation. Furthermore, through the training and application of deep learning models and multiple classification models, it can be applied to various types of deepfake content, ensuring good detection capabilities even in diverse environments.

[0057] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0058] Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 2 is a flowchart of a deepfake content detection method according to an embodiment of the present invention. Figure 3 is a schematic diagram of a deep learning model identifying the authenticity of a first image frame according to an embodiment of the present invention. Figure 4 is a schematic diagram of multiple facial feature points according to an embodiment of the present invention. Figure 5 is a flowchart of a deepfake content detection method according to an embodiment of the present invention. Figure 6 is a schematic diagram of detecting deepfake content based on authenticity assessment value and facial feature information according to an embodiment of the present invention. Figure 7 is a flowchart of a deepfake content detection method according to an embodiment of the present invention. Figure 8 is a schematic diagram of detecting deepfake content based on audio features according to an embodiment of the present invention. Figures 9A and 9B are schematic diagrams of providing warnings according to an embodiment of the present invention.

Claims

1. A method for detecting deepfake content, comprising: The process involves: acquiring a video including a first image frame; using a deep learning model to identify the authenticity of the first image frame and obtaining an authenticity evaluation value output by the deep learning model; identifying multiple facial feature points of the first image frame to determine at least one facial feature based on the multiple facial feature points; and determining whether the video is real or fake content based on the authenticity evaluation value and the at least one facial feature information. The deep learning model includes a feature extraction network, a recursive neural network, and a classification neural network sequentially connected. The step of using the deep learning model to identify the authenticity of the first image frame and obtaining the authenticity evaluation value output by the deep learning model includes: performing image preprocessing on the first image frame to obtain a standardized facial image; inputting the standardized facial image into the feature extraction network of the deep learning model to obtain an image feature vector output by the feature extraction network. The image feature vector of the first image frame is fed into the recursive neural network of the deep learning model to obtain a state output of the recursive neural network; and the state output is fed into the classification neural network to output the authenticity evaluation value of the first image frame.

2. The deepfake content detection method as described in claim 1, wherein the step of determining whether the video is real content or fake content based on the authenticity assessment value and the information of at least one facial feature includes: The authenticity assessment value and the information of at least one facial feature are input into a classification model; And based on the classification results of the classification model, determine whether the video is real content or fake content.

3. The deepfake content detection method as described in claim 1, wherein the step of identifying the plurality of facial feature points of the first image frame to determine the at least one facial feature information based on the plurality of facial feature points includes: Identify the plurality of eye feature points in the first image frame; And determine eye feature information in the at least one facial feature information based on the eye feature points, wherein the eye feature information includes the eye width-to-height ratio or blink information.

4. The deepfake content detection method as described in claim 1, wherein the step of identifying the plurality of facial feature points of the first image frame to determine the at least one facial feature information based on the plurality of facial feature points includes: Identify the plurality of mouth feature points in the first image frame; And determine one mouth feature information from the at least one facial feature information based on the mouth feature points, wherein the mouth feature information includes the mouth width-to-height ratio or lip synchronization information.

5. The deepfake content detection method as described in claim 1 further includes: Extract multiple audio features from the audio data of the video; The multiple audio features are combined into an integrated audio feature; And using a classification model based on the integrated audio features, determine whether the video is real content or fake content.

6. The deepfake content detection method as described in claim 5, wherein the audio features include Mel frequency cepstral coefficients (MFCC), spectral contrast, formant distribution, and pitch features.

7. The deepfake content detection method as described in claim 1, wherein the step of acquiring the video including the first image frame comprises: Run an application and retrieve the video being played by the application.

8. The deepfake content detection method described in claim 1 further includes: In response to the determination that the video is fake content, a warning is issued.

9. An electronic device comprising: A storage device that records multiple instructions; The system also includes a processor coupled to the storage device, configured to: acquire a video including a first image frame; use a deep learning model to identify the authenticity of the first image frame and obtain an authenticity assessment value output by the deep learning model; identify multiple facial feature points of the first image frame to determine at least one facial feature based on the multiple facial feature points; and determine whether the video is real or fake content based on the authenticity assessment value and the at least one facial feature information, wherein the deep learning model includes a feature extraction network, a recursive neural network, and a classification neural network connected in sequence, and the processor is further configured to: perform an image preprocessing on the first image frame to obtain a standardized facial image; input the standardized facial image to the feature extraction network of the deep learning model to obtain an image feature vector output by the feature extraction network; The image feature vector of the first image frame is fed into the recursive neural network of the deep learning model to obtain a state output of the recursive neural network; and the state output is fed into the classification neural network to output the authenticity evaluation value of the first image frame.

Images