Real-time face swapping detection method, system, device and medium based on deep learning

Abstract

The application discloses a kind of real-time face changing detection method, system, equipment and medium based on deep learning, including real-time acquisition key frame image sequence in video stream;According to the preset deep learning model processing key frame image sequence, obtain the face changing probability score of each key frame image;According to the preset frequency domain analysis algorithm processing key frame image sequence, obtain the frequency domain anomaly score of each key frame image;According to the preset light flow analysis algorithm processing key frame image sequence, obtain the motion coherence score of each adjacent key frame image;According to the preset multi-modal fusion algorithm processing each face changing probability score, frequency domain anomaly score and motion coherence score, obtain the face changing detection result of key frame image sequence.The application improves the efficiency of real-time dynamic face changing detection by the fusion of a variety of processing algorithms.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a real-time face swapping detection method, system, device and medium based on deep learning. BACKGROUND

[0002] With the development of face swapping technology, malicious face swapping videos are increasingly used on social media and live streaming platforms. The misuse of these technologies has led to problems of personal privacy and information security. Therefore, an efficient and accurate face swapping detection method is needed to address the challenge of recognizing fake faces in live streaming.

[0003] However, some existing face swapping detection methods, such as machine learning algorithms, can be disturbed by factors such as facial expressions and poses, or have weak generalization ability due to insufficient training, which can lead to problems such as poor real-time performance, insufficient detection accuracy, and high false positive rate. These methods cannot effectively cope with the constantly advancing face swapping technology. SUMMARY

[0004] To address the deficiencies of the prior art, the present application provides a real-time face swapping detection method, system, device and medium based on deep learning, which improves the efficiency of real-time dynamic face swapping detection.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a real-time face swapping detection method based on deep learning, comprising:

[0006] real-time acquisition of a sequence of key frame images in a video stream;

[0007] processing the sequence of key frame images according to a preset deep learning model to obtain face swapping probability scores for each key frame image; wherein the deep learning model is trained using generated counterfeit data generated by a generative adversarial network;

[0008] processing the sequence of key frame images according to a preset frequency domain analysis algorithm to obtain frequency domain anomaly scores for each key frame image;

[0009] processing the sequence of key frame images according to a preset optical flow analysis algorithm to obtain motion coherence scores for each adjacent key frame image;

[0010] processing each of the face swapping probability scores, frequency domain anomaly scores and motion coherence scores according to a preset multi-modal fusion algorithm to obtain face swapping detection results for the sequence of key frame images; wherein the face swapping detection results include: no face swapping, malicious face swapping and non-malicious face swapping.

[0011] Compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of the data of the video stream can provide a real-time data source for subsequent processing, and meanwhile, extraction of the key frame image sequence can effectively reduce the calculation amount of subsequent processing and improve the processing efficiency; processing of each key frame image according to the deep learning model obtained by pre-training using the generative adversarial network can improve the image feature extraction and detection capability; processing of each key frame image according to the frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing of each key frame image according to the optical flow analysis algorithm can capture the dynamic changes of the face and detect the coherence and naturalness of the face motion; multi-modal fusion of the deep learning model recognition result, the frequency domain analysis result and the optical flow analysis result, in combination with multiple features and analysis methods, can output the face swapping detection result, and improve the accuracy and reliability of detection.

[0012] In some embodiments of the first aspect of the present application, the real-time acquisition of the key frame image sequence in the video stream comprises:

[0013] acquiring original video stream data;

[0014] processing the original video stream data according to a preset frame number control rule to obtain a down-sampling frame sequence;

[0015] processing the down-sampling frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0016] Compared with the prior art, the above embodiments have the following beneficial effects: acquisition of the original video stream data ensures the completeness and real-time nature of the data, down-sampling processing of the original video stream data according to the preset frame number control rule reduces the calculation load and improves the processing speed, screening of the key frame image sequence makes the data for subsequent processing representative, improves the detection precision and reliability, and reduces false positives and omissions.

[0017] In some embodiments of the first aspect of the present application, the processing of the down-sampling frame sequence according to the preset key frame detection algorithm to obtain a key frame image sequence comprises:

[0018] calculating the pixel difference value or the optical flow change value of adjacent frame images in the down-sampling frame sequence;

[0019] according to a preset first threshold range, regarding each adjacent frame image whose pixel difference value or optical flow change value meets the first threshold range as each key frame image;

[0020] cropping the face region image of each key frame image to obtain a key frame image sequence.

[0021] Compared with the prior art, the above-mentioned embodiment has the following beneficial effects: the frame with large change identified by the pixel difference or the light flow change value can contain more information; the threshold range is set to ensure that the screened frame is representative enough; the face region image of each key frame image is cropped to focus on the face region, reduce the interference of irrelevant background, reduce the calculation amount, and improve the detection accuracy at the same time.

[0022] In some embodiments of the first aspect of the application, the sequence of down-sampled frames is processed according to a preset key frame detection algorithm to obtain a sequence of key frame images, further comprising:

[0023] The comprehensive distance change value of each feature point of the face in the adjacent frame images in the sequence of down-sampled frames is calculated;

[0024] According to a preset second threshold range, each adjacent frame image with a comprehensive distance change value meeting the second threshold range is taken as a key frame;

[0025] The face region image of each key frame image is cropped to obtain a sequence of key frame images.

[0026] Compared with the prior art, the above-mentioned embodiment has the following beneficial effects: the comprehensive distance change value of the feature points is taken as the second selection standard of the key frame, which can more comprehensively screen the key frame, the second threshold range is set to ensure that the screened frame is representative in terms of feature point change, the face region image of each key frame image is cropped to focus on the face region, reduce the interference of irrelevant background, reduce the calculation amount, and improve the detection accuracy at the same time.

[0027] In some embodiments of the first aspect of the application, the sequence of key frame images is processed according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image, comprising:

[0028] According to the Fourier transform or wavelet transform algorithm, the frequency domain mode of each key frame image is extracted;

[0029] The frequency domain anomaly score of each key frame image is calculated by comparing the frequency domain mode of each key frame image with that of a normal face image.

[0030] Compared with the prior art, the above-mentioned embodiment has the following beneficial effects: frequency domain analysis can reveal the high-frequency and low-frequency components in the image, which is helpful for detecting the abnormal mode left by face swapping operation, and the detection accuracy and reliability are improved by comparing and quantifying the abnormal degree.

[0031] In some embodiments of the first aspect of the application, the sequence of key frame images is processed according to a preset light flow analysis algorithm to obtain a motion continuity score of each adjacent key frame image, comprising:

[0032] Calculate the displacement of each feature point of the face in each adjacent key frame;

[0033] According to each displacement, determine whether the motion trajectory of the face is abnormal, and obtain the motion coherence score of each adjacent key frame image.

[0034] Compared with the prior art, the above embodiment has the following beneficial effects: the optical flow analysis calculates the displacement and motion trajectory of each feature point of the face, captures the dynamic change of the face, detects whether the face motion is abnormal, and improves the detection accuracy.

[0035] In some embodiments of the first aspect of the application, the processing of each of the face swapping probability score, the frequency domain anomaly score, and the motion coherence score according to the preset multi-modal fusion algorithm to obtain the face swapping detection result of the key frame image sequence comprises:

[0036] According to a preset vector machine model or a random forest model, processing the face swapping probability score, the frequency domain anomaly score, and the motion coherence score of each key frame image to obtain the face swapping probability of each key frame image;

[0037] Weighted calculation of the face swapping probability of each key frame image to obtain the face swapping detection result of the key frame image sequence.

[0038] Compared with the prior art, the above embodiment has the following beneficial effects: the multi-modal fusion algorithm combines multiple features, and through weighted calculation, comprehensively considers various scores, and finally obtains an accurate face swapping detection result, improving the comprehensiveness and reliability of the detection.

[0039] In a second aspect, the embodiments of the application also provide a real-time face swapping detection system based on deep learning, comprising: a data acquisition module, a neural module, a frequency domain analysis module, an optical flow analysis module, and a fusion processing module;

[0040] The data acquisition module is configured to acquire a key frame image sequence in a video stream in real time.

[0041] The neural module is configured to process the key frame image sequence according to a preset deep learning model to obtain a face swapping probability score of each key frame image; wherein the deep learning model is obtained by training a generated countermeasure network using generated fake data.

[0042] The frequency domain analysis module is configured to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image.

[0043] The optical flow analysis module is configured to process the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image.

[0044] The fusion processing module is configured to process the face replacement probability score, the frequency domain anomaly score and the motion continuity score according to a preset multi-modal fusion algorithm to obtain a face replacement detection result of the key frame image sequence; wherein the face replacement detection result includes: no face replacement, malicious face replacement and non-malicious face replacement.

[0045] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the data of the video stream is acquired in real time, which can provide a real-time data source for subsequent processing, and meanwhile, the extraction of the key frame image sequence can effectively reduce the calculation amount of subsequent processing and improve the processing efficiency; the deep learning model obtained by using the generative adversarial network for pre-training is used to process each key frame image, which can improve the image feature extraction and detection capability; the frequency domain analysis algorithm is used to process each key frame image, which can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; the optical flow analysis algorithm is used to process each key frame image, which can capture the dynamic changes of the face and detect the continuity and naturalness of the face motion; the deep learning model recognition result, the frequency domain analysis result and the optical flow analysis result are multi-modal fused, a variety of features and analysis methods are combined, and the face replacement detection result is output, which improves the accuracy and reliability of detection.

[0046] In a third aspect, the embodiments of the present application also provide an electronic device for real-time face replacement detection based on deep learning, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program implements the steps of the real-time face replacement detection method based on deep learning when loaded into the processor.

[0047] In a fourth aspect, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the real-time face replacement detection method based on deep learning when executed by a processor. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 FIG. 1 is a flowchart of a real-time face replacement detection method based on deep learning provided in some embodiments of the present application.

[0049] Figure 2 FIG. 3 is a structural diagram of a real-time face replacement detection system based on deep learning provided in some embodiments of the present application.

[0050] Figure 3 FIG. 4 is a structural diagram of an electronic device for real-time face replacement detection based on deep learning provided in some embodiments of the present application. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0052] Embodiment one:

[0053] Please refer to Figure 1 A real-time face changing detection method based on deep learning provided by the embodiment of the present application includes steps S1 to S5.

[0054] Step S1: Real-time acquisition of key frame image sequences in a video stream.

[0055] In some embodiments of the present application, step S1 can be implemented through the following preferred implementation:

[0056] S11: Acquisition of original video stream data; wherein the original video stream data can be a local video or acquired from a network end in real time;

[0057] S12: Processing of the original video stream data according to a preset frame number control rule to obtain a down-sampling frame sequence;

[0058] The frame number control rule can be adjusted according to actual conditions. For example, if the frame number of a video stream acquired from a network end in real time is relatively high, such as a 60 frames per second video, in order to reduce the calculation load of the detection system, down-sampling can be performed to become 10 frames or 15 frames per second. If the detection scene is relatively complex, for example, the background color is relatively many or the number of characters is relatively many, the frame rate control can also be higher to ensure a relatively good balance between detection efficiency and detection accuracy. In actual application, in addition to frame control, GPU or other hardware acceleration capabilities can also be combined for accelerated detection. In actual testing, the present scheme realizes a detection delay of less than 100 milliseconds in a 60 frames per second high-definition live video, which is sufficient to meet the real-time requirements of most live scenes.

[0059] S13: Processing of the down-sampling frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0060] In the preferred implementation, the original video stream data is acquired to ensure the integrity and real-time of the data, the original video stream data is down-sampled according to the preset frame number control rule to reduce the calculation load and improve the processing speed, the key frame image sequence is screened out, the data for subsequent processing is representative, the detection precision and reliability are improved, and the false positives and false negatives are reduced.

[0061] Furthermore, the process of processing the downsampled frame sequence according to the preset keyframe detection algorithm can be implemented through the following preferred embodiments:

[0062] Calculate the pixel difference or optical flow change value between adjacent frames in the downsampled frame sequence;

[0063] According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value conforms to the first threshold range is taken as a key frame image.

[0064] By using pixel differences or optical flow variations, frames with significant information changes between adjacent frames can be identified. These frames with significant changes are more likely to contain more information, and therefore, they can be filtered out as data sources for subsequent processing.

[0065] The face region images of each keyframe image are cropped to obtain a keyframe image sequence. During the cropping process, a face detection algorithm can be used to identify the face region.

[0066] In this preferred embodiment, frames with significant variations are identified by pixel differences or optical flow changes, which can contain more information; a threshold range is set to ensure that the selected frames are sufficiently representative; face region images of each key frame are cropped to focus on the face region, reducing interference from irrelevant backgrounds, reducing computational load, and improving detection accuracy.

[0067] Furthermore, the process of processing the downsampled frame sequence according to the preset keyframe detection algorithm can also be implemented through the following preferred embodiments:

[0068] Calculate the combined distance change value of each facial feature point in adjacent frame images in the downsampled frame sequence;

[0069] According to the preset second threshold range, each adjacent frame image whose comprehensive distance change value conforms to the second threshold range is taken as a key frame;

[0070] When the overall information of the video frame does not change much, the focus can be on the face area. The comprehensive distance change value of the feature points on the face (such as eyes, mouth, and facial angles) can be used as the second selection criterion for key frames, which can more comprehensively filter key frames and reduce key frame omissions.

[0071] The face region images of each keyframe image are cropped to obtain a keyframe image sequence; similarly, a face recognition algorithm can be used to identify the face region during cropping.

[0072] In the preferred embodiment, the comprehensive distance change value of the feature points is used as the second selection criterion of the key frame, which can comprehensively screen the key frame, set a second threshold range, ensure that the screened frame is representative in terms of feature point change, crop the face region image of each key frame image, focus on the face region, reduce the interference of irrelevant background, reduce the calculation amount, and improve the detection accuracy.

[0073] Step S2: processing the key frame image sequence according to a preset deep learning model to obtain a face changing probability score of each key frame image; wherein the deep learning model is obtained by training using fake data generated by a generative adversarial network;

[0074] The deep learning model can be a convolutional neural network, a time series model, or other deep learning models. It can also be integrated with different algorithms to build a new model as a face detection model through ensemble learning. During training, the model can be trained using multiple image datasets containing different light, expression, angle, and background, or multiple trained using fake data generated by a generative adversarial network to improve the detection ability of the model.

[0075] In the embodiment, the deep learning model pre-trained using the generative adversarial network can improve the image feature extraction capability and enhance the detection accuracy and robustness.

[0076] Step S3: processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image.

[0077] Preferably, the frequency domain anomaly score can be obtained by the following preferred embodiment:

[0078] According to the Fourier transform or wavelet transform algorithm, the frequency domain mode of each key frame image is extracted;

[0079] The frequency domain mode of each key frame image is compared with that of a normal face image, and the frequency domain anomaly score of each key frame image is calculated.

[0080] In step S3, the frequency domain analysis algorithm can be other than the Fourier transform or wavelet transform algorithm, such as a discrete cosine transform or other similar algorithm. When extracting the frequency domain mode, the detection threshold is adaptively adjusted according to the dynamic characteristics of the video. For example, for a video with large light fluctuations, the sensitivity of the frequency domain analysis can be dynamically adjusted to enable the system to work stably in different environments. At the same time, when performing detection, a sliding window method can be used to continuously detect several frames, and the consistency of the detection results of the continuous frames is determined, such as scene consistency, to exclude accidental abnormalities in a single frame and ensure the time sequence consistency and continuity of the analysis of the expression, frequency domain, and motion characteristics.

[0081] In the preferred embodiment, the frequency domain analysis can reveal the high-frequency and low-frequency components in the image, which helps to detect abnormal patterns left by face swapping operations, and improves the accuracy and reliability of detection by comparing and quantifying the degree of abnormality.

[0082] Step S4: processing the sequence of key frame images according to a preset optical flow analysis algorithm to obtain a motion continuity score of each adjacent key frame image.

[0083] Preferably, the motion continuity score can be obtained by the following preferred implementation:

[0084] Calculate the displacement of each feature point of the face in each adjacent key frame.

[0085] According to the displacement, determine whether the motion trajectory of the face is abnormal to obtain the motion continuity score of each adjacent key frame image.

[0086] In step S4, the optical flow analysis algorithm is not limited and can be a local optical flow calculation method, a global optical flow calculation method, an optical flow algorithm based on polynomial expansion, or other deep learning optical flow estimation methods. When performing optical flow analysis, the detection threshold can be adaptively adjusted according to the dynamic characteristics of the video. For example, for a video with large light fluctuations, the sensitivity of the optical flow analysis can be dynamically adjusted to enable the system to work stably in different environments. At the same time, when performing detection, a sliding window method can be used to continuously detect several frames, and the consistency of the detection results of the continuous frames is determined, such as scene consistency, to exclude accidental abnormalities in a single frame and ensure the time sequence consistency and continuity of the analysis of the expression, frequency domain, and motion characteristics.

[0087] In the preferred embodiment, the optical flow analysis calculates the displacement and motion trajectory of each feature point of the face, captures the dynamic changes of the face, detects whether the facial motion is abnormal, and improves the accuracy of detection.

[0088] Step S5: processing the face replacement probability scores, the frequency domain anomaly scores and the motion coherence scores according to a preset multi-modal fusion algorithm to obtain a face replacement detection result of the key frame image sequence.

[0089] In some embodiments of the present application, step S5 can be implemented through the following preferred implementation:

[0090] According to a preset vector machine model or random forest model, processing the face replacement probability scores, the frequency domain anomaly scores and the motion coherence scores of each key frame image to obtain a face replacement probability of each key frame image.

[0091] Weighted calculation of the face replacement probability of each key frame image to obtain the face replacement detection result of the key frame image sequence.

[0092] In step S5, the multi-modal fusion algorithm can use other fusion models or rule-based methods to weight and fuse the detection results of static and dynamic features in addition to the vector machine model or random forest model. Further, after obtaining the detection result, it can be sent to a monitoring personnel or a client, or it can also be linked to an alarm system to provide real-time warning and reminder when the recognition result is face replacement.

[0093] In the preferred embodiment, the multi-modal fusion algorithm combines multiple features, and through weighted calculation, comprehensively considers various scores to finally obtain an accurate face replacement detection result, improving the comprehensiveness and reliability of the detection.

[0094] In addition, according to the real-time face replacement detection method based on deep learning provided in the embodiment, in some actual system application scenarios, such as in the evaluation on the test data sets FaceForensics++ and DFDC, the system realizes a detection accuracy of more than 90%, and the sensitivity (recall rate) of detecting fake facial features is also more than 85%; in the actual live test environment, the average detection delay of the system is 80 milliseconds, and the false positive rate is less than 5%, showing the practicability and high efficiency of the system in complex application scenarios.

[0095] In summary, compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of data of a video stream can provide a real-time data source for subsequent processing, and meanwhile, extraction of a key frame image sequence can effectively reduce the calculation amount of subsequent processing and improve processing efficiency; processing of each key frame image according to a deep learning model obtained by pre-training using a generative adversarial network can improve image feature extraction and detection capability; processing of each key frame image according to a frequency domain analysis algorithm can capture frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve detection sensitivity; processing of each key frame image according to an optical flow analysis algorithm can capture dynamic changes of the face and detect the coherence and naturalness of face movement; multi-modal fusion of deep learning model recognition results, frequency domain analysis results and optical flow analysis results, in combination with multiple features and analysis methods, can output face swapping detection results, and improve detection accuracy and reliability.

[0096] Embodiment two:

[0097] Please refer to Figure 2 Based on the same inventive concept, the embodiment of the present application discloses a real-time face swapping detection system based on deep learning, which comprises a data acquisition module M1, a neural module M2, a frequency domain analysis module M3, an optical flow analysis module M4 and a fusion processing module M5.

[0098] The data acquisition module M1 acquires a key frame image sequence in a video stream in real time.

[0099] The data acquisition module M1 comprises an original acquisition unit, a down-sampling unit and an extraction unit; the original acquisition unit is used to acquire original video stream data; the down-sampling unit processes the original video stream data according to a preset frame number control rule to obtain a down-sampled frame sequence; and the extraction unit is used to process the down-sampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0100] The data acquisition module M1 of the embodiment acquires original video stream data to ensure the completeness and real-time nature of the data, performs down-sampling processing on the original video stream data according to a preset frame number control rule to reduce calculation load and improve processing speed, screens out a key frame image sequence, so that the data for subsequent processing is representative, detection precision and reliability are improved, and false positives and false negatives are reduced.

[0101] As a preferred, the extraction unit is used to process the down-sampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence, comprising:

[0102] calculating pixel difference values or optical flow change values of adjacent frame images in the down-sampled frame sequence;

[0103] According to a preset first threshold range, each adjacent frame image with a pixel difference value or an optical flow change value meeting the first threshold range is taken as a key frame image.

[0104] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0105] In the preferred embodiment, the frame with a large change in pixel difference or optical flow change value can contain more information, the threshold range is set to ensure that the selected frame is representative, and the face region image of each key frame image is cropped to focus on the face region, reduce the interference of irrelevant background, reduce the calculation amount, and improve the detection accuracy.

[0106] Further, the extraction unit is configured to process the down-sampling frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence, and can further include:

[0107] The integrated distance change value of each feature point of the face in adjacent frame images in the down-sampling frame sequence is calculated.

[0108] According to a preset second threshold range, each adjacent frame image with an integrated distance change value meeting the second threshold range is taken as a key frame.

[0109] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0110] In the preferred embodiment, the integrated distance change value of the feature point is used as a second selection criterion for the key frame, which can more comprehensively select the key frame, the second threshold range is set to ensure that the selected frame is representative in terms of feature point change, the face region image of each key frame image is cropped to focus on the face region, reduce the interference of irrelevant background, reduce the calculation amount, and improve the detection accuracy.

[0111] The neural module M2 is configured to process the key frame image sequence according to a preset deep learning model to obtain a face swapping probability score of each key frame image, wherein the deep learning model is trained using fake data generated by a generative adversarial network.

[0112] The neural module M2 of the embodiment processes each key frame image according to the deep learning model pre-trained using the generative adversarial network, which can improve the image feature extraction capability and enhance the detection accuracy and robustness.

[0113] The frequency domain analysis module M3 is configured to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image.

[0114] The frequency domain analysis module M3 comprises a frequency domain extraction unit and a frequency domain calculation unit; the frequency domain extraction unit is configured to extract the frequency domain mode of each key frame image according to a Fourier transform or wavelet transform algorithm; and the frequency domain calculation unit is configured to compare the frequency domain mode of each key frame image with that of a normal human face image and calculate the frequency domain anomaly score of each key frame image.

[0115] The frequency domain analysis module M3 of this embodiment can reveal the high-frequency and low-frequency components in the image through frequency domain analysis, which helps to detect abnormal patterns left by face swapping operations, improves the accuracy and reliability of detection by comparing and quantifying the degree of abnormality.

[0116] The optical flow analysis module M4 is configured to process the sequence of key frame images according to a preset optical flow analysis algorithm to obtain the motion continuity score of each adjacent key frame image.

[0117] The optical flow analysis module M4 comprises a displacement calculation unit and a continuity calculation unit; the displacement calculation unit is configured to calculate the displacement of each feature point of a face in each adjacent key frame; and the continuity calculation unit is configured to determine whether the motion trajectory of the face is abnormal according to each displacement to obtain the motion continuity score of each adjacent key frame image.

[0118] The optical flow analysis module M4 of this embodiment can calculate the displacement and motion trajectory of each feature point of a face through optical flow analysis, capture the dynamic changes of the face, and detect whether the face motion is abnormal, thereby improving the accuracy of detection.

[0119] The fusion processing module M5 is configured to process the face swapping probability score, the frequency domain anomaly score and the motion continuity score of each key frame image according to a preset multi-modal fusion algorithm to obtain the face swapping detection result of the sequence of key frame images; wherein the face swapping detection result comprises no face swapping, malicious face swapping and non-malicious face swapping.

[0120] The fusion processing module M5 comprises a first probability calculation unit and a result output unit; the first probability calculation unit is configured to process the face swapping probability score, the frequency domain anomaly score and the motion continuity score of each key frame image according to a preset vector machine model or random forest model to obtain the face swapping probability of each key frame image; and the result output unit is configured to calculate the face swapping probability of each key frame image by weighting to obtain the face swapping detection result of the sequence of key frame images.

[0121] The fusion processing module M5 of this embodiment can obtain an accurate face swapping detection result by combining multiple features through a multi-modal fusion algorithm, comprehensively considering various scores through weighted calculation, thereby improving the comprehensiveness and reliability of detection.

[0122] In summary, compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of the data of the video stream can provide a real-time data source for subsequent processing, and meanwhile, extraction of the key frame image sequence can effectively reduce the calculation amount of subsequent processing and improve the processing efficiency; processing of each key frame image according to the deep learning model obtained by pre-training using the generative adversarial network can improve the image feature extraction and detection capability; processing of each key frame image according to the frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing of each key frame image according to the optical flow analysis algorithm can capture the dynamic changes of the face and detect the coherence and naturalness of the face motion; multi-modal fusion of the deep learning model recognition result, the frequency domain analysis result and the optical flow analysis result, in combination with multiple features and analysis methods, outputs the face swapping detection result, and improves the accuracy and reliability of detection.

[0123] The division of each module described above is only logical functional division, and there can be another division manner in actual implementation, for example, multiple modules can be combined or integrated into another system, for example, directly integrated into an existing video monitoring, security protection and content review system.

[0124] Embodiment three

[0125] Figure 3 A structural diagram of an electronic device for real-time face swapping detection based on deep learning is shown in the present application. As shown in Figure 3 The face swapping detection device can include a processor N1, a memory N2, a data interface N3 and a communication bus N4.

[0126] The processor N1, the memory N2 and the data interface N3 communicate with each other through the communication bus N4; the data interface N3 is used for data communication with other additional devices; the processor N1 is used for executing the program N5, and can specifically execute the related steps in the above-mentioned embodiment of the real-time face swapping detection method based on deep learning.

[0127] Specifically, the program N5 can include program codes, and the program codes include computer executable instructions.

[0128] The processor N1 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the real-time face swapping detection device based on deep learning can be the same type of processors, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

[0129] The memory N2 is for storing the program N5, and can comprise a high-speed RAM memory and / or a non-volatile memory such as at least one disk memory.

[0130] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Furthermore, embodiments of the present application are not described with reference to any particular programming language.

[0131] Embodiment Four

[0132] The embodiment of the present application further provides a computer readable storage medium, the storage medium stores at least one executable instruction, and the executable instruction enables the deep learning based real-time face changing detection device / system to perform the deep learning based real-time face changing detection method in any method embodiment described above when the deep learning based real-time face changing detection device / system runs.

[0133] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. Similarly, in order to simplify the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description of it. Among them, the claims following the detailed description are hereby expressly incorporated into the detailed description, wherein each claim itself is a separate embodiment of the present application.

[0134] Those skilled in the art can understand that the modules in the device in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and furthermore can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. A real-time face-swapping detection method based on deep learning, characterized in that, include: Real-time acquisition of keyframe image sequences from a video stream; The keyframe image sequence is processed according to a preset deep learning model to obtain a face-swapping probability score for each keyframe image; wherein, the deep learning model is trained using fake data generated by a generative adversarial network; The keyframe image sequence is processed according to a preset frequency domain analysis algorithm to obtain the frequency domain anomaly score of each keyframe image. The keyframe image sequence is processed according to a preset optical flow analysis algorithm to obtain the motion coherence score of each adjacent keyframe image. The face-swapping probability score, frequency domain anomaly score, and motion coherence score are processed according to a preset multimodal fusion algorithm to obtain the face-swapping detection result of the keyframe image sequence; wherein, the face-swapping detection result includes: no face-swapping, malicious face-swapping, and non-malicious face-swapping. The real-time acquisition of keyframe image sequences from the video stream includes: Obtain raw video stream data; According to the preset frame rate control rules, the original video stream data is processed to obtain a downsampled frame sequence; The downsampled frame sequence is processed according to a preset keyframe detection algorithm to obtain a keyframe image sequence; The step of processing the downsampled frame sequence according to a preset keyframe detection algorithm to obtain a keyframe image sequence includes: Calculate the combined distance change value of each facial feature point in adjacent frame images in the downsampled frame sequence; According to the preset second threshold range, each adjacent frame image whose comprehensive distance change value conforms to the second threshold range is taken as a key frame; The face region images of each keyframe image are cropped to obtain the keyframe image sequence.

2. The real-time face-swapping detection method based on deep learning as described in claim 1, characterized in that, The step of processing the downsampled frame sequence according to a preset keyframe detection algorithm to obtain a keyframe image sequence includes: Calculate the pixel difference or optical flow change value between adjacent frames in the downsampled frame sequence; According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value conforms to the first threshold range is taken as a key frame image. The face region images of each keyframe image are cropped to obtain the keyframe image sequence.

3. The real-time face-swapping detection method based on deep learning as described in claim 2, characterized in that, The step of processing the keyframe image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score for each keyframe image includes: Extract the frequency domain pattern of each keyframe image based on Fourier transform or wavelet transform algorithms; By comparing the frequency domain patterns of each keyframe image with those of a normal face image, a frequency domain anomaly score is calculated for each keyframe image.

4. The real-time face-swapping detection method based on deep learning as described in claim 3, characterized in that, The step of processing the keyframe image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score for each adjacent keyframe image includes: Calculate the displacement of each facial feature point in each adjacent keyframe; Based on the displacements described, it is determined whether the motion trajectory of the face is abnormal, and the motion coherence score of each adjacent keyframe image is obtained.

5. The real-time face-swapping detection method based on deep learning as described in claim 4, characterized in that, The step of processing the face-swapping probability score, frequency domain anomaly score, and motion coherence score according to a preset multimodal fusion algorithm to obtain the face-swapping detection result of the keyframe image sequence includes: Based on a preset vector machine model or random forest model, the face-swapping probability score, frequency domain anomaly score, and motion coherence score of each keyframe image are processed to obtain the face-swapping probability of each keyframe image. The face-swapping probability of each keyframe image is calculated by weighting, and the face-swapping detection result of the keyframe image sequence is obtained.

6. A real-time face-swapping detection system based on deep learning, characterized in that, include: Data acquisition module, neural network module, frequency domain analysis module, optical flow analysis module, and fusion processing module; The data acquisition module is used to acquire keyframe image sequences in the video stream in real time. The neural module is used to process the keyframe image sequence according to a preset deep learning model to obtain a face-swapping probability score for each keyframe image; wherein, the deep learning model is trained using fake data generated by a generative adversarial network; The frequency domain analysis module is used to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain the frequency domain anomaly score of each key frame image. The optical flow analysis module is used to process the key frame image sequence according to a preset optical flow analysis algorithm to obtain the motion coherence score of each adjacent key frame image. The fusion processing module is used to process each of the face-swapping probability scores, frequency domain anomaly scores, and motion coherence scores according to a preset multimodal fusion algorithm to obtain the face-swapping detection results of the keyframe image sequence; wherein, the face-swapping detection results include: no face-swapping, malicious face-swapping, and non-malicious face-swapping; The data acquisition module includes: a raw acquisition unit, a downsampling unit, and an extraction unit; the raw acquisition unit is used to acquire raw video stream data; the downsampling unit processes the raw video stream data according to a preset frame number control rule to obtain a downsampled frame sequence; the extraction unit is used to process the downsampled frame sequence according to a preset keyframe detection algorithm to obtain a keyframe image sequence. The extraction unit is used to process the downsampled frame sequence according to a preset keyframe detection algorithm to obtain a keyframe image sequence, including: Calculate the combined distance change value of each facial feature point in adjacent frame images in the downsampled frame sequence; According to the preset second threshold range, each adjacent frame image whose comprehensive distance change value conforms to the second threshold range is taken as a key frame; The face region images of each keyframe image are cropped to obtain the keyframe image sequence.

7. An electronic device for real-time face-swapping detection based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of a real-time face-swapping detection method based on deep learning according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a real-time face-swapping detection method based on deep learning according to any one of claims 1-5.

Images