Real-time face change detection method, system and equipment based on deep learning and medium

Abstract

The invention discloses a real-time face change detection method, system and device based on deep learning and a medium. The method comprises the following steps: acquiring a key frame image sequence in a video stream in real time; processing the key frame image sequence according to a preset deep learning model to obtain a face change probability score of each key frame image; processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image; processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image; and according to a preset multi-modal fusion algorithm, processing the face change probability score, the frequency domain anomaly score and the motion coherence score to obtain a face change detection result of the key frame image sequence. According to the method, the efficiency of real-time dynamic face change detection is improved by combining fusion of various processing algorithms.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a real-time face-changing detection method, system, device and medium based on deep learning. Background Art

[0002] With the development of face-swapping technology, the use of malicious face-swapping videos on social media and live broadcast platforms has gradually increased. The abuse of these technologies has led to personal privacy and information security issues. Therefore, an efficient and accurate face-swapping detection method is needed to address the challenge of identifying forged faces in live broadcasts.

[0003] However, some existing face-swapping detection methods, such as machine learning algorithms, may be affected by factors such as facial expressions and postures, or have poor generalization capabilities due to insufficient training, and are prone to problems such as poor real-time performance, insufficient detection accuracy, and high false alarm rates, and are unable to effectively cope with the ever-improving face-swapping technology. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present application provides a real-time face-changing detection method, system, device and medium based on deep learning to improve the efficiency of real-time dynamic face-changing detection.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a real-time face-changing detection method based on deep learning, comprising:

[0006] Acquire key frame image sequences in video streams in real time;

[0007] Processing the key frame image sequence according to a preset deep learning model to obtain a face-swapping probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network;

[0008] Processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score for each key frame image;

[0009] Processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain motion coherence scores of adjacent key frame images;

[0010] The face-changing probability scores, frequency domain anomaly scores and motion continuity scores are processed according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence; wherein the face-changing detection result includes: no face-changing, malicious face-changing and non-malicious face-changing.

[0011] Compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of video stream data can provide a real-time data source for subsequent processing. At the same time, extracting a key frame image sequence can effectively reduce the computational complexity of subsequent processing and improve processing efficiency; processing each key frame image according to a deep learning model obtained by pre-training using a generative adversarial network can improve image feature extraction and detection capabilities; processing each key frame image according to a frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing each key frame image according to an optical flow analysis algorithm can capture the dynamic changes of the face and detect the continuity and naturalness of the face movement; multi-modally fusion of the deep learning model recognition results, frequency domain analysis results, and optical flow analysis results, combined with a variety of features and analysis methods, output face-changing detection results, and improve the accuracy and reliability of detection.

[0012] In some embodiments of the first aspect of the present application, the real-time acquisition of a key frame image sequence in a video stream includes:

[0013] Get the original video stream data;

[0014] According to a preset frame number control rule, the original video stream data is processed to obtain a downsampled frame sequence;

[0015] The downsampled frame sequence is processed according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0016] Compared with the prior art, the above-mentioned embodiment has the following beneficial effects: obtaining the original video stream data, ensuring the integrity and real-time performance of the data, downsampling the original video stream data according to the preset frame number control rules, reducing the computing load, improving the processing speed, screening out the key frame image sequence, making the data for subsequent processing representative, improving the detection accuracy and reliability, and reducing false alarms and missed alarms.

[0017] In some embodiments of the first aspect of the present application, the step of processing the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence includes:

[0018] Calculating pixel difference values ​​or optical flow change values ​​of adjacent frame images in the downsampled frame sequence;

[0019] According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value meets the first threshold range is used as each key frame image;

[0020] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0021] Compared with the prior art, the above embodiments have the following beneficial effects: by using pixel differences or optical flow change values, frames with larger changes can be identified, which can contain more information; a threshold range is set to ensure that the screened frames are sufficiently representative; the face area image of each key frame image is cropped to focus on the face area, reduce interference from irrelevant background, reduce the amount of calculation, and improve the accuracy of detection.

[0022] In some embodiments of the first aspect of the present application, the downsampled frame sequence is processed according to a preset key frame detection algorithm to obtain a key frame image sequence, further comprising:

[0023] Calculating the comprehensive distance change value of each facial feature point of adjacent frame images in the downsampled frame sequence;

[0024] According to a preset second threshold range, each adjacent frame image whose comprehensive distance change value meets the second threshold range is used as each key frame;

[0025] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0026] Compared with the prior art, the above embodiment has the following beneficial effects: by using the comprehensive distance change value of the feature points as the second selection criterion for the key frames, the key frames can be screened more comprehensively, and the second threshold range can be set to ensure that the screened frames are representative in terms of feature point changes, and the face area images of each key frame image are cropped to focus on the face area, reduce interference from irrelevant background, reduce the amount of calculation, and improve the accuracy of detection.

[0027] In some embodiments of the first aspect of the present application, processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image includes:

[0028] Extract the frequency domain pattern of each key frame image according to Fourier transform or wavelet transform algorithm;

[0029] The frequency domain patterns of each key frame image and the normal face image are compared, and the frequency domain anomaly score of each key frame image is calculated.

[0030] Compared with the prior art, the above embodiments have the following beneficial effects: frequency domain analysis can reveal high-frequency and low-frequency components in the image, which helps to detect abnormal patterns left by face-changing operations, and improves the accuracy and reliability of detection by comparing and quantifying the degree of abnormality.

[0031] In some embodiments of the first aspect of the present application, the step of processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image includes:

[0032] Calculate the displacement of each feature point of the face in each adjacent key frame;

[0033] According to each of the displacements, it is determined whether the motion trajectory of the face is abnormal, and the motion continuity score of each adjacent key frame image is obtained.

[0034] Compared with the prior art, the above embodiment has the following beneficial effects: optical flow analysis calculates the displacement and motion trajectory of each feature point of the face, captures the dynamic changes of the face, detects whether there are abnormalities in facial movement, and improves the accuracy of detection.

[0035] In some embodiments of the first aspect of the present application, the face-changing probability score, the frequency domain anomaly score, and the motion coherence score are processed according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence, including:

[0036] According to a preset vector machine model or random forest model, the face-swapping probability score, frequency domain anomaly score and motion coherence score of each key frame image are processed to obtain the face-swapping probability of each key frame image;

[0037] The face-changing probability of each key frame image is weightedly calculated to obtain the face-changing detection result of the key frame image sequence.

[0038] Compared with the prior art, the above embodiment has the following beneficial effects: the multimodal fusion algorithm combines multiple features, performs weighted calculations, and comprehensively considers various scores to ultimately obtain accurate face-changing detection results, thereby improving the comprehensiveness and reliability of detection.

[0039] In a second aspect, an embodiment of the present application further provides a real-time face-changing detection system based on deep learning, including: a data acquisition module, a neural module, a frequency domain analysis module, an optical flow analysis module and a fusion processing module;

[0040] Wherein, the data acquisition module is used to obtain the key frame image sequence in the video stream in real time;

[0041] The neural module is used to process the key frame image sequence according to a preset deep learning model to obtain a face-swapping probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network;

[0042] The frequency domain analysis module is used to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image;

[0043] The optical flow analysis module is used to process the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image;

[0044] The fusion processing module is used to process each of the face-changing probability scores, frequency domain anomaly scores and motion continuity scores according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence; wherein the face-changing detection result includes: no face-changing, malicious face-changing and non-malicious face-changing.

[0045] Compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of video stream data can provide a real-time data source for subsequent processing. At the same time, extracting a key frame image sequence can effectively reduce the computational complexity of subsequent processing and improve processing efficiency; processing each key frame image according to a deep learning model obtained by pre-training using a generative adversarial network can improve image feature extraction and detection capabilities; processing each key frame image according to a frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing each key frame image according to an optical flow analysis algorithm can capture the dynamic changes of the face and detect the continuity and naturalness of the face movement; multi-modally fusion of the deep learning model recognition results, frequency domain analysis results, and optical flow analysis results, combined with a variety of features and analysis methods, output face-changing detection results, and improve the accuracy and reliability of detection.

[0046] In the third aspect, an embodiment of the present application also provides an electronic device for real-time face-changing detection based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the steps of the real-time face-changing detection method based on deep learning are implemented.

[0047] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the real-time face-changing detection method based on deep learning are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 : A flowchart of a real-time face-changing detection method based on deep learning provided in some embodiments of the present invention.

[0049] Figure 2 : A structural diagram of a real-time face-changing detection system based on deep learning provided in some embodiments of the present invention.

[0050] Figure 3 : A structural diagram of an electronic device for real-time face-changing detection based on deep learning provided in some embodiments of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1:

[0053] Please refer to Figure 1 , a real-time face-changing detection method based on deep learning provided by an embodiment of the present invention, comprising steps S1 to S5:

[0054] Step S1: Acquire a key frame image sequence in a video stream in real time.

[0055] In some embodiments of the present application, step S1 can be implemented by the following preferred implementations:

[0056] S11: Obtaining original video stream data; wherein the original video stream data may be a local video, or may be obtained in real time from a network terminal;

[0057] S12: Processing the original video stream data according to a preset frame number control rule to obtain a downsampled frame sequence;

[0058] Among them, the frame rate control rules can be adjusted according to actual conditions. For example, if the frame rate of the video stream obtained in real time from the network is relatively high, such as 60 frames per second, in order to reduce the computing load of the detection system, it can be downsampled to 10 or 15 frames per second. If the detection scene is more complex, for example, there are more colors in the background, or there are more people, the frame rate control can also be higher to ensure a better balance between detection efficiency and detection accuracy. In practical applications, in addition to frame control, GPU or other hardware acceleration capabilities can also be combined to accelerate detection. In actual tests, this solution has achieved a detection delay of less than 100 milliseconds in high-definition live video at 60 frames per second, which is sufficient to meet the real-time requirements of most live broadcast scenarios.

[0059] S13: Processing the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0060] In this preferred embodiment, the original video stream data is obtained to ensure the integrity and real-time nature of the data, and the original video stream data is downsampled according to a preset frame rate control rule to reduce the computing load, increase the processing speed, and filter out key frame image sequences so that the data for subsequent processing is representative, thereby improving detection accuracy and reliability and reducing false alarms and missed alarms.

[0061] Furthermore, the processing of the downsampled frame sequence according to the preset key frame detection algorithm can be implemented by the following preferred implementations:

[0062] Calculating pixel difference values ​​or optical flow change values ​​of adjacent frame images in the downsampled frame sequence;

[0063] According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value meets the first threshold range is used as each key frame image;

[0064] Among them, through the pixel difference or optical flow change value, the frames with large information changes between adjacent frames can be identified. These frames with large changes are more likely to contain more information. Therefore, these frames with large changes can be screened out as the data source for subsequent processing.

[0065] The face region image of each key frame image is cropped to obtain a key frame image sequence, wherein a face detection algorithm can be used to identify the face region during cropping.

[0066] In this preferred embodiment, frames with larger changes are identified through pixel differences or optical flow change values, which can contain more information; a threshold range is set to ensure that the screened frames are sufficiently representative; the face area image of each key frame image is cropped to focus on the face area, reduce interference from irrelevant background, reduce calculation amount, and improve detection accuracy.

[0067] Furthermore, the processing of the downsampled frame sequence according to the preset key frame detection algorithm can also be implemented by the following preferred implementations:

[0068] Calculating the comprehensive distance change value of each facial feature point of adjacent frame images in the downsampled frame sequence;

[0069] According to a preset second threshold range, each adjacent frame image whose comprehensive distance change value meets the second threshold range is used as each key frame;

[0070] Among them, when the overall information of the video picture does not change much, we can focus on the face area. By using the comprehensive distance change value of the feature points on the face (such as eyes, mouth, and facial angles) as the second selection criterion for key frames, we can more comprehensively screen key frames and reduce key frame omissions.

[0071] The face region image of each key frame image is cropped to obtain a key frame image sequence; similarly, when cropping, a face recognition algorithm can be used to identify the face region.

[0072] In this preferred embodiment, the comprehensive distance change value of the feature points is used as the second selection criterion for the key frames, so that the key frames can be screened more comprehensively, and the second threshold range can be set to ensure that the screened frames are representative in terms of feature point changes. The face area images of each key frame image are cropped, focusing on the face area, reducing interference from irrelevant background, reducing the amount of calculation, and improving the accuracy of detection.

[0073] Step S2: Processing the key frame image sequence according to a preset deep learning model to obtain a face-swapping probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network;

[0074] Among them, the deep learning model can be a convolutional neural network, a time series model or other deep learning models. It can also integrate different algorithms through an integrated learning method to build a new model as a face detection model. During training, a variety of image data sets containing different lighting, expressions, angles and backgrounds can be used to train the model, or forged data generated by a generative adversarial network can be used for multiple training to improve the detection ability of the model.

[0075] In this embodiment, each key frame image is processed according to the deep learning model obtained by pre-training using a generative adversarial network, which can improve the image feature extraction capability and enhance the accuracy and robustness of detection.

[0076] Step S3: Processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score for each key frame image.

[0077] Preferably, the frequency domain anomaly score can be obtained by the following preferred implementation:

[0078] Extract the frequency domain pattern of each key frame image according to Fourier transform or wavelet transform algorithm;

[0079] The frequency domain patterns of each key frame image and the normal face image are compared, and the frequency domain anomaly score of each key frame image is calculated.

[0080] Among them, in step S3, in addition to the Fourier transform or wavelet transform algorithm, the frequency domain analysis algorithm can also use other frequency domain analysis algorithms, such as discrete cosine transform or other similar algorithms; when extracting the frequency domain pattern, the detection threshold is adaptively adjusted according to the dynamic characteristics of the video; for example, for videos with large light fluctuations, the sensitivity of the frequency domain analysis can also be dynamically adjusted so that the system can work stably in different environments; at the same time, when performing detection, a sliding window method can be used to continuously detect several frames, and the consistency of the detection results of consecutive frames is judged, such as scene consistency, to eliminate accidental anomalies in a single frame, thereby ensuring that the analysis of expression, frequency domain and motion characteristics has temporal consistency and coherence.

[0081] In this preferred embodiment, frequency domain analysis can reveal high-frequency and low-frequency components in the image, which helps to detect abnormal patterns left by the face-swapping operation, and improve the accuracy and reliability of detection by comparing and quantifying the degree of abnormality.

[0082] Step S4: Processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain motion coherence scores of adjacent key frame images.

[0083] Preferably, the motion coherence score can be obtained by the following preferred implementation:

[0084] Calculate the displacement of each feature point of the face in each adjacent key frame;

[0085] According to each of the displacements, it is determined whether the motion trajectory of the face is abnormal, and the motion continuity score of each adjacent key frame image is obtained.

[0086] Among them, the optical flow analysis algorithm used in step S4 is not further limited, and can be a local optical flow calculation method, a global optical flow calculation method, an optical flow algorithm based on polynomial expansion, or other methods such as deep learning optical flow estimation; when performing optical flow analysis, the detection threshold can also be adaptively adjusted according to the dynamic characteristics of the video; for example, for videos with large light fluctuations, the sensitivity of the optical flow analysis can also be dynamically adjusted so that the system can work stably in different environments; at the same time, when performing detection, a sliding window method can also be used to continuously detect several frames, and the consistency of the detection results of consecutive frames can be judged, such as scene consistency, to eliminate accidental anomalies in a single frame, thereby ensuring that the analysis of expression, frequency domain and motion features has temporal consistency and coherence.

[0087] In this preferred embodiment, optical flow analysis calculates the displacement and motion trajectory of each feature point of the face, captures the dynamic changes of the face, detects whether there is any abnormality in facial movement, and improves the accuracy of detection.

[0088] Step S5: Processing the face-changing probability scores, frequency domain anomaly scores and motion continuity scores according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence.

[0089] In some embodiments of the present application, step S5 can be implemented by the following preferred implementations:

[0090] According to a preset vector machine model or random forest model, the face-swapping probability score, frequency domain anomaly score and motion coherence score of each key frame image are processed to obtain the face-swapping probability of each key frame image;

[0091] The face-changing probability of each key frame image is weightedly calculated to obtain the face-changing detection result of the key frame image sequence.

[0092] Among them, in step S5, the multimodal fusion algorithm, in addition to the vector machine model or the random forest model, can also use other fusion models, or rule-based methods to weightedly fuse the detection results of static and dynamic features; further, after obtaining the detection results, they can be sent to the monitoring personnel or the client, or the alarm system can be linked to issue a real-time warning reminder when the recognition result is a face-changing.

[0093] In this preferred embodiment, the multimodal fusion algorithm combines multiple features, performs weighted calculations, and comprehensively considers various scores to ultimately obtain accurate face-changing detection results, thereby improving the comprehensiveness and reliability of detection.

[0094] In addition, according to the real-time face-changing detection method based on deep learning provided by this embodiment, in some actual system application scenarios, such as in the evaluation on the test data sets FaceForens i cs++ and DFDC, the system achieved a detection accuracy of more than 90%, and the sensitivity (recall rate) in detecting forged facial features also exceeded 85%; in the actual live broadcast test environment, the average detection delay of the system was 80 milliseconds, and the false alarm rate was less than 5%, showing the practicality and efficiency of the system in complex application scenarios.

[0095] In summary, compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of video stream data can provide a real-time data source for subsequent processing. At the same time, extracting a key frame image sequence can effectively reduce the computational complexity of subsequent processing and improve processing efficiency; processing each key frame image according to a deep learning model obtained by pre-training using a generative adversarial network can improve image feature extraction and detection capabilities; processing each key frame image according to a frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing each key frame image according to an optical flow analysis algorithm can capture the dynamic changes of the face and detect the continuity and naturalness of the face movement; multi-modally fusion of the deep learning model recognition results, frequency domain analysis results, and optical flow analysis results, combined with a variety of features and analysis methods, output face-changing detection results, and improve detection accuracy and reliability.

[0096] Embodiment 2:

[0097] Please refer to Figure 2 Based on the same inventive concept, a real-time face-changing detection system based on deep learning disclosed in an embodiment of the present invention includes: a data acquisition module M1, a neural module M2, a frequency domain analysis module M3, an optical flow analysis module M4 and a fusion processing module M5;

[0098] The data acquisition module M1 acquires the key frame image sequence in the video stream in real time.

[0099] The data acquisition module M1 includes: an original acquisition unit, a downsampling unit and an extraction unit; the original acquisition is used to obtain original video stream data; the downsampling unit processes the original video stream data according to a preset frame number control rule to obtain a downsampled frame sequence; the extraction unit is used to process the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence.

[0100] The data acquisition module M1 of this embodiment obtains the original video stream data to ensure the integrity and real-time nature of the data, downsamples the original video stream data according to the preset frame rate control rules, reduces the computing load, improves the processing speed, and screens out the key frame image sequence, so that the data for subsequent processing is representative, improves the detection accuracy and reliability, and reduces false alarms and missed alarms.

[0101] Preferably, the extraction unit is used to process the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence, including:

[0102] Calculating pixel difference values ​​or optical flow change values ​​of adjacent frame images in the downsampled frame sequence;

[0103] According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value meets the first threshold range is used as each key frame image;

[0104] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0105] In this preferred embodiment, frames with larger changes are identified through pixel differences or optical flow change values, which can contain more information; a threshold range is set to ensure that the screened frames are sufficiently representative; the face area image of each key frame image is cropped to focus on the face area, reduce interference from irrelevant background, reduce calculation amount, and improve detection accuracy.

[0106] Furthermore, the extraction unit is used to process the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence, and may also include:

[0107] Calculating the comprehensive distance change value of each facial feature point of adjacent frame images in the downsampled frame sequence;

[0108] According to a preset second threshold range, each adjacent frame image whose comprehensive distance change value meets the second threshold range is used as each key frame;

[0109] The face region image of each key frame image is cropped to obtain a key frame image sequence.

[0110] In this preferred embodiment, the comprehensive distance change value of the feature points is used as the second selection criterion for the key frames, so that the key frames can be screened more comprehensively, and the second threshold range can be set to ensure that the screened frames are representative in terms of feature point changes. The face area images of each key frame image are cropped, focusing on the face area, reducing interference from irrelevant background, reducing the amount of calculation, and improving the accuracy of detection.

[0111] The neural module M2 is used to process the key frame image sequence according to a preset deep learning model to obtain a face-changing probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network.

[0112] The neural module M2 of this embodiment processes each key frame image according to the deep learning model obtained by pre-training using a generative adversarial network, which can improve the image feature extraction capability and enhance the accuracy and robustness of detection.

[0113] The frequency domain analysis module M3 is used to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image.

[0114] The frequency domain analysis module M3 includes: a frequency domain extraction unit and a frequency domain calculation unit; the frequency domain extraction unit is used to extract the frequency domain pattern of each key frame image according to the Fourier transform or wavelet transform algorithm; the frequency domain calculation unit is used to compare the frequency domain patterns of each key frame image and the normal face image, and calculate the frequency domain abnormality score of each key frame image.

[0115] The frequency domain analysis module M3 of this embodiment can reveal the high-frequency and low-frequency components in the image through frequency domain analysis, which helps to detect abnormal patterns left by the face-changing operation and improve the accuracy and reliability of detection by comparing and quantifying the degree of abnormality.

[0116] The optical flow analysis module M4 is used to process the key frame image sequence according to a preset optical flow analysis algorithm to obtain motion continuity scores of adjacent key frame images.

[0117] The optical flow analysis module M4 includes: a displacement calculation unit and a coherence calculation unit; the displacement calculation unit is used to calculate the displacement of each feature point of the face in each adjacent key frame; the coherence calculation unit is used to determine whether the motion trajectory of the face is abnormal based on each displacement, and obtain the motion coherence score of each adjacent key frame image.

[0118] The optical flow analysis module M4 of this embodiment calculates the displacement and motion trajectory of each feature point of the face through optical flow analysis, captures the dynamic changes of the face, detects whether there is any abnormality in facial movement, and improves the accuracy of detection.

[0119] The fusion processing module M5 is used to process each of the face-changing probability scores, frequency domain anomaly scores and motion continuity scores according to a preset multimodal fusion algorithm to obtain the face-changing detection results of the key frame image sequence; wherein the face-changing detection results include: no face-changing, malicious face-changing and non-malicious face-changing.

[0120] The fusion processing module M5 includes: a first probability calculation unit and a result output unit; the first probability calculation unit is used to process the face-changing probability score, frequency domain anomaly score and motion continuity score of each key frame image according to a preset vector machine model or random forest model to obtain the face-changing probability of each key frame image; the result output unit weightedly calculates the face-changing probability of each key frame image to obtain the face-changing detection result of the key frame image sequence.

[0121] The fusion processing module M5 of this embodiment uses a multimodal fusion algorithm, combines multiple features, performs weighted calculations, and comprehensively considers various scores to ultimately obtain accurate face-changing detection results, thereby improving the comprehensiveness and reliability of detection.

[0122] In summary, compared with the prior art, the embodiments of the present application have the following beneficial effects: real-time acquisition of video stream data can provide a real-time data source for subsequent processing. At the same time, extracting a key frame image sequence can effectively reduce the computational complexity of subsequent processing and improve processing efficiency; processing each key frame image according to a deep learning model obtained by pre-training using a generative adversarial network can improve image feature extraction and detection capabilities; processing each key frame image according to a frequency domain analysis algorithm can capture the frequency domain features of the image, detect subtle changes that are difficult to find in the spatial domain, and improve the sensitivity of detection; processing each key frame image according to an optical flow analysis algorithm can capture the dynamic changes of the face and detect the continuity and naturalness of the face movement; multi-modally fusion of the deep learning model recognition results, frequency domain analysis results, and optical flow analysis results, combined with a variety of features and analysis methods, output face-changing detection results, and improve detection accuracy and reliability.

[0123] The division of the modules described above is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, such as directly integrated into an existing video surveillance, security protection and content review system.

[0124] Embodiment three:

[0125] Figure 3 The structure diagram of an electronic device for real-time face-changing detection based on deep learning is presented. Figure 3 As shown, the face-changing detection device may include: a processor N1, a memory N2, a data interface N3 and a communication bus N4.

[0126] Among them: the processor N1, the memory N2, and the data interface N3 communicate with each other through the communication bus N4; the data interface N3 is used for data communication with other additional devices; the processor N1 is used to execute the program N5, which can specifically execute the relevant steps in the above-mentioned embodiment of the real-time face-changing detection method based on deep learning.

[0127] Specifically, the program N5 may include program codes, and the program codes include computer executable instructions.

[0128] The processor N1 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the real-time face-changing detection device based on deep learning may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0129] The memory N2 is used to store the program N5. The memory N2 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0130] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present application are not directed to any particular programming language.

[0131] Embodiment 4:

[0132] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is run on a real-time face-changing detection device / system based on deep learning, the real-time face-changing detection device / system based on deep learning executes the real-time face-changing detection method based on deep learning in any of the above method embodiments.

[0133] In the specification provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. Similarly, in order to simplify the application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are thus clearly incorporated into the specific embodiment, wherein each claim itself is used as a separate embodiment of the application.

[0134] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. A real-time face-changing detection method based on deep learning, characterized in that: include: Acquire key frame image sequences in video streams in real time; Processing the key frame image sequence according to a preset deep learning model to obtain a face-swapping probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network; Processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score for each key frame image; Processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain motion coherence scores of adjacent key frame images; The face-changing probability scores, frequency domain anomaly scores and motion continuity scores are processed according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence; wherein the face-changing detection result includes: no face-changing, malicious face-changing and non-malicious face-changing.

2. A real-time face-changing detection method based on deep learning as claimed in claim 1, characterized in that: The real-time acquisition of a key frame image sequence in a video stream includes: Get the original video stream data; According to a preset frame number control rule, the original video stream data is processed to obtain a downsampled frame sequence; The downsampled frame sequence is processed according to a preset key frame detection algorithm to obtain a key frame image sequence.

3. A real-time face-changing detection method based on deep learning as claimed in claim 2, characterized in that: The step of processing the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence includes: Calculating pixel difference values ​​or optical flow change values ​​of adjacent frame images in the downsampled frame sequence; According to a preset first threshold range, each adjacent frame image whose pixel difference value or optical flow change value meets the first threshold range is used as each key frame image; The face region image of each key frame image is cropped to obtain a key frame image sequence.

4. A real-time face-changing detection method based on deep learning as claimed in claim 3, characterized in that: The step of processing the downsampled frame sequence according to a preset key frame detection algorithm to obtain a key frame image sequence further includes: Calculating the comprehensive distance change value of each facial feature point of adjacent frame images in the downsampled frame sequence; According to a preset second threshold range, each adjacent frame image whose comprehensive distance change value meets the second threshold range is used as each key frame; The face region image of each key frame image is cropped to obtain a key frame image sequence.

5. A real-time face-changing detection method based on deep learning as claimed in claim 4, characterized in that: The step of processing the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image includes: Extract the frequency domain pattern of each key frame image according to Fourier transform or wavelet transform algorithm; The frequency domain patterns of each key frame image and the normal face image are compared, and the frequency domain anomaly score of each key frame image is calculated.

6. A real-time face-changing detection method based on deep learning as claimed in claim 5, characterized in that: The step of processing the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image includes: Calculate the displacement of each feature point of the face in each adjacent key frame; According to each of the displacements, it is determined whether the motion trajectory of the face is abnormal, and the motion continuity score of each adjacent key frame image is obtained.

7. A real-time face-changing detection method based on deep learning as claimed in claim 6, characterized in that: The step of processing the face-changing probability scores, the frequency domain anomaly scores and the motion continuity scores according to a preset multimodal fusion algorithm to obtain the face-changing detection result of the key frame image sequence includes: According to a preset vector machine model or random forest model, the face-swapping probability score, frequency domain anomaly score and motion coherence score of each key frame image are processed to obtain the face-swapping probability of each key frame image; The face-changing probability of each key frame image is weightedly calculated to obtain the face-changing detection result of the key frame image sequence.

8. A real-time face-changing detection system based on deep learning, characterized in that: include: Data acquisition module, neural module, frequency domain analysis module, optical flow analysis module and fusion processing module; Wherein, the data acquisition module is used to obtain the key frame image sequence in the video stream in real time; The neural module is used to process the key frame image sequence according to a preset deep learning model to obtain a face-swapping probability score for each key frame image; wherein the deep learning model is trained using forged data generated by a generative adversarial network; The frequency domain analysis module is used to process the key frame image sequence according to a preset frequency domain analysis algorithm to obtain a frequency domain anomaly score of each key frame image; The optical flow analysis module is used to process the key frame image sequence according to a preset optical flow analysis algorithm to obtain a motion coherence score of each adjacent key frame image; The fusion processing module is used to process each of the face-changing probability scores, frequency domain anomaly scores and motion continuity scores according to a preset multimodal fusion algorithm to obtain a face-changing detection result of the key frame image sequence; wherein the face-changing detection result includes: no face-changing, malicious face-changing and non-malicious face-changing.

9. An electronic device for real-time face-changing detection based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the steps of the real-time face-changing detection method based on deep learning according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a real-time face-changing detection method based on deep learning are implemented according to any one of claims 1 to 7.

Images