Video authenticity identification method and electronic equipment

Abstract

The invention provides a video authenticity identification method and electronic equipment, and the method comprises the steps: obtaining a target image frame sequence of a to-be-identified video, respectively obtaining the corresponding multi-modal features of each target image frame in the target image frame sequence, obtaining a multi-modal feature sequence corresponding to the target image frame sequence, and obtaining a multi-modal feature sequence corresponding to the multi-modal feature sequence; the multi-modal feature sequence is analyzed based on the LSTM to obtain an analysis result of the target image frame sequence in the time dimension, the analysis result is input into the discriminator to obtain an authenticity identification result of the to-be-identified video, and the authenticity of the video can be accurately identified by comprehensively analyzing the image frame sequence.

Description

Technical Field

[0001] The present application relates to the technical field of video authentication, and in particular to a method and electronic device for authenticating a video. Background Art

[0002] In today's digital age, the rapid development of deep learning technology has led to the continuous improvement of deepfake technology. Malicious actors can use deepfake technology to easily generate fake videos that are almost indistinguishable from the real thing. These fake videos are like a flood and have a catastrophic impact on all levels of society. For example, in the field of news dissemination, fake videos may lead to the rapid spread of false news, mislead public opinion, and disrupt social order; at the personal level, deepfake videos can be used for defamation, causing irreparable damage to personal reputation and image; in the commercial and financial fields, fraudsters may use fake videos to commit fraudulent activities, causing huge economic losses to companies and individuals. Therefore, how to identify the authenticity of videos has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a video authenticity identification method and electronic device to solve the above-mentioned technical problems.

[0004] On the one hand, a method for identifying authenticity of a video is provided, comprising: Obtain a target image frame sequence of the video to be identified; For each target image frame in the target image frame sequence, the corresponding multimodal features are respectively obtained to obtain a multimodal feature sequence corresponding to the target image frame sequence; the multimodal features include an image feature vector of the target image frame, a motion feature vector and an audio feature vector corresponding to the target image frame; the motion feature vector is a vector calculated based on the target image frame and the next target image frame adjacent to the target image frame and used to reflect the motion features of the target object in the target image frame; Analyzing the multimodal feature sequence based on LSTM to obtain an analysis result of the target image frame sequence in a time dimension; The analysis result is input into the discriminator to obtain the authenticity identification result of the video to be identified.

[0005] In one embodiment, obtaining a target image frame sequence of a video to be identified includes: Decomposing the video to be identified into a continuous sequence of original image frames; Preprocessing is performed on each original image frame in the original image frame sequence to obtain a target image frame sequence.

[0006] In one of the embodiments, the preprocessing of each original image frame in the original image frame sequence to obtain a target image frame sequence includes: For each original image frame in the original image frame sequence, adjusting an image parameter of the original image frame through a normalization operation; the image parameter includes at least one of image size, image brightness, and image contrast; The original image frame after the normalization operation is subjected to noise reduction processing to obtain a target image frame.

[0007] In one embodiment, adjusting the image parameters of each original image frame in the original image frame sequence by a normalization operation includes: By formula Adjusting the pixel value of each pixel in the original image frame; represents the pixel value of a certain pixel in the original image frame, represents the pixel value of the corresponding pixel point after the image size of the original image frame is adjusted, represents the minimum pixel value in the original image frame, represents the maximum pixel value in the original image frame, Indicates the minimum pixel value within the target mapping range, Indicates the maximum pixel value within the target mapping range; By formula ΔL Adjusting the brightness value of each pixel in the original image frame; Indicates that the pixel coordinates in the original image frame are The brightness value at ΔL Indicates the brightness adjustment increment coefficient; or through the formula Adjusting the contrast value of each pixel in the original image frame; represents the average brightness of the original image frame, Indicates the contrast adjustment factor.

[0008] In one embodiment, for each target image frame in the target image frame sequence, respectively acquiring the corresponding multimodal features thereof to obtain a multimodal feature sequence corresponding to the target image frame sequence includes: For each of the target image frames, a convolutional neural network is used to extract features of the target image frame to obtain an image feature vector of the target image frame; Calculating a motion feature vector of the target image frame based on a pixel value of the target image frame and a pixel value of a next target image frame adjacent to the target image frame; Extracting audio data of the video to be identified, performing feature extraction on a frame of target audio data in the audio data corresponding to the target image frame, and obtaining an audio feature vector corresponding to the target image frame; The image feature vector, the motion feature vector and the audio feature vector are fused to obtain multimodal features of the target image frame.

[0009] In one of the embodiments, calculating the motion feature vector of the target image frame based on the pixel value of the target image frame and the pixel value of the next target image frame adjacent to the target image frame includes: Determining the moving direction of the target object according to the gradients of the pixel difference between the target image frame and the next target image frame in the horizontal and vertical directions; Calculating the moving speed of the target object according to the displacement and interval time of the target object between the target image frame and the next target image frame; A motion feature vector of the target image frame is calculated based on the motion direction and the motion speed.

[0010] In one embodiment, the step of extracting features from a frame of target audio data in the audio data corresponding to the target image frame to obtain an audio feature vector corresponding to the target image frame includes: Spectral features and Mel-frequency cepstrum coefficients are extracted from the target audio data to obtain an audio feature vector of the target audio data.

[0011] In one embodiment, the analyzing the multimodal feature sequence based on LSTM to obtain the analysis result of the target image frame sequence in the time dimension includes: The multimodal feature sequence is analyzed by the following formula: ; ; ; ; ;in, represents the input gate, represents the forget gate, represents the output gate, Represents a memory unit, represents the activation function, and Represent the weight matrix and bias vector of the input gate respectively, and Represent the weight matrix and bias vector of the forget gate respectively, and Represent the weight matrix and bias vector of the memory unit respectively, and Represent the weight matrix and bias vector of the output gate respectively, Indicates The multimodal features input at each moment, Indicates -1 hidden state at a time.

[0012] In one embodiment, the discriminator is formed by fusing a plurality of classifiers of different categories, and the analysis result is input into the discriminator to obtain the authenticity identification result of the video to be identified, including: Input the hidden state at the last moment into the discriminator; The discriminator is based on Determining the authenticity of the video to be identified; Indicates The weight of the classifier, Indicates The classifier determines the probability that the video to be identified is a real video, Represents the total number of classifiers.

[0013] Furthermore, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above methods.

[0014] The video authenticity identification method and electronic device provided by the present application obtain a target image frame sequence of a video to be identified, obtain the corresponding multimodal features for each target image frame in the target image frame sequence, obtain a multimodal feature sequence corresponding to the target image frame sequence, analyze the multimodal feature sequence based on LSTM, obtain an analysis result of the target image frame sequence in the time dimension, input the analysis result into a discriminator to obtain an authenticity identification result of the video to be identified, and accurately identify the authenticity of the video by performing a comprehensive analysis of the image frame sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a process for video authenticity identification provided by an embodiment of the present application; Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] Existing video identification methods are mainly based on some simple external feature analysis, such as video resolution, frame rate, encoding format, etc. However, the makers of deep fake videos can carefully adjust these parameters to make the fake videos indistinguishable from the real videos in these aspects. Among traditional methods, identification methods based on manual features and machine learning have a certain effect, but their identification accuracy is low in the face of complex deep fake videos. It can be expressed as: in, is the number of true positives (the number of real videos correctly identified), is the number of false positives (the number of fake videos mistakenly identified as real videos), is the number of true negatives (the number of correctly identified fake videos), is the number of false negatives (the number of real videos misclassified as fake videos). In complex deep fake scenarios, the performance of these methods is The value is low and the efficiency is not high, which is difficult to meet actual needs.

[0018] Some existing technologies attempt to identify from multiple dimensions. For example, from the texture features of the image Starting from the specific texture analysis algorithm, the texture complexity, direction and other information of the image are calculated; from the consistency of facial expressions Angle analysis, based on the physiological laws of human facial expressions, builds a model to determine whether the changes in facial expressions of people in the video are in line with common sense. However, these methods have obvious shortcomings. For non-face-related deep fake video content, methods based on facial expression analysis are completely ineffective, and these methods are easily affected by image noise N and video quality Q.

[0019] In summary, existing identification technologies generally lack the ability to comprehensively, deeply and comprehensively analyze video frame sequences. With the continuous updating and iteration of deepfake technology, existing identification methods are increasingly unable to cope with new challenges. They have serious deficiencies in identification accuracy, robustness and versatility, and are unable to effectively prevent the rampant spread of fake videos in the network environment.

[0020] In view of this, the present application embodiment provides a video authenticity identification method, see Figure 1 As shown, the following steps are included: S11: Obtain a target image frame sequence of the video to be identified.

[0021] S12: For each target image frame in the target image frame sequence, respectively obtain its corresponding multimodal features to obtain a multimodal feature sequence corresponding to the target image frame sequence; the multimodal features include an image feature vector of the target image frame, a motion feature vector and an audio feature vector corresponding to the target image frame; the motion feature vector is a vector calculated based on the target image frame and the next target image frame adjacent to the target image frame, and is used to reflect the motion characteristics of the target object in the target image frame.

[0022] S13: Analyze the multimodal feature sequence based on LSTM to obtain an analysis result of the target image frame sequence in the time dimension.

[0023] S14: Input the analysis result into an identifier to obtain an authenticity identification result of the video to be identified.

[0024] In step S11, the video to be identified may be decomposed into a continuous sequence of original image frames; each original image frame in the sequence of original image frames is preprocessed to obtain a target image frame sequence.

[0025] Specifically, let the input video be V(t), where t represents time, and the video to be identified is decomposed into a continuous sequence of original image frames through the frame extraction algorithm. , where n is the number of frames extracted. Then, each original image frame in the original image frame sequence is preprocessed to obtain the corresponding target image frame , each target image frame constitutes a target image frame sequence. Specifically, the preprocessing includes: For each original image frame in the original image frame sequence, the image parameters of the original image frame are adjusted through a normalization operation; the image parameters include at least one of image size, image brightness and image contrast; and the original image frame processed by the normalization operation is subjected to noise reduction processing to obtain a target image frame.

[0026] The step of adjusting the image parameters of each original image frame in the original image frame sequence by a normalization operation includes: By formula Adjusting the pixel value of each pixel in the original image frame; represents the pixel value of a certain pixel in the original image frame, represents the pixel value of the corresponding pixel point after the image size of the original image frame is adjusted, represents the minimum pixel value in the original image frame, represents the maximum pixel value in the original image frame, Indicates the minimum pixel value within the target mapping range, Indicates the maximum pixel value within the target mapping range. For example, if the goal is to normalize the pixel values ​​in the original image frame to the range of 0 to 255, then It is 255. It is 0. and The value of can be determined according to the actual image processing requirements and the characteristics of the target image.

[0027] By formula ΔL Adjusting the brightness value of each pixel in the original image frame; Indicates that the pixel coordinates in the original image frame are The brightness value at ΔL Indicates the brightness adjustment increment coefficient; or through the formula Adjusting the contrast value of each pixel in the original image frame; represents the average brightness of the original image frame, Indicates the contrast adjustment factor.

[0028] The normalization operation can ensure the consistency of the image, which is convenient for subsequent processing. In the embodiment of the present application, a filtering algorithm, such as mean filtering and median filtering, can be used to perform noise reduction processing to reduce interference factors in the image.

[0029] The above step S12 may include the following sub-steps: Sub-step 1: For each of the target image frames, use a convolutional neural network to extract features of the target image frame to obtain an image feature vector of the target image frame.

[0030] Sub-step 2: Calculate the motion feature vector of the target image frame based on the pixel value of the target image frame and the pixel value of the next target image frame adjacent to the target image frame.

[0031] Sub-step three: extracting the audio data of the video to be identified, performing feature extraction on a frame of target audio data in the audio data corresponding to the target image frame, and obtaining an audio feature vector corresponding to the target image frame.

[0032] Sub-step four: fusing the image feature vector, the motion feature vector and the audio feature vector to obtain multimodal features of the target image frame.

[0033] In the above sub-step 1, CNN (Convolutional Neural Networks) is used to To extract features. CNN consists of multiple convolutional layers, pooling layers, and activation functions. The convolutional layer uses the convolution kernel K Perform convolution operation with the target image frame to extract the local image features of the target image frame. Assume the convolution kernel size is , the step length is s , filled with p , then the convolution operation can be expressed as: in is the convolution result, j Represents the position index of the convolution kernel on the image. After multiple convolutional layers and pooling layers, CNN can automatically learn the deep features in the image, such as edges, textures, shapes, etc., and obtain the image feature vector .

[0034] In the above sub-step 2, the moving direction of the target object can be determined according to the gradient of the pixel difference between the target image frame and the next target image frame in the horizontal and vertical directions; the moving speed of the target object can be calculated according to the displacement and interval time of the target object between the target image frame and the next target image frame; and the moving feature vector of the target image frame can be calculated based on the moving direction and the moving speed. .

[0035] Assume that two adjacent target image frames are and , then the target image frame can be calculated and The pixel difference between , by calculating the gradient of the pixel difference in the horizontal and vertical directions to determine the movement direction of the target object , through the target object in the target image frame and The displacement of the target object is determined based on the position of the target image frame. and The interval time and the displacement between them are used to calculate the speed of the target object. .

[0036] The specific method to determine the direction and speed of movement is as follows: The direction of motion is determined by calculating the horizontal and vertical gradients.

[0037] For the calculated pixel difference , respectively calculate its horizontal direction and vertical direction The gradient on .

[0038] Horizontal gradient The adjacent pixels can be Difference calculation in direction, for example using the central difference method: ， in Indicates the pixel position.

[0039] Vertical gradient Similarly, it is calculated by the difference between adjacent pixels in the y direction: .

[0040] Then determine the direction of movement based on the horizontal and vertical gradients, and use the inverse tangent function 。

[0041] Calculate velocity by combining displacement and interval time. The target object is in the target image frame and The position in can be determined by the coordinate change of the feature points (such as edges, corners, etc.) of the target object in the two frames. Assume that the target object is The coordinates of the feature points in are ,exist The coordinates of the feature points in are , then displacement.

[0042] Known target image frame and The interval time between (usually the inverse of the video frame rate), then the speed .

[0043] Method for determining motion feature vector based on motion direction and speed Motion feature vector Can be designed to include direction of motion and speed In vector form, for example Such motion feature vectors can comprehensively reflect the changes in the motion state of the target object between image frames. After being fused with the image feature vector and the audio feature vector, they can provide richer information for subsequent LSTM-based time dimension analysis, which helps to more accurately identify the authenticity of the video.

[0044] In the above sub-step 3, the audio data of the video to be identified is , according to the time interval corresponding to the target image frame sequence, the audio data is divided into multiple frames of audio data, and the first Frame target image frame and the first +1 frame of target image frame corresponding to the time period between the frame of this audio data as The target audio data corresponding to the target image frame. The audio feature extraction of the target audio data includes: extracting the spectrum features and Mel frequency cepstrum coefficients of the target audio data, thereby obtaining the audio feature vector of the target audio data. Among them, it can be obtained by Fourier transform Calculate the spectrum.

[0045] For each target image frame, its corresponding image feature vector, motion feature vector and audio feature vector can be obtained. By fusing the image feature vector, motion feature vector and audio feature vector, the multimodal feature corresponding to the target image frame can be obtained. By combining the multimodal features corresponding to each target image frame in time sequence, the multimodal feature sequence corresponding to the target image frame sequence can be obtained.

[0046] It should be noted that fusing the image feature vector, the motion feature vector and the audio feature vector includes: fusing the image feature vector , motion feature vector and the audio feature vector For splicing, for example, As a multimodal feature corresponding to the target image frame.

[0047] Step S13 includes: The multimodal feature sequence is analyzed by the following formula: ; ; ; ; ; in, represents the input gate, represents the forget gate, represents the output gate, Represents a memory unit, represents the activation function, and Represent the weight matrix and bias vector of the input gate respectively, and Represent the weight matrix and bias vector of the forget gate respectively, and Represent the weight matrix and bias vector of the memory unit respectively, and Represent the weight matrix and bias vector of the output gate respectively, Indicates The multimodal features of the input at each moment, Indicates -1 hidden state at a time.

[0048] The activation function may be a sigmoid activation function, which can effectively process long-term dependencies in sequence data through LSTM (Long Short Term Memory) and capture the changing patterns of video frame sequences in the time dimension.

[0049] During the LSTM analysis, the time interval information between frames is Δt As an additional input, the model can better understand the temporal evolution of the video content. By fusing the time interval information with the feature vector, the model's ability to understand the dynamic changes of the video is further improved.

[0050] In the above step S14, the hidden state at the last moment can be Input the discriminator; the discriminator is based on Determining the authenticity of the video to be identified; Indicates The weight of the classifier, Indicates A classifier determines the probability that the video to be identified is a real video, Represents the total number of classifiers.

[0051] It should be noted that in the embodiment of the present application, a fully connected layer can be used to construct a classifier, and classifiers of different categories are fused to form a discriminator. Specifically, the hidden state of the last moment in the LSTM is input into each classifier to obtain the classification results of each classifier, and finally based on Determine the authenticity of the video to be authenticated.

[0052] The classifier is trained with a large amount of real video and deep fake video data, and can output the probability that the video is a real video. The calculation of the classifier can be expressed as: .

[0053] in, and Represent the weight matrix and bias vector of the classifier respectively, represents the activation function. In an optional implementation, the discriminator can Compared with the preset probability threshold, if If the probability is greater than or equal to a preset probability threshold, the video to be identified is determined to be a real video; otherwise, the video to be identified is determined to be a deep fake video. The size of the preset probability threshold can be flexibly set by the developer.

[0054] In the embodiments of the present application, by utilizing multimodal features of the video such as images, motion, and audio, and deeply analyzing the temporal changes in the frame sequence, it is possible to more accurately identify deepfake videos. Compared with traditional methods, the accuracy rate is significantly improved in the identification of complex scenes and high-quality deepfake videos. The data preprocessing step reduces the interference of factors such as image noise. The CNN in the feature extraction process can automatically learn robust features, and the effective processing of time series changes by LSTM enables the identification technology to maintain good performance in the face of videos with different frame rates and different shooting conditions. Robustness index It can be measured by the change of accuracy under different interference conditions. The value remains stable, which is better than the existing technology. The present invention can not only identify deep fake videos based on face replacement, but also effectively identify other types of deep fake videos, such as object replacement, scene tampering, etc., through multimodal features and frame sequence analysis. Generality index The success rate of identifying different types of deep fake videos can be used to evaluate the effectiveness of the present invention in identifying various types of deep fake videos. Higher value.

[0055] It should be noted that the CNN, LSTM, classifier and discriminator mentioned in the above content are all obtained through training. This process is introduced in detail below.

[0056] 1. Data Preparation (1) Collect training data Collect a large number of real videos and known deep fake videos from multiple channels. The sources of real videos include but are not limited to major news organizations, film and television production companies, content shared by social network users, surveillance video repositories, etc. These real videos cover different scene types (such as indoors, outdoors, daytime, nighttime, etc.), shooting equipment (different models of cameras, mobile phones, etc.) and shooting styles (professional shooting, amateur shooting). For deep fake videos, collect samples generated using different deep fake technologies, such as face replacement and expression tampering videos generated by various variants of generative adversarial networks (GANs) (such as DCGAN, StyleGAN, etc.), as well as object replacement and scene synthesis videos generated by technologies such as autoencoders. Ensure that the collected dataset is large enough and diverse enough to cover various situations that may occur in reality.

[0057] During the collection process, each video is annotated with detailed metadata, including video source, shooting equipment information, scene type, whether it contains audio, etc. These metadata will facilitate targeted processing in subsequent data screening, preprocessing, and model training.

[0058] (2) Data labeling For each collected video, it is labeled according to the pre-defined labeling criteria. For example, for real videos, it is marked as "0", and for deep fake videos, it is marked as "1".

[0059] When annotating deepfake videos, we further annotate their deepfake types (such as face replacement, object replacement, scene tampering, etc.) and the deepfake technical features that may be used (such as specific GAN architecture or other algorithm features). The annotated dataset forms a structured set that can be represented as ,in It is Videos, It is its authenticity label, Is the relevant metadata information.

[0060] 2. Model Training (1) Initialize model parameters According to the characteristics of different network modules, appropriate parameter initialization methods are used. For the convolutional neural network (CNN) part, the Xavier initialization method is used to initialize the convolution kernel weights. For the weight matrix and bias vector in the long short-term memory network (LSTM), they are initialized according to the normal distribution, where the mean of the weight matrix is ​​set to 0 and the standard deviation is adjusted according to the input and output dimensions to ensure appropriate gradient propagation. For the parameters of the fully connected layer, Xavier initialization or other suitable initialization strategies are also used to ensure that the model has a good parameter distribution in the early stage of training, which is conducive to the subsequent optimization process.

[0061] During the initialization process, the scale and complexity of the model are taken into consideration, and the range of initialization parameters is appropriately adjusted for large-scale models to avoid gradient vanishing or gradient exploding problems. At the same time, specific initialization settings are made for the parameters of some special layers (such as the BatchNormalization layer) to ensure that they can effectively normalize the data during training.

[0062] (2) Training process The labeled data set is divided into training set, validation set and test set. Usually the division is 70%, 20% and 10%. The training set data is input into the model and processed according to the process of data preprocessing, feature extraction, sequence analysis and identification module.

[0063] In the data preprocessing stage, for each video, frame extraction is performed according to the established algorithm to ensure that the extracted frame sequence is complete and no information is lost. In the normalization process, the brightness and contrast are accurately adjusted according to the original pixel value distribution of the image so that the visual features of the image are within an appropriate range. At the same time, advanced denoising algorithms (such as denoising methods based on wavelet transform) are used to remove noise in the image and improve image quality.

[0064] In the feature extraction stage, CNN performs multi-layer convolution and pooling operations on each frame. The size, step size, and padding parameters of the convolution kernel are optimized through multiple experiments to fully extract the local and global features of the image.

[0065] When calculating motion features, the pixel changes between adjacent frames are accurately analyzed, and optical flow algorithms (such as the Lucas-Kanade optical flow algorithm) are used to accurately calculate the object's movement direction and speed.

[0066] For audio feature extraction, a high-precision audio processing library is used to calculate the spectrum and Mel-frequency cepstral coefficients (MFCC) to ensure the accuracy of audio features.

[0067] In the sequence analysis module, LSTM processes the input multimodal feature vector and time interval information. The appropriate number of LSTM units and hidden layer dimensions are set, and the optimal network structure is determined through a large number of experiments.

[0068] During the training process, the cross entropy loss function is used to measure the difference between the identification results output by the model and the true label. Combined with the Adam optimizer, the learning rate is dynamically adjusted according to the characteristics of the training data. The initial value of the learning rate is determined after multiple experiments. During the training process, the learning rate decay strategy is adopted. For example, after a certain number of training rounds (such as 5 rounds), the learning rate is multiplied by a decay factor less than 1 (such as 0.9) to ensure that the model can converge stably.

[0069] After each training round, the model is evaluated on the validation set. The accuracy, loss, recall, and F1 values ​​on the validation set are calculated. The hyperparameters of the model are adjusted according to the evaluation results of the validation set, such as increasing or decreasing the number of network layers, adjusting the parameters of the convolution kernel, etc. If overfitting occurs on the validation set (such as the validation loss value increases for several consecutive rounds), take appropriate measures, such as adding a dropout layer, adjusting the regularization parameters, or stopping training early.

[0070] 3. Model Testing (1) Test data preparation Collect new real videos and deep fake videos from channels with different sources but similar distribution characteristics as the training data as test sets. These test data should include various types of real scenes and deep fake videos generated by different technologies to comprehensively evaluate the generalization ability of the model in practical applications.

[0071] During the collection process, the test videos are also annotated with detailed metadata, consistent with the annotation method of the training data, so as to facilitate in-depth analysis of the test results.

[0072] (2) Testing process The test set data is input into the trained model, and the model processes each video according to the trained process and outputs the identification result. For each test video, the identification probability and the final judgment result (true or false) output by the model are recorded.

[0073] Calculate the various evaluation indicators of the model on the test set, including accuracy, recall, F1 value, false positive rate, and false negative rate. The accuracy calculation formula is: ,in are true positives (correctly identified deepfake videos), are true negatives (real videos that were correctly identified), is a false positive (misclassifying a real video as a deepfake), is a false negative (misclassifying a deep fake video as a real video). The recall rate is calculated as: , the F1 value calculation formula is: ,in Through the comprehensive analysis of these indicators, the performance of the model is comprehensively evaluated, and the performance of the model in test samples of different types and difficulties is determined, providing a reference for the practical application of the model.

[0074] IV. Practical Application (1) Online video platforms The identification technology of the present invention is integrated into the video uploading and reviewing process of the platform in the form of a plug-in or a system module. When a user uploads a video, the video data is first transmitted to the identification system.

[0075] In the identification system, the data preprocessing module immediately processes the video, decomposes it into a frame sequence, and performs normalization and denoising operations. Subsequently, the feature extraction module extracts image, motion and audio features, and the sequence analysis module analyzes the frame sequence features in the time dimension. Finally, the identification module outputs the probability of whether the video is real or fake. If the identification result shows that the video is a deep fake video, a warning message is sent to the platform, and the platform will reject the upload of the video and send a prompt to the user to inform him that there is a problem with the uploaded video and it may violate the platform regulations. At the same time, the relevant information of the suspected deep fake video (such as the source of the video, details of the identification results, etc.) is recorded in the platform's log system for subsequent analysis and monitoring.

[0076] Regularly conduct spot checks on videos posted on the platform and use an identification system to re-evaluate the authenticity of the videos to respond to possible new deep fake technologies or missed detections, ensuring the long-term reliability of the platform's content.

[0077] (2) News media industry When news media organizations receive external video materials or internally produced videos, they pass them as input to this identification technology system. In the news editing process, a special video identification link is set up to ensure that all videos used for news reports are strictly authenticated.

[0078] When the video enters the identification system, it is processed by various modules to obtain the identification result. If the video is identified as authentic and has a high credibility (probability close to 1), it can smoothly enter the news production process. If the video is identified as a suspected deep fake video (the probability is within a certain range, such as 0.3 - 0.7), the video will be handed over to a professional news verification team to further confirm the authenticity of the video in combination with other information sources (such as interview records, on-site evidence, etc.). If the video is determined to be a deep fake video, it is prohibited to use it in news reports, and the source of the video will be investigated to prevent false information from spreading through the news media and maintain the credibility of the news industry and the public's right to know.

[0079] News media organizations can establish their own video identification database to store identified video information (including identification results, video features, sources, etc.) for subsequent data analysis and model updates to adapt to the ever-changing deep fake technology environment.

[0080] Based on the same inventive concept, see Figure 2 As shown, an embodiment of the present application also provides an electronic device, which includes a processor 201 and a memory 202, in which a computer program is stored. The processor 201 and the memory 202 communicate through a communication bus, and the processor 201 executes the computer program to implement the steps of the method in the above embodiment, which will not be repeated here.

[0081] Understandably, Figure 2 The structure shown is for illustration only. The electronic device may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown.

[0082] The processor 201 can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0083] The memory 202 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0084] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0085] It should be noted that the diagram provided in the present embodiment only illustrates the basic concept of the present invention in a schematic manner, so the diagram only shows the components related to the present invention rather than drawing according to the number, shape and size of the components during actual implementation. The type, quantity and ratio of each component during actual implementation can be a random change, and the component layout type may also be more complicated. The structure, ratio, size, etc. illustrated in the drawings of the present specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can implement, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effect that the present invention can produce and the purpose that can be achieved. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of the present invention. The change or adjustment of its relative relationship should also be regarded as the scope of the present invention without substantially changing the technical content.

[0086] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for identifying the authenticity of a video, characterized in that: include: Obtain a target image frame sequence of the video to be identified; For each target image frame in the target image frame sequence, the corresponding multimodal features are respectively obtained to obtain a multimodal feature sequence corresponding to the target image frame sequence; the multimodal features include an image feature vector of the target image frame, a motion feature vector corresponding to the target image frame, and an audio feature vector; the The motion feature vector is a vector calculated based on the target image frame and the next target image frame adjacent to the target image frame and used to reflect the motion feature of the target object in the target image frame; Analyzing the multimodal feature sequence based on LSTM to obtain an analysis result of the target image frame sequence in a time dimension; The analysis result is input into the discriminator to obtain the authenticity identification result of the video to be identified.

2. The video authenticity identification method according to claim 1, characterized in that: The step of obtaining a target image frame sequence of a video to be identified comprises: Decomposing the video to be identified into a continuous sequence of original image frames; Preprocessing is performed on each original image frame in the original image frame sequence to obtain a target image frame sequence.

3. The video authenticity identification method according to claim 2, characterized in that: The preprocessing of each original image frame in the original image frame sequence to obtain a target image frame sequence comprises: For each original image frame in the original image frame sequence, adjusting the image parameters of the original image frame by a normalization operation; The image parameters include at least one of image size, image brightness and image contrast; The original image frame after the normalization operation is subjected to noise reduction processing to obtain a target image frame.

4. The video authenticity identification method according to claim 3, characterized in that: The step of adjusting the image parameters of each original image frame in the original image frame sequence by a normalization operation includes: By formula Adjusting the pixel value of each pixel in the original image frame; represents the pixel value of a certain pixel in the original image frame, represents the pixel value of the corresponding pixel point after the image size of the original image frame is adjusted, represents the minimum pixel value in the original image frame, represents the maximum pixel value in the original image frame, Indicates the minimum pixel value within the target mapping range, Indicates the maximum pixel value within the target mapping range; By formula ΔL Adjusting the brightness value of each pixel in the original image frame; Indicates that the pixel coordinates in the original image frame are The brightness value at ΔL Indicates the brightness adjustment increment coefficient; or through the formula Adjusting the contrast value of each pixel in the original image frame; represents the average brightness of the original image frame, Indicates the contrast adjustment factor.

5. The video authenticity identification method according to claim 1, characterized in that: The step of acquiring the corresponding multimodal features of each target image frame in the target image frame sequence to obtain a multimodal feature sequence corresponding to the target image frame sequence includes: For each of the target image frames, a convolutional neural network is used to extract features of the target image frame to obtain an image feature vector of the target image frame; Calculating a motion feature vector of the target image frame based on a pixel value of the target image frame and a pixel value of a next target image frame adjacent to the target image frame; Extracting audio data of the video to be identified, performing feature extraction on a frame of target audio data in the audio data corresponding to the target image frame, and obtaining an audio feature vector corresponding to the target image frame; The image feature vector, the motion feature vector and the audio feature vector are fused to obtain multimodal features of the target image frame.

6. The video authenticity identification method according to claim 5, characterized in that: The step of calculating the motion feature vector of the target image frame based on the pixel value of the target image frame and the pixel value of the next target image frame adjacent to the target image frame comprises: Determining the moving direction of the target object according to the gradients of the pixel difference between the target image frame and the next target image frame in the horizontal and vertical directions; Calculating the moving speed of the target object according to the displacement and interval time of the target object between the target image frame and the next target image frame; A motion feature vector of the target image frame is calculated based on the motion direction and the motion speed.

7. The video authenticity identification method according to claim 5, characterized in that: The step of extracting features from a frame of target audio data in the audio data corresponding to the target image frame to obtain an audio feature vector corresponding to the target image frame includes: Spectral features and Mel-frequency cepstrum coefficients are extracted from the target audio data to obtain an audio feature vector of the target audio data.

8. The video authenticity identification method according to claim 1, characterized in that: The analyzing the multimodal feature sequence based on LSTM to obtain the analysis result of the target image frame sequence in the time dimension includes: The multimodal feature sequence is analyzed by the following formula: ； ； ； ； ;in, represents the input gate, represents the forget gate, represents the output gate, Represents a memory unit, represents the activation function, and Represent the weight matrix and bias vector of the input gate respectively, and Represent the weight matrix and bias vector of the forget gate respectively, and Represent the weight matrix and bias vector of the memory unit respectively, and Represent the weight matrix and bias vector of the output gate respectively, Indicates The multimodal features of the input at each moment, Indicates -1 hidden state at a time.

9. The video authenticity identification method according to claim 1, characterized in that: The discriminator is formed by fusing a plurality of classifiers of different categories, and the analysis result is input into the discriminator to obtain the authenticity identification result of the video to be identified, including: Input the hidden state at the last moment into the discriminator; The discriminator is based on Determining the authenticity of the video to be identified; Indicates The weight of the classifier, Indicates The classifier determines the probability that the video to be identified is a real video, Represents the total number of classifiers.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 9.

Images