Video detection method and device, intelligent terminal and computer readable storage medium

Abstract

This invention discloses a video detection method, apparatus, smart terminal, and computer-readable storage medium. The method includes: acquiring a file to be detected, which consists of several image frames and corresponding audio data; extracting initial optical flow features corresponding to the several image frames and initial voiceprint features corresponding to the audio data according to preset extraction rules; and determining the detection result corresponding to the file to be detected based on the initial optical flow features and initial voiceprint features. This invention can effectively improve the screening efficiency for videos containing fake content.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a video detection method, apparatus, smart terminal, and computer-readable storage medium. Background Technology

[0002] With the development of 5G technology and the rapid iteration of mobile phones, more and more users are watching video programs through video platforms. To attract viewers, some users resort to face-swapping technology and fake audio to spread false information and harm the interests of others. Common face-swapping technologies include DeepFake and Face2face, which use deep learning to replace faces. Their working mechanism is similar to translation models in natural language processing, using deep neural networks to detect facial movements of people in the original video and then synthesizing the facial information of the target person to make them perform the same movements. At the same time, the background and audio of the video remain unchanged, achieving a level of realism that is almost indistinguishable to the naked eye. Audio replacement mainly involves replacing keywords in the original audio using computer synthesis technology; or re-recording an entire audio segment to replace the audio in the original video.

[0003] Currently, the primary method for detecting fake videos is manual review. However, based on the DeepFake and Face2face technologies mentioned above, it's difficult for the naked eye to distinguish between real and fake videos. Furthermore, the barrier to entry for posting videos on video platforms is extremely low, resulting in a massive influx of video submissions every day. For example, on YouTube, approximately 300 hours of video are posted every minute—a figure that already exceeds the capacity for manual review and continues to grow. Therefore, a rapid and automated method for identifying fake videos is of great importance. Summary of the Invention

[0004] This invention provides a video detection method, apparatus, smart terminal, and computer-readable storage medium, aiming to solve the problem of low efficiency in detecting fake videos in the prior art.

[0005] To achieve the above objectives, the present invention provides a video detection method, comprising:

[0006] Obtain the file to be detected, which consists of several image frames and corresponding audio data;

[0007] According to the preset extraction rules, the initial optical flow features corresponding to several image frames and the initial voiceprint features corresponding to audio data are extracted respectively.

[0008] Based on the initial optical flow characteristics and initial acoustic signature characteristics, the detection result corresponding to the file to be detected is determined.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a video detection device, comprising:

[0010] The acquisition module is used to acquire the file to be detected, which consists of several image frames and corresponding audio data;

[0011] The extraction module is used to extract optical flow features corresponding to several image frames and voiceprint features corresponding to audio data according to preset extraction rules.

[0012] The determination module is used to determine the detection result corresponding to the file to be detected based on optical flow characteristics and acoustic characteristics.

[0013] In addition, to achieve the above objectives, the present invention also provides a smart terminal, which includes a memory, a processor, and a video detection program stored in the memory and executable on the processor. When the processor executes the video detection program, it implements the steps of the video detection method described above.

[0014] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a video detection program, which, when executed by a processor, implements the steps of the video detection method described above.

[0015] This invention provides a video detection method. It obtains initial optical flow features by extracting optical flow features from image frames in a video file, and obtains corresponding initial voiceprint features by extracting voiceprint features from audio data in the video file. Then, it determines whether the initial optical flow features and initial voiceprint features are fake, thereby determining the detection result for the file to be detected. Although fake videos and real videos have very small differences in a single frame, they differ in the movement of individual pixels. Fake videos cannot account for every pixel. Therefore, their optical flow features will have certain anomalies, thus determining whether the video is fake. Audio replacement mainly involves replacing keywords in the original audio using computer synthesis technology; or re-recording the entire audio segment to replace the audio in the original video. However, the voiceprint features of such audio are unstable. Therefore, based on the optical flow features and voiceprint features of the video file, it is possible to effectively determine whether the video file is fake. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the video detection method of the present invention;

[0017] Figure 2 This is an overall implementation flowchart of a preferred embodiment of the video detection method of the present invention;

[0018] Figure 3 This is a schematic diagram illustrating the specific classification process in a preferred embodiment of the video detection method of the present invention;

[0019] Figure 4This is a schematic diagram of a preferred embodiment of the smart terminal of the present invention;

[0020] Figure 5 This is a schematic diagram of the operating environment of a preferred embodiment of the smart terminal of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] The video detection method of the preferred embodiment of the present invention is implemented using a smart terminal as the execution subject. For example... Figure 1 As shown, the video detection method includes the following steps:

[0023] Step S100: The smart terminal acquires the file to be tested.

[0024] Specifically, when a video file needs to be detected, it is treated as the file to be detected and acquired, which can be done through methods such as reading or importing. A video file is essentially composed of several image frames and corresponding audio data. Each image frame consists of an image and its corresponding time point, while the audio consists of sampled signal points and their corresponding times; the two are mutually corresponding.

[0025] Furthermore, since this embodiment primarily targets tampering with faces and voices, simple landscape videos, such as documentaries about natural scenery, are excluded from the detection scope. Therefore, not all video files need to be included in the detection; files requiring detection can be filtered out through facial recognition within the videos. Figure 2 As shown, this process can be specifically described as follows:

[0026] The smart terminal obtains the video file to be reviewed and inputs the video file into a trained face recognition model for face recognition, thereby obtaining the individual face images and the corresponding number of faces in the video file;

[0027] If the number of faces is zero, the smart terminal determines that the video file is a valid file;

[0028] If the number of faces is greater than zero, the smart terminal determines that the video file is the file to be detected.

[0029] Specifically, the smart terminal first acquires all the video files to be reviewed, and then inputs each video file into a trained face recognition model. This face recognition model is mainly for detecting faces in the video files and can use recognition methods based on facial feature statistics, such as Hidden Markov Model (HMM) or methods based on Hausdorff distance. Current face recognition models are relatively mature, and the use of different face recognition methods does not affect the execution of this embodiment. Therefore, the various models will not be described in detail here.

[0030] After inputting the face recognition model, the system controls the face recognition model to perform face recognition on the video file, thereby extracting the face images present in the video file. If the number of face images is zero, it means that the video file does not contain people, and therefore, the video file is directly determined to be a qualified file; if the number of face images is greater than zero, it means that the video file contains faces, and therefore, it is determined to be a file to be detected.

[0031] In step S200, according to the preset extraction rules, the smart terminal extracts the initial optical flow features corresponding to several image frames and the initial voiceprint features corresponding to audio data.

[0032] Specifically, fake videos and camera-captured videos may be difficult to distinguish in a single frame, but they will differ in the movement of individual pixels. Camera-captured videos have smooth and natural light flow, but fake videos cannot capture every single pixel. Therefore, optical flow features can be used to discern subtle differences in pixel movement around a face in a video. Optical flow (or optic flow) is a concept related to motion detection of objects within a field of view. It describes the motion of an observed target, surface, or edge relative to the observer's movement. In this embodiment, the optical flow feature refers to the optical flow between image frames.

[0033] Furthermore, fake audio in the file to be detected is often achieved by replacing keywords in the original audio using computer synthesis technology; or by re-recording the entire audio segment to replace the audio in the original video. Therefore, it can be judged by the stability of the voiceprint features in a video audio segment.

[0034] In this embodiment, extraction rules for extracting initial optical flow features and initial voiceprint features are preset. According to the extraction rules, the smart terminal extracts the initial optical flow features corresponding to the image frame and the initial voiceprint features corresponding to the audio data, respectively.

[0035] For the initial optical flow features, this embodiment can employ a deep learning-based optical flow prediction algorithm (FlowNet), such as the Flow Convolutional Neural Networks (Flow-CNN) optical flow prediction algorithm, the PWC-Net optical flow prediction model, etc., or conventional optical flow prediction algorithms, such as the Lucas-Kanade optical flow prediction algorithm. Taking the deep learning-based optical flow prediction algorithm as an example, the initial optical flow features include a first optical flow feature and a second optical flow feature. In this embodiment, all image frames are first input into a trained optical flow prediction model. This model performs optical flow recognition on image frames adjacent to each corresponding time step to obtain the first optical flow feature. Simultaneously, according to a preset acquisition interval, prediction frames are determined within the image frames. For example, if the acquisition interval is one image frame, then the first image frame is acquired as the prediction frame, then the third image frame is acquired as the prediction frame, and so on, to obtain several prediction frames. Then, the prediction frames are input into the trained optical flow prediction model, which performs optical flow recognition on prediction frames adjacent to each corresponding time step to obtain the second optical flow feature.

[0036] For the initial voiceprint features, this embodiment can employ deep learning-based voiceprint feature extraction algorithms, such as Gaussian Mixture Model (GMM), or traditional voiceprint feature algorithms, such as algorithms for extracting Linear Prediction Cepstral Coefficients (LPCCs) and Mel-Frequency Cepstral Coefficients (MFCCs). It is worth noting that since fake audio is often obtained by subtracting or replacing elements from existing real audio, in this embodiment, the initial voiceprint features corresponding to the audio data are not extracted as a whole. Instead, the audio data is divided into multiple segments, resulting in segmented audio. Voiceprint extraction is then performed on each segment to obtain the initial voiceprint features corresponding to each segment.

[0037] This embodiment can be implemented using different models or algorithms for extracting optical flow features and for extracting voiceprint features.

[0038] Furthermore, since videos vary in length, ranging from a few seconds to tens of hours, extracting optical flow from every frame of every video file would result in a massive amount of data and be extremely time-consuming. For audio data, if a video is too short, the segmented audio may not meet the length requirement for extracting the initial voiceprint features; if it is too long, the differences in initial voiceprint features between the segmented audio may be smoothed out, making accurate identification impossible. Therefore, in this embodiment, the extraction rules for initial optical flow features and initial voiceprint features include not only the list extraction method but also image sub-rules and audio sub-rules.

[0039] Based on the initial optical flow features, image sub-rules are used to define the image frames for which optical flow extraction is required. First, according to the image sub-rules, the smart terminal selects target images from several image frames to generate a target image sequence.

[0040] For example, the image sub-rule further filters and selects the image frames used to calculate the first optical flow feature, limiting the type, size, and corresponding time of the extracted image frames. In a preferred image sub-rule of this embodiment, the starting frame is first determined based on the extracted face image. For example, if a face image originates from an image frame at time 0.1s, this image frame is used as the starting frame. Then, using the time corresponding to the starting frame as the starting time, image frames with the same preset sampling number are sequentially selected as the target frame set according to a preset first time interval. For example, if the first time interval is 1s and the preset sampling number is 10, then one image frame is selected as the target frame every 1s until the number of target frames is 10. Then, the starting frame and the target frame set are sorted according to the order of their corresponding times to generate the target image sequence.

[0041] Furthermore, when multiple face images exist, the earliest corresponding image frame from the source image frames can be selected as the starting frame. Alternatively, the image frame corresponding to each face image can be used as the starting frame. After obtaining multiple target image sequences, integration processing is performed based on the target frames and starting frames in the image sequences. For example, duplicate image frames are deleted, and consecutive image frames are stitched together. The integrated target image sequence is then output.

[0042] The target image sequence is input into a trained optical flow extraction model to extract optical flow features corresponding to the target image sequence.

[0043] Specifically, since the process of extracting optical flow features corresponding to all image frames has been described above, a similar method can be used to extract optical flow features from the target image sequence, so it will not be repeated here.

[0044] Based on the initial voiceprint features, audio sub-rules are set. These sub-rules define the audio signals for which voiceprint extraction is required. According to these sub-rules, the smart terminal selects the target audio sequence from the audio data.

[0045] Specifically, the audio sub-rules may include the length of the audio sequence composed of selected audio signals, the position of the selected first audio signal, etc. In this embodiment, a preferred audio sub-rule is as follows: based on the image frames corresponding to each face image, determine a starting frame among several image frames; using the time corresponding to the starting frame as the starting time, extract the sequence from the audio data according to a preset second time interval to obtain the target audio sequence. For example, if the second time interval is one minute, then using the time corresponding to the starting frame, for example, 0.1s, as the starting time, and then selecting a one-minute-long audio sequence from the audio data as the target audio sequence.

[0046] The target audio sequence is input into a trained voiceprint extraction model to extract voiceprint features, resulting in initial voiceprint features corresponding to the target audio sequence. Since the process of extracting initial voiceprint features for all audio data has already been described above, a similar method can be used to extract voiceprint features from the target audio sequence, so it will not be repeated here.

[0047] Step S300: Based on the initial optical flow characteristics and initial voiceprint characteristics, the smart terminal determines the detection result corresponding to the file to be detected.

[0048] Specifically, based on the initial optical flow features and initial voiceprint features, the smart terminal can determine the detection result corresponding to the file to be detected. Taking the first to twelfth image frames as an example, regarding the initial optical flow features, firstly, based on the first optical flow features between the first and second image frames, the predicted optical flow features between the first and third image frames are predicted. Then, the predicted optical flow features are compared with the second optical flow features. If there is a large deviation between the predicted optical flow features and the second optical flow features, the optical flow features of the file to be detected are considered abnormal.

[0049] For the initial voiceprint features, the similarity of the voiceprint features corresponding to each segment of audio is calculated. If the similarity value of a certain voiceprint feature with other voiceprint features is less than a preset similarity threshold, it is determined that the voiceprint features of the file to be detected are abnormal. In addition, it can also be judged based on the difference in similarity between voiceprint features. For example, if the audio data is divided into 3 segments, and the extracted voiceprint features are A, B, C, and D, if the similarity between A and B and C is 90% and 95% respectively, while the similarity between D and A, B, and C is 70%, 60%, and 65%, the difference is significant, and it can also be considered that the voiceprint features are abnormal.

[0050] If either the initial voiceprint feature or the initial optical flow feature is abnormal, the file to be tested is determined to be an abnormal file. If both are normal, the file to be tested is determined to be a qualified file. Furthermore, different weight values ​​can be set for the initial voiceprint feature and the initial optical flow feature, based on the degree of abnormality of the initial voiceprint feature and the presence of an abnormality in either feature. For example, if the weight value of the initial voiceprint feature is higher, then even if the initial voiceprint feature is normal but the initial optical flow feature is abnormal, the file will still be determined to be a normal file. This setting can be adjusted according to user needs.

[0051] Furthermore, the above methods of determination are rather crude and simplistic. To improve the accuracy of the determination, this embodiment uses a trained feature detection model to determine the detection result corresponding to the file to be detected. The specific process is as follows:

[0052] A10. Input the initial optical flow features and initial voiceprint features into the trained feature detection model for feature classification to obtain the classification result corresponding to the file to be detected.

[0053] Specifically, the feature detection model is trained on a base model using a large number of pre-labeled optical flow and voiceprint features. Optional feature detection models include residual neural networks (RNNs) or convolutional neural networks (CNNs), such as the ResNet-50 model in residual neural networks and the VGGish model in CNNs. The pre-labeled information indicates whether the optical flow feature and the voiceprint are real or fake. After training, the initial optical flow and voiceprint features are input into the trained feature detection model, which classifies the input features based on the previous training results. Then, based on the classification results, the classification result for the file to be detected is determined. This determination can be achieved by adjusting the weights of the features based on their similarity and differences.

[0054] Furthermore, to reduce the computational cost of classifying the initial optical flow features and initial voiceprint features, in the first real-time method of this embodiment, the feature detection model includes a dimensionality reduction model and a feature classification model. The dimensionality reduction model is used to reduce the dimensionality of the initial optical flow features and initial voiceprint features. The initial optical flow features are input into the trained dimensionality reduction model, which then performs dimensionality reduction on the initial optical flow features to obtain dimensionality-reduced optical flow features; similarly, the initial voiceprint features are input into the dimensionality reduction model, which performs dimensionality reduction on the initial voiceprint features to obtain dimensionality-reduced voiceprint features. The dimensionality reduction model can employ principal component analysis (PCA), low variance filtering, missing values ​​ratio, or other models, and can be adjusted according to user needs.

[0055] Furthermore, whether it is the initial optical flow features and initial voiceprint features obtained based on all image frames and audio data, or the initial optical flow features and initial voiceprint features obtained based on the starting frame determined by the face image, there is a certain correlation between the two in terms of time and information acquisition. Therefore, the initial optical flow features and initial voiceprint features can be classified as a whole.

[0056] In the second real-time method of this embodiment, the feature detection model includes an optical flow hidden layer, a voiceprint hidden layer, a fully connected layer, and a classifier. Initial optical flow features are input into the optical flow hidden layer to obtain the target optical flow features corresponding to the initial optical flow features; and initial voiceprint features are input into the voiceprint hidden layer to obtain the target voiceprint features corresponding to the initial voiceprint features. A hidden layer refers to a neural network that abstracts the features of the input data to another dimension to represent its more abstract characteristics. Hidden layers can be convolutional layers in convolutional neural networks, encoders and decoders in recurrent neural networks, etc. The specific selection can be based on the characteristics of the input data, i.e., the initial optical flow features and the initial voiceprint features, as well as the training effect. Figure 3 As shown, the hidden layers used are either the hidden layers of the Bi-LSTM model in a Long Short-Term Memory (LSTM) artificial neural network or the hidden layers of the Bi-GRU model in a Gated Recurrent Unit (GRU). Therefore, the target optical flow features and target voiceprint features are more abstract features than the initial optical flow features and initial voiceprint features. The target optical flow features and target voiceprint features are then input into a fully connected layer, which fuses the target optical flow features and target voiceprint features to generate fused features. The fused features are then input into the classifier mentioned above, which calculates a first probability value for a normal classification result and a second probability value for an abnormal classification result corresponding to the fused features. Based on the first probability value and the second probability value, the classification result corresponding to the file to be detected is determined.

[0057] In addition, such as Figure 3 As shown, the input values ​​of the hidden layer can be dimension-reduced optical flow features and dimension-reduced audio features. Combining the first and second implementation methods of this embodiment, a third combined implementation method can be obtained. Since the content is repetitive, it will not be described again here.

[0058] A20. When the classification result is normal, the document to be tested is determined to be a qualified document.

[0059] Specifically, if the classification result is normal, it means that the file to be tested has not shown any face replacement or false audio, and therefore the file to be tested is determined to be a qualified file.

[0060] A30. When the classification result is abnormal, the file to be detected is determined to be an abnormal file.

[0061] Specifically, if the classification result is normal, it indicates that the file to be detected has face replacement and false audio, thus determining that the file to be detected is an abnormal file.

[0062] Furthermore, if Figure 4 As shown, based on the above video detection method, the present invention also provides a video detection device 100, which includes:

[0063] The acquisition module 110 is used to acquire the file to be detected, wherein the file to be detected consists of several image frames and corresponding audio data;

[0064] The extraction module 120 is used to extract the optical flow features corresponding to the image frame and the voiceprint features corresponding to the audio data according to the preset extraction rules.

[0065] The determination module 130 is used to determine the detection result corresponding to the file to be detected based on optical flow characteristics and acoustic characteristics.

[0066] The acquisition module 110 is preceded by a coarse detection module, which is specifically used for:

[0067] Obtain the video file and input it into the trained face recognition model to perform face recognition, thereby obtaining the face images and the corresponding number of faces in the video file;

[0068] If the number of faces is zero, the video file is considered a valid file.

[0069] If the number of faces is greater than zero, then the video file is determined to be the file to be detected.

[0070] The extraction rules include image sub-rules; the extraction module 120 includes an optical flow feature extraction module, which includes:

[0071] The first selection unit is used to select the target image in the image frame according to the image sub-rules and generate the target image sequence;

[0072] The first extraction unit is used to input the target image sequence into the trained optical flow extraction model to extract optical flow and obtain the optical flow features corresponding to the target image sequence.

[0073] Specifically, the first selection unit is used for:

[0074] Based on each face image, determine the starting frame among several image frames;

[0075] Taking the time corresponding to the starting frame as the starting time, and according to the preset first time interval, image frames with the same preset sampling number are selected sequentially as the target frame set;

[0076] The starting frame and the target frame sets are sorted according to the chronological order of their corresponding times to generate a target image sequence.

[0077] The extraction rules include audio sub-rules; the extraction module 120 also includes a voiceprint feature extraction module, which includes:

[0078] The second selection unit is used to select the target audio sequence in the audio data according to the audio sub-rules;

[0079] The second extraction unit is used to input the target image sequence into the trained optical flow extraction model to extract optical flow and obtain the optical flow features corresponding to the target image sequence.

[0080] Specifically, the second selection unit is used for:

[0081] Based on the image frames corresponding to each face image, determine the starting frame in the image frame;

[0082] Taking the time corresponding to the starting frame as the starting time, the sequence in the audio data is extracted according to the preset second time interval to obtain the target audio sequence.

[0083] The determining module includes:

[0084] The classification unit is used to input the initial optical flow features and initial voiceprint features into the trained feature detection model for feature classification, and obtain the classification result corresponding to the file to be detected.

[0085] The determination unit is used to determine that the document to be tested is a qualified document when the classification result is normal; or,

[0086] If the classification result is abnormal, the file to be detected is determined to be an abnormal file.

[0087] The feature detection model includes a dimensionality reduction model and a feature classification model; the classification unit includes:

[0088] The dimensionality reduction subunit is used to input the initial optical flow features into the trained dimensionality reduction model for dimensionality reduction processing, thereby obtaining the dimensionality-reduced optical flow features; and

[0089] The initial voiceprint features are input into the dimensionality reduction model for dimensionality reduction processing to obtain the dimensionality-reduced voiceprint features.

[0090] The first classification subunit is used to input the dimension-reduced optical flow features and dimension-reduced voiceprint features into the feature classification model for feature classification and to determine the classification result corresponding to the file to be detected.

[0091] The feature detection model includes an optical flow hidden layer, a voiceprint hidden layer, a fully connected layer, and a classifier; the classification unit includes:

[0092] The feature extraction subunit is used to input the initial optical flow features into the optical flow hidden layer to obtain the target optical flow features corresponding to the initial optical flow features; and

[0093] The initial voiceprint features are input into the voiceprint hidden layer to obtain the target voiceprint features corresponding to the initial voiceprint features;

[0094] The feature fusion subunit is used to input the target optical flow features and target acoustic signature features into the fully connected layer for feature fusion to generate fused features;

[0095] The computational subunit is used to input the fused features into the classifier to calculate the first probability value of the fused feature corresponding to the classification result being normal and the second probability value of the corresponding classification result being abnormal.

[0096] The second classification subunit is used to determine the classification result corresponding to the file to be detected based on the first probability value and the second probability value.

[0097] Furthermore, such as Figure 5 As shown, based on the above video detection method, the present invention also provides a smart terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the smart terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0098] In some embodiments, the memory 20 may be an internal storage unit of the smart terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the smart terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the smart terminal. Furthermore, the memory 20 may include both internal and external storage units of the smart terminal. The memory 20 is used to store application software and various types of data installed on the smart terminal, such as program code for installing the smart terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a video detection program 40, which can be executed by the processor 10 to implement the video detection method of this application.

[0099] In some embodiments, processor 10 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as performing video detection methods.

[0100] In some embodiments, display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 30 is used to display information on the smart terminal and to display a visual user interface. The components 10-30 of the smart terminal communicate with each other via a system bus.

[0101] In one embodiment, the above-described video detection method is implemented when the processor 10 executes the video detection program 40 in the memory 20.

[0102] The present invention also provides a computer-readable storage medium storing a video detection program, which, when executed by a processor, implements the steps of the video detection method described above.

[0103] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0104] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A video detection method, characterized in that, include: Obtain the file to be detected, which consists of several image frames and corresponding audio data; According to the preset extraction rules, the initial optical flow features corresponding to the plurality of image frames and the initial voiceprint features corresponding to the audio data are extracted respectively. Based on the initial optical flow features and the initial acoustic signature features, the detection result corresponding to the file to be detected is determined; The step of determining the detection result corresponding to the file to be detected based on the initial optical flow features and the initial voiceprint features includes: inputting the initial optical flow features and the initial voiceprint features into a trained feature detection model for feature classification to obtain the classification result corresponding to the file to be detected; when the classification result is normal, the file to be detected is determined to be a qualified file; or, when the classification result is abnormal, the file to be detected is determined to be an abnormal file. The feature detection model includes an optical flow hidden layer, a voiceprint hidden layer, a fully connected layer, and a classifier. The step of inputting the initial optical flow features and the initial voiceprint features into the trained feature detection model for feature classification to obtain the classification result corresponding to the file to be detected includes: inputting the initial optical flow features into the optical flow hidden layer, where the optical flow hidden layer abstracts the initial optical flow features to another dimension space to obtain the target optical flow features corresponding to the initial optical flow features; inputting the initial voiceprint features into the voiceprint hidden layer, where the voiceprint hidden layer abstracts the initial voiceprint features to another dimension space to obtain the target voiceprint features corresponding to the initial voiceprint features; inputting the target optical flow features and the target voiceprint features into the fully connected layer for feature fusion to generate fused features; inputting the fused features into the classifier to calculate a first probability value for a normal classification result and a second probability value for an abnormal classification result; and determining the classification result corresponding to the file to be detected based on the first probability value and the second probability value. The classification results are used to indicate whether the file to be detected contains face replacement or false audio. The step of determining the detection result corresponding to the file to be detected based on the initial optical flow features and the initial voiceprint features further includes: predicting the predicted optical flow features between the first image frame and the third image frame based on the first optical flow features between the first image frame and the second image frame; comparing the predicted optical flow features and the second optical flow features; if there is a deviation between the predicted optical flow features and the second optical flow features, then the optical flow features of the file to be detected are considered to be abnormal; the step of obtaining the second optical flow features includes determining the predicted frames in the image frames according to a preset acquisition interval; inputting the predicted frames into a trained optical flow prediction model; the optical flow prediction model performs optical flow recognition on the predicted frames adjacent to each corresponding time to obtain the second optical flow features; The extraction rules include image sub-rules; the step of extracting the initial optical flow features corresponding to the plurality of image frames according to the preset extraction rules includes: selecting a target image from the plurality of image frames according to the image sub-rules to generate a target image sequence; inputting the target image sequence into a trained optical flow extraction model for optical flow extraction to obtain optical flow features corresponding to the target image sequence; the image sub-rules also filter and screen the image frames used to calculate the first optical flow features, limiting the type, size, and corresponding time of the extracted image frames.

2. The method according to claim 1, characterized in that, Before obtaining the file to be detected, the method further includes: Acquire a video file and input the video file into a trained face recognition model for face recognition to obtain each face image and the corresponding number of faces in the video file; If the number of faces is greater than zero, then the video file is determined to be the file to be detected.

3. The method according to claim 2, characterized in that, The step of selecting target images from the plurality of image frames according to the image sub-rules and generating a target image sequence includes: Based on the individual face images, determine the starting frame among the plurality of image frames; Taking the time corresponding to the starting frame as the starting time, and according to the preset first time interval, image frames with the same preset sampling number are selected sequentially as the target frame set. The starting frame and the target frame set are sorted according to the chronological order of their corresponding times to generate a target image sequence.

4. The method according to claim 2, characterized in that, The extraction rules include audio sub-rules; the step of extracting the initial voiceprint features corresponding to the audio data according to the preset extraction rules includes: According to the audio sub-rules, the target audio sequence in the audio data is selected; The target audio sequence is input into a trained voiceprint extraction model to extract voiceprint features, thereby obtaining the initial voiceprint features corresponding to the target audio sequence.

5. The method according to claim 4, characterized in that, The step of selecting the target audio sequence from the audio data according to the audio sub-rule includes: Based on the image frames corresponding to each of the face images, determine the starting frame in the image frames; Taking the time corresponding to the starting frame as the starting time, the sequence in the audio data is extracted according to the preset second time interval to obtain the target audio sequence.

6. The method according to claim 1, characterized in that, The feature detection model includes a dimensionality reduction model and a feature classification model; the step of inputting the initial optical flow features and the initial voiceprint features into the trained feature detection model for feature classification to obtain the classification result corresponding to the file to be detected includes: The initial optical flow features are input into a trained dimensionality reduction model for dimensionality reduction processing to obtain dimensionality-reduced optical flow features; and The initial voiceprint features are input into the dimensionality reduction model for dimensionality reduction processing to obtain dimensionality-reduced voiceprint features; The reduced optical flow features and the reduced voiceprint features are input into the feature classification model for feature classification to determine the classification result corresponding to the file to be detected.

7. A video detection device, characterized in that, include: The acquisition module is used to acquire the file to be detected, which consists of several image frames and corresponding audio data; The extraction module is used to extract the initial optical flow features corresponding to the plurality of image frames and the initial voiceprint features corresponding to the audio data according to preset extraction rules. The determination module is used to determine the detection result corresponding to the file to be detected based on the initial optical flow features and the initial acoustic features; The video detection device is further configured to input the initial optical flow features and the initial voiceprint features into a trained feature detection model for feature classification to obtain the classification result corresponding to the file to be detected; when the classification result is normal, the file to be detected is determined to be a qualified file; or, when the classification result is abnormal, the file to be detected is determined to be an abnormal file. The feature detection model includes an optical flow hidden layer, a voiceprint hidden layer, a fully connected layer, and a classifier; the video detection device is also used to input the initial optical flow features into the optical flow hidden layer, and the optical flow hidden layer abstracts the initial optical flow features to another dimension space to obtain the target optical flow features corresponding to the initial optical flow features; The initial voiceprint features are input into the voiceprint hiding layer, which abstracts the initial voiceprint features into another dimension space to obtain the target voiceprint features corresponding to the initial voiceprint features. The target optical flow features and the target voiceprint features are input into the fully connected layer for feature fusion to generate fused features; the fused features are input into the classifier to calculate a first probability value for a normal classification result and a second probability value for an abnormal classification result; based on the first probability value and the second probability value, the classification result corresponding to the file to be detected is determined; the classification result is used to indicate whether the file to be detected contains face replacement or audio fraud. The video detection device is further configured to predict the optical flow features between the first image frame and the third image frame based on the first optical flow features between the first image frame and the second image frame, compare the predicted optical flow features with the second optical flow features, and if there is a deviation between the predicted optical flow features and the second optical flow features, then the optical flow features of the file to be detected are considered to be abnormal; the step of obtaining the second optical flow features includes determining the predicted frames in the image frames according to a preset acquisition interval; inputting the predicted frames into a trained optical flow prediction model, wherein the optical flow prediction model performs optical flow recognition on the predicted frames adjacent to each corresponding time to obtain the second optical flow features; The extraction rules include image sub-rules; the video detection device is further configured to select target images from the plurality of image frames according to the image sub-rules, generate a target image sequence; input the target image sequence into a trained optical flow extraction model for optical flow extraction, and obtain optical flow features corresponding to the target image sequence; The image sub-rules further filter and select the image frames used to calculate the first optical flow feature, limiting the type, size, and corresponding time of the extracted image frames.

8. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a video detection program stored in the memory and executable on the processor. When the processor executes the video detection program, it implements the steps of the video detection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a video detection program, which, when executed by a processor, implements the steps of the video detection method as described in any one of claims 1-6.

Images