Multimodal feature-based counterfeit video discrimination method, apparatus and device, and medium

Abstract

The invention discloses a forged video discrimination method, device and equipment based on multi-modal features and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: extracting a picture frame and an audio from a video; identifying a target from the picture frame, and further determining picture features of the target; identifying audio features of the audio; dynamically outputting a first weight about time synchronization, a second weight about time-space consistency and a third weight about detecting whether different level features are abnormal according to the image features and the audio features of the image frames and the audios, and extracting time sequence features of the video; and judging whether the video is forged according to the first weight, the second weight, the third weight and the time sequence feature. By analyzing the time synchronization of the picture frame and the audio, analyzing the time-space consistency of the picture frame and the audio and detecting whether different level features of the picture frame and the audio are abnormal, whether the video is forged or not can be comprehensively and accurately detected, and the judgment accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for distinguishing forged videos based on multimodal features. Background Art

[0002] Although existing multimodal forged video detection methods can determine the authenticity of videos, they are difficult to effectively deal with different types of video content. At the same time, existing methods usually only focus on the characteristics of a single modality or simple audio and video synchronization detection, lacking comprehensive analysis capabilities. The limitations of existing technologies make it difficult to detect complex and dynamic forged videos with high detection accuracy and high misjudgment rate. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment and medium for distinguishing forged videos based on multimodal features, so as to improve the accuracy of distinguishing whether a video has been forged.

[0004] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for distinguishing fake videos based on multimodal features, the method comprising the following steps:

[0005] Extract picture frames and audio from the video;

[0006] Recognize the target from the picture frame, and then determine the picture features of the target; wherein the picture features include the detection features and motion features of the target in the picture frame;

[0007] Identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio;

[0008] Dynamically outputting a first weight of the time synchronization between the picture frame and the audio, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture features and the audio features, and extracting time series features of the video;

[0009] Whether the video is forged is determined based on the first weight, the second weight, the third weight and the time series feature.

[0010] In some embodiments, the identifying a target from the picture frame comprises the following steps:

[0011] Extracting a plurality of the targets from the continuous picture frames using a target detection model;

[0012] Determining the image features of the target comprises the following steps:

[0013] Identifying the detection features of the target using a target recognition model;

[0014] The target tracking model is used to identify the trajectory data of the target as the motion feature; wherein the trajectory data includes the center of mass position, velocity, acceleration, motion direction, trajectory curvature and trajectory length of the target.

[0015] In some embodiments, the identifying the trajectory data of the target as the motion feature using the target tracking model comprises the following steps:

[0016] Calculating the centroid position of the target over time based on a bounding box detecting the target;

[0017] The calculation formula for the center of mass position is:

[0018]

[0019] Among them, (x c ,y c ) represents the centroid position, represents the time series of the centroid position changing with the timestamp t; x, y represent the horizontal and vertical coordinates of the lower left corner of the bounding box, and w, h represent the width and height of the bounding box;

[0020] Calculating the instantaneous speed of the target and the corresponding speed magnitude;

[0021] The calculation formula of the instantaneous speed is:

[0022]

[0023] in, represents the instantaneous speed, Δt is the time interval between adjacent picture frames;

[0024] The calculation formula of the speed is:

[0025]

[0026] Among them, v t Indicates the magnitude of the speed, represents the instantaneous velocity in the x direction, represents the instantaneous velocity in the y direction;

[0027] Calculating the instantaneous acceleration of the target and the corresponding acceleration magnitude;

[0028] The calculation formula of the instantaneous acceleration is:

[0029]

[0030] in, represents the instantaneous acceleration;

[0031] The calculation formula for the acceleration magnitude is:

[0032]

[0033] Among them, a t represents the magnitude of the acceleration, represents the magnitude of acceleration in the x direction, Indicates the magnitude of acceleration in the y direction;

[0034] Calculating the azimuth of the target as the moving direction;

[0035] The calculation formula of the azimuth angle is:

[0036]

[0037] Among them, θ t represents the azimuth; arctan 2 is a two-parameter inverse tangent function;

[0038] Calculating the trajectory curvature of the target based on three consecutive centroid positions;

[0039] The calculation formula of the trajectory curvature is:

[0040]

[0041] Among them, κ t represents the trajectory curvature; Δθ t =θ t+1 -θ t ,Δs t is the path length;

[0042] Calculating the trajectory length of the target based on the time series of the center of mass position;

[0043] The calculation formula of the trajectory length is:

[0044]

[0045] Wherein, L represents the length of the trajectory; N represents the total number of timestamps.

[0046] In some embodiments, identifying the audio features of the audio comprises the following steps:

[0047] Denoising and filtering the audio, thereby adjusting the sampling rate of the audio to the sampling rate of an automatic speech recognition model, and then using the automatic speech recognition model to recognize the text features from the audio;

[0048] Inputting the audio into a sentiment classification model to obtain the sentiment feature;

[0049] The audio is input into a background sound classification model to obtain the background sound features.

[0050] In some embodiments, the step of inputting the audio into a sentiment classification model to obtain the sentiment feature comprises the following steps:

[0051] Extracting a pitch sequence from the audio using an audio analysis library; wherein the pitch sequence includes a pitch average, a pitch variance, and a pitch extreme value;

[0052] Calculating the square value of the audio signal amplitude in each preset time window in the audio to obtain an energy sequence statistically; wherein the energy sequence includes average energy, energy variance and energy extreme value;

[0053] Determine the total duration of the audio and detect the number of sentences or words in the audio, and then determine the speaking rate; wherein the total duration of the audio, the number of sentences or words in the audio and the speaking rate are used as a duration sequence;

[0054] The pitch sequence, the energy sequence and the duration sequence are input into the emotion classification model to obtain the emotion feature.

[0055] In some embodiments, dynamically outputting a first weight of the time synchronization between the picture frame and the audio according to the picture features and the audio features, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal comprises the following steps:

[0056] Combining the vectorized picture features and the vectorized audio features and inputting them into an adaptive feature extraction module;

[0057] The adaptive feature extraction module is used to dynamically output the first weight, the second weight, and the third weight according to the changing frequency of the picture feature or the changing frequency of the audio feature.

[0058] In some embodiments, the step of determining whether the video is forged based on the first weight, the second weight, the third weight, and the time series feature comprises the following steps:

[0059] Inputting the time series features into an audio and picture linkage analysis module, and then using the audio and picture linkage analysis module to output a linkage feature score;

[0060] Inputting the time series features into a multimodal fusion module based on spatiotemporal features, and then using the multimodal fusion module based on spatiotemporal features to output a spatiotemporal consistency score;

[0061] Inputting the time series features into a multi-level anomaly detection module, and then using the multi-level anomaly detection module to output an anomaly detection score;

[0062] multiplying the first weight by the linkage feature score, multiplying the second weight by the spatiotemporal consistency score, and multiplying the third weight by the anomaly detection score, and then accumulating the three multiplication results as a final score;

[0063] If the final score is greater than or equal to a preset threshold, the video is judged to be forged;

[0064] If the final score is less than the preset threshold, the video is determined to be not forged.

[0065] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a forged video identification device based on multimodal features, the device comprising:

[0066] A video preprocessing unit, used for extracting picture frames and audio from the video;

[0067] A picture analysis unit, used to identify a target from the picture frame, and then determine the picture features of the target; wherein the picture features include detection features and motion features of the target in the picture frame;

[0068] An audio analysis unit, configured to identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio;

[0069] A feature processing unit, configured to dynamically output a first weight of the time synchronization between the picture frame and the audio, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture feature and the audio feature, and extract the time series feature of the video;

[0070] A video identification unit is used to identify whether the video has been forged based on the first weight, the second weight, the third weight and the time series feature.

[0071] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned method for distinguishing forged videos based on multimodal features is implemented.

[0072] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for distinguishing forged videos based on multimodal features is implemented.

[0073] The embodiments of the present application include at least the following beneficial effects:

[0074] The present application can extract picture frames and audio from a video; identify a target from a picture frame, and then determine the picture features of the target; wherein the picture features include detection features and motion features of the target in the picture frame; identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio; dynamically output a first weight of the picture frame and audio regarding time synchronization, a second weight regarding spatiotemporal consistency, and a third weight regarding detecting whether features at different levels are abnormal based on the picture features and audio features, and extract the time series features of the video; determine whether the video has been forged based on the first weight, the second weight, the third weight, and the time series features. The present application can comprehensively and accurately detect whether a video has been forged by analyzing the time synchronization between the picture frame and the audio, analyzing the spatiotemporal consistency between the picture frame and the audio, and detecting whether features at different levels of the picture frame and the audio are abnormal, thereby improving the accuracy of the determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 A schematic diagram of a flow chart of a method for identifying forged videos based on multimodal features provided in an embodiment of the present application;

[0077] Figure 2 An example flow chart for determining the authenticity of a video provided in an embodiment of the present application;

[0078] Figure 3 An example flow chart of video preprocessing provided in an embodiment of the present application;

[0079] Figure 4 An example flowchart of picture preprocessing provided in an embodiment of the present application;

[0080] Figure 5 An example flow chart of motion pattern analysis provided in an embodiment of the present application;

[0081] Figure 6An example flow chart of text feature vectorization of audio provided in an embodiment of the present application;

[0082] Figure 7 An example flow chart of audio emotion and intonation feature vectorization provided in an embodiment of the present application;

[0083] Figure 8 An example flow chart of a method for identifying forged videos based on multimodal features provided in an embodiment of the present application;

[0084] Fig. 9 A schematic diagram of the structure of a forged video identification device based on multimodal features provided in an embodiment of the present application;

[0085] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0087] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0088] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0090] Before describing the embodiments of the present application in detail, some related technologies involved in the embodiments of the present application are first described. The related technologies involved in the embodiments of the present application are as follows:

[0091] Existing forged video detection solutions include:

[0092] 1. Single-mode video feature detection: Existing forged video detection technologies often use single-mode feature analysis of video frames, relying on methods such as image texture and optical flow analysis. Typical technologies include: Convolutional neural network (CNN) detection: Extract pixel-level and edge features of video frames to identify forgery traces such as abnormal lighting and unnatural expressions. Optical flow analysis: Detect the motion trajectory between consecutive frames in the video and identify forgeries by judging the physical rationality of the motion.

[0093] 2. Single-mode audio feature detection: Audio forgery detection is usually performed by analyzing the consistency of audio content and background sound. Typical technologies include: Automatic speech recognition (ASR, used to convert language in audio into text): Analyze the speech content in the audio and perform semantic comparison with the video screen. Audio feature analysis: Extract pitch, energy and background sound to determine whether the emotional expression and background sound match the video content.

[0094] 3. Detection based on simple multimodal fusion: Some existing technologies simply combine audio and video features to improve detection accuracy, but the fusion depth is insufficient. The main methods include: Audio and video synchronization detection: Analyze the synchronization of the speaker's lip movement and voice, and detect whether the audio and video are aligned. Simple feature fusion: Combine video image and audio emotional features, but lack deep spatiotemporal modeling and multi-level analysis capabilities, and it is difficult to deal with complex forged videos.

[0095] Deficiencies of existing technology:

[0096] Although existing multimodal forged video detection methods can combine audio and video features, their fusion methods are often relatively simple and lack flexible adaptive adjustment capabilities, making it difficult to effectively deal with different types of video content. At the same time, these methods usually only focus on the features of a single modality or simple audio and video synchronization detection, and lack the ability to comprehensively analyze multi-level anomalies (such as pixel, frame level, and semantic level). The limitations of existing technologies make it difficult to detect complex and dynamic forged videos with low detection accuracy and high misjudgment rate.

[0097] Therefore, the embodiments of the present application provide a method, device, equipment and medium for distinguishing forged videos based on multimodal features. The technical solution of the present application includes: extracting picture frames and audio from the video; identifying the target from the picture frame, and then determining the picture features of the target; wherein the picture features include the detection features and motion features of the target in the picture frame; identifying the audio features of the audio; wherein the audio features include the text features, emotional features and background sound features of the audio; dynamically outputting the first weight of the picture frame and the audio about the time synchronization, the second weight about the spatiotemporal consistency and the third weight about detecting whether the features at different levels are abnormal according to the picture features and the audio features, and extracting the time series features of the video; judging whether the video has been forged according to the first weight, the second weight, the third weight and the time series features. The present application can comprehensively and accurately detect whether the video has been forged by analyzing the time synchronization between the picture frame and the audio, analyzing the spatiotemporal consistency between the picture frame and the audio, and detecting whether the features at different levels of the picture frame and the audio are abnormal, thereby improving the accuracy of the judgment.

[0098] The embodiments of the present application provide a method, device, equipment and medium for distinguishing fake videos based on multimodal features, and relate to the field of artificial intelligence technology. The method, device, equipment and medium for distinguishing fake videos based on multimodal features provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but are not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0099] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0100] Reference Figure 1 The embodiment of the present application provides a method for identifying forged videos based on multimodal features. The method may include but is not limited to steps S100 to S140, which are as follows:

[0101] S100: extracting picture frames and audio from the video.

[0102] It can be understood that this embodiment can extract continuous picture frames and audio from the video.

[0103] S110: Identify a target from the picture frame, and then determine the picture features of the target; wherein the picture features include detection features and motion features of the target in the picture frame.

[0104] Furthermore, the step of identifying a target from the picture frame in S110 includes the following steps S111:

[0105] S111: extracting a plurality of the targets from the continuous picture frames using a target detection model.

[0106] As another further implementation, determining the target image features in S110 includes the following steps S112-S113:

[0107] S112: Identify the detection features of the target using a target recognition model;

[0108] S113: Using a target tracking model to identify trajectory data of the target as the motion feature; wherein the trajectory data includes the center of mass position, velocity, acceleration, motion direction, trajectory curvature and trajectory length of the target.

[0109] More specifically, S113 may include the following steps:

[0110] Calculating the centroid position of the target over time based on a bounding box detecting the target;

[0111] The calculation formula of the center of mass position is:

[0112]

[0113] Among them, (x c ,y c ) represents the centroid position, represents the time series of the centroid position changing with the timestamp t; x, y represent the horizontal and vertical coordinates of the lower left corner of the bounding box, and w, h represent the width and height of the bounding box;

[0114] Calculating the instantaneous speed of the target and the corresponding speed magnitude;

[0115] The calculation formula of the instantaneous speed is:

[0116]

[0117] in, represents the instantaneous speed, Δt is the time interval between adjacent picture frames;

[0118] The calculation formula of the speed is:

[0119]

[0120] Among them, v t Indicates the magnitude of the speed, represents the instantaneous velocity in the x direction, represents the instantaneous velocity in the y direction;

[0121] Calculating the instantaneous acceleration of the target and the corresponding acceleration magnitude;

[0122] The calculation formula of the instantaneous acceleration is:

[0123]

[0124] in, represents the instantaneous acceleration;

[0125] The calculation formula for the acceleration magnitude is:

[0126]

[0127] Among them, a t represents the magnitude of the acceleration, represents the magnitude of acceleration in the x direction, Indicates the magnitude of acceleration in the Y direction;

[0128] Calculating the azimuth of the target as the moving direction;

[0129] The calculation formula of the azimuth angle is:

[0130]

[0131] Among them, θ t represents the azimuth; arctan 2 is a two-parameter inverse tangent function;

[0132] Calculating the trajectory curvature of the target based on three consecutive centroid positions;

[0133] The calculation formula of the trajectory curvature is:

[0134]

[0135] Among them, κ t represents the trajectory curvature; Δθ t =θ t+1 -θ t ,Δs t is the path length;

[0136] Calculating the trajectory length of the target based on the time series of the center of mass position;

[0137] The calculation formula of the trajectory length is:

[0138]

[0139] Wherein, L represents the length of the trajectory; N represents the total number of timestamps.

[0140] S120: Identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio.

[0141] Further, S120 may include the following steps S121 to S123:

[0142] S121: performing noise reduction and filtering on the audio, thereby adjusting the sampling rate of the audio to the sampling rate of an automatic speech recognition model, and then using the automatic speech recognition model to recognize the text features from the audio;

[0143] S122: inputting the audio into a sentiment classification model to obtain the sentiment feature;

[0144] S123: Input the audio into a background sound classification model to obtain the background sound features.

[0145] More specifically, S122 may include the following steps S1221 to S1224:

[0146] S1221: extracting a pitch sequence from the audio using an audio analysis library; wherein the pitch sequence includes a pitch average, a pitch variance, and a pitch extreme value;

[0147] S1222: Calculate the square value of the audio signal amplitude in each preset time window in the audio to obtain an energy sequence by statistics; wherein the energy sequence includes average energy, energy variance and energy extreme value;

[0148] S1223: Determine the total duration of the audio and detect the number of sentences or words in the audio, and then determine the speaking rate; wherein the total duration of the audio, the number of sentences or words in the audio and the speaking rate are used as a duration sequence;

[0149] S1224: Input the pitch sequence, the energy sequence and the duration sequence into the emotion classification model to obtain the emotion feature.

[0150] S130: Dynamically output the first weight of the picture frame and the audio regarding time synchronization, the second weight regarding time-space consistency, and the third weight regarding detecting whether features at different levels are abnormal based on the picture features and the audio features, and extract the time series features of the video.

[0151] Furthermore, dynamically outputting a first weight of the time synchronization between the picture frame and the audio, a second weight of the spatiotemporal consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture features and the audio features in S130 includes the following steps S131-S132:

[0152] S131: Combining the vectorized picture features and the vectorized audio features and inputting them into an adaptive feature extraction module;

[0153] S132: Utilize the adaptive feature extraction module to dynamically output the first weight, the second weight, and the third weight according to the changing frequency of the picture feature or the changing frequency of the audio feature.

[0154] S140: Determine whether the video is forged based on the first weight, the second weight, the third weight and the time series feature.

[0155] Reference Figure 2 , S140 may further include the following steps S141 to S146:

[0156] S141: inputting the time series feature into an audio and picture linkage analysis module, and then using the audio and picture linkage analysis module to output a linkage feature score;

[0157] S142: inputting the time series features into a multimodal fusion module based on spatiotemporal features, and then using the multimodal fusion module based on spatiotemporal features to output a spatiotemporal consistency score;

[0158] S143: inputting the time series features into a multi-level anomaly detection module, and then using the multi-level anomaly detection module to output an anomaly detection score;

[0159] S144: multiplying the first weight by the linkage feature score, multiplying the second weight by the spatiotemporal consistency score, and multiplying the third weight by the anomaly detection score, and then accumulating the three multiplication results as a final score;

[0160] S145: If the final score is greater than or equal to a preset threshold, the video is determined to be forged;

[0161] S146: If the final score is less than the preset threshold, the video is determined to be not forged.

[0162] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0163] The purpose of this embodiment is to provide a method for detecting forged videos based on multimodal features, which can accurately determine the authenticity of videos by dynamically adjusting the feature extraction strategies of audio and video, analyzing the linkage features, spatiotemporal consistency, and multi-level anomalies between audio and video. This method can effectively detect a variety of forged features in forged videos, including audio and video time dislocation, incoherent motion trajectories, and logical inconsistencies at the semantic level, and has excellent forgery detection capabilities, especially in complex scenes. The following are the functions and corresponding technical features implemented by this embodiment:

[0164] 1. Adaptive dynamic feature extraction.

[0165] Analyze the visual and audio features of the video and dynamically adjust the feature extraction weights of subsequent modules, so that the system can adaptively select feature focuses for different video content.

[0166] Technical features adopted: Enhance the flexibility and accuracy of detection by adaptively adjusting the feature extraction strategy.

[0167] 2. Linked analysis of audio and video.

[0168] Detect the time synchronization of audio and video, evaluate the matching of lip shape and speech, and the coordination of movement and sound, and generate a linkage feature score.

[0169] Technical features used: Analyze the consistency of audio and video linkage and detect misalignment in forged videos.

[0170] 3. Multimodal fusion based on spatiotemporal features.

[0171] Through spatiotemporal modeling, the motion features of consecutive picture frames are aligned with the audio features to detect the spatiotemporal consistency of video content.

[0172] Technical features adopted: Capturing the inconsistency of video in time and space through the fusion of temporal and spatial features.

[0173] 4. Multi-level anomaly detection.

[0174] Detect video anomaly features at the pixel, frame, and semantic levels to identify lighting inconsistencies, motion anomalies, and logical inconsistencies in audio and video content.

[0175] Technical features adopted: Multi-level detection improves detection accuracy, especially anomaly identification at the semantic level.

[0176] 5. Weighted joint decision strategy.

[0177] The scores output by each module are combined according to the weights set by the adaptive module to ultimately determine the authenticity of the video.

[0178] Technical features adopted: Weighted decision strategy combined with multimodal features to achieve accurate judgment of complex videos.

[0179] Specifically, the steps of the scheme of this embodiment include: data preprocessing, picture feature extraction, audio feature extraction, and multimodal model discrimination. The following is a detailed description:

[0180] 1. Data preprocessing.

[0181] This process extracts synchronized audio and picture frames from the video, such as Figure 3 shown.

[0182] 1.1 Video decoding: Use the decoding tool to load the video.

[0183] 1.2 Audio Extraction: Extract audio files from video and record timestamps at the same time.

[0184] 1.3 Frame extraction: traverse the video and extract the images frame by frame.

[0185] 1.4 Timestamp synchronization: align audio and picture frames according to timestamps.

[0186] 2. Picture feature extraction.

[0187] The process includes: picture preprocessing, target motion pattern analysis, and continuous picture understanding.

[0188] 2.1 The image preprocessing process includes first using the target detection model to extract the main target from the continuous image frames, and then applying the target recognition model to analyze the detection features Z (type, identity, color, etc.) of the target, such as a thin face, a person wearing sunglasses, a blue taxi, a bare tree, etc. For details, please refer to Figure 4 .

[0189] 2.2 Movement pattern analysis, the process is as follows Figure 5 As shown, the DeepSORT algorithm is used to track a specific target and obtain trajectory data. Specifically:

[0190] a. Centroid calculation.

[0191] For each bounding box, calculate the centroid location:

[0192]

[0193] Time series: records the position sequence of each centroid and the corresponding timestamp t.

[0194] b. Speed ​​calculation.

[0195] Instantaneous speed:

[0196]

[0197] Wherein, Δt is the time interval between adjacent frames.

[0198] c. Speed:

[0199]

[0200] d. Acceleration calculation.

[0201] Instantaneous acceleration:

[0202]

[0203] Acceleration magnitude:

[0204]

[0205] e. Direction of movement.

[0206] Direction angle calculation:

[0207]

[0208] Among them, arctan 2 is a two-parameter inverse tangent function that returns the direction angle of a vector.

[0209] f. Curvature of trajectory.

[0210] Curvature approximation: For three consecutive points Curvature κ t It can be approximated as:

[0211]

[0212] Among them, Δθ t =θ t+1 -θ t ,Δs t is the path length.

[0213] g. Track length.

[0214] Cumulative path length:

[0215]

[0216] 2.3 Motion information vectorization.

[0217] The extracted motion features of specific targets are combined into a feature vector f for subsequent analysis, where Z is the detection feature output by the target recognition model.

[0218] f=[v t ,a t ,θ t ,L,Z];

[0219] 2.4 Combine the motion vectors of all targets in the picture, and the feature vector is F picture :

[0220] F picture =[f1,f2,f3...f n ].

[0221] 3. Audio feature extraction.

[0222] Automatic speech recognition (ASR), emotional intonation analysis, and background sound recognition technology are used to extract audio features.

[0223] 3.1 ASR text conversion process Figure 6 As shown. Audio preprocessing includes noise reduction and filtering to remove background noise, filtering, and adjusting the sampling rate to the standard format required by the model. After that, the ASR model generates text and maps the text into a vector V text .

[0224] 3.2 Sentiment and intonation analysis feature vectorization, such as Figure 7The audio analysis library Praat is used to extract pitch sequences to reflect the speaker's intonation changes. Its statistical features include the average pitch, pitch variance, and pitch extremes, which reflect the overall pitch level, fluctuation, and extreme changes of the speech, respectively, and are denoted as v pitch :

[0225] v pitch =[mean pitch, pitch variance, min pitch, max pitch];

[0226] By calculating the square value of the audio signal amplitude in each short-time window, its statistical features include average energy, energy variance and energy extreme value, which respectively reflect the loudness, fluctuation and extreme changes of the speech, denoted as v energy :

[0227] v energy =[mean energy,energy variance,min energy,max energy];

[0228] The speech rate is estimated by analyzing the total duration of the audio file and the number of sentences or words detected in the speech. Its characteristics include the number of words or syllables pronounced per second, the pause time between words (which helps to judge emotions), and the duration of the entire speech, denoted as v duration :

[0229] v duration =[speaking rate,pause duration,total duration];

[0230] Then v pitch 、v energy 、v duration After the combination, it is sent to the sentiment classification model to obtain the one-hot encoding of different emotions and their probability distribution b emotion . (For example, positive is [1,0,0], neutral is [0,1,0], negative is [0,0,1], and the model outputs the probability distribution of sentiment categories as [0.1,0.7,0.2]).

[0231] 3.3 Background sound recognition feature vectorization.

[0232] Extract background sound from the audio and feed it into the background sound classification model to identify the type of background sound. The output category is represented as a one-hot encoded vector v bg_audio :

[0233] For example, “traffic noise” can be represented as [1,0,0,0] (assuming four types of background noise);

[0234] Probability distribution: The probability predicted by the model is like [0.7, 0.1, 0.15, 0.05].

[0235] 3.4 Feature vector fusion.

[0236] The final eigenvector can be expressed as v audio =[v text ,v emotion ,v bg_audio ].

[0237] 4. Multimodal feature discrimination, that is, how to use the above features to determine whether the video is forged (synthesized) or real.

[0238] Connect the image and audio features in time series Multimodal , then sent to Figure 8 The module shown determines whether the video is fake or real, where:

[0239] X Multimodal (t) = [F picture (t),B audio (t)], t=1,2,...;

[0240] 4.1 Adaptive feature extraction module.

[0241] This module performs a preliminary analysis of X Multimodal (t) Determine the main features of the time series (motion trajectory, emotional changes, etc.), and then dynamically configure the feature weights in the subsequent modules based on the type and frequency of the input features. If there are many motion features in the picture features, increase the weight of the motion features in the subsequent modules; if the emotions or background sounds in the audio features change significantly, increase the weight of the audio emotions and background sounds in the subsequent modules. Then output the feature extraction strategy and feature vector output of the time series.

[0242] 4.2 Audio and picture linkage analysis module.

[0243] The purpose of this module is to detect the time synchronization between audio and picture, determine the temporal consistency of multimodal features, capture the phenomenon of dislocation or incoordination in forged videos, use the attention mechanism, analyze the synchronization of picture and audio features, evaluate the coordination between action and sound, and output the linkage feature score S link .

[0244] 4.3 Multimodal fusion module based on spatiotemporal features.

[0245] The goal of this module is to analyze the spatiotemporal consistency of the video, integrate the time series features of the picture frames and audio, use 3D convolutional networks to model spatiotemporal features, detect forged features between consecutive frames (such as abnormal motion trajectories), and output a spatiotemporal consistency score through multimodal fusion and weighting mechanism to judge the coherence of the video in time and space, and output the spatiotemporal consistency score S spatial .

[0246] 4.4 Multi-level anomaly detection module.

[0247] This module detects anomalies through features at different levels (pixel level, frame level, semantic level) to identify forgery traces in video details. Pixel-level anomaly detection: Use frequency domain analysis to detect inconsistencies in light, shadow, texture or details of each frame to identify forgery traces. Frame-level anomaly detection: Detect the continuity of motion trajectories between consecutive frames to determine whether they conform to physical motion laws. Semantic-level anomaly detection: Combine the ASR text content of the audio with the semantics of the scene to determine whether the two are consistent and output anomaly detection score S abnormal For example, the video footage may show an outdoor scene, while the audio content discusses an indoor environment.

[0248] 4.5 Weighted joint decision strategy.

[0249] The output scores of each module are weighted and combined for judgment, as shown below:

[0250] S final =w1·S link +w2·S spatial +w3·S abnormal ;

[0251] Set the threshold: for the final score S final Set the preset threshold τ, if S final ≥τ, the video is judged as "fake"; otherwise, it is "real".

[0252] Dynamic weight adjustment: Dynamically adjust each weight through the adaptive module, and adjust the emphasis of the weight according to the different input video content (such as audio-dominated, picture-dominated). For example: when audio is more critical, increase the weight of the first weight w1 so that the audio and video linkage analysis accounts for a larger proportion in the final judgment. When the motion characteristics are more significant, increase the weight of the second weight w2 so that the spatial and temporal consistency has a greater impact on the judgment. If there are many abnormal details, increase the weight of the third weight w3 to make the abnormal detection results more influential.

[0253] In summary, the technical features and beneficial effects of this embodiment are as follows:

[0254] Technical feature 1: Adaptive dynamic feature extraction module.

[0255] Technical features: This module analyzes the multimodal features of the video (image, audio) and dynamically adjusts the feature extraction strategy to ensure that subsequent modules can focus on analyzing different types of video content. Specifically, the adaptive dynamic feature extraction module can identify facial features, background sounds, emotional changes, motion trajectories, etc. in the video, and adjust the weights and analysis focus of each module based on these features.

[0256] Beneficial effects: By introducing an adaptive dynamic feature extraction module, this embodiment can adaptively optimize the feature extraction strategy according to the characteristics of the video, thereby significantly improving the accuracy and efficiency of forgery detection. Compared with the prior art, this module can more flexibly process various types of video content (such as dialogue videos, sports videos, etc.), while the existing methods usually use fixed feature extraction strategies and cannot dynamically adapt to different content, resulting in low detection accuracy or high misjudgment rate.

[0257] Technical feature 2: Audio and video linkage analysis module.

[0258] Technical features: Through dynamic time warping (DTW) or attention mechanism, the audio and video features are synchronously analyzed in the time dimension to detect the linkage features of audio and video in the video, such as the synchronization of voice and lip shape, and the matching degree of action and sound. This module generates linkage feature scores by analyzing the time synchronization of audio and video, and identifies the phenomenon of audio and video misalignment or inconsistency in forged videos.

[0259] Beneficial effects: This module enhances the accuracy of forgery detection by capturing the temporal consistency of audio and video, especially the matching of lip shape and voice, and the coordination of action and sound. Compared with the prior art, this embodiment can not only detect forgery features of a single modality (such as video only or audio only), but also detect linkage inconsistencies between audio and video. This capability is generally lacking in the prior art, giving this embodiment a clear advantage in dealing with complex forged videos.

[0260] Technical feature 3: Multimodal fusion module based on spatiotemporal features.

[0261] Technical features: This embodiment uses a 3D convolutional network or video Transformer model to achieve the fusion of image and audio features in time and space, and detect the spatiotemporal consistency of consecutive frames. This module aligns and models the motion features of the video screen (such as target trajectory, acceleration, etc.) with the audio features (such as emotional changes, background sound, etc.) on the time axis, thereby identifying the incoherence of forged videos in the spatiotemporal dimension.

[0262] Beneficial effects: Through the spatiotemporal consistency analysis of multimodal features, this embodiment can capture anomalies in forged videos in the continuous time dimension. Compared with the existing technology, this embodiment has stronger time and space perception capabilities. Most of the existing technologies lack in-depth modeling between consecutive frames, making it difficult to identify the more subtle time and space inconsistency features in forged videos. The multimodal fusion module of this embodiment overcomes this defect and greatly improves the detection effect of complex forged videos.

[0263] Technical feature 4: Multi-level anomaly detection mechanism.

[0264] Technical features: The multi-level anomaly detection mechanism conducts in-depth analysis of video features at the pixel level, frame level, and semantic level. The module detects light and shadow or texture anomalies in video frames at the pixel level, analyzes the consistency of motion trajectories and physical motion laws at the frame level, and detects the logical consistency of audio and video content at the semantic level (such as the consistency of voice content with video scenes).

[0265] Beneficial effects: This module achieves higher detection accuracy based on existing detection methods through multi-level and multi-dimensional anomaly detection. In particular, semantic-level detection can ensure the consistency of video content at a high level of logic. This capability is extremely scarce in the prior art, which can usually only detect low-level features (such as pixel or frame-level features) and is difficult to effectively judge the semantic consistency of the video. This embodiment achieves integrated detection from low-level details to high-level logic through a multi-level anomaly detection mechanism, and has extremely strong robustness and accuracy in complex scenarios.

[0266] Technical feature 5: Multimodal weighted joint decision strategy.

[0267] Technical features: This embodiment proposes a multi-modal weighted joint decision strategy, which combines the audio and video linkage feature score, spatiotemporal consistency score, and multi-level anomaly detection score according to dynamic weights to form the final forgery detection score. The weights of each module are adjusted through the adaptive module, so that the system can flexibly adjust the decision strategy according to different content types.

[0268] Beneficial effects: The weighted joint decision strategy ensures that the detection results of each module play a reasonable role in the final decision. Compared with the fixed weight mechanism of the prior art, the weighted mechanism of this embodiment can better adapt to the characteristics of different video contents. By adaptively adjusting the weights, this embodiment shows higher flexibility and accuracy in complex video forgery detection tasks, effectively reduces the false positive rate, and improves the detection capability of the overall system.

[0269] Reference Fig. 9The embodiment of the present application further provides a forged video identification device based on multimodal features, which can implement the forged video identification method based on multimodal features. The device includes:

[0270] A video preprocessing unit, used for extracting picture frames and audio from the video;

[0271] A picture analysis unit, used to identify a target from the picture frame, and then determine the picture features of the target; wherein the picture features include detection features and motion features of the target in the picture frame;

[0272] An audio analysis unit, configured to identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio;

[0273] A feature processing unit, configured to dynamically output a first weight of the time synchronization between the picture frame and the audio, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture feature and the audio feature, and extract the time series feature of the video;

[0274] A video identification unit is used to identify whether the video has been forged based on the first weight, the second weight, the third weight and the time series feature.

[0275] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0276] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the forged video identification method based on multimodal features when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0277] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0278] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0279] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0280] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the forged video identification method based on multimodal features in the embodiments of this application;

[0281] Input / output interface 1003, used to implement information input and output;

[0282] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0283] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0284] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0285] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the forged video identification method based on multimodal features is implemented.

[0286] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0287] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0288] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0289] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0290] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0291] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0292] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0293] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0294] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0295] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0296] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0297] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0298] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A forged video identification method based on multimodal features, characterized in that: The method comprises the following steps: Extract picture frames and audio from the video; Recognize the target from the picture frame, and then determine the picture features of the target; wherein the picture features include the detection features and motion features of the target in the picture frame; Identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio; Dynamically outputting a first weight of the time synchronization between the picture frame and the audio, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture features and the audio features, and extracting time series features of the video; Whether the video is forged is determined based on the first weight, the second weight, the third weight and the time series feature.

2. The method for identifying forged videos based on multimodal features according to claim 1, characterized in that: The step of identifying a target from the picture frame comprises the following steps: Extracting a plurality of the targets from the continuous picture frames using a target detection model; Determining the image features of the target comprises the following steps: Identifying the detection features of the target using a target recognition model; The target tracking model is used to identify the trajectory data of the target as the motion feature; wherein the trajectory data includes the center of mass position, velocity, acceleration, motion direction, trajectory curvature and trajectory length of the target.

3. The method for identifying forged videos based on multimodal features according to claim 1, characterized in that: The method of using a target tracking model to identify the trajectory data of the target as the motion feature comprises the following steps: Calculating the centroid position of the target over time based on a bounding box detecting the target; The calculation formula for the center of mass position is: Among them, (x c ,y c ) represents the centroid position, represents the time series of the centroid position changing with the timestamp t; x, y represent the horizontal and vertical coordinates of the lower left corner of the bounding box, and w, h represent the width and height of the bounding box; Calculating the instantaneous speed of the target and the corresponding speed magnitude; The calculation formula of the instantaneous speed is: in, represents the instantaneous speed, Δt is the time interval between adjacent picture frames; The calculation formula of the speed is: Among them, v t Indicates the magnitude of the speed, represents the instantaneous velocity in the x direction, represents the instantaneous velocity in the y direction; Calculating the instantaneous acceleration of the target and the corresponding acceleration magnitude; The calculation formula of the instantaneous acceleration is: in, represents the instantaneous acceleration; The calculation formula for the acceleration magnitude is: Among them, a t represents the magnitude of the acceleration, represents the magnitude of acceleration in the x direction, Indicates the magnitude of acceleration in the y direction; Calculating the azimuth of the target as the moving direction; The calculation formula of the azimuth angle is: Among them, θ t represents the azimuth; arctan 2 is a two-parameter inverse tangent function; Calculating the trajectory curvature of the target based on three consecutive centroid positions; The calculation formula of the trajectory curvature is: Among them, κ t represents the trajectory curvature; Δθ t =θ t+1 -θ t ,Δs t is the path length; Calculating the trajectory length of the target based on the time series of the center of mass position; The calculation formula of the trajectory length is: Wherein, L represents the length of the trajectory; N represents the total number of timestamps.

4. The method for identifying forged videos based on multimodal features according to claim 1, characterized in that: The step of identifying the audio features of the audio comprises the following steps: Denoising and filtering the audio, thereby adjusting the sampling rate of the audio to the sampling rate of an automatic speech recognition model, and then using the automatic speech recognition model to recognize the text features from the audio; Inputting the audio into a sentiment classification model to obtain the sentiment feature; The audio is input into a background sound classification model to obtain the background sound features.

5. The method for identifying forged videos based on multimodal features according to claim 4, characterized in that: The step of inputting the audio into a sentiment classification model to obtain the sentiment feature comprises the following steps: Extracting a pitch sequence from the audio using an audio analysis library; wherein the pitch sequence includes a pitch average, a pitch variance, and a pitch extreme value; Calculating the square value of the audio signal amplitude in each preset time window in the audio to obtain an energy sequence statistically; wherein the energy sequence includes average energy, energy variance and energy extreme value; Determine the total duration of the audio and detect the number of sentences or words in the audio, and then determine the speaking rate; wherein the total duration of the audio, the number of sentences or words in the audio and the speaking rate are used as a duration sequence; The pitch sequence, the energy sequence and the duration sequence are input into the emotion classification model to obtain the emotion feature.

6. The method for identifying forged videos based on multimodal features according to claim 1, characterized in that: The method of dynamically outputting a first weight of time synchronization between the picture frame and the audio, a second weight of spatiotemporal consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture features and the audio features comprises the following steps: Combining the vectorized picture features and the vectorized audio features and inputting them into an adaptive feature extraction module; The adaptive feature extraction module is used to dynamically output the first weight, the second weight, and the third weight according to the changing frequency of the picture feature or the changing frequency of the audio feature.

7. The method for identifying forged videos based on multimodal features according to any one of claims 1 to 6, characterized in that: The step of determining whether the video has been forged according to the first weight, the second weight, the third weight and the time series feature comprises the following steps: Inputting the time series features into an audio and picture linkage analysis module, and then using the audio and picture linkage analysis module to output a linkage feature score; Inputting the time series features into a multimodal fusion module based on spatiotemporal features, and then using the multimodal fusion module based on spatiotemporal features to output a spatiotemporal consistency score; Inputting the time series features into a multi-level anomaly detection module, and then using the multi-level anomaly detection module to output an anomaly detection score; multiplying the first weight by the linkage feature score, multiplying the second weight by the spatiotemporal consistency score, and multiplying the third weight by the anomaly detection score, and then accumulating the three multiplication results as a final score; If the final score is greater than or equal to a preset threshold, the video is judged to be forged; If the final score is less than the preset threshold, the video is determined to be not forged.

8. A forged video identification device based on multimodal features, characterized in that: The device comprises: A video preprocessing unit, used for extracting picture frames and audio from the video; A picture analysis unit, used to identify a target from the picture frame, and then determine the picture features of the target; wherein the picture features include detection features and motion features of the target in the picture frame; An audio analysis unit, configured to identify audio features of the audio; wherein the audio features include text features, emotional features, and background sound features of the audio; A feature processing unit, configured to dynamically output a first weight of the time synchronization between the picture frame and the audio, a second weight of the time-space consistency, and a third weight of detecting whether features at different levels are abnormal according to the picture feature and the audio feature, and extract the time series feature of the video; A video identification unit is used to identify whether the video has been forged based on the first weight, the second weight, the third weight and the time series feature.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the forged video identification method based on multimodal features as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying a forged video based on multimodal features according to any one of claims 1 to 7 is implemented.

Images