Video content counterfeiting detection method and device, equipment and medium

Abstract

The invention relates to the technical field of image detection, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a video content counterfeiting detection method, device, equipment and medium, and the method comprises the steps: obtaining a to-be-detected video stream, carrying out the frame sampling and standardization processing of the to-be-detected video stream, and generating a to-be-detected video sequence; performing visual double-branch feature extraction on the to-be-detected video sequence to obtain a universal visual feature, a local counterfeit feature and an audio feature; fusing the universal visual features, the local counterfeit features and the audio features to obtain audio and video consistency features; mapping the audio and video consistency feature to a low-dimensional decoupling space and carrying out feature decoupling to obtain a target counterfeit feature; and performing expansion convolution on the target counterfeiting feature to obtain a target counterfeiting probability sequence, and determining authenticity of the to-be-detected video stream according to the target counterfeiting probability sequence. According to the invention, the content counterfeiting detection efficiency and detection accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method, apparatus, device, and medium for detecting video content forgery. Background Technology

[0002] With the rapid development of generative AI technologies such as generative adversarial networks and decoupled models, the quality of deepfake content has significantly improved, posing a serious challenge to forgery detection on user-generated content (UGC) platforms. Current mainstream deepfake detection methods mainly rely on supervised learning, requiring a large amount of labeled data to train the model. However, these methods face two major bottlenecks in real-world scenarios: first, the diverse distribution of UGC (such as lighting, pose, and compression artifacts) limits the model's generalization ability; second, the rapid iteration of new forgery technologies makes supervised models easily obsolete.

[0003] For example, in the healthcare field, deepfake technology can be used by criminals to forge critical documents such as medical images and medical records. In telemedicine diagnosis, if the images uploaded by patients are deepfake, doctors' diagnoses based on erroneous images will seriously mislead treatment plans and endanger patients' lives and health. Current mainstream deepfake detection methods rely on supervised learning, requiring a large amount of labeled data to train the model. However, in medical scenarios, the image features of different diseases vary greatly, and the distribution of forgeries is highly diverse. For example, images taken by different devices differ in resolution, color, etc., and lighting conditions also affect image quality. These factors limit the generalization ability of detection models based on supervised learning.

[0004] For example, in the fintech industry, where data authenticity and security are extremely critical—such as in online identity authentication and electronic contract signing—deepfake technology can be used to forge user facial images, fingerprints, or contract documents. Criminals use deepfakes to generate fake user facial videos, bypassing online identity authentication systems to conduct illegal transfers and loans, causing significant losses to financial institutions and users. General forgery detection technologies struggle to detect subtle forgeries in financial transactions. Their feature decoupling process lacks sufficient constraint on forgery-specific information, leading to the loss of key detection clues, posing a serious challenge to deepfake detection in fintech scenarios.

[0005] Therefore, improving the efficiency and accuracy of content forgery detection has become an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for detecting video content forgery, the main purpose of which is to solve the problems of low efficiency and inaccuracy in content forgery detection.

[0007] Firstly, to achieve the above objectives, the present invention provides a video content forgery detection method, comprising: The video stream to be detected is acquired, and frame sampling and standardization processing are performed on the video stream to be detected to generate a video sequence to be detected. Visual dual-branch feature extraction is performed on the video sequence to be detected to obtain the corresponding general visual features, local forgery features and audio features; Temporal attention fusion is performed on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features; The audio-video consistency features are mapped to a preset low-dimensional decoupling space, and the audio-video consistency features mapped to the low-dimensional decoupling space are decoupled to obtain multiple target forgery features. The target forgery features are subjected to feature dilation convolution to obtain a target forgery probability sequence, and the authenticity of the video stream to be detected is determined based on the target forgery probability sequence.

[0008] Secondly, the present invention also provides a video content forgery detection device, comprising: The video sequence preprocessing module is used to acquire the video stream to be detected, perform frame sampling and standardization processing on the video stream to be detected, and generate the video sequence to be detected. The dual-branch feature extraction module is used to perform visual dual-branch feature extraction on the video sequence to be detected, and obtain the corresponding general visual features, local forgery features and audio features. The multi-feature fusion module is used to perform temporal attention fusion on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features. The forgery feature decoupling module is used to map the audio-video consistency features to a preset low-dimensional decoupling space, and to perform feature decoupling on the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features; The video forgery detection module is used to perform feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence, and determine the authenticity of the video stream to be detected based on the target forgery probability sequence.

[0009] Thirdly, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the video content forgery detection method described above.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the video content forgery detection method described above.

[0011] In this embodiment of the invention, a phased video preprocessing technique significantly improves the operational efficiency and model performance of computer vision tasks. It reduces computational load while preserving action temporal features, eliminates feature shifts caused by device differences, and increases video analysis throughput. The dual-branch architecture effectively captures the global authenticity of the video and accurately locates pixel-level forgery traces, extracting local forgery features. Simultaneously, it combines features such as Mel-frequency spectrum and fundamental frequency of the audio signal to form cross-modal feature complementarity, significantly improving detection robustness and accuracy, and providing an efficient and reliable method for identifying forged content. Through a temporal cross-attention fusion mechanism, it achieves efficient collaboration and consistency in audio and video feature construction. The model can adaptively capture temporal synchronization anomalies between audio and video, improving detection accuracy in forgery detection tasks. Through the decoupling design of spatial and temporal dimensions, it achieves accurate separation and efficient representation of forgery features. By capturing temporal misalignments in audio and video, it constructs temporal forgery features and supports modular extension to detect unknown forgery types, significantly enhancing the model's generalization ability and robustness. Through the feature dilation convolution mechanism, it significantly improves the temporal modeling ability and probability prediction accuracy of video forgery detection, avoiding the loss of local information caused by fixed receptive fields in traditional methods. Temporal smoothing processing effectively suppresses inter-frame noise interference, improves the stability of probability sequences, and enhances detection accuracy and efficiency. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a video content forgery detection method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video content forgery detection method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a process for extracting visual features from visual content based on a preset visual dual branch, according to an embodiment of the present invention. Figure 4 This is a schematic diagram of a video content forgery detection device according to an embodiment of the present invention; Figure 5 A schematic diagram of an electronic device for implementing a video content forgery detection method according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of an electronic device that implements a video content forgery detection method according to an embodiment of the present invention.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] This application provides a method for detecting video content spoofing. The execution subject of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the device provided in this application: a server, a terminal, or other electronic equipment. In other words, the video content spoofing detection method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] This invention discloses a method for detecting video content forgery, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the video stream to be detected through the client. A phased video preprocessing technique significantly improves the efficiency and model performance of computer vision tasks, reducing computation while preserving action temporal features, eliminating feature shifts caused by device differences, and increasing video analysis throughput. In the dual-branch architecture, the global authenticity of the video is effectively captured, and pixel-level forgery traces are accurately located. Local forgery features are extracted, and combined with features such as Mel-frequency spectrum and fundamental frequency of the audio signal, cross-modal feature complementarity is formed, significantly improving detection robustness and accuracy, and providing an efficient and reliable method for identifying forged content. Through a time-cross-attention fusion mechanism, efficient collaborative and consistent modeling of audio and video features is achieved, enabling self-regulation. This invention adapts to capturing temporal synchronization anomalies between audio and video, improving detection accuracy in forgery detection tasks. Through decoupling design of spatial and temporal dimensions, it achieves precise separation and efficient representation of forgery features. It constructs temporal forgery features by capturing temporal misalignments in audio and video, supporting modular extension to detect unknown forgery types, significantly enhancing the model's generalization ability and robustness. The feature dilation convolution mechanism significantly improves the temporal modeling capability and probability prediction accuracy of video forgery detection, avoiding the local information loss caused by fixed receptive fields in traditional methods. Temporal smoothing effectively suppresses inter-frame noise interference, improving the stability of the probability sequence and increasing detection accuracy and efficiency. Finally, the target forgery probability output is fed back to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0019] Reference Figure 2The diagram shown is a flowchart illustrating a video content forgery detection method according to an embodiment of the present invention. In this embodiment, the video content forgery detection method includes: S1. Obtain the video stream to be detected, perform frame sampling and standardization processing on the video stream to be detected, and generate the video sequence to be detected.

[0020] In this embodiment of the invention, the video stream to be detected refers to a continuous video data sequence that needs to be analyzed, processed, or detected, and is usually transmitted or stored in a real-time or non-real-time manner; it can be historical video recorded by a monitoring system, live content transmitted over the network, or video files stored on a local device.

[0021] In detail, the hardware interface of the capture card is accessed directly through the device SDK to obtain raw video frames. The raw data, which is usually in uncompressed RGB or YUV format, is compressed and encoded (such as H.264 or H.265) through hardware encoders (such as NVIDIA NVENC) or software libraries (such as x264) to reduce bandwidth usage and adapt to the transmission protocol.

[0022] Furthermore, a multimedia framework can be used to parse the container formats of video files such as MP4, AVI, and MOV, extract the video track data, decode the compressed video frames (such as H.264 encoded I / P / B frames) into processable pixel data (such as BGR and RGB formats), ensure the temporal consistency between frames, and thus obtain the video stream to be detected.

[0023] For example, in the healthcare field, to detect forged content in surgical videos or medical images, surgical footage is captured in real time by an endoscope camera with an integrated 5G module. After being encoded and compressed, the footage is transmitted to a cloud analysis platform via an encrypted channel that meets medical data security standards. The platform parses the video stream, extracts pixel-level motion features for each frame using optical flow, and detects abnormal operations (such as unauthorized personnel entering the camera or sudden changes in instrument trajectory) using a pre-trained model. Simultaneously, blockchain technology is used to verify the hash value, comparing the metadata (such as device ID and timestamp) of the newly captured video with on-chain records to identify traces of tampering.

[0024] For example, in the fintech field, for scenarios involving remote account opening or electronic contract signing, real-time video streams of users are captured from the camera of a mobile app. Low-latency transmission protocols are used, combined with liveness detection technologies (such as blink frequency and micro-expression analysis) to generate dynamic biometric vectors. Simultaneously, NLP is used to parse user voice commands and perform temporal alignment detection with lip movements in the video to identify AI face-swapping or voice cloning attacks. In securities trading monitoring, real-time market data streams are pulled from exchange APIs. By transcribing the anchor's voice and combining it with OCR-recognized subtitle information, a multimodal content fingerprint database is constructed. Cosine similarity algorithms are then used to compare historical real video clips to quickly locate forged market manipulation statements.

[0025] In this embodiment of the invention, the step of performing frame sampling and normalization processing on the video stream to be detected to generate a video sequence to be detected includes: The video stream to be detected is subjected to uniform frame sampling in the time dimension to obtain a key video frame sequence; Each key video frame in the key video frame sequence is subjected to resolution normalization processing to obtain a normalized video frame sequence. The standardized video frame sequence is subjected to color space conversion to obtain a color-unified video frame sequence; The color-uniform video frame sequence is batch-processed to obtain the video sequence to be detected.

[0026] In detail, a time-interval equidistant frame extraction strategy is adopted to extract key frames from a continuous video stream. Based on the video's frame rate (e.g., 30 FPS) and a preset sampling interval (e.g., 1 frame per second), a timestamp threshold for each frame is calculated. All frames in the video stream are traversed, and only frames whose timestamps match the threshold are retained to generate a key video frame sequence.

[0027] For example, a 10-minute 30FPS video (18,000 frames in total) can be sampled at 1 frame per second, and only 600 frames can be retained as keyframes. The data volume is compressed to 1 / 30 of the original video. By reducing redundant frames, the computational load is reduced, while the temporal dynamic features of the video are preserved. This is suitable for scenarios that require the analysis of action continuity, such as behavior recognition and anomaly detection.

[0028] Furthermore, for potential resolution differences (such as 720P, 1080P) in the keyframe sequence, a bilinear interpolation algorithm is used for uniform scaling to determine the target resolution (such as 224×224 pixels). Then, the width and height of each frame image are scaled proportionally. If the original resolution is higher than the target value, the number of pixels is reduced by downsampling. If it is lower than the target value, the pixel value is supplemented by interpolation to ensure that all frames have the same spatial dimension.

[0029] To eliminate color deviations under different devices or lighting conditions, the RGB color space frame is converted into a standardized color space (such as YUV or grayscale), the RGB three-channel values ​​are separated, and then the new channel values ​​are calculated using a linear transformation formula (such as the weighting coefficient in YUV conversion).

[0030] For example, when converting to grayscale, the system assigns weights of 0.299, 0.587, and 0.114 to the R, G, and B channels, respectively. After weighted summation, a single-channel grayscale value is obtained, which can reduce color interference and highlight image texture and structural features. It is especially suitable for tasks that are sensitive to changes in illumination, such as object detection and face recognition.

[0031] Furthermore, the standardized single-frame images are combined into batch processing sequences in chronological order to improve the parallel computing efficiency of the model. The sequence is divided into multiple subsets according to the preset batch size (e.g., 32 frames). The frames in each subset are temporally continuous and have the same dimension. For example, after the 600-frame key sequence is divided into batches of 32, 18 complete batches and 1 remaining batch (containing 12 frames) are generated. Sequences that are not large enough to be batched are filled with zeros. This allows multiple frames of data to be processed at once, making full use of the parallel computing power of the GPU and significantly improving the inference speed (the measured efficiency is improved by 5-8 times).

[0032] In this embodiment of the invention, the phased video preprocessing technology significantly improves the running efficiency and model performance of computer vision tasks. In the time dimension, the uniform frame sampling strategy compresses redundant video frames into a key frame sequence, reducing the amount of computation while preserving the temporal features of the action. Resolution standardization and color space conversion eliminate feature shifts caused by device differences by unifying image size and color representation. The batch processing mechanism converts single-frame input into parallel data blocks, improving the throughput of video analysis.

[0033] S2. Visual dual-branch feature extraction is performed on the video sequence to be detected to obtain the corresponding general visual features, local forgery features and audio features.

[0034] In this embodiment of the invention, general visual features and local forgery features are extracted through a visual dual-branch approach. The first branch fuses multi-scale visual features to obtain global semantic information, while the second branch locates the forgery region through pyramid decomposition and residual analysis. At the same time, features such as Mel spectrum and fundamental frequency are extracted from the audio signal, realizing multimodal feature joint analysis of video content and providing comprehensive feature support for forgery detection.

[0035] In this embodiment of the invention, the step of performing visual bi-branch feature extraction on the video sequence to be detected to obtain corresponding general visual features, local forgery features, and audio features includes: Extract the visual content and audio signals from the video sequence to be detected; Based on the preset visual dual-branch, visual features are extracted from the visual content to obtain the corresponding general visual features and local forgery features; The audio signal is pre-emphasized to obtain a first audio signal; The first audio signal is subjected to frame-by-frame windowing processing to obtain an audio frame sequence; Perform time-frequency transformation on each audio frame in the audio frame sequence to obtain spectral features; Mel frequency cepstral coefficient analysis is performed on the spectral features to obtain the Mel spectral features, and the fundamental frequency features and energy features are extracted from the spectral features; The audio features are obtained by concatenating the Mel spectrum features, the fundamental frequency features, and the energy features.

[0036] In detail, audio feature detection uses multi-stage signal processing and feature fusion technology to accurately extract key features that reflect the authenticity and consistency of speech content; it pre-emphasizes the original audio signal and uses a first-order high-pass filter to enhance high-frequency components (such as the frequency bands of fricatives and plosives) to compensate for the low-frequency attenuation caused by oral cavity and vocal tract resonance during speech signal transmission, making the spectrum flatter.

[0037] For example, by increasing the amplitude of frequencies above 1kHz, the detailed features of consonants can be highlighted, providing richer acoustic clues for subsequent forgery detection.

[0038] Furthermore, the pre-emphasized signal is subjected to frame-by-frame windowing processing, dividing the continuous audio stream into short frames with a fixed duration (e.g., 25ms), with each frame overlapping by 50% to reduce boundary effects; at the same time, each frame signal is multiplied by a Hamming window, and the spectral leakage (i.e., the diffusion of signal energy in the frequency domain) is reduced through the gradual attenuation characteristics of the window function.

[0039] In this process, time-frequency transformation is performed on each windowed signal frame. The Fast Fourier Transform (FFT) is used to convert the time-domain signal into a frequency-domain spectrum. The FFT decomposes the signal into sinusoidal components of different frequencies through complex number operations, generating amplitude and phase spectra. The amplitude spectrum reflects the energy distribution of each frequency band and is the basis for subsequent feature extraction. For example, the FFT result of a speech segment will show the fundamental frequency (vocal cord vibration frequency) and the energy peaks of its harmonics, while fake audio (such as speech cloning) may have abnormal harmonic structure due to defects in the generation model.

[0040] Based on the spectral characteristics, Mel frequency cepstral coefficient (MFCC) analysis is performed. First, the linear frequency scale is converted into a Mel scale (simulating the nonlinear perception of pitch by the human ear). Then, the spectrum is weighted and summed through a Mel filter bank to generate energy values ​​for multiple Mel frequency bands. The output of the filter bank is logarithmized and subjected to discrete cosine transform (DCT) to extract the cepstral coefficients as MFCC features. MFCC can capture the vocal tract characteristics of speech (such as the shape of the vocal organs) and is robust to noise and channel changes. Fundamental frequency features (the vocal cord vibration frequency is calculated by autocorrelation or cepstral method) and energy features (the sum of squares or logarithmic energy of each frame of signal) are extracted from the spectrum to reflect the pitch and loudness information of speech, respectively.

[0041] Specifically, the MFCC features (usually the first 13 dimensions), fundamental frequency features (1 dimension), and energy features (1 dimension) are concatenated along the feature dimensions to form an audio feature vector that integrates multi-dimensional acoustic information. This feature includes both the physiological generation mechanism of speech (fundamental frequency and energy) and the vocal tract filtering characteristics (MFCC), which can effectively distinguish between real speech and synthesized speech (such as AI voice replacement), detect speech tampering (such as splicing and speed adjustment), and provide key audio evidence for verifying the authenticity of video content.

[0042] For example, in the medical and health scenario, visual dual-branch feature extraction technology can be applied to medical image forgery detection and voice-assisted diagnosis of illness. Taking medical image forgery detection as an example, the hospital obtains the medical image video sequence of the patient to be tested, in which there may be cases where criminals forge images to cover up the illness or create false diagnoses.

[0043] First, this invention extracts the visual content and audio signals from the video sequence. The visual content includes each frame of medical images, and the audio signal may be a recording of a doctor explaining the condition. The visual content is processed according to a preset visual dual-branch. One branch extracts general visual features, such as the overall structure of the image and the outline of organs, which are common features. The other branch focuses on local forgery features to accurately identify possible tampering traces in the image, such as abnormal edges and unnatural textures.

[0044] For audio signals, pre-emphasis processing is first performed to highlight the high-frequency components, resulting in the first audio signal. Then, the signal is segmented and windowed to form an audio frame sequence. Time-frequency transformation is performed on each frame to obtain spectral features. Mel-frequency cepstral coefficient analysis is used to obtain Mel-frequency spectral features, and fundamental frequency and energy features are extracted simultaneously. Finally, these features are concatenated to form the audio features. Combining visual and audio features, the system can accurately determine whether medical images have been forged, ensuring diagnostic accuracy and protecting patients' rights.

[0045] like Figure 3As shown in this embodiment of the invention, the step of extracting visual features from the visual content based on a preset visual dual-branch to obtain corresponding general visual features and local forgery features includes: Obtain the first visual branch and the second visual branch from the preset visual dual branches, and use the first visual branch to extract the low-level visual features of the visual content. Extract the mid-level and high-level semantic features of the visual content, and perform feature fusion on the low-level visual features, the mid-level semantic features, and the high-level semantic features to obtain multi-scale global semantic features; The multi-scale global semantic features are subjected to dimensionality reduction and normalization to obtain general visual features; The second visual branch is used to perform multi-scale pyramid decomposition on the visual content to obtain video image data at different resolution levels. High-frequency residual analysis is performed on the video image data at each scale to obtain pixel-level residual feature maps; Local anomaly detection is performed on the pixel-level residual feature map to obtain the abnormal region; Connectivity analysis and morphological post-processing are performed on the abnormal region to obtain the forged region, and local forged features are extracted from the forged region.

[0046] In detail, visual frames and audio signals are separated from the video sequence to be detected. In the visual dual-branch architecture, the first branch focuses on global semantic feature extraction. It adopts the hierarchical structure of convolutional neural network (CNN) and scans the visual content layer by layer through convolutional kernels of different sizes. Low-level convolutional kernels (such as 3×3) capture details such as edges and textures to generate low-level visual features; mid-level convolutional kernels (such as 5×5) aggregate local region features to extract mid-level semantic features such as body parts or motion patterns; high-level convolutional kernels (such as 7×7) integrate global context to identify high-level semantic features such as scene categories or complex behaviors.

[0047] Furthermore, the feature pyramid network (FPN) is used to concatenate multi-level features across channels and apply attention weights to fuse low-level details and high-level semantic information, forming multi-scale global semantic features. Principal component analysis (PCA) is then used to reduce the feature dimensionality, and batch normalization (BatchNorm) is used to eliminate data distribution bias, ultimately generating a general visual feature that can be generalized to various forgery detection tasks.

[0048] Specifically, the second branch focuses on local forgery trace mining. It uses the Laplacian pyramid decomposition algorithm to decompose visual content into multiple resolution levels (such as original resolution, 1 / 2 resolution, 1 / 4 resolution), with each level retaining structural information at different scales. For image data at each scale, it calculates the residual map (i.e., high-frequency components) with the adjacent high-resolution layer. By statistically analyzing the distribution characteristics of the residual pixels, such as the mean and variance, it generates pixel-level residual feature maps. The high-frequency components of real content usually exhibit a natural random distribution, while forged areas (such as face swapping or object tampering) will have abnormally clustered high-frequency components due to splicing boundaries or inconsistent content.

[0049] In detail, the Isolation Forest unsupervised algorithm is used to detect anomalies in the residual feature map and mark potential forgery pixels. Adjacent anomalies are merged through connected component labeling to form candidate forgery regions. Morphological opening operation (erosion followed by dilation) is used to eliminate noise interference, optimize region boundaries, and extract local forgery features such as texture and edges of the forgery regions.

[0050] For example, in a fintech scenario, this technology can be used for anti-counterfeiting of financial transaction videos and emotion analysis of customer service voice messages. In terms of anti-counterfeiting of financial transaction videos, financial institutions process a large number of transaction videos, which poses a risk of fraudsters forging videos to commit fraud. Visual content and audio signals are extracted from the transaction video sequence to be tested. The visual content covers the transaction scene, people's actions, etc., and the audio signal is the dialogue between the two parties in the transaction. Through a two-branch visual approach, one branch extracts general visual features, such as the overall layout of the transaction environment; the other branch mines local forgery features and identifies unnatural facial expressions and movements of people in the video.

[0051] In audio processing, the process involves pre-emphasis, frame-by-frame windowing, and time-frequency transformation to obtain Mel-frequency features, fundamental frequency features, and energy features, which are then concatenated into audio features. Comprehensive analysis of these features can effectively detect video forgery and prevent financial fraud. In customer service voice emotion analysis, the voice signals of customer service representatives and customers are extracted, and audio features are obtained according to the above process. Combined with visual features extracted from visual content (such as the facial expressions and body movements of customer service representatives and customers), customer emotions can be accurately judged, helping financial institutions optimize service quality, improve customer satisfaction, and ensure the sound operation of financial business.

[0052] In this embodiment of the invention, in the dual-branch architecture, the first branch uses a multi-scale convolutional network to fuse low-level texture and high-level semantic information to generate general visual features, effectively capturing the global authenticity of the video; the second branch uses pyramid decomposition and high-frequency residual analysis to accurately locate pixel-level forgery traces and extract local forgery features; at the same time, it combines the features of the audio signal such as Mel spectrum and fundamental frequency to form cross-modal feature complementarity, which significantly improves the detection robustness and accuracy, and provides an efficient and reliable solution for identifying forged content.

[0053] S3. Perform temporal attention fusion on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features.

[0054] In this embodiment of the invention, after enhancing visual features through spatial attention, audio features are aligned in the time dimension, and a cross-modal correlation weight matrix is ​​calculated to achieve dynamic fusion of visual and audio features. Finally, a fusion feature reflecting the consistency of audio and video is generated, which effectively improves the accuracy of forgery detection.

[0055] In this embodiment of the invention, the step of performing temporal cross-attention fusion on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features includes: Spatial attention weighting is applied to the general visual features and the local forgery features to obtain enhanced visual features; Obtain the temporal dimension of the enhanced visual features, and expand the audio features temporally based on the temporal dimension to obtain the target audio features; Calculate the correlation weight matrix between the enhanced visual features and the target audio features; Based on the correlation weight matrix, the enhanced visual features and the target audio features are fused using cross-modal features to obtain audio-video consistency features.

[0056] In detail, attention weights are applied to general visual features and local forged features in the spatial dimension, and a spatial attention mask is generated through convolution operations. This mask dynamically assigns weights based on the salience of each spatial location in the feature map (such as edge strength and texture complexity).

[0057] For example, a self-attention mechanism is used to calculate the correlation between each pixel in the feature map and other pixels, generating a normalized weight distribution. The weights are then multiplied with the original features channel by channel to highlight key regions (such as fake boundaries and abnormal textures) and suppress redundant information, resulting in enhanced visual features.

[0058] Since visual features naturally include a time dimension (frame sequence), while audio features are usually short-time spectra corresponding to a single frame, time dimension expansion is required to match the time resolution of visual features. Linear or spline interpolation is performed on the Mel spectrum or MFCC coefficients of audio features to align their time steps with the visual frame rate, ensuring that audio features and visual features have the same contextual range in the time dimension.

[0059] The calculation of the correlation weight matrix between the enhanced visual features and the target audio features reflects the temporal alignment of the two modal features. The visual feature vector and the audio feature vector are multiplied by an outer product to generate high-dimensional interactive features, which are then compressed into scalar weights through a fully connected layer. The cosine of the angle between the visual features and the audio features in the feature space can also be calculated as the confidence of temporal alignment to represent the correlation strength between the visual features and the corresponding audio features in each frame.

[0060] Specifically, after the weight matrix is ​​normalized row by row, it is multiplied element by element with the audio features to generate a time-aligned audio context vector. This vector is then concatenated or added to the visual features, preserving both local details and global consistency. The resulting audio-video consistency features contain both spatial information of visual forgery traces and temporal clues of audio anomalies, significantly improving the detection capability for complex forgery types such as spliced ​​videos and deepfakes.

[0061] For example, in healthcare scenarios, time-cross-attention fusion technology can significantly improve the accuracy and reliability of video analysis in telemedicine consultations and medical teaching. Taking telemedicine consultations as an example, doctors diagnose patients through video. First, they extract general visual features from the video, covering common information such as the patient's overall body posture and ward environment, as well as local forgery features, such as possible traces of local tampering in forged medical record images. By applying spatial attention weights to these visual features, the model will focus on key areas, such as the patient's lesion site or suspicious modifications to the medical record, thereby obtaining enhanced visual features.

[0062] Simultaneously, audio features are extracted from the video, including sound information such as the patient's description of their condition and the doctor's inquiries and conversations. After obtaining the time dimension of the enhanced visual features, the audio features are extended in time based on this dimension, so that the audio features are precisely aligned with the visual features on the time axis to obtain the target audio features. Then, the correlation weight matrix between the enhanced visual features and the target audio features is calculated. This matrix can reflect the degree of correlation between visual and audio information at different times.

[0063] For example, when a patient describes specific symptoms, the corresponding facial expressions and audio information at specific times show a high correlation. By performing cross-modal feature fusion based on this matrix, deeply combining visual and audio information, audio-video consistency features are obtained. This helps doctors to more comprehensively and accurately assess a patient's condition, avoiding misdiagnosis due to insufficient information from a single modality. In medical teaching video analysis, this technology can detect whether videos contain fake demonstrations, ensuring the authenticity and effectiveness of the teaching content.

[0064] In this embodiment of the invention, a time-cross attention fusion mechanism is used to achieve efficient collaborative and consistent modeling of audio and video features. Spatial attention is used to focus on key areas of visual forgery, and then the audio and video time sequence is aligned by extending the time dimension. Finally, a dynamic correlation weight matrix is ​​used to achieve deep interaction of cross-modal features, which can adaptively capture time synchronization anomalies between audio and video, thereby improving the detection accuracy in forgery detection tasks.

[0065] S4. Map the audio-video consistency features to a preset low-dimensional decoupling space, and decouple the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features.

[0066] In this embodiment of the invention, when mapping audio-video consistency features to a preset low-dimensional decoupling space, a multi-stage feature decoupling and dimensionality reduction technique is used to perform layer normalization on the input audio-video consistency features, thereby eliminating the numerical instability problem caused by the difference in the dimensions of features of different modalities.

[0067] In detail, principal component analysis (PCA) is used to whiten the feature covariance matrix, and eigenvalue decomposition is performed. The original features are projected onto an orthogonal basis space to unify the variance of each dimension to 1. An adversarial disentanglement network is used to construct the initial disentanglement space. The high-dimensional audio and video features are compressed into a preset low-dimensional space (such as 64 dimensions) through a fully connected layer to generate a latent representation vector.

[0068] Two independent discriminators can be designed to determine whether the latent representation contains modality type information (visual / audio) and task-independent information (such as background noise and illumination changes). The encoder and discriminator are optimized through adversarial training. The encoder tries to generate modality-independent latent representations, while the discriminator tries to distinguish modality sources as much as possible. By minimizing the discriminator loss (such as cross-entropy) and maximizing the encoder confusion loss, the latent representation is forced to gradually strip away modality-specific attributes.

[0069] Specifically, sparse regularization (such as L1 regularization) is applied to each dimension of the latent space, so that a few dimensions dominate the feature representation and the remaining dimensions approach zero. By analyzing the activation patterns of non-zero dimensions, their corresponding semantics (such as "lip-syncing anomaly" and "spectral distortion") can be manually labeled, which enhances the interpretability of the decoupled space and enables the mapping of audio and video consistency features into a low-dimensional decoupled space.

[0070] In this embodiment of the invention, the feature decoupling of the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features includes: The audio-visual consistency features are weighted spatially according to a preset spatial importance weight to obtain spatial dimension features; The spatial dimension features are subjected to layer normalization to obtain spatial forgery features; A time-dependent analysis is performed on the audio-video consistency features to obtain temporal dimension features; The time-series features are subjected to a nonlinear transformation to obtain time forgery features; The spatial forgery feature and the temporal forgery feature are concatenated to obtain the target forgery feature.

[0071] In detail, to capture the differences in the contribution of different spatial regions in video frames to forgery detection, a self-attention mechanism is used to calculate spatial importance weights. Specifically, the feature map of the low-dimensional decoupled space is divided into several spatial grids (such as 8×8 regions). A linear transformation is performed on the feature vectors in each grid to generate query vectors and key vectors. By calculating the dot product similarity between the query vector and all key vectors and normalizing it through an activation function, the attention weight distribution of each spatial grid is obtained. For example, in deepfakes, the edge regions of faces usually have higher weights. The original feature map is multiplied element-wise with the attention weights to achieve spatial dimension weighting and generate spatial dimension features.

[0072] Furthermore, to eliminate the numerical instability that may be introduced during the spatial weighting process, the spatial dimension features are subjected to layer normalization. The mean and variance are calculated along the feature channel dimension, and the feature values ​​of each channel are standardized (mean is 0, variance is 1). Then, the feature distribution is adjusted by a learnable scaling factor and offset, so that the spatial forgery features have a more stable numerical range while maintaining spatial saliency, which is beneficial for subsequent cross-module feature fusion.

[0073] Specifically, to address the temporal continuity of video sequences, a temporal convolutional network (TCN) is used to analyze the temporal dependencies of audio and video features. The TCN expands the receptive field by dilated causal convolution while avoiding the leakage of future information. For temporal feature sequences in low-dimensional decoupled space, convolutional kernels with different dilation rates (such as 1, 2, 4) are used to extract multi-scale temporal patterns layer by layer.

[0074] For example, small dilation rate convolutions capture short-term motion between frames (such as blink frequency), while large dilation rate convolutions capture long-term semantic changes (such as synchronization deviation between lip movements and speech). By dynamically adjusting the contribution ratio of each temporal feature through gating mechanisms (such as GLU), temporal dimension features are generated, preserving forgery clues in the time dimension, such as temporal misalignment or abnormal repetition patterns between audio and video.

[0075] In detail, to enhance the expressive power of time-forged features, a nonlinear transformation is performed on the temporal dimension features. A deep transformation module is constructed using residual connections. The feature dimension is compressed through 1×1 convolution, and then nonlinearity is introduced through an activation function. Finally, the original dimension is restored through convolution and added to the input features. The residual structure alleviates the gradient vanishing problem in deep networks. A channel attention mechanism (such as the SE module) is introduced to dynamically weight the transformed features, highlighting key temporal channels (such as features corresponding to high-frequency noise). This makes the final generated time-forged features more discriminative and can effectively distinguish the temporal anomalies between natural videos and forged videos.

[0076] Optionally, spatial forgery features and temporal forgery features can be merged by concatenation or weighted summation to generate multiple target forgery features. Spatial dominant features (focusing on texture forgery) and temporal dominant features (focusing on temporal forgery) can be defined, or feature subsets can be automatically divided through cluster analysis.

[0077] For example, in the healthcare scenario, audio-video consistency feature decoupling technology can help verify the authenticity of remote medical diagnosis and prevent the counterfeiting of medical training content. Taking remote medical diagnosis as an example, patients and doctors conduct diagnosis and treatment through video calls. During this process, audio-video data containing images of the patient's symptoms and audio descriptions are generated. These audio-video data are mapped to a low-dimensional decoupling space to obtain audio-video consistency features. Based on the preset spatial importance weights, the features are weighted according to the spatial dimensions.

[0078] For example, the spatial location of the patient's facial lesions in the image is more critical, so it is given a higher weight, thus highlighting the spatial dimension features of the key areas. After layer normalization, the differences in data distribution are eliminated, and spatial forgery features are obtained. At the same time, time dependency analysis is carried out on the audio-video consistency features. The time information such as pauses in the patient's speech and changes in speech rate when describing symptoms contains important clues. Through analysis, the temporal dimension features are obtained, and after nonlinear transformation, the complex patterns hidden in the time series are mined, resulting in temporal forgery features. Finally, the spatial forgery features and temporal forgery features are concatenated to obtain the target forgery features. If abnormal target forgery features are detected, it may mean that the video has been tampered with or the patient's description is false, ensuring the accuracy of remote diagnosis.

[0079] In medical training, this technology can detect whether training videos contain fake operational demonstrations, ensuring that medical staff learn real and effective skills.

[0080] In this embodiment of the invention, the decoupling design of spatial and temporal dimensions achieves accurate separation and efficient representation of forgery features. The spatial dimension uses an attention mechanism to focus on areas with significant forgery traces (such as face edges and texture anomalies). Layer normalization further suppresses spatial noise. Temporal dependency analysis and nonlinear transformation construct temporal forgery features by capturing audio and video temporal misalignments. This supports modular extension to the detection of unknown forgery types, significantly enhancing the model's generalization ability and robustness.

[0081] For example, in a fintech scenario, this technology can be used for anti-counterfeiting of financial product promotion videos and verification of the authenticity of customer service audio and video. In terms of financial product promotion, criminals may forge fake videos to mislead investors. By decoupling the audio and video consistency features of the promotion video, and by weighting the spatial dimension, important spatial information such as product display and key actions of the presenter is highlighted. After layer normalization, spatial forgery features are obtained. The logical coherence of the explanation and the time nodes of key content in the temporal dimension are analyzed. After nonlinear transformation, temporal forgery features are obtained. After splicing, target forgery features are obtained, thereby determining the authenticity of the video.

[0082] In customer service, the audio and video data of customer-customer communication are processed in the same way. Spatial dimension focuses on customer facial expressions and gestures, while temporal dimension analyzes the rhythm of the conversation and emotional changes. If the target's forgery characteristics are abnormal, it may indicate that customer service is misleading customers or that customers are making malicious complaints. This helps financial institutions maintain a fair and just trading environment, ensure the compliant and secure operation of financial business, and enhance customers' trust in financial institutions.

[0083] S5. Perform feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence, and determine the authenticity of the video stream to be detected based on the target forgery probability sequence.

[0084] In this embodiment of the invention, the target forgery features are processed by feature channel expansion and multi-scale dilated convolution. After temporal smoothing and activation function transformation, a forgery probability sequence is generated, which ultimately realizes the determination of the authenticity of the video stream. Dilated convolution is used to expand the receptive field to capture temporal anomalies, and probabilistic processing is combined to improve the reliability of forgery detection.

[0085] In this embodiment of the invention, performing feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence includes: Multiple target forgery features are subjected to feature channel expansion processing to obtain multiple expanded forgery features; The extended forgery features are processed by multi-scale dilated convolution using multiple preset dilated convolution kernels to obtain convolutional forgery features; The convolutional forgery features are smoothed in terms of temporal dimension to obtain smoothed forgery features; Extract the exponential linear unit values ​​of the smoothed forgery features, and perform exponential probability transformation on the exponential linear unit values ​​to obtain the target forgery probability sequence.

[0086] In detail, to enhance the expressive power of the forged features, the channel dimension of multiple target forged features is expanded. The number of channels of each feature map is increased through convolution operations (e.g., from 64 dimensions to 256 dimensions). While keeping the spatial and temporal dimensions unchanged, more learnable parameter channels are introduced. The feature map after channel expansion has richer semantic information, providing a more flexible feature combination basis for subsequent multi-scale convolution.

[0087] Furthermore, a temporal awareness module is constructed using dilated convolution. Convolutional kernels with different dilation rates capture the multi-scale temporal dependencies of forged features. Each convolutional kernel samples feature points at different intervals in the temporal dimension. For example, a convolutional kernel with a dilation rate of 1 focuses on local inter-frame changes (such as lip-sync differences in three consecutive frames), while a convolutional kernel with a dilation rate of 8 captures long-distance temporal patterns (such as abnormal fluctuations in the audio spectrum of the entire video segment).

[0088] To eliminate temporal jitter that may be introduced by multi-scale convolution (such as probability abrupt changes caused by single-frame noise), temporal smoothing of convolutional spoofing features is performed. The Causal Moving Average method is adopted, which sums the feature values ​​of each feature point and several frames before and after it (such as ±3 frames) in the temporal dimension with weights decreasing according to a Gaussian distribution to highlight the contribution of the center frame. The smoothed features are more coherent in time and can effectively suppress the interference of local outliers on the final probability prediction.

[0089] Specifically, the smooth forgery features are nonlinearly transformed using the Exponential Linear Unit (ELU) to enhance the model's ability to fit complex forgery patterns. The ELU introduces smooth decay in the negative region (instead of the hard truncation of ReLU), which alleviates the gradient vanishing problem while preserving the linearity in the positive region. Subsequently, the ELU output is mapped to the [0,1] probability space through the Sigmoid activation function to generate a target forgery probability sequence. The probability value at each time step represents the confidence that the corresponding video frame or segment is forged. For example, a probability value of 0.9 indicates that the segment is very likely to have been tampered with. Finally, the authenticity of the video stream to be detected is determined by threshold judgment (e.g., probability > 0.7 is judged as forgery) or temporal aggregation (e.g., calculating the average probability of the entire video).

[0090] In this embodiment of the invention, the temporal modeling capability and probability prediction accuracy of video forgery detection are significantly improved through the feature dilation convolution mechanism. Feature channel expansion enhances feature expression capability through dimensionality increase operation, and can simultaneously capture forgery clues in multiple dimensions such as texture and temporal sequence. Multi-scale dilation convolution uses convolution kernels with different dilation rates to achieve seamless fusion of short-, medium- and long-range temporal dependencies, avoiding the loss of local information caused by fixed receptive fields in traditional methods. Temporal smoothing processing effectively suppresses inter-frame noise interference, improves the stability of probability sequences, and enhances detection accuracy and efficiency.

[0091] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] like Figure 4 The diagram shown is a functional block diagram of a video content forgery detection device provided in an embodiment of the present invention.

[0093] This disclosure provides a video content forgery detection device, which corresponds one-to-one with the video content forgery detection method described in the above embodiments. For example... Figure 4 As shown, the video content forgery detection device 100 can be installed in an electronic device. According to its functions, the video content forgery detection device 100 includes a video sequence preprocessing module 101, a dual-branch feature extraction module 102, a multi-feature fusion module 103, a forgery feature decoupling module 104, and a video forgery recognition module 105. Detailed descriptions of each functional module are as follows: The video sequence preprocessing module 101 is used to acquire the video stream to be detected, perform frame sampling and standardization processing on the video stream to be detected, and generate the video sequence to be detected. The dual-branch feature extraction module 102 is used to perform visual dual-branch feature extraction on the video sequence to be detected, and obtain the corresponding general visual features, local forgery features and audio features. The multi-feature fusion module 103 is used to perform temporal cross-attention fusion on the general visual features, the local forgery features and the audio features to obtain audio-video consistency features; The forgery feature decoupling module 104 is used to map the audio-video consistency features to a preset low-dimensional decoupling space, and to perform feature decoupling on the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features; The video forgery identification module 105 is used to perform feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence, and determine the authenticity of the video stream to be detected based on the target forgery probability sequence.

[0094] In one embodiment, when the video sequence preprocessing module 101 performs frame sampling and normalization processing on the video stream to be detected to generate the video sequence to be detected, it is used to: The video stream to be detected is subjected to uniform frame sampling in the time dimension to obtain a key video frame sequence; Each key video frame in the key video frame sequence is subjected to resolution normalization processing to obtain a normalized video frame sequence. The standardized video frame sequence is subjected to color space conversion to obtain a color-unified video frame sequence; The color-uniform video frame sequence is batch-processed to obtain the video sequence to be detected.

[0095] In one embodiment, when the dual-branch feature extraction module 102 performs visual dual-branch feature extraction on the video sequence to be detected to obtain the corresponding general visual features, local forgery features, and audio features, it is used to: Extract the visual content and audio signals from the video sequence to be detected; Based on the preset visual dual-branch, visual features are extracted from the visual content to obtain the corresponding general visual features and local forgery features; The audio signal is pre-emphasized to obtain a first audio signal; The first audio signal is subjected to frame-by-frame windowing processing to obtain an audio frame sequence; Perform time-frequency transformation on each audio frame in the audio frame sequence to obtain spectral features; Mel frequency cepstral coefficient analysis is performed on the spectral features to obtain the Mel spectral features, and the fundamental frequency features and energy features are extracted from the spectral features; The audio features are obtained by concatenating the Mel spectrum features, the fundamental frequency features, and the energy features.

[0096] In one embodiment, when the dual-branch feature extraction module 102 performs visual feature extraction on the visual content according to a preset visual dual-branch to obtain corresponding general visual features and local forgery features, it is used to: Obtain the first visual branch and the second visual branch from the preset visual dual branches, and use the first visual branch to extract the low-level visual features of the visual content. Extract the mid-level and high-level semantic features of the visual content, and perform feature fusion on the low-level visual features, the mid-level semantic features, and the high-level semantic features to obtain multi-scale global semantic features; The multi-scale global semantic features are subjected to dimensionality reduction and normalization to obtain general visual features; The second visual branch is used to perform multi-scale pyramid decomposition on the visual content to obtain video image data at different resolution levels. High-frequency residual analysis is performed on the video image data at each scale to obtain pixel-level residual feature maps; Local anomaly detection is performed on the pixel-level residual feature map to obtain the abnormal region; Connectivity analysis and morphological post-processing are performed on the abnormal region to obtain the forged region, and local forged features are extracted from the forged region.

[0097] In one embodiment, when the multi-feature fusion module 103 performs temporal cross-attention fusion of the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features, it is used to: Spatial attention weighting is applied to the general visual features and the local forgery features to obtain enhanced visual features; Obtain the temporal dimension of the enhanced visual features, and expand the audio features temporally based on the temporal dimension to obtain the target audio features; Calculate the correlation weight matrix between the enhanced visual features and the target audio features; Based on the correlation weight matrix, the enhanced visual features and the target audio features are fused using cross-modal features to obtain audio-video consistency features.

[0098] In one embodiment, when the forgery feature decoupling module 104 performs feature decoupling on the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features, it is used to: The audio-visual consistency features are weighted spatially according to a preset spatial importance weight to obtain spatial dimension features; The spatial dimension features are subjected to layer normalization to obtain spatial forgery features; A time-dependent analysis is performed on the audio-video consistency features to obtain temporal dimension features; The time-series features are subjected to a nonlinear transformation to obtain time forgery features; The spatial forgery feature and the temporal forgery feature are concatenated to obtain the target forgery feature.

[0099] In one embodiment, when the video forgery recognition module 105 performs feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence, it is used to: Multiple target forgery features are subjected to feature channel expansion processing to obtain multiple expanded forgery features; The extended forgery features are processed by multi-scale dilated convolution using multiple preset dilated convolution kernels to obtain convolutional forgery features; The convolutional forgery features are smoothed in terms of temporal dimension to obtain smoothed forgery features; Extract the exponential linear unit values ​​of the smoothed forgery features, and perform exponential probability transformation on the exponential linear unit values ​​to obtain the target forgery probability sequence.

[0100] In this invention, the specific limitations of a video content forgery detection device can be found in the above-described limitations of a video content forgery detection method, and will not be repeated here. Each module in the aforementioned video content forgery detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a video content forgery detection method on the server side.

[0102] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a video content forgery detection method on the client side.

[0103] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The video stream to be detected is acquired, and frame sampling and standardization processing are performed on the video stream to be detected to generate a video sequence to be detected. Visual dual-branch feature extraction is performed on the video sequence to be detected to obtain the corresponding general visual features, local forgery features and audio features; Temporal attention fusion is performed on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features; The audio-video consistency features are mapped to a preset low-dimensional decoupling space, and the audio-video consistency features mapped to the low-dimensional decoupling space are decoupled to obtain multiple target forgery features. The target forgery features are subjected to feature dilation convolution to obtain a target forgery probability sequence, and the authenticity of the video stream to be detected is determined based on the target forgery probability sequence.

[0104] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0105] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0106] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0107] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0108] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: The video stream to be detected is acquired, and frame sampling and standardization processing are performed on the video stream to be detected to generate a video sequence to be detected. Visual dual-branch feature extraction is performed on the video sequence to be detected to obtain the corresponding general visual features, local forgery features and audio features; Temporal attention fusion is performed on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features; The audio-video consistency features are mapped to a preset low-dimensional decoupling space, and the audio-video consistency features mapped to the low-dimensional decoupling space are decoupled to obtain multiple target forgery features. The target forgery features are subjected to feature dilation convolution to obtain a target forgery probability sequence, and the authenticity of the video stream to be detected is determined based on the target forgery probability sequence.

[0109] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0110] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0111] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0112] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0117] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for detecting video content forgery, characterized in that, The method includes: The video stream to be detected is acquired, and frame sampling and standardization processing are performed on the video stream to be detected to generate a video sequence to be detected. Visual dual-branch feature extraction is performed on the video sequence to be detected to obtain the corresponding general visual features, local forgery features and audio features; Temporal attention fusion is performed on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features; The audio-video consistency features are mapped to a preset low-dimensional decoupling space, and the audio-video consistency features mapped to the low-dimensional decoupling space are decoupled to obtain multiple target forgery features. The target forgery features are subjected to feature dilation convolution to obtain a target forgery probability sequence, and the authenticity of the video stream to be detected is determined based on the target forgery probability sequence.

2. The video content forgery detection method as described in claim 1, characterized in that, The step of performing frame sampling and normalization processing on the video stream to be detected to generate a video sequence to be detected includes: The video stream to be detected is subjected to uniform frame sampling in the time dimension to obtain a key video frame sequence; Each key video frame in the key video frame sequence is subjected to resolution normalization processing to obtain a normalized video frame sequence. The standardized video frame sequence is subjected to color space conversion to obtain a color-unified video frame sequence; The color-uniform video frame sequence is batch-processed to obtain the video sequence to be detected.

3. The video content forgery detection method as described in claim 1, characterized in that, The step of performing visual dual-branch feature extraction on the video sequence to be detected to obtain corresponding general visual features, local forgery features, and audio features includes: Extract the visual content and audio signals from the video sequence to be detected; Based on the preset visual dual-branch, visual features are extracted from the visual content to obtain corresponding general visual features and local forgery features; The audio signal is pre-emphasized to obtain a first audio signal; The first audio signal is subjected to frame-by-frame windowing processing to obtain an audio frame sequence; Perform time-frequency transformation on each audio frame in the audio frame sequence to obtain spectral features; Mel frequency cepstral coefficient analysis is performed on the spectral features to obtain the Mel spectral features, and the fundamental frequency features and energy features are extracted from the spectral features; The audio features are obtained by concatenating the Mel spectrum features, the fundamental frequency features, and the energy features.

4. The video content forgery detection method as described in claim 3, characterized in that, The step of extracting visual features from the visual content based on a preset visual dual-branch model to obtain corresponding general visual features and local forgery features includes: Obtain the first visual branch and the second visual branch from the preset visual dual branches, and use the first visual branch to extract the low-level visual features of the visual content. Extract the mid-level and high-level semantic features of the visual content, and perform feature fusion on the low-level visual features, the mid-level semantic features, and the high-level semantic features to obtain multi-scale global semantic features; The multi-scale global semantic features are subjected to dimensionality reduction and normalization to obtain general visual features; The visual content is decomposed into a multi-scale pyramid using the second visual branch to obtain video image data at different resolution levels. High-frequency residual analysis is performed on the video image data at each scale to obtain pixel-level residual feature maps; Local anomaly detection is performed on the pixel-level residual feature map to obtain the abnormal region; Connectivity analysis and morphological post-processing are performed on the abnormal region to obtain the forged region, and local forged features are extracted from the forged region.

5. The video content forgery detection method as described in claim 1, characterized in that, The step of performing temporal cross-attention fusion on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features includes: Spatial attention weighting is applied to the general visual features and the local forgery features to obtain enhanced visual features; Obtain the temporal dimension of the enhanced visual features, and expand the audio features temporally based on the temporal dimension to obtain the target audio features; Calculate the correlation weight matrix between the enhanced visual features and the target audio features; Based on the correlation weight matrix, the enhanced visual features and the target audio features are fused using cross-modal features to obtain audio-video consistency features.

6. The video content forgery detection method as described in claim 1, characterized in that, The audio-video consistency features mapped to the low-dimensional decoupling space are decoupled to obtain multiple target forgery features, including: The audio-visual consistency features are weighted spatially according to a preset spatial importance weight to obtain spatial dimension features; The spatial dimension features are subjected to layer normalization to obtain spatial forgery features; A time-dependent analysis is performed on the audio-video consistency features to obtain temporal dimension features; The time-series features are subjected to a nonlinear transformation to obtain time forgery features; The spatial forgery feature and the temporal forgery feature are concatenated to obtain the target forgery feature.

7. The video content forgery detection method as described in claim 1, characterized in that, The step of performing feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence includes: Multiple target forgery features are subjected to feature channel expansion processing to obtain multiple expanded forgery features; The extended forgery features are processed by multi-scale dilated convolution using multiple preset dilated convolution kernels to obtain convolutional forgery features; The convolutional forgery features are smoothed in terms of temporal dimension to obtain smoothed forgery features; Extract the exponential linear unit values ​​of the smoothed forgery features, and perform exponential probability transformation on the exponential linear unit values ​​to obtain the target forgery probability sequence.

8. A video content forgery detection device, characterized in that, The device includes: The video sequence preprocessing module is used to acquire the video stream to be detected, perform frame sampling and standardization processing on the video stream to be detected, and generate the video sequence to be detected. The dual-branch feature extraction module is used to perform visual dual-branch feature extraction on the video sequence to be detected, and obtain the corresponding general visual features, local forgery features and audio features. The multi-feature fusion module is used to perform temporal attention fusion on the general visual features, the local forgery features, and the audio features to obtain audio-video consistency features. The forgery feature decoupling module is used to map the audio-video consistency features to a preset low-dimensional decoupling space, and to perform feature decoupling on the audio-video consistency features mapped to the low-dimensional decoupling space to obtain multiple target forgery features; The video forgery detection module is used to perform feature dilation convolution on multiple target forgery features to obtain a target forgery probability sequence, and determine the authenticity of the video stream to be detected based on the target forgery probability sequence.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video content forgery detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video content forgery detection method as described in any one of claims 1 to 7.

Images