A video-level deepfake detection method and system for local audiovisual tampering

Abstract

This invention relates to a video-level forgery detection method and system for local audiovisual tampering. The method divides the input video into multiple time windows, extracting visual and audio features from each window. An audiovisual heterogeneous graph containing visual and audio nodes is constructed using the time windows as basic units. Time-aware intra-modal interactions and bidirectional cross-modal interactions are performed on this graph. Furthermore, a multi-scale temporal fusion mechanism combines local window evidence with long-range context. Finally, through joint prediction using a video-level main task and a window-level auxiliary task, the method outputs a video-level forgery detection result and a window-level anomaly response score. This allows the system to retain local window evidence while outputting the authenticity judgment of the entire video, explicitly modeling intra-modal temporal relationships and cross-modal correspondences, and improving detection stability by combining multi-scale context.

Description

Technical Field

[0001] This application belongs to the field of video information detection technology, specifically relating to a video-level forgery detection method and system for local audiovisual tampering. Background Technology

[0002] With the development of deep generative models, speech synthesis, lip-syncing, face replay, and multimodal editing technologies, the barrier to generating audio and video forgery content is constantly decreasing. Existing deepfake detection methods typically treat the entire video as the detection object, classifying it as real or fake based on visual features, audio features, or combined audiovisual features.

[0003] However, in real-world distribution scenarios, forged content doesn't necessarily cover the entire video. Attackers can replace or tamper with only short segments of the video, such as partially replacing audio content, changing a few keywords, modifying partial lip movements, inserting short segments of audio-visual desynchronization, or creating audiovisual semantic inconsistencies only for a small period of time in the video. Existing technologies have at least the following shortcomings: 1. The tampered segments are short in duration, and the abnormal signals account for a low percentage of the entire video, making them easily masked by a large amount of untampered content.

[0004] 2. The visual images, speech actions, and contextual semantics before and after partial tampering may maintain strong continuity, making it difficult to reliably capture anomalies by relying solely on the global representation of the entire video.

[0005] 3. Local speech replacement, local lip-sync shift, and local semantic changes usually manifest as short-term audiovisual inconsistencies, requiring simultaneous modeling of audio, visual, and their cross-modal correspondences.

[0006] 4. Existing video-level detection methods often employ global pooling, global alignment, or whole-segment feature aggregation, which can easily weaken local abnormal responses during the aggregation process.

[0007] 5. Although existing multimodal detection methods can utilize audio and visual information, they lack a local evidence preservation mechanism oriented towards time windows, making it difficult to simultaneously consider short-term anomaly modeling and long-term context judgment. Summary of the Invention

[0008] The purpose of this application is to overcome the shortcomings of the prior art and provide a video-level forgery detection method and system for local audiovisual tampering. This method and system can retain local window evidence while outputting the authenticity judgment of the entire video, explicitly model the intramodal temporal relationship and cross-modal correspondence, and improve the detection stability by combining multi-scale context.

[0009] The technical problem solved by this application is achieved through the following technical solution: A method for temporal localization of audio and video forgery segments is disclosed. The method divides the input video into multiple time windows, extracting visual and audio features from each window. An audiovisual heterogeneous graph containing visual and audio nodes is constructed using the time windows as basic units. Temporally aware intra-modal interactions and bidirectional cross-modal interactions are performed on this graph. Furthermore, a multi-scale temporal fusion mechanism is used to combine local window evidence with long-range context. Finally, a joint prediction by a video-level main task and a window-level auxiliary task outputs a video-level forgery detection result and a window-level anomaly response score. The steps of the method are as follows: S1, Input representation and window partitioning, used to obtain a time-organized audiovisual window representation; S2, Heterogeneous graph construction, used to establish visual window nodes, audio window nodes, and the local connection relationships between them; S3, window-level graph interaction, is used to perform intra-modal attention and cross-modal attention; S4, Multi-scale temporal fusion, used to fuse short-scale, medium-scale and long-scale contexts; S5, multi-task prediction, is used to simultaneously generate video-level classification results and window-level auxiliary responses; S6. Output the results, providing video-level detection results and window-level anomaly response scores.

[0010] Moreover, S1 specifically refers to: First, acquire the audio and video samples to be detected, and divide them into multiple continuous time windows along the time axis. Let the total duration of the input video be T, and the preset window length be T. w The input video is divided into W1, W2, ..., W... N The i-th window is denoted as W. i The center time of the window is denoted as ci, and the time interval between any two windows i and j is denoted as Δt. ij , ci = (i - 1 / 2) × T w ; Δtij = |ci - cj|; For each window, visual and audio segments are obtained separately. The visual segments are input into the video feature extraction branch to obtain visual window features; the audio segments are input into the audio feature extraction branch to obtain audio window features. After feature extraction, the two types of window features are temporally encoded and linearly projected to map them to a unified latent space. Window partitioning can use continuous non-overlapping windows, sliding windows, or adaptive windows. The window length can be adjusted based on the video frame rate, audio sampling rate, duration of fake segments, and actual deployed resources.

[0011] Moreover, S2 specifically involves: for each time window, establishing a visual window node vi and an audio window node ai respectively. The audiovisual heterogeneous graph includes modal inner edges and cross-modal edges. Modal inner edges are used to connect window nodes that are temporally adjacent or close in time within the same modality. Cross-modal edges are used to connect visual window nodes and audio window nodes, representing the audiovisual correspondence within a local time range. Let the intramodal interaction radius be rintra and the cross-modal interaction radius be rcross. When the time distance between the centers of two same-modal window nodes does not exceed rintra, an intramodal edge is established between them; when the time distance between the centers of the visual window node and the audio window node does not exceed rcross, a cross-modal edge is established between them. Intramodal neighborhood: The set of nodes in the same modal window whose temporal distance is greater than 0 and does not exceed rintra; Cross-modal neighborhood: A set of heteromodal window nodes whose temporal distance does not exceed rcross; Both intramodal edges and cross-modal edges are bidirectional directed connections.

[0012] Moreover, S3 specifically refers to the window-level graph interaction, which includes intramodal attention learning and cross-modal attention learning. Intramodal attention learning is performed within the visual modality and the audio modality, respectively, to enhance the local temporal changes between adjacent windows in the same modality. In intramodal interaction, the attention weight is determined by both content relevance and temporal relationship. Intramodal attention score = content relevance score + time bias score; Cross-modal attention learning is used to enable bidirectional information exchange between visual and audio nodes. Visual nodes receive information from audio nodes in their cross-modal neighborhood, and audio nodes receive information from visual nodes in their cross-modal neighborhood. This mechanism can explicitly model local audiovisual mismatch cues such as mismatch between speech content and lip movements, abnormal audio-visual synchronization caused by local audio replacement, and inconsistency between visual facial expression changes and audio semantics. Intramodal attention learning and cross-modal attention learning are implemented using single-head attention, multi-head attention, graph attention, graph convolution, or other message passing mechanisms.

[0013] Moreover, S4 specifically refers to: the multi-scale temporal fusion is performed after window-level graph interaction, along with context feedback; The visual node representation and audio node representation in the previous window node are input into gating fusion to obtain a window-level fusion sequence; The fused sequence is input into multi-scale temporal modeling to extract short-scale, medium-scale, and long-scale temporal information, respectively. The results of multi-scale temporal modeling are used to form long-range contexts, enabling local window evidence to be judged in conjunction with the context of the entire video segment. The long-range context is fed back to the window node through context feedback, resulting in the updated window node. Fusion context = Gated weight × Global context + (1 - Gated weight) × Multi-scale local context + Differential context; Multi-scale temporal modeling can be achieved through temporal convolutions with different kernel sizes, different dilation rates, and different time lengths, or through Transformer, state-space model, temporal pyramid, or other long- and short-range temporal modeling structures.

[0014] Furthermore, S5 specifically involves: setting up a video-level classification branch and a window-level auxiliary classification branch; the video-level classification branch performs global pooling on the window representation after multi-scale fusion and outputs the forgery probability of the entire video through the video-level classification head; the window-level auxiliary classification branch classifies the window representation and outputs the abnormal response score for each window. When the probability of video-level forgery is higher than a preset threshold, the video is output as a forged video; otherwise, the video is output as a real video. The window-level anomaly response score is used to assist in the analysis of suspicious anomaly time regions. Windows with response scores higher than a preset threshold are marked as anomaly windows, and temporally adjacent anomaly windows are merged into candidate anomaly segments, and their start and end times and confidence levels are output.

[0015] A temporal localization system for audio and video forgery segments is disclosed. The system is used to execute the localization method described above. The localization system includes an input acquisition module, a window partitioning module, a bimodal feature extraction module, an audiovisual heterogeneous graph construction module, a time-aware intramodal interaction module, a bidirectional cross-modal interaction module, a multi-scale temporal fusion module, a context feedback module, a video-level classification module, a window-level auxiliary classification module, a training optimization module, and an abnormal time region analysis module.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention divides the input video into time windows and retains local audiovisual evidence at the window level, which can reduce the risk of short-term tampering being covered by long-term real content.

[0017] 2. This invention constructs an audiovisual heterogeneous graph containing visual nodes and audio nodes, which can simultaneously model intramodal temporal relationships and cross-modal correspondences.

[0018] 3. This invention enables the model to focus on windows of the same modality that are close in time and related in content through a time-aware intramodal interaction mechanism, which helps to capture local temporal abrupt changes.

[0019] 4. This invention can explicitly model cross-modal anomalies such as local audio-visual asynchrony, lip-sync mismatch, and local semantic inconsistency through a two-way cross-modal interaction mechanism.

[0020] 5. This invention, through a multi-scale temporal fusion mechanism, simultaneously introduces short-scale, medium-scale, and long-scale local patterns and global context, which helps to improve the stability of video-level discrimination.

[0021] 6. The present invention uses a context feedback mechanism to ensure that each window representation contains both local anomaly evidence and long-range contextual information, thus balancing local sensitivity and overall robustness.

[0022] 7. This invention uses joint training of video-level main tasks and window-level auxiliary tasks to ensure that video-level detection results are consistent with local anomaly responses.

[0023] 8. This invention not only outputs the true or false result of the entire video segment, but also outputs the window-level abnormal response score, which can provide auxiliary basis for manual review, suspicious segment backtracking, content security risk control and subsequent fine-grained positioning.

[0024] 9. The window length, intramodal interaction radius, cross-modal interaction radius, and multi-scale temporal branches of the present invention can all be adjusted according to the actual application scenario, and have strong engineering adaptability.

[0025] 10. This invention can be used in combination with different visual encoders, audio encoders and timing modeling modules, and has strong scalability. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram illustrating the input representation and heterogeneous graph construction of the present invention; Figure 3 This is a schematic diagram of the single-layer graph interaction module of the present invention; Figure 4 This is a schematic diagram of the multi-scale temporal fusion and contextual feedback of the present invention. Detailed Implementation

[0027] The present application will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present application.

[0028] An innovative method for temporal localization of audio and video forgery segments is proposed, which involves: dividing the input video into multiple time windows and extracting visual and audio features from each window; constructing an audiovisual heterogeneous graph containing visual and audio nodes using the time windows as basic units; performing time-aware intramodal interactions and bidirectional cross-modal interactions on this graph; further combining local window evidence with long-range context through a multi-scale temporal fusion mechanism; and finally, outputting video-level forgery detection results and window-level anomaly response scores through joint prediction of video-level main tasks and window-level auxiliary tasks.

[0029] The overall process is as follows Figure 1 As shown, the overall process of this invention includes input representation and window partitioning 100, heterogeneous graph construction 110, window-level graph interaction 120, multi-scale temporal fusion 130, multi-task prediction 140, and output 150. Input representation and window partitioning 100 is used to obtain a temporally organized audiovisual window representation; heterogeneous graph construction 110 is used to establish visual window nodes, audio window nodes, and the local connections between them; window-level graph interaction 120 is used to perform intra-modal attention and cross-modal attention; multi-scale temporal fusion 130 is used to fuse short-scale, medium-scale, and long-scale context; multi-task prediction 140 is used to simultaneously generate video-level classification results and window-level auxiliary responses; and output 150 is used to provide video-level detection results and window-level anomaly response scores.

[0030] The above process is not limited to specific visual encoders, audio encoders, temporal modeling networks, or classification head structures. Any implementation that divides the input video into windows, constructs an audiovisual heterogeneous graph, performs intramodal and cross-modal window interactions, and jointly outputs video-level results and window-level responses can be used as an implementation method of this invention.

[0031] (a) Input representation and window division like Figure 1 Input representation and window division 100 and Figure 2 As shown in Figure 200, the audio-visual input and window partitioning and feature representation are as follows: First, the audio-visual samples to be detected are acquired and divided into multiple continuous time windows along the time axis. Let the total duration of the input video be T, and the preset window length be Tw. The input video is divided into W1, W2, ..., WN. The i-th window is denoted as Wi, the center time of the window is denoted as ci, and the time interval between any two windows i and j is denoted as Δtij.

[0032] ci = (i - 1 / 2) × Tw; Δtij = |ci - cj|; For each window, visual and audio segments are obtained separately. The visual segment is input into the video feature extraction branch to obtain visual window features; the audio segment is input into the audio feature extraction branch to obtain audio window features. After feature extraction, the two types of window features are temporally encoded and linearly projected to map them to a unified latent space.

[0033] Window partitioning can use continuous, non-overlapping windows, sliding windows, or adaptive windows. The window length can be adjusted based on the video frame rate, audio sampling rate, duration of the forged segment, and actual deployed resources.

[0034] (II) Construction of Audiovisual Heterogeneous Diagrams like Figure 1 Heterogeneous graph construction 110 and Figure 2 As shown in Figure 220, this invention constructs an audiovisual heterogeneous graph using time windows as the basic unit. For each time window, a visual window node vi and an audio window node ai are created respectively. Figure 2 In this context, v represents the video window node, and a represents the audio window node.

[0035] Audiovisual heterogeneous graphs include intramodal edges and crossmodal edges. Intramodal edges are used to connect window nodes that are temporally adjacent or close in time within the same modality; crossmodal edges are used to connect visual window nodes and audio window nodes, representing audiovisual correspondences within a local temporal range.

[0036] Let the intramodal interaction radius be rintra and the cross-modal interaction radius be rcross. When the time distance between the centers of two window nodes in the same modality does not exceed rintra, an intramodal edge is established between them; when the time distance between the centers of the visual window node and the audio window node does not exceed rcross, a cross-modal edge is established between them.

[0037] Intramodal neighborhood: The set of nodes in the same modal window whose temporal distance is greater than 0 and does not exceed rintra. Cross-modal neighborhood: A set of heteromodal window nodes whose temporal distance does not exceed rcross. In a preferred embodiment, intramodal edges employ bidirectional directed connections, and cross-modal edges also employ bidirectional directed connections, thus simultaneously establishing information transmission directions from vision to audio and from audio to vision. This composition method enables the model to focus on the audiovisual context most relevant to the current window within a local timeframe, preventing distant and irrelevant content from prematurely participating in local anomaly modeling.

[0038] (III) Window-level graph interaction like Figure 1 Window-level graph interaction 120 and Figure 3As shown, window-level graph interaction includes intra-modal attention learning 300 and cross-modal attention learning 310. Figure 3 In the diagram, v represents the video window node, a represents the audio window node, and the curved arrow indicates that the window node is being updated.

[0039] Intramodal attention learning 300 is performed within both the visual and audio modalities to reinforce local temporal changes between adjacent windows within the same modality, such as abrupt visual lip movements, abnormal facial expressions, abrupt audio waveform changes, or local speech variations. In intramodal interaction, attention weights are determined by both content relevance and temporal relationships.

[0040] Intramodal attention score = Content relevance score + Temporal bias score; Cross-modal attention learning 310 is used for bidirectional information exchange between visual and audio nodes. Visual nodes receive information from audio nodes in their cross-modal neighborhood, and audio nodes receive information from visual nodes in their cross-modal neighborhood. This mechanism can explicitly model local audiovisual mismatch cues such as mismatch between speech content and lip movements, abnormal audio-visual synchronization caused by local audio substitution, and inconsistency between visual facial expression changes and audio semantics.

[0041] Intramodal attention learning 300 and cross-modal attention learning 310 can be implemented using single-head attention, multi-head attention, graph attention, graph convolution, or other message passing mechanisms. This invention does not limit the specific number of attention heads, hidden dimensions, or normalization method.

[0042] (iv) Multi-scale temporal fusion and contextual feedback like Figure 1 Multi-scale temporal fusion in 130 and Figure 4 As shown, this invention performs multi-scale temporal fusion and contextual feedback after window-level graph interaction. Figure 4 In the diagram, 400 represents the window node before the update, 410 represents gated fusion, 420 represents multi-scale temporal modeling, 430 represents long-range context, 440 represents context feedback, and 450 represents the window node after the update.

[0043] Multi-scale temporal fusion and contextual feedback include the following steps: 1. Input the visual node representation and audio node representation in the previous window node 400 into the gating fusion 410 to obtain the window-level fusion sequence.

[0044] 2. Input the fused sequence into multi-scale time series modeling 420, and extract short-scale, medium-scale, and long-scale time series information respectively.

[0045] 3. Based on the results of multi-scale temporal modeling, a long-range context 430 is formed, enabling local window evidence to be judged in conjunction with the context within the entire video segment.

[0046] 4. The long-range context is injected back to the window node through context feedback 440, resulting in the updated window node 450.

[0047] Fusion context = gating weight × global context + (1 - gating weight) × multi-scale local context + differential context; Multi-scale temporal modeling 420 can be achieved through temporal convolutions with different kernel sizes, dilation rates, and phase lengths, or through Transformers, state-space models, temporal pyramids, or other long- and short-range temporal modeling structures. Through this mechanism, each window node retains both local audiovisual interaction information and obtains long-range contextual constraints, thereby improving the stability of video-level discrimination under local audiovisual tampering conditions.

[0048] (v) Multi-task prediction and output like Figure 1 As shown in the multi-task prediction 140 and output 150, this invention sets up a video-level classification branch and a window-level auxiliary classification branch. The video-level classification branch performs global pooling on the window representation after multi-scale fusion and outputs the forgery probability of the entire video through the video-level classification head; the window-level auxiliary classification branch classifies the window representation and outputs the abnormal response score of each window.

[0049] When the probability of video-level forgery exceeds a preset threshold, the video is output as forged; otherwise, it is output as genuine. Window-level anomaly response scores can be used to assist in analyzing suspicious anomaly time regions. Specifically, windows with response scores exceeding a preset threshold can be marked as anomaly windows, and temporally adjacent anomaly windows can be merged into candidate anomaly segments, outputting their start and end times and confidence levels.

[0050] (vi) Joint training methods During the training phase, joint optimization is performed using both video-level and window-level labels. For samples with locally tampered intervals, the overlap ratio between each window and the real tampered interval is calculated. When the overlap ratio reaches a preset threshold, the window is marked as a fake window; otherwise, it is marked as a real window.

[0051] For samples that only have video-level labels but no locally tampered interval annotations, the video-level labels can be assigned to the windows in the sample, or training can be performed using only video-level loss.

[0052] Total loss = Video-level loss weight × Video-level classification loss + Window-level loss weight × Window-level auxiliary classification loss; The video-level classification loss and window-level auxiliary classification loss can be cross-entropy loss, weighted cross-entropy loss, Focal Loss, or a combination thereof. Preferably, Focal Loss is used when there is a local imbalance between positive and negative samples due to tampering. This joint training method enables the model to improve its response to local abnormal windows while optimizing the judgment of the authenticity of the entire video.

[0053] (vii) Detection system This invention also provides a video-level forgery detection system for local audiovisual tampering, comprising an input acquisition module, a window segmentation module, a bimodal feature extraction module, an audiovisual heterogeneous graph construction module, a time-aware intramodal interaction module, a bidirectional cross-modal interaction module, a multi-scale temporal fusion module, a contextual feedback module, a video-level classification module, a window-level auxiliary classification module, a training optimization module, and an abnormal time region analysis module.

[0054] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A method for temporal localization of audio and video forgery segments, characterized in that: The method divides the input video into multiple time windows and extracts the visual and audio features of each window respectively. An audiovisual heterogeneous graph containing visual and audio nodes is constructed using time windows as the basic unit; time-aware intramodal interactions and bidirectional cross-modal interactions are performed on this graph; further, local window evidence is combined with long-range context through a multi-scale temporal fusion mechanism; finally, video-level forgery detection results and window-level anomaly response scores are output through joint prediction of video-level main tasks and window-level auxiliary tasks. The steps of the method are as follows: S1, Input representation and window division, used to obtain a time-organized audiovisual window representation; S2, Heterogeneous graph construction, used to establish visual window nodes, audio window nodes, and the local connections between them; S3, window-level graph interaction, is used to perform intra-modal attention and cross-modal attention; S4, Multi-scale temporal fusion, used to fuse short-scale, medium-scale and long-scale contexts; S5, multi-task prediction, is used to simultaneously generate video-level classification results and window-level auxiliary responses; S6. Output the results, providing video-level detection results and window-level anomaly response scores.

2. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S1 is: First, acquire the audio and video samples to be detected, and divide them into multiple continuous time windows along the time axis. Let the total duration of the input video be T, and the preset window length be T. w The input video is divided into W1, W2, ..., W... N Let the i-th window be denoted as W. i The center time of the window is denoted as ci, and the time interval between any two windows i and j is denoted as Δt. ij , ci = (i - 1 / 2) × T w ; Δtij = |ci - cj|; For each window, visual and audio segments are obtained separately. The visual segments are input into the video feature extraction branch to obtain the visual window features; the audio segments are input into the audio feature extraction branch to obtain the audio window features. After feature extraction is completed, temporal encoding and linear projection are performed on the two types of window features to map them to a unified hidden space; Window partitioning can use continuous non-overlapping windows, sliding windows, or adaptive windows. The window length can be adjusted based on the video frame rate, audio sampling rate, duration of fake segments, and actual deployed resources.

3. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: S2 specifically involves: for each time window, establishing a visual window node vi and an audio window node ai. The audiovisual heterogeneous graph includes modal inner edges and cross-modal edges. Modal inner edges are used to connect window nodes that are temporally adjacent or close in time within the same modality. Cross-modal edges are used to connect visual window nodes and audio window nodes, representing the audiovisual correspondence within a local time range. Let the intramodal interaction radius be rintra and the cross-modal interaction radius be rcross. When the time distance between the centers of two same-modal window nodes does not exceed rintra, an intramodal edge is established between them; when the time distance between the centers of the visual window node and the audio window node does not exceed rcross, a cross-modal edge is established between them. Intramodal neighborhood: The set of nodes in the same modal window whose temporal distance is greater than 0 and does not exceed rintra; Cross-modal neighborhood: A set of heteromodal window nodes whose temporal distance does not exceed rcross; Both intramodal edges and cross-modal edges are bidirectional directed connections.

4. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S3 is as follows: the window-level graph interaction includes intramodal attention learning and cross-modal attention learning. Intramodal attention learning is performed within the visual modality and the audio modality, respectively, to enhance the local temporal changes between adjacent windows in the same modality. In intramodal interaction, the attention weight is determined by both content relevance and temporal relationship. Intramodal attention score = content relevance score + time bias score; Cross-modal attention learning is used to enable bidirectional information exchange between visual and audio nodes. Visual nodes receive information from audio nodes in their cross-modal neighborhood, and audio nodes receive information from visual nodes in their cross-modal neighborhood. This mechanism can explicitly model local audiovisual mismatch cues such as mismatch between speech content and lip movements, abnormal audio-visual synchronization caused by local audio replacement, and inconsistency between visual facial expression changes and audio semantics. Intramodal attention learning and cross-modal attention learning are implemented using single-head attention, multi-head attention, graph attention, graph convolution, or other message passing mechanisms.

5. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S4 is: the multi-scale temporal fusion is performed after window-level graph interaction, along with context feedback; The visual node representation and audio node representation in the previous window node are input into gating fusion to obtain a window-level fusion sequence; The fused sequence is input into multi-scale temporal modeling to extract short-scale, medium-scale, and long-scale temporal information, respectively. The results of multi-scale temporal modeling are used to form long-range contexts, enabling local window evidence to be judged in conjunction with the context of the entire video segment. The long-range context is fed back to the window node through context feedback, resulting in the updated window node. Fusion context = Gated weight × Global context + (1 - Gated weight) × Multi-scale local context + Differential context; Multi-scale temporal modeling can be achieved through temporal convolutions with different kernel sizes, different dilation rates, and different time lengths, or through Transformer, state-space model, temporal pyramid, or other long- and short-range temporal modeling structures.

6. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: S5 specifically involves setting up a video-level classification branch and a window-level auxiliary classification branch. The video-level classification branch performs global pooling on the window representation after multi-scale fusion and outputs the forgery probability of the entire video through the video-level classification head. The window-level auxiliary classification branch classifies the window representation and outputs the abnormal response score for each window. If the probability of video-level forgery is higher than a preset threshold, output that the video is a forged video; otherwise, output that the video is a real video. Window-level anomaly response scores are used to assist in the analysis of suspicious anomaly time periods; Window with a response score higher than a preset threshold is marked as an abnormal window, and temporally adjacent abnormal windows are merged into candidate abnormal segments, and their start and end times and confidence scores are output.

7. A system for locating audio and video forgery segments in a time sequence, characterized in that: The system is used to execute the localization method as described in any one of claims 1 to 6. The localization system includes an input acquisition module, a window partitioning module, a bimodal feature extraction module, an audiovisual heterogeneous graph construction module, a time-aware intramodal interaction module, a bidirectional cross-modal interaction module, a multi-scale temporal fusion module, a context feedback module, a video-level classification module, a window-level auxiliary classification module, a training optimization module, and an abnormal time region analysis module.

Images