Audio and video counterfeit segment time sequence positioning method and system

Abstract

The present application relates to a kind of audio and video counterfeit segment timing positioning method and system, facing audio and video depth counterfeit content analysis scene, video modal feature and audio modal feature are jointly modeled, and through hierarchical timing coding, multiscale feature fusion, candidate quality evaluation and boundary positioning processing, the automatic positioning of the start time of counterfeit segment is realized.The present application is deployed in general electronic equipment by program mode, for audio and video content review, media authenticity verification, digital forensics analysis and related content security scene, the present application can improve candidate sorting reliability and boundary positioning precision, improve the positioning precision of counterfeit segment in short time local scene and candidate sorting quality.

Description

Technical Field

[0001] This application belongs to the fields of artificial intelligence, multimedia content security analysis and multimodal temporal information processing technology, specifically involving a method and system for temporal localization of audio and video forgery segments. Background Technology

[0002] With the development of deepfake generation and cross-modal editing technologies, the forms of forgery in audio and video content have gradually expanded from whole-segment manipulation to fragmented manipulation within local time intervals. Compared to video-level detection tasks that only output true / false results, practical applications require determining the start time, end time, and duration of forged segments to support applications such as content review, authenticity verification, and digital forensics. Currently, existing technologies typically employ a temporal localization framework, which generates candidate segments after temporal modeling of audio and video features and outputs classification and boundary prediction results to achieve forged segment localization.

[0003] However, such schemes typically have the following shortcomings: First, the ranking of candidate segments mainly relies on the classification results, which makes it difficult to reflect the degree of matching between candidate boundaries and real boundaries, resulting in inconsistencies between the ranking results and the actual localization quality. Second, boundary prediction often adopts continuous regression, which is insufficient for representing forged segments with blurred boundaries, unclear transitions, or local uncertainties. Third, for audio and video forgery scenarios where audio anomalies, video anomalies, and cross-modal mismatches coexist, existing schemes still have room for improvement in terms of multimodal joint modeling, hierarchical temporal representation, and high-quality candidate retention.

[0004] In scenarios involving the localization of forged audio and video segments, the forged intervals are typically short in duration, with weak anomalous signals near the boundaries. Furthermore, the anomalies may originate from changes in video content, audio content, or mismatches in the audiovisual correspondence. These characteristics render traditional methods relying on classification result ranking and continuous boundary regression inaccurate in high-precision localization scenarios. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of the prior art and provide a method and system for temporal localization of audio and video forgery segments, so as to improve the reliability of candidate sorting and the accuracy of boundary positioning, and improve the positioning accuracy and candidate sorting quality of forgery segments in short-term local scenes.

[0006] The technical problem solved by this application is achieved through the following technical solution: A method for temporal localization of audio and video forgery segments, the method comprising the following steps: S1. Obtain the audio and video data to be analyzed, and construct joint temporal features based on video modal features and audio modal features; S2. Perform hierarchical temporal coding on the joint temporal features to obtain feature representations at multiple resolution levels; S3. The feature representations of the multiple resolution levels are fused to obtain multi-scale fused features for forged fragment localization; S4. Based on the multi-scale fusion features, output the classification results, localization quality results, and boundary prediction results of the candidate segments; S5. Reorder the candidate segments according to the classification results and the localization quality results, and decode the boundaries of the candidate segments according to the boundary prediction results to obtain the localization results of the forged segments.

[0007] Moreover, the first step is to acquire the audio and video data to be analyzed, including video sequences and corresponding audio sequences; Subsequently, in the feature extraction stage, video modal features and audio modal features are extracted respectively. The video modal features include visual representation features and motion-related features, and the audio modal features include acoustic representation features. In the feature construction stage, the video modal features and audio modal features are subjected to time alignment, channel concatenation, resampling and mapping processing to map them to a unified temporal grid and a unified hidden dimension to obtain embedded features. The embedded features are input into the hierarchical temporal backbone and continuously downsampled to form backbone features at multiple resolution levels. The backbone features are then input into a multi-scale fusion module to generate multi-level fusion features. During the prediction stage, classification results, localization quality results, and boundary distribution results are output simultaneously for each scale location. The multi-scale prediction results are then subjected to score recalibration, candidate screening, boundary decoding, soft suppression, and voting correction. Finally, the start time, end time, and corresponding confidence level of the forged fragment are output.

[0008] Furthermore, S2 and S3 specifically involve the following steps: the embedded features first enter the hierarchical temporal coding backbone, then sequentially pass through local window coding, offset window coding, and multi-level downsampling to form multiple levels of backbone features; subsequently, the backbone features are fused into multiple levels of fused features through top-down upsampling and lateral fusion; local window coding is used to model short-range dependencies within a local time window to enhance the local structural representation near the boundary of short-term forged fragments; offset window coding is used to establish cross-window interactions between adjacent windows to compensate for the information fragmentation caused by simple local window modeling; multi-level downsampling abstracts the features layer by layer to form hierarchical representations with different resolutions; thereafter, high-level semantic information is passed down to lower levels through a multi-scale fusion structure, and low-level details are used to supplement high-level semantics, thereby achieving collaborative modeling of local boundary cues and long-range contextual semantics.

[0009] Furthermore, S4 specifically involves: a quality head set outside the classification head, used to explicitly output the localization quality results of candidate segments corresponding to each scale position; during the inference phase, the localization quality results are fused with the classification results, so that the ranking score simultaneously reflects the probability of a candidate segment being forged and the degree of matching between the candidate segment boundary and the target boundary; when a candidate segment has a high classification score but a large boundary deviation, its comprehensive score will be suppressed due to the low localization quality result; when a candidate segment has a relatively low classification score but its boundary matches the target segment better, its comprehensive score can be improved, thereby obtaining a better position in the final ranking; The boundary header outputs the left and right boundary distributions for each position. These distributions can be trained through distribution supervision to enable the model to learn the probability distribution of the boundary in a discrete interval, rather than learning a single continuous landing point. When there are fuzzy transitions, gradual intervals, or local multiple solutions at the boundary of a forged fragment, the model can still express the boundary uncertainty in a distributional form, thereby improving the boundary representation capability in short fragment scenarios. In the boundary decoding stage, the left and right boundary offsets are determined based on the predicted left and right boundary distributions and further decoded into the start and end times of the candidate fragments. Interval geometric constraints are introduced to constrain the consistency between the predicted boundary and the target boundary from the perspective of overall interval matching. This is achieved through a joint boundary supervision approach that combines distributed modeling, boundary decoding, and interval geometric constraints.

[0010] Moreover, S5 specifically involves: firstly summarizing the point-by-point outputs for all scale locations, where each location includes at least the classification result, the localization quality result, and the boundary distribution result; then, in the score recalibration stage, fusing the classification result and the localization quality result to obtain the comprehensive score of the candidate segments; and in the candidate selection stage, preferably performing threshold filtering on the comprehensive score first, and then retaining the candidate segments with higher scores. Subsequently, the boundary decoding method obtains the start and end times of candidate segments by decoding the reference position and the predicted offset. In the soft suppression stage, a soft suppression strategy is used to continuously reduce the scores of highly overlapping candidate segments. In the voting correction stage, the boundary is further corrected by using neighboring high-scoring candidates to further improve the stability and accuracy of the final output boundary. Through a continuous post-processing link of score recalibration, candidate screening, boundary decoding, soft suppression and voting correction, high-quality candidates are retained and redundant candidates with large boundary deviations are suppressed.

[0011] A temporal localization system for audio and video forgery segments, used to execute the localization method, the system comprising: The data acquisition module is used to acquire the audio and video data to be analyzed. A joint feature construction module is used to extract video-side features and audio-side features, and to perform time alignment, resampling and embedding mapping on the video-side features and audio-side features; The hierarchical temporal coding module is used to perform hierarchical temporal coding on the embedded features and output multi-level backbone features; A multi-scale fusion module is used to fuse the multi-level backbone features and output multi-level fused features; The joint prediction module is used to output classification results, localization quality results, and boundary distribution results; The candidate reordering and boundary decoding module is used to perform score recalibration, candidate filtering, boundary decoding, soft suppression, and vote correction on candidate segments; The results output module is used to output the start time, end time, and confidence level of the forged fragment.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention, by jointly constructing video modal features and audio modal features, can form a multimodal temporal representation at a unified time scale, thereby improving the feature utilization efficiency in the task of locating audio and video forged segments; 2. This invention, through hierarchical temporal coding and multi-scale fusion, can take into account both local boundary details and long-range contextual information, thereby improving the temporal representation capability of short-term forged fragments; 3. This invention ranks candidate segments by fusing classification results and localization quality results, which can improve the consistency between candidate ranking results and actual localization quality. 4. This invention enhances boundary representation capabilities and improves boundary positioning accuracy through boundary distribution prediction and interval geometric constraints; 5. This invention is deployed on general electronic devices through a program, and has good engineering feasibility and application adaptability. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the hierarchical temporal coding and multi-scale fusion structure of the present invention; Figure 3 This is a schematic diagram of the quality-perceived reordering of the present invention; Figure 4 This is a schematic diagram of the boundary monitoring of the present invention; Figure 5 This is a schematic diagram of the reasoning process of the present invention; Figure 6 This is a schematic diagram illustrating the training supervision of the present invention. Detailed Implementation

[0014] The present application will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present application.

[0015] like Figure 1 As shown, a method for temporal localization of audio and video forgery segments is innovative in that the steps of the method are as follows: S1. Obtain the audio and video data to be analyzed, and construct joint temporal features based on video modal features and audio modal features; S2. Perform hierarchical temporal coding on the joint temporal features to obtain feature representations at multiple resolution levels; S3. The feature representations of the multiple resolution levels are fused to obtain multi-scale fused features for forged fragment localization; S4. Based on the multi-scale fusion features, output the classification results, localization quality results, and boundary prediction results of the candidate segments; S5. Reorder the candidate segments according to the classification results and the localization quality results, and decode the boundaries of the candidate segments according to the boundary prediction results to obtain the localization results of the forged segments.

[0016] like Figure 1 As shown, the overall process of this invention includes stages such as input, feature extraction, feature construction, backbone network, multi-scale fusion, prediction, inference, and output. In the figure, 101 represents input, 102 represents feature extraction, 103 represents feature construction, 104 represents backbone network, 105 represents multi-scale fusion, 106 represents prediction, and 107 represents inference. First, the audio and video data to be analyzed is acquired, including video sequences and corresponding audio sequences. Subsequently, in the feature extraction stage, video modal features and audio modal features are extracted respectively. The video modal features include visual representation features and motion-related features, and the audio modal features include acoustic representation features. In the feature construction stage, the video modal features and audio modal features are subjected to time alignment, channel concatenation, resampling and mapping processing to map them to a unified temporal grid and a unified hidden dimension to obtain embedded features. The embedded features are input into the hierarchical temporal backbone and continuously downsampled to form backbone features at multiple resolution levels. The backbone features are then input into a multi-scale fusion module to generate multi-level fusion features. During the prediction stage, classification results, localization quality results, and boundary distribution results are output simultaneously for each scale location. The multi-scale prediction results are then subjected to score recalibration, candidate screening, boundary decoding, soft suppression, and voting correction. Finally, the start time, end time, and corresponding confidence level of the forged fragment are output.

[0017] like Figure 2As shown in the figure, 201 represents the input, 202 represents the backbone encoding, and 203 represents multi-scale fusion. The embedded features first enter the hierarchical temporal encoding backbone, and then pass through local window encoding, offset window encoding, and multi-level downsampling to form multiple levels of backbone features. Subsequently, the backbone features are fused into multiple levels of fused features through top-down upsampling and lateral fusion. Local window encoding is used to model short-range dependencies within a local time window to enhance the local structural expression near the boundary of short-term forged fragments. Offset window encoding is used to establish cross-window interactions between adjacent windows to compensate for the information fragmentation caused by simple local window modeling. Multi-level downsampling abstracts the features layer by layer to form hierarchical representations with different resolutions. After that, the high-level semantic information is passed down to the low level through the multi-scale fusion structure, and the low-level details are used to supplement the high-level semantics, thereby realizing the collaborative modeling of local boundary cues and long-range contextual semantics.

[0018] like Figure 3 As shown in the figure, 301 represents the classification and sorting results, and 302 represents the re-sorting results. Figure 3 The left side shows the sorting results when candidate segments are sorted only based on classification results; Figure 3 The right side illustrates the process of fusing the classification results and quality results to obtain a comprehensive score after introducing the quality head, and then re-ranking the candidate segments according to the comprehensive score. The quality head, located outside the classification head, is used to explicitly output the localization quality results of candidate segments at each scale position. During the inference phase, the localization quality results are fused with the classification results, so that the ranking score simultaneously reflects the probability of a candidate segment being forged and the degree of matching between the candidate segment boundary and the target boundary. When a candidate segment has a high classification score but a large boundary deviation, its comprehensive score will be suppressed due to the low localization quality result; conversely, when a candidate segment has a relatively low classification score but its boundary matches the target segment better, its comprehensive score can be improved, thus obtaining a better position in the final ranking. like Figure 4 As shown in the figure, 401 represents distribution modeling, 402 represents boundary decoding, and 403 represents interval geometric constraints. The boundary header does not directly output a single continuous boundary value, but instead outputs the left and right boundary distributions respectively. For each position, the boundary header outputs the left and right boundary distributions respectively. These distribution results can be trained through distribution supervision, enabling the model to learn the probability distribution of the boundary in a discrete interval, rather than learning a single continuous landing point. When the forged fragment boundary exhibits fuzzy transitions, gradual intervals, or local multiple solutions, the model can still express boundary uncertainty in a distributional form, thereby improving the boundary representation capability in short fragment scenarios. In the boundary decoding stage, the left and right boundary offsets are determined based on the predicted left and right boundary distributions and further decoded into the start and end times of the candidate fragment. Interval geometric constraints are introduced to constrain the consistency between the predicted boundary and the target boundary from the perspective of overall interval matching. This is achieved through a joint boundary supervision approach that combines distributed modeling, boundary decoding, and interval geometric constraints.

[0019] like Figure 5 As shown in the figure, 501 represents multi-scale prediction, 502 represents score recalibration, 503 represents candidate screening, 504 represents boundary decoding, 505 represents soft suppression, and 506 represents voting correction. Figure 5 The post-processing chain of the inference stage of this invention is presented. First, the point-by-point outputs of all scale locations are summarized, where each location includes at least the classification result, the localization quality result, and the boundary distribution result; then, in the score recalibration stage, the classification result and the localization quality result are fused to obtain the comprehensive score of the candidate segments; in the candidate selection stage, preferably, threshold filtering is first performed on the comprehensive score, and then the candidate segments with higher scores are retained. Subsequently, the boundary decoding method obtains the start and end times of candidate segments by decoding the reference position and the predicted offset. In the soft suppression stage, a soft suppression strategy is used to continuously reduce the scores of highly overlapping candidate segments. In the voting correction stage, the boundary is further corrected by using neighboring high-scoring candidates to further improve the stability and accuracy of the final output boundary. Through a continuous post-processing link of score recalibration, candidate screening, boundary decoding, soft suppression and voting correction, high-quality candidates are retained and redundant candidates with large boundary deviations are suppressed.

[0020] like Figure 6 As shown in the figure, 601 corresponds to the original ground truth segment interval, 602 corresponds to the center sampling region, and 603 corresponds to the supervision label after label smoothing. The figure illustrates the construction method of training supervision using the ground truth segment center and the center sampling radius. To reduce the impact of boundary noise on classification supervision, a center sampling strategy is adopted during the training phase, treating only feature positions within the center region of the ground truth segment as high-confidence positive samples. This strategy makes the training focus more on the center region of the ground truth segment, thereby reducing the adverse effects of blurred labels near the boundary on classification learning. Based on center sampling, this invention further employs a label smoothing strategy to smooth the classification labels, thereby alleviating the overconfidence of the classification head in positive samples and improving training stability. Center sampling and label smoothing together constitute an important supervision strategy in the training phase of this invention, used to improve the classification robustness in short segment boundary scenarios.

[0021] Parameter configuration of preferred embodiments In a preferred embodiment, the present invention may employ the following parameter configuration: input modalities are visual features and audio features; input feature dimensions are 4096 and 2048, respectively; maximum sequence length is 576; backbone hidden dimension is 256; number of attention heads is 4; window size is 7; training optimizer is AdamW; initial learning rate is 0.001; number of training epochs is 10; weight decay is 0.05; batch size is 32; center sampling radius is 1.5; label smoothing coefficient is 0.1; number of discrete intervals in boundary distribution is 16; quality loss weight is 0.5; distribution supervision loss weight is 0.5; inference filtering threshold is 0.001; soft suppression smoothing coefficient is 0.50; upper limit of soft suppression front candidates is 1000. The above parameters are merely a configuration in a preferred embodiment and do not constitute a limitation on the scope of protection of the present invention.

[0022] The visual features can be extracted by feature extraction networks such as TSN, and the audio features can be extracted by audio representation networks such as BYOL-A. For different data scales, computing resources, and business scenarios, those skilled in the art can make equivalent adjustments to the feature extraction network type, input dimension, number of layers, and training parameters.

[0023] Implementation effect verification By testing the audio / video forgery segment localization task, the improved performance of this invention in terms of candidate ranking consistency, boundary localization accuracy, and localization stability under high threshold conditions can be verified. Test results show that this invention improves the retention of high-quality candidates by introducing a fusion mechanism between localization quality results and classification results; enhances boundary alignment accuracy through boundary distribution prediction and interval geometric constraints; and strengthens the collaborative expression of local boundary information and contextual semantic information through hierarchical temporal coding and multi-scale fusion.

[0024] The solution of this invention has achieved better positioning results on publicly available test data, indicating that the invention has good applicability in the scenario of locating audio and video forged segments.

[0025] A temporal localization system for audio and video forgery segments, used to execute the aforementioned localization method, is innovative in that: the system comprises: The data acquisition module is used to acquire the audio and video data to be analyzed. A joint feature construction module is used to extract video-side features and audio-side features, and to perform time alignment, resampling and embedding mapping on the video-side features and audio-side features; The hierarchical temporal coding module is used to perform hierarchical temporal coding on the embedded features and output multi-level backbone features; A multi-scale fusion module is used to fuse the multi-level backbone features and output multi-level fused features; The joint prediction module is used to output classification results, localization quality results, and boundary distribution results; The candidate reordering and boundary decoding module is used to perform score recalibration, candidate filtering, boundary decoding, soft suppression, and vote correction on candidate segments; The results output module is used to output the start time, end time, and confidence level of the forged fragment.

[0026] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A method for temporal localization of audio and video forgery segments, characterized in that: The steps of the method are as follows: S1. Obtain the audio and video data to be analyzed, and construct joint temporal features based on video modal features and audio modal features; S2. Perform hierarchical temporal coding on the joint temporal features to obtain feature representations at multiple resolution levels; S3. The feature representations of the multiple resolution levels are fused to obtain multi-scale fused features for forged fragment localization; S4. Based on the multi-scale fusion features, output the classification results, localization quality results, and boundary prediction results of the candidate segments; S5. Reorder the candidate segments according to the classification results and the localization quality results, and decode the boundaries of the candidate segments according to the boundary prediction results to obtain the localization results of the forged segments.

2. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: First, acquire the audio and video data to be analyzed, including video sequences and corresponding audio sequences; Subsequently, in the feature extraction stage, video modal features and audio modal features are extracted respectively. The video modal features include visual representation features and motion-related features, and the audio modal features include acoustic representation features. In the feature construction stage, the video modal features and audio modal features are subjected to time alignment, channel splicing, resampling and mapping processing to map them to a unified temporal grid and a unified hidden dimension to obtain embedded features; The embedded features are input into the hierarchical temporal backbone and continuously downsampled to form backbone features at multiple resolution levels. The backbone features are then input into the multi-scale fusion module to generate multi-level fused features. During the prediction phase, classification results, localization quality results, and boundary distribution results are output simultaneously for each scale location. Score recalibration, candidate screening, boundary decoding, soft suppression, and voting correction are performed on the multi-scale prediction results, and finally the start time, end time, and corresponding confidence level of the forged fragment are output.

3. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S2 and S3 are as follows: the embedded features first enter the hierarchical temporal coding backbone, and then pass through local window coding, offset window coding, and multi-level downsampling to form multiple levels of backbone features; subsequently, the backbone features are fused into multiple levels of fused features through top-down upsampling and lateral fusion; local window coding is used to model short-range dependencies within a local time window to enhance the local structural representation near the boundary of short-term forged fragments; offset window coding is used to establish cross-window interactions between adjacent windows to compensate for the information fragmentation caused by simple local window modeling; multi-level downsampling allows the features to be abstracted layer by layer to form hierarchical representations with different resolutions; thereafter, high-level semantic information is passed down to lower levels through a multi-scale fusion structure, and low-level details are used to supplement high-level semantics, thereby achieving collaborative modeling of local boundary cues and long-range contextual semantics.

4. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S4 involves: setting a quality head outside the classification head to explicitly output the localization quality results of candidate segments at each scale position; during the inference phase, fusing the localization quality results with the classification results so that the ranking score simultaneously reflects the probability of a candidate segment being forged and the degree of matching between the candidate segment boundary and the target boundary; when a candidate segment has a high classification score but a large boundary deviation, its overall score will be suppressed due to the low localization quality result; when a candidate segment has a relatively low classification score but its boundary matches the target segment better, its overall score can be improved, thereby obtaining a better position in the final ranking; The boundary header outputs the left and right boundary distributions for each position. These distributions can be trained through distribution supervision to enable the model to learn the probability distribution of the boundary in a discrete interval, rather than learning a single continuous landing point. When there are fuzzy transitions, gradual intervals, or local multiple solutions at the boundary of a forged fragment, the model can still express the boundary uncertainty in a distributional form, thereby improving the boundary representation capability in short fragment scenarios. In the boundary decoding stage, the left and right boundary offsets are determined based on the predicted left and right boundary distributions and further decoded into the start and end times of the candidate fragments. Interval geometric constraints are introduced to constrain the consistency between the predicted boundary and the target boundary from the perspective of overall interval matching. This is achieved through a joint boundary supervision approach that combines distributed modeling, boundary decoding, and interval geometric constraints.

5. The method for time-series localization of audio and video forged segments according to claim 1, characterized in that: Specifically, S5 involves: first, summarizing the point-by-point outputs for all scale locations, where each location includes at least the classification result, the localization quality result, and the boundary distribution result; then, in the score recalibration stage, fusing the classification result and the localization quality result to obtain the comprehensive score of the candidate segment. In the candidate selection stage, it is preferable to first perform threshold filtering on the comprehensive score, and then retain the candidate segments with higher scores; Subsequently, the boundary decoding method uses the reference position and predicted offset to decode the start and end times of the candidate segments; In the soft suppression stage, a soft suppression strategy is used to continuously reduce the score of highly overlapping candidate segments; During the voting correction stage, the boundary is further corrected by using neighboring high-scoring candidates to improve the stability and accuracy of the final output boundary. Through a continuous post-processing chain of score recalibration, candidate screening, boundary decoding, soft suppression and voting correction, high-quality candidates are retained and redundant candidates with large boundary deviations are suppressed.

6. A system for locating forged audio and video segments in a specific time sequence, characterized in that: The system is used to perform the positioning method as described in any one of claims 1 to 5, the system comprising: The data acquisition module is used to acquire the audio and video data to be analyzed. A joint feature construction module is used to extract video-side features and audio-side features, and to perform time alignment, resampling and embedding mapping on the video-side features and audio-side features; The hierarchical temporal coding module is used to perform hierarchical temporal coding on the embedded features and output multi-level backbone features; A multi-scale fusion module is used to fuse the multi-level backbone features and output multi-level fused features; The joint prediction module is used to output classification results, localization quality results, and boundary distribution results; The candidate reordering and boundary decoding module is used to perform score recalibration, candidate filtering, boundary decoding, soft suppression, and vote correction on candidate segments; The results output module is used to output the start time, end time, and confidence level of the forged fragment.

Images