Methods, terminals, and media for long-form video understanding of multimodal audio and video data

Abstract

This invention discloses a method, terminal, and medium for long-video understanding of multimodal audio and video data. The method includes: extracting bimodal features of audio and video based on multi-view dynamic alignment and hierarchical masking iteratively; removing redundant features and strengthening global core discriminative features based on a dynamic prototype library and hierarchical pruning; performing deep fusion of cross-modal features using bidirectional cross-attention; performing long-term pre-training based on spatiotemporal density correlation modeling; and performing hierarchical partitioning and dual candidate pooling of the pre-trained features, inputting them into a large language model to complete deep parsing of the long video. This invention effectively solves the problems of feature redundancy and long-term semantic fragmentation, improving the accuracy and coherence of long-video understanding.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal video understanding technology, and in particular to a method, terminal, and medium for long video understanding of multimodal audio and video data. Background Technology

[0002] With the rapid development of digital media technology and the explosive growth of video content, long-form video has become an important carrier for internet content dissemination, intelligent analysis, and industrial applications, playing a crucial role in video content understanding, intelligent recommendation, security monitoring, education, and entertainment. Long-form video understanding technology can extract core semantic information and capture long-term logical connections from massive amounts of audio and video temporal data, providing accurate content analysis and decision support for various intelligent applications. It is a research hotspot in the fields of computer vision and artificial intelligence. Especially in scenarios such as intelligent video retrieval, immersive interactive experiences, and industrial video inspection, high-precision and highly coherent long-form video understanding capabilities can significantly improve the content parsing efficiency and scenario adaptability of intelligent systems.

[0003] Although long-video understanding technology has made some progress, long-video data itself has inherent characteristics such as large temporal span, complex modal information, and high feature redundancy. Furthermore, the actual acquired audio and video data often faces quality defects such as blurry images, object occlusion, and cluttered audio backgrounds, posing numerous challenges to the accurate understanding of long videos. Existing long-video understanding methods are mostly based on traditional multimodal feature extraction and fusion strategies, employing modality-specific encoders combined with fixed processing mechanisms for feature parsing. This not only easily overlooks key features when processing defective audio and video data, leading to a significant reduction in feature representation quality, but also struggles to effectively remove redundant information such as spatial repetition, temporal redundancy, and intramodal invalid information in multimodal features, resulting in the dilution of core discriminative features and a substantial decrease in the efficiency and accuracy of feature representation.

[0004] Existing technologies lack efficient adaptation mechanisms for heterogeneous audio and video features in cross-modal fusion, making it difficult to break down information barriers between modalities and fully explore cross-modal semantic relationships and long-term logic. In long-term feature processing, fixed sampling and modeling strategies are often employed, easily leading to problems such as spatiotemporal information fragmentation and event density imbalance. This results in models failing to effectively learn the spatiotemporal attributes and long-term patterns of long videos, causing long-term semantic breaks. Furthermore, the total number of long-term feature tokens after processing often exceeds the context window capacity of large language models. Existing fixed-ratio token compression methods cannot accurately remove redundant tokens and easily damage core temporal logic, leading to fragmented analytical logic in the understanding of long videos by large language models, severely affecting the coherence and reliability of long video understanding. These intertwined problems result in significant deficiencies in existing long video understanding technologies regarding the robustness of feature extraction, the efficiency of fusion, the accuracy of long-term modeling, and adaptability to large models, making it difficult to meet the demands for deep and accurate understanding of long videos in practical applications. Summary of the Invention

[0005] To address the above problems, this invention provides a method, terminal, and medium for long video understanding of multimodal audio and video data, which effectively solves the problems of feature redundancy and long-term semantic fragmentation, and improves the accuracy and coherence of long video understanding tasks.

[0006] In a first aspect, the present invention provides a long video understanding method for multimodal audio and video data, comprising: S1, based on multi-view dynamic alignment and hierarchical masking iteration, features are extracted from the original audio and video bimodal data to obtain audio and video bimodal features; S2, based on a dynamic prototype library and hierarchical pruning, performs redundant purification of audio and video dual-modal features, removes multimodal redundant features, and strengthens global core discriminative features; S3 employs bidirectional cross-attention to perform cross-modal feature deep fusion of enhanced audio and video features; S4, based on spatiotemporal density correlation modeling, performs long-time processing on the deeply fused audio and video features to obtain pre-trained features; S5 performs hierarchical segmentation and dual candidate pooling of pre-trained features, and inputs them into a large language model to complete deep parsing of long videos.

[0007] Furthermore, S1 specifically includes: S11: Obtain the audio and video dual-modal data of the original long video and construct the video light field features and frequency domain audio field features from multiple perspectives; S12, based on video light field features and frequency domain audio field features, quantifies and normalizes the reliability of local audio and video regions, and through an adaptive dynamic offset mechanism, weighted aligns the multi-view features to the central viewpoint to obtain the aligned multi-view feature set. S13, construct an adaptive binary initial mask based on the aligned multi-view features, perform single-layer feature optimization by combining weight correction and modality aggregation, and perform hierarchical mask iterative optimization by accumulating and updating historical masks; S14: Render the optimized features from the target perspective to obtain the initial unobstructed audio and video features. Then, repair the feature details through modal concatenation residual convolution to output robust audio and video dual-modal features.

[0008] By combining multi-view dynamic alignment and hierarchical masking iteration with local reliability quantization and adaptive offset mechanisms, it can effectively repair quality defects such as occlusion and blurring in the original audio and video, and output robust bimodal features with the help of modal concatenated residual convolution, significantly improving the completeness and accuracy of feature extraction.

[0009] Furthermore, S2 specifically includes: S21. Based on the audio and video dual-modal features, the reliability of the features is quantified from both local and global dimensions, and the quantization results are normalized. S22, a dynamic prototype library is built based on global reliability, and feature filtering is performed to remove temporally redundant features, resulting in an audio and video feature set; S23 employs a hierarchical pruning strategy to sequentially remove modal and spatial redundant features from the audio and video feature set, obtaining a core feature subset. The core feature subset is then weighted and its features are aggregated. The dynamic prototype library is updated and iterative optimization is completed, outputting the enhanced global core features of the audio and video.

[0010] By utilizing dual-dimensional reliability quantization, a dynamic prototype library, and a hierarchical pruning strategy, redundant features within and in the modality space can be accurately eliminated. Furthermore, by updating the prototype library through weight enhancement and feature aggregation, the global core discriminative features are effectively strengthened, thereby improving the efficiency and discriminative power of feature representation.

[0011] Furthermore, the hierarchical pruning strategy specifically includes: The audio and video feature set is pruned according to the pruning ratio to remove redundant features within the modality; Calculate the cosine similarity of the flattened vectors of audio and video features before and after pruning. If the cosine similarity is greater than the preset validity threshold, the pruning is deemed valid. Calculate the centered cosine similarity between the pruned effective features and the dynamic prototype library, and remove features whose centered cosine similarity is greater than a preset similarity threshold to remove spatially redundant information.

[0012] In hierarchical pruning, cosine similarity is introduced to verify the effectiveness of pruning, and spatially redundant features are removed by centered cosine similarity. This ensures that key information is not lost while maximizing the compression of redundancy, thereby improving the safety and feature quality of the pruning process.

[0013] Furthermore, S3 specifically includes: S31, based on the enhanced global core features of audio and video, completes the unification of feature dimensions and distribution calibration through linear transformation, layer normalization and random deactivation operation, and obtains dimension-adapted single-modal features. S32 employs a bidirectional cross-attention mechanism to perform bidirectional semantic interaction on dimension-adapted audio and video features, and fuses the dimension-adapted audio and video features with the interaction results through residual connections. S33 performs dimensional concatenation on the interactively fused audio and video features, inputs the concatenated features into a fully connected network containing the ReLU activation function, and outputs cross-modal fused features.

[0014] By employing a bidirectional cross-attention mechanism combined with residual connections, deep semantic interaction and complementary information fusion of audio and video features are achieved. Furthermore, cross-modal fusion features are output through dimensional unification and nonlinear mapping, effectively breaking down information barriers between modalities and enhancing the semantic richness of the fused features.

[0015] Furthermore, S4 specifically includes: S41, based on cross-modal fusion features, is decomposed into short video segment features through overlapping truncation and generates global temporal metadata. Combined with topological and temporal localization embedding to mine spatiotemporal correlation, spatiotemporally enhanced segment features are obtained through fusion. S42 quantifies the variance of segment features and normalizes it to obtain the event density value, which is then fused with the spatiotemporally enhanced segment features. The density equalization sampling and contrastive learning are combined to balance the differences in event density among the segments and output the enhanced features. S43, enhance the feature mapping back to the long video temporal coordinates, weighted fuse the overlapping region features and reconstruct the long temporal sequence to obtain the long temporal reconstructed features, introduce a long temporal self-attention layer to capture global semantic associations and obtain the global association matrix; S44: Based on the global correlation matrix, the long-term reconstructed feature value is quantified in multiple dimensions and an effective information mask is generated. After filtering out noisy features, feature gaps are filled, and long-term regular core features are obtained through attention pooling. S45 performs layer normalization on the core features of long-term patterns and outputs pre-trained features.

[0016] Based on spatiotemporal density correlation modeling, through overlapping segment decomposition, event density quantization, density equalization sampling and contrastive learning, long-term self-attention and multi-task loss optimization, it can overcome the semantic fragmentation and density imbalance problems in long videos and output pre-trained features that focus on long-term patterns.

[0017] Furthermore, the method of combining topological and temporal localization embedding to mine spatiotemporal correlations and fusing them to obtain spatiotemporally enhanced fragment features specifically includes: Cosine similarity is used to measure the semantic association strength between features of any two frames within a short video clip, and an adjacency matrix is ​​constructed. By introducing the graph Laplace decomposition method, the adjacency matrix is ​​decomposed to obtain the graph Laplace matrix; Eigenvalue decomposition is performed on the graph Laplacian matrix, and the eigenvectors obtained from the eigenvalue decomposition are mapped to the same dimensional space as the fragment features to obtain the topological embedding; The sinusoidal position code used to characterize the relative position of each frame within a segment is concatenated with the global periodic embedding used to characterize the global temporal attributes of the segment. The concatenated vector is then compressed to the same dimension as the segment features to obtain the temporal localization embedding. Fragment features are added element-wise with topological embedding and temporal localization embedding to obtain spatiotemporally enhanced fragment features.

[0018] By using graph Laplacian decomposition to generate topological embeddings, and combining sinusoidal positional encoding and global periodic embedding to construct temporal localization embeddings, and adding them element-wise with fragment features, we can deeply explore the spatiotemporal correlations between frames within a fragment and significantly enhance the spatiotemporal expressive power of fragment features.

[0019] Furthermore, S5 specifically includes: S51 treats each feature vector of the pre-trained features as a token and binds it to a time-series label. Candidate key time periods are generated through an adaptive sliding window. The temporal and semantic correlation between the token and the candidate key time periods is quantified to classify the tokens. S52. Based on the quantization results, the tokens are processed in layers. The core features of each time period are selected as global layer tokens and the remaining features are selected as local layer tokens. The comprehensive reliability score of the local layer tokens and the semantic correlation between the global layer tokens and the local layer tokens are calculated. S53, construct a dual-candidate pool to classify and manage tokens. Global layer tokens are directly entered into the pool, and local layer tokens are calculated based on comprehensive reliability score and semantic relevance. Tokens are sorted and filtered according to their comprehensive scores and then entered into the pool. S54, constrained by the context window capacity of the large language model, prioritizes the selection of global layer tokens, and selects local layer tokens with high comprehensive scores from the remaining capacity. The selected tokens are concatenated and rearranged according to the original temporal order to obtain a compressed token set, which is then input into the large language model to complete the deep parsing of long videos.

[0020] By performing hierarchical segmentation and dual candidate pooling of pre-trained features, the core tokens of the global layer are retained first, and the tokens of the local layer are selected based on their merits. This approach maintains the original temporal logic while adapting to the context window capacity of a large language model, effectively improving the coherence and accuracy of deep analysis of long videos.

[0021] Secondly, the present invention also provides a computer terminal, comprising: Memory, which stores executable programs; A processor is configured to run the program, wherein the program executes the long video understanding method for multimodal audio and video data during runtime.

[0022] Thirdly, the present invention also provides a computer-readable storage medium comprising a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to execute the long video understanding method for multimodal audio and video data.

[0023] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the long video understanding method for multimodal audio and video data.

[0024] Compared with existing technologies, the beneficial effects of this invention are as follows: Through multi-view dynamic alignment and hierarchical masking iteration, it effectively repairs quality defects such as occlusion and blurring in audio and video, extracting robust bimodal features; based on a dynamic prototype library and hierarchical pruning, it accurately removes redundant multimodal features, strengthens global core discriminative features, and improves feature representation efficiency and discriminative power; employing bidirectional cross-attention and residual connections, it breaks down information barriers between heterogeneous audio and video features, achieving deep cross-modal semantic fusion; based on long-term temporal pre-training using spatiotemporal density correlation modeling, it overcomes the problems of semantic fragmentation and event density imbalance in long videos, outputting high-value features focusing on long-term patterns; finally, through a token compression method using hierarchical partitioning and dual candidate pool selection, it retains core temporal logic while adapting to the context window of a large language model, significantly improving the accuracy and coherence of deep analysis of long videos. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 A comparison chart of data availability between embodiments of the present invention and traditional long video understanding methods under different viewpoint offsets; Figure 3 A comparison chart of data availability between embodiments of the present invention and traditional long video understanding methods under different mask iteration layers; Figure 4 A comparison chart showing the understanding accuracy of the embodiments of the present invention and traditional long video understanding methods under different long-sequence pooling window numbers; Figure 5 This is a comparison chart showing the accuracy of long video question answering between embodiments of the present invention and traditional long video understanding methods under different compression ratios. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.

[0028] This invention provides a method for understanding long videos using multimodal audio and video data, such as... Figure 1 As shown, the specific steps include: S1, based on multi-view dynamic alignment and hierarchical masking iteration, extracts features from the original audio and video bimodal data to obtain audio and video bimodal features.

[0029] Furthermore, S1 includes the following steps: S11: Obtain the audio and video dual-modal data of the original long video, construct the video light field and frequency domain audio field from multiple perspectives, and complete the initial feature representation of the audio and video multi-view field.

[0030] S12 quantifies and normalizes the reliability of local audio and video regions. Through an adaptive dynamic offset mechanism, the multi-view features are weighted and aligned to the central viewpoint to obtain the aligned multi-view feature set.

[0031] Specifically, the reliability of a local video region is measured by the local texture variance, while the reliability of a local audio region is measured by the ratio of the frequency domain energy of a single frequency point to the average frequency domain energy of the entire frequency band. The calculation formula is as follows: ; ; in, The local texture variance within a 3x3 window of the video; For time frames; From a video perspective; For spatial coordinates, For height, Width; This is the offset in the height direction; This is the offset in the width direction; Features of light field in multi-view video; The mean of the features within a 3×3 window; This represents the audio frequency domain energy spectrum value, i.e., the multi-view frequency domain audio field characteristics; From an audio perspective; For frequency; This refers to the audio channel number; The average frequency domain energy across the entire frequency band; This represents the total number of frequency points in the frequency domain. This is to sum the energy over all frequency points.

[0032] Normalize the reliability coefficient: ; ; in, For the video modality at time t, viewpoint Regional reliability coefficient at spatial location (h, w); It is a natural exponential function; This is the normalized scaling factor; This is the texture variance reliability threshold; For the audio modality at time t, viewpoint Frequency f and frequency reliability coefficient at channel c; This represents the frequency domain energy value at a single frequency point. It represents the average frequency domain energy across the entire frequency band.

[0033] To align multi-view features with the central viewpoint and eliminate feature misalignment caused by viewpoint differences, a dynamic offset mechanism is adopted. The offset is determined by both the viewpoint difference and the normalized reliability coefficient. ; ; in, These are video features after multi-view dynamic offset alignment; This represents the video's viewpoint offset. The center viewpoint coordinates of the video; The audio features are after multi-view dynamic offset alignment; This represents the audio's viewpoint offset. The center viewpoint coordinates of the audio.

[0034] After completing the multi-view feature offset alignment, in order to adapt to the new feature distribution and ensure the consistency of the mask and the offset features in the spatial dimension, the local reliability features after offset alignment are recalculated to obtain the video local reliability features. Audio local reliability characteristics .

[0035] S13: An adaptive binary initial mask is constructed based on the aligned multi-view features, and single-layer feature optimization is completed by combining weight correction and modality aggregation.

[0036] Specifically, the expression for the adaptive binary initialization mask is as follows: ; ; in, Provide the initial binary occlusion mask for the first layer of the video modality; For static area mask of video, k is the time frame offset, and I is the pixel value; The threshold for texture variance discrimination; The initial binary occlusion mask for the first layer of the audio modality; This is the frequency domain energy of a single frequency point recalculated after multi-view alignment; The average frequency domain energy of the entire frequency band is recalculated after multi-view alignment; The threshold for determining the proportion of audio energy.

[0037] The aligned audio and video features are input into the fully connected layer to generate initial audio and video features. A weight correction and weak feature preservation mechanism is adopted, and an adaptive binary initial mask is used to weight the initial audio and video features. ; ; in, These are the output video features after weighting using the first layer of masking; These are the initial video features; The retention factor; The output audio features are those after weighting by the first layer of masking; These are the initial audio features.

[0038] After completing the feature weight correction, the i-th layer video features and audio features The input modality aggregation block is used for modality aggregation. This block consists of a channel attention module and a spatial attention module, which respectively generate video modal color maps. and audio modal color map and video modal alpha graph and audio modal alpha graph .

[0039] Specifically, the channel attention module... and Perform global average pooling along the spatial / frequency dimensions to extract video channel descriptors respectively. and audio channel descriptors Subsequently and Video channel weights are generated by performing two layers of bottlenecked linear transformation and activation. and audio channel weights , video features and video channel weight and audio features and audio channel weights By multiplying each channel, we obtain the video modal color map. and audio modal color map .

[0040] Spatial attention module and Average pooling along the channel dimension yields the video modal space response map. Audio modal space response diagram ,Will and Video modality alpha maps are generated by convolutional layers and sigmoid activation, respectively. and audio modal alpha graph .

[0041] Layered mask iterative optimization is achieved through cumulative updates of historical masks. In each iteration, the alpha map of the previous layer and the initial mask are used for cumulative updates to generate the occlusion prior mask for the current layer. Then, nonlinear displacement compensation is used to obtain the final occlusion mask, and the features are weighted for optimization. The above layered mask update and feature optimization steps are repeated until the preset number of iterations is reached, thereby achieving the repair of occluded regions and feature alignment.

[0042] Specifically, the formula for calculating the occlusion prior mask is as follows: ; ; in, The (i+1)th layer occlusion prior mask for the video modality; These are the initial mask weight coefficients; The alpha graph of the i-th layer of the video modality; The (i+1)th layer occlusion prior mask for the audio modality; Let be the alpha graph of the i-th layer of the audio modality.

[0043] The formula for calculating the final occlusion mask of layer i+1 is as follows: ; ; ; ; ; in, The final occlusion mask for the (i+1)th layer of the video modality; Let be the dynamic offset step size of the video viewpoint of the i-th layer; For offset compensation in the height direction; To compensate for the offset in the width direction; This is the final occlusion mask for the (i+1)th layer of the audio modality; The dynamic offset step size for the audio viewpoint of the i-th layer; For frequency direction compensation offset; This is a threshold parameter used to adjust the compensation intensity in the video modality; This refers to the video modal height compensation coefficient. This is the video modal width compensation coefficient; It is a symbolic function; This refers to the audio compensation coefficient; Balance coefficients for aligning audio frequency domain features.

[0044] S14: Render the optimized features from the target perspective to obtain the initial unobstructed audio and video features. Then, repair the feature details through modal concatenation residual convolution to output robust audio and video dual-modal features.

[0045] S2, based on a dynamic prototype library and hierarchical pruning, performs redundant purification of audio and video dual-modal features, eliminates multimodal redundant features, and strengthens global core discriminative features.

[0046] Furthermore, S2 includes the following steps: S21. Based on the audio and video dual-modal features, calculate the local feature response value and the global feature contribution to complete the dual-dimensional quantization and normalization of feature reliability. The specific formula is as follows: ; ; ; ; in, This refers to the local reliability of the video, i.e., the local feature response value of the video. This refers to the local reliability of the audio signal, i.e., the local characteristic response value of the audio signal. Let t be the global reliability of the video, i.e., the contribution of the video's global features; Let t be the global reliability of the audio in frame t, i.e., the contribution of the global audio features.

[0047] S22, a dynamic prototype library built based on global reliability.

[0048] Specifically, the global reliability of audio and video features is linearly normalized to [0, 100]. Within a sliding window, the linearly normalized global reliability is sorted from high to low. A preset number of features are selected as the high-discriminative core features of the current window. Features that move out of the window are automatically removed as the window slides. The high-discriminative core features are then updated as reference prototypes to form a video dynamic prototype library. and audio dynamic prototyping library .

[0049] A dual-rule filtering process is performed based on global reliability to remove temporally redundant features, resulting in an effective set of video features. and audio feature set .

[0050] Specifically, for video modalities, if the global reliability of the video after linear normalization... Or continuous low reliability ( ) number of frames If the feature is retained, then for audio modalities, if the global reliability of the linearly normalized audio is... Or continuous low reliability ( ) number of frames If a feature is found to be valid, it is retained; the remaining features are deemed temporally redundant and are removed, ultimately resulting in a valid set of video features. and audio feature set .

[0051] S23 employs a hierarchical pruning strategy, sequentially removing video feature sets. and audio feature set The modalities and spatial redundancy features are used to obtain the core feature subset.

[0052] Specifically, the pruning ratio is as follows: ; in, This refers to the pruning ratio; m is the pruning strength coefficient; p represents the mean local reliability of the current feature.

[0053] According to the pruning ratio Pruning is performed to remove redundant features within the modality, and the effectiveness of the pruning is verified: ; ; in, Effective cosine similarity pruning for video features; A one-dimensional vector representing the flattened video features after pruning; A one-dimensional vector representing the video features before pruning; Effective cosine similarity pruning for audio features; The flattened one-dimensional vector of audio features after pruning; The audio features flattened before pruning are represented as a one-dimensional vector. Let L2 be the vector norm.

[0054] If the cosine similarity of the pruned features is greater than the preset validity threshold, the pruning validity verification is considered successful. Based on the verified audio and video features, the centered cosine similarity with the dynamic prototype library is calculated respectively. Audio and video features with a centered cosine similarity greater than the preset similarity threshold are removed, i.e., spatially redundant features are removed. The formula for calculating the centered cosine similarity is as follows: ; ; in, Centralized cosine similarity between video features and a dynamic prototype library; , Video and audio features were used to verify the effectiveness of the pruning process. , To verify the mean values ​​of the video and audio features after they have passed the verification; , The feature mean of the video and audio dynamic prototype library; The centralized cosine similarity between audio features and the dynamic prototype library.

[0055] The retained core feature subset is weighted and aggregated. The dynamic prototype library is updated based on the aggregated features. The redundant purification process described above is repeated to strengthen the core features, and the strengthened global core features of the video are output. Audio global core features .

[0056] S3 employs bidirectional cross-attention to perform cross-modal feature deep fusion of enhanced audio and video features.

[0057] Furthermore, S3 includes the following steps: S31, based on the global core features of audio and video, completes the feature dimension unification and distribution calibration through linear transformation, layer normalization, and random deactivation operations to obtain dimension-adapted single-modal features. The specific formula is as follows: ; ; in, Video features adapted to dimensions; Audio features adapted to dimensions; The mean of the global core features of the video; The mean of the global core features of the audio; This is element-wise multiplication; This is a linear transformation operation; For layer normalization operation; This is a random deactivation operation.

[0058] S32 employs a bidirectional cross-attention mechanism to perform bidirectional semantic interaction on dimension-adapted audio and video features. It then fuses the dimension-adapted audio and video features with the interaction results through residual connections, thereby overlaying cross-modal complementary information while retaining core single-modal discriminative features. The specific formula is as follows: ; ; in, Features of the fused video; For cross-modal attention; These are the audio features after fusion.

[0059] S33 performs dimensional concatenation on the interactively fused audio and video features to achieve shallow aggregation. The concatenated features are then input into a fully connected network containing a ReLU activation function to mine cross-modal nonlinear semantic associations. Simultaneously, feature dimensionality compression and deep aggregation are completed, outputting cross-modal fused features that fuse the core semantics of audio and video. The specific formula is as follows: ; ; in, This is a cross-modal splicing feature; For feature splicing operations; Features for cross-modal audio-visual fusion; This is the activation function.

[0060] S4, based on spatiotemporal density correlation modeling, performs long-term temporal processing on the deeply fused audio and video features to obtain pre-trained features.

[0061] Furthermore, S4 includes the following steps: S41, based on cross-modal fusion features, decomposes the video into short video segment features through overlapping extraction. It generates global temporal metadata, including the start and end times of the segment, the temporal step size, and the segment number. It combines topology and temporal localization embedding to mine spatiotemporal correlations and fuses them to obtain spatiotemporally enhanced segment features, thus completing the segmentation of long video features.

[0062] Specifically, for any two frames within segment i Cosine similarity is used to measure the semantic association strength between features in two frames: ; in, The semantic association strength between the features of the two frames; Each is a fragment The Middle Frame and the The feature vector of a frame.

[0063] An adjacency matrix is ​​constructed based on semantic association strength, and then a graph Laplacian is introduced to decompose the adjacency matrix: ; in, It is the identity matrix; Let be the adjacency matrix of segment i; Let be the degree matrix of segment i; Let be the normalized graph Laplace matrix of segment i.

[0064] right Eigenvalue decomposition yields graph Laplacian eigenvalues ​​and eigenvectors. The decomposed eigenvalues ​​are sorted in descending order, and the eigenvectors corresponding to the first predetermined number of eigenvalues ​​are obtained. These eigenvectors are then mapped to fragment features using a learnable linear layer. In the same dimensional space, topological embedding is obtained. ; through sinusoidal position encoding Representing the relative positions of frames within a segment using global periodic embedding. Characterizing the global temporal properties of this segment, and The fragments are then stitched together and compressed using a learnable linear projection layer to match the fragment features. The same dimensions yield temporal localization embedding. ; fragment features Adding the two embeddings element-wise yields spatiotemporally enhanced fragment features: ; in, These are the spatiotemporally enhanced fragment features.

[0065] S42, quantize the fragment feature variance and normalize it to obtain the event density value, and then compare it with the spatiotemporally enhanced fragment features. Fusion combines density-balanced sampling with contrastive learning to balance the differences in event density among different segments.

[0066] Specifically, the formulas for quantifying and normalizing the variance of segment features are as follows: ; ; in, Let V be the variance of the features of the i-th segment, used to characterize the event density of that segment; For feature dimensions; Let be the feature value of the i-th segment, the t-th frame, and the d-th dimension; Let be the normalized variance of the i-th segment, i.e., the event density value; The maximum variance of all segments within the current training batch; It represents the minimum variance of all segments within the current training batch.

[0067] Event density values ​​are learned through a linear layer. Mapped to spatiotemporally enhanced fragment features Density embedding vectors of the same dimension ,Will and Element-wise multiplication is performed to achieve a deep fusion of density properties and spatiotemporal features. Simultaneously, this learnable linear layer will... Numerical adaptive generation of differential correction coefficients: for low-density segments ( Less than the first threshold ), generating an amplification factor greater than 1 to improve its semantic discriminability; for high-density segments ( Greater than the second threshold A smoothing coefficient less than 1 is generated to suppress noise interference. This yields a unified fusion feature that represents the balanced intensity of each density segment. .

[0068] To balance the number of samples of different density segments during training, a segment-level density-balanced sampling pool is constructed, which includes high-density and medium-density segments. Low-density fragments are dynamically sampled in a 1:1:1 ratio. Based on unified fusion features... Positive sample pairs are constructed from adjacent time segments of the same density class, and negative sample pairs are constructed from non-adjacent time segments of different density classes, providing a balanced sample base for contrastive learning.

[0069] The constructed positive and negative sample pairs are input into the shared feature encoding layer to extract the corresponding positive and negative sample features, and the density-balanced contrastive loss is calculated based on the InfoNCE loss function. This loss is used to supervise and adjust the feature distance of segments. Based on this, the features optimized by density-balanced contrastive loss are sequentially normalized using LayerNorm and regularized using Dropout, ultimately outputting enhanced features with clear spatiotemporal correlation and balanced representation of each density segment. .

[0070] S43 will enhance features Mapping back to the temporal coordinates of long videos, weighted fusion of overlapping region features and reconstruction of long temporal sequences, and introduction of a long temporal self-attention layer to capture global semantic associations.

[0071] Specifically, since there are overlapping areas between adjacent segments, it is necessary to perform weighted fusion on the overlapping positions to eliminate boundary discontinuities. The weighted fusion formula is as follows: ; in, The fused features are for the t-th frame; These are weighting coefficients; Features of two adjacent segments at overlapping locations.

[0072] For non-overlapping regions, the features of the corresponding segments are directly used, and the features of each region in the long video are spliced ​​and fused together according to the original temporal sequence to obtain a complete long temporal feature sequence. This sequence is then subjected to layer normalization to obtain the long temporal reconstructed features. ;Will The global correlation matrix is ​​obtained by inputting a lightweight long-term self-attention layer. It captures global semantic associations across segments and quantifies the long-term value contribution of features.

[0073] S44 quantifies the long-term reconstructed feature value in multiple dimensions and generates an effective information mask. After filtering out low-value noise features, it fills in feature gaps and obtains long-term regular core features through attention pooling.

[0074] Specifically, based on the global association matrix Adjacency matrix and event density value Calculate the overall information value score for each frame: ; in, The score represents the overall information value of frame t. These are learnable weights within the model; Global correlation matrix The row mean; For local spatiotemporal saliency masks, derived from the adjacency matrix The row mean is obtained by normalization; To obtain an event density consistency score, the event density value is calculated. The standard deviation is obtained by normalizing and then inverting it.

[0075] Set dynamic threshold Generate a valid information mask, and combine the mask with long-term reconstructed features. Element-wise multiplication yields high-value effective features, while low-value noise features are filtered out. ; ; in, A valid information mask; These are high-value, effective features.

[0076] To maintain temporal continuity, a learnable linear interpolation layer is employed. By performing feature completion at the masked positions, complete long-term high-value effective features can be obtained. ,based on Calculate the mask attention weights: ; in, For mask attention weights; It is a normalized exponential function; These are learnable attention parameters; This is a transpose.

[0077] Based on mask attention weights Long-term high-value effective features The aggregated features are then condensed using a pooling window to extract long-term regularity core features. .

[0078] S45 uses multi-task loss constraints to jointly optimize the model parameters in S41-S44, completes high-precision pre-training of long-term features, and outputs pre-trained features that focus on the core patterns of long videos.

[0079] Specifically, multi-task losses include periodic consistency loss, cross-entropy loss, value scoring loss, and density balance comparison loss.

[0080] The periodic consistency loss enables the model to learn the periodic patterns in the video, and the calculation formula is as follows: ; in, This is due to periodic consistency loss; Batch size; This represents the long-term periodic component value of the b-th sample and the d-th feature dimension, from... Extract from; Embedded for global cycles; It is an absolute value.

[0081] For event sorting, leverage long-term pattern core features The system autonomously learns the sequence of events during key periods and calculates the cross-entropy loss by comparing it with a real two-dimensional sorting matrix generated from global temporal metadata. .

[0082] The value rating loss enables the model to autonomously evaluate the information value of each frame, and the calculation formula is as follows: ; in, Loss due to value rating; N is the total number of elements involved in the calculation within the batch; n is the sample index; To predict the overall information value score, based on generate; To determine the true comprehensive information value score, based on generate.

[0083] The formula for calculating the total loss is as follows: ; ; in, This represents the total loss of the long-sequence module. For loss weights; For density equalization contrast loss; This represents the total loss of the model.

[0084] Based on total loss Backpropagation is used to optimize the model parameters until convergence, and then the core features of long-term patterns are analyzed. Layer normalization is performed to ultimately output pre-trained features free of redundant noise and suitable for downstream tasks involving long videos. .

[0085] S5 performs hierarchical segmentation and dual candidate pool selection on the pre-trained feature tokens, and inputs them into a large language model to complete the deep parsing of long videos.

[0086] Furthermore, S5 includes the following steps: S51, based on pre-trained features , and each of its feature vectors Treat each token as a single token and associate each token with a time stamp. One-to-one binding, generating candidate key time periods through an adaptive sliding window. The system quantifies the temporal and semantic correlation between tokens and candidate key time periods to achieve accurate token segmentation. The number of pre-trained feature vectors, This represents the number of candidate key time periods.

[0087] Specifically, calculate the temporal correlation: ; in, For temporal correlation; Let be the pre-trained feature set for the i-th candidate key time period; Let be the start timestamp of the i-th candidate key time period; Let be the end timestamp of the i-th candidate key time period; The midpoint of the i-th candidate critical time period; This represents the time tolerance threshold.

[0088] Calculate semantic relevance: ; in, Semantic relevance; It is the mean of all Token feature vectors within the i-th candidate key time period.

[0089] Calculate the overall correlation between the token and the candidate key time period: ; in, To assess overall relevance; These are the weighting coefficients.

[0090] For each token, find the candidate key time period that has the highest overall relevance and assign the token to that time period.

[0091] S52, based on the quantification results, perform hierarchical processing on the tokens: For each candidate key time period, calculate the feature mean of its internal token subset, select the token with the highest cosine similarity to the feature mean as the core feature of that time period, and select the core features of each time period as the global layer tokens. The remainder are local layer tokens ( The multimodal reliability scores of the global layer token and the local layer token, as well as the semantic relevance of the local layer token, are calculated respectively.

[0092] Specifically, calculate the local reliability of token features: ; in, For the characteristic of local reliability of a single token; This is the normalized feature vector of the token; Let Variance be the variance.

[0093] Calculate the global and local relevance of tokens: ; ; in, Let be the global relevance of the t-th frame; Let be the local correlation of the t-th frame; This is the normalized feature vector corresponding to the current temporal position in the global layer; This is the normalized feature vector corresponding to the current temporal position of the local layer; This is the mean vector of the global layer tokens; This is the mean vector of the local layer tokens.

[0094] Calculate the combined reliability score of the global layer token and the local layer token: ; ; in, The overall reliability score for global layer tokens; The overall reliability score for local layer tokens; , These are weighting coefficients; They are respectively The normalized eigenvectors; for The normalized eigenvectors; Calculate the semantic association between local layer tokens and global layer tokens: ; in, This refers to the semantic association between local layer tokens and global layer tokens. Let be the normalized feature vector of the j-th local layer token; This is the normalized feature vector of the global layer token corresponding to the time period to which the local layer token belongs.

[0095] S53. Construct a dual-candidate pool to classify and manage tokens. Global layer tokens are directly added to the pool, while local layer tokens are calculated based on a comprehensive score. Tokens are sorted and filtered according to the comprehensive score, and the filtered tokens are added to the pool.

[0096] Specifically, the comprehensive reliability score based on local layer tokens. and semantic relevance Calculate the overall score of local layer tokens Those whose overall score is higher than the preset score threshold will be included in the pool. The formula for calculating the overall score is as follows: ; in, This is a reliability weighting factor; This is a weighting coefficient for semantic relevance.

[0097] S54, constrained by the context window capacity of the large language model, prioritizes the selection of global layer tokens, and uses the remaining capacity to select the best among the comprehensive scores. The high-level local layer tokens are concatenated and rearranged according to the original temporal order to obtain a suitable compressed token set. The compressed token set is then input into a large language model to complete the deep analysis of long videos and quantify the accuracy of the results.

[0098] Specifically, the accuracy of the results is quantified using the following formula: ; in, Score the accuracy of the results; These are the weighting coefficients; The degree of match between the answer and the video; The relevance of the answer to the task.

[0099] To verify the effectiveness of the multimodal feature processing proposed in this invention for long video understanding, a theoretical analysis is conducted from three dimensions: feature repair, redundancy removal, and long-term semantic fragmentation. The results demonstrate that the method can generate high-quality feature representations that meet the requirements of long video understanding tasks.

[0100] Proof: A weak feature preservation mechanism is used to avoid early deletion of valid information. Through historical mask accumulation and nonlinear displacement compensation, the feature representation of the occluded region is gradually repaired in multiple iterations. When the number of iterations is sufficient, the repaired features converge to the neighborhood of the true unoccluded features, thus satisfying the feature repair requirement.

[0101] High-value features are selected through a two-dimensional reliability assessment, low-reliability dimensions are eliminated through dynamic pruning, and core information is ensured to be preserved through cosine similarity verification. This process makes the loss of task-related information between the compressed features and the original features controllable, thus satisfying the redundancy removal requirement.

[0102] Through density-balanced pre-training and contrastive learning, the model learns density-invariant feature representations. High-value temporal steps are selected using value scoring and dynamic thresholding, and these key information points are forcibly focused on during attention pooling. Based on source coding theory, high-value information is prioritized for encoding within a limited context window, ensuring the coherence of long-term semantics.

[0103] In this embodiment, based on real long video datasets (EPIC-KITCHENS-100, ActivityNetCaptions), different parameters (viewpoint offset, number of mask iteration layers, number of long-sequence pooling windows, and compression ratio) are used to evaluate the usability of this invention for multimodal long video understanding. The results of the comparative experiments are as follows: Figures 2-5 As shown in the figure, the data availability comparison between this embodiment and the traditional long video understanding method under different viewpoint offsets is as follows. Figure 2As shown in the figure; a comparison of data availability between this embodiment and traditional long video processing methods under different occlusion mask iteration layers is shown in the figure. Figure 3 As shown in the figure; a comparison of the understanding accuracy of this embodiment and the traditional long video understanding method under different long-sequence pooling window numbers is shown in the figure. Figure 4 As shown in the figure; a comparison of the final video question-answering accuracy of the embodiments of the present invention and the traditional long video understanding method under different compression ratios is shown in the figure. Figure 5 As shown. By Figures 2-5 It can be seen that the data availability of this embodiment is better than that of the traditional method under different viewpoint offsets and different mask iteration layers. The understanding accuracy of this embodiment is better than that of the traditional method under different long-term pooling window numbers. The video question answering accuracy of this embodiment is better than that of the traditional method under different compression ratios.

[0104] It should be noted that the present invention is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments that have the same structure and perform the same effects as the technical concept within the scope of the present invention are included within the scope of the present invention. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of the present invention, are also included within the scope of the present invention.

Claims

1. A method for understanding long videos of multimodal audio and video data, characterized in that, include: S1, based on multi-view dynamic alignment and hierarchical masking iteration, features are extracted from the original audio and video bimodal data to obtain audio and video bimodal features; S2, based on a dynamic prototype library and hierarchical pruning, performs redundant purification of audio and video dual-modal features, removes multimodal redundant features, and strengthens global core discriminative features; S3 employs bidirectional cross-attention to perform cross-modal feature deep fusion of enhanced audio and video features; S4, based on spatiotemporal density correlation modeling, performs long-time processing on the deeply fused audio and video features to obtain pre-trained features; S5 performs hierarchical segmentation and dual candidate pooling of pre-trained features, and inputs them into a large language model to complete deep parsing of long videos.

2. The long video understanding method for multimodal audio and video data according to claim 1, characterized in that, S1 specifically includes: S11: Obtain the audio and video dual-modal data of the original long video and construct the video light field features and frequency domain audio field features from multiple perspectives; S12, based on video light field features and frequency domain audio field features, quantifies and normalizes the reliability of local audio and video regions, and through an adaptive dynamic offset mechanism, weighted aligns the multi-view features to the central viewpoint to obtain the aligned multi-view feature set. S13, construct an adaptive binary initial mask based on the aligned multi-view features, perform single-layer feature optimization by combining weight correction and modality aggregation, and perform hierarchical mask iterative optimization by accumulating and updating historical masks; S14: Render the optimized features from the target perspective to obtain the initial unobstructed audio and video features. Then, repair the feature details through modal concatenation residual convolution to output robust audio and video dual-modal features.

3. The long video understanding method for multimodal audio and video data according to claim 1, characterized in that, S2 specifically includes: S21. Based on the audio and video dual-modal features, the reliability of the features is quantified from both local and global dimensions, and the quantization results are normalized. S22, a dynamic prototype library is built based on global reliability, and feature filtering is performed to remove temporally redundant features, resulting in an audio and video feature set; S23 employs a hierarchical pruning strategy to sequentially remove modal and spatial redundant features from the audio and video feature set, obtaining a core feature subset. The core feature subset is then weighted and its features are aggregated. The dynamic prototype library is updated and iterative optimization is completed, outputting the enhanced global core features of the audio and video.

4. The long video understanding method for multimodal audio and video data according to claim 3, characterized in that, The layered pruning strategy specifically includes: The audio and video feature set is pruned according to the pruning ratio to remove redundant features within the modality; Calculate the cosine similarity of the flattened vectors of audio and video features before and after pruning. If the cosine similarity is greater than the preset validity threshold, the pruning is deemed valid. Calculate the centered cosine similarity between the pruned effective features and the dynamic prototype library, and remove features whose centered cosine similarity is greater than a preset similarity threshold to remove spatially redundant information.

5. The long video understanding method for multimodal audio and video data according to claim 1, characterized in that, S3 specifically includes: S31, based on the enhanced global core features of audio and video, completes the unification of feature dimensions and distribution calibration through linear transformation, layer normalization and random deactivation operation, and obtains dimension-adapted single-modal features. S32 employs a bidirectional cross-attention mechanism to perform bidirectional semantic interaction on dimension-adapted audio and video features, and fuses the dimension-adapted audio and video features with the interaction results through residual connections. S33 performs dimensional concatenation on the interactively fused audio and video features, inputs the concatenated features into a fully connected network containing the ReLU activation function, and outputs cross-modal fused features.

6. The long video understanding method for multimodal audio and video data according to claim 1, characterized in that, S4 specifically includes: S41, based on cross-modal fusion features, is decomposed into short video segment features through overlapping truncation and generates global temporal metadata. Combined with topological and temporal localization embedding to mine spatiotemporal correlation, spatiotemporally enhanced segment features are obtained through fusion. S42 quantifies the variance of segment features and normalizes it to obtain the event density value, which is then fused with the spatiotemporally enhanced segment features. The density equalization sampling and contrastive learning are combined to balance the differences in event density among the segments and output the enhanced features. S43, enhance the feature mapping back to the long video temporal coordinates, weighted fuse the overlapping region features and reconstruct the long temporal sequence to obtain the long temporal reconstructed features, introduce a long temporal self-attention layer to capture global semantic associations and obtain the global association matrix; S44: Based on the global correlation matrix, the long-term reconstructed feature value is quantified in multiple dimensions and an effective information mask is generated. After filtering out noisy features, feature gaps are filled, and long-term regular core features are obtained through attention pooling. S45 performs layer normalization on the core features of long-term patterns and outputs pre-trained features.

7. The long video understanding method for multimodal audio and video data according to claim 6, characterized in that, The method of combining topological and temporal localization embedding to mine spatiotemporal correlations and fusing them to obtain spatiotemporally enhanced fragment features specifically includes: Cosine similarity is used to measure the semantic association strength between features of any two frames within a short video clip, and an adjacency matrix is ​​constructed. By introducing the graph Laplace decomposition method, the adjacency matrix is ​​decomposed to obtain the graph Laplace matrix; Eigenvalue decomposition is performed on the graph Laplacian matrix, and the eigenvectors obtained from the eigenvalue decomposition are mapped to the same dimensional space as the fragment features to obtain the topological embedding; The sinusoidal position code used to characterize the relative position of each frame within a segment is concatenated with the global periodic embedding used to characterize the global temporal attributes of the segment. The concatenated vector is then compressed to the same dimension as the segment features to obtain the temporal localization embedding. Fragment features are added element-wise with topological embedding and temporal localization embedding to obtain spatiotemporally enhanced fragment features.

8. The long video understanding method for multimodal audio and video data according to claim 1, characterized in that, S5 specifically includes: S51 treats each feature vector of the pre-trained features as a token and binds it to a time-series label. Candidate key time periods are generated through an adaptive sliding window. The temporal and semantic correlation between the token and the candidate key time periods is quantified to classify the tokens. S52. Based on the quantization results, the tokens are processed in layers. The core features of each time period are selected as global layer tokens and the remaining features are selected as local layer tokens. The comprehensive reliability score of the local layer tokens and the semantic correlation between the global layer tokens and the local layer tokens are calculated. S53, construct a dual-candidate pool to classify and manage tokens. Global layer tokens are directly entered into the pool, and local layer tokens are calculated based on comprehensive reliability score and semantic relevance. Tokens are sorted and filtered according to their comprehensive scores and then entered into the pool. S54 uses the context window capacity of the large language model as a constraint to select tokens from the dual candidate pool, prioritizing global layer tokens. With the remaining capacity, it selects local layer tokens with high comprehensive scores. The selected tokens are concatenated and rearranged according to the original temporal order to obtain a compressed token set, which is then input into the large language model to complete the deep parsing of long videos.

9. A computer terminal, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 8.

Images