Multi-granularity fusion video clip retrieval method based on audio importance perception

Abstract

The present invention discloses a multi-granularity fusion video clip retrieval method based on audio importance perception, which belongs to the technical field of multimodal video clip retrieval. The method comprises: constructing a visual branch, a fusion branch and an audio branch for retrieval; after obtaining visual-text fusion features and audio-text fusion features, predicting the audio importance score, performing multi-granularity fusion on the two fusion features based on the importance score to obtain the total fusion feature, and then predicting the third video clip retrieval result; and during training, introducing retrieval loss, audio importance prediction loss, knowledge distillation loss between branches, and significance comparison loss between fusion features. Such training enables each branch after training to significantly improve retrieval accuracy, adapt to the uncertainty of audio modality, improve machine-learning cues robustness, enhance single-modal performance, and improve system practicality.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal video clip retrieval, and in particular relates to a multi-granularity fusion video clip retrieval method based on audio importance perception. Background Art

[0002] Video Moment Retrieval (VMR) involves accurately locating semantically relevant start and end time segments from a long video based on a natural language query. This task is widely used in fields such as video understanding, intelligent surveillance, human-computer interaction, and video search, and is a core issue in multimodal content understanding. Because it involves complex cross-modal semantic alignment and reasoning mechanisms, it has long been a key research area of interest in both academia and industry.

[0003] Currently, mainstream VMR methods focus on joint modeling of visual and textual modalities. These methods typically construct two-stream networks, cross-modal attention mechanisms, or contrastive learning frameworks to extract semantic features from video frames and language queries and establish a correspondence between the two, thereby achieving precise localization of semantic segments. While these methods have improved visual-semantic alignment to a certain extent, they generally overlook the equally important audio modality in videos.

[0004] In reality, audio carries a wealth of semantic clues in videos. For example, conversations, human voices, background sounds, and unstructured noise can all provide critical supplementary information for understanding query statements. In some scenes with visual ambiguity or similar actions, audio can even be the sole basis for distinguishing semantic behaviors. For example, a person laughing and a person talking may appear nearly identical visually, but their audio representations may be completely different. Consequently, in recent years, some research has begun to explore the integration of audio information into VMR models to improve their semantic understanding capabilities.

[0005] Representative works include PMI-LOC, UMT, and ADPN. PMI-LOC introduces trimodal features of RGB, optical flow, and audio, and designs a cross-modal attention mechanism for fusion; UMT constructs a unified multimodal Transformer network to integrate visual and audio information; and ADPN proposes a variant of the Transformer architecture to optimize the audio fusion path from the perspective of semantic consistency and complementarity. These methods have demonstrated the potential of audio modality in fragment retrieval tasks and achieved performance improvements over traditional bimodal methods.

[0006] However, the above methods generally have a key problem: they fail to model and perceive the importance of audio information and lack a selection mechanism based on semantic relevance. Specifically, these methods usually treat audio, visual, and text features equally, and perform joint reasoning through average fusion or simple attention mechanisms, failing to fully consider the differences in the contribution of audio under different video and query semantics. In practical applications, audio signals exhibit extremely high diversity and uncertainty: on the one hand, audio is an indispensable modality in semantics such as laughter, shouting, and clapping; but on the other hand, the audio signals in a large number of videos may only be background music, environmental noise, or even semantically silent clips. Such information not only does not help complete the task, but may even introduce interference and mislead the model's judgment.

[0007] A further challenge is that there is currently no publicly available large-scale dataset of audio importance labels, meaning that models lack supervisory signals regarding the semantic contribution of audio during training. This prevents existing methods from learning from the data when to focus on audio and when to suppress its interference, thus limiting the effective utilization of the audio modality. This problem has yet to be fundamentally addressed using existing technology.

[0008] In summary, while existing video clip retrieval methods have initially incorporated audio modalities, they generally lack the ability to perceive the semantic importance of audio. This inability to dynamically adjust the weight and role of audio features based on different query content and video scenes results in significant performance bottlenecks when handling complex and diverse cross-modal semantic reasoning tasks. Therefore, it is urgent to propose a retrieval method that can dynamically perceive the semantic importance of audio and adaptively adjust the fusion of audio and visual modalities based on its contribution, in order to further improve the accuracy and robustness of video clip retrieval. Summary of the Invention

[0009] In view of the above, the purpose of the present invention is to provide a multi-granularity fusion video clip retrieval method based on audio importance perception, aiming to achieve a more flexible and accurate semantic reasoning mechanism between audio, visual and text modalities, thereby improving the robustness and retrieval accuracy of the model in multimodal complex scenarios.

[0010] To achieve the above-mentioned object of the invention, an embodiment provides a multi-granularity fusion video clip retrieval method based on audio importance perception, comprising the following steps:

[0011] Constructing a learning framework, which includes an input unit and a segment retrieval prediction unit, the input unit is used to input video, audio segments and query text, the segment retrieval prediction unit includes a visual branch, a fusion branch and an audio branch, the visual branch extracts visual-text fusion features based on the video frame and the query text and predicts the first video segment retrieval result, the audio branch extracts audio-text fusion features based on the audio segment and the query text and predicts the second video segment retrieval result, the fusion branch predicts the audio importance score based on the visual-text fusion features and the audio-text fusion features, and then performs multi-granular fusion of the two fusion features based on the audio importance score to obtain the total fusion feature and predict the third video segment retrieval result;

[0012] A retrieval loss is constructed based on the retrieval results of each video clip, and pseudo-labels are constructed based on the corresponding retrieval losses of the visual branch and the audio branch. An audio importance prediction loss is constructed based on the pseudo-labels and the predicted audio importance scores. A knowledge distillation loss is constructed between the fusion branch and the visual branch and the audio branch respectively, and a saliency contrast loss is introduced between the three fused features.

[0013] After the learning framework is trained using all loss functions and the learning framework parameters are optimized, video segment retrieval is performed based on at least one branch in the input unit and the segment retrieval prediction unit.

[0014] Optionally, the visual branch includes a visual encoder, a text encoder, a visual-text fusion module, and a first segment location predictor, and predicts the first video segment retrieval result after extracting visual-text fusion features based on the video frame and the query text, including:

[0015] After using the visual encoder and text encoder to extract visual features and semantic features from the video frames and query text respectively, the visual-text fusion module is used to perform semantic interaction on the visual features and semantic features, and the context query attention mechanism is adopted to extract the most relevant context features activated by the query semantics as the visual-text fusion features. The first segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the visual-text fusion features as the first video segment retrieval result.

[0016] Optionally, the audio branch includes an audio encoder, a text encoder, an audio-text fusion module, and a second segment location predictor, and extracts audio-text fusion features based on the audio segment and the query text, and predicts the second video segment retrieval result, including:

[0017] After using the audio encoder and text encoder to extract audio features and semantic features from the audio clip and query text respectively, the audio-text fusion module is used to perform semantic interaction on the audio features and semantic features, and the context query attention mechanism is used to extract the most relevant context features activated by the query semantics as the audio-text fusion features. The second segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the audio-text fusion features as the second video segment retrieval result.

[0018] Optionally, the fusion branch includes an importance-aware multi-granularity fusion module and a third segment location predictor, wherein the importance-aware multi-granularity fusion module includes an audio importance predictor and a multi-granularity fusion submodule. After predicting the audio importance score based on the visual-text fusion feature and the audio-text fusion feature, the two fusion features are multi-granularly fused based on the audio importance score to obtain a total fusion feature, and then the third video segment retrieval result is predicted, including:

[0019] The audio importance predictor is used to predict the audio importance score based on the visual-text fusion features and the audio-text fusion features. The multi-granularity fusion submodule is used to perform weighted summation of the visual-text fusion features and the audio-text fusion features at the local level, event level, and global level based on the audio importance score to achieve multi-granularity fusion to obtain the total fusion feature. The third segment positioning predictor is used to predict the start and end positions of the target video segment based on the total fusion feature as the third video segment retrieval result.

[0020] Optionally, the audio importance predictor includes a global feature aggregation submodule and a multi-layer perceptron, and predicts the audio importance score based on the visual-text fusion feature and the audio-text fusion feature, including:

[0021] The global feature aggregation submodule uses attention pooling to perform global feature aggregation on the visual-text fusion features and the audio-text fusion features to obtain two global semantic features. The two global semantic features are then concatenated and input into a multi-layer perceptron for feature interaction between modalities and prediction of the audio importance score.

[0022] Optionally, based on the audio importance score, weighted summation of the visual-text fusion feature and the audio-text fusion feature is performed at the local level, event level, and global level to achieve multi-granularity fusion to obtain the overall fusion feature, including:

[0023] For local-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively subjected to a multi-core convolutional neural network to extract multiple local features of each, which are then spliced and input into their corresponding multi-layer perceptrons to obtain locally enhanced video features and audio features. The locally enhanced video features and audio features are then weighted and fused element by element using the audio importance score as the weight to obtain the local perceptual fusion feature.

[0024] For event-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively extracted through the slot attention sub-module to extract their own event features. After the respective event features and the original fusion features are extracted through their corresponding cross-attention sub-modules, the features output by the cross-attention sub-modules are weighted and fused element by element using the audio importance score as the weight to obtain the event-aware fusion feature.

[0025] For global-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively subjected to the attention summation operation to obtain their own global features. The respective global features and each element in the original fusion feature are then concatenated and passed through their respective corresponding multi-layer perceptrons to obtain the globally enhanced video features and audio features. The globally enhanced video features and audio features are then weighted and fused element by element using the audio importance score as the weight to obtain the global perceptual fusion feature.

[0026] For multi-granularity fusion, a group of Bi-GRU is introduced for local perception fusion features, event perception fusion features, and global perception fusion features. The fusion features at each level are combined in pairs to reconstruct the cross-perception relationship between them. The fusion results are then spliced and the total fusion features are obtained through a multi-layer perceptron.

[0027] Optionally, construct pseudo labels based on the retrieval losses corresponding to the visual branch and the audio branch, including:

[0028] ;

[0029] in, and denote the retrieval losses of the audio branch and the visual branch respectively, represents the temperature hyperparameter, represents the initial pseudo label, Represents the pseudo label output after processing, is the lower threshold, is the upper threshold.

[0030] Optionally, the audio importance prediction loss constructed based on the pseudo labels and the predicted audio importance scores adopts binary cross entropy loss.

[0031] Optionally, a knowledge distillation loss is constructed between the fusion branch and the visual branch and the audio branch, respectively, using the KL divergence between the third video clip retrieval result of the fusion branch and the first video clip retrieval result and the second video clip retrieval result of the visual branch and the audio branch, respectively;

[0032] The significant contrast loss within and outside the true value interval is introduced for the three fusion features, which is expressed as :

[0033] ;

[0034] in, It is the feature sequence after the fusion feature is compressed to 1 through the linear layer. Represents a randomly selected feature within the true value interval, Represents a randomly selected feature outside the true value range, loss Mandatory Features and features To open a distance.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Based on the obtained visual-text fusion features and audio-text fusion features, the present invention predicts the audio importance score, performs multi-granularity fusion on the two fusion features based on the importance score to obtain the total fusion feature, and then predicts the retrieval result of the third video clip. During training, retrieval loss, audio importance prediction loss, knowledge distillation loss between branches, and significance comparison loss between fusion features are introduced. In this way, each branch after training can significantly improve the retrieval accuracy, adapt to the uncertainty of audio modality, improve the robustness of machine clues, enhance the single modal performance, and improve the practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flowchart of a multi-granularity fusion video clip retrieval method based on audio importance perception provided by an embodiment;

[0039] Figure 2 It is a structural and training diagram of the learning framework provided in the embodiment;

[0040] Figure 3 Schematic diagram of the structure of the multi-granularity fusion submodule provided in the embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0042] The inventive concept of the present invention is to address the following common problems in existing video clip retrieval technologies: First, existing methods mostly focus on the joint modeling of visual and textual modalities. Although cross-modal semantic alignment has been achieved to a certain extent, the rich semantic information contained in the audio modality in the video is generally ignored. Especially in scenes where speech, laughter, and other visual sounds are difficult to distinguish, audio often plays a key role. The lack of full utilization of the audio modality will lead to misjudgment or fuzzy recognition in the model's semantic understanding and clip positioning. Secondly, although some methods have attempted to introduce audio modalities for trimodal fusion, there is a common problem of treating multimodal features equally, and failing to make dynamic judgments based on the semantic value of audio information in different scenarios. Due to the lack of audio importance supervision labels that can be used for training, the existing technology has difficulty in effectively identifying whether the audio has key semantics in model design, resulting in limited performance improvement and even the introduction of noise interference in some scenarios.

[0043] The embodiments of the present invention provide a video clip retrieval solution that can perceive the importance of audio semantics and dynamically regulate the fusion of audio and visual information. It aims to achieve a more flexible and accurate effective joint reasoning mechanism for semantics between audio, visual and text modalities, thereby improving the robustness and accuracy of the model in video clip retrieval in multimodal complex scenarios.

[0044] The embodiment provides a multi-granularity fusion video segment retrieval method based on audio importance perception, which aims to retrieve the start and end frame pairs of a specific segment that semantically matches the natural language query 𝑄 from an uncropped video 𝑉. , }, In addition, for each frame in the video, the corresponding audio clip can be extracted as a supplementary modality to provide contextual information to enhance the retrieval effect. Figure 1 As shown, the following steps are included:

[0045] S1, constructs a learning framework, which includes an input unit and a fragment retrieval prediction unit. The input unit is used to input video, audio fragments and query text. The fragment retrieval prediction unit includes a visual branch, a fusion branch and an audio branch, all of which are used to retrieve video fragments and obtain video fragment retrieval results.

[0046] In the embodiment, the learning framework is as follows Figure 2 As shown, the input unit is used to input video, audio clips, and query text. Among them, the video frame is captured from the video as the input of the visual modality, the query text is input using natural language as the input of the text modality, and the audio clip is used as the input of the audio modality.

[0047] In an embodiment, the visual branch included in the segment retrieval prediction unit extracts visual-text fusion features based on the video frame and the query text and predicts the first video segment retrieval result. Figure 2 As shown, the visual branch includes a visual encoder, a text encoder, a visual-text fusion module, and a first segment positioning predictor. The process of predicting the first video segment retrieval result is: after using the visual encoder and the text encoder to extract visual features and semantic features from the video frame and the query text respectively, the visual-text fusion module is used to perform semantic interaction on the visual features and semantic features, and the context query attention mechanism is used to extract the most relevant context features activated by the query semantics as the visual-text fusion features, and the first segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the visual-text fusion features as the first video segment retrieval result.

[0048] In the embodiment, the audio branch included in the segment retrieval prediction unit extracts audio-text fusion features based on the audio segment and the query text and predicts the second video segment retrieval result. Figure 2 As shown, the audio branch includes an audio encoder, a text encoder, an audio-text fusion module, and a second segment positioning predictor. The process of predicting the second video segment retrieval result is: after using the audio encoder and the text encoder to extract audio features and semantic features from the audio segment and the query text respectively, the audio-text fusion module is used to perform semantic interaction on the audio features and semantic features, and the context query attention mechanism is used to extract the most relevant context features activated by the query semantics as audio-text fusion features, and the second segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the audio-text fusion features as the second video segment retrieval result.

[0049] In the specific implementation, for the visual modality, the pre-trained visual CNN model is used to extract the original visual features of the video frame , and enhanced by a visual encoder consisting of a feedforward network (FFN), convolutional layer and Transformer layer to obtain visual features For audio modality, a pre-trained audio-aware CNN model is used to extract raw audio features of the audio clips. , and enhanced by an audio encoder that is consistent with the structure and visual encoder to obtain audio features For the text modality, the query text is initialized directly using the GloVe word vector. Since the query text may have different semantic alignments with the visual modality and the audio modality, two independent text encoders (with the same structure as the audio encoder) are further used to encode the query text respectively, thereby obtaining modality-specific enhanced semantic features, namely and To highlight the key semantic alignment between the visual modality and the given query text, the visual features are fused in the visual-text fusion module. and semantic features Apply the context-query attention mechanism to obtain visual-text fusion features At the same time, in the audio-text fusion module, the audio features and semantic features Apply contextual query attention mechanism to obtain audio-text fusion features .

[0050] In the embodiment, the fusion branch included in the segment retrieval prediction unit predicts the audio importance score based on the visual-text fusion feature and the audio-text fusion feature, and then performs multi-granular fusion of the two fusion features based on the importance score to obtain the total fusion feature and predict the third video segment retrieval result. Figure 2 As shown, the fusion branch includes an importance-aware multi-granularity fusion module and a third segment location predictor, wherein the importance-aware multi-granularity fusion module includes an audio importance predictor and a multi-granularity fusion sub-module. The importance-aware multi-granularity fusion module is a multi-granularity fusion structure guided by the audio importance predictor. This predictor is trained to identify and emphasize audio cues with semantic relevance, enabling the fusion module to selectively fuse effective audio signals with visual features. Guided by the predicted audio importance scores, the multi-granularity fusion process can effectively filter out irrelevant or noisy audio content, while aggregating meaningful cross-modal information at multiple time scales, thereby improving retrieval performance.

[0051] Specifically, the process of the fusion branch predicting the retrieval result of the third video clip includes: using the audio importance predictor to predict the audio importance score based on the visual-text fusion features and the audio-text fusion features, using the multi-granularity fusion sub-module to perform weighted summation of the visual-text fusion features and the audio-text fusion features at the local level (video frame-audio clip alignment), event level (semantic event aggregation), and global level (overall semantic abstraction) based on the audio importance score, fully capturing the complementary information between audio and vision, and reconstructing and integrating the features at all levels into a unified multimodal fusion representation through Bi-GRU to achieve multi-granularity fusion to obtain the total fusion feature, and using the third clip positioning predictor to predict the start and end positions of the target video clip based on the total fusion feature as the third video clip retrieval result.

[0052] The audio importance predictor adopts a lightweight design to dynamically estimate the importance of audio in each video-query pair. The specific audio importance predictor includes a global feature aggregation submodule and a multi-layer perceptron. It predicts the audio importance score based on the visual-text fusion feature and the audio-text fusion feature. It includes: using the global feature aggregation submodule to use the attention pooling operation to respectively perform the visual-text fusion feature and the audio-text fusion feature. and audio-text fusion features Perform global feature aggregation to obtain two global semantic features and Then, the two global semantic features and After splicing, the input is sent to the multi-layer perceptron (MLP) for feature interaction between modalities, enabling it to reason about the relative importance of audio based on visual context and predict the audio importance score through activation functions such as sigmoid. , the audio importance score 𝑝 is used to guide the subsequent multimodal fusion process and achieve dynamic fusion control by adjusting the contribution of the audio modality.

[0053] Given that audio modalities are inherently more noisy and variable than visual signals, simple fusion strategies may not be sufficient to fully exploit the complementarity between audio and vision. To this end, this paper proposes a multi-granularity fusion submodule that performs hierarchical fusion from three perspectives: local, event, and global. Dynamically predicted audio importance scores guide weighted fusion, achieving multi-granularity fusion to obtain an overall fused feature.

[0054] For local level fusion, such as Figure 3As shown in (a), in order to achieve frame-by-frame alignment and fine-grained fusion of visual frames and audio clips, a symmetrical multi-core one-dimensional convolutional network is constructed to more deeply perceive the local relationship between video frames and audio clips. Specifically, the visual-text fusion feature and audio-text fusion features Each of the multiple local features is extracted through a multi-core convolutional neural network and , and then splice the inputs to the corresponding multi-layer perceptron (MLP) to obtain the locally enhanced video features and audio characteristics , and then through the audio importance score p As weights for locally enhanced video features and audio characteristics Perform weighted element-by-element fusion to obtain local perception fusion features ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, represents the kernel size of the convolutional network, Represents the convolution kernel Corresponding to the feature dimension of the output, Conv1 represents one-dimensional convolution, n represents the total number of convolution kernels, LN Representation layer normalization.

[0059] For event-level fusion, in order to capture the event-oriented semantic matching relationship between vision and audio for activity understanding, such as Figure 3 As shown in (b), first, the visual-text fusion feature and audio-text fusion features The slot attention sub-module uses a set of learnable event slots to aggregate similar visual / audio clips into multiple events and extract their respective event features. and , their respective event characteristics and and the original fusion features and Extract features through their corresponding cross-attention sub-modules and Then, the audio importance score p Features output as weights across attention submodules and Perform weighted element-by-element fusion to obtain event-aware fusion features :

[0060] ;

[0061] in, SlotAttn Represents the slot attention mechanism, and across the attention submodules, the original features / feature As a query item, the event features extracted and As key and value pairs, and Weighted element-wise fusion is performed in the same way as local-level fusion.

[0062] For global level fusion, in order to match visual and audio context from a global perspective, such as Figure 3 As shown in (c), visual-text fusion features and audio-text fusion features After the attention pooling operation, the global features of each element are obtained. Then, the global features of each element are concatenated with the original fusion features and the global enhanced video features are obtained through the corresponding multi-layer perceptron (MLP). and audio characteristics , and then through the audio importance score p As weights for globally enhanced video features and audio characteristics Perform weighted element-by-element fusion to obtain global perception fusion features ,The weighted element-wise fusion here still adopts the same method as the local level fusion.

[0063] For multi-granularity fusion, since there are different perceptual correlations between fusion features from different granularity levels, the local perceptual fusion features , event perception fusion features , and global perception fusion features , introduce a set of Bi-GRU (bidirectional gated recurrent units), combine the fusion features of each level in pairs to reconstruct the cross-perception relationship between them, and then splice the fusion results and map them to the original feature space dimension d through a multi-layer perceptron to obtain the total fusion feature F .

[0064] In the embodiment, the first segment location predictor used by the visual branch, the second segment location predictor used by the visual branch, and the third segment location predictor used by the fusion branch all adopt the same structure, which is composed of a convolutional layer, a Transformer layer, and a linear layer. Each segment location predictor predicts the logits of the starting position and the ending position of the video segment based on the input features, thereby achieving accurate positioning of the target segment in the video. For the first segment location predictor, it is based on the feature vision-text fusion feature. Predict the logits of the start and end positions of a video clip and , and then select the maximum value subscript as the predicted final starting position or ending position through softmax and use it as the first video segment retrieval result; for the second segment positioning predictor, it is based on the characteristic audio-text fusion feature Predict the logits of the start and end positions of a video clip and , similarly, the maximum value subscript is selected through softmax as the predicted final starting position or ending position and used as the second video segment retrieval result; for the third segment positioning predictor, it is based on the total fusion feature F Predict the logits of the start and end positions of a video clip and , similarly, the subscript of the maximum value is selected through softmax as the predicted final starting position or ending position and used as the third video segment retrieval result.

[0065] S2, constructs a retrieval loss based on the retrieval results of each video clip, constructs pseudo-labels based on the retrieval losses corresponding to the visual branch and the audio branch, and constructs an audio importance prediction loss based on the pseudo-labels and the predicted audio importance scores. A knowledge distillation loss is constructed between the fusion branch and the visual branch and the audio branch respectively, and a significance contrast loss is introduced for the three fusion features within and outside the true value interval.

[0066] In the embodiment, a retrieval loss is constructed as a core loss for each video segment retrieval result of each branch to ensure that each branch has the ability to retrieve video segments. Specifically, each retrieval loss is the cross entropy between the predicted starting position of the video segment in the video segment retrieval result and the true value of the starting position, and the cross entropy between the predicted ending position of the video segment and the true value of the ending position. Taking the retrieval loss of the fusion branch as an example, the specific retrieval loss is Expressed as:

[0067] ;

[0068] in, CE represents the cross entropy loss function, and It is a 0-1 sequence provided during training, which is 1 only at the start / end position, indicating the true value of the position.

[0069] In the embodiment, since the audio importance predictor lacks true importance annotation, a loss-aware pseudo-importance label generator is designed to construct pseudo-labels as supervisory signals during the training process. Inspired by the observation that neural networks tend to learn from "simple" samples first during training, and such samples usually exhibit smaller training losses. Based on this, the retrieval loss of each video-query pair under the audio branch and the visual branch is compared. The modality with smaller loss is considered to provide more relevant information and should therefore be assigned a higher pseudo-importance score. The pseudo-labels constructed based on the retrieval losses corresponding to the visual branch and the audio branch include:

[0070] ;

[0071] in, and denote the retrieval losses of the audio branch and the visual branch respectively, represents the temperature hyperparameter, represents the pseudo importance score of the audio modality, Represents the pseudo importance score of the processed output and serves as a pseudo label, is the lower threshold, is the upper threshold. Below threshold When , the audio is regarded as a non-information mode and its contribution will be suppressed. On the contrary, if Above threshold , it indicates that audio plays a dominant role in retrieval.

[0072] Then based on the pseudo-label and the predicted audio importance score Constructing audio importance prediction loss , using binary cross entropy loss:

[0073] ;

[0074] in, represents the batch size, Indicates the The index of the sample. Audio importance score In the subsequent multi-granularity fusion stage, it serves as a key control parameter to guide the selective fusion of audio and visual features. In order to prevent unstable prediction results from misleading the fusion process in the early stages of training, the initial fusion weight is set to a neutral value of 0.5, and the audio importance score is gradually increased as training progresses. This curriculum-like strategy helps build robust multimodal interactions in the early stages while mitigating the impact of noise in early importance estimates.

[0075] In the embodiment, the fusion branch naturally captures richer and more comprehensive semantic representations by jointly modeling audio and visual cues. However, in practical applications, the audio signal may be missing, damaged or unavailable during the reasoning stage. In order to ensure that the unimodal branch, especially the visual branch, can still maintain strong retrieval capabilities under such conditions, a cross-modal knowledge distillation strategy is introduced to transfer the joint semantic knowledge learned in the fusion branch to the unimodal branch, thereby enhancing its ability to work independently during the reasoning stage, especially in scenarios such as audio loss. Specifically, we regard the fusion branch as a teacher network (teacher) and distill its knowledge into the student network (student), namely the visual branch and the audio branch, especially the visual branch, so that it can inherit the modal complementarity information contained in the fusion branch, and achieve good retrieval results even with only visual input. To this end, the knowledge distillation loss is constructed by minimizing the Kullback-Leibler (KL) divergence between the third video clip retrieval result of the fusion branch and the first video clip retrieval result and the second video clip retrieval result output by the visual branch and the audio branch. :

[0076] ;

[0077] in, and Represents the logits of the starting and ending positions in the video clip retrieval results predicted by the student network (visual branch or audio branch), and Represents the logits of the starting and ending positions in the video clip retrieval results predicted by the teacher network (fusion branch), is the temperature coefficient, is the softmax function. The distillation process of the integrated visual branch and audio branch, the final knowledge distillation loss For the visual branch With audio branch The sum of .

[0078] In the embodiment, three fusion features are simultaneously 、 as well as F The significant contrast loss is introduced inside and outside the true value interval respectively. This loss increases the model's attention to key information by widening the feature gap inside and outside the true value interval. It is specifically expressed as :

[0079]

[0080] in, is the fusion feature ( 、 or F ) is the feature sequence after the linear layer compresses the dimension to 1, Represents a randomly selected feature within the true value interval, Represents a randomly selected feature outside the true value range, loss Mandatory Features and features To open a distance.

[0081] Then the total loss function of the entire learning framework is for:

[0082]

[0083] in, 、 and is the balance coefficient of each loss item.

[0084] S3, after using all loss functions to train the learning framework and optimize the learning framework parameters, perform video segment retrieval based on at least one branch in the input unit and the segment retrieval prediction unit.

[0085] In this embodiment, a total loss function is used to train the learning framework and optimize its parameters. After optimization, video clip retrieval is performed based on at least one branch in the input unit and the segment retrieval prediction unit. This means that the system can select either a fusion branch or a single-modal branch for retrieval based on demand, providing good flexibility and generalization capabilities. The visual-audio fusion branch is generally preferred.

[0086] Through the above technical solution, the present invention achieves automatic perception and dynamic weighting of the contribution of audio modalities in videos, effectively solving the problem of insufficient audio information utilization and even the introduction of noise in existing methods, and improving the robustness and accuracy of video clip retrieval in complex multimodal scenarios. Specific technical effects demonstrated through experiments are:

[0087] 1. Significantly improve search accuracy

[0088] By introducing an audio importance prediction module, this paper effectively avoids the interference of low-quality audio on model performance. It also enhances the complementarity between different modalities through a multi-granular semantic fusion strategy. Experimental results on two public benchmark datasets, Charades-STA and ActivityNet Captions, show that the proposed method outperforms existing mainstream methods across multiple evaluation metrics. For the Charades-STA dataset, the R1@5 / R1@7 / mIOU indicators are improved by 0.87% / 9.76% / 5.33% respectively, and on the ActivityNet Captions data, the R1@5 / R1@7 / mIOU indicators are improved by 8.84% / 12.01% / 6.81% respectively. Compared with the method without introducing audio, the R1@5 / R1@7 / mIOU indicators on the Charades-STA dataset of the present invention are improved by 9.78% / 11.92% / 5.42% respectively, and the R1@5 / R1@7 / mIOU indicators on the ActivityNet Captions dataset are improved by 8.84% / 8.55% / 4.58% respectively. Among them, R1@5 represents the ratio of test samples whose IOU (intersection over union) between the first-ranked interval returned by the retrieval and the true value interval is greater than 0.5 to all test samples, while R1@7 represents the ratio of test samples whose IOU is greater than 0.7 to all test samples, and mIOU represents the average IOU of all test samples.

[0089] 2. Adapt to audio modal uncertainty and improve model robustness

[0090] The audio importance prediction mechanism proposed in this paper can dynamically evaluate the actual contribution of audio modalities in different scenarios, thereby automatically adjusting the level of audio participation during the inference phase. Experiments have shown that the present invention can maintain high retrieval accuracy even in scenarios with strong noise interference or missing audio information. The performance of the model using the audio importance perception module is much lower than that of the version without this module when the audio noise level increases, verifying the system's robustness and practical adaptability.

[0091] 3. Realize modular design with good compatibility and scalability

[0092] The audio importance predictor and multi-granularity fusion submodule proposed in this paper are highly pluggable and can be integrated as universal modules into other multimodal retrieval frameworks. When combined with existing methods such as EMB and EAMAT, their performance is significantly improved, demonstrating the versatility and portability of this invention.

[0093] 4. Enhance single-mode performance and improve system practicality

[0094] To address real-world situations where audio is missing or unavailable, this paper introduces a cross-modal knowledge distillation mechanism to transfer the joint knowledge from the fusion branch to the visual and audio branches, significantly improving the independent retrieval capabilities of the visual branch in the absence of audio input. Experiments demonstrate that when using only the visual modality during the inference phase, the proposed method exhibits minimal performance degradation, with R1@5 decreasing by only 1.46%, R1@7 by only 1.79%, and mIoU by 1.17%, demonstrating excellent deployment flexibility and practical value.

[0095] In summary, the present invention not only achieves a performance breakthrough in technical indicators, but also enhances the adaptability and scalability of the system through modular design and a universal fusion mechanism. At the same time, it demonstrates excellent robustness and applicability in actual scenarios, and has high application value and promotion prospects.

[0096] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-granularity fusion video clip retrieval method based on audio importance perception, characterized by: The following steps are involved: Constructing a learning framework, which includes an input unit and a segment retrieval prediction unit, the input unit is used to input video, audio segments and query text, the segment retrieval prediction unit includes a visual branch, a fusion branch and an audio branch, the visual branch extracts visual-text fusion features based on the video frame and the query text and predicts the first video segment retrieval result, the audio branch extracts audio-text fusion features based on the audio segment and the query text and predicts the second video segment retrieval result, the fusion branch predicts the audio importance score based on the visual-text fusion features and the audio-text fusion features, and then performs multi-granular fusion of the two fusion features based on the audio importance score to obtain the total fusion feature and predict the third video segment retrieval result; A retrieval loss is constructed based on the retrieval results of each video clip, and pseudo-labels are constructed based on the corresponding retrieval losses of the visual branch and the audio branch. An audio importance prediction loss is constructed based on the pseudo-labels and the predicted audio importance scores. A knowledge distillation loss is constructed between the fusion branch and the visual branch and the audio branch respectively. At the same time, a saliency comparison loss is introduced for the three fused features, both within the true value range and outside the true value range. After the learning framework is trained using all loss functions and the learning framework parameters are optimized, video segment retrieval is performed based on at least one branch in the input unit and the segment retrieval prediction unit.

2. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: The visual branch includes a visual encoder, a text encoder, a visual-text fusion module, and a first segment location predictor. It extracts visual-text fusion features based on the video frame and the query text and predicts the first video segment retrieval result, including: After using the visual encoder and text encoder to extract visual features and semantic features from the video frames and query text respectively, the visual-text fusion module is used to perform semantic interaction on the visual features and semantic features, and the context query attention mechanism is adopted to extract the most relevant context features activated by the query semantics as the visual-text fusion features. The first segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the visual-text fusion features as the first video segment retrieval result.

3. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: The audio branch includes an audio encoder, a text encoder, an audio-text fusion module, and a second segment location predictor. It extracts audio-text fusion features based on the audio segment and query text and predicts the second video segment retrieval result, including: After using the audio encoder and text encoder to extract audio features and semantic features from the audio clip and query text respectively, the audio-text fusion module is used to perform semantic interaction on the audio features and semantic features, and the context query attention mechanism is used to extract the most relevant context features activated by the query semantics as the audio-text fusion features. The second segment positioning predictor is used to predict the start and end positions of the video segment corresponding to the query semantics based on the audio-text fusion features as the second video segment retrieval result.

4. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: The fusion branch includes an importance-aware multi-granularity fusion module and a third segment location predictor. The importance-aware multi-granularity fusion module includes an audio importance predictor and a multi-granularity fusion submodule. After predicting the audio importance score based on the visual-text fusion feature and the audio-text fusion feature, the two fusion features are multi-granularly fused based on the audio importance score to obtain the total fusion feature and predict the third video segment retrieval result, including: The audio importance predictor is used to predict the audio importance score based on the visual-text fusion features and the audio-text fusion features. The multi-granularity fusion submodule is used to perform weighted summation of the visual-text fusion features and the audio-text fusion features at the local level, event level, and global level based on the audio importance score to achieve multi-granularity fusion to obtain the total fusion feature. The third segment positioning predictor is used to predict the start and end positions of the target video segment based on the total fusion feature as the third video segment retrieval result.

5. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 4 is characterized in that: The audio importance predictor includes a global feature aggregation submodule and a multi-layer perceptron, and predicts the audio importance score based on visual-text fusion features and audio-text fusion features, including: The global feature aggregation submodule uses attention pooling to perform global feature aggregation on the visual-text fusion features and the audio-text fusion features to obtain two global semantic features. The two global semantic features are then concatenated and input into a multi-layer perceptron for feature interaction between modalities and prediction of the audio importance score.

6. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 4 is characterized in that: Based on the audio importance score, the visual-text fusion features and audio-text fusion features are weightedly summed at the local, event, and global levels to achieve multi-granularity fusion and obtain the overall fusion features, including: For local-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively subjected to a multi-core convolutional neural network to extract multiple local features of each, which are then spliced and input into their corresponding multi-layer perceptrons to obtain locally enhanced video features and audio features. The locally enhanced video features and audio features are then weighted and fused element by element using the audio importance score as the weight to obtain the local perceptual fusion feature. For event-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively extracted through the slot attention sub-module to extract their own event features. After the respective event features and the original fusion features are extracted through their corresponding cross-attention sub-modules, the features output by the cross-attention sub-modules are weighted and fused element by element using the audio importance score as the weight to obtain the event-aware fusion feature. For global-level fusion, the visual-text fusion feature and the audio-text fusion feature are respectively subjected to the attention summation operation to obtain their own global features. Then, each element of their respective global features and the original fusion features are spliced together and then passed through their corresponding multi-layer perceptrons to obtain the globally enhanced video features and audio features. The globally enhanced video features and audio features are then weighted and fused element by element using the audio importance score as the weight to obtain the global perceptual fusion feature. For multi-granularity fusion, a group of Bi-GRU is introduced for local perception fusion features, event perception fusion features, and global perception fusion features. The fusion features at each level are combined in pairs to reconstruct the cross-perception relationship between them. The fusion results are then spliced and the total fusion features are obtained through a multi-layer perceptron.

7. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: For each branch, the retrieval loss is constructed based on each video clip retrieval result as: the cross entropy between the predicted starting position of the video clip in the video clip retrieval result and the true value of the starting position, and the cross entropy between the predicted ending position of the video clip and the true value of the ending position.

8. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: Construct pseudo labels based on the retrieval losses corresponding to the visual branch and the audio branch, including: ； in, and denote the retrieval losses of the audio branch and the visual branch respectively, represents the temperature hyperparameter, represents the initial pseudo label, Represents the pseudo label output after processing, is the lower threshold, is the upper threshold.

9. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: The audio importance prediction loss constructed based on pseudo labels and predicted audio importance scores adopts binary cross entropy loss.

10. The multi-granularity fusion video clip retrieval method based on audio importance perception according to claim 1 is characterized in that: The knowledge distillation loss constructed between the fusion branch and the visual branch and the audio branch respectively adopts the KL divergence between the third video segment retrieval result of the fusion branch and the first video segment retrieval result and the second video segment retrieval result of the visual branch and the audio branch respectively; The significant contrast loss within and outside the true value interval is introduced for the three fusion features, which is expressed as : ； in, It is the feature sequence after the fusion feature is compressed to 1 through the linear layer. Represents a randomly selected feature within the true value interval, Represents a randomly selected feature outside the true value range, loss Mandatory Features and features To open a distance.

Images