Video clip retrieval method, video clip retrieval device and storage medium

Abstract

This application relates to a video clip retrieval method, a video clip retrieval device, and a storage medium. The method comprises: obtaining a first video feature sequence and a corresponding text feature sequence, encoding the first video feature sequence based on a Mamba model to obtain a second video feature sequence; obtaining initial weights of the second video feature sequence and the text feature sequence based on a correlation matrix between the second video feature sequence and the text feature sequence; updating the weight of the second video feature sequence based on the weight of the text feature sequence, and updating the weight of the text feature sequence based on the weight of the second video feature sequence in the Mamba model; and predicting a video clip corresponding to the text feature sequence based on the updated second video feature sequence and text feature sequence. This method can solve the problems of low computational efficiency and accuracy in video clip retrieval.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a video clip retrieval method, a video clip retrieval device, and a storage medium. Background Art

[0002] With the rapid development of multimedia technology and Internet applications, video content is growing exponentially. Users often need to quickly and accurately locate specific clips in videos, such as finding key knowledge points in educational videos, identifying important events in news videos, or extracting exciting moments in sports videos.

[0003] The task of video clip retrieval mainly uses natural language queries to locate segments in the video that correspond to semantic information. Traditional methods are usually based on manual annotation or simple feature matching, which are inefficient and lack accuracy. In recent years, artificial intelligence technologies represented by deep learning have promoted the development of this field. In the early multimodal information fusion processing from text to image or video, convolutional neural networks and recurrent neural networks were often used to represent the information of the two modalities respectively. Related technologies have shown that the Transformer architecture based on the attention mechanism can more effectively integrate text information into video representation. However, the self-attention of the model based on the Transformer architecture has a high computational overhead on long sequences, and the prediction results rely on the attention mechanism, which may distract from irrelevant content, resulting in low computational efficiency and accuracy of video clip retrieval.

[0004] There is currently no effective solution to the problems of low computational efficiency and accuracy in video clip retrieval in related technologies. Summary of the Invention

[0005] Based on this, it is necessary to provide a video clip retrieval method, a video clip retrieval device and a storage medium that can solve the problems of low computational efficiency and accuracy of video clip retrieval in response to the above technical problems.

[0006] In a first aspect, a video clip retrieval method is provided in this embodiment, the method comprising:

[0007] Obtaining a first video feature sequence and a corresponding text feature sequence, and encoding the first video feature sequence based on a Mamba model to obtain a second video feature sequence;

[0008] Obtaining initial weights of the second video feature sequence and the text feature sequence according to a correlation matrix between the second video feature sequence and the text feature sequence;

[0009] In the Mamba model, the weight of the second video feature sequence is updated according to the weight of the text feature sequence, the weight of the text feature sequence is updated according to the weight of the second video feature sequence, and based on the updated second video feature sequence and the text feature sequence, a video clip corresponding to the text feature sequence is predicted.

[0010] In some embodiments, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, includes:

[0011] Obtaining the weight of the video segment in the current state according to the weight of each video segment in the second video feature sequence and the weight of each text vector in the text feature sequence in the historical state;

[0012] The weight of the text vector in the current state is obtained according to the weight of the text vector and the weight of the video clip in the historical state.

[0013] In some embodiments, in the Mamba model, before updating the weight of the second video feature sequence according to the correlation matrix and updating the weight of the text feature sequence, the method further includes:

[0014] The video segments are scaled based on the correlation between each video segment in the second video feature sequence and the text feature sequence.

[0015] In some embodiments, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, includes:

[0016] ;

[0017] ;

[0018] Wherein, f(·) represents the state update function of the Mamba model, g(·) represents the weighted information extracted according to the correlation matrix, the weighted information includes the vectors in the sequence and the weights of the vectors, α is the preset first hyperparameter, and β is the preset second hyperparameter; h represents the state of the sequence in the Mamba model, v is the vector in the second video feature sequence, t is the vector in the text feature sequence, n is the number of state updates of the second video feature sequence, and l is the number of state updates of the text feature sequence.

[0019] In some embodiments, obtaining a first video feature sequence includes:

[0020] Tiling and linear projection are performed on the video to be retrieved to obtain a video sequence;

[0021] A global temporal marker is added to the video sequence obtained by projection to obtain the first video feature sequence.

[0022] In some embodiments, predicting a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence includes:

[0023] According to the updated second video feature sequence, the text feature sequence and the global temporal marker, a video segment corresponding to the text feature sequence is predicted.

[0024] In some embodiments, obtaining the Mamba model includes:

[0025] According to a data set containing text and video, a text feature sequence and a video feature sequence for training are obtained;

[0026] Obtaining a first video segment predicted by the Mamba model based on the text feature sequence and the video feature sequence used for training, and a second video segment in the data set corresponding to the text feature sequence;

[0027] constructing a loss function based on the difference between the first video clip and the second video clip;

[0028] Parameters of the Mamba model are adjusted according to the loss function.

[0029] In some embodiments, constructing a loss function based on the difference between the first video segment and the second video segment includes:

[0030] A first loss function and a second loss function are constructed according to the difference between the first video segment and the second video segment; wherein,

[0031] The first loss function is an intersection-over-union loss function constructed based on time information of the first video clip and time information of the second video clip, or the first loss function is a binary cross entropy loss function constructed based on a pre-assigned significance score of the first video clip;

[0032] The second loss function includes at least one of the following: a KL divergence loss function, a contrastive learning loss function, and a significance order constraint loss function.

[0033] In a second aspect, a video clip retrieval device is provided in this embodiment, the device comprising:

[0034] A preprocessing module, which obtains a first video feature sequence and a corresponding text feature sequence, and inputs the first video feature sequence into a Mamba model to obtain a second video feature sequence;

[0035] a first cross-modal interaction module, which obtains initial weights of the second video feature sequence and the text feature sequence according to a correlation matrix between the second video feature sequence and the text feature sequence;

[0036] a second cross-modal interaction module, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence;

[0037] The prediction module is configured to predict a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence.

[0038] In a third aspect, a computer device is provided in this embodiment, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the video clip retrieval method described in the first aspect when executing the computer program.

[0039] The above-mentioned video clip retrieval method, video clip retrieval device and storage medium use Mamba as an encoder to capture the dependency relationship between the various features of the first video feature sequence over a long period of time and encode them to obtain a second video feature sequence after feature processing. By calculating the correlation matrix and guiding the status update of text and video in the Mamba model based on the weights obtained from the correlation, cross-modal feature interaction is efficiently realized, and the cross-modal alignment capability of video and text is improved, so that the model can accurately understand the semantics of the query text and find matching video clips, thereby improving efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A diagram illustrating an application environment of a video clip retrieval method according to an embodiment;

[0041] Figure 2 1 is a flow chart of a video clip retrieval method according to an embodiment;

[0042] Figure 3 is a flowchart of a video clip retrieval method according to another embodiment;

[0043] Figure 4 is a structural block diagram of a video clip retrieval device in one embodiment;

[0044] Figure 5FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] Several attention-based Transformer architectures have been proposed for video clip retrieval. For example, the Moment-DETR model pioneered a Transformer-based encoder-decoder architecture, treating clip retrieval as a straightforward set prediction problem. This model was able to predict clip coordinates and saliency scores end-to-end, achieving breakthroughs in video clip retrieval. However, this approach underutilized query information and ignored the relevance between the query and video content. Subsequently, the QD-DETR model, based on Moment-DETR, emphasized incorporating contextual information from the query into the video representation, enhancing the model's query sensitivity and improving the accuracy of clip retrieval. However, the QD-DETR model overlooked the complementary role of audio information in video semantics. Furthermore, the CG-DETR model adjusts the Transformer's cross-modal attention allocation based on relevance. By reallocating the total amount of attention using dummy tokens, relevance learning refines word-level interactions, and instance-adaptive saliency tokens extract key cues, the CG-DETR model improves the accuracy of temporal localization in videos without increasing inference cost. However, the self-attention mechanism in these Transformer-based models has a significant computational overhead for long sequences, limiting their efficiency and ability to model long videos. Furthermore, fine-grained video-text alignment relies on an attention mechanism, which can distract from irrelevant content.

[0047] Based on this, the embodiment of the present application provides a video clip retrieval method. The video clip retrieval method provided in this embodiment can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, tablets, IoT devices, etc. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0048] In one embodiment, Figure 2As shown, a video clip retrieval method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0049] Step S202: Obtain a first video feature sequence and a corresponding text feature sequence, and encode the first video feature sequence based on a Mamba model to obtain a second video feature sequence.

[0050] The first video feature sequence is associated with the text feature sequence. The original query text corresponding to the text feature sequence is a description of the original video corresponding to the first video feature sequence. Optionally, a video containing L segments and a query text containing N words are obtained, and the query text describes the target event in the video. The features of the video and the query text are extracted to obtain the first video feature sequence {v1, v2, ..., v L} and text feature sequence {q1,q2,…,q N}.

[0051] Mamba is a selective state-space model. Compared to the SSM (State Space Model) in sequence modeling, it can adaptively select contextual information through a dynamic input-dependent parameter adjustment mechanism, filtering out redundant information while enhancing key features. Mamba is used as an encoder for sequence modeling. Optionally, after inputting a first video feature sequence into the Mamba model, Mamba captures the long-term dependencies between the features in the first video feature sequence and outputs a second video feature sequence that is compatible with downstream tasks.

[0052] Step S204 : obtaining initial weights of the second video feature sequence and the text feature sequence according to the correlation matrix between the second video feature sequence and the text feature sequence.

[0053] The correlation matrix between each video feature vector in the second video feature sequence and the text feature sequence can be calculated. Alternatively, a video segment consisting of multiple video feature vectors in the second video feature sequence can be obtained, and the correlation matrix between each video segment and the entire text feature sequence can be calculated.

[0054] When it is determined based on the correlation matrix that the correlation is high, a higher initial weight can be assigned to the video feature vector or the video segment composed of multiple video features; otherwise, a lower initial weight can be assigned. Similarly, based on the correlation matrix, when it is determined that the correlation between the text feature vector in the text feature sequence and the second video feature sequence is high, a higher initial weight can be assigned to the text feature vector; otherwise, a lower initial weight can be assigned. On this basis, the relationship between the correlation and the weight can be set as a linear correlation or a nonlinear correlation. The specific corresponding relationship can be set according to needs and is not limited here. Optionally, the correlation corresponding to each text vector in the text feature sequence can be extracted from the correlation matrix as the weight of the text vector, and the correlation corresponding to each video segment in the second video feature sequence can be extracted as the weight of the video segment.

[0055] Step S206: In the Mamba model, the weight of the second video feature sequence is updated according to the weight of the text feature sequence, the weight of the text feature sequence is updated according to the weight of the second video feature sequence, and the video clip corresponding to the text feature sequence is predicted based on the updated second video feature sequence and text feature sequence.

[0056] The weight of the text feature sequence corresponds to the relevance of the text feature sequence, and the weight of the second video feature sequence is also related to the relevance of the second video feature sequence. In the Mamba model, in conjunction with Mamba's state update function, the second video feature sequence and the text feature sequence are updated based on their respective weights, resulting in a second video feature sequence and its updated weight, and a text feature sequence and its updated weight. Optionally, while updating the weight of the second video feature sequence based on the state update function, the weight of the text feature sequence is used as an influencing factor, and the output of the state update function and the influencing factor are combined to obtain the second video feature sequence with the updated weight. Similarly, while updating the weight of the text feature sequence based on the state update function, the weight of the second video feature sequence is used as an influencing factor, and the output of the state update function and the influencing factor are combined to obtain the text feature sequence with the updated weight. By allowing the states of the video and text modalities to interact during the Mamba propagation process, the matching degree between the second video feature sequence and the text feature sequence can be strengthened, which is conducive to improving the accuracy of predicting video clips.

[0057] Optionally, the Mamba model includes an MLP (Multilayer Perceptron) temporal prediction layer. This layer learns the mapping relationship between the updated second video feature sequence and the text feature sequence, and obtains the temporal information of the video segment based on the text feature sequence. This temporal information can include the start time, end time, center coordinates, segment width, and other information. Optionally, the updated second video feature sequence and the text feature sequence can be fused to obtain a fused representation sequence. The MLP temporal prediction layer receives this fused representation sequence and outputs the temporal information of the video segment. Alternatively, other networks, such as fully connected layers and Bayesian prediction layers, can be used to implement predictions.

[0058] In the aforementioned video clip retrieval method, the Mamba state-space model replaces the traditional Transformer model for sequence modeling, avoiding attention calculations. Mamba also converts self-attention calculations into state-space propagation, significantly improving model inference speed and reducing computational complexity. Mamba's state propagation mechanism effectively captures the dependencies between text and video sequences over long timescales. By explicitly calculating the correlation matrix between video and text, Mamba-based correlation-guided cross-modal interaction between video and text is implemented. This strengthens the match between video content and query text, improves the accuracy of cross-modal alignment, and enhances the precision of video clip retrieval.

[0059] In one embodiment, in the Mamba model, before updating the weight of the second video feature sequence according to the correlation matrix and updating the weight of the text feature sequence, the method further includes: scaling the video segments based on the correlation between each video segment in the second video feature sequence and the text feature sequence.

[0060] The second video feature sequence includes multiple video clips, and the correlation of each video clip with respect to the entire text feature sequence is obtained according to the correlation matrix.

[0061] Optionally, for the second video feature sequence V={v1,v2,…,v L} and text feature sequence Q={q1,q2,…,q N}, calculate the correlation matrix , including: linearly projecting V and Q into a unified semantic space , , where W q is the visual projection matrix of the text feature sequence, W q is the visual projection matrix of the video feature sequence, and then the cosine similarity is calculated to form a correlation matrix:

[0062] ;

[0063] Among them, C l,n It reflects the correlation between the video clip l and the text word n.

[0064] For each video segment v l Calculate its relevance to the entire query Q:

[0065] ;

[0066] By s l As a weighted pair v l Zoom , achieving the effect of adjusting the strength of video features according to correlation.

[0067] In this embodiment, the relevance matrix is ​​used to filter out the most relevant segments, reducing the interference of irrelevant content. This allows for global segment saliency screening, which reduces the weight of low-relevance video segments, minimizing their impact on the final output during Mamba transmission, thereby improving computational efficiency and accuracy.

[0068] In one embodiment, the weight of the second video feature sequence is updated according to the weight of the text feature sequence, and the weight of the text feature sequence is updated according to the weight of the second video feature sequence, including: obtaining the weight of the video segment in the current state according to the weight of each video segment in the second video feature sequence and the weight of each text vector in the text feature sequence under the historical state; obtaining the weight of the text vector in the current state according to the weight of the text vector and the weight of the video segment under the historical state.

[0069] During Mamba transmission, the state update of a video clip depends not only on its own historical state but also on the influence of its textual information. Optionally, text weighting information is extracted from the correlation matrix. The text weighting information includes text vectors and their weights in the text feature sequence. Based on this text weighting information, the video state is updated based on the historical state of the second video feature sequence, resulting in an updated, weighted second video feature sequence.

[0070] Similarly, during Mamba transmission, the state update of a video clip depends not only on its own historical state but also on the influence of the textual information. Optionally, video weighting information is extracted from the correlation matrix, including the video vectors and their weights in the second video feature sequence. Based on this video weighting information, the video state update is guided based on the historical state of the first textual feature sequence, resulting in an updated weighted second video feature sequence.

[0071] Optionally, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, includes:

[0072] ;

[0073] ;

[0074] Wherein, f(·) represents the state update function of the Mamba model, g(·) represents the weighted information extracted according to the correlation matrix, the weighted information includes the vectors in the sequence and the weights of the vectors, α is the preset first hyperparameter, and β is the preset second hyperparameter; h represents the state of the sequence in the Mamba model, v is the vector in the second video feature sequence, t is the vector in the text feature sequence, n is the number of state updates of the second video feature sequence, and l is the number of state updates of the text feature sequence.

[0075] The weighted information includes: text weighted information or video weighted information. The text weighted information includes the text vectors and the weights of the text vectors in the text feature sequence. The video weighted information includes the video segments and the weights of the video segments in the second video feature sequence. α is used to control the intensity of the cross-modal influence of the text weighted information on the second video feature sequence. The text weighted information is calculated as follows: , the result of the weighted summation As an additional cross-modal signal, it guides the update of the state of the second video feature sequence. β is used to measure the strength of the cross-modal influence of the video weighting information on the text feature sequence; the video weighting information is calculated as follows , the result of the weighted summation As an additional cross-modal signal, it guides the update of the second video feature sequence state.

[0076] In this embodiment, two interactive modes are used to allow the states of video and text modalities to interact during the Mamba communication process, achieving cross-modal feature fusion. This improves the cross-modal alignment of video and text, enabling accurate understanding of the semantics of text feature sequences corresponding to query text and finding matching video clips. This improves the accuracy of video retrieval.

[0077] In one embodiment, obtaining the first video feature sequence includes: tiling and linearly projecting the video to be retrieved to obtain a video sequence; and adding a global temporal marker to the projected video sequence to obtain the first video feature sequence.

[0078] Among them, the standard Mamba is designed for one-dimensional sequences. In order to process the visual task, the video to be retrieved is tiled, including converting the two-dimensional image of each video frame in the video to be retrieved into a two-dimensional block and projecting the two-dimensional block into a dimensional vector. Optionally, the two-dimensional image frame Convert to flat 2D patches , where H×W represents the size of the image frame, C is the number of channels in the image, P is the size of the image patch, and J is the number of patches in each image. p Project onto a vector of size D and add position embedding , the formula is as follows:

[0079] ;

[0080] in, represents the jth block of the image frame, is a learnable projection matrix, Represents an entire sequence of blocks.

[0081] Then, a global time sequence tag is added to the video sequence by position embedding. The global time sequence tag is used to aggregate the timing information and context summary of the entire video and can be used as an additional token. Send to the lth layer of Mamba and output the second video feature sequence While Mamba encodes the first video feature sequence, global temporal tagging can learn global features of the entire video clip.

[0082] Furthermore, in one embodiment, a video segment corresponding to the text feature sequence is obtained based on the updated second video feature sequence and the text feature sequence, including: predicting a video segment corresponding to the text feature sequence based on the updated second video feature sequence, the text feature sequence and the global timing marker.

[0083] In the Mamba model, during the state update of the second video feature sequence and text feature sequence, global temporal tags can also be combined with text features to enhance video query understanding and participate in the final segment prediction as a global representation, which can improve prediction accuracy.

[0084] Optionally, the second video feature sequence, the text feature sequence, and the global temporal marker are fused. This generates a representation that incorporates global features. This representation is then directly mapped into time segment predictions using an MLP. Compared to traditional Transformer methods that require token lookups to calculate the starting position of time boundaries, this approach eliminates the need for Hungarian matching or additional filtering steps, resulting in improved computational efficiency.

[0085] In one embodiment, obtaining a text feature sequence includes obtaining a query text. A pre-trained CLIP (Contrastive Language–Image Pre-training) model is used to embed the query text to obtain a semantic vector representation of the text. Positional encoding is added to the text vector sequence to preserve the structural order of sentences, and modality identifiers are added to distinguish the text modality. This enables the model to understand the source of the input sequence.

[0086] In one embodiment, obtaining a Mamba model includes: obtaining a text feature sequence and a video feature sequence for training based on a data set containing text and video; obtaining a first video segment predicted by the Mamba model based on the text feature sequence and the video feature sequence for training, and a second video segment corresponding to the text feature sequence in the data set; constructing a loss function based on the difference between the first video segment and the second video segment; and adjusting parameters of the Mamba model based on the loss function.

[0087] The text feature sequence and video feature sequence for training can be obtained from the dataset using the same method used to construct the text feature sequence and the second video feature sequence. The first video segment is a video segment predicted during the Mamba model training process, and the second video segment is a real video segment corresponding to the text feature sequence. One or more loss functions can be constructed based on the actual application. After constructing the loss function, the Mamba model parameter update can be stopped if the model converges or if the number of iterative updates reaches a specified number.

[0088] A first loss function can be constructed separately, and Mamba training can be performed based on the first loss function. Alternatively, a first loss function and a second loss function can be constructed, and Mamba training can be performed using a multi-task joint training strategy. Furthermore, a loss function is constructed based on the difference between the first video clip and the second video clip, including: constructing a first loss function and a second loss function based on the difference between the first video clip and the second video clip; wherein the first loss function is an intersection-over-union loss function constructed based on the time information of the first video clip and the time information of the second video clip, or the first loss function is a binary cross entropy loss function constructed based on the pre-assigned significance score of the first video clip; the second loss function includes at least one of the following: a KL divergence loss function, a contrastive learning loss function, and a significance order constraint loss function.

[0089] The first loss function is a primary task loss. When the first loss function is an intersection-over-union (IoU) loss function constructed based on the degree of matching between the temporal information of the first video segment and the temporal information of the second video segment, a video event bias corresponding to the text feature sequence can be obtained based on the first video segment. The IoU loss function encourages the predicted segment to match the real segment. The first loss function is as follows:

[0090] ;

[0091] Where A represents the area of ​​the predicted box corresponding to the first video segment output according to the text feature sequence prediction, and B represents the area of ​​the real box corresponding to the second video segment.

[0092] When the first loss function is a binary cross-entropy loss function constructed based on the pre-assigned saliency score of the first video segment and the difference between the first video segment and the second video segment, the saliency scores of the multiple video segments corresponding to the second video feature sequence can be predicted, and then the first video segment corresponding to the text feature sequence can be determined based on the saliency scores. Optionally, saliency scores are assigned to different video segments, and the true labels are determined based on the saliency scores. The probability distribution of the matching between the multiple video segments corresponding to the second video feature sequence and the text feature sequence is predicted, and the cross entropy between the probability distribution and the true label is calculated using BCE (Binary Cross-Entropy Loss). The first loss function is as follows:

[0093] ;

[0094] Among them, y i represents the true label, y i The value is 0 or 1, p i is the probability value predicted by the model; p iIt can be output by Sigmoid function, with a range of [0, 1], p i It can also be output through other activation functions, for example, through the output of the Probit function or the Clamped Linear function; N is the number of video clips corresponding to the second video feature sequence.

[0095] The second loss function is an auxiliary task loss that can improve cross-modal alignment capabilities and optimize correlation calculations.

[0096] When the second loss function includes the contrastive learning loss function, the segment-sentence contrastive learning loss can be calculated. The segment-sentence contrastive learning loss is used to align the second video feature sequence and the text feature sequence in the common embedding space, thereby improving the stability of the correlation matrix calculation. The formula is as follows:

[0097] ;

[0098] Where sim(V,Q) represents the similarity between the second video feature sequence V and the text feature sequence Q in the embedding space. P represents the matched video and text features, and Q' represents the negative example text, i.e., the incorrectly matched video and text features.

[0099] When the second loss function includes the KL (Kullback-Leibler) divergence loss, the loss can be used to measure the difference between the variational distribution and the prior distribution, so that the variational distribution is close to the prior distribution. The variational distribution is the conditional distribution of the video and text inferred based on the second video feature sequence and the text feature sequence. Optionally, the second video feature sequence and the text feature sequence are input into the motion prior encoder, and the normal distribution composed of the mean and covariance matrix output by the motion prior encoding is obtained. The motion prior encoder is a neural network module that is responsible for extracting dynamic features (such as motion trajectories and temporal changes) from the second video feature sequence, and combining the semantic information of the text through an attention mechanism or feature splicing, and outputting the parameters of the variational distribution, including the mean and covariance matrix. The prior distribution is a standard normal distribution composed of 0 and 1. Optionally, the correlation matrix distillation loss is calculated, and the specific formula is as follows:

[0100] ;

[0101] Among them, μ i represents the mean of the variational distribution, σ i represents the standard deviation of the variational distribution, and z represents the dimension of the predefined latent space.

[0102] When the second loss function includes the saliency order constraint loss function, a soft label vector y of length T is constructed smooth, connecting the video and text alignment relationship through the soft label vector. The weight within the GT segment (Ground Truth Segment) is set to 1, and the weight of the event region outside the GT segment gradually decreases starting from the GT segment boundary. The GT segment is the second video segment manually annotated. The formula is as follows:

[0103] ;

[0104] Among them, p aliency It is the prediction of salient tokens on the video timeline. The saliency order constraint loss function is used to ensure that the distribution of salient tokens on the timeline matches the GT segment.

[0105] After constructing the first and second loss functions, all loss functions are jointly optimized and gradient updated. Taking the intersection-over-union loss as the first loss function and the KL divergence loss function, contrastive learning loss function, and significance order constraint loss function as an example, based on the joint optimization and gradient update of the loss functions, we get:

[0106] ;

[0107] Here, λ1, λ2, λ3, and λ4 control the weights of different loss functions. Use an existing optimizer and implement a gradient clipping learning rate strategy to ensure training stability. Alternatively, you can choose optimizers such as Adam, AdamW, and SGD. The same principles apply to constructing other primary and secondary loss functions and are not detailed here.

[0108] In one embodiment, after the Mamba model is trained based on the loss function, evaluation metrics may be calculated to evaluate Mamba. The evaluation metrics may include one or more of the following: recall, average intersection-over-union, and overall average precision.

[0109] Optionally, the recall rate is used to evaluate the effectiveness of video and text-based predictions. A search is considered effective when the prediction results (t_start, t_end) of the first k first video segments returned based on the query text (text feature sequence) completely cover the actual time period (the second video segment). The recall rate is calculated as follows:

[0110] ;

[0111] Among them, n topk is the total number of successfully retrieved fragments in the first k results of all queries, N relevantis the total number of video clips associated with the second video feature sequence in the test set, and k is a positive integer. For example, k can be set to a positive integer such as 1 or 5. Recall@1 corresponds to a test set where a GT clip appears in the first predicted result, and Recall@5 corresponds to a test set where a GT clip appears in the top five predicted results, and a retrieval is considered successful.

[0112] Optionally, the degree of overlap between the predicted first video segment and the true segment (the second video segment) is measured by mIoU (Mean Intersection over Union).

[0113] ;

[0114] Where N is the total number of all detected segments, S i is the i-th video clip, G i is the i-th real video clip, |·| represents the length of the video clip, ∩ represents the intersection of video clips, and ∪ represents the union of video clips. Generally speaking, the value of mIoU is between 0 and 1, where 1 represents a complete match and 0 represents a complete mismatch.

[0115] Optionally, mAP@m (mean Average Precision) is used to calculate the AP (average precision) of the search results corresponding to the query text (text feature sequence). The higher the ranking of the video clip in the search results, the higher the AP value. The AP values ​​of the search results corresponding to multiple query texts (text feature word order) are averaged to obtain the overall average precision:

[0116] ;

[0117] Among them, Q is the number of query texts, APi is the AP@m of a query statement.

[0118] In one embodiment, a method for video clip retrieval is implemented based on a state-space model-based visual architecture called Vision Mamba. Figure 3 A schematic diagram of another video retrieval method is provided. Figure 3 As shown, video clip retrieval includes: feature extraction, relevance-guided cross-modal interaction and prediction.

[0119] Feature extraction includes video feature extraction and text feature extraction. When extracting video features, the original video (L, C, H, W) is tiled and linearly projected (L, J, C × H × W) to obtain the first video feature sequence. Using the pre-trained video backbone network Vision Mamba, the feature sequence is extracted from the original video feature sequence. Each video is represented as a set of embedding vectors {v1, v2, ..., v L}, each vector dimension is D, and the fusion representation sequence (L×J,D) is obtained, which is the second video feature sequence. When extracting text features, for the query text containing N words {q1,q2,…,q N}, using the pre-trained text model CLIP to extract text features, resulting in a text feature sequence (N, D). Position encoding and modality identity encoding are performed on the extracted second video feature sequence and text feature sequence: explicit position encoding is introduced for each of the video and text features to capture sequence position information, and modality identity encoding is introduced to clearly distinguish between video and text features. Furthermore, a global temporal marker (GTT) is introduced as a special position for the entire global feature.

[0120] In the forward propagation stage of the model, correlation-guided cross-modal interaction is performed, including: calculating the video and text word correlation matrix C∈R^(L×N) based on the second video feature sequence and the text feature sequence, and using the correlation matrix to screen out the most relevant segments and weaken the interference of irrelevant content: for each video segment v in the second video feature sequence l Calculate its relevance s to the entire query Q l By s l As a weighted pair v l Scaling is performed. By introducing a correlation matrix, the correlation between video clips and text words is explicitly calculated to guide Mamba's state updates. The states of the video and text modalities interact during Mamba propagation, forming a video-text fusion feature sequence. By combining the SSM coupled update strategy, the state propagation of the video modality is influenced by the text modality, and vice versa, achieving cross-modal feature fusion. Global convolution kernels are used to pre-compile cross-modal interactions to maintain the computational efficiency of the Mamba structure and avoid attention calculations.

[0121] Through the MLP time prediction layer, it receives the feature sequence processed by the correlation-guided cross-modal interaction module and outputs the predicted values ​​of the start and end time of the video clip.

[0122] When training models for implementing video retrieval methods, publicly available datasets can be used as training data. Alternatively, datasets such as Charades-STA and ActivityNet Captions can be used. When the retrieval goal is to locate the corresponding time segment in a video based on the query text, Charades-STA is suitable. When using ActivityNet Captions, the characteristics of this dataset can be exploited to retain the entire long text information and utilize Mamba to efficiently process long sequences. During training, a multi-text shared video feature and sliding window segment training strategy are employed to enhance long video localization capabilities and model robustness. Multi-task joint training can also be employed.

[0123] In this embodiment, this paper uses Vim (Vision Mamba) as an encoder to efficiently encode temporal features of video frame sequences and redesigns the cross-modal interaction strategy based on the SSM architecture. Vim combines bidirectional sequence modeling of visual tasks on the basis of the Mamba architecture, taking into account both forward and backward information to achieve global context modeling, thereby simultaneously improving efficiency and long sequence modeling performance.

[0124] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0125] Based on the same inventive concept, embodiments of the present application also provide a video clip retrieval device for implementing the aforementioned video clip retrieval method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more video clip retrieval device embodiments provided below can be found in the aforementioned limitations of the video clip retrieval method and will not be further elaborated here.

[0126] In one embodiment, Figure 4 As shown, a video clip retrieval device is provided, comprising:

[0127] A preprocessing module obtains a first video feature sequence and a corresponding text feature sequence, and inputs the first video feature sequence into a Mamba model to obtain a second video feature sequence;

[0128] The first cross-modal interaction module obtains initial weights of the second video feature sequence and the text feature sequence according to a correlation matrix between the second video feature sequence and the text feature sequence;

[0129] The second cross-modal interaction module, in the Mamba model, updates the weight of the second video feature sequence according to the weight of the text feature sequence, and updates the weight of the text feature sequence according to the weight of the second video feature sequence;

[0130] The prediction module is used to predict a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence.

[0131] In some embodiments, the second cross-modal interaction module updates the weight of the second video feature sequence according to the weight of the text feature sequence, and updates the weight of the text feature sequence according to the weight of the second video feature sequence, including: obtaining the weight of the video segment in the current state according to the weight of each video segment in the second video feature sequence and the weight of each text vector in the text feature sequence under the historical state; obtaining the weight of the text vector in the current state according to the weight of the text vector and the weight of the video segment under the historical state.

[0132] In some embodiments, before the second cross-modal interaction module updates the weight of the second video feature sequence according to the correlation matrix and updates the weight of the text feature sequence in the Mamba model, the execution method also includes: scaling the video segments based on the correlation between each video segment in the second video feature sequence and the text feature sequence.

[0133] Optionally, in the Mamba model, the second cross-modal interaction module updates the weight of the second video feature sequence according to the weight of the text feature sequence, and updates the weight of the text feature sequence according to the weight of the second video feature sequence, including:

[0134] ;

[0135] ;

[0136] Wherein, f(·) represents the state update function of the Mamba model, g(·) represents the weighted information extracted according to the correlation matrix, the weighted information includes the vectors in the sequence and the weights of the vectors, α is the preset first hyperparameter, and β is the preset second hyperparameter; h represents the state of the sequence in the Mamba model, v is the vector in the second video feature sequence, t is the vector in the text feature sequence, n is the number of state updates of the second video feature sequence, and l is the number of state updates of the text feature sequence.

[0137] In some embodiments, the preprocessing module obtains the first video feature sequence, including: tiling and linearly projecting the video to be retrieved to obtain a video sequence; adding a global temporal marker to the projected video sequence to obtain the first video feature sequence.

[0138] Furthermore, the prediction module predicts a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence, including: predicting a video segment corresponding to the text feature sequence based on the updated second video feature sequence, the text feature sequence and the global timing mark.

[0139] In some embodiments, a preprocessing module obtains a Mamba model, including: obtaining a text feature sequence and a video feature sequence for training based on a data set containing text and video; obtaining a first video segment predicted by the Mamba model based on the text feature sequence and the video feature sequence for training, and a second video segment corresponding to the text feature sequence in the data set; constructing a loss function based on the difference between the first video segment and the second video segment; and adjusting parameters of the Mamba model based on the loss function.

[0140] Optionally, constructing a loss function based on the difference between the first video clip and the second video clip includes: constructing a first loss function and a second loss function based on the difference between the first video clip and the second video clip; wherein the first loss function is an intersection-over-union loss function constructed based on the time information of the first video clip and the time information of the second video clip, or the first loss function is a binary cross entropy loss function constructed based on the significance score pre-assigned to the first video clip; the second loss function includes at least one of the following: a KL divergence loss function, a contrastive learning loss function, and a significance order constraint loss function.

[0141] Each module in the video clip retrieval device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0142] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a video clip retrieval method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projector, etc. The input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0147] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0148] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video clip retrieval method, characterized in that: The method comprises: Obtaining a first video feature sequence and a corresponding text feature sequence, and encoding the first video feature sequence based on a Mamba model to obtain a second video feature sequence; Obtaining initial weights of the second video feature sequence and the text feature sequence according to a correlation matrix between the second video feature sequence and the text feature sequence; In the Mamba model, the weight of the second video feature sequence is updated according to the weight of the text feature sequence, the weight of the text feature sequence is updated according to the weight of the second video feature sequence, and a video segment corresponding to the text feature sequence is predicted based on the updated second video feature sequence and the text feature sequence; Wherein, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, includes: ； ； f(·) represents the state update function of the Mamba model, g(·) represents the weighted information extracted according to the correlation matrix, the weighted information includes the vectors in the sequence and the weights of the vectors, α is the preset first hyperparameter, and β is the preset second hyperparameter; h represents the state of the sequence in the Mamba model, v is the vector in the second video feature sequence, t is the vector in the text feature sequence, n is the number of state updates of the second video feature sequence, and l is the number of state updates of the text feature sequence.

2. The method according to claim 1, characterized in that Updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, comprising: Obtaining the weight of the video segment in the current state according to the weight of each video segment in the second video feature sequence and the weight of each text vector in the text feature sequence in the historical state; The weight of the text vector in the current state is obtained according to the weight of the text vector and the weight of the video clip in the historical state.

3. The method according to claim 1 or 2, characterized in that In the Mamba model, before updating the weight of the second video feature sequence according to the correlation matrix and updating the weight of the text feature sequence, the method further includes: The video segments are scaled based on the correlation between each video segment in the second video feature sequence and the text feature sequence.

4. The method according to claim 1, wherein Obtaining a first video feature sequence, including: Tiling and linear projection are performed on the video to be retrieved to obtain a video sequence; A global temporal marker is added to the video sequence obtained by projection to obtain the first video feature sequence.

5. The method according to claim 4, characterized in that Predicting a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence includes: According to the updated second video feature sequence, the text feature sequence and the global temporal marker, a video segment corresponding to the text feature sequence is predicted.

6. The method according to claim 1, characterized in that Obtain the Mamba model, including: According to a data set containing text and video, a text feature sequence and a video feature sequence for training are obtained; Obtaining a first video segment predicted by the Mamba model based on the text feature sequence and the video feature sequence used for training, and a second video segment in the data set corresponding to the text feature sequence; constructing a loss function based on the difference between the first video clip and the second video clip; Parameters of the Mamba model are adjusted according to the loss function.

7. The method according to claim 6, characterized in that Constructing a loss function based on a difference between the first video segment and the second video segment, comprising: A first loss function and a second loss function are constructed according to the difference between the first video segment and the second video segment; wherein, The first loss function is an intersection-over-union loss function constructed based on time information of the first video clip and time information of the second video clip, or the first loss function is a binary cross entropy loss function constructed based on a pre-assigned significance score of the first video clip; The second loss function includes at least one of the following: a KL divergence loss function, a contrastive learning loss function, and a significance order constraint loss function.

8. A video clip retrieval device, characterized in that: The device comprises: A preprocessing module, which obtains a first video feature sequence and a corresponding text feature sequence, and inputs the first video feature sequence into a Mamba model to obtain a second video feature sequence; a first cross-modal interaction module, which obtains initial weights of the second video feature sequence and the text feature sequence according to a correlation matrix between the second video feature sequence and the text feature sequence; a second cross-modal interaction module, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence; A prediction module, configured to predict a video segment corresponding to the text feature sequence based on the updated second video feature sequence and the text feature sequence; Wherein, in the Mamba model, updating the weight of the second video feature sequence according to the weight of the text feature sequence, and updating the weight of the text feature sequence according to the weight of the second video feature sequence, includes: ； ； f(·) represents the state update function of the Mamba model, g(·) represents the weighted information extracted according to the correlation matrix, the weighted information includes the vectors in the sequence and the weights of the vectors, α is the preset first hyperparameter, and β is the preset second hyperparameter; h represents the state of the sequence in the Mamba model, v is the vector in the second video feature sequence, t is the vector in the text feature sequence, n is the number of state updates of the second video feature sequence, and l is the number of state updates of the text feature sequence.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video clip retrieval method according to any one of claims 1 to 7 are implemented.

Images