Partial correlation video retrieval method based on bidirectional cross-modal collaborative alignment mechanism

Abstract

The application relates to a partial correlation video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism and belongs to the technical field of computer vision and multi-modal information processing. The method comprises the following steps: generating multiple semantic representations through keyword detection and clustering, and constructing a semantic-visual association library; generating dynamic semantic anchor points by using semantic features similar to visual features, and selecting corresponding semantic features from the semantic-visual association library according to keywords in a query; designing a feature purification module guided by the dynamic semantic anchor points and a visual semantic feature injection module at a text level, and collaboratively optimizing visual features and text query features; matching the visual features and the text query features optimized by the feature purification and semantic injection modules based on a similarity score, and selecting a video with the highest score as a final retrieval result. The application improves the matching precision between text queries and video contents, and generates a retrieval result more in line with semantic intentions.

Description

Technical Field

[0001] This invention relates to a partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism, belonging to the field of computer vision and multimodal information processing technology. Background Technology

[0002] With the rapid development of the internet and multimedia technologies, video data has experienced explosive growth. How to efficiently and accurately retrieve target videos that meet users' semantic needs from massive amounts of video data has become an important research topic in the fields of computer vision and multimodal information processing. Text-to-video retrieval, as a typical application of cross-modal retrieval, aims to retrieve semantically relevant video content from video databases based on text queries. It has wide application value in scenarios such as video management, intelligent recommendation, security monitoring, and content moderation. Most existing video retrieval methods are based on the assumption of fully relevant matching, meaning that the text query and video content are assumed to have a high degree of consistency at the overall semantic level. However, in practical applications, users' text queries often only describe local events, local objects, or local semantic fragments in the video, while the video content contains a large amount of redundant information unrelated to the query. This "partially relevant" retrieval requirement makes it difficult for traditional global feature alignment or simple similarity matching methods to accurately capture the true semantic relationship between text and video, resulting in decreased retrieval accuracy.

[0003] To address these issues, researchers have proposed a series of retrieval methods based on attention mechanisms, keyframe modeling, and cross-modal alignment. However, existing methods still have certain limitations. On the one hand, most methods focus on unidirectional cross-modal mapping, neglecting the bidirectional semantic interaction between text and video, resulting in the failure to fully leverage the complementarity between visual and semantic features. On the other hand, existing methods typically rely on static representations for modeling key semantics, making it difficult to dynamically adjust the semantic focus based on specific queries, thus affecting the retrieval performance of some relevant videos. Furthermore, visual features often contain a large amount of noise information irrelevant to the query, and there is a lack of effective semantic guidance mechanisms to purify and optimize them.

[0004] Therefore, how to construct a method that can achieve bidirectional cross-modal collaborative alignment between text and video, and dynamically model key semantic information for some relevant retrieval scenarios, has become a technical problem that urgently needs to be solved in the current video retrieval field. Summary of the Invention

[0005] To overcome the problems of insufficient matching accuracy and limited semantic alignment capability of existing video retrieval methods in partially relevant retrieval scenarios, this invention proposes a partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism to achieve more accurate and efficient semantic matching between text queries and video content.

[0006] The technical solution of this invention is: a partially relevant video retrieval method based on a bidirectional cross-modal cooperative alignment mechanism, the method comprising:

[0007] Extract visual features from the video and text query features from the text query;

[0008] A semantic-visual association library is constructed by generating various semantic representations through keyword detection and clustering.

[0009] Dynamic semantic anchors are generated using semantic features similar to visual features, and corresponding semantic features are selected from the semantic-visual association library based on keywords in the query. Furthermore, a feature purification module guided by dynamic semantic anchors and a visual semantic feature injection module at the text level are designed to perform collaborative optimization processing on visual features and text query features.

[0010] Visual features and text query features optimized by the feature purification and semantic injection modules are matched based on similarity scores, and the video with the highest score is selected as the final retrieval result; and corresponding features are selected according to the matching scores to dynamically update the semantic-visual association library.

[0011] During the training phase, the semantic-visual association library, the semantic-guided feature purification module, and the text-level visual semantic feature injection module are jointly optimized.

[0012] During the inference phase, feature matching is performed based on similarity scores, and the trained relevant video retrieval model is used to retrieve relevant videos.

[0013] Furthermore, the method includes the following specific steps:

[0014] Step 1: Obtain video data and text query data, and input the video data into the visual feature extraction network and the text query data into the text query feature extraction network to obtain the corresponding video features and text query features.

[0015] Step 2: Perform multi-scale processing on the acquired video features. By using different scale averaging methods, obtain coarse-grained feature representations and fine-grained feature representations of the video features.

[0016] Step 3: Construct a semantic-visual association library to associate semantic information with visual information;

[0017] Part-of-speech analysis is performed on the text query data to extract a series of keywords. The keywords are then encoded, and the features of the encoded keywords are clustered to obtain multiple semantic representations. The semantic representations corresponding to the keywords are stored as a semantic-visual association library.

[0018] Step 4: Construct a feature purification module guided by dynamic semantic anchors. Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors; under the guidance of the dynamic semantic anchors, perform feature purification processing on the video features to obtain purified video features.

[0019] Step 5: Construct a visual semantic feature injection module to inject additional semantic features into text query features;

[0020] Based on the keywords contained in the text query, the corresponding visual semantic features are retrieved from the semantic-visual association library, and these visual semantic features are injected into the text query features.

[0021] Step 6: Similarity score calculation. The visual features after feature purification and semantic feature injection are compared with the text query features. Then, the similarity scores of coarse-grained and fine-grained features are averaged, and the video with the highest score is selected as the final search result.

[0022] Step 7: For the dynamic update of the semantic-visual association library, the feature with the highest similarity score between the text and video features is used as the visual feature of the text to update the semantic-visual feature of the keyword corresponding to the text in the semantic-visual association library.

[0023] Step 8: Train the feature purification module and visual semantic feature injection module guided by dynamic semantic anchors.

[0024] Furthermore, Step 2 specifically includes the following steps:

[0025] The acquired video features are first subjected to average pooling to unify the number of features contained in each video. This is used as a fine-grained feature representation. Then, the sampled features are subjected to average pooling with the interval set to 4 to obtain the number of video features. This feature is represented as a coarse-grained feature; the process of multi-scale processing of video features is represented as follows:

[0026] ;

[0027] ;

[0028] in, This indicates the average pooling operation. This represents the video features extracted through a 3D convolutional network. This indicates the number of frames in the video. This represents the fine-grained feature representation obtained after the first average pooling. This represents the coarse-grained feature representation obtained after the second average pooling.

[0029] Furthermore, Step 3 specifically includes the following steps:

[0030] First, SpaCy was used to perform part-of-speech tagging on all text in the dataset, selecting verbs and nouns as keywords. Then, the GloVe model was used to extract word-level features from these keywords, and the K-Means clustering algorithm was used to cluster these features, thus establishing a one-to-many mapping relationship between semantics and keywords. This mapping relationship was stored as a semantic-visual association library. Furthermore, a separate feature space was constructed for each semantic meaning to store the visual features corresponding to that semantic meaning, forming a total of... Semantic categories.

[0031] Furthermore, Step 4 specifically includes the following steps:

[0032] Step 4.1: Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors: specifically including:

[0033] For dynamic semantic anchors, multi-scale processing is first performed to obtain fine-grained feature representations. Representation of coarse-grained features Transform into global feature representation and Then, the top 100 most similar features to the global features are selected from the semantic-visual association library. Using semantic features to construct dynamic semantic anchors and The generation process for fine-grained dynamic semantic anchors is the same as that for coarse-grained dynamic semantic anchors. Specifically, the generation process for fine-grained dynamic semantic anchors is as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] in, This indicates the average pooling operation. This represents the fine-grained global features obtained after average pooling. For semantic data in the semantic-visual association library, For a number from 1 to The set of integers, This represents the semantic features stored in the semantic-visual association library. This indicates the calculation of cosine similarity. This indicates selecting the index corresponding to the highest score. This represents the top TOP values ​​selected from the semantic-visual association database that are most similar to the global features. An index of semantic features, express The first in One index, This represents the Softmax function. Represents fine-grained dynamic semantic anchors;

[0039] Step 4.2: Guided by the dynamic semantic anchor point, perform feature purification processing on the video features to obtain purified video features, specifically including:

[0040] The feature purification process for both fine-grained and coarse-grained feature representations of video features is the same. Specifically, the operation for fine-grained feature representation is as follows:

[0041] Fine-grained feature representation is obtained by multi-scale processing. Input to A parallel Transformer module is used to obtain features that can simultaneously capture long-range time dependencies and local dynamic changes. Subsequently, a cross-attention mechanism was introduced, along with fine-grained dynamic semantic anchors. As a query, the output of the Transformer module The interaction is modeled as keys and values, and a fusion weight is learned using a regularization strategy; finally, this weight is used to optimize the output of each Transformer. Weighted fusion is performed to obtain fine-grained visual features after feature purification. The purification process of fine-grained video features is represented as follows:

[0042] ;

[0043] ;

[0044] ;

[0045] in, express Layer-parallel Transformer module, This represents the features obtained after passing through the parallel Transformer module. For cross-attention modules, This represents a regularization operation. The weights are used to perform weighted fusion of the outputs of each Transformer. Represents scalar multiplication. This represents fine-grained visual features after feature purification.

[0046] Furthermore, Step 5 specifically includes: parsing the text query using SpaCy to extract keywords. Furthermore, it retrieves semantic concepts corresponding to these keywords from the semantic-visual association database. Subsequently, the retrieved semantic features are injected into the text query features. The operations for obtaining fine-grained sentence features and coarse-grained sentence features are the same; among them, fine-grained sentence features... The acquisition process is defined as follows:

[0047] ;

[0048] ;

[0049] ;

[0050] in, This indicates a text query. This indicates the number of words contained in the text query, and SpaCy indicates that the text query is parsed. This indicates the keywords contained in the text query. Indicates the number of keywords. This indicates that the semantic concept corresponding to the keyword is retrieved from the semantic-visual association database. This indicates the semantic concept corresponding to the keyword. This represents the features obtained from a text query using a pre-trained text model. Represents the features corresponding to semantic concepts. This indicates a feature concatenation operation. Represents the Transformer layer. Represents the attention layer. Represents fine-grained sentence features.

[0051] Further, Step 6 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both coarse-grained and fine-grained levels; then, taking the maximum similarity score at each granularity as the final score for that granularity; finally, averaging the final scores at both granularities and selecting the video with the highest score as the final retrieval result; the final retrieval score... The process of obtaining is defined as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] in, This represents fine-grained visual features after feature purification. This represents coarse-grained visual features after feature purification. This represents the calculation of the cosine similarity function. This represents the function that takes the maximum value. and These represent the fine-grained and coarse-grained sentence features after semantic feature injection, respectively. and These represent the similarity scores of the video text pairs at the coarse-grained and fine-grained levels, respectively. This represents the final search score.

[0056] Furthermore, Step 7 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both the coarse-grained and fine-grained levels. and Then, based on the index of the maximum similarity score obtained for each granularity, fine-grained features are obtained after the first average pooling. And the coarse-grained features obtained after the second average pooling. Select from the selected video features and update the semantic features in the semantic-visual association library corresponding to the text query using the selected video features. Then, obtain the final semantic features by dividing by the number of updates. The specific update process is defined as follows:

[0057] ;

[0058] ;

[0059] ;

[0060] in, and The numbers represent the coarse-grained and fine-grained similarity scores obtained by calculating cosine similarity between visual features and text query features after feature purification and semantic feature injection, respectively. This indicates selecting the index corresponding to the highest score. and These represent the feature indices with the highest scores in fine-grained and coarse-grained programming, respectively. and They represent and The fine-grained and coarse-grained features corresponding to the index, and These represent the first and second parts of the semantic-visual association library, respectively. The fine-grained and coarse-grained features corresponding to each semantic meaning Indicates the first The number of semantic updates, and Indicates the first The final fine-grained features and final coarse-grained features of each semantic.

[0061] Furthermore, Step 8 specifically includes the following steps:

[0062] Step 8.1: Optimize the parameters in the feature purification module and the visual semantic feature injection module using the BertAdam optimizer;

[0063] Step 8.2, when training the feature purification module and the visual semantic feature injection module, two loss functions are defined: contrastive loss and triplet loss. These two functions will convert the scores obtained in Step 6 into the training loss function. and Calculate the loss against negative samples in the same batch;

[0064] The fine-grained contrast loss is defined as follows:

[0065] ;

[0066] in and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, and These represent the video within a batch. All text negative samples and for video All video negative samples, To represent logarithmic calculation, This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained contrast loss; the coarse-grained contrast loss is obtained in the same way.

[0067] The fine-grained triplet loss is defined as follows:

[0068] ;

[0069] and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, This represents the function that takes the maximum value. These are predefined margins, where This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained triplet loss; the coarse-grained triplet loss is obtained in the same way.

[0070] The present invention also provides a partially relevant video retrieval system based on a bidirectional cross-modal collaborative alignment mechanism, the system comprising: a module for executing the partially relevant video retrieval method based on the bidirectional cross-modal collaborative alignment mechanism.

[0071] The beneficial effects of this invention are:

[0072] 1. Improve the matching accuracy and semantic consistency of some related video retrieval: This invention constructs a "bidirectional cross-modal collaborative alignment mechanism" to introduce dynamic semantic anchors at the visual feature end for guidance and purification, and injects visual semantic information at the text query feature end for reinforcement, so that video content and text query can form complementary alignment at the semantic and visual levels, thereby significantly improving the matching accuracy between text intent and video segments and generating retrieval results that are more in line with semantic intent.

[0073] 2. Enhanced robustness and generalization ability in complex scenarios: This invention obtains coarse-grained and fine-grained feature representations through multi-scale processing, and performs noise filtering and information focusing on video features under the guidance of dynamic semantic anchors, effectively alleviating problems such as background redundancy, occlusion, action changes and semantic loss in real videos; at the same time, the semantic-visual association library provides stable semantic priors, enabling the model to maintain high retrieval robustness and generalization ability when facing long videos, weakly related segments or incomplete query expressions;

[0074] 3. Improve the efficiency of cross-modal feature collaborative optimization and reduce interference from irrelevant information: The dynamic semantic anchor guided feature purification module designed in this invention can adaptively filter key visual information related to the query in coarse and fine granular features, reducing the interference of irrelevant fragments on similarity calculation; while the visual semantic feature injection module further bridges the semantic gap between text description and video content, making the cross-modal alignment process more efficient and the retrieval results ranking more stable and reliable.

[0075] 4. Achieve dynamic updates to the semantic-visual association library and continuously improve retrieval performance: This invention selects the most relevant visual features based on the similarity scores between text and video features to update the semantic-visual association library, enabling the semantic representation to gradually align with the distribution of real videos during training and inference. This dynamic update mechanism can continuously optimize the consistency between semantic priors and visual representations, avoiding performance degradation caused by an outdated semantic library, thereby achieving more efficient and stable partial related video retrieval in different data domains and application scenarios. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the process structure of the method of the present invention;

[0077] Figure 2 This is a structural diagram of the feature purification module guided by the dynamic semantic anchor point in the method of the present invention;

[0078] Figure 3 A structural diagram of the visual semantic feature injection module of the method of the present invention;

[0079] Figure 4 This is a structural diagram of the similarity score calculation module of the method of the present invention;

[0080] Figure 5 This is a diagram illustrating the dynamic update structure of the semantic-visual association library in the method of this invention. Detailed Implementation

[0081] Example 1: As Figures 1-5 As shown, a partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism includes the following specific steps:

[0082] Step 1: Obtain video data and text query data, and input the video data into the visual feature extraction network and the text query data into the text query feature extraction network to obtain the corresponding video features and text query features.

[0083] Furthermore, Step 1 includes preprocessing:

[0084] The preprocessing includes: inputting video into a pre-trained 3D convolutional network to extract video features, and inputting text into a pre-trained text model to extract text features. Before training begins, the weight parameters of all linear and embedding layers are randomly initialized using a normal distribution with a mean of 0 and a standard deviation as the initialization range parameter, to ensure the stability of the weight distribution and facilitate model convergence.

[0085] When the module is a layer normalization layer, its bias parameter is initialized to 0 and its scale parameter is initialized to 1, so as to maintain the ability of the normalization operation to stably adjust the feature distribution in the early stage of training.

[0086] When the module is a one-dimensional convolutional layer (Conv1d), its default parameter initialization method is called to reset the convolutional kernel weights and bias parameters to ensure that the convolutional layer parameters meet the initial training requirements of the network.

[0087] Furthermore, when the module is a linear layer containing a bias term, its bias parameter is further initialized to 0 to reduce the impact of the initial bias on the feature mapping result, thereby improving the stability and controllability of the model training process.

[0088] Finally, the acquired video and text features are unified in dimensionality using single-layer fully connected layers, reducing the feature dimensionality to [the minimum required level]. dimension.

[0089] Step 2: Perform multi-scale processing on the acquired video features. By using different scale averaging methods, obtain coarse-grained feature representations and fine-grained feature representations of the video features.

[0090] Furthermore, Step 2 specifically includes the following steps:

[0091] The acquired video features are first subjected to average pooling to unify the number of features contained in each video. This is used as a fine-grained feature representation. Then, the sampled features are subjected to average pooling with the interval set to 4 to obtain the number of video features. This feature is used as a coarse-grained feature representation; wherein, the average pooling is achieved by uniformly dividing the original video feature sequence along the time dimension and averaging the feature vectors within each division interval. The process of multi-scale processing of video features is represented as follows:

[0092] ;

[0093] ;

[0094] in, This indicates the average pooling operation. This represents the video features extracted through a 3D convolutional network. This indicates the number of frames in the video. This represents the fine-grained feature representation obtained after the first average pooling. This represents the coarse-grained feature representation obtained after the second average pooling.

[0095] Step 3: Construct a semantic-visual association library to associate semantic information with visual information;

[0096] Part-of-speech analysis is performed on the text query data to extract a series of keywords. The keywords are then encoded, and the features of the encoded keywords are clustered to obtain multiple semantic representations. The semantic representations corresponding to the keywords are stored as a semantic-visual association library.

[0097] Furthermore, Step 3 specifically includes the following steps:

[0098] First, SpaCy is used to perform part-of-speech tagging on all text in the dataset, selecting verbs and nouns as keywords. Then, the GloVe model is used to extract word-level features from these keywords, and the K-Means clustering algorithm is used to cluster these features, thus establishing a one-to-many mapping relationship between semantics and keywords. This mapping relationship is stored as a semantic-visual association library. Further, a separate feature space is constructed for each semantic meaning to store the corresponding visual features. Each semantic feature space stores two features at different granularities and records the update count for each semantic meaning, forming a total of [database name missing]. Semantic categories.

[0099] Step 4: Construct a feature purification module guided by dynamic semantic anchors. Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors; under the guidance of the dynamic semantic anchors, perform feature purification processing on the video features to obtain purified video features.

[0100] Furthermore, Step 4 specifically includes the following steps:

[0101] Step 4.1: Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors: specifically including:

[0102] For dynamic semantic anchors, multi-scale processing is first performed to obtain fine-grained feature representations. Representation of coarse-grained features Transform into global feature representation and Then, the top 100 most similar features to the global features are selected from the semantic-visual association library. Using semantic features to construct dynamic semantic anchors and The generation process for fine-grained dynamic semantic anchors is the same as that for coarse-grained dynamic semantic anchors. Specifically, the generation process for fine-grained dynamic semantic anchors is as follows:

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] in, This indicates the average pooling operation. This represents the fine-grained global features obtained after average pooling. For semantic data in the semantic-visual association library, For a number from 1 to The set of integers, This represents the semantic features stored in the semantic-visual association library. This indicates the calculation of cosine similarity. This indicates selecting the index corresponding to the highest score. This represents the top TOP values ​​selected from the semantic-visual association database that are most similar to the global features. An index of semantic features, express The first in One index, This represents the Softmax function. Represents fine-grained dynamic semantic anchors;

[0108] Step 4.2: Guided by the dynamic semantic anchor point, perform feature purification processing on the video features to obtain purified video features, specifically including:

[0109] The feature purification process for both fine-grained and coarse-grained feature representations of video features is the same. Specifically, the operation for fine-grained feature representation is as follows:

[0110] Fine-grained feature representation is obtained by multi-scale processing. Input to A parallel Transformer module is used to obtain features that can simultaneously capture long-range time dependencies and local dynamic changes. Subsequently, a cross-attention mechanism was introduced, along with fine-grained dynamic semantic anchors. As a query, the output of the Transformer module The interaction is modeled as keys and values, and a fusion weight is learned using a regularization strategy; finally, this weight is used to optimize the output of each Transformer. Weighted fusion is performed to obtain fine-grained visual features after feature purification. The purification process of fine-grained video features is represented as follows:

[0111] ;

[0112] ;

[0113] ;

[0114] in, express Layer-parallel Transformer module, This represents the features obtained after passing through the parallel Transformer module. For cross-attention modules, This represents a regularization operation. The weights are used to perform weighted fusion of the outputs of each Transformer. Represents scalar multiplication. This represents fine-grained visual features after feature purification.

[0115] Step 5: Construct a visual semantic feature injection module to inject additional semantic features into text query features;

[0116] Based on the keywords contained in the text query, the corresponding visual semantic features are retrieved from the semantic-visual association library, and these visual semantic features are injected into the text query features.

[0117] Furthermore, Step 5 specifically includes: parsing the text query using SpaCy to extract keywords. Furthermore, it retrieves semantic concepts corresponding to these keywords from the semantic-visual association database. Subsequently, the retrieved semantic features are injected into the text query features. The operations for obtaining fine-grained sentence features and coarse-grained sentence features are the same; among them, fine-grained sentence features... The acquisition process is defined as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] in, This indicates a text query. This indicates the number of words contained in the text query, and SpaCy indicates that the text query is parsed. This indicates the keywords contained in the text query. Indicates the number of keywords. This indicates that the semantic concept corresponding to the keyword is retrieved from the semantic-visual association database. This indicates the semantic concept corresponding to the keyword. This represents the features obtained from a text query using a pre-trained text model. Represents the features corresponding to semantic concepts. This indicates a feature concatenation operation. Represents the Transformer layer. Represents the attention layer. Represents fine-grained sentence features.

[0122] Step 6: Similarity score calculation. The visual features after feature purification and semantic feature injection are compared with the text query features. Then, the similarity scores of coarse-grained and fine-grained features are averaged, and the video with the highest score is selected as the final search result.

[0123] Further, Step 6 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both coarse-grained and fine-grained levels; then, taking the maximum similarity score at each granularity as the final score for that granularity; finally, averaging the final scores at both granularities and selecting the video with the highest score as the final retrieval result; the final retrieval score... The process of obtaining is defined as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] in, This represents fine-grained visual features after feature purification. This represents coarse-grained visual features after feature purification. This represents the calculation of the cosine similarity function. This represents the function that takes the maximum value. and These represent the fine-grained and coarse-grained sentence features after semantic feature injection, respectively. and These represent the similarity scores of the video text pairs at the coarse-grained and fine-grained levels, respectively. This represents the final search score.

[0128] Step 7: For the dynamic update of the semantic-visual association library, the feature with the highest similarity score between the text and video features is used as the visual feature of the text to update the semantic-visual feature of the keyword corresponding to the text in the semantic-visual association library.

[0129] Furthermore, Step 7 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both the coarse-grained and fine-grained levels. and Then, based on the index of the maximum similarity score obtained for each granularity, fine-grained features are obtained after the first average pooling. And the coarse-grained features obtained after the second average pooling. Select from the selected video features and update the semantic features in the semantic-visual association library corresponding to the text query using the selected video features. Then, obtain the final semantic features by dividing by the number of updates. The specific update process is defined as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] in, and The numbers represent the coarse-grained and fine-grained similarity scores obtained by calculating cosine similarity between visual features and text query features after feature purification and semantic feature injection, respectively. This indicates selecting the index corresponding to the highest score. and These represent the feature indices with the highest scores in fine-grained and coarse-grained programming, respectively. and They represent and The fine-grained and coarse-grained features corresponding to the index, and These represent the first and second parts of the semantic-visual association library, respectively. The fine-grained and coarse-grained features corresponding to each semantic meaning Indicates the first The number of semantic updates, and Indicates the first The final fine-grained features and final coarse-grained features of each semantic.

[0134] Step 8: Train the feature purification module and visual semantic feature injection module guided by dynamic semantic anchors.

[0135] Furthermore, Step 8 specifically includes the following steps:

[0136] Step 8.1: Optimize the parameters in the feature purification module and the visual semantic feature injection module using the BertAdam optimizer;

[0137] Step 8.2, when training the feature purification module and the visual semantic feature injection module, two loss functions are defined: contrastive loss and triplet loss. These two functions will convert the scores obtained in Step 6 into the training loss function. and Calculate the loss against negative samples in the same batch;

[0138] The fine-grained contrast loss is defined as follows:

[0139] ;

[0140] in and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, and These represent the video within a batch. All text negative samples and for video All video negative samples, To represent logarithmic calculation, This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained contrast loss; the coarse-grained contrast loss is obtained in the same way.

[0141] The fine-grained triplet loss is defined as follows:

[0142] ;

[0143] and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, This represents the function that takes the maximum value. These are predefined margins, where This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained triplet loss; the coarse-grained triplet loss is obtained in the same way.

[0144] This invention also provides a partially relevant video retrieval system based on a bidirectional cross-modal cooperative alignment mechanism, the system comprising:

[0145] The preprocessing module is used to acquire video data and text data, and input the video data into a visual feature extraction network and the text data into a text query feature extraction network, respectively, to obtain the corresponding video features and text query features.

[0146] The multi-scale processing module is used to perform multi-scale processing on the acquired video features. By using averaging methods at different scales, it obtains coarse-grained and fine-grained feature representations of the video features.

[0147] A semantic-visual association library is used to perform part-of-speech analysis on text data to extract a series of keywords, encode the keywords, cluster the features of the encoded keywords to obtain multiple semantic representations, and store the semantic representations corresponding to the keywords.

[0148] A feature purification module guided by dynamic semantic anchors is used to input the coarse-grained feature representation and the fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors; under the guidance of the dynamic semantic anchors, the video features are purified to obtain purified video features;

[0149] The visual semantic feature injection module is used to inject additional semantic features into text query features; it retrieves the corresponding visual semantic features from the semantic-visual association library based on the keywords contained in the text query and injects these visual semantic features into the text query features.

[0150] The similarity score calculation module is used to calculate the similarity between visual features after feature purification and semantic feature injection and text query features. Then, the similarity scores of coarse-grained and fine-grained features are averaged, and the video with the highest score is selected as the final search result.

[0151] The dynamic update module of the semantic-visual association library is used to update the semantic-visual features of the corresponding keywords in the semantic-visual association library by using the feature with the highest similarity score between the text and video features as the visual feature of the text.

[0152] The training module is used to train the feature purification module and the visual semantic feature injection module guided by dynamic semantic anchors in the training content;

[0153] To verify the effectiveness of the method in this invention, ActivityNet-Captions and TVR were used as datasets. The ActivityNet-Captions dataset contains approximately 20,000 YouTube videos with an average duration of 117.6 seconds. Each video is associated with an average of 3.7 text queries, each containing approximately 14.8 words, for a total of approximately 8,500 keywords. The TVR dataset contains approximately 20,000 video clips from six different types of TV series, each with an average duration of approximately 8 minutes. Each video is associated with an average of 5 text queries, each consisting of approximately 13 words, for a total of approximately 13,800 keywords. The algorithm of this invention is developed based on PyTorch and trained on a single NVIDIA RTX 4090 GPU. The I3D network is used to extract video features from the ActivityNet-Captions dataset, and both I3D and ResNet152 networks are applied to extract video features from the TVR dataset. For text queries, RoBERTa is used as the text encoder. During training, early stopping is applied with a patience factor of 10 epochs out of a total of 100 epochs to prevent overfitting. The initial learning rate is set to 0.0001. This method employs two parallel Transformer modules. The number of clusters is set to 200, the Top-K value is set to 5, and the dimension is... 384, downsampling It is 128.

[0154] This invention uses ranking-based metrics R@K (K=1,5,10,100) and SumR to evaluate the model. R@K represents the percentage of text queries that retrieve at least one correctly matching video among the top K search results. SumR is defined as the sum of R@1, R@5, R@10, and R@100. Higher R@K and SumR scores indicate better retrieval performance.

[0155] Tables 1 and 2 summarize the performance of this invention on the ActivityNet-Captions and TVR datasets, respectively, and compare it with methods such as PEAN, GMMFormer, ARL, PBU, and HLFormer. Experimental results show that this invention outperforms the comparison methods in all evaluation metrics, including R@K (K=1, 5, 10, 100) and SumR. On the ActivityNet-Captions dataset shown in Table 1, compared with the HLFormer method, this invention improves the Recall@1, Recall@5, Recall@10, and Recall@100 metrics by 0.8%, 1.2%, 1.0%, and 0.4%, respectively, and the SumR metric by 3.4%. On the TVR dataset shown in Table 2, compared with the HLFormer method, this invention improves the Recall@1, Recall@5, Recall@10, and Recall@100 metrics by 0.6%, 1.1%, 1.5%, and 1.2%, respectively, and the SumR metric by 4.5%, significantly demonstrating its performance advantage.

[0156] Table 1 shows the performance of different methods on the ActivityNet-Captions dataset.

[0157] Table 2 shows the performance of different methods on the TVR dataset.

[0158] The specific embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism, characterized in that: The method includes: Extract visual features from the video and text query features from the text query; A semantic-visual association library is constructed by generating various semantic representations through keyword detection and clustering. Dynamic semantic anchors are generated using semantic features similar to visual features, and corresponding semantic features are selected from the semantic-visual association library based on keywords in the query. Furthermore, a feature purification module guided by dynamic semantic anchors and a visual semantic feature injection module at the text level are designed to perform collaborative optimization processing on visual features and text query features. Visual features and text query features optimized by the feature purification and semantic injection modules are matched based on similarity scores, and the video with the highest score is selected as the final retrieval result; and corresponding features are selected according to the matching scores to dynamically update the semantic-visual association library. During the training phase, the semantic-visual association library, the semantic-guided feature purification module, and the text-level visual semantic feature injection module are jointly optimized. During the inference phase, feature matching is performed based on similarity scores, and the trained relevant video retrieval model is used to retrieve relevant videos.

2. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 1, characterized in that: The method includes the following specific steps: Step 1: Obtain video data and text query data, and input the video data into the visual feature extraction network and the text query data into the text query feature extraction network to obtain the corresponding video features and text query features. Step 2: Perform multi-scale processing on the acquired video features. By using different scale averaging methods, obtain coarse-grained feature representations and fine-grained feature representations of the video features. Step 3: Construct a semantic-visual association library to associate semantic information with visual information; Part-of-speech analysis is performed on the text query data to extract a series of keywords. The keywords are then encoded, and the features of the encoded keywords are clustered to obtain multiple semantic representations. The semantic representations corresponding to the keywords are stored as a semantic-visual association library. Step 4: Construct a feature purification module guided by dynamic semantic anchors. Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors; under the guidance of the dynamic semantic anchors, perform feature purification processing on the video features to obtain purified video features. Step 5: Construct a visual semantic feature injection module to inject additional semantic features into text query features; Based on the keywords contained in the text query, the corresponding visual semantic features are retrieved from the semantic-visual association library, and these visual semantic features are injected into the text query features. Step 6: Similarity score calculation. The visual features after feature purification and semantic feature injection are compared with the text query features. Then, the similarity scores of coarse-grained and fine-grained features are averaged, and the video with the highest score is selected as the final search result. Step 7: For the dynamic update of the semantic-visual association library, the feature with the highest similarity score between the text and video features is used as the visual feature of the text to update the semantic-visual feature of the keyword corresponding to the text in the semantic-visual association library. Step 8: Train the feature purification module and visual semantic feature injection module guided by dynamic semantic anchors.

3. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: The specific steps of Step 2 include: The acquired video features are first subjected to average pooling to unify the number of features contained in each video. This is used as a fine-grained feature representation. Then, the sampled features are subjected to average pooling with the interval set to 4 to obtain the number of video features. This feature is represented as a coarse-grained feature; the process of multi-scale processing of video features is represented as follows: ； ； in, This indicates the average pooling operation. This represents the video features extracted through a 3D convolutional network. This indicates the number of frames in the video. This represents the fine-grained feature representation obtained after the first average pooling. This represents the coarse-grained feature representation obtained after the second average pooling.

4. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: The specific steps in Step 3 include: First, SpaCy was used to perform part-of-speech tagging on all text in the dataset, selecting verbs and nouns as keywords. Then, the GloVe model was used to extract word-level features from these keywords, and the K-Means clustering algorithm was used to cluster these features, thus establishing a one-to-many mapping relationship between semantics and keywords. This mapping relationship was stored as a semantic-visual association library. Furthermore, a separate feature space was constructed for each semantic meaning to store the visual features corresponding to that semantic meaning, forming a total of... Semantic categories.

5. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: The specific steps in Step 4 include: Step 4.1: Input the coarse-grained feature representation and fine-grained feature representation into the feature purification module; select semantic features similar to video features from the semantic-visual association library, and aggregate the semantic features to generate dynamic semantic anchors: specifically including: For dynamic semantic anchors, multi-scale processing is first performed to obtain fine-grained feature representations. Representation of coarse-grained features Transform into global feature representation and Then, the top 100 most similar features to the global features are selected from the semantic-visual association library. Using semantic features to construct dynamic semantic anchors and The generation process for fine-grained dynamic semantic anchors is the same as that for coarse-grained dynamic semantic anchors. Specifically, the generation process for fine-grained dynamic semantic anchors is as follows: ； ； ； ； in, This indicates the average pooling operation. This represents the fine-grained global features obtained after average pooling. For semantic data in the semantic-visual association library, For a number from 1 to The set of integers, This represents the semantic features stored in the semantic-visual association library. This indicates the calculation of cosine similarity. This indicates selecting the index corresponding to the highest score. This represents the top TOP values ​​selected from the semantic-visual association database that are most similar to the global features. An index of semantic features, express The first in One index, This represents the Softmax function. Represents fine-grained dynamic semantic anchors; Step 4.2: Guided by the dynamic semantic anchor point, perform feature purification processing on the video features to obtain purified video features, specifically including: The feature purification process for both fine-grained and coarse-grained feature representations of video features is the same. Specifically, the operation for fine-grained feature representation is as follows: Fine-grained feature representation is obtained by multi-scale processing. Input to A parallel Transformer module is used to obtain features that can simultaneously capture long-range time dependencies and local dynamic changes. Subsequently, a cross-attention mechanism was introduced, along with fine-grained dynamic semantic anchors. As a query, the output of the Transformer module The interaction is modeled as keys and values, and a fusion weight is learned using a regularization strategy; finally, this weight is used to optimize the output of each Transformer. Weighted fusion is performed to obtain fine-grained visual features after feature purification. The purification process of fine-grained video features is represented as follows: ； ； ； in, express Layer-parallel Transformer module, This represents the features obtained after passing through the parallel Transformer module. For cross-attention modules, This represents a regularization operation. The weights are used to perform weighted fusion of the outputs of each Transformer. Represents scalar multiplication. This represents fine-grained visual features after feature purification.

6. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: Step 5 specifically includes: parsing the text query using SpaCy to extract keywords. Furthermore, it retrieves semantic concepts corresponding to these keywords from the semantic-visual association database. Subsequently, the retrieved semantic features are injected into the text query features. The operations for obtaining fine-grained sentence features and coarse-grained sentence features are the same; among them, fine-grained sentence features... The acquisition process is defined as follows: ； ； ； in, This indicates a text query. This indicates the number of words contained in the text query, and SpaCy indicates that the text query is parsed. This indicates the keywords contained in the text query. Indicates the number of keywords. This indicates that the semantic concept corresponding to the keyword is retrieved from the semantic-visual association database. This indicates the semantic concept corresponding to the keyword. This represents the features obtained from a text query using a pre-trained text model. Represents the features corresponding to semantic concepts. This indicates a feature concatenation operation. Represents the Transformer layer. Represents the attention layer. Represents fine-grained sentence features.

7. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: Step 6 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both coarse-grained and fine-grained levels; then, taking the maximum similarity score at each granularity as the final score for that granularity; finally, averaging the final scores at both granularities and selecting the video with the highest score as the final retrieval result; the final retrieval score... The process of obtaining is defined as follows: ； ； ； in, This represents fine-grained visual features after feature purification. This represents coarse-grained visual features after feature purification. This represents the calculation of the cosine similarity function. This represents the function that takes the maximum value. and These represent the fine-grained and coarse-grained sentence features after semantic feature injection, respectively. and These represent the similarity scores of the video text pairs at the coarse-grained and fine-grained levels, respectively. This represents the final search score.

8. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: Step 7 specifically includes: calculating the similarity scores of the visual features and text query features after feature purification and semantic feature injection using cosine similarity at both the coarse-grained and fine-grained levels. and Then, based on the index of the maximum similarity score obtained for each granularity, fine-grained features are obtained after the first average pooling. And the coarse-grained features obtained after the second average pooling. Select from the selected video features and update the semantic features in the semantic-visual association library corresponding to the text query using the selected video features. Then, obtain the final semantic features by dividing by the number of updates. The specific update process is defined as follows: ； ； ； in, and The numbers represent the coarse-grained and fine-grained similarity scores obtained by calculating cosine similarity between visual features and text query features after feature purification and semantic feature injection, respectively. This indicates selecting the index corresponding to the highest score. and These represent the feature indices with the highest scores in fine-grained and coarse-grained programming, respectively. and They represent and The fine-grained and coarse-grained features corresponding to the index, and These represent the first and second parts of the semantic-visual association library, respectively. The fine-grained and coarse-grained features corresponding to each semantic meaning Indicates the first The number of semantic updates, and Indicates the first The final fine-grained features and final coarse-grained features of each semantic.

9. The partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism according to claim 2, characterized in that: The specific steps in Step 8 include: Step 8.1: Optimize the parameters in the feature purification module and the visual semantic feature injection module using the BertAdam optimizer; Step 8.2, when training the feature purification module and the visual semantic feature injection module, two loss functions are defined: contrastive loss and triplet loss. These two functions will convert the scores obtained in Step 6 into the training loss function. and Calculate the loss against negative samples in the same batch; The fine-grained contrast loss is defined as follows: ； in and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, and These represent the video within a batch. All text negative samples and for video All video negative samples, To represent logarithmic calculation, This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained contrast loss; the coarse-grained contrast loss is obtained in the same way. The fine-grained triplet loss is defined as follows: ； and These represent the video within a batch. Its negative samples The score and for the text Its negative samples The score, This represents the function that takes the maximum value. These are predefined margins, where This represents all matching video-text pairs within a batch. This indicates the number of all matching video-text pairs within a batch. This represents the fine-grained triplet loss; the coarse-grained triplet loss is obtained in the same way.

10. A partially related video retrieval system based on a bidirectional cross-modal collaborative alignment mechanism, characterized in that, The system includes a module for performing the partially relevant video retrieval method based on a bidirectional cross-modal collaborative alignment mechanism as described in any one of claims 1 to 9.

Images