Video text retrieval method, system, device and medium based on multi semantic space

Abstract

This invention discloses a video text retrieval method, system, device, and medium based on multiple semantic spaces: video feature encoding yields video features and multiple video semantic subspaces; text feature encoding yields text features and multiple text semantic subspaces; cross-modal subspace learning reduces the semantic gap between two modalities; same-modal subspace learning yields semantically distinct subspaces; both same-modal and cross-modal losses are applied simultaneously to the video text retrieval task; through subspace fusion matching, the similarity scores of multiple semantic spaces are weighted and fused to fully utilize multiple semantic subspaces to achieve mutual retrieval between video and text; the retrieval system includes a feature encoding module and a subspace learning module, including a cross-modal subspace learning module and a same-modal subspace learning module. This invention adaptively learns and represents semantic subspaces of different modalities by maintaining differences within the same-modal subspace and consistency across cross-modal subspaces.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and language, and mainly involves mutual retrieval between video and text. More specifically, it relates to a video text retrieval method, system, device and medium based on multi-semantic space. Background Technology

[0002] Multimedia retrieval research has been developing for a long time, but for some time scholars have focused on retrieval of single-modal data. In a single-modal data retrieval scenario, users input text content to retrieve other desired text content or input an image to retrieve other similar images. However, in today's rapidly developing internet era, single-modal data retrieval struggles to meet users' ever-growing data retrieval needs. Therefore, in recent years, multimodal data retrieval methods have gradually attracted researchers' attention. With the development of short videos, text and video retrieval has become a research focus in the field of cross-modal retrieval.

[0003] The retrieval between video and text is essentially a retrieval between the semantics of two different modalities. Both video and text data possess rich semantics; for example, videos contain various semantic elements such as background, music, characters, and actions, while text contains verbs, nouns, polysemous words, and modal particles. Therefore, retrieving video from text is equivalent to simultaneously matching multiple different textual semantic information with multiple different video semantic information. Representing video and text as separate vectors for retrieval is equivalent to merging the various semantics of video and text into a common vector space to calculate similarity for matching. Due to the complexity and diversity of video and text semantics, achieving accurate matching through single-vector retrieval is extremely difficult. MEE and HGR address these issues by encoding video and text into a common semantic space for matching, respectively, based on the multi-semantic nature of video and text. This can alleviate the problem of mutual retrieval between video and text in complex and diverse semantic scenarios to some extent. However, manually finding suitable semantic features for representation requires experience and is time-consuming. Furthermore, to match multiple semantic spaces of video and text, it is necessary to consider whether a certain semantic space of the video can match a certain semantic space of the text. If we force a matching of semantic spaces that are obviously not compatible, the final result may become worse. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a video-text retrieval method, system, device, and medium based on multi-semantic space. This method not only adaptively generates multiple differentiated semantic subspaces but also fully utilizes these subspaces through multi-semantic space fusion to improve video-text matching performance. Semantic subspace generation and matching reduce the semantic gap between two modalities through model training, thereby maintaining consistency across modal spaces. However, differences must be maintained for semantic spaces within the same modality. The adaptively generated semantic subspaces are characterized by maintaining differences within the same modality and consistency across modalities. This approach eliminates the need for manual mining and representation of different semantic information in videos or texts and allows for highly flexible settings of the number of semantic subspaces and subspace matching methods to improve retrieval performance based on actual application scenarios.

[0005] The objective of this invention is achieved through the following technical solutions.

[0006] A video text retrieval method based on multiple semantic spaces includes the following process:

[0007] Step 1: Feature Encoding

[0008] ① Video feature encoding to obtain video features v feat and multiple video semantic subspaces

[0009] First, frames in the video are sampled chronologically. Then, a CNN model pre-trained on the ImageNet dataset is used to extract a corresponding feature vector for each sampled image. Each extracted feature vector is 2048-dimensional. The two extracted feature vectors are then concatenated into a 4096-dimensional feature vector. The resulting video frame-level features are represented as a feature sequence {v1, v2, ..., v...}. n}, where v i This represents the feature vector of the i-th frame in an n-frame video, and then attention pooling is used to further obtain the video frame-level aggregated feature v. f ;

[0010] The video frame-level feature sequence {v1,v2,...,v n The sequence features of the video are extracted using a bidirectional gated recurrent unit (BRU). The hidden state sequence output by the BRU is denoted as... in This represents the hidden state of the bidirectional gated recurrent unit at time step j, and then attention pooling is used to further obtain the video sequence aggregation feature v. s ;

[0011] Aggregate the frame-level features of the video v f With sequence aggregation feature vs The final feature representation v of the video is obtained by splicing. feat The video features are mapped into k semantic subspaces using k fully connected layers, denoted as:

[0012]

[0013] ② Text feature encoding to obtain text features t feat and multiple text semantic subspaces

[0014] A pre-trained word vector matrix is ​​obtained by pre-training an existing dual-encoding model. Words in the text are then converted into word vectors through this pre-trained matrix, thus transforming the text into word vectors, denoted as: {t1, t2, ..., t...} m Word vectors are processed through a bidirectional gated recurrent unit to obtain a word vector sequence. These two sequences are then processed through attention pooling to obtain word vector aggregation features and word vector sequence aggregation features, respectively. The final text feature representation t is obtained by concatenating the word vector aggregation features and word vector sequence aggregation features. feat Then, k fully connected layers are used to map k text semantic subspaces, denoted as:

[0015] Step 2: Subspace Learning

[0016] ① Cross-modal subspace learning

[0017] The cosine similarity function calculates the distance between vectors of different modalities, making similar cross-modal samples closer together and dissimilar cross-modal samples farther apart, thus deriving the loss function for cross-modal matching in a single semantic subspace, and further obtaining the loss function L for cross-modal matching in k subspaces. cross Video and text cross-modal subspace learning mines positive and negative samples through a loss function of cross-modal matching of k subspaces;

[0018] ② Same-modal subspace learning

[0019] The semantic subspaces of video and text are transformed into probability distributions. For multiple semantic subspace distributions of the same modality, the KL divergence is calculated pairwise, and the sum of the differences is maximized to obtain the loss function that maximizes the semantic space difference of the video. Loss function that maximizes the semantic space difference of text

[0020] Video and text utilize the aforementioned cross-modal matching loss function L cross Cross-modal semantic subspace consistency optimization is performed to reduce the semantic gap between two modalities, utilizing the aforementioned loss function. and We perform semantic difference optimization within the same modality to obtain semantically distinct subspaces; we apply both same-modality and cross-modality losses to the video text retrieval task to derive a weighted loss function L.

[0021] Step 3: Subspace Fusion and Matching

[0022] By weighted fusion of similarity scores from multiple semantic subspaces, the mutual retrieval between videos and text can be fully utilized.

[0023] The video frame-level aggregation feature v mentioned in the first step f

[0024]

[0025]

[0026] Among them, v f It is a video frame-level aggregation feature, γ i This represents the frame-level aggregation weight of the i-th video frame. This represents the video frame-level attention weights, where ReLU is the activation function;

[0027] Hidden state of the bidirectional gated loop unit at time step j Represented as:

[0028]

[0029]

[0030]

[0031] Wherein, Concat represents vector concatenation; and These represent the forward and reverse gated loop units, respectively. and These represent the hidden states of the forward and reverse GRUs at time step j, respectively.

[0032] Video sequence aggregation features v s Represented as:

[0033]

[0034]

[0035] Among them, v s It is a video sequence aggregation feature, β i This represents the aggregate weight of the i-th frame video sequence. This represents the attention weights of the video sequence;

[0036] The final feature representation of the video v feat as follows:

[0037] v feat =Concat(v f ,v s )

[0038] The loss function for cross-modal matching of a single semantic subspace as described in step two:

[0039]

[0040] in, This represents the loss function for matching the i-th semantic subspace, and D represents the size of the corpus dictionary; and They represent the video respectively. Relevant and irrelevant texts are also referred to as videos. Positive and negative samples in the i-th semantic subspace; and Respectively represent the text Relevant and irrelevant videos are also referred to as text. The positive and negative samples of the i-th semantic subspace; S represents the cosine similarity function; α represents the distance constant;

[0041] The cross-modal matching loss function for the k subspaces mentioned in the second step is expressed as follows:

[0042]

[0043] The probability distribution of the i-th semantic subspace of the video in the second step is represented as follows: The Softmax function transforms the values ​​of a vector into a probability distribution; similarly, it yields the probability distribution of the i-th semantic subspace of the text.

[0044] The loss function that maximizes semantic space differences in the video described in step two. for:

[0045]

[0046] The loss function that maximizes semantic space differences in the text described in step two. for:

[0047]

[0048] The weighted loss function L mentioned in the second step is expressed as:

[0049]

[0050] Among them, w cross This represents the weights for cross-modal semantic subspace consistency optimization, w same This represents the weights for optimizing the differences in the semantic subspace of the same modality.

[0051] The similarity score fusion representation in the third step is as follows:

[0052]

[0053] Where, μ o This represents the weight of similarity in a certain semantic subspace, and satisfies... S represents the cosine similarity function; in the similarity matrix Before calculating the weighted sum, the similarity matrix is ​​normalized.

[0054] A video text retrieval system based on adaptive multi-semantic space representation includes:

[0055] (1) Feature coding module, including video feature coding module and text feature coding module;

[0056] The video feature encoding module is used to obtain the feature vector of each frame of video, extract the sequence features of the video through a bidirectional gated recurrent unit, and use attention pooling to obtain the frame-level aggregated features v of the video. f With sequence aggregation feature v s The final feature representation v of the video is obtained by splicing. feat Then, k fully connected layers are used to map the video features into k video semantic subspaces, denoted as:

[0057] The text feature encoding module is used to convert text into word vectors. A bidirectional gated recurrent unit is used to obtain a sequence of word vectors. These are then processed by attention pooling to obtain aggregated word vector features and aggregated word vector sequence features, respectively. These are then concatenated to obtain the final representation t of the text features. feat Then, using k fully connected layers, the layer is mapped into k text semantic subspaces, denoted as:

[0058] (2) Subspace learning modules, including cross-modal subspace learning modules and same-modal subspace learning modules;

[0059] The cross-modal subspace learning module utilizes the cross-modal matching loss function L of k subspaces. cross Perform cross-modal semantic subspace consistency optimization to reduce the semantic gap between two modalities;

[0060] The same-modal subspace learning module utilizes a loss function that maximizes the semantic space difference of the video. And text that maximizes the difference in semantic space Perform semantic difference optimization within the same modality to obtain semantically distinct subspaces;

[0061] Both intramodal and crossmodal losses are applied simultaneously to the video text retrieval task, resulting in a weighted loss function L;

[0062] (3) The subspace fusion matching module fully utilizes multiple semantic subspaces to achieve mutual retrieval between videos and text by weighted fusion of similarity scores from multiple semantic spaces.

[0063] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described video text retrieval method based on multiple semantic spaces.

[0064] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described video text retrieval method based on multiple semantic spaces.

[0065] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0066] Video-to-text retrieval is essentially a retrieval between the semantics of video and text data from two different modalities. To efficiently represent the diverse semantics of video and text, this invention proposes an Adaptive Semantic Subspace Representation (ASSR) method. This method shares a multi-semantic space feature extraction network and adaptively learns and represents the semantic subspaces of different modalities by maintaining differences within the same modal subspace and consistency across different modalities. Furthermore, this invention improves retrieval performance by employing multi-semantic subspace fusion matching for video and text retrieval. Attached Figure Description

[0067] Figure 1 This is a framework diagram of the video text retrieval method based on multiple semantic spaces according to the present invention.

[0068] Figure 2 The impact of the amount of semantic space on retrieval performance. Detailed Implementation

[0069] The present invention will now be further described with reference to the accompanying drawings.

[0070] like Figure 1 As shown, the specific implementation process of the video text retrieval method based on multi-semantic space of this invention is as follows:

[0071] Step 1: Feature Encoding

[0072] ① Video feature encoding to obtain video features v feat and multiple video semantic subspaces

[0073] For videos, frames are first sampled chronologically. Then, a convolutional neural network (CNN) model pre-trained on the ImageNet dataset is used to extract a corresponding feature vector for each sampled image. Each extracted feature vector is 2048-dimensional. The two extracted feature vectors are then concatenated into a 4096-dimensional feature vector, thus completing the extraction of video frame-level features. The CNN models include ResNet-101 and ResNet-152, which can use a concat operation to concatenate two 2048-dimensional vectors. For example, given A images, ResNet-101 and ResNet-152 are used to extract a 2048-dimensional feature vector for each image, and the concat operation is used to concatenate them into a 4096-dimensional vector, resulting in A 4096-dimensional feature vectors for A images.

[0074] The video frame-level features obtained from the pre-trained model are represented as a feature sequence {v1, v2, ..., v...} n}, where v i Let represent the feature vector of the i-th frame in an n-frame video. Then, attention pooling is used to further aggregate the video feature vectors into a fixed-length feature set. The formula for attention pooling is as follows:

[0075]

[0076]

[0077] Among them, v f It is a video frame-level aggregation feature, γ i This represents the frame-level aggregation weight of the i-th video frame. This represents the video frame-level attention weights, and ReLU is the activation function.

[0078] Furthermore, the video frame-level feature sequence {v1,v2,...,v... n The sequence features of the video can be extracted using a bidirectional gated recurrent unit (biGRU), and the hidden state of the bidirectional gated recurrent unit at time step j is... Represented as:

[0079]

[0080]

[0081]

[0082] Wherein, Concat represents vector concatenation; and These represent the forward and reverse gated loop units, respectively. and Let represent the hidden states of the forward and reverse GRUs at time step j, respectively.

[0083] The hidden state sequence output by the bidirectional gated recurrent unit is denoted as... Similar to video frame-level feature vector aggregation, attention pooling is used to aggregate H... v Pooling is performed as a fixed-length feature, as shown below:

[0084]

[0085]

[0086] Among them, v s It is a video sequence aggregation feature, β i This represents the aggregate weight of the i-th frame video sequence. This represents the attention weights of the video sequence.

[0087] Finally, the frame-level aggregation features v of the video are... f With sequence aggregation feature v s The final feature representation v of the video is obtained by splicing. frat v feat =Concat(v f ,v s After obtaining the final feature representation of the video, k fully connected layers are used to map the video features into k video semantic subspaces, denoted as: The representation of the i-th semantic subspace Represented as:

[0088]

[0089] in, This represents the i-th fully connected layer.

[0090] ② Text feature encoding to obtain text features t feat and multiple text semantic subspaces

[0091] For text features, this invention obtains a pre-trained word vector matrix by pre-training an existing dual encoding (DE) model. Words in the text are converted into word vectors through the pre-trained word vector matrix, thus transforming the text into word vectors, denoted as: {t1,t2,...,t...} m}, where ti This represents the word vector of the i-th word in the text. The text feature encoding is identical to the video, except that the resulting word vectors differ from those in the video. Word vectors are processed through a bidirectional gated recurrent unit to obtain a sequence of word vectors. These are then processed through attention pooling to obtain aggregated word vector features and aggregated word vector sequence features, respectively. The aggregated word vector features and the aggregated word vector sequence features are concatenated to obtain the final text feature representation t. feat Then, k fully connected layers are used to map k text semantic subspaces, denoted as: The i-th text semantic subspace is represented as

[0092] Step 2: Subspace Learning

[0093] Feature encoding maps both video and text to k semantic subspaces, denoted as: and To enable content from multiple semantic spaces to express video or text from multiple semantic dimensions, this invention reduces the semantic gap between modalities through cross-modal semantic space learning, while maintaining the differences between semantic subspaces within the same modality. Compared to using manually defined and represented multi-semantic space models, this invention is more flexible, as the semantic matching process for multiple semantic subspaces can be handled by neural network learning.

[0094] ① Cross-modal subspace learning

[0095] This invention utilizes a loss function for cross-modal subspace learning of video and text to mine positive and negative samples through cross-modal matching of k subspaces. Specifically, given a vector of data from a certain modality ( or For a given modality, the distance between data vectors (positive samples) from another modality should be closer in the common space, while dissimilar vectors (negative samples) should be farther apart. The cosine similarity function calculates the distance between vectors from different modalities, making similar cross-modal samples closer and dissimilar cross-modal samples farther apart, resulting in the loss function for cross-modal matching within a single semantic subspace:

[0096]

[0097] in, This represents the loss function for matching the i-th semantic subspace, and D represents the size of the corpus dictionary; and They represent the video respectively. Relevant and irrelevant texts are also referred to as videos. Positive and negative samples in the i-th semantic subspace; and Respectively represent the text Relevant and irrelevant videos are also referred to as text. The positive and negative samples of the i-th semantic subspace; S represents the cosine similarity function; α represents the distance constant.

[0098] This leads to the k-subspace cross-modal matching loss function L. cross , is represented as:

[0099]

[0100] ② Same-modal subspace learning

[0101] To achieve the difference in semantic subspaces within the same modality, the KL divergence metric is used to measure the difference in probability distributions. Specifically, the semantic subspaces of video and text are first transformed into probability distributions. For example, the probability distribution of the i-th semantic subspace of a video can be represented as: The Softmax function can transform the values ​​of a vector into a probability distribution; similarly, it can be used to obtain the probability distribution of the i-th semantic subspace of a text.

[0102] Because all semantic spaces within the same modality need to be pairwise distinct, for multiple semantic subspaces distributed within the same modality, it is necessary to calculate the KL divergence pairwise and maximize the sum of the differences. Therefore, the loss function for maximizing semantic space differences in a video is... for:

[0103]

[0104] Similarly, we can obtain the loss function that maximizes the semantic space difference of the text.

[0105]

[0106] Video and text utilize the aforementioned cross-modal matching loss function L cross (L cross Constrained video-text retrieval neural networks (CNNs) can optimize cross-modal semantic subspace consistency to reduce the semantic gap between two modalities by utilizing the aforementioned loss function. and Optimizing for semantic differences within the same modality yields semantically distinct subspaces. Both same-modality and cross-modality losses are applied simultaneously to the video text retrieval task, resulting in a weighted loss function L:

[0107]

[0108] Among them, w crossThis represents the weights for cross-modal semantic subspace consistency optimization, w same This represents the weights for optimizing the differences in the same-modal semantic subspace. The weighted loss function L is used to measure the importance of cross-modal loss and same-modal loss.

[0109] Step 3: Subspace Fusion and Matching

[0110] Video-to-text retrieval is essentially a semantic search between video and text. While the subspace learning and matching operations described above allow for the direct selection of a set of subspaces for matching videos and text, choosing the specific subspace to use as the final retrieval space is difficult. Therefore, since we don't know how to select the best set of subspaces, why not use all subspaces for matching? Ultimately, by weighted fusion of similarity scores from multiple semantic subspaces, we can fully utilize these subspaces to achieve mutual retrieval between videos and text. The similarity score fusion is represented as follows:

[0111]

[0112] Where, μ i This represents the weight of similarity in a certain semantic subspace, and satisfies... S represents the cosine similarity function. It's important to note here that when dealing with the similarity matrix... Before calculating the weighted sum, the similarity matrix needs to be normalized to avoid the similarity scores of a certain subspace being too large or too small, which would affect the fusion effect.

[0113] The weighted fusion process first calculates the similarity between the i-th video semantic subspace and the i-th text semantic subspace. In other words, it refers to the degree of matching between the i-th video semantic subspace and the i-th text semantic subspace, and assigns different weights (μ) to different matching subspaces. i Finally, similarity scores (Sim(s)) are obtained across multiple semantic spaces. v ,s t This enables mutual retrieval between the semantics of video and text data modalities.

[0114] Based on the above principles and processes, this invention proposes a video text retrieval system based on adaptive multi-semantic space representation, comprising the following:

[0115] (1) Feature coding module, including video feature coding module and text feature coding module;

[0116] The video feature encoding module is used to obtain the feature vector of each frame of video, extract the sequence features of the video through a bidirectional gated recurrent unit, and use attention pooling to obtain the frame-level aggregated features v of the video. fWith sequence aggregation feature v s The final feature representation v of the video is obtained by splicing. feat Then, k fully connected layers are used to map the video features into k video semantic subspaces, denoted as:

[0117] The text feature encoding module is used to convert text into word vectors. A bidirectional gated recurrent unit is used to obtain a sequence of word vectors. These are then processed by attention pooling to obtain aggregated word vector features and aggregated word vector sequence features, respectively. These are then concatenated to obtain the final representation t of the text features. feat Then, using k fully connected layers, the layer is mapped into k text semantic subspaces, denoted as:

[0118] (2) Subspace learning modules, including cross-modal subspace learning modules and same-modal subspace learning modules;

[0119] The cross-modal subspace learning module utilizes the cross-modal matching loss function L of k subspaces. cross Perform cross-modal semantic subspace consistency optimization to reduce the semantic gap between two modalities;

[0120] The same-modal subspace learning module utilizes a loss function that maximizes the semantic space difference of the video. And text that maximizes the difference in semantic space Perform semantic difference optimization within the same modality to obtain semantically distinct subspaces;

[0121] Both intramodal and crossmodal losses are applied simultaneously to the video text retrieval task, resulting in a weighted loss function L;

[0122] (3) The subspace fusion matching module fully utilizes multiple semantic subspaces to achieve mutual retrieval between videos and text by weighted fusion of similarity scores from multiple semantic spaces.

[0123] Based on the above principles and processes, this invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The device is characterized in that the processor executes the computer program to implement the above-mentioned video text retrieval method based on multiple semantic spaces.

[0124] Based on the above principles and processes, this invention proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the aforementioned video text retrieval method based on multiple semantic spaces.

[0125] Example 1

[0126] Validate the performance of the ASSR method on the MSR-VTT dataset. Compare the performance of the ASSR method with classic video text retrieval models MEE, CE, TCE, W2V++, HGR, DE, and SFEM. Perform video text retrieval using the ASSR method:

[0127] Dataset partitioning: The ASSR method uses three different partition sets on the MSR-VTT dataset: Official, Test1k-Miech, and Test1k-Yu.

[0128] ASSR method parameter configuration: For the Official and Test1k-Miech partition sets of the MSR-VTT data, the number of common subspaces in this invention is set to 2, and the dimension of each semantic space is set to 1024. The α parameter in the loss function is set to 0.1, and the dimension of the hidden state of the gated recurrent unit module is set to 512 (the dimension of the hidden state of the bidirectional gated recurrent unit is 1024). The weights w in the cross-modal loss function... cross The weight w of the same-modal loss function is set to 0.5. same Set to 5.0. For the Test1k-Miech partition set, the hyperparameters that differ from the other two partition sets include the dimension of each semantic space and the α parameter in the loss function, which are set to 2048 and 0.15 respectively. Other parameter settings are the same as the other two partition sets.

[0129] Table 1 ASSR method Performance comparison with the comparative algorithm on three different split datasets of MSR-VTT.

[0130]

[0131] The results in Table 1 show that the ASSR method outperforms all compared algorithms in overall retrieval performance across the three different partition sets. The ASSR method's encoding module uses only simple bidirectional gated recurrent units and attention pooling, making it much simpler than the DE and SFEM models in the existing comparison algorithms. However, ASSR achieves better retrieval performance than more complex models. This indicates that for cross-modal retrieval tasks, multi-subspace representation and fusion retrieval of video and text can improve retrieval performance. Therefore, for cross-modal retrieval tasks, effective subspace generation does not necessarily require a very complex feature encoder to represent video and text. Even with a relatively simple encoding structure, appropriate use of subspace learning and fusion methods can improve the expressive power of cross-modal subspaces, thereby improving cross-modal retrieval performance.

[0132] Example 2

[0133] The performance of the ASSR method was validated on the TGIF dataset. In the TGIF dataset, a GIF has only one corresponding text description in the training and validation sets, while a GIF has three corresponding text descriptions in the test set, making it more challenging than MSR-VTT. The chosen comparison algorithms included Order, DeViSE, VSE++, Corr-AE, PVSE, DE, and SFEM. The parameter configuration of the ASSR method on the TGIF dataset was the same as that on the Official and Test1k-Miech partition sets of the MSR-VTT dataset.

[0134] Table 2 Performance comparison of the ASSR method with the comparison algorithms on the TGIF dataset.

[0135]

[0136] The results in Table 2 show that the ASSR method significantly outperforms DE on the TGIF dataset. While ASSR has a much simpler structure than SFEM, its superior subspace representation capabilities allow it to achieve powerful retrieval performance. Furthermore, in the TGIF training and validation data, each GIF corresponds to only one text description, while the test set has three text descriptions per GIF. This demonstrates that the ASSR method can improve retrieval performance even with insufficient training data by fully utilizing the representation capabilities of the subspace. From the design philosophy of the ASSR method, the representation of the cross-modal common space and its optimization methods directly affect its retrieval performance. Moreover, when the encoding capability is sufficiently strong, further exploiting the representation of the common space can further enhance the retrieval performance of the ASSR method, essentially using a more powerful feature extractor based on the ASSR concept.

[0137] Example 3

[0138] Validating the ASSR method performance on the MSVD dataset. Since many contrastive algorithms experimented on the MSVD dataset have uncertain text sampling in the dataset, to fairly compare performance differences between algorithms, the text descriptions in MSVD were not sampled. Instead, the open-source DE and SFEM code was rerun using the full text descriptions. ASSR and the contrastive algorithms (DE and SFEM) used the exact same dataset and features, and the performance of the three algorithms was compared.

[0139] Table 3 Performance comparison of the ASSR method with the comparative algorithms on the MSVD dataset.

[0140]

[0141] As shown in Table 3, the ASSR method outperforms the DE model in most metrics on the MSVD dataset. However, for SFEM, the improvement in text-to-video retrieval is more significant than that in video-to-text retrieval. Specifically, for text-to-video retrieval, ASSR improves upon DE by 0.5%, 2.4%, 1.8%, and 16.7% in R@1, R@5, R@10, and MedR metrics, respectively, and by 0.9%, 2.0%, 2.1%, and 16.7% compared to SFEM. For video-to-text retrieval, ASSR improves upon DE by 12.4%, 2.9%, 1.0%, and 20% in R@1, R@5, R@10, and MedR metrics, respectively. However, compared to SFEM, it only improves by 1.1% in R@10. ASSR improves upon both DE and SFEM by 1.8% and 1.9% in the total R@K for text-to-video retrieval, respectively. Compared to DE, ASSR improved the total R@K of video-to-text retrieval by 3.9%, but showed no improvement compared to ASSR. ASSR also improved the overall SumR (sum of revenue) for video and text retrieval by 2.9% and 0.5% respectively compared to DE.

[0142] Table 4 shows the experimental results of the same-modal loss function.

[0143]

[0144] Example 4

[0145] To demonstrate the effectiveness of the semantic subspace and feature extraction designed in this invention in video text retrieval, we discuss the use of the same-modal loss function and the impact of the number of subspaces on the performance of the ASSR method on the MSR-VTT Official partition set.

[0146] Table 4 presents the experimental results with and without using the same-modal loss function. The results show that using the same-modal loss function significantly improves the performance of both text-to-video and video-to-text retrieval. This indicates that the same-modal loss function introduces differences between multiple subspaces within the same modality, facilitating the fusion of multiple subspaces and thus improving cross-modal retrieval performance. Furthermore, the results without using the same-modal loss function show that the overall performance index SumR is better than that of the DE algorithm compared in Table 1. This demonstrates that the fusion matching of multiple semantic spaces itself can improve the retrieval performance of the ASSR method, and adding the same-modal loss function can further enhance retrieval performance on this basis. The experimental results on the impact of the number of subspaces on the performance of the ASSR method are as follows: Figure 2As shown in the figure, the impact of the number of subspaces on performance is not clearly predictable; more is not necessarily better, nor is less. In the current experiment, using two semantic spaces per modality yielded the best results. The reason why the impact of the number of subspaces on performance is not clearly predictable is because the number of subspaces is affected by the structure and parameters of the ASSR method. Therefore, to obtain better experimental results through subspace fusion, multiple parameter tuning experiments are needed, taking into account various factors.

[0147] In summary, to eliminate the need for manual selection and mining of multi-semantic representations in videos and texts, this invention proposes a video-text retrieval method based on adaptive multi-semantic subspace representation. The adaptive multi-semantic subspace representation (ASSR) method requires only a relatively simple structure to achieve good retrieval results, demonstrating the effectiveness of multi-semantic subspace representation. We verified the impact of the number of subspaces on the retrieval performance of this invention, as well as the impact of using a loss function optimized for differences within the same modality subspace. This invention features consistency between different modality subspaces while maintaining differences within subspaces of the same modality.

[0148] Although the functions and working processes of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific functions and working processes described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these are within the protection scope of the present invention.

Claims

1. A method for video text retrieval based on multi-semantic space, characterized in that, The process includes the following: Step 1: Feature Encoding ① video feature encoding, obtaining video features and a plurality of video semantic subspaces First, frames in the video are sampled in chronological order. Then, a CNN model pre-trained on the ImageNet dataset is used to extract a corresponding feature vector for each sampled image. Each extracted feature vector is 2048-dimensional. Finally, two extracted feature vectors are concatenated into a 4096-dimensional feature vector. The resulting video frame-level feature representation is a feature sequence. ,in express n The first frame of the video The feature vectors of each video frame are then used to further obtain frame-level aggregated features using attention pooling. ; Video frame-level feature sequences The video sequence features are extracted using a bidirectional gated recurrent unit (BRU). The hidden state sequence output by the BRU is denoted as... ,in Indicates the bidirectional gated loop unit number The hidden states at each time step are then used, and attention pooling is used to further obtain the aggregated features of the video sequence. ; Aggregate frame-level features of video With sequence aggregation features The final feature representation of the video is obtained by splicing. ;use Each fully connected layer maps video features to... A video semantic subspace, denoted as: ; ②text feature encoding, obtaining text features and a plurality of text semantic subspaces By pre-training an existing dual-encoding model, a pre-trained word vector matrix is ​​obtained. Words in the text are then converted into word vectors using this pre-trained matrix, thus transforming the text into word vectors, denoted as: Word vectors are processed through a bidirectional gated recurrent unit to obtain a word vector sequence. These two sequences are then processed through attention pooling to obtain word vector aggregation features and word vector sequence aggregation features, respectively. The final text feature representation is obtained by concatenating the word vector aggregation features and word vector sequence aggregation features. Then use Each fully connected layer is mapped to A text semantic subspace, denoted as: ; Step 2: Subspace Learning ① Cross-modal subspace learning The cosine similarity function calculates the distance between vectors of different modalities, making similar cross-modal samples closer together and dissimilar cross-modal samples farther apart, thus deriving the loss function for cross-modal matching in a single semantic subspace, and further obtaining... Subspace cross-modal matching loss function Video and text cross-modal subspace learning through k Loss function for cross-modal matching in subspaces to mine positive and negative samples; ② Same-modal subspace learning The semantic subspaces of video and text are transformed into probability distributions respectively. For multiple semantic subspace distributions of the same modality, probability distributions are calculated pairwise. KL By maximizing the sum of the divergences, we obtain the loss function that maximizes the semantic space difference in the video. Loss function that maximizes the semantic space difference of text ; Video and text utilize the aforementioned cross-modal matching loss function Cross-modal semantic subspace consistency optimization is performed to reduce the semantic gap between two modalities, utilizing the aforementioned loss function. and By performing intramodal semantic difference optimization, semantically distinct subspaces are obtained; intramodal and cross-modal losses are applied simultaneously to the video text retrieval task, resulting in a weighted loss function. ; Step 3: Subspace Fusion and Matching By weighted fusion of similarity scores from multiple semantic subspaces, the mutual retrieval between videos and text can be fully utilized. 2.The method of claim 1, wherein, The video frame-level aggregation features mentioned in the first step : ， ， in, It is a video frame-level aggregation feature. Indicates the first Frame-level aggregated weights for video frames. This represents the video frame-level attention weights. ReLU It is an activation function; bidirectional gated recurrent unit hidden state of the time step is represented as: ， ， ， in, This indicates vector concatenation; and These represent the forward and reverse gated loop units, respectively. and These represent the forward and reverse GRUs at the 1st and 2nd respectively. The hidden state of the time step; Video sequence aggregation features is represented as: ， ， wherein, is a video sequence aggregate feature, denotes a frame video sequence aggregate weight, denotes a video sequence attention weight; Final feature representation of the video as follows: 。 3. The video text retrieval method based on multiple semantic spaces according to claim 1, characterized in that, The loss function for cross-modal matching of a single semantic subspace as described in step two: ， in, It indicates the first The loss function for matching semantic subspaces. D This indicates the size of the corpus dictionary; and They represent the video respectively. Relevant and irrelevant texts are also referred to as videos. No. Positive and negative samples in a semantic subspace; and Respectively represent the text Relevant and irrelevant videos are also referred to as text. No. Positive and negative samples in a semantic subspace; This represents the cosine similarity function; Indicates a distance constant; In the second step, the The subspace cross-modal matching loss function is represented as: 。 4.The method of claim 1, wherein, The second step in the video The probability distribution of each semantic subspace is represented as follows: ,in The function transforms the values ​​of a vector into a probability distribution; similarly, it obtains the probability distribution of the text. Probability distribution of semantic subspaces . 5.The method of claim 1, wherein, loss function that maximizes semantic space difference of the video in the second step is: ， loss function maximizing semantic space difference of the text in the second step is: ， the weighted loss function in the second step is represented as: ， wherein, represents the weight of the cross-modal semantic subspace consistency optimization, represents the weight of the intra-modal semantic subspace difference optimization.

6. The video text retrieval method based on multiple semantic spaces according to claim 1, characterized in that, The similarity score fusion representation in the third step is as follows: ， in, This represents the weight of similarity in a certain semantic subspace, and satisfies... , Represents the cosine similarity function; in the similarity matrix Before calculating the weighted sum, the similarity matrix is ​​normalized.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video text retrieval method based on multiple semantic spaces as described in any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the video text retrieval method based on multiple semantic spaces as described in any one of claims 1 to 6.

Images