Text-based video retrieval method and device, equipment and storage medium

Abstract

The invention discloses a text-based video retrieval method and device, equipment and a storage medium, and the method comprises the steps: obtaining text content used for retrieving videos, constructing sentence features and word features of the text content, carrying out video coarse screening based on the text content, obtaining a plurality of candidate videos, constructing video features and frame features of each candidate video, and carrying out video retrieval based on the video features and the frame features. And calculating video sentence similarity, video word similarity, frame sentence similarity and frame word similarity of the candidate videos, thereby calculating semantic similarity scores of the candidate videos, and determining the candidate video with the highest semantic similarity score as a retrieval output video of the text content. Therefore, by calculating the global and local multi-granularity similarity between the text and the matched video, the semantic similarity score finally used for evaluating the similarity with the text content is more accurate and reliable, so that the accuracy of video retrieval based on the text is improved.

Description

Technical Field

[0001] This application relates to the field of video retrieval technology, and more specifically, to a text-based video retrieval method, apparatus, device, and storage medium. Background Technology

[0002] In fields such as video surveillance and security, text-to-video cross-modal retrieval has become a core technical requirement for efficiently locating target videos. By inputting descriptive text, matching videos can be quickly retrieved, significantly improving the utilization efficiency of video data and meeting the demands for accurate and efficient retrieval in real-world scenarios, thus possessing significant practical value.

[0003] Existing text-video retrieval technologies have developed various implementation schemes, such as multi-scale retrieval methods based on the Mamba model, fine-grained cross-modal alignment models, retrieval schemes enhanced with spatiotemporal relationships, and semantically driven retrieval systems based on large language models. However, in terms of the targeting of visual feature extraction and cross-modal similarity calculation, the need for multi-granular semantic alignment between global and local aspects has not been fully considered, resulting in low accuracy of similarity calculation and low precision of retrieval results.

[0004] Improving the accuracy of text-based video retrieval is an issue that needs attention. Summary of the Invention

[0005] In view of the above problems, this application provides a text-based video retrieval method, apparatus, device and storage medium to improve the accuracy of text-based video retrieval.

[0006] To achieve the above objectives, the following specific solutions are proposed:

[0007] A text-based video retrieval method includes:

[0008] Obtain the text content used for video retrieval, and construct sentence features and word features of the text content;

[0009] Based on the text content, a coarse screening of videos is performed to obtain multiple candidate videos, and video features and frame features of each candidate video are constructed.

[0010] For each candidate video, calculate the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features.

[0011] For each candidate video, a semantic similarity score is calculated based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity of the candidate video.

[0012] The candidate video with the highest semantic similarity score is determined as the retrieved video of the text content.

[0013] Optionally, a semantic similarity score is calculated for the candidate videos based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity, including:

[0014] The video word similarity, the frame sentence similarity, and the frame word similarity are scalarized to obtain scalarized video word similarity, scalarized frame sentence similarity, and scalarized frame word similarity.

[0015] The similarity vector is obtained by concatenating the video word similarity, the frame sentence similarity, the frame word similarity, and the video sentence similarity.

[0016] Calculate the granularity aggregation weight vector based on the similarity vector;

[0017] The semantic similarity score of the candidate video is obtained by weighting and summing the similarity vector using the granular aggregation weight vector.

[0018] Optionally, the method further includes:

[0019] The text content is input into a pre-trained text video retrieval model, which outputs the retrieved video results.

[0020] The training process of the text video retrieval model includes:

[0021] Obtain training samples, which include multiple text data and multiple video data;

[0022] Construct sentence embedding features and word embedding features for each text content, and construct video embedding features and frame embedding features for each video data;

[0023] For each video data sentence embedding feature and word embedding feature, and for each text data video embedding feature and frame embedding feature, calculate the video sentence similarity scalar between the video embedding feature and the sentence embedding feature, the video word similarity vector between the video embedding feature and the word embedding feature, the frame sentence similarity vector between the frame embedding feature and the sentence embedding feature, and the frame word similarity vector between the frame embedding feature and the word embedding feature.

[0024] For each video data and each text data, a multi-granularity aggregated similarity score of the text data to the video data is calculated based on the video sentence similarity scalar, the video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector between the video data and the text data.

[0025] The multi-granularity aggregated similarity score of each text data with respect to each video data is used as the input to the loss function of the text video retrieval basic model. The first loss value of the text video retrieval basic model for training with respect to the text data and the video data is calculated, and the model parameters of the text video retrieval basic model are optimized based on the first loss value until the text video retrieval model is obtained after training with all text data and all video data.

[0026] Optionally, based on the video sentence similarity scalar, the video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector between the video data and the text data, a multi-granularity aggregated similarity score of the text data to the video data is calculated, including:

[0027] The video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector are scalarized respectively to obtain scalarized video word similarity scalar, scalarized frame sentence similarity scalar, and scalarized frame word similarity scalar.

[0028] The video word similarity scalar, the frame sentence similarity scalar, the frame word similarity scalar, and the video sentence similarity scalar are concatenated to obtain a similarity concatenation vector;

[0029] Calculate the similarity weight vector based on the similarity concatenation vector;

[0030] The text data is aggregated with the video data by weighting and summing the similarity vectors using the similarity weights.

[0031] Optionally, based on the similarity concatenation vector, a similarity weight vector is calculated, including:

[0032] The similarity weight vector is obtained by performing linear Softmax aggregation on the concatenated similarity vector using the following formula:

[0033]

[0034] in, For linear layers, initialized as a 4x4 identity matrix. Let be the similarity weight vector. For the i-th text data For the i-th video data Multi-granularity similarity vectors The similarity weight vector has a dimension of 1*4.

[0035] Optionally, the method further includes:

[0036] Using the multi-granularity similarity vector of each text data point to each video data point as input to the multi-granularity similarity balance loss function of the text-video retrieval base model, a second loss value is calculated for training the text-video retrieval base model on the given text data and video data. The model parameters of the text-video retrieval base model are then optimized based on this second loss value. The multi-granularity similarity balance loss function is:

[0037]

[0038] in, This is the second loss value. For sample batch size, For the i-th text data For the i-th video data A multi-granularity similarity vector.

[0039] A text-based video retrieval device includes:

[0040] A text-level feature construction unit is used to acquire text content for video retrieval and construct sentence features and word features of the text content;

[0041] A video-level feature construction unit is used to perform coarse screening of videos based on the text content, obtain multiple candidate videos, and construct video features and frame features for each candidate video.

[0042] A multi-granularity similarity calculation unit is used to calculate, for each candidate video, the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features.

[0043] The semantic similarity score calculation unit is used to calculate the semantic similarity score of each candidate video based on the video sentence similarity, the video word similarity, the frame sentence similarity, and the frame word similarity.

[0044] The video retrieval unit is used to determine the candidate video with the highest semantic similarity score as the retrieval output video of the text content.

[0045] Optionally, the semantic similarity score calculation unit includes:

[0046] The scalarization unit is used to perform scalarization processing on the video word similarity, the frame sentence similarity, and the frame word similarity respectively, to obtain scalarized video word similarity, scalarized frame sentence similarity, and scalarized frame word similarity;

[0047] A scalar concatenation unit is used to concatenate the video word similarity, the frame sentence similarity, the frame word similarity, and the video sentence similarity to obtain a similarity vector;

[0048] The weight vector calculation unit is used to calculate the granular aggregated weight vector based on the similarity vector.

[0049] The weighted summation unit is used to sum the similarity vector by the granular aggregated weight vector to obtain the semantic similarity score of the candidate video.

[0050] A text-based video retrieval device, including a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is used to execute the program to implement the various steps of the text-based video retrieval method described above.

[0053] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the text-based video retrieval method described above.

[0054] By employing the above technical solution, this application obtains the text content used for video retrieval, constructs sentence features and word features of the text content, performs coarse screening of videos based on the text content to obtain multiple candidate videos, constructs video features and frame features for each candidate video, calculates the video-sentence similarity between the candidate video's video features and sentence features, the video-word similarity between the candidate video's video features and word features, the frame-sentence similarity between the candidate video's frame features and sentence features, and the frame-word similarity between the candidate video's frame features and word features for each candidate video, calculates the semantic similarity score of the candidate video based on the video-sentence similarity, video-word similarity, frame-sentence similarity, and frame-word similarity, and determines the candidate video with the highest semantic similarity score as the retrieval output video for the text content. Therefore, by calculating global and local multi-granularity similarity between the text and the matched video, the final semantic similarity score used to evaluate the similarity with the text content becomes more accurate and reliable, thereby improving the accuracy of text-based video retrieval. Attached Figure Description

[0055] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0056] Figure 1 A schematic diagram illustrating a text-based video retrieval process provided in this application embodiment;

[0057] Figure 2 A schematic diagram illustrating a process for calculating semantic similarity scores, provided for an embodiment of this application;

[0058] Figure 3 A schematic diagram of a device structure for implementing text-based video retrieval, provided as an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the structure of a device for implementing text-based video retrieval, provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a computer, cloud, or server.

[0062] Next, combined Figure 1 The text-based video retrieval method of this application may include the following steps:

[0063] Step S110: Obtain the text content used for video retrieval, and construct sentence features and word features of the text content.

[0064] Understandably, by acquiring the content of the retrieved text and constructing sentence and word features, a multi-dimensional and accurate representation of the text's semantics can be directly achieved. Sentence features capture the global semantics of the text, fully conveying the overall context and core intent of the retrieval request, avoiding the loss of the overall meaning of the text due to focusing only on local aspects. Word features, on the other hand, can focus on the specific semantics of individual words in the text, refining key information in the retrieval request, such as details about the described object and attributes. The combination of the two can cover the global and local multi-granular semantics of the text, avoiding one-sided or coarse semantic representation, ensuring that the retrieval intent of the text is comprehensively and accurately transformed into a feature form that can be used for subsequent calculations, thereby improving the completeness and accuracy of the text's semantic expression from the source.

[0065] Step S120: Perform coarse screening of videos based on text content to obtain multiple candidate videos, and construct video features and frame features for each candidate video.

[0066] Understandably, using text-based coarse screening of videos can directly narrow down the search scope, avoid complex subsequent processing of all videos, reduce unnecessary computational resource consumption, and significantly improve search efficiency. Simultaneously, constructing video and frame features for the selected candidate videos enables multi-granular representation of their semantics. Video features carry the global semantic information of the candidate videos, while frame features capture the detailed semantics of local frames. The combination of these two ensures that the semantic information of the candidate videos is fully covered, avoiding semantic information loss due to single feature representations, and providing comprehensive feature support for subsequent accurate matching.

[0067] Step S130: For each candidate video, calculate the video sentence similarity between the video features and sentence features of the candidate video, the video word similarity between the video features and word features of the candidate video, the frame sentence similarity between the frame features and sentence features of the candidate video, and the frame word similarity between the frame features and word features of the candidate video.

[0068] Understandably, by calculating similarity across four different dimensions, the semantic relationship between text and candidate videos can be captured comprehensively, from global to local. Video sentence similarity focuses on the matching of the overall intent of the candidate video with the overall intent of the text; video word similarity relates to the fit between the overall video and the keywords of the text; frame sentence similarity connects the semantics of a local frame in the video with the overall meaning of the text; and frame word similarity refines the matching details between a local frame in the video and individual words in the text. These four similarity metrics cover different semantic matching scenarios, avoiding the one-sidedness of matching caused by a single similarity calculation, ensuring that the semantic fit between text and candidate videos is fully explored, and improving the comprehensiveness and detail of semantic relationship judgment.

[0069] Step S140: For each candidate video, calculate the semantic similarity score of the candidate video based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity.

[0070] Understandably, calculating semantic similarity scores using a combination of four similarity metrics integrates multi-dimensional matching information and balances the contributions of semantic associations at different granularities. Each of the four similarity metrics has its own emphasis; using any single similarity metric alone cannot fully reflect the true fit between the text and the candidate video. However, by comprehensively calculating the similarity score, the bias of any one dimension can be prevented from excessively influencing the overall judgment, while simultaneously integrating effective matching information from all dimensions. This fusion approach makes the final semantic similarity score more closely reflect the actual semantic relationship between the text and the candidate video, avoiding scoring biases caused by information fragmentation and improving the accuracy and reliability of the score.

[0071] Step S150: Determine the candidate video with the highest semantic similarity score as the retrieval output video of the text content.

[0072] Understandably, identifying the candidate video with the highest semantic similarity score as the output directly pinpoints the target video that best matches the text retrieval needs. The highest semantic similarity score indicates that the candidate video performs optimally in both global and local semantic matching, best aligning with the retrieval intent expressed in the text. This effectively avoids search results deviating from the intended purpose due to selecting suboptimal videos. This filtering method ensures that the retrieval output accurately responds to the core requirements of the input text, allowing users to quickly obtain videos that meet their expectations. This directly improves the accuracy and effectiveness of search results, satisfying the core needs of text-to-video retrieval scenarios.

[0073] The text-based video retrieval method provided in this embodiment obtains the text content used for video retrieval, constructs sentence and word features of the text content, performs coarse screening of videos based on the text content to obtain multiple candidate videos, constructs video features and frame features for each candidate video, calculates the video-sentence similarity between the candidate video's video features and sentence features, the video-word similarity between the candidate video's video features and word features, the frame-sentence similarity between the candidate video's frame features and sentence features, and the frame-word similarity between the candidate video's frame features and word features for each candidate video, calculates the semantic similarity score of the candidate video based on the video-sentence similarity, video-word similarity, frame-sentence similarity, and frame-word similarity, and determines the candidate video with the highest semantic similarity score as the retrieval output video for the text content. Therefore, by calculating global and local multi-granularity similarity between the text and the matched video, the final semantic similarity score used to evaluate the similarity with the text content is more accurate and reliable, thereby improving the accuracy of text-based video retrieval.

[0074] In some embodiments of this application, the process of calculating the semantic similarity score of candidate videos based on video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity, as mentioned in the above embodiments, is described. (Refer to...) Figure 2 The process may include:

[0075] Step S210: Perform scalarization processing on the video word similarity, frame sentence similarity, and frame word similarity respectively to obtain scalarized video word similarity, scalarized frame sentence similarity, and scalarized frame word similarity.

[0076] Video sentence similarity can be represented as the dot product similarity between sentence-conditionalized video features and sentence features. It is a 1-dimensional scalar, i.e., a single numerical value. The video features and sentence features are d-dimensional vectors, where d is the feature dimension. Video word similarity can be represented as the vector calculated by the matrix multiplication of sentence-conditionalized video features and word features, with a shape of [1, m], where m is the number of words, and word features are [m, d] vectors. Frame sentence similarity can be represented as the similarity vector calculated by the matrix multiplication of word-conditionalized frame features and sentence features, with a shape of [m, 1], and sentence features are [m, d] vectors. Frame word similarity can be represented as the similarity vector calculated by the matrix multiplication of word-conditionalized frame features and word features, with a shape of [m, m].

[0077] Specifically, since video word similarity, frame sentence similarity, and frame word similarity are not 1-dimensional scalars, they all need to be scalarized. The scalarization process for video word similarity can be achieved by taking the mean or maximum value of a 1×m vector, compressing it into a 1-dimensional scalar; the scalarization process for frame sentence similarity can be achieved by taking the mean or maximum value of an m×1 vector, compressing it into a 1-dimensional scalar; and the scalarization process for frame word similarity can be achieved by extracting key similarities (e.g., retaining the top 10% significant values) from an m×m matrix through similarity recombination, then taking the mean or maximum value, compressing it into a 1-dimensional scalar.

[0078] Step S220: Concatenate the video word similarity, frame sentence similarity, frame word similarity, and video sentence similarity to obtain a similarity vector.

[0079] Specifically, the four similarity scalars—video word similarity, frame sentence similarity, frame word similarity, and video sentence similarity—can be concatenated in sequence to form a 1×4 similarity vector. .here This represents the i-th candidate video. Represents text content.

[0080] Step S230: Calculate the granularity aggregation weight vector based on the similarity vector.

[0081] Specifically, the granularity of the aggregated weight vector can be calculated using linear softmax aggregation:

[0082]

[0083] in, It is a linear layer, and its matrix is ​​4x4. It is a 1×4 vector.

[0084] Step S240: The similarity vector is weighted and summed by granular aggregation weight vector to obtain the semantic similarity score of the candidate video.

[0085] Specifically, the semantic similarity score of candidate videos can be obtained by weighted summation using the following formula:

[0086]

[0087] in, For text content For candidate videos semantic similarity score, Represents the granularity aggregation weight vector The kth element, Represents a similarity vector The kth element.

[0088] In some embodiments of this application, another text-based video retrieval method is provided, which specifically involves inputting text content into a pre-trained text video retrieval model and outputting retrieval video results.

[0089] The training process for the text video retrieval model may include:

[0090] S1. Obtain training samples.

[0091] The training samples can include multiple text data points and multiple video data points. Text and video data can be paired to form multiple text-video data pairs. Understandably, the subsequent model training process mainly focuses on training the similarity judgment of these text-video data pairs.

[0092] S2. Construct sentence embedding features and word embedding features for each text content, and construct video embedding features and frame embedding features for each video data.

[0093] Among them, video embedding features can be represented as Let represent the video embedding feature of the i-th text data with respect to the i-th video data, which is a sentence-conditional video embedding feature. The frame embedding feature can be represented as... Let represent the frame embedding feature of the i-th text data with respect to the i-th video data, which is a word-conditional frame embedding feature. The sentence embedding feature can be represented as... Word embedding features can be represented as .

[0094] Sentence-level embedding features can be obtained from the global sentence embeddings extracted by the CLIP text encoder. Word-level embedding features can be obtained from the embeddings of all word tokens output by the CLIP text encoder. Frame-level features can be obtained from the [CLS] token embeddings extracted from each frame by the CLIP video encoder, enhanced by a temporal encoder (e.g., dimension d=512). Video-level features can be generated by aggregating frame-level features using a language-video attention module.

[0095] More specifically, , Where LN (layer normalization) represents the layer normalization function, and FC (fully connected layer) represents the fully connected layer function. It's weight. It is a language-video attention module, and its calculation method is as follows:

[0096]

[0097] in, Three projection matrices , and The dimensions of all are [d, dp]. The dimension is [dp, d], where dp represents the projection dimension. For the query, the dimension is [1, dp]. The key is [n, dp], and the dimension is [n, dp], where n represents the number of video frames. The value is [n, dp]. It's a video. The final frame-level embedding features have dimensions [n,d].

[0098] , This represents the Transformer Encoder, where P (PositionEmbedding) represents the position embedding. Indicates from video The frame-level embedding features are extracted from independent frames, which do not consider the temporal information of the video. The dimension is [n,d]. It uses positional embedding The temporal encoder is used to model temporal relationships. It is a 3-layer Transformer encoder that includes a multi-head self-attention mechanism with 8 attention heads and a feedforward network. The dimensions of query, key, and value features are all set to 512.

[0099] S3. For the sentence embedding features and word embedding features of each video data, and the video embedding features and frame embedding features of each text data, calculate the video sentence similarity scalar between the video embedding features and the sentence embedding features, the video word similarity vector between the video embedding features and the word embedding features, the frame sentence similarity vector between the frame embedding features and the sentence embedding features, and the frame word similarity vector between the frame embedding features and the word embedding features.

[0100] Among them, the video sentence similarity scalar is The video word similarity vector is The sentence similarity vector of the frames is The word similarity vector of the frame is .

[0101] S4. For each video data and each text data, calculate the multi-granularity aggregated similarity score of the text data to the video data based on the video sentence similarity scalar, video word similarity vector, frame sentence similarity vector, and frame word similarity vector between the video data and the text data.

[0102] Specifically, the process of calculating the multi-granularity aggregated similarity score of text data to video data based on the video sentence similarity scalar, video word similarity vector, frame sentence similarity vector, and frame word similarity vector can include:

[0103] S41. Perform scalarization on the video word similarity vector, frame sentence similarity vector, and frame word similarity vector respectively to obtain scalarized video word similarity scalar, scalarized frame sentence similarity scalar, and scalarized frame word similarity scalar.

[0104] Specifically, the average or maximum value of the video word similarity vector (1×m vector) can be taken and compressed into a 1-dimensional scalar; the average or maximum value of the frame sentence similarity vector (m×1 vector) can be taken and compressed into a 1-dimensional scalar; the key similarity of the frame word similarity vector (m×m matrix) can be extracted through similarity recombination (SR module) (such as retaining the top 10% significant values), and then the average or maximum value can be taken and compressed into a 1-dimensional scalar.

[0105] S42. Concatenate the video word similarity scalar, the frame sentence similarity scalar, the frame word similarity scalar, and the video sentence similarity scalar to obtain a similarity concatenation vector.

[0106] Specifically, the four similarity scalars can be concatenated sequentially into a 1×4 similarity concatenation vector, denoted as .

[0107] S43. Calculate the similarity weight vector based on the similarity concatenation vector.

[0108] Specifically, the similarity concatenation vector can be linearly aggregated using the following formula to obtain the similarity weight vector:

[0109]

[0110] in, For linear layers, initialized as a 4x4 identity matrix. This is the similarity weight vector. For the i-th text data For the i-th video data Multi-granularity similarity vectors This is a similarity weight vector with a dimension of 1*4.

[0111] Understandably, through linear Softmax aggregation, similarities at different granularities are unified into a fixed-dimensional vector, which facilitates subsequent model processing.

[0112] S44. By weighting and summing the similarity vectors using similarity weights, we obtain the multi-granularity aggregated similarity score of text data for video data.

[0113] Specifically, the semantic similarity score of candidate videos can be obtained by weighted summation using the following formula:

[0114]

[0115] in, For text data For video data semantic similarity score, Represents the similarity weight vector The kth element, Represents similarity concatenation vectors The kth element.

[0116] S5. Using the multi-granularity aggregated similarity score of each text data with respect to each video data as the input of the loss function of the text video retrieval basic model, calculate the first loss value of the text video retrieval basic model for training on the text data and the video data, and optimize the model parameters of the text video retrieval basic model based on the first loss value, until the text video retrieval model is obtained after training with all text data and all video data.

[0117] Among them, the text-based video retrieval basic model is an existing model that can roughly retrieve videos based on text.

[0118] Understandably, the multi-granularity aggregated similarity score of each text data point for each video data point can be used as input to the InfoNCE loss function of the text-video retrieval base model, enabling it to distinguish between matching positive samples and non-matching negative samples in contrastive learning.

[0119] In addition, the training process of the text-video retrieval base model can also utilize a multi-granularity similarity balance loss function. Specifically, the multi-granularity similarity vector of each text data point to each video data point can be used as the input to the multi-granularity similarity balance loss function of the text-video retrieval base model. The second loss value of the text-video retrieval base model trained on the text data point and the video data point can be calculated, and the model parameters of the text-video retrieval base model can be optimized based on the second loss value.

[0120] The multi-granularity similarity balance loss function is:

[0121]

[0122] in, This is the second loss value. For sample batch size, For the i-th text data For the i-th video data A multi-granularity similarity vector.

[0123] Understandably, by minimizing variance, the contributions of similarity at different granularities can be balanced, thereby avoiding over- or under-representation problems.

[0124] The apparatus for implementing text-based video retrieval provided in the embodiments of this application will be described below. The apparatus for implementing text-based video retrieval described below can be referred to in correspondence with the method for implementing text-based video retrieval described above.

[0125] See Figure 3 , Figure 3 This is a schematic diagram of a device structure for implementing text-based video retrieval, as disclosed in an embodiment of this application.

[0126] like Figure 3 As shown, the device may include:

[0127] The text-level feature construction unit 11 is used to obtain text content for retrieving videos and construct sentence features and word features of the text content;

[0128] The video-level feature construction unit 12 is used to perform coarse screening of videos based on the text content, obtain multiple candidate videos, and construct video features and frame features for each candidate video.

[0129] The multi-granularity similarity calculation unit 13 is used to calculate, for each candidate video, the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features.

[0130] The semantic similarity score calculation unit 14 is used to calculate the semantic similarity score of each candidate video based on the video sentence similarity, the video word similarity, the frame sentence similarity, and the frame word similarity.

[0131] The video retrieval determination unit 15 is used to determine the candidate video with the highest semantic similarity score as the retrieval output video of the text content.

[0132] Optionally, the specific implementation logic of the other units of the text-based video retrieval device corresponds one-to-one with the text-based video retrieval method mentioned in the foregoing embodiments. For detailed implementation process, please refer to the foregoing embodiments, which will not be repeated here.

[0133] The text-based video retrieval device provided in this application embodiment can be applied to text-based video retrieval devices, such as terminals: mobile phones, computers, etc. Optionally, Figure 4 The hardware structure block diagram of a text-based video retrieval device is shown, with reference to... Figure 4 The hardware structure of a text-based video retrieval device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0134] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0135] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0136] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0137] The memory stores a program, which the processor can call. The program is used for:

[0138] Obtain the text content used for video retrieval, and construct sentence features and word features of the text content;

[0139] Based on the text content, a coarse screening of videos is performed to obtain multiple candidate videos, and video features and frame features of each candidate video are constructed.

[0140] For each candidate video, calculate the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features.

[0141] For each candidate video, a semantic similarity score is calculated based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity of the candidate video.

[0142] The candidate video with the highest semantic similarity score is determined as the retrieved video of the text content.

[0143] Optionally, the refined and extended functions of the program can be found in the description above.

[0144] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0145] Obtain the text content used for video retrieval, and construct sentence features and word features of the text content;

[0146] Based on the text content, a coarse screening of videos is performed to obtain multiple candidate videos, and video features and frame features of each candidate video are constructed.

[0147] For each candidate video, calculate the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features.

[0148] For each candidate video, a semantic similarity score is calculated based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity of the candidate video.

[0149] The candidate video with the highest semantic similarity score is determined as the retrieved video of the text content.

[0150] Optionally, the refined and extended functions of the program can be found in the description above.

[0151] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0153] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text-based video retrieval method, characterized in that, include: Obtain the text content used for video retrieval, and construct sentence features and word features of the text content; Based on the text content, a coarse screening of videos is performed to obtain multiple candidate videos, and video features and frame features of each candidate video are constructed. For each candidate video, calculate the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features. For each candidate video, a semantic similarity score is calculated based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity of the candidate video. The candidate video with the highest semantic similarity score is determined as the retrieved video of the text content.

2. The method according to claim 1, characterized in that, Based on the video sentence similarity, video word similarity, frame sentence similarity, and frame word similarity of the candidate videos, a semantic similarity score is calculated, including: The video word similarity, the frame sentence similarity, and the frame word similarity are scalarized to obtain scalarized video word similarity, scalarized frame sentence similarity, and scalarized frame word similarity. The similarity vector is obtained by concatenating the video word similarity, the frame sentence similarity, the frame word similarity, and the video sentence similarity. Calculate the granularity aggregation weight vector based on the similarity vector; The semantic similarity score of the candidate video is obtained by weighting and summing the similarity vector using the granular aggregation weight vector.

3. The method according to claim 1, characterized in that, Also includes: The text content is input into a pre-trained text video retrieval model, which outputs the retrieved video results. The training process of the text video retrieval model includes: Obtain training samples, which include multiple text data and multiple video data; Construct sentence embedding features and word embedding features for each text content, and construct video embedding features and frame embedding features for each video data; For each video data sentence embedding feature and word embedding feature, and for each text data video embedding feature and frame embedding feature, calculate the video sentence similarity scalar between the video embedding feature and the sentence embedding feature, the video word similarity vector between the video embedding feature and the word embedding feature, the frame sentence similarity vector between the frame embedding feature and the sentence embedding feature, and the frame word similarity vector between the frame embedding feature and the word embedding feature. For each video data and each text data, a multi-granularity aggregated similarity score of the text data to the video data is calculated based on the video sentence similarity scalar, the video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector between the video data and the text data. The multi-granularity aggregated similarity score of each text data with respect to each video data is used as the input to the loss function of the text video retrieval basic model. The first loss value of the text video retrieval basic model for training with respect to the text data and the video data is calculated, and the model parameters of the text video retrieval basic model are optimized based on the first loss value until the text video retrieval model is obtained after training with all text data and all video data.

4. The method according to claim 3, characterized in that, Based on the video sentence similarity scalar, the video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector between the video data and the text data, a multi-granularity aggregated similarity score of the text data to the video data is calculated, including: The video word similarity vector, the frame sentence similarity vector, and the frame word similarity vector are scalarized respectively to obtain scalarized video word similarity scalar, scalarized frame sentence similarity scalar, and scalarized frame word similarity scalar. The video word similarity scalar, the frame sentence similarity scalar, the frame word similarity scalar, and the video sentence similarity scalar are concatenated to obtain a similarity concatenation vector; Calculate the similarity weight vector based on the similarity concatenation vector; The text data is aggregated with the video data by weighting and summing the similarity vectors using the similarity weights.

5. The method according to claim 3, characterized in that, Based on the aforementioned similarity concatenation vector, a similarity weight vector is calculated, including: The similarity weight vector is obtained by performing linear Softmax aggregation on the concatenated similarity vector using the following formula: in, For linear layers, initialized as a 4x4 identity matrix. Let be the similarity weight vector. For the i-th text data For the i-th video data Multi-granularity similarity vectors The similarity weight vector has a dimension of 1*4.

6. The method according to any one of claims 3-5, characterized in that, Also includes: Using the multi-granularity similarity vector of each text data point to each video data point as input to the multi-granularity similarity balance loss function of the text-video retrieval base model, a second loss value is calculated for training the text-video retrieval base model on the given text data and video data. The model parameters of the text-video retrieval base model are then optimized based on this second loss value. The multi-granularity similarity balance loss function is: in, This is the second loss value. For sample batch size, For the i-th text data For the i-th video data A multi-granularity similarity vector.

7. A text-based video retrieval device, characterized in that, include: A text-level feature construction unit is used to acquire text content for video retrieval and construct sentence features and word features of the text content; A video-level feature construction unit is used to perform coarse screening of videos based on the text content, obtain multiple candidate videos, and construct video features and frame features for each candidate video. A multi-granularity similarity calculation unit is used to calculate, for each candidate video, the video sentence similarity between the video features of the candidate video and the sentence features, the video word similarity between the video features of the candidate video and the word features, the frame sentence similarity between the frame features of the candidate video and the sentence features, and the frame word similarity between the frame features of the candidate video and the word features. The semantic similarity score calculation unit is used to calculate the semantic similarity score of each candidate video based on the video sentence similarity, the video word similarity, the frame sentence similarity, and the frame word similarity. The video retrieval unit is used to determine the candidate video with the highest semantic similarity score as the retrieval output video of the text content.

8. The apparatus according to claim 7, characterized in that, The semantic similarity score calculation unit includes: The scalarization unit is used to perform scalarization processing on the video word similarity, the frame sentence similarity, and the frame word similarity respectively, to obtain scalarized video word similarity, scalarized frame sentence similarity, and scalarized frame word similarity; A scalar concatenation unit is used to concatenate the video word similarity, the frame sentence similarity, the frame word similarity, and the video sentence similarity to obtain a similarity vector; The weight vector calculation unit is used to calculate the granular aggregated weight vector based on the similarity vector. The weighted summation unit is used to sum the similarity vector by the granular aggregated weight vector to obtain the semantic similarity score of the candidate video.

9. A text-based video retrieval device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the text-based video retrieval method as described in any one of claims 1-6.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the text-based video retrieval method as described in any one of claims 1-6.

Images