Video searching method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a video search method, device and equipment and a medium, and the method comprises the steps: determining a first similarity between each candidate video and a video search text based on a first text feature vector and a plurality of first video feature vectors, selecting a preset number of candidate videos which are ranked in the front, and storing the selected candidate videos in the video search text; the videos serve as a plurality of target videos matched with the video search text; generating a second video feature vector of each target video in the spatial dimension and a third video feature vector of each target video in the time dimension based on the plurality of target videos and the first text feature vector; and based on the first text feature vector, the second video feature vector and the third video feature vector, determining a target similarity between each target video and the video search text, and sorting the plurality of target videos according to a descending order of the target similarities. By adopting a video search mode of recalling and then sorting, the video search efficiency is effectively improved while the accuracy of video search is ensured.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and data analysis technology, and in particular to a video search method, device, equipment and medium. Background Art

[0002] With the rapid development of the Internet and computer technology, the number of online videos has increased dramatically. For example, the Internet may generate a huge amount of videos every day. At present, when users search for short videos they want to watch, they can enter text information in the search bar of short video applications to search. However, video searches in related technologies are usually performed through text information such as video titles and tags, lacking semantic understanding of the video itself, which leads to inaccurate video search results. Summary of the invention

[0003] The present invention provides a video search method, apparatus, computer equipment and medium to solve the technical problem that video search is usually performed through text information such as video titles and tags, lacks semantic understanding of the video itself, and thus leads to inaccurate video search results.

[0004] In a first aspect, a video search method is provided, comprising:

[0005] Obtaining a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0006] Based on the first text feature vector and the plurality of first video feature vectors, determining a first similarity between each candidate video and the video search text, and sorting the plurality of candidate videos in descending order according to the first similarities, and selecting a preset number of candidate videos ranked first as the plurality of target videos matching the video search text;

[0007] Based on the multiple target videos and the first text feature vector, generating a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video;

[0008] Based on the first text feature vector, the second video feature vector and the third video feature vector, a target similarity between each target video and the video search text is determined, and the multiple target videos are sorted in descending order according to the target similarity.

[0009] In a second aspect, a video search device is provided, comprising:

[0010] An acquisition module, used to acquire a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0011] A first determination module is used to determine a first similarity between each candidate video and the video search text based on the first text feature vector and the plurality of first video feature vectors, and to sort the plurality of candidate videos in descending order according to the first similarities, and to select a preset number of candidate videos ranked first as the plurality of target videos matching the video search text;

[0012] A generating module, used for generating a second video feature vector in a spatial dimension and a third video feature vector in a temporal dimension for each target video based on a plurality of target videos and a first text feature vector;

[0013] The second determination module is used to determine the target similarity between each target video and the video search text based on the first text feature vector, the second video feature vector and the third video feature vector, and sort the multiple target videos in descending order according to the target similarity.

[0014] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video search method when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video search method are implemented.

[0016] In the scheme implemented by the above-mentioned video search method, device, computer equipment and storage medium, first, feature extraction is performed on the video search text to obtain a first text feature vector, and feature extraction is performed on each candidate video to obtain multiple first video feature vectors. According to the first text feature vector and the first video feature vector, the similarity between the video search text and each candidate video is calculated, and multiple target videos matching the video search text are screened from a large number of candidate videos according to the similarity. The features of the text and the features of the video are directly associated, and fine-grained visual and text features are used to achieve alignment, thereby reducing the information loss of the video and text during alignment, enhancing the performance of video-text search, and thus improving the accuracy of video search. Thereafter, feature extraction is performed on multiple target videos, and the first text feature vector is interacted with the extracted feature vector to obtain a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video. According to the second video feature vector, the third video feature vector and the first text feature vector, the target similarity between the video search text and each target video is calculated, and then based on the calculated target similarity, the multiple target videos are sorted, and finally the multiple target videos are displayed according to the sorting result. The text features are fused with the video features of the selected target videos, and then the similarity is calculated, added, and sorted. There is no need to fuse the text features with the video features of all candidate videos, which greatly reduces the amount of calculation when applied online. Through the above method, the video search is divided into the video recall stage and the sorting stage. The video search method of recalling first and then sorting is adopted, which effectively improves the efficiency of video search while ensuring the accuracy of video search. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is a schematic diagram of an application environment of a video search method in one embodiment of the present invention;

[0019] Figure 2 is a flow chart of a video search method according to an embodiment of the present invention;

[0020] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0021] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0022] Figure 5 is a structural schematic diagram of a video search device in one embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0024] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The video search method provided by the embodiment of the present invention can be applied to Figure 1 In an application environment, the client communicates with the server through a network. The server receives the video search text sent by the client, and extracts text features of the video search text from a large number of candidate videos stored in a candidate video library to obtain a first text feature vector; and extracts features of each candidate video in the candidate video library to obtain a first video feature vector of each candidate video. Thereafter, the similarity between each candidate video and the video search text is calculated according to the text feature vector and the video feature vector, and multiple target videos with high similarity to the video search text are screened from multiple candidate videos to achieve video recall. Thereafter, the recalled multiple target videos are re-sorted, and the multiple target videos are fed back to the client according to the final sorting result, and the multiple target videos are displayed to the user according to the sorting result. In the present invention, firstly, based on the text features corresponding to the text and the video features corresponding to each video, multiple target videos matching the text are screened out, and then the multiple target videos are sorted based on the text features and the video features. Through the two-stage video search of first recalling and then sorting, the accuracy of the search results is improved through fine-grained information search, while taking into account the time efficiency of video search. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0027] See also Figure 2 As shown, Figure 2A schematic flow chart of a video search method provided by an embodiment of the present invention includes the following steps:

[0028] S10: Obtain a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0029] The video search method provided by the present invention can be applied to searching video scenes based on text descriptions, wherein the video search text is a search text input by a user, and feature extraction is performed on the video search text to obtain text features corresponding to the video search text, which are recorded as the first text feature vector.

[0030] The candidate videos may be videos stored on the server. Feature extraction is performed on each candidate video to obtain video features corresponding to each candidate video, which are recorded as first video feature vectors.

[0031] In actual application scenarios, candidate videos are material videos stored in the candidate video library, which are used to create and publish all video content of marketing short videos. When relevant personnel make videos, they need a large number of videos for marketing material content. At this time, relevant personnel enter video search text, where the video search text can be product search text, brand search text, tutorial search text, etc., such as "How to use facial cream", "How to make homemade pizza", "Funny short video collection", etc.

[0032] In one embodiment of the present application, Figure 3 As shown, a specific solution for obtaining a first text feature vector and a first video feature vector is provided. In S10, the first text feature vector of the video search text and multiple first video feature vectors of multiple candidate videos are obtained, which specifically includes the following steps S11-S15:

[0033] S11: Obtain video search text and multiple candidate videos;

[0034] S12: extracting text features from the video search text using a pre-trained text encoder to obtain a first text feature vector;

[0035] S13: performing frame extraction processing on each candidate video according to a preset frame rate to obtain multiple video frames, and generating a first image sequence based on each video frame;

[0036] Wherein, the first image sequence includes a plurality of image blocks;

[0037] For steps S11-S13, the server receives the video search text sent by the client, and uses a pre-trained text encoder to extract features from the video search text to obtain a first text feature vector. The pre-trained text encoder is a pre-trained neural network model for text feature extraction.

[0038] Furthermore, after obtaining a large number of candidate videos, each candidate video is subjected to frame extraction processing according to a preset frame rate. For example, if the preset frame rate is four frames per second and the candidate video is 10 seconds, a total of 40 video frames are extracted. Thereafter, based on a preset image size (e.g., 16×16 pixels), each frame is divided into multiple image patches to form an image sequence corresponding to each video frame, which is recorded as a first image sequence.

[0039] In the above way, extracting frames from the video and patchifying it can effectively reduce the computational complexity. At the same time, it allows the model to focus more on local details and help extract more fine-grained features.

[0040] Optionally, the text encoder may be a BERT (Bidirectional Encoder Representations from Transformers) model.

[0041] S14: inputting a plurality of image blocks in the first image sequence of each video frame into a pre-trained image encoder in time order to obtain a first time feature vector corresponding to each video frame;

[0042] S15: Performing maximum pooling processing on multiple first time feature vectors of multiple video frames of each candidate video to obtain a first video feature vector corresponding to each candidate video.

[0043] For steps S14-S15, for any candidate video, the patches (multiple image blocks) of each frame are input into the pre-trained image encoder in time order for feature extraction to obtain the first time feature vector of each video frame. These first time feature vectors are subjected to a maximum pooling operation to obtain the first video feature vector of the candidate video.

[0044] In the above manner, the extracted temporal features are subjected to a maximum pooling operation to select the maximum value on each feature dimension, thereby retaining the strongest features on each dimension to capture the most significant and representative features in the video.

[0045] Optionally, the image encoder may be a ViT (Vision Transformer) model.

[0046] S20: Based on the first text feature vector and the plurality of first video feature vectors, determining a plurality of first similarities between the video search text and the plurality of candidate videos, and sorting the plurality of candidate videos from large to small according to the first similarities, and selecting a preset number of candidate videos ranked first as target videos;

[0047] After obtaining the first text feature vector and the first video feature vector, since the dimensions of the features extracted by different encoding methods may be different, in order to avoid the situation where features with large dimensions dominate the video search results, it is necessary to map each feature to the same space for similarity calculation. After mapping to the same space, a common similarity measurement method (such as cosine similarity) can be used to calculate the similarity between the first video feature vector of each candidate video and the first text feature vector of the video search text in turn, as the similarity between each candidate video and the video search text, recorded as the first similarity.

[0048] Thereafter, the candidate videos are sorted in descending order according to the first similarity, and a preset number of candidate videos ranked first are selected as target videos matching the video search text.

[0049] Optionally, different preset numbers can be set according to different usage scenarios, and this application does not make any specific limitations here.

[0050] Compared with the video search method in the related art that textualizes or labels the video and searches the text to display the results of related videos, the present application directly associates the features of the text with the features of the video, and searches the video content directly through the text features, without the need for textualization / labeling of the video, thereby significantly reducing the amount of calculation, making the search simpler and faster, and at the same time, can deeply explore the unique information of the video and text, thereby reducing the information loss when the video and text are aligned, and enhancing the performance of video-text search.

[0051] S30: Based on the multiple target videos and the first text feature vector, generate a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video;

[0052] S40: Determine the target similarity between each target video and the video search text based on the first text feature vector, the second video feature vector and the third video feature vector, and sort the multiple target videos in descending order of target similarity.

[0053] For steps S30-S40, in the related art, after recalling multiple target videos that match the video search text, the multiple target videos are usually directly displayed to the user. However, the visual information in the video changes in the time dimension, and directly using MeanPooling to represent the visual information cannot fully capture the interactive relationship between the text and the video. Therefore, after screening a large number of candidate videos and retaining only a few target videos with high relevance, the present application proposes to extract spatial and temporal video features for each target video to obtain multiple first spatial image feature vectors and multiple second temporal feature vectors of multiple target videos, and to perform feature fusion on the extracted spatial feature vectors and temporal feature vectors with the text feature vector of the video search text, respectively, and to generate multiple second video feature vectors corresponding to the multiple target videos based on the first text feature vector and the multiple first spatial image feature vectors, and to generate multiple third video feature vectors corresponding to the multiple target videos based on the first text feature vector and the multiple second temporal feature vectors, so as to obtain video spatial representations of the target videos in different dimensions. Afterwards, the second video feature vector, the third video feature vector and the first text feature vector are used to calculate the similarity between each target video and the video search text, recorded as the target similarity, and the multiple target videos are sorted in descending order according to the target similarity. Finally, the sorted multiple target videos are fed back to the client, and the multiple target videos are displayed to the user according to the sorting results.

[0054] It can be seen from the above scheme that in the stage of sorting the multiple selected target videos, by fusing the text features with the video features, the search paradigm is changed from a double tower to a single tower, which can fully capture the complex interactions and relationships between the text and video features, and then focus on the key correlation between the two, and then calculate the target similarity between each target video and the video search text based on the features after the interaction, as the basis for sorting, this application only fuses the features of the target video with the text features, while improving the accuracy of the sorting results, taking into account the time efficiency of the video search.

[0055] In one embodiment of the present application, Figure 4 As shown, a specific scheme for generating a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video is provided. In S30, based on multiple target videos and the first text feature vector, a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video are generated, specifically including the following steps S31-S36:

[0056] S31: performing frame extraction processing on a plurality of target videos according to a preset frame rate to obtain a plurality of target video frames, and generating a second image sequence based on each target video frame;

[0057] Wherein, the second image sequence includes a plurality of image blocks;

[0058] S32: Processing the second image sequence by a pre-trained image encoder to obtain features of each image block;

[0059] S33: performing mean pooling processing on multiple features of multiple image blocks to obtain a first spatial image feature vector corresponding to each target video frame;

[0060] For steps S31-S33, each target video is sequentially framed according to a preset frame rate to obtain a plurality of target video frames corresponding to each target video. Thereafter, each frame is divided into a plurality of image patches based on a preset image size (e.g., 16×16 pixels) to form an image sequence corresponding to each target video frame, which is recorded as a second image sequence.

[0061] Furthermore, multiple image blocks of each frame are input into a pre-trained image encoder, and the features of each patch of the current frame are output, and then the features of all patches are mean pooled to obtain the first spatial image feature vector corresponding to each target video frame.

[0062] S34: performing feature fusion on the first text feature vector of the video search text and the multiple first spatial image feature vectors of each target video through a cross attention mechanism to obtain a second video feature vector corresponding to each target video;

[0063] After obtaining multiple first spatial image feature vectors of multiple target video frames extracted from each target video, the first text feature vector and multiple first spatial image feature vectors are used as dual-stream inputs and are respectively sent to the pre-trained text interaction model. A cross-attention mechanism is introduced between the video features and the text features. Through the weighted summation strategy, the video features and the text features are fused to obtain a comprehensive feature representation. Specifically, Nv fine-grained first spatial image feature vectors with a feature vector dimension of d and first text feature vectors with a feature vector dimension of d are input into the pre-trained text interaction model. By calculating the similarity between the fine-grained first spatial image feature vector and the first text feature vector, the video search text pair N is obtained through softmax. v The similarity weight of the first spatial image feature vector is obtained, and after multiplying the weight with the first spatial image feature vector, the second video feature vector of each target video in the spatial dimension is obtained by weighting, thereby enhancing the degree of attention to key information.

[0064] S35: inputting a plurality of image blocks in the second image sequence of each target video frame into a pre-trained image encoder in time order to obtain a plurality of second time feature vectors corresponding to each target video;

[0065] S36: Through the cross-attention mechanism, the first text feature vector of the video search text and the multiple second time feature vectors of each target video are feature fused to obtain a third video feature vector corresponding to each target video.

[0066] For steps S35-36, multiple image blocks in the second image sequence are input into the pre-trained image encoder in time sequence to obtain a time feature vector of each target video frame, which is recorded as a second time feature vector. Thereafter, multiple second time feature vectors of multiple target video frames after each target video is extracted are feature fused with the first text feature vector of the video search text to obtain a video feature vector of each target video in the time dimension, which is recorded as a third video feature vector.

[0067] It can be seen from the above scheme that by fusing the image features of each target video with the text features, the search paradigm is changed from a double-tower to a single-tower. Combining spatial and temporal representations can better understand the semantic relationship between text and video and effectively capture cross-modal correlations.

[0068] In one embodiment of the present application, a specific scheme for sorting multiple target videos is provided. In S40, based on the first text feature vector, the second video feature vector and the third video feature vector, the target similarity between each target video and the video search text is determined, and the multiple target videos are sorted in descending order according to the target similarity, which specifically includes the following steps S41-S44:

[0069] S41: Calculating a second similarity between the video search text and each target video based on the first text feature vector and the second video feature vector;

[0070] S42: Calculating a third similarity between the video search text and each target video based on the first text feature vector and the third video feature vector;

[0071] S43: Based on the second similarity and the third similarity, a target similarity between the video search text and each target video is obtained.

[0072] For steps S41-S43, each feature is mapped to the same space for similarity calculation. After being mapped to the same space, a common similarity measurement method (such as cosine similarity) can be used to calculate the similarity between the second video feature vector of each target video and the first text feature vector of the video search text in turn, as the second similarity between each target video and the video search text. At the same time, the similarity between the third video feature vector of each target video and the first text feature vector of the video search text is calculated in turn, as the third similarity between each target video and the video search text. Thereafter, the second similarity and the third similarity are averaged to obtain the total similarity between the video search text and each target video, which is recorded as the target similarity.

[0073] It can be seen from the above scheme that by calculating the second similarity and the third similarity separately and adding them up, it is possible to evaluate the degree of match between the video and the text in different dimensions of image features and time features, thereby obtaining a more comprehensive target similarity, which helps to screen out videos that meet the text description in multiple dimensions and improve the accuracy of sorting.

[0074] S44: sorting the multiple target videos in descending order according to the target similarity.

[0075] The multiple target videos are sorted in descending order according to the target similarity. Thereafter, the multiple target videos are fed back to the client according to the sorting result, so that the client displays the multiple target videos to the user according to the sorting result.

[0076] It can be seen from the above scheme that after screening out multiple target videos that match the video search text from a large number of candidate videos, the multiple target videos are re-sorted, thereby achieving screening and sorting of videos at different levels, ensuring that the target videos finally displayed to users are highly relevant content that has been verified by multiple times, so as to give priority to displaying videos that are more relevant to the user's search, allowing users to find content of interest more quickly, thereby improving user satisfaction and usage experience.

[0077] In one embodiment of the present application, a specific model training scheme is provided, which specifically includes the following steps S50-S80:

[0078] S50: construct a multimodal image-text search model, wherein the multimodal image-text search model is composed of a text encoder and an image encoder;

[0079] S60: Obtain a video sample set and a text sample set;

[0080] For steps S50-S60, in order to accurately search the video, it is necessary to pre-train the multimodal image-text search model to obtain a pre-trained multimodal image-text search model. The multimodal image-text search model adopts a double-tower structure, which is a CLIP (Contrastive Language-Image Pre-training) model composed of a text encoder and an image encoder, wherein the image encoder realizes feature extraction of image information, and the text encoder realizes feature extraction of text information. Based on the structure of the text encoder in the current CLIP model, the relevant parameters of the text encoder are initialized to obtain a pre-constructed text encoder; based on the structure of the image encoder in the current CLIP model, the relevant parameters of the image encoder are initialized to obtain a pre-constructed image encoder.

[0081] Afterwards, a Python script with data crawling capabilities is used to crawl large-scale subtitled video data from a designated website to form a video sample set. Each video data corresponds to text description data. The text description data is formed by extracting the subtitles of each video to form a text sample set to reflect the main content of the video data. The two modal samples have a one-to-one correspondence.

[0082] S70: Calculate the Spearman constraint loss and contrastive learning loss based on the video sample set and the text sample set;

[0083] The multimodal image-text search model is trained through Spearman constraint loss and contrastive learning loss. Spearman constraint loss compares the similarity between different modalities (i.e., image and text), so that the model can better capture the correspondence between multimodal features and enhance the alignment effect. Contrastive learning loss can effectively improve the model's ability to distinguish similarities by maximizing the similarity between positive sample pairs (i.e., matching images and texts) and minimizing the similarity between negative sample pairs (i.e., mismatched images and texts).

[0084] By combining contrastive learning and Spearman constraints in the above way, the model can have a deeper understanding of each modality, improve the fusion effect between modalities, and help achieve better performance in comprehensive applications.

[0085] In one embodiment of the present application, a specific loss function construction scheme is provided. In S70, Spearman constraint loss and contrastive learning loss are calculated based on the video sample set and the text sample set, which specifically includes the following steps S71-S78:

[0086] S71: Based on the video sample set and the text sample set, a plurality of video-text training data pairs are formed, wherein each video training data pair includes video training data and its corresponding text training data;

[0087] After crawling a large number of videos, the corresponding videos are segmented based on the text sentences in the text description content to obtain the video clips corresponding to each text sentence. The text sentences are then used as text training data and the corresponding video clips are used as video training data to form a video-text training data pair.

[0088] S72: Processing the text training data through a text encoder to obtain a second text feature vector;

[0089] S73: Process the video training data through an image encoder to obtain a second spatial image feature vector and a third temporal feature vector;

[0090] For steps S72-S73, each text training data is input into a pre-constructed text encoder in turn to obtain the second text feature vector of the text training data. Thereafter, each video data is framed according to a preset frame rate to obtain multiple key frames corresponding to each video training data, and each key frame is patchified according to a set image size to obtain patches of each key frame, and the patches are input into a pre-constructed image encoder to obtain the features of each patch in each key frame. After mean pooling the image features of all patches, multiple second spatial image feature vectors corresponding to each video training data are obtained. The patches of each key frame are input into a pre-constructed image encoder in chronological order to obtain a third temporal feature vector corresponding to each key frame.

[0091] S74: performing maximum pooling processing on the third time feature vector to generate a fourth video feature vector for each video data;

[0092] For any video training data, feature extraction is performed on it to obtain the third time feature vector of each extracted key frame, and then max pooling is performed on multiple third time feature vectors to obtain the fourth video feature vector of the video training data.

[0093] S75: generating a fifth video feature vector for each video data based on the second text feature vector and the second spatial image feature vector;

[0094] The second text feature vector and the plurality of second space image feature vectors are input into a pre-trained text interaction model, and the second text feature vector and the plurality of second space image feature vectors are feature fused through a cross attention mechanism to obtain a fifth video feature vector corresponding to each video training data.

[0095] S76: generating a sixth video feature vector for each video data based on the second text feature vector and the third time feature vector;

[0096] The second text feature vector and multiple third time feature vectors are input into the pre-trained text interaction model, and the second text feature vector and multiple third time feature vectors are feature fused through the cross attention mechanism to obtain the sixth video feature vector corresponding to each video training data.

[0097] In the related art, Mean Pooling is used to merge multiple frame representations of a video into one video representation to construct a loss function. However, unlike the relatively small amount of visual information in an image, the visual information in a video varies in the visual dimension. Obviously, the accuracy of using Mean Pooling alone to represent the visual information is low. Therefore, this application proposes to combine image features with time features to complement each other and enhance the robustness of system training.

[0098] S77: Calculate the Spearman constraint based on the fifth video feature vector and the sixth video feature vector;

[0099] Among them, Spearman is used to evaluate the correlation between two variables X and Y, where the correlation between variables can be described by a monotonic function, and its value is represented by the Greek letter ρ. The Spearman constraint loss is in the channel dimension, and the Spearman correlation coefficient is calculated for the channel of the video feature and the channel of the text feature in a training batch, thereby realizing the evaluation of the monotonic correlation of two different channels. Therefore, using the Spearman constraint formula, the monotonic correlation coefficient of all current training data on the fifth video feature vector and the sixth video feature vector is calculated, and then the Spearman constraint is calculated.

[0100] In one embodiment of the present application, a specific Spearman constraint loss function construction scheme is provided. In S77, a Spearman constraint loss function is constructed based on the fifth video feature vector and the sixth video feature vector, which specifically includes the following steps S771-S772:

[0101] S771: Calculate the monotonic correlation coefficient between the fifth video feature vector and the sixth video feature vector;

[0102] S772: Calculate the Spearman constraint based on the monotonic correlation coefficient.

[0103] For steps S771-S772, first, the i-th elements in the fifth video feature vector and the sixth video feature vector are sorted respectively, and a ranking is assigned to each element. Thereafter, the difference in the rank values ​​of each value in different video feature vectors is calculated. The difference in rank values ​​and the dimension of the video feature vector are input into the mathematical expression of the Spearman constraint:

[0104]

[0105] Wherein, ρ is the monotonic correlation coefficient; n is the dimension of the video feature vector; d i is the difference in the rank value of the i-th element in the video feature vector, i∈n.

[0106] Thereafter, the Spearman constraint loss is calculated based on the current Spearman correlation coefficient.

[0107] S78: Calculate contrastive learning loss based on the second text feature vector and the fourth video feature vector.

[0108] Based on the second text feature vector and the fourth video feature vector, the cosine similarities of the positive sample pairs (video-text training data pairs) and the negative sample pairs (unmatched video-text training data pairs) are calculated respectively, and imported into the mathematical expression of the InfoNCE loss function to obtain the InfoNCE loss.

[0109] Through the above approach, a parameter-free text interaction method is implemented for fine-grained video representation learning, and additional Spearman constraints are embedded to optimize cross-modal representation learning.

[0110] In actual application scenarios, the CLIP model comparison learning framework is pre-built, and the text training data is input into the text encoder (such as the BERT model), and the text feature vector T (i.e., the second text feature vector) with dimension d is output. The video training data is extracted, for example, 4 frames per second, all frames (10 seconds of video is 40 frames) are patchified, and the patches of each frame are input into the spatial image encoder (such as the ViT model), and the features of each patch of the current frame are output. After mean pooling the features of all patches, N is finally obtained. v (40 frames) of spatial image feature vectors V s (i.e., the second spatial image feature vector). v V s Input the text interaction model to obtain the final video space representation V1 (i.e., the fifth video feature vector). Afterwards, all frames (40 frames) are taken as a whole and input into ViT in chronological order to obtain the temporal feature vector V corresponding to each frame. t (i.e., the third time eigenvector). Similarly, N v The time feature vector V t The text feature vector T is input into the text interaction model to obtain the final video space representation V2 (i.e., the sixth video feature vector). v The time feature vector V tPerform max pooling to obtain a video representation V3 (i.e., the fourth video feature vector). Subsequently, the Spearman constraint is used to calculate the monotonic correlation coefficient of all training data in the current training batch on V1 and V2. InfoNCE is used to calculate the loss of the text vector T and the video representation V3 (i.e., the similarity of the positive sample pair and the negative sample pair is calculated respectively). Finally, the sum of the two losses is calculated, and the gradient of the loss is back-propagated to the parameters in the training model.

[0111] S80: Based on Spearman loss and contrastive learning loss, backpropagate and calculate gradients of the parameters in the text encoder and image encoder until the loss converges, and output the trained multimodal image-text search model.

[0112] Calculate the weighted sum of Spearman loss and contrastive learning loss as the loss function of the training model, backpropagate the loss function, calculate the gradient of the loss relative to the model parameters (text encoder parameters and image encoder parameters), use the optimizer (such as Adam, SGD) to iteratively update the multi-model graph-text search model parameters according to the gradient calculation results, until the loss converges or the maximum number of iterations is reached, and output the trained multi-model graph-text search model.

[0113] It can be seen that in the above scheme, first, feature extraction is performed on the video search text to obtain the first text feature vector, and feature extraction is performed on each candidate video to obtain multiple first video feature vectors. According to the first text feature vector and the first video feature vector, the similarity between the video search text and each candidate video is calculated, and multiple target videos matching the video search text are screened from a large number of candidate videos according to the similarity. The features of the text and the features of the video are directly associated, and fine-grained visual and text features are used to achieve alignment, thereby reducing the information loss of the video and text during alignment, enhancing the performance of video-text search, and thus improving the accuracy of video search. Thereafter, feature extraction is performed on multiple target videos, and the first text feature vector is interacted with the extracted feature vector to obtain the second video feature vector of each target video in the spatial dimension and the third video feature vector in the time dimension. According to the second video feature vector, the third video feature vector and the first text feature vector, the target similarity between the video search text and each target video is calculated, and then based on the calculated target similarity, the multiple target videos are sorted, and finally the multiple target videos are displayed according to the sorting result. The text features are fused with the video features of the selected target videos, and then the similarity is calculated, added, and sorted. There is no need to fuse the text features with the video features of all candidate videos, which greatly reduces the amount of calculation when applied online. Through the above method, the video search is divided into the video recall stage and the sorting stage. The video search method of recalling first and then sorting is adopted, which effectively improves the efficiency of video search while ensuring the accuracy of video search.

[0114] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0115] In one embodiment, a video search device is provided, which corresponds one-to-one to the video search method in the above embodiment. Figure 5 As shown, the video search device includes: an acquisition module 101, a first determination module 102, a generation module 103, and a second determination module 104. The functional modules are described in detail as follows:

[0116] An acquisition module 101 is used to acquire a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0117] A first determination module 102 is used to determine a first similarity between each candidate video and the video search text based on the first text feature vector and the plurality of first video feature vectors, and to sort the plurality of candidate videos in descending order according to the first similarities, and to select a preset number of candidate videos ranked first as the plurality of target videos matching the video search text;

[0118] A generating module 103, configured to generate a second video feature vector in a spatial dimension and a third video feature vector in a temporal dimension for each target video based on the multiple target videos and the first text feature vector;

[0119] The second determination module 104 is used to determine the target similarity between each target video and the video search text based on the first text feature vector, the second video feature vector and the third video feature vector, and sort the multiple target videos in descending order of target similarity.

[0120] In one embodiment, the acquisition module 101 is specifically configured to:

[0121] Obtain video search text and multiple candidate videos;

[0122] Using a pre-trained text encoder, extract text features from the video search text to obtain a first text feature vector;

[0123] According to a preset frame rate, each candidate video is subjected to frame extraction processing to obtain a plurality of video frames, and a first image sequence is generated based on each video frame, wherein the first image sequence includes a plurality of image blocks;

[0124] Inputting a plurality of image blocks in a first image sequence of each video frame into a pre-trained image encoder in time order to obtain a first time feature vector corresponding to each video frame;

[0125] Multiple first time feature vectors of multiple video frames of each candidate video are subjected to maximum pooling processing to obtain a first video feature vector corresponding to each candidate video.

[0126] In one embodiment, the generating module 103 is specifically configured to:

[0127] According to a preset frame rate, a plurality of target videos are subjected to frame extraction processing to obtain a plurality of target video frames, and a second image sequence is generated based on each target video frame, wherein the second image sequence includes a plurality of image blocks;

[0128] Processing the second image sequence through a pre-trained image encoder to obtain features of each image block;

[0129] Performing mean pooling processing on multiple features of multiple image blocks to obtain a first spatial image feature vector corresponding to each target video frame;

[0130] Through the cross attention mechanism, the first text feature vector of the video search text and the multiple first spatial image feature vectors of each target video are feature fused to obtain the second video feature vector corresponding to each target video;

[0131] Inputting a plurality of image blocks in the second image sequence of each target video frame into a pre-trained image encoder in time order to obtain a plurality of second time feature vectors corresponding to each target video;

[0132] Through the cross-attention mechanism, the first text feature vector of the video search text and multiple second time feature vectors of each target video are fused to obtain the third video feature vector corresponding to each target video.

[0133] In one embodiment, the second determining module 104 is specifically configured to:

[0134] Calculating a second similarity between the video search text and each target video based on the first text feature vector and the second video feature vector;

[0135] Calculating a third similarity between the video search text and each target video based on the first text feature vector and the third video feature vector;

[0136] Based on the second similarity and the third similarity, obtaining a target similarity between the video search text and each target video;

[0137] Sort multiple target videos in descending order according to target similarity.

[0138] In one embodiment, the device further comprises:

[0139] A construction module is used to construct a multimodal image-text search model, wherein the multimodal image-text search model is composed of a text encoder and an image encoder;

[0140] The acquisition module 101 is also used to acquire a video sample set and a text sample set;

[0141] A calculation module is used to calculate the Spearman constraint loss and contrastive learning loss based on the video sample set and the text sample set;

[0142] The training module is used to perform backpropagation and gradient calculation on the parameters in the text encoder and image encoder based on the Spearman constraint loss and contrastive learning loss until the loss converges, and output the trained multimodal image-text search model.

[0143] In one embodiment, the computing module is specifically configured to:

[0144] Based on the video sample set and the text sample set, a plurality of video-text training data pairs are formed, wherein each video training data pair includes video training data and its corresponding text training data;

[0145] Processing the text training data through a text encoder to obtain a second text feature vector;

[0146] Processing the image training data through an image encoder to obtain a second spatial image feature vector and a third temporal feature vector;

[0147] Performing maximum pooling processing on the third time feature vector to generate a fourth video feature vector for each video data;

[0148] generating a fifth video feature vector for each video data based on the second text feature vector and the second spatial image feature vector;

[0149] Generate a sixth video feature vector for each video data based on the second text feature vector and the third time feature vector;

[0150] Calculating a Spearman constraint based on the fifth video feature vector and the sixth video feature vector;

[0151] Based on the second text feature vector and the fourth video feature vector, a contrastive learning loss is calculated.

[0152] In one embodiment, the computing module is further configured to:

[0153] calculating a monotonic correlation coefficient between the fifth video feature vector and the sixth video feature vector;

[0154] Based on the monotonic correlation coefficient, Spearman constraints are calculated.

[0155] The present invention provides a video search device. First, feature extraction is performed on a video search text to obtain a first text feature vector, and feature extraction is performed on each candidate video to obtain multiple first video feature vectors. Based on the first text feature vector and the first video feature vector, the similarity between the video search text and each candidate video is calculated, and multiple target videos matching the video search text are screened from a large number of candidate videos according to the similarity. The features of the text and the features of the video are directly associated, and fine-grained visual and text features are used to achieve alignment, thereby reducing the information loss of the video and text during alignment, enhancing the performance of video-text search, and thus improving the accuracy of video search. Thereafter, feature extraction is performed on multiple target videos, and the first text feature vector is interacted with the extracted feature vector to obtain a second video feature vector and a third video feature vector of each target video. Based on the second video feature vector, the third video feature vector and the first text feature vector, the target similarity between the video search text and each target video is calculated, and then based on the calculated target similarity, the multiple target videos are sorted, and finally the multiple target videos are displayed according to the sorting result. The text features are fused with the video features of the selected target videos, and then the similarity is calculated, added, and sorted. There is no need to fuse the text features with the video features of all candidate videos, which greatly reduces the amount of calculation when applied online. Through the above method, the video search is divided into the video recall stage and the sorting stage. The video search method of recalling first and then sorting is adopted, which effectively improves the efficiency of video search while ensuring the accuracy of video search.

[0156] For the specific definition of the video search device, please refer to the definition of the video search method above, which will not be repeated here. Each module in the above-mentioned video search device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a video search method.

[0158] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a video search method.

[0159] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0160] Obtaining a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0161] Based on the first text feature vector and the plurality of first video feature vectors, determining a first similarity between each candidate video and the video search text, and sorting the plurality of candidate videos in descending order according to the first similarities, and selecting a preset number of candidate videos ranked first as the plurality of target videos matching the video search text;

[0162] Based on the multiple target videos and the first text feature vector, generating a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video;

[0163] Based on the first text feature vector, the second video feature vector and the third video feature vector, a target similarity between each target video and the video search text is determined, and the multiple target videos are sorted in descending order according to the target similarity.

[0164] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0165] Obtaining a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos;

[0166] Based on the first text feature vector and the plurality of first video feature vectors, determining a first similarity between each candidate video and the video search text, and sorting the plurality of candidate videos in descending order according to the first similarities, and selecting a preset number of candidate videos ranked first as the plurality of target videos matching the video search text;

[0167] Based on the multiple target videos and the first text feature vector, generating a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video;

[0168] Based on the first text feature vector, the second video feature vector and the third video feature vector, a target similarity between each target video and the video search text is determined, and the multiple target videos are sorted in descending order according to the target similarity.

[0169] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0171] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0172] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video search method, characterized in that: include: Obtaining a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos; Based on the first text feature vector and the multiple first video feature vectors, determine a first similarity between each candidate video and the video search text, sort the multiple candidate videos in descending order according to the first similarity, and select a preset number of candidate videos ranked first as multiple target videos matching the video search text; Based on the multiple target videos and the first text feature vector, generate a second video feature vector in the spatial dimension and a third video feature vector in the temporal dimension for each target video; Based on the first text feature vector, the second video feature vector and the third video feature vector, a target similarity between each target video and the video search text is determined, and the multiple target videos are sorted in descending order according to the target similarity.

2. The method according to claim 1, characterized in that The step of obtaining a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos specifically includes: Obtaining the video search text and the multiple candidate videos; Extracting text features from the video search text using a pre-trained text encoder to obtain the first text feature vector; Perform frame extraction processing on each candidate video according to a preset frame rate to obtain multiple video frames, and generate a first image sequence based on each video frame, wherein the first image sequence includes multiple image blocks; Inputting a plurality of image blocks in the first image sequence of each video frame into a pre-trained image encoder in time order to obtain a first time feature vector corresponding to each video frame; The multiple first time feature vectors of the multiple video frames of each candidate video are subjected to maximum pooling processing to obtain the first video feature vector corresponding to each candidate video.

3. The method according to claim 1, characterized in that The step of generating a second video feature vector in a spatial dimension and a third video feature vector in a temporal dimension for each target video based on the multiple target videos and the first text feature vector specifically includes: According to a preset frame rate, the multiple target videos are subjected to frame extraction processing to obtain multiple target video frames, and a second image sequence is generated based on each target video frame, wherein the second image sequence includes multiple image blocks; Processing the second image sequence by a pre-trained image encoder to obtain features of each image block; Performing mean pooling processing on multiple features of the multiple image blocks to obtain a first spatial image feature vector corresponding to each target video frame; By using a cross attention mechanism, feature fusion is performed on the first text feature vector of the video search text and the multiple first spatial image feature vectors of each target video to obtain the second video feature vector corresponding to each target video; Inputting a plurality of image blocks in the second image sequence of each target video frame into a pre-trained image encoder in time order to obtain a plurality of second time feature vectors corresponding to each target video; Through the cross-attention mechanism, the first text feature vector of the video search text and the multiple second time feature vectors of each target video are feature fused to obtain the third video feature vector corresponding to each target video.

4. The method according to claim 1, characterized in that: The step of determining the target similarity between each target video and the video search text based on the first text feature vector, the second video feature vector, and the third video feature vector, and sorting the multiple target videos in descending order of target similarity specifically includes: Calculating a second similarity between the video search text and each target video based on the first text feature vector and the second video feature vector; Calculating a third similarity between the video search text and each target video based on the first text feature vector and the third video feature vector; Based on the second similarity and the third similarity, obtaining a target similarity between the video search text and each target video; The multiple target videos are sorted in descending order according to the target similarity.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the first text feature vector of the video search text and the multiple first video feature vectors of the multiple candidate videos, the method further includes: Constructing a multimodal image-text search model, wherein the multimodal image-text search model is composed of a text encoder and an image encoder; Obtain a video sample set and a text sample set; Based on the video sample set and the text sample set, calculating the Spearman constraint loss and the contrastive learning loss; Based on the Spearman constraint loss and the contrastive learning loss, back propagation and gradient calculation are performed on the parameters in the text encoder and the image encoder until the loss converges, and the trained multimodal image-text search model is output.

6. The method according to claim 5, characterized in that The step of calculating the Spearman constraint loss and the contrastive learning loss based on the video sample set and the text sample set specifically includes: Based on the video sample set and the text sample set, a plurality of video-text training data pairs are formed, wherein each video training data pair includes video training data and its corresponding text training data; Processing the text training data by the text encoder to obtain a second text feature vector; Processing the image training data by the image encoder to obtain a second spatial image feature vector and a third temporal feature vector; Performing maximum pooling processing on the third time feature vector to generate a fourth video feature vector for each video data; generating a fifth video feature vector for each video data based on the second text feature vector and the second spatial image feature vector; Generate a sixth video feature vector for each video data based on the second text feature vector and the third time feature vector; Calculating the Spearman constraint based on the fifth video feature vector and the sixth video feature vector; The contrastive learning loss is calculated based on the second text feature vector and the fourth video feature vector.

7. The method according to claim 6, characterized in that The step of calculating the Spearman constraint based on the fifth video feature vector and the sixth video feature vector specifically includes: Calculating a monotonic correlation coefficient between the fifth video feature vector and the sixth video feature vector; Based on the monotonic correlation coefficient, the Spearman constraint is calculated.

8. A video search device, characterized in that: include: An acquisition module, used to acquire a first text feature vector of a video search text and a plurality of first video feature vectors of a plurality of candidate videos; A first determination module is used to determine a first similarity between each candidate video and the video search text based on the first text feature vector and the multiple first video feature vectors, and to sort the multiple candidate videos in descending order according to the first similarity, and select a preset number of candidate videos ranked first as multiple target videos matching the video search text; A generating module, configured to generate a second video feature vector in a spatial dimension and a third video feature vector in a temporal dimension for each target video based on the multiple target videos and the first text feature vector; The second determination module is used to determine the target similarity between each target video and the video search text based on the first text feature vector, the second video feature vector and the third video feature vector, and sort the multiple target videos in descending order of target similarity.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video search method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video search method according to any one of claims 1 to 7 are implemented.

Images