Method, apparatus and device for multimodal video search ranking and storage medium
By calculating the similarity of feature vectors and K-means clustering, setting an automatic threshold, and calculating the total relevance and average relevance of videos, the problem of inaccurate similarity matching in multimodal video search is solved, and the search accuracy is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 深圳市恒扬数据股份有限公司
- Filing Date
- 2023-04-10
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies cannot accurately match the similarity between videos in multimodal video search, resulting in low search accuracy.
The base database and query feature vectors are obtained through feature processing. After calculating the similarity, K-means clustering is performed, an automatic threshold is set, relevant frames are selected, and the total relevance and average relevance of the videos are calculated and sorted.
It improves the accuracy of multimodal video search, avoids the problem of too many or too few relevant frames caused by manually setting thresholds, and improves search accuracy by comprehensively considering frame similarity and ranking position information.
Smart Images
Figure CN116610835B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video search technology, and in particular to a method, apparatus, device, and storage medium for multimodal video search ranking. Background Technology
[0002] With the development of technology, big data and media applications are being used in every aspect of people's lives. The accuracy requirements for video and image searches are also getting higher and higher. In the process of multimodal video search, keyframe processing is usually performed on the video to transform it into multimodal image search. This can greatly reduce the amount of computation and algorithm complexity of video search. However, the disadvantage of this method is that it will eventually obtain the similarity of images rather than the similarity of videos, resulting in insufficient accuracy of video search results.
[0003] Therefore, given the low accuracy of video search similarity calculation, improving the accuracy of video search is an urgent problem to be solved by existing technologies. Summary of the Invention
[0004] The main objective of this application is to provide a method, apparatus, device, and storage medium for multimodal video search and ranking, aiming to solve the technical problems of existing technologies for multimodal video search and ranking that cannot match the similarity between videos and have low accuracy in video search matching.
[0005] To achieve the aforementioned objectives, this application proposes a method for multimodal video search and ranking, the method comprising:
[0006] The base video data is processed to obtain the base feature vector, and the data to be queried is processed by a multimodal model to obtain the query feature vector.
[0007] Calculate the similarity between the query feature vector and the base database feature vector;
[0008] K-means clustering is performed on all the similarities, and an automatic threshold is set according to the similarity ranking of the cluster centers obtained after clustering to obtain the relevant frames;
[0009] The total relevance of each video is calculated based on the relevant frames, and the average relevance of each video is calculated and sorted to obtain the final result of the video search.
[0010] Furthermore, the step of performing feature processing on the base video data to obtain the base feature vector includes:
[0011] The videos in the base library are subjected to frame extraction at video intervals to filter out duplicate images exceeding the threshold and images with a variation range below the preset threshold.
[0012] Features are extracted from the filtered images, and frames with high similarity between adjacent frames are filtered to obtain the final video keyframes.
[0013] The keyframe features of the video are stored in a vector database to obtain the base feature vector.
[0014] Furthermore, the data to be queried includes text data and / or image data to be queried;
[0015] The step of performing multimodal model processing on the data to be queried to obtain the query feature vector includes:
[0016] The text data and / or image data to be queried are processed by a multimodal model to obtain a 512-dimensional query feature vector.
[0017] Further, the step of calculating the similarity between the query feature vector and the base database feature vector includes:
[0018] Through formula Calculate the similarity S between the query feature vector and the base database feature vector, where Q is the query feature vector and G is the base database feature vector.
[0019] Further, the step of performing K-means clustering on all the similarities, setting an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering, and obtaining the relevant frames includes:
[0020] Randomly select k similarity samples from the dataset as initial cluster centers, where k is a hyperparameter representing the number of clusters;
[0021] For each similarity sample in the dataset, calculate its distance to the k cluster centers, and assign it to the class corresponding to the cluster center with the smallest distance;
[0022] For each of the aforementioned clusters, recalculate its cluster center position;
[0023] Then, for each sample in the dataset, calculate its distance to the k cluster centers and assign it to the class corresponding to the cluster center with the smallest distance. Continue until the preset condition is met, and then output the sample status of the current K cluster centers to obtain the similarity of the K cluster centers.
[0024] The similarity of the K cluster centers is sorted in descending order, and the similarity at K / 3 is taken as the automatic threshold. Frames with similarity greater than the automatic threshold are selected as relevant frames.
[0025] Further, the step of calculating the total relevance of each video based on the relevant frames includes:
[0026] Sort all the relevant frames and record the similarity and location information of each relevant video frame;
[0027] According to the formula Calculate the total relevance for each video, where rel(i) represents the similarity of the i-th frame, Ri i The position sorting result of the i-th frame is given, and m is the total number of relevant frames in a video.
[0028] Furthermore, the step of simultaneously calculating the average relevance of each video and performing a final ranking to obtain the final result of the video search includes:
[0029] The average relevance of a video is obtained by dividing its total relevance by the number of relevant frames. Videos are then ranked based on their average relevance to obtain the final search ranking results.
[0030] This application also provides an apparatus for multimodal video search and ranking, the apparatus comprising:
[0031] The processing module is used to perform feature processing on the base video data to obtain the base feature vector, and to perform multimodal model processing on the data to be queried to obtain the query feature vector;
[0032] The vector similarity module is used to calculate the similarity between the query feature vector and the base database feature vector;
[0033] An automatic thresholding module is used to perform K-means clustering on all the similarities, and set an automatic threshold according to the similarity ranking of the cluster centers obtained after clustering to obtain the relevant frames.
[0034] The relevance calculation module is used to calculate the total relevance of each video based on the relevant frames, and at the same time calculate the average relevance of each video and perform the final sorting to obtain the final result of the video search.
[0035] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.
[0036] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.
[0037] This application provides a method for ranking in multimodal video search. It dynamically adjusts a threshold based on the features of the search query and the underlying database. Then, it selects frames from each video that are relevant to the query, ranks all relevant frames above the threshold, calculates the final relevance of each video using a formula, and finally ranks each video according to its average relevance to obtain the final video search result. This method avoids the problem of manually setting the threshold too high or too low, which can lead to too many or too few relevant frames and affect video similarity calculation. Furthermore, it comprehensively considers the similarity of relevant frames and their ranking position information, amplifying the influence of keyframes in the video and improving the accuracy of dynamic video search. Attached Figure Description
[0038] Figure 1 This is a flowchart illustrating a method for multimodal video search and ranking according to an embodiment of this application;
[0039] Figure 2 This is a schematic block diagram of a device for multimodal video search and ranking according to an embodiment of this application;
[0040] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.
[0041] Figure 4 This is a schematic diagram of similarity K-means clustering according to an embodiment of this application.
[0042] Figure 5 This is a schematic diagram of video-related frames according to an embodiment of this application.
[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0045] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.
[0046] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0047] Reference Figure 1 This invention provides a method for multimodal video search and ranking, including steps S1-S4, specifically:
[0048] S1. Perform feature processing on the base video data to obtain the base feature vector, and perform multimodal model processing on the data to be queried to obtain the query feature vector;
[0049] First, the videos in the base database undergo video feature processing to obtain some keyframe features, which are then stored in a vector database to obtain the base database feature vectors. Specifically, during the feature processing, the video first undergoes intermittent frame extraction to filter out a large number of repetitive or little-changing images. Then, features are extracted from the filtered images, and finally, frames with high similarity between adjacent frames are filtered out to obtain the final video keyframes.
[0050] The text / image data to be queried is then processed by a multimodal model to obtain a 512-dimensional query feature vector.
[0051] S2. Calculate the similarity between the query feature vector and the base database feature vector;
[0052] Since the query feature vector and the base database feature vector are both extracted through a multimodal model and have the same dimension, the similarity between the query vector and the base database vector can be obtained through matrix operations.
[0053] S3. Perform K-means clustering on all the similarities, and set an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering to obtain the relevant frames;
[0054] After obtaining the similarity scores, K-means clustering is performed on all similarity scores. Assuming the number of clusters is K, where K is the number of videos in the base library, this will result in K cluster centers. The similarity scores of the K cluster centers are then obtained. These K similarity scores are sorted in descending order, and the similarity score at K / 3 is taken as the automatic threshold. Frames with similarity scores greater than the automatic threshold are considered relevant frames.
[0055] S4. Calculate the total relevance of each video based on the relevant frames, and calculate the average relevance of each video and sort them to obtain the final result of the video search.
[0056] First, sort all the relevance scores and record the similarity and location information of each video's related frames. Then, calculate the relevance of the videos according to preset rules. Finally, calculate the average relevance of the videos and sort them according to the average relevance scores. This is the final video sorting result.
[0057] As described above, in the multimodal search process, the threshold can be dynamically adjusted based on the features of the search query and the base database. Then, the frames of each video that are relevant to the query are selected based on the threshold. All relevant frames exceeding the threshold are then sorted, and the final relevance of each video is calculated using a formula. Finally, each video is sorted according to its average relevance to obtain the final video search result. This avoids the situation where manually setting the threshold too high or too low can lead to too many or too few relevant frames, affecting the video similarity calculation. Furthermore, by comprehensively considering the similarity of relevant frames and their ranking position information, the influence of keyframes in the video is amplified, improving the accuracy of dynamic video search.
[0058] In one embodiment, the step of performing feature processing on the base video data to obtain the base feature vector includes:
[0059] S201. Perform video interval frame extraction on the videos in the base library to filter out duplicate images that exceed the threshold and images whose variation range is below the preset threshold.
[0060] S202. Extract features from the filtered images, filter out frames with high similarity between adjacent frames, and obtain the final video keyframes.
[0061] S203. Save the video keyframe features in a vector database to obtain the base database feature vector.
[0062] In the feature processing of this video, the video first undergoes frame-by-frame extraction at intervals to filter out a large number of repetitive or little-changing images. Then, features are extracted from the filtered images, and finally, frames with high similarity between adjacent frames are filtered out to obtain the final video keyframes. This processing stage is a relatively common video preprocessing stage, and this patent will not describe it in detail.
[0063] In one embodiment, the data to be queried includes text data and / or image data to be queried;
[0064] The step of performing multimodal model processing on the data to be queried to obtain the query feature vector includes:
[0065] S301: The text data and / or image data to be queried are processed by a multimodal model to obtain the 512-dimensional query feature vector.
[0066] In this embodiment, data from different modalities can be processed and fused using different methods. For example, text data can be converted into feature vectors through word embedding, while video and audio data can have features extracted using convolutional neural networks and recurrent neural networks.
[0067] In one embodiment, the step of calculating the similarity between the query feature vector and the base database feature vector includes:
[0068] S401, via formula Calculate the similarity S between the query feature vector and the base database feature vector, where Q is the query feature vector and G is the base database feature vector.
[0069] Since the query feature vector and the base database feature vector are both extracted using a multimodal model and have the same dimensions, the formula... By performing matrix operations on the query feature vector and the base database feature vector, the similarity between the query vector and the base database vector can be obtained.
[0070] In one embodiment, the steps of performing K-means clustering on all the similarities, setting an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering, and obtaining the relevant frames include:
[0071] S501. Randomly select k similarity samples from the dataset as initial cluster centers, where k is a hyperparameter representing the number of clusters;
[0072] S502. For each similarity sample in the dataset, calculate its distance to the k cluster centers, and assign it to the class corresponding to the cluster center with the smallest distance.
[0073] S503. For each of the clusters, recalculate its cluster center position;
[0074] S504. Then, for each sample in the dataset, calculate its distance to the k cluster centers and assign it to the class corresponding to the cluster center with the smallest distance. After the preset condition is met, output the sample status of the current K cluster centers to obtain the similarity of the K cluster centers.
[0075] S505. Sort the similarity of the K cluster centers in descending order, take the similarity at K / 3 as the automatic threshold, and select the frames that are greater than the automatic threshold as relevant frames.
[0076] In this embodiment, K-means clustering is performed on all similarities. Assuming the number of clusters is K, where K is the number of videos in the base library, K cluster centers will be obtained. The similarity of the K cluster centers will be obtained. These K similarities will be sorted in descending order, and the similarity at K / 3 will be taken as the automatic threshold. Frames with similarity greater than the automatic threshold are considered relevant frames.
[0077] Specifically, k samples are randomly selected as initial cluster centers (k is a hyperparameter representing the number of clusters. Its value can be determined based on prior knowledge or verification methods); then, for each sample in the dataset, its distance to the k cluster centers is calculated, and it is assigned to the cluster corresponding to the cluster center with the smallest distance; then, for each cluster that has been assigned, its cluster center position is recalculated; then, the two steps s502-s503 above are repeated iteratively until a certain termination condition is reached. This termination condition can be a pre-set number of iterations, or a condition such as the cluster center position remaining unchanged.
[0078] The purpose of setting an automatic threshold in this embodiment is to prevent the number of filtered videos from being too many or too few, ensuring that the number of filtered videos remains stable within a reasonable range. For example... Figure 4 As shown, the distribution of similarity scores generally shows a higher number of values in the middle and a lower number of values at the extremes. Through clustering, high-similarity videos are grouped into one category, and low-similarity videos are grouped into another, such as K1, K2, ..., Kn, where the number of categories equals the number of videos. By selecting the similarity of each cluster center, we transform the problem of sorting the similarity of an entire database into the problem of sorting K similarities, significantly accelerating the automatic threshold calculation.
[0079] In one embodiment, the step of calculating the total relevance of each video based on the relevant frames includes:
[0080] S601. Sort all the relevant frames and record the similarity and location information of each video relevant frame;
[0081] S602, According to the formula Calculate the total relevance for each video, where rel(i) represents the similarity of the i-th frame, Ri i The position sorting result of the i-th frame is given, and m is the total number of relevant frames in a video.
[0082] In this embodiment, all relevance scores are first sorted from high to low, and the similarity and location information of each relevant video frame are recorded. Then, the formula is used to... The overall relevance of the videos is calculated, and finally, the videos are ranked according to their average relevance. This is the final video ranking result. In the formula, rel(i) represents the similarity of the i-th frame, and R... i The position sorting result of the i-th frame is given, and m is the total number of relevant frames in a video.
[0083] The calculation process of this embodiment can be understood in detail through an example below.
[0084] Typically, the similarity of the selected relevant frames from each video will vary, and some videos may not have any relevant frames, such as... Figure 5 As shown, there are 3 videos, each with related frames, and the similarity is not the same. By using the method in this embodiment, we will determine the total relevance of each video and obtain the order of the video search results.
[0085] The similarity of the three videos containing related frames is sorted in descending order, resulting in the following table.
[0086]
[0087] Thus, the similarity and positional relationship of each related frame in the video can be represented by the following table:
[0088]
[0089] The sum of relevance for each video is calculated as follows:
[0090]
[0091]
[0092]
[0093] In one embodiment, the steps described above, which simultaneously calculate the average relevance of each video and perform a final ranking to obtain the final results of the video search, include:
[0094] S701. Divide the total relevance of the video by the number of relevant frames in the video to obtain the average relevance of the video. Sort the videos according to the average relevance to obtain the final search ranking result of the videos.
[0095] In this embodiment, based on the total relevance of the video that has already been obtained, the average relevance of the video is obtained by dividing the total relevance of the video by the number of relevant frames of the video. The videos are then sorted according to the average relevance to obtain the final search ranking result of the video.
[0096] The final detailed calculation results are shown in the table below:
[0097]
[0098] The final video search results show Video 2, Video 3, and Video 1. If sorted by average video similarity, Video 2 should be last, while Video 1 and Video 3 are indistinguishable in order.
[0099] As can be seen from the examples above, it is impossible to sort videos 1 and 3 by average similarity, and the influence of frames with high similarity in the video is ignored. If only the maximum similarity among the relevant frames of each video is taken as the video similarity for comparison, the information of most other relevant frames is ignored, which will also affect the accuracy of sorting. However, the video similarity calculation method in this embodiment takes into account the influence of frame similarity and sorting position, increases the weight of frames ranked higher, and can also comprehensively balance the influence of all relevant frames in the video by calculating the average relevance of the video.
[0100] Reference Figure 2 In one embodiment, the above-described apparatus for multimodal video search and ranking includes:
[0101] Processing module 100 is used to perform feature processing on the base video data to obtain the base feature vector, and to perform multimodal model processing on the data to be queried to obtain the query feature vector;
[0102] Vector similarity module 200 is used to calculate the similarity between the query feature vector and the base database feature vector;
[0103] The automatic threshold module 300 is used to perform K-means clustering on all the similarities, set an automatic threshold according to the similarity ranking of the cluster centers obtained after clustering, and obtain the relevant frames.
[0104] The relevance calculation module 400 is used to calculate the total relevance of each video based on the relevant frames, and at the same time calculate the average relevance of each video and perform final sorting to obtain the final result of video search.
[0105] In one embodiment, the processing module 100 includes:
[0106] The filtering unit is used to extract frames from the videos in the base library at video intervals, and filter out duplicate images that exceed the threshold and images whose range of variation is lower than the preset threshold.
[0107] The secondary filtering unit is used to extract features from the filtered image, filter frames with high similarity between adjacent frames, and obtain the final video keyframes.
[0108] The feature storage unit is used to store the video keyframe features in a vector database to obtain the base database feature vector.
[0109] In one embodiment, the data to be queried includes text data and / or image data to be queried; the processing module 100 further includes:
[0110] A multimodal processing unit is used to process the text data and / or image data to be queried through a multimodal model to obtain a 512-dimensional query feature vector.
[0111] In one embodiment, the vector similarity module 200 described above includes:
[0112] Formula calculation unit, used to calculate formulas Calculate the similarity S between the query feature vector and the base database feature vector, where Q is the query feature vector and G is the base database feature vector.
[0113] In one embodiment, the automatic threshold module 300 includes:
[0114] The sample selection unit is used to randomly select k similar samples from the dataset as initial cluster centers, where k is a hyperparameter representing the number of clusters;
[0115] The calculation and classification unit is used to calculate the distance from each similarity sample in the dataset to the k cluster centers, and assign it to the class corresponding to the cluster center with the smallest distance;
[0116] A recalculation unit is used to recalculate the cluster center position for each of the clusters;
[0117] The loop unit is used to calculate the distance from each sample in the dataset to the k cluster centers and assign it to the class corresponding to the cluster center with the smallest distance, until the preset condition is met, and then output the sample status of the current K cluster centers, thereby obtaining the similarity of the K cluster centers.
[0118] The threshold generation unit is used to sort the similarity of the K cluster centers in descending order, take the similarity at K / 3 as the automatic threshold, and select the frames that are greater than the automatic threshold as relevant frames.
[0119] In one embodiment, the relevance calculation module 400 includes:
[0120] A frame bit recording unit is used to sort all the related frames and record the similarity and position information of each video related frame;
[0121] The total relevance calculation unit is used to calculate the total relevance based on the formula. Calculate the total relevance for each video, where rel(i) represents the similarity of the i-th frame, Ri i The position sorting result of the i-th frame is given, and m is the total number of relevant frames in a video.
[0122] In one embodiment, the relevance calculation module 400 further includes:
[0123] The average relevance calculation unit is used to divide the total relevance of the video by the number of relevant frames in the video to obtain the average relevance of the video. The videos are then sorted according to the average relevance to obtain the final search ranking results.
[0124] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the process of multimodal video search and ranking. The network interface allows communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. The above-mentioned computer program, when executed by a processor, implements a method for multimodal video search ranking, comprising the following steps: performing feature processing on the base library video data to obtain a base library feature vector, and performing multimodal model processing on the data to be queried to obtain a query feature vector; calculating the similarity between the query feature vector and the base library feature vector; performing K-means clustering on all the similarities, setting an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering, and obtaining relevant frames; calculating the total relevance of each video based on the relevant frames, and simultaneously calculating the average relevance of each video and performing a final ranking to obtain the final result of the video search.
[0125] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.
[0126] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for ranking multimodal video search, comprising the following steps: performing feature processing on base library video data to obtain base library feature vectors, and performing multimodal model processing on the data to be queried to obtain query feature vectors; calculating the similarity between the query feature vectors and the base library feature vectors; performing K-means clustering on all the similarities, setting an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering, and obtaining relevant frames; calculating the total relevance of each video based on the relevant frames, and simultaneously calculating the average relevance of each video and performing a final ranking to obtain the final result of the video search.
[0127] It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0130] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for multimodal video search and ranking, characterized in that, The method includes: The base video data is processed to obtain the base feature vector, and the data to be queried is processed by a multimodal model to obtain the query feature vector. Calculate the similarity between the query feature vector and the base database feature vector; K-means clustering is performed on all the similarities, and an automatic threshold is set according to the similarity ranking of the cluster centers obtained after clustering to obtain the relevant frames; The total relevance of each video is calculated based on the relevant frames, and the average relevance of each video is calculated and sorted to obtain the final result of the video search. The data to be queried includes text data and / or image data to be queried; The step of performing multimodal model processing on the data to be queried to obtain the query feature vector includes: The text data and / or image data to be queried are processed by a multimodal model to obtain a 512-dimensional query feature vector; The step of calculating the similarity between the query feature vector and the base database feature vector includes: Through formula Calculate the similarity S between the query feature vector and the base database feature vector, where Q is the query feature vector and G is the base database feature vector; The steps of performing K-means clustering on all the aforementioned similarities, setting an automatic threshold based on the similarity ranking of the cluster centers obtained after clustering, and obtaining the relevant frames include: Randomly select k similarity samples from the dataset as initial cluster centers, where k is a hyperparameter representing the number of clusters; For each similarity sample in the dataset, calculate its distance to the k cluster centers, and assign it to the class corresponding to the cluster center with the smallest distance; For each of the aforementioned clusters, recalculate its cluster center position; Then, for each sample in the dataset, calculate its distance to the k cluster centers and assign it to the class corresponding to the cluster center with the smallest distance. Continue until the preset condition is met, and then output the sample status of the current K cluster centers to obtain the similarity of the K cluster centers. The similarity of the K cluster centers is sorted in descending order, and the similarity at K / 3 is taken as the automatic threshold. Frames with similarity greater than the automatic threshold are selected as relevant frames. The step of calculating the total relevance of each video based on the relevant frames includes: Sort all the relevant frames and record the similarity and location information of each relevant video frame; According to the formula Calculate the total relevance for each video, where, Represents the similarity of the i-th frame. The position sorting result of the i-th frame is given, and m is the total number of relevant frames in a video.
2. The method for multimodal video search and ranking according to claim 1, characterized in that, The step of performing feature processing on the base video data to obtain the base feature vector includes: The videos in the base library are subjected to frame extraction at video intervals to filter out duplicate images exceeding the threshold and images with a variation range below the preset threshold. Features are extracted from the filtered images, and frames with high similarity between adjacent frames are filtered to obtain the final video keyframes. The keyframe features of the video are stored in a vector database to obtain the base feature vector.
3. The method for multimodal video search and ranking according to claim 1, characterized in that, The step of simultaneously calculating the average relevance of each video and performing a final ranking to obtain the final results of the video search includes: The average relevance of a video is obtained by dividing its total relevance by the number of relevant frames. Videos are then ranked based on their average relevance to obtain the final search ranking results.
4. An apparatus for multimodal video search and ranking, characterized in that, The apparatus is used to implement the multimodal video search and ranking method of claim 1, comprising: The processing module is used to perform feature processing on the base video data to obtain the base feature vector, and to perform multimodal model processing on the data to be queried to obtain the query feature vector; The vector similarity module is used to calculate the similarity between the query feature vector and the base database feature vector; An automatic thresholding module is used to perform K-means clustering on all the similarities, and set an automatic threshold according to the similarity ranking of the cluster centers obtained after clustering to obtain the relevant frames. The relevance calculation module is used to calculate the total relevance of each video based on the relevant frames, and at the same time calculate the average relevance of each video and perform the final sorting to obtain the final result of the video search.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.