A video retrieval method and system based on multimodal information fusion

By vectorizing video data and combining the weighted fusion of multimodal large models, the problem of insufficient information utilization in traditional video search methods is solved, more accurate video search results and user intention recognition are achieved, and video retrieval is improved accuracy and efficiency.

CN118626672BActive Publication Date: 2024-11-05SHANDONG INSPUR SCI RES INST CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411095068.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-05
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

Traditional video retrieval methods rely on single modal information and cannot effectively utilize the multimodal characteristics of video, resulting in the search results being not comprehensive and accurate enough, and it is difficult to understand the user's query intention, and cannot provide results that truly meet user needs.

Method used

By obtaining video data, including video digest, audio, subtitles, covers and keyframes, it is vectorized and mapped to high-dimensional vector space, combined with multimodal large models for weighted fusion, and matching the optimal search scheme according to user intentions. Realize video retrieval.

Benefits of technology

It improves the representation accuracy of video key information and the accuracy of video retrieval, and provides more accurate and efficient video content retrieval services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626672B_ABST
    Figure CN118626672B_ABST
Patent Text Reader

Abstract

The present invention proposes a video retrieval method and system based on multimodal information fusion, which belongs to the technical field of artificial intelligence equipment, and includes: acquiring a video and processing it to obtain video data, the video data including video summary, audio, subtitles, cover and key frames; vectorizing the video data to obtain a video data vector, and mapping it to the same high-dimensional vector database for storage; acquiring a user query, analyzing the user query to obtain the user intent, matching the optimal retrieval scheme according to the user intent, and retrieving a target video from the vector database; achieving accurate representation of key video information, improving the ability to extract key video information and the accuracy of video retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a video retrieval method and system based on multimodal information fusion. Background Art

[0002] With the rapid development of Internet technology and the widespread dissemination of multimedia content, video has become one of the main media for people to obtain information, entertainment and education. With the rapid growth of the number of videos, efficient and accurate video retrieval technology has become particularly important. However, traditional video retrieval methods usually rely on single-modal information of videos, such as text descriptions or metadata of videos, ignoring the multimodal characteristics of videos themselves, such as images, audio and subtitles. This approach leads to incomplete and inaccurate retrieval results. In addition, due to the lack of effective means to understand the user's query intent, traditional video retrieval systems often find it difficult to provide results that truly meet user needs. In recent years, with the advancement of deep learning and natural language processing technology, especially the development of multimodal large models, new ideas and technical means have been provided to solve the above problems. These models can process data from different modalities and map them into a unified high-dimensional vector space, so as to better retain and utilize the multidimensional information of videos. Despite this, current technologies still face severe challenges. How to effectively extract different modal information of videos and achieve more efficient fusion utilization needs to be solved urgently. Summary of the invention

[0003] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0004] In a first aspect, the present invention provides a video retrieval method based on multimodal information fusion, comprising:

[0005] Acquire the video and process it to obtain video data, which includes video summary, audio, subtitles, cover and key frames;

[0006] Vectorize the video data to obtain video data vectors, and map them to the same high-dimensional vector database for storage;

[0007] Obtain user queries, analyze the user queries to obtain user intent, match the best retrieval scheme according to the user intent, and retrieve the target video from the vector database; the retrieval scheme is obtained by pre-setting the weights of multiple vectors in the video data vector and performing weighted fusion.

[0008] Furthermore, the steps of obtaining the video and processing the video data include: generating a video summary, splitting the audio and video channels, processing the audio, processing the cover, and extracting key frames;

[0009] Video summary generation: The video information is summarized by combining the multimodal large model tool with the prompt engineering to obtain the video summary;

[0010] Audio and video channel splitting: Use built-in tools to split video files of various formats into channels to obtain video and audio information; built-in tools can use PCIE channel splitting.

[0011] Audio processing: Convert audio to text using the ASR algorithm to obtain text subtitles;

[0012] Cover processing: extract the video cover image and obtain the cover text and cover image through the OCR recognition model;

[0013] Keyframe Extraction: Outputs a list of video keyframes by inputting the video into the keyframe extraction model.

[0014] Furthermore, the step of vectorizing the video data to obtain a video data vector and mapping it to the same high-dimensional vector database for storage includes:

[0015] The video summary, audio information, text subtitles, cover text, cover image, and key frame list are vectorized by a multimodal model to obtain summary vectors, audio vectors, subtitle vectors, cover text vectors, cover image vectors, and key frame vectors;

[0016] The obtained summary vector, audio vector, subtitle vector, cover text vector, cover image vector, key frame vector and corresponding metadata video summary, text subtitle, cover text are packaged and stored in the vector database.

[0017] Furthermore, the step of obtaining user queries and analyzing the user queries to obtain user intentions includes: analyzing the user queries through a large model Conduct analysis to determine user intent.

[0018] Furthermore, the step of matching the best search scheme according to the user intention includes: setting different search schemes according to the user intention, sending the user question into the embedding model for vectorization to obtain the query vector, and calculating the similarity score between the query vector and the summary vector, the audio vector, the subtitle vector, the cover text vector, the cover image vector, and the key frame vector respectively. - The final score is calculated by the weightedranker strategy :

[0019]

[0020] in, is the similarity score corresponding to the vector, The corresponding weight is the weight corresponding to the vector;

[0021] Sort the retrieved videos according to the final score S to get the relevant videos.

[0022] In a second aspect, the present invention provides a video retrieval system based on multimodal information fusion, comprising:

[0023] The data processing module is configured to: acquire the video and process it to obtain video data, where the video data includes video summary, audio, subtitles, cover and key frames;

[0024] The database establishment module is configured to: vectorize the video data to obtain video data vectors, and map them to the same high-dimensional vector database for storage;

[0025] The query retrieval module is configured to: obtain user queries, analyze user queries to obtain user intent, match the optimal retrieval scheme according to user intent, and retrieve the target video from the vector database; the retrieval scheme is obtained by pre-setting the weights of multiple vectors in the video data vector and performing weighted fusion.

[0026] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the video retrieval method based on multimodal information fusion described in the first aspect.

[0027] In a fourth aspect, the present invention provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the video retrieval method based on multimodal information fusion described in the first aspect.

[0028] Compared with the prior art, the advantages and positive effects of the present invention are:

[0029] The present invention processes video summaries, audio, subtitles, covers, key frames and other information and maps them to the same high-dimensional vector space for storage, and performs weighted fusion of different vectors according to different retrieval requirements to generate retrieval tools with multiple combinations. It further performs intent recognition through intelligent agents based on user input of pictures or texts, distributes the input to designated retrieval tools, and performs intent recognition judgment, thereby achieving accurate representation of key video information and improving the ability to extract key video information and the accuracy of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 The present invention is a flowchart of a video retrieval method based on multimodal information fusion. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0034] Terminology explanation:

[0035] Multimodal large model: Multimodal large model can process and fuse information from different modalities and map it to the same high-dimensional vector space, thereby retaining and utilizing the multidimensional information of the video to a greater extent. This method can not only improve the accuracy of video retrieval, but also better understand and identify the user's query intent and provide retrieval results that better meet user needs. The development of multimodal large model provides new possibilities for video retrieval technology.

[0036] The video retrieval method of multimodal information fusion can process various information such as video summary, audio, subtitles, cover, keyframes, etc., and uniformly map them to high-dimensional vector space for storage. According to different retrieval requirements, these vectors can be weighted fused to generate various combinations of retrieval tools. When users input pictures or texts, the intelligent agent can recognize the intent, distribute the input to the specified retrieval tool, perform intent recognition judgment, and match the optimal retrieval scheme for video retrieval.

[0037] For example, the audio information in the video can be converted into text subtitles by using the advanced ASR (Automatic Speech Recognition) algorithm, the text information can be extracted from the video cover by the OCR (Optical Character Recognition) model, and the key frames of the video can be extracted by the key frame extraction model. After the information is processed by the multimodal embedding model, the corresponding vector is generated and stored in the database. During retrieval, the user query is analyzed for intent, the appropriate retrieval scheme is selected, and the retrieval is performed according to the weight of different modal information to finally obtain relevant videos.

[0038] This video retrieval method based on multimodal information fusion not only improves the accuracy and efficiency of video retrieval in theory, but also provides users with more accurate and efficient video content retrieval services in practice.

[0039] Embodiment 1, as Figure 1 As shown, the present application provides a video retrieval method based on multimodal information fusion, including: processing video summaries, audio, subtitles, covers, key frames and other information and mapping them to the same high-dimensional vector space for storage, and weighted fusion of different vectors according to different retrieval requirements to generate multiple combinations of retrieval tools, and further based on the user input of pictures or texts, perform intent recognition through intelligent agents, distribute the input to the designated retrieval tool, and perform intent recognition judgment. The specific steps are as follows:

[0040] Step 1: Summary generation: Use the multimodal large model tool combined with the prompt engineering to summarize the video information and obtain the video summary. .

[0041] Step 2: Split the audio and video channels. Use the built-in tool to split the video files of various formats into channels to obtain video information. and audio information .

[0042] Step 3: Audio processing: Use the ASR algorithm to convert the audio into text to obtain text subtitles .

[0043] Step 4: Cover processing: extract the video cover image and obtain the cover text through the OCR recognition model and cover image .

[0044] Step 5: key frame extraction: output the video key frame list by inputting the video into the key frame extraction model .

[0045] Step 6: Vectorize the data and summarize the video using a multimodal embedding model , audio information , Text subtitles , Cover Text , Cover Image , Keyframe List Vectorize and get the summary vector , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector .

[0046] Step 7: Create a video library and convert the summary vector obtained in step 6 , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector and corresponding metadata video summary , Text subtitles , Cover Text Pack and store in vector database.

[0047] Step 8: Intent recognition, querying users through a large model Conduct analysis to determine user intent.

[0048] Step 9: Video retrieval. Set different retrieval schemes according to user intent. Each retrieval scheme has a certain effect on the summary vector. , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector Different weights are set, and finally the optimal retrieval scheme is matched according to the user's intention to retrieve the target video.

[0049] Among them, step 3 uses the paraformer model to analyze the audio information Perform feature extraction and output as a text The timestamp of each word is used to further segment the text using punctuation marks, and the time interval corresponding to the text is given according to the timestamp of the beginning and end characters of each text segment, thereby generating subtitle information. .

[0050] Step 4: Use the OCR model to perform text recognition on the cover image, and further organize and splice the OCR results through the large language model to obtain the cover text Or directly process the cover image through a multimodal large model to obtain the cover text .

[0051] Step 5: Use the TransNetV2 model to analyze the video information Perform inference recognition and output a list of video key frames .

[0052] Step 8, collect a batch of queries to classify the intent, such as image search video category, sound retrieval category, picture retrieval category, knowledge retrieval category, etc., and set the prompt words of the intent recognition module according to these intent classifications.

[0053] Step 9: Set different retrieval schemes according to user intent and feed the user question into the embedding model for vectorization to obtain the query vector. , respectively calculate its summaries vector , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector Similarity score - The final score is calculated by the weightedranker strategy :

[0054]

[0055] in, is the similarity score corresponding to the vector, The corresponding weight is the vector.

[0056] Sort the retrieved videos according to the final score S to obtain related videos.

[0057] The present invention improves the representation accuracy of key video information by fusing multimodal information such as video summary, audio, subtitles, cover, key frames, etc. It also improves the ability to extract key video information and the accuracy of video retrieval by performing intent analysis and retrieval tool allocation based on user questions through intelligent agents.

[0058] A video retrieval method based on multimodal information fusion, specifically taking multimedia video retrieval as an example, comprises the following steps:

[0059] Establish a multimedia video library and use the Video-LLaMa model to summarize the video information and obtain the video summary .

[0060] Use the ffmpeg tool to split the audio channel and video channel of the video in the database to obtain video information and audio information .

[0061] For audio information, the paraformer speech translation model is used to obtain a text segment and the start and end timestamps corresponding to each character. The langchain tool is used to segment the text according to punctuation marks to obtain several text segments. The start timestamp of the first character and the end timestamp of the last character of each text segment are extracted as the start and end timestamps of the text segment. Finally, all text segments and their corresponding time intervals are integrated to generate subtitle text. .

[0062] Extract the cover image of the first frame of the video for OCR recognition to obtain several text segments, which are then organized and spliced ​​through a large language model to obtain the cover text If the OCR result is empty, the cover is summarized through the multimodal large model to obtain the cover text , and the BM25 algorithm is used to calculate the cover text With video summary and subtitle text Similarity Score:

[0063]

[0064] It is a document With query The relevance score of . Yes Query The i-th term in , It is a term The inverse document frequency of It is a term The frequency of occurrence in document D, is the length of document D, is the average length of all documents in the corpus, and is a tuning parameter, usually .0 and .

[0065] If the score is less than 5, use Replaces the Cover Text.

[0066] Through the TransNetV2 model, video information Perform inference recognition and output a list of video key frames .

[0067] Video Summarization via Multimodal Embedding Model ImageBind , audio information , Text subtitles , Cover Text , Cover Image , Keyframe List Vectorize and get the summary vector , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector .

[0068] The summary vector obtained in step 6 , audio vector , subtitle vector , Cover text vector , Cover Image Vector , keyframe vector and corresponding metadata video summary , Text subtitles , Cover Text Package and store into vector database to build multimedia video vector library.

[0069] By collecting a batch of queries, we can classify the intents, such as searching for videos by image, searching for sounds by pictures, searching for knowledge by pictures, etc. We can set the prompt words for the intent recognition module based on these intent classifications. The following is an example of a prompt word template:

[0070] Prompt word template

[0071]

[0072] Different retrieval schemes are set according to user intent, and the user question is sent to the embedding model for vectorization to obtain the query vector. The similarity scores between the query vector and the summary vector, audio vector, subtitle vector, cover text vector, cover image vector, and key frame vector are calculated respectively. The final score S is calculated by the weightedranker strategy:

[0073]

[0074] in, is the similarity score corresponding to the vector, The corresponding weight is the vector.

[0075] Sort the retrieved videos according to the final score S to get the relevant videos.

[0076] Embodiment 2, the present invention provides a video retrieval system based on multimodal information fusion, comprising:

[0077] The data processing module is configured to: acquire the video and process it to obtain video data, where the video data includes video summary, audio, subtitles, cover and key frames;

[0078] The database establishment module is configured to: vectorize the video data to obtain video data vectors, and map them to the same high-dimensional vector database for storage;

[0079] The query retrieval module is configured to: obtain user queries, analyze user queries to obtain user intent, match the optimal retrieval scheme according to user intent, and retrieve the target video from the vector database; the retrieval scheme is obtained by pre-setting the weights of multiple vectors in the video data vector and performing weighted fusion.

[0080] Embodiment 3, a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods of Embodiment 1.

[0081] Embodiment 4, the present invention provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute any method of Embodiment 1.

[0082] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0083] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0084] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0085] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0086] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0087] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A video retrieval method based on multimodal information fusion, characterized in that: include: Acquire the video and process it to obtain video data, which includes video summary, audio, subtitles, cover and key frames; The steps of obtaining video and processing it to obtain video data include: video summary generation, audio and video channel splitting, audio processing, cover processing and key frame extraction; video summary generation: the video information is summarized and summarized through the multimodal large model tool combined with the prompt project to obtain the video summary. The multimodal large model tool combined with the prompt project adopts the Video-LLaMa model; audio and video channel splitting: the video files of various formats are split into channels through the built-in tool to obtain video information and audio information. The built-in tool adopts the ffmpeg tool; audio processing: the audio is converted into text through the ASR algorithm to obtain text subtitles, and the paraformer speech translation model is used to obtain a text, and the start timestamp and end timestamp corresponding to each character are obtained. The text is segmented according to punctuation marks through the langchain tool to obtain several text segments, and the start timestamp of the first character and the end timestamp of the last character of each text segment are extracted as the start and end timestamps of the text, and finally all text segments and their corresponding time intervals are integrated to generate subtitle text ; Cover processing: extract the video cover image and obtain the cover text and cover image through the OCR recognition model; Key frame extraction: output the video key frame list by inputting the video into the key frame extraction model. The video input key frame extraction model adopts the TransNetV2 model, and the TransNetV2 model is used to infer and recognize the video information and output the video key frame list; Vectorize the video data to obtain a video data vector, and map it to the same high-dimensional vector database for storage; the steps of vectorizing the video data to obtain a video data vector, and mapping it to the same high-dimensional vector database for storage include: vectorizing the video summary, audio information, text subtitles, cover text, cover image, and video key frame list through the multimodal embedding model ImageBind to obtain a summary vector, an audio vector, a subtitle vector, a cover text vector, a cover image vector, and a video key frame vector; packaging the obtained summary vector, audio vector, subtitle vector, cover text vector, cover image vector, video key frame vector, and corresponding metadata video summary, text subtitles, and cover text into a vector database; Obtaining user queries, analyzing the user queries to obtain user intent, matching the best retrieval scheme according to the user intent, and retrieving the target video from the vector database; the retrieval scheme is obtained by presetting weights of multiple vectors in the video data vector and performing weighted fusion; The step of matching the best search scheme according to the user intention includes: setting different search schemes according to the user intention, sending the user question into the embedding model for vectorization to obtain the query vector, and calculating the similarity score between the query vector and the summary vector, the audio vector, the subtitle vector, the cover text vector, the cover image vector, and the key frame vector respectively. The final score is calculated by the weightedranker strategy : in, is the similarity score corresponding to the vector, The corresponding weight is the weight corresponding to the vector; Sort the retrieved videos according to the final score S to obtain relevant videos; Calculate cover text using BM25 algorithm With video summary and subtitle text Similarity Score: is the relevance score of document D to query Q, is the i-th term in the query Q, It is a term The inverse document frequency of It is a term The frequency of occurrence in document D, is the length of document D, is the average length of all documents in the corpus, and is the adjustment parameter, and .

2. The video retrieval method based on multimodal information fusion according to claim 1, characterized in that: The steps of obtaining user queries and analyzing the user queries to obtain user intentions include: analyzing the user queries through a large model Conduct analysis to determine user intent.

3. A video retrieval system based on multimodal information fusion, characterized in that: include: The data processing module is configured to: acquire the video and process it to obtain video data, where the video data includes video summary, audio, subtitles, cover and key frames; The steps of obtaining video and processing it to obtain video data include: video summary generation, audio and video channel splitting, audio processing, cover processing and key frame extraction; video summary generation: the video information is summarized and summarized through the multimodal large model tool combined with the prompt project to obtain the video summary. The multimodal large model tool combined with the prompt project adopts the Video-LLaMa model; audio and video channel splitting: the video files of various formats are split into channels through the built-in tool to obtain video information and audio information. The built-in tool adopts the ffmpeg tool; audio processing: the audio is converted into text through the ASR algorithm to obtain text subtitles, and the paraformer speech translation model is used to obtain a text, and the start timestamp and end timestamp corresponding to each character are obtained. The text is segmented according to punctuation marks through the langchain tool to obtain several text segments, and the start timestamp of the first character and the end timestamp of the last character of each text segment are extracted as the start and end timestamps of the text, and finally all text segments and their corresponding time intervals are integrated to generate subtitle text ; Cover processing: extract the video cover image and obtain the cover text and cover image through the OCR recognition model; Key frame extraction: output the video key frame list by inputting the video into the key frame extraction model. The video input key frame extraction model adopts the TransNetV2 model, and the TransNetV2 model is used to infer and recognize the video information and output the video key frame list; The database establishment module is configured to: vectorize the video data to obtain the video data vector, and map it to the same high-dimensional vector database for storage; the steps of vectorizing the video data to obtain the video data vector, and mapping it to the same high-dimensional vector database for storage include: vectorizing the video summary, audio information, text subtitles, cover text, cover image, and video key frame list through the multimodal embedding model ImageBind to obtain the summary vector, audio vector, subtitle vector, cover text vector, cover image vector, and video key frame vector; packaging the obtained summary vector, audio vector, subtitle vector, cover text vector, cover image vector, video key frame vector and corresponding metadata video summary, text subtitle, and cover text into the vector database; The query retrieval module is configured to: obtain a user query, analyze the user query to obtain the user's intention, match the optimal retrieval scheme according to the user's intention, and retrieve the target video from the vector database; the retrieval scheme is obtained by presetting the weights of multiple vectors in the video data vector and performing weighted fusion; The step of matching the best search scheme according to the user intention includes: setting different search schemes according to the user intention, sending the user question into the embedding model for vectorization to obtain the query vector, and calculating the similarity score between the query vector and the summary vector, the audio vector, the subtitle vector, the cover text vector, the cover image vector, and the key frame vector respectively. The final score is calculated by the weightedranker strategy : in, is the similarity score corresponding to the vector, The corresponding weight is the weight corresponding to the vector; Sort the retrieved videos according to the final score S to obtain relevant videos; Calculate cover text using BM25 algorithm With video summary and subtitle text Similarity Score: is the relevance score of document D to query Q, is the i-th term in the query Q, It is a term The inverse document frequency of It is a term The frequency of occurrence in document D, is the length of document D, is the average length of all documents in the corpus, and is the adjustment parameter, and .

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the video retrieval method based on multimodal information fusion as described in any one of claims 1-2.

5. An electronic device, characterized in that: It comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the video retrieval method based on multimodal information fusion as described in any one of claims 1-2.

Citation Information

Patent Citations

  • Video information processing method and device based on video information processing model

    CN112861580A

  • Method for intelligently identifying and searching programs based on audio

    CN114547373A

  • Multi-modal large model assisted unsupervised cross-modal video retrieval method and device

    CN118427396A

  • Artificial intelligence data search and distribution method and system

    CN118467851A