A multimodal large model driven video understanding and retrieval method
Through the combination of multimodal large-modal model-driven video understanding and SBERT model, the problem of insufficient manual labeling dependence and semantic understanding in video retrieval is solved, efficient understanding and accurate retrieval of massive video data is achieved, and scene understanding and retrieval are improved. accuracy.
Patent Information
- Application Number
- CN202411276188.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Existing video retrieval methods rely on large-scale manual labeling data to be difficult to scale, semantic understanding and scene recognition capabilities are insufficient, it is difficult to effectively integrate multimodal information for searching, and different search methods are difficult to unify.
The video understanding method driven by multimodal large model is adopted, and the video is converted into text description through speech recognition and multimodal model, and vector search and full-text search are combined with the SBERT model to integrate the advantages of different search methods.
It realizes the understanding and retrieval of massive video data, improves the ability to understand complex semantics and scenes, reduces the dependence on manual labeled data, and improves the retrieval accuracy and generalization capabilities.
Smart Images

Figure CN118779492B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video retrieval technology, and in particular to a video understanding and retrieval method driven by a multimodal large model. Background Art
[0002] With the growth of video data and the widespread application of video in social media, entertainment, surveillance, and education, research on how to efficiently understand and retrieve video content can enable the system to extract valuable information from massive data. Video understanding and retrieval technology has made great progress in recent years, especially the introduction of deep learning and large model technology, which enables these technologies to show higher accuracy and robustness when processing complex video content. For example, large models such as GPT-4V can learn richer and more abstract representations from picture frames through large-scale pre-training, improving the performance of video understanding and retrieval.
[0003] The development of the automatic speech recognition system (ASR) converts speech into text, enabling the system to process and understand the user's voice input and effectively combine it with text data. For example, DeepSpeech developed by Baidu and Whisper released by OpenAI. The speech recognition system can extract the speech in each video, convert the speech into text, and segment the text.
[0004] The success of large-scale pre-trained models, such as the Transformer model and the BERT (Bidirectional Encoder Representations from Transformers) model in natural language processing, has also led to progress in the application of multimodal understanding. The SBERT (Sentence-BERT) model is a variant of the BERT model. By optimizing the Siamese Network and the Triplet Loss function, it is more efficient in calculating the similarity between sentence pairs. A common way to apply the SBERT model to video understanding is to convert video slices into text descriptions, and then use the SBERT model to calculate the embedding vector of each text description.
[0005] At the same time, multimodal models that combine images and text are also making great progress. For example, the CLIP model released by OpenAI achieves cross-modal understanding by learning how to compare text and images. In the field of video understanding, automatic understanding and description of video content can be achieved by sampling video frames and generating image descriptions through visual-text models (such as GPT-4V).
[0006] In the field of video retrieval, video content can be converted into feature vectors, and efficient index structures (such as inverted index, KD tree, LSH, etc.) can be used to accelerate the retrieval process. At the same time, for video feature matching and similarity calculation, similarity metrics (such as Euclidean distance, cosine similarity) can be used to calculate the similarity between query features and features in the video library, so as to find the video most similar to the query.
[0007] There are some video retrieval methods in the prior art. For example, prior art 1 maps the extracted video features into the semantic space by constructing a deep neural network model to generate a semantic vector; prior art 2 processes the narration audio through a large language model to obtain the operation step text, calculates the frame-by-frame similarity of the operation step text, and determines the target video segment in each video segment; prior art 3 performs video retrieval based on attention and knowledge distillation; prior art 4 performs cross-modal video retrieval based on dual-branch dynamic distillation learning, and handles cross-modal video retrieval from a new perspective. However, the above prior art also faces the following problems: 1) The existing methods rely on the training of large-scale manually annotated video data, which is difficult to expand to massive video data. 2) The existing video retrieval system still has deficiencies in semantic understanding and scene recognition, especially in the understanding of complex scenes and abstract concepts. Conventional feature extraction methods often fail to capture high-level semantic information. 3) Video data is inherently multimodal, containing visual and auditory information. How to effectively integrate these multimodal information for retrieval is a major challenge. 4) Various retrieval methods search for video features with different focuses. How to effectively integrate these different retrieval methods and perform unified retrieval is also a major challenge. Summary of the invention
[0008] In order to solve the above technical problems, the present invention provides a video understanding and retrieval method driven by a multimodal large model. The present invention is based on the following two aspects when performing video understanding: on the one hand, the voice in the video is extracted and converted into text, and then the text is segmented and the labels are recorded, so that the system can integrate multimodal information for understanding; on the other hand, the video is frame sampled, and the multimodal large model is used to convert the frame group into text description, which enhances the system's ability to understand complex semantics and scenes. The present invention is based on the following two aspects when performing video retrieval: first, for each video clip, the SBERT model is used to calculate its vector and perform vector retrieval; second, full-text retrieval is performed on the text in the video clip. The present invention combines these retrieval methods to perform hybrid retrieval, integrating the advantages of different methods and returning the best results.
[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0010] A multimodal large model driven video understanding and retrieval method comprises the following steps:
[0011] Step 1: Preprocess the video files to be retrieved:
[0012] Segment the video file to obtain video clips; perform frame sampling on each video clip at a set frequency to form a frame group corresponding to the video clip; add a silent audio track to the video clip without audio;
[0013] Step 2: extracting audio files from each video clip, and using a pre-trained speech recognition model to convert the speech in the audio files into text as a text representation of the auditory elements of the video clip;
[0014] Step 3: Using the pre-trained multimodal large model, convert the image information of each frame image in the frame group corresponding to the video clip into text as the text representation of the visual element of the video clip;
[0015] Step 4: merge the text representation of the auditory elements of the video clip and the text representation of the visual elements of the video clip to obtain the text representation of the video clip, calculate the vector of the text representation of each video clip through the SBERT model, and obtain the encoding vector of the video clip; store the encoding vector of the video clip and the text representation of the video clip in the database to complete the video understanding process;
[0016] Step 5: Query the video clips in the database through natural language query statements: calculate the query statement vector through the SBERT model, and perform vector retrieval through the query statement vector and the video clip encoding vector in the database; input the query statement into the large language model, parse the keywords in the query statement, and perform full-text retrieval through the keywords and the text representation of the video clips in the database; use the inverse sort fusion to comprehensively sort the results of the vector retrieval and the full-text retrieval results to give the final retrieval results, completing the video retrieval process.
[0017] Furthermore, the image information of the image includes:
[0018] Semantic information, including objects in the image identified by object detection, scene types in the image identified by scene classification, and human behaviors in the image analyzed by behavior recognition;
[0019] Color information refers to the color of each element in the image;
[0020] Texture information refers to the statistical description of the surface characteristics of each element in the image;
[0021] Shape information refers to the outline and structure of each element in the image;
[0022] Spatial information, including position information and composition information; the position information refers to the specific position of each element in the image; the composition information refers to the arrangement and layout of each element in the image;
[0023] The elements include objects, characters and scenes.
[0024] Furthermore, in step 5, the full-text search is performed using keywords and text representations of video clips in the database, specifically including:
[0025] An inverted index table is constructed for the text representation of the video clips in the database; when performing a full-text search, the keywords are segmented and then searched in the inverted index table.
[0026] Compared with the prior art, the beneficial technical effects of the present invention are:
[0027] 1. Compared with the existing video retrieval methods that require large-scale manually annotated video data, the present invention uses a multimodal large model to understand the video content. It does not require large-scale manually annotated video data, has higher generalization ability, and is easier to expand to massive video data in the real world.
[0028] 2. The present invention uses a pre-trained speech recognition model to convert the audio of the video into text form, so as to understand the video from an auditory perspective.
[0029] 3. The present invention performs frame sampling on the video and converts it into text description using a large multimodal model, so as to understand the video from a visual perspective, enhance the understanding of the complex semantics of the video and the understanding of the scene, achieve better retrieval effects, and have a certain degree of interpretability.
[0030] 4. The present invention comprehensively utilizes the visual elements and auditory elements of the video for multimodal understanding, and effectively integrates different retrieval methods for hybrid retrieval, thereby improving the accuracy of the retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A flowchart for video understanding of the present invention;
[0032] Figure 2 Flowchart of video retrieval for the present invention. DETAILED DESCRIPTION
[0033] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.
[0034] The present invention mainly aims at the problem of video clip retrieval and provides a fine-grained video clip retrieval method that does not rely on video annotation information. The purpose is to use a multimodal large model to understand the video content, including its visual elements and auditory elements, and build a video database based on the understood video content. During retrieval, a hybrid retrieval of full-text retrieval and vector retrieval is performed based on query statements in natural language form to determine the target video clip.
[0035] In order to solve the problems of high feature training cost, weak complex scene understanding ability, and difficulty in video information retrieval in traditional video retrieval methods, the present invention adopts a combination of video understanding technology based on a multimodal large model and natural language hybrid retrieval technology. The natural language description of the video clips is obtained through the video and image understanding capabilities of the large model, and then vectorization is used for data storage and retrieval.
[0036] The present invention aims at the problem of video understanding and retrieval. First, multiple video files are preprocessed (see step 1). Then, a video understanding method is proposed, which can understand the speech context information of the video (see step 2) and obtain the image information of the video (see step 3). After combining the two important information, a natural language sentence query is realized, that is, the corresponding target video segment is located according to the input natural language query sentence, and the specific time point in the target video segment can be marked (see step 4).
[0037] In summary, the present invention includes four steps: (1) integrating video files to be retrieved from multiple sources and multiple fields, performing necessary preprocessing, and obtaining video clips; (2) extracting the speech from each video clip, using a pre-trained speech recognition model to convert the speech into text, and obtaining a text representation of the auditory elements of the video. The speech recognition model here can be a pre-trained speech recognition model such as DeepSpeech and Whisper; (3) using a multimodal large model to understand the video picture, converting the video clip into text, and obtaining a text representation of the visual elements of the video. The multimodal large model here can be a commercial reasoning interface (API) such as GPT-4o and Claude3.5, or an open source pre-trained local reasoning model such as GLM-4V; (4) for a query statement in natural language form, using the SBERT model to directly calculate the vector of the query statement, and performing vector retrieval in the vector index in the database; further, the present invention uses the large model to parse out the keywords in the query statement, and then performs full-text retrieval, combining the two retrieval methods to achieve hybrid retrieval.
[0038] The details of steps 1 to 4 are as follows:
[0039] Step 1: Preprocessing of multi-source, multi-domain video files. During the testing phase, the video data sources are diversified. The video files of the present invention include videos of different scenes, environments, camera angles and lighting conditions to ensure the applicability of the system to different types of videos. The video sources involved in the present invention are rich, including intersection monitoring video data, long videos and short videos disclosed on the Internet, and open-source test videos in scientific and technological literature; the content covers multiple fields; the duration ranges from 30 seconds to 30 minutes. For the above-mentioned videos to be retrieved, the present invention adopts a unified preprocessing method: (1) The video is segmented at fixed time intervals (such as 30 seconds) to form video clips, and 5 seconds of redundancy is left between video clips to avoid missing important information; (2) For each video clip, frame sampling is performed at a certain frequency (such as extracting 1 frame per second) to form a frame group corresponding to the video clip; (3) For each video clip, its audio file is extracted, and for video clips that have not yet added audio, a soundless audio track is automatically added to it.
[0040] Step 2: Auditory information understanding. For the speech in the audio file of each video clip, the pre-trained speech recognition model is used to convert the speech into text. For the extracted auditory element text representation, the corresponding encoding is performed in step 3.
[0041] Step 3: Visual information understanding. Each frame of a video often contains a lot of information, including semantic information, color information, texture information, shape information, and spatial information. Multimodal large models such as Claude3.5, GPT-4o, and GLM-4v have strong image understanding capabilities and can completely convert image information into text, and then encode it to obtain semantic representation. The information extracted in this step covers a wealth of video information, including color information (the color of each element), texture information (statistical description of the surface features of each element), shape information (the outline and structure of each element); spatial information, including location information and composition information; semantic information, including objects in the image identified and located through object detection, such as face recognition, vehicle recognition, etc., scene types in the video identified through scene classification, such as indoor, outdoor, urban, natural, etc., and human behaviors in the video analyzed through behavior recognition, such as running, jumping, shaking hands, etc.
[0042] Furthermore, the texture information includes both the texture of the element surface in the usual sense, even if the element surface presents uneven grooves, and the color pattern on the smooth surface of the element, where the color pattern is different from the color mentioned above.
[0043] Textures have three main flags:
[0044] 1) A certain local sequence is repeated over a larger area than the sequence;
[0045] 2) The sequence is composed of non-random arrangements of basic elements;
[0046] 3) Each part is roughly a uniform entity, with roughly the same structural size anywhere in the texture area;
[0047] For example, “a girl wearing a skirt with a floral pattern on it”, here “floral pattern” is also considered a texture.
[0048] Furthermore, the position information in the spatial information indicates the specific position of each element in the image, that is, the coordinates in the two-dimensional space. The position information in the spatial information refers to the arrangement and layout of each element in the image, and the elements include objects, people, scenes, etc.
[0049] In this stage, the multimodal large model is used to convert the video clips into text representations of visual elements, and the text representations of auditory elements in step 2 are combined to obtain the text representations of the video clips. For the text representation of each video clip, the SBERT model is used to calculate its vector representation and encode the text representation of the video clip. Steps 1 to 3 are the video understanding process. For the overall process, see Figure 1 .
[0050] Step 4: Video segment retrieval. The purpose of the retrieval system is to find the most matching video segment in the video database based on the user's natural language query. Figure 2 The retrieval system includes two aspects: on the one hand, the SBERT model is used to directly calculate the vector of the query statement, and the vector search is performed in the built vector index; on the other hand, the large language model is used to parse the keywords in the query statement in natural language form, and then a full-text search is performed. For the above two retrieval methods, the present invention adopts reciprocal sort fusion (RRF) for hybrid retrieval, and finally obtains a comprehensive ranking of the two retrieval results and gives the positioning. The advantage of reciprocal sort fusion is that it does not use the relevant score, but only relies on the ranking calculation, which can well integrate the two different retrieval methods. According to the query statement entered by the user, the above hybrid retrieval strategy is used to obtain the matching item with the largest comprehensive similarity, and output the ID and time point of its video clip to achieve the effect of precise positioning.
[0051] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.
[0052] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
Claims
1. A multimodal large model driven video understanding and retrieval method, characterized by: The following steps are involved: Step 1: Preprocess the video files to be retrieved: Segment the video file to obtain video segments; perform frame sampling on each video segment at a set frequency to form a frame group corresponding to the video segment; add a silent audio track to the video segment without audio; Step 2: Extract audio files from each video clip and use a pre-trained speech recognition model to convert the speech in the audio files into text as the text representation of the auditory elements of the video clip; Step 3: Using the pre-trained multimodal large model, the image information of each frame in the frame group corresponding to the video clip is converted into text as the text representation of the visual elements of the video clip; Step 4: Merge the text representation of the auditory elements of the video clip and the text representation of the visual elements of the video clip to obtain the text representation of the video clip. Use the SBERT model to calculate the vector of the text representation of each video clip to obtain the video clip encoding vector; store the video clip encoding vector and the text representation of the video clip in the database to complete the video understanding process; Step 5: Query the video clips in the database using a natural language query. The SBERT model calculates the query vector, and performs vector retrieval by comparing the query vector with the encoded vectors of the video clips in the database. The query is then fed into the large language model, which parses the keywords in the query. Full-text retrieval is then performed using the keywords and the text representation of the video clips in the database. The results of vector retrieval and full-text retrieval are comprehensively sorted using reciprocal sort fusion to give the final retrieval results and complete the video retrieval process.
2. The multimodal large model driven video understanding and retrieval method according to claim 1, characterized in that: The image information of an image includes: Semantic information, including objects in images identified through object detection, scene types in images identified through scene classification, and human behaviors analyzed through behavior recognition; Color information refers to the color of each element in the image; Texture information refers to the statistical description of the surface characteristics of each element in the image; Shape information refers to the outline and structure of each element in the image; Spatial information, including position information and composition information; the position information refers to the specific position of each element in the image; the composition information refers to the arrangement and layout of each element in the image; The elements include objects, characters and scenes.
3. The multimodal large model driven video understanding and retrieval method according to claim 1, characterized in that: In step 5, the full-text search is performed using keywords and text representations of video clips in the database, specifically including: An inverted index table is constructed for the text representation of the video clips in the database; when performing a full-text search, the keywords are segmented and then searched in the inverted index table.
