Multi-granularity video retrieval method and device based on multi-modal large model, computer equipment and readable storage medium

Abstract

The invention discloses a multi-granularity video retrieval method and device based on a multi-modal large model, computer equipment and a readable storage medium, and the method comprises the steps: firstly obtaining video query information input by a user, carrying out the intention recognition to obtain a query field, rewriting the query information and the field to obtain a video query vector, and carrying out the retrieval of the video query vector; and retrieving in a preset retrieval video knowledge base according to the vector and the field, and finally obtaining the target retrieval video content, so as to improve the efficiency and accuracy of video retrieval and adapt to multi-field retrieval requirements.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular, to a multi-granularity video retrieval method, device, computer device and readable storage medium based on a multi-modal large model. Background Art

[0002] In the current field of video retrieval, traditional methods have many deficiencies, such as high consumption of computing resources, difficulty in retrieving long videos, poor semantic alignment, and low retrieval recall rate, etc., making it difficult to meet the accurate and efficient retrieval needs of users in different fields and specific topics. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-granularity video retrieval method, device, computer device and readable storage medium based on a multi-modal large model.

[0004] In a first aspect, an embodiment of the present invention provides a multi-granularity video retrieval method based on a multi-modal large model, including:

[0005] Obtain video query information input by a user;

[0006] Perform intent recognition on the video query information to obtain a query field corresponding to the video query information;

[0007] Rewrite the query content of the video query information and the query field to obtain a video query vector;

[0008] Retrieve in a preset retrieved video knowledge base according to the video query vector and the query field to obtain target retrieved video content.

[0009] In a possible implementation manner, the rewriting the query content of the video query information and the query field to obtain a video query vector includes:

[0010] Encode the video query information and the query field through a text encoder and a video encoder to obtain a to-be-query text encoding vector and a to-be-query video encoding vector corresponding to the video query information and the query field;

[0011] Use the to-be-query text encoding vector and the to-be-query video encoding vector as the video query vector.

[0012] In a possible implementation manner, retrieving in a preset retrieved video knowledge base according to the video query vector and the query field to obtain target retrieved video content includes:

[0013] Calculate the similarity between the text query encoding vector and the video query encoding vector and the candidate text encoding vectors and candidate video encoding vectors corresponding to the query domain stored in the predefined retrieval video knowledge base respectively, to obtain the candidate retrieval video content with the highest similarity;

[0014] If the similarity corresponding to the candidate retrieval video content with the highest similarity exceeds the predefined similarity threshold, then use the candidate retrieval video content with the highest similarity as the target retrieval video content;

[0015] If the similarity corresponding to the candidate retrieval video content with the highest similarity does not exceed the predefined similarity threshold, then perform a full-scale retrieval in the predefined retrieval video knowledge base to obtain the target retrieval video content; the full-scale retrieval includes cross-retrieval between the text query encoding vector and all candidate video encoding vectors and cross-retrieval between the video query encoding vector and all candidate text encoding vectors.

[0016] In a possible implementation, the predefined retrieval video knowledge base is constructed in the following manner, including:

[0017] Obtain the video to be processed;

[0018] Classify the video to be processed by the zero-shot method based on the multimodal large model to obtain the video category of the video to be processed;

[0019] According to the business scenario corresponding to the video category, select the segmentation strategy corresponding to the business scenario to segment the video to be processed, to obtain multiple video segments;

[0020] For each of the video segments, perform descriptive subtitle generation, speech recognition, and pedestrian detection operations respectively, and perform structured processing to obtain the structured expression corresponding to the video to be processed, where the structured expression includes the video classification information of the video to be processed, the time stamps, subtitle contents, speech recognition contents, and the identifiers and coordinates of the target detection and tracking information corresponding to each video segment;

[0021] Perform text and video encoding processing on the structured expression corresponding to the video to be processed through a multimodal encoder to obtain the text encoding vector to be processed and the video encoding vector to be processed corresponding to the video to be processed;

[0022] Store the text encoding vector to be processed and the video encoding vector to be processed into the initial vector database to obtain the predefined retrieval video knowledge base.

[0023] In a possible implementation, according to the service scenario corresponding to the video category, select a segmentation strategy corresponding to the service scenario to segment the video to be processed, and obtain a plurality of video segments, including:

[0024] If there is a visual narrative structure in the service scenario corresponding to the video category, normalize a plurality of video frames included in the video to be processed;

[0025] Extract features from the normalized plurality of video frames to obtain video frame features corresponding to each of the video frames;

[0026] Calculate the similarity metric between consecutive video frames based on the video frame features; the similarity metric is obtained based on at least two similarity calculation methods;

[0027] Determine a preset similarity threshold according to the at least two similarity calculation methods, and use the target frames whose similarity metric between consecutive video frames exceeds the preset similarity threshold as shot transition points;

[0028] Perform clustering based on a plurality of the shot transition points to obtain a plurality of clustering clusters, and each of the clustering clusters represents a shot;

[0029] Classify based on the type corresponding to each of the clustering clusters, output the start frame and end frame of each of the shots, and obtain the plurality of video segments based on shot segmentation;

[0030] If there is no visual narrative structure in the service scenario corresponding to the video category, evenly segment the video to be processed at a preset interval to obtain the plurality of video segments.

[0031] In a possible implementation, for each of the video segments, generate descriptive subtitles, including:

[0032] Process each of the video segments through a preset visual large model to obtain the descriptive subtitles corresponding to each of the video segments, including event time, event location, character relationship, character behavior, spatial relationship, and video content summary.

[0033] In a possible implementation, for each of the video segments, perform pedestrian detection respectively, including:

[0034] Perform pedestrian detection through YOLOv9 and use DeepSORT for pedestrian tracking.

[0035] In a second aspect, an embodiment of the present invention provides a multi-granularity video retrieval device based on a multi-modal large model, including:

[0036] An acquisition module, configured to acquire video query information input by a user; perform intent recognition on the video query information to obtain a query domain corresponding to the video query information; rewrite the video query information and the query domain to obtain a video query vector;

[0037] A retrieval module, configured to retrieve in a preset retrieved video knowledge base according to the video query vector and the query domain to obtain target retrieved video content.

[0038] In a third aspect, an embodiment of the present invention provides a computer device, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls a computer device where the readable storage medium is located to execute the method described in the first aspect.

[0040] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a multi-granularity video retrieval method, device, computer device and readable storage medium based on a multi-modal large model disclosed by the present invention, by acquiring video query information input by a user, performing intent recognition to obtain a query domain, rewriting the query information and the domain to obtain a video query vector, and then retrieving in a preset retrieved video knowledge base according to this vector and the domain, finally obtaining target retrieved video content, aiming to improve the efficiency and accuracy of video retrieval and meet the retrieval requirements of multiple domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 It is a schematic flowchart of the steps of the multi-granularity video retrieval method based on a multi-modal large model provided by an embodiment of the present invention;

[0043] Figure 2 It is a schematic flowchart of video preprocessing provided by an embodiment of the present invention;

[0044] Figure 3 It is a schematic flowchart of video segmentation provided by an embodiment of the present invention;

[0045] Figure 4A schematic flowchart of video subtitle generation provided by an embodiment of the present invention;

[0046] Figure 5 A schematic block diagram of the speech recognition process in a video provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of video structuring provided by an embodiment of the present invention;

[0048] Figure 7 A schematic block diagram of the structure of a multi-granularity video retrieval device based on a multi-modal large model provided by an embodiment of the present invention;

[0049] Figure 8 A schematic block diagram of the structure of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0051] The following will describe in detail the specific implementation manners of the present invention with reference to the accompanying drawings.

[0052] To solve the technical problems in the foregoing background art, Figure 1 A schematic flowchart of a multi-granularity video retrieval method based on a multi-modal large model provided by an embodiment of the present disclosure is as follows. The multi-granularity video retrieval method based on a multi-modal large model will be introduced in detail below.

[0053] Step S201: Obtain video query information input by a user;

[0054] Step S202: Perform intent recognition on the video query information to obtain a query domain corresponding to the video query information;

[0055] Step S203: Rewrite the query content of the video query information and the query domain to obtain a video query vector;

[0056] Step S204: Retrieve in a preset retrieved video knowledge base according to the video query vector and the query domain to obtain target retrieved video content.

[0057] In an embodiment of the present invention, by way of example, it is assumed that the server receives video query information input by a user as: "I want to watch the latest technology lectures on the application of artificial intelligence in the medical field."

[0058] The server first receives and records this input, preparing to enter the subsequent processing flow.

[0059] The server processes the query information input by the user according to the designed intent recognition process. It uses the pre-set intent recognition prompt template and the Qwen2-VL7B algorithm model to determine the field to which the user's query intent belongs.

[0060] In this example, the server analyzes the user's input "I want to watch the latest science and technology lectures on the application of artificial intelligence in the medical field". The model will match and judge the input with 46 pre-defined field categories. Finally, it is determined that the user's query field is "science and technology lectures".

[0061] The server then rewrites the user's original query information and the identified query field. It organizes them as "In the field of science and technology lectures, I want to watch the latest on the application of artificial intelligence in the medical field".

[0062] Then, the server uses BGE3 as the text encoder and ViCLIP as the video encoder to encode the rewritten query content respectively, so as to obtain the video query vector.

[0063] The BGE3 model processes the rewritten text and converts it into a numerical vector that can capture semantic information. For example, it may generate a vector of a specific dimension according to the words, grammatical structures and semantic relationships in the text, where the value of each element represents certain features of the text.

[0064] The ViCLIP model encodes the video features that may be relevant to the query. If relevant video examples or clips are provided to it, it will convert the visual features of these videos into vector representations for subsequent similarity calculation and retrieval operations with the vectors encoded from the text.

[0065] The server first retrieves in the video knowledge base of the specific field of "science and technology lectures".

[0066] Suppose a large number of lecture videos on different science and technology topics have been stored in the video knowledge base of this field. For each stored video, the server has generated corresponding text encoding vectors and video encoding vectors through preprocessing, multi-modal encoding and other steps in the offline video knowledge base construction process.

[0067] The server calculates the cosine similarity between the previously generated text encoding vector and video encoding vector of the user query and the text encoding vectors and video encoding vectors of each video in the "science and technology lectures" field respectively.

[0068] For example, for a science and technology lecture video on "Applications of Artificial Intelligence in the Medical Field", the server calculates the cosine similarity between the user query vector and the video vector. If the similarity is high (e.g., exceeding the preset threshold of 0.8), it is considered that the video is highly relevant to the user's query.

[0069] The server sorts the retrieval results in descending order of similarity and selects the top K (e.g., top 10) video contents as the preliminary target retrieval results.

[0070] If the similarities of the videos retrieved within the "science and technology lecture" field are all lower than 0.8, the server will expand the retrieval scope, retrieve all contents in the entire video knowledge base, calculate the cosine similarity again, and screen out the top K video contents that meet the requirements.

[0071] In addition, the server also performs similarity weighting on the retrieval of text-to-text (such as video subtitles, speech recognition content, target retrieval and tracking content) and text-to-video to obtain more accurate retrieval results that meet the user's needs.

[0072] Finally, the server returns the video content that best meets the user's needs to the user, such as providing relevant video links, titles, introductions, etc., enabling users to quickly find the video resources they expect.

[0073] Through the above detailed scenario examples, the specific operations and effects of the multi-granularity video retrieval method based on the multi-modal large model in practical applications can be more clearly understood, so as to better meet the video retrieval needs of users in different fields and specific topics.

[0074] In the embodiment of the present invention, the query content of the video query information and the query field is rewritten to obtain a video query vector, which can be implemented through the following examples.

[0075] The video query information and the query field are encoded through a text encoder and a video encoder to obtain the to-be-query text encoding vector and the to-be-query video encoding vector corresponding to the video query information and the query field;

[0076] The to-be-query text encoding vector and the to-be-query video encoding vector are used as the video query vector.

[0077] In the embodiment of the present invention, by way of example, it is assumed that the video query information input by the user received by the server is "I want to find some documentary films about tourist attractions introductions". After intention recognition, the query field is determined to be "documentary films".

[0078] The server first rewrites this query information and the query domain, and organizes it into "In the documentary field, I want to find some content about the introduction of tourist attractions."

[0079] Next, the server uses the text encoder BGE3 to encode the rewritten text. BGE3 will analyze and process the vocabulary, grammar structure, and semantic relationships in this text. For example, it will extract key information such as "documentary", "tourist attraction", "introduction", etc., and transform this information into a numerical vector of a specific dimension. The value of each element in this vector represents certain characteristics of these key information, such as word frequency, semantic weight, etc. After encoding, the encoded vector of the text to be queried is obtained.

[0080] At the same time, the server uses the video encoder ViCLIP to process the video features that may be related to the query. Suppose there are some example documentary video clips related to the introduction of tourist attractions in the server. ViCLIP will analyze the visual features of these video clips, such as the color of the picture, composition, movement of objects, etc. Then, ViCLIP transforms these visual features into another numerical vector of a specific dimension, that is, the encoded vector of the video to be queried.

[0081] Finally, the server combines the obtained encoded vector of the text to be queried and the encoded vector of the video to be queried as the video query vector.

[0082] For example, the encoded vector of the text to be queried may be a 512-dimensional vector, where the values of some dimensions are relatively high, representing the importance of key concepts such as "documentary" and "tourist attraction" in the text. The encoded vector of the video to be queried may be a 1024-dimensional vector, reflecting the visual features in the video clip.

[0083] The server uses these two vectors as an overall video query vector for subsequent similarity calculation and retrieval operations in the preset retrieved video knowledge base to find the video content that best matches the user's query.

[0084] In practical applications, this encoding method can effectively transform the user's query requirements into a computable and comparable vector form, enabling the server to quickly and accurately screen out the content that meets the user's intention from a large amount of video data.

[0085] In the embodiment of the present invention, according to the video query vector and the query domain, retrieve in the preset retrieved video knowledge base to obtain the target retrieved video content, and the implementation can be executed through the following examples.

[0086] Calculate the similarity between the text encoding vector to be queried and the video encoding vector to be queried, and the candidate text encoding vector and candidate video encoding vector corresponding to the query domain stored in the preset retrieval video knowledge base respectively, to obtain the candidate retrieval video content with the highest similarity;

[0087] If the similarity corresponding to the candidate retrieval video content with the highest similarity exceeds the preset similarity threshold, then use the candidate retrieval video content with the highest similarity as the target retrieval video content;

[0088] If the similarity corresponding to the candidate retrieval video content with the highest similarity does not exceed the preset similarity threshold, perform a full-scale retrieval in the preset retrieval video knowledge base to obtain the target retrieval video content; the full-scale retrieval includes cross-retrieval between the text encoding vector to be queried and all candidate video encoding vectors and cross-retrieval between the video encoding vector to be queried and all candidate text encoding vectors.

[0089] In the embodiment of the present invention, by way of example, it is assumed that the server receives the video query information input by the user as "I want to watch teaching course videos about food making", and determines the query domain as "food making" through intent recognition. After rewriting the query information and the query domain, the server obtains the text encoding vector to be queried and the video encoding vector to be queried through the text encoder BGE3 and the video encoder ViCLIP.

[0090] The server first finds the candidate text encoding vector and candidate video encoding vector corresponding to the "food making" domain in the preset retrieval video knowledge base.

[0091] For the similarity calculation of the text encoding vector, the server may use the cosine similarity algorithm. Assume that the text encoding vector to be queried is [0.5, 0.3, 0.2, 0.1, 0.4,...], and a certain candidate text encoding vector is [0.4, 0.2, 0.3, 0.15, 0.35,...]. The server calculates the cosine value of the included angle between these two vectors as their similarity. For the video encoding vector, a similar calculation is also performed.

[0092] Assume that after calculation, it is found that the similarity of one of the candidate retrieval video contents is the highest, for example, reaching 0.85. If the preset similarity threshold is 0.8, since 0.85 exceeds 0.8, the server will directly use this candidate retrieval video content as the target retrieval video content.

[0093] But if the highest similarity is only 0.7 and does not exceed the preset similarity threshold of 0.8, the server will perform a full-scale retrieval.

[0094] In the full-scale retrieval, the server performs cross-retrieval of the text encoding vector to be queried with all candidate video encoding vectors in the knowledge base, and also performs cross-retrieval of the video encoding vector to be queried with all candidate text encoding vectors.

[0095] For example, the server finds a new candidate video encoding vector among the full-scale candidate video encoding vectors, and its similarity to the text encoding vector to be queried reaches 0.9. Similarly, in the cross-retrieval of the video encoding vector to be queried and the full-scale candidate text encoding vectors, new high-similarity combinations may also be found.

[0096] The server synthesizes the results of these cross-retrievals, sorts them again from high to low according to similarity, finally determines the target retrieval video content that meets the requirements, and provides it to the user.

[0097] Through such a detailed and comprehensive retrieval process, the server can find the video content that best meets the user's needs as accurately as possible, improving the user's retrieval experience and satisfaction.

[0098] In an actual large-scale video knowledge base, there may be thousands or even more candidate encoding vectors. With its efficient computing power and optimized algorithms, the server can quickly complete these complex calculation and comparison operations, and provide high-quality retrieval results for users in a timely manner.

[0099] In the embodiment of the present invention, the preset retrieval video knowledge base is constructed in the following manner, and the implementation can be performed through the following examples.

[0100] Obtain the video to be processed;

[0101] Classify the video to be processed by the zero-shot method based on the multi-modal large model to obtain the video category of the video to be processed;

[0102] According to the business scenario corresponding to the video category, select the segmentation strategy corresponding to the business scenario to segment the video to be processed to obtain multiple video segments;

[0103] For each of the video segments, perform operations of generating descriptive subtitles, speech recognition, and pedestrian detection respectively, and perform structured processing to obtain the structured expression corresponding to the video to be processed. The structured expression includes the video classification information of the video to be processed, the time stamps, subtitle contents, speech recognition contents, and the identifiers and coordinates of the target detection and tracking information corresponding to each video segment;

[0104] Perform text and video encoding processing on the structured expression corresponding to the video to be processed through a multi-modal encoder to obtain the text encoding vector to be processed and the video encoding vector to be processed corresponding to the video to be processed;

[0105] Store the text encoding vector to be processed and the video encoding vector to be processed in the initial vector database to obtain the preset retrieval video knowledge base.

[0106] In an embodiment of the present invention, for example, the server obtains a video to be processed, such as a two-hour teaching video about "home decoration".

[0107] First, the server classifies this video to be processed based on a multimodal large model using the zero-shot method. The model will classify it into the video category of "home decoration" according to the features in the video, such as the displayed decoration materials, decoration styles, and the content explained.

[0108] Since this video belongs to the teaching category, the server selects a segmentation strategy suitable for teaching videos according to the business scenario corresponding to the "home decoration" category. For example, the two-hour video is segmented into 12 video segments at intervals of 10 minutes.

[0109] For each video segment, the server performs relevant operations respectively. For example, descriptive subtitles are generated through a multimodal large model. Suppose the subtitle of one video segment is "In this segment, from 15 minutes to 25 minutes, it explains how to choose the floor material suitable for the living room and shows several different material samples. At the same time, there is a clear voice of the narrator introducing the advantages and disadvantages of each material. No pedestrians are detected in the picture." The server also recognizes the voice in the video, converts the voice of the narrator into text. At the same time, pedestrian detection is performed. If no pedestrians are detected, the corresponding identification and coordinates are empty. After processing each video segment, structured processing is performed to obtain the structured expression corresponding to this video to be processed. The structured expression includes the video classification information of "home decoration", the time stamps corresponding to each video segment (such as 15 - 25 minutes), the subtitle content (such as the above description about the floor material), the voice recognition content (such as the narrator's introduction about the advantages and disadvantages of the material), and the identification and coordinates of the object detection and tracking information (if no pedestrians, it is empty).

[0110] Next, the server performs text and video encoding processing on this structured expression through multimodal encoders, such as the text encoder BGE3 and the video encoder ViCLIP. BGE3 converts the text information of the subtitle and voice recognition into the text encoding vector to be processed, and ViCLIP converts the visual content of each video segment into the video encoding vector to be processed.

[0111] Finally, the server stores the generated text encoding vector to be processed and the video encoding vector to be processed in the initial vector database. In this way, when the user conducts a search, the server can quickly find video content that matches the user's needs from this preset video knowledge base for retrieval.

[0112] For example, when the user enters "I want to find a home decoration teaching video about the selection of living room floor materials", the server can quickly find relevant video segments from the knowledge base and provide them to the user to meet their needs.

[0113] In the actual processing process, the server may need to process a large number of videos to be processed simultaneously. However, relying on its powerful computing power and efficient algorithms, it can complete these complex construction steps in an orderly manner and provide users with rich and accurate video retrieval services.

[0114] In the embodiment of the present invention, for the business scenario corresponding to the video category, a segmentation strategy corresponding to the business scenario is selected to segment the video to be processed, and multiple video segments are obtained. The following examples can be used for implementation.

[0115] If there is a visual narrative structure in the business scenario corresponding to the video category, normalize the multiple video frames included in the video to be processed;

[0116] Extract features from the normalized multiple video frames to obtain video frame features corresponding to each video frame;

[0117] Calculate the similarity measure between consecutive video frames based on the video frame features; the similarity measure is obtained based on at least two similarity calculation methods;

[0118] Determine a preset similarity threshold according to the at least two similarity calculation methods, and use the target frames whose similarity measure between consecutive video frames exceeds the preset similarity threshold as shot transition points;

[0119] Cluster based on multiple shot transition points to obtain multiple clusters, and each cluster represents a shot;

[0120] Classify based on the type corresponding to each cluster, and output the start frame and end frame of each shot to obtain the multiple video segments based on shot segmentation;

[0121] If there is no visual narrative structure in the business scenario corresponding to the video category, evenly segment the video to be processed at a preset interval to obtain the multiple video segments.

[0122] In an embodiment of the present invention, exemplarily, the server receives a video to be processed, such as a "movie" category. Since movies usually have an obvious visual narrative structure, the server will segment it according to the following steps:

[0123] First, the server normalizes the numerous video frames included in this movie. For example, the resolution of all video frames is uniformly adjusted to a standard size, and the pixel values are standardized for subsequent processing and calculation.

[0124] Then, feature extraction is performed on the normalized video frames. The server may use a deep learning algorithm, such as ResNet50, to extract the features of each video frame. These features may include information such as color, texture, and shape.

[0125] Next, the similarity measure between consecutive video frames is calculated based on the extracted video frame features. The server will use both Euclidean distance and cosine similarity for these similarity calculations. Through the analysis of a large amount of data and empirical summary, the server determines a preset similarity threshold.

[0126] Suppose the threshold for Euclidean distance is set to 5, and the threshold for cosine similarity is set to 0.8. When the similarity measure between consecutive video frames is lower than these two thresholds, these frames are marked as shot transition points.

[0127] Based on the determined multiple shot transition points, the server performs a clustering operation. Similar shot transition points will be grouped into a clustering cluster, and each clustering cluster represents a shot.

[0128] Finally, the server classifies according to the characteristics and types of each clustering cluster, and accurately outputs the start frame and end frame of each shot, thereby obtaining multiple video segments based on shot segmentation.

[0129] Another video to be processed is of the "traffic monitoring" category. Since there is usually no visual narrative structure in this type of business scenario, the server will segment this video to be processed at a preset interval, such as every 30 seconds. In this way, the long traffic monitoring video is divided into multiple video segments of equal duration, which is convenient for subsequent processing and analysis.

[0130] For example, for a suspense movie, the server successfully identifies the shot transition points between the intense chase scene and the warm conversation scene through the above complex calculations and analyses, and reasonably segments the movie into different paragraphs. For a traffic monitoring video, the server stably divides it into multiple segments at fixed time intervals, and each segment can be analyzed and processed independently.

[0131] In actual operation, the server can complete these segmentation tasks quickly and accurately, regardless of the type and characteristics of the video, and can provide a good foundation for subsequent processing, improve the efficiency and quality of video processing to meet different application requirements.

[0132] In the embodiment of the present invention, descriptive subtitles are generated for each of the video segments, which may be implemented through the following examples.

[0133] Each of the video segments is processed through a preset visual macro model to obtain the descriptive subtitles corresponding to each of the video segments, including event time, event location, character relationships, character behaviors, spatial relationships, and video content summary.

[0134] In the embodiment of the present invention, illustratively, the server receives a series of video segments, for example, one of the video segments is about a wonderful moment of a basketball game.

[0135] The server inputs this video segment into a preset visual model, such as Qwen2-VL 7B, for processing.

[0136] First, the model analyzes the images and possible audio information in the video to extract features related to the event time. For example, through the game time logo on the player's jersey, the time displayed on the mobile phone of the audience, or the time information mentioned in the commentary, it is determined that this wonderful moment occurred with about 5 minutes left in the third quarter of the game, thus obtaining a description of the event time.

[0137] Next, the model determined that the event took place in a large sports stadium in a certain city based on the characteristics of the venue in the video, surrounding environmental signs, and billboards in the background.

[0138] Then, the model further analyzes the interactions and cooperation between players in the video, as well as the communication between coaches and players, and sorts out the relationships between characters, such as clarifying the tacit passing cooperation between teammates, or the tactical guidance relationship between coaches and players.

[0139] At the same time, the model accurately describes the character's behavior by observing the player's actions, such as shooting, dribbling, and defending. For example, a player jumped high and made a beautiful three-point shot.

[0140] The model also determines the spatial relationship based on the positional relationship between elements such as players, basketball and court in the video, such as when a player shoots from a certain position outside the three-point line and the positioning of other players under the basket.

[0141] Finally, the model summarizes the content of the entire video segment, describing it as "With 5 minutes left in the third quarter, in [sports venue name], teammates passed the ball tacitly, [player name] jumped high at [specific location] outside the three-point line and shot a wonderful three-pointer. The players on the court were well positioned and finally scored successfully. This wonderful moment pushed the game to a climax."

[0142] For example, another video segment is about a cooking tutorial.

[0143] The server also inputs it into the preset visual model.

[0144] The model determined that the event took place in the morning by recognizing the images and audio of the food preparation stage.

[0145] Based on the layout of the kitchen and the surrounding environment, it was determined that the incident took place in a home kitchen.

[0146] Analyze the collaborative relationship between the chef and the assistant, and clarify the relationship between the characters.

[0147] Observe the chef's actions of cutting vegetables and cooking, and describe the character's behavior, such as the chef skillfully cutting the vegetables into even pieces.

[0148] Determine the spatial relationship between various kitchen utensils, ingredients, and people, such as the pot on the stove, vegetables on the chopping board, and the chef standing in front of the stove to operate.

[0149] Finally, it was summarized as "In the family kitchen in the morning, the chef and the assistant worked closely together. The chef skillfully cut the vegetables into pieces on the chopping board and stir-fried them in a pot on the stove to prepare a delicious dish."

[0150] In this way, the server generates detailed and accurate descriptive subtitles for each video segment, providing rich and valuable information for subsequent processing and user retrieval.

[0151] In the embodiment of the present invention, pedestrian detection is performed for each of the video segments, which can be implemented through the following examples.

[0152] Pedestrian detection is performed through YOLOv9, and pedestrian tracking is performed using DeepSORT.

[0153] In the embodiment of the present invention, illustratively, the server receives a video segment to be processed, such as a surveillance video of a city street.

[0154] First, the server uses the YOLOv9 algorithm to detect pedestrians. YOLOv9 analyzes each frame of the video segment. It divides the image into multiple grids and then predicts whether there are pedestrians in each grid.

[0155] For example, in a certain frame of an image, YOLOv9 pays attention to various regions in the image. For a region containing a pedestrian, it makes a judgment and marks the position of the pedestrian based on the pedestrian's features, such as body contour, head shape, positions of limbs, etc. Suppose in this frame of the image, YOLOv9 detects three pedestrians and marks the coordinate ranges of each of them in the image.

[0156] After completing pedestrian detection, the server then uses DeepSORT for pedestrian tracking. DeepSORT combines the pedestrian detection results detected by YOLOv9 and the tracking information in the previous frames.

[0157] Suppose in several consecutive frames of an image, YOLOv9 detects the same pedestrian. DeepSORT extracts the appearance features of this pedestrian, such as clothing color, hairstyle, height ratio, etc., and combines the movement information of the pedestrian, such as moving speed, direction, etc., to determine the position changes of the same pedestrian in different frames.

[0158] For example, a pedestrian wearing a red coat walks from the left side to the right side of the screen. DeepSORT continuously tracks the position of this pedestrian in each frame based on its appearance features and movement trajectory, and assigns a unique identifier to it so that the server can accurately record and analyze the action trajectory of this pedestrian.

[0159] Another example is that another video segment is a surveillance video inside a shopping mall. YOLOv9 also detects each frame and discovers multiple pedestrians. Then DeepSORT comes into play. Even in the case of a relatively dense crowd and brief occlusions between pedestrians, it can accurately track each pedestrian. For example, a pedestrian pushing a shopping cart shuttles between the shelves, and DeepSORT can continuously track his actions without losing the tracking target due to interference from other pedestrians or brief occlusions.

[0160] Through the collaborative work of YOLOv9 and DeepSORT, the server can effectively detect and track pedestrians in the video segment, providing accurate and detailed pedestrian information for subsequent analysis and processing, such as the number of pedestrians, activity trajectories, staying times, etc. This is of great significance for applications such as traffic management, security monitoring, and pedestrian flow analysis.

[0161] To more clearly describe the solution provided by the embodiments of the present invention, a relatively complete implementation manner is provided below.

[0162] Video information usually includes various contents such as image frames in a time series, video titles, video tags, video subtitles, and audio. Among them, the image frames in a time series are essential basic information in the video, while other information such as titles, tags, subtitles, and audio may be missing or incomplete depending on the specific situation. Based on these characteristics of video information, the present invention designs a complete multi-modal and multi-granularity video retrieval system. This system has strong adaptability to vertical domain tasks and generalization ability, and also performs excellently in retrieval efficiency and depth of semantic understanding. This system has the following characteristics:

[0163] Strong adaptability to domain tasks: It can effectively meet the retrieval requirements in different domains.

[0164] Strong generalization ability: It maintains stable performance in diverse data environments.

[0165] High retrieval efficiency: It quickly responds to user queries and provides efficient retrieval results.

[0166] Strong semantic understanding: It deeply understands the semantic information of video content and improves the retrieval accuracy.

[0167] According to the characteristics of video information and the task requirements of video retrieval, this system is divided into two parts: 1. Offline video knowledge base construction; 2. Online user video retrieval. The design idea of this system is as follows:

[0168] Design of the offline video knowledge base construction system

[0169] The offline video knowledge base construction system mainly includes three parts: a video preprocessing module, a multi-modal encoding module, and a vector database construction module;

[0170] Video preprocessing module

[0171] The video preprocessing module is the basis of the video retrieval workflow, including core steps such as video classification, video segmentation, video subtitle generation, speech recognition, and pedestrian retrieval and tracking. When processing long videos, first use a multi-modal large model combined with the prompt technology to carefully classify the video content. This key step not only optimizes the video retrieval process, improves the retrieval efficiency and accuracy, but also lays a solid foundation for the entire preprocessing process.

[0172] Then, according to the characteristics of the video content, use different video segmentation techniques to cut the long video into a series of shorter segments and accurately record the timestamp positions of these segments in the original video file. This step not only facilitates the generation and structured processing of video subtitles, but also ensures the traceability and integrity of the video content.

[0173] Video subtitle generation automatically generates video subtitles through an advanced video multi-modal large model, enhancing the associated information between different modalities in the video content. For videos with voice information, advanced speech recognition technology is used to convert voice signals into text, further enriching the video's text database.

[0174] Object detection and tracking algorithms accurately detect and track pedestrians in the video, which is crucial for deeply understanding the state of the people in the video and plays an important role especially in fields such as security monitoring and crowd flow analysis. The design framework of the entire video preprocessing module is clearly shown in the following figure, which not only reflects a profound understanding of video processing technology but also demonstrates the persistent pursuit of in-depth analysis of video content. Through this module, the original video data can be transformed into valuable information, providing a solid foundation for subsequent analysis and decision-making.

[0175] Video classification:

[0176] If traditional algorithms are used for video classification, large-scale data collection and network model training are required. To reduce data collection costs and training costs, the zero-shot ability of the multi-modal large model is directly utilized for video classification. Through the exploration of 12 major fields including entertainment, education, news and documentary, sports, life, technology and business, games, animation, blogs, enterprise and marketing, security and monitoring, and culture and art, the videos are subdivided into a total of 46 video categories, including movies, TV series, variety shows, music videos (MVs), academic lectures, teaching courses, training tutorials, cultural popular science, news reports, documentaries, talk shows, live broadcasts, sports event broadcasts, sports event highlights, sports training tutorials, travel videos, food production, health and fitness, home decoration, product demonstrations, commercial advertisements, technology lectures, stock market analysis, game live broadcasts, game guides, esports events, game reviews, 2D animations, 3D animations, animated movies, animated series, Vlogs, short films, social media videos, personal sharing videos, corporate promotional videos, product introductions, marketing campaign videos, customer testimonials, traffic monitoring, public security monitoring, home security monitoring, dance performances, drama performances, art exhibitions, and cultural festival activities.

[0177] After research and analysis, the open-source model Qwen2-VL, which has powerful capabilities in the field of video understanding, is used to achieve video classification. Qwen2-VL is a series of multimodal large language models developed by the Qwen team at Alibaba. It is the latest model in the Qwen model family, aiming to enhance the understanding of images and videos and provide advanced performance in various vision-language tasks. It has strong long-video understanding capabilities. The Qwen2-VL series performs comparably to top models such as GPT-4o and Claude3.5-Sonnet in various multimodal scenarios. The specific implementation method for achieving video classification through Qwen2-VL is as follows:

[0178] First, design a prompt template for video classification, and the specific content is as follows:

[0179] # Role

[0180] You are a video classification expert.

[0181] # Task Description

[0182] You will conduct a detailed classification evaluation of a series of video contents. Please determine the most likely classification for each video based on its specific content and assign a confidence score to this classification. The score ranges from 0 (completely unsure) to 1 (completely sure). The following is a list of video classifications for selection, including a total of 46 categories:

[0183] 1. Movie, 2. TV Series, 3. Variety Show, 4. Music Video (MV), 5. Academic Lecture, 6. Teaching Course, 7. Training Tutorial, 8. Cultural Popularization, 9. News Report, 10. Documentary, 11. Interview Program, 12. Live Broadcast, 13. Live Sports Event, 14. Sports Event Highlights, 15. Sports Training Tutorial, 16. Travel Video, 17. Food Making, 18. Health and Fitness, 19. Home Decoration, 20. Product Demonstration, 21. Commercial Advertisement, 22. Technology Lecture, 23. Stock Market Analysis, 24. Game Live Broadcast, 25. Game Strategy, 26. E-sports Event, 27. Game Review, 28. 2D Animation, 29. 3D Animation, 30. Animated Movie, 31. Animated Series, 32. Vlog, 33. Short Film, 34. Social Media Video, 35. Personal Sharing Video, 36. Corporate Promotional Video, 37. Product Introduction, 38. Marketing Activity Video, 39. Customer Testimony, 40. Traffic Monitoring, 41. Public Security Monitoring, 42. Home Security Monitoring, 43. Dance Performance, 44. Drama Performance, 45. Art Exhibition, 46. Cultural Festival Activity.

[0184] **Input**:

[0185] A video

[0186] **Output**:

[0187] The field and confidence level of the user's query.

[0188] # Requirements

[0189] You select the most appropriate category based on the specific situation of the video content and give the corresponding confidence score;

[0190] Input the video in combination with the prompt template and call the Qwen2-VL7B model for video classification. Please refer to Figure 2 .

[0191] Video segmentation:

[0192] Video segmentation technology is a key link in the field of video processing, which can be achieved through two main methods: average segmentation and detection segmentation based on shot detection in the video. A shot, that is, the transition between different shots, is a common narrative technique in movies, TV dramas, advertisements, animations, and some online short video content. These video types rely on a clear visual narrative structure to effectively convey the story or information. Therefore, the identification of shots is crucial for the segmentation of these videos.

[0193] However, in many surveillance scenarios, the video content often does not follow the traditional narrative structure and lacks obvious shots. For example, in traffic surveillance and factory surveillance scenarios, continuous surveillance videos usually do not have shot transitions, which makes the method of video segmentation based on shots no longer applicable.

[0194] For different scenarios of video retrieval, two segmentation strategies are adopted. For videos with obvious shot features such as movies, TV dramas, and advertisements, a video segmentation method based on shot detection is used to maintain the coherence and integrity of the narrative. For most surveillance videos, due to the lack of shots, the average segmentation method is adopted to equally divide the long video into multiple segments. This method controls the length of each segment by setting the `segment_length` parameter. By default, the length of each video segment is 2 seconds. The video shot algorithm is implemented through the following steps:

[0195] 1. Preprocessing: Scale the video frames to a unified size and normalize them for easy processing;

[0196] 2. Feature extraction: Use ResNet50 to extract the features of different frames

[0197] 3. Calculate the similarity measure between consecutive frames: Use both Euclidean distance and cosine similarity as the similarity measures.

[0198] 4. Shot boundary detection: Set thresholds according to the results of two similarity calculations. Frames exceeding the thresholds are considered shot transition points. Merge frames for clustering analysis, and each cluster represents a shot.

[0199] 5. Shot segmentation: Classify the shots and output the start and end frames of each shot.

[0200] This flexible segmentation strategy not only adapts to the characteristics of different video contents, but also provides convenience for video retrieval and subsequent processing, ensuring the efficiency and accuracy of video analysis. Through refined segmentation, video features can be extracted more effectively, the precision of video retrieval can be improved, and it also lays a foundation for in-depth analysis of video content. Please refer to Figure 3 。

[0201] Video caption generation:

[0202] Video caption generation is a task that combines computer vision and natural language processing. Its goal is to automatically convert the scenes or events in a video into captions described in natural language. This task not only requires the model to understand the video content, but also needs to generate accurate, coherent and informative text. Video caption generation is an important research direction in the field of video understanding and multimodal representation, and has a wide range of applications in practical applications, playing an important role in video retrieval and human-computer interaction. This system uses the Qwen2-VL 7B model for the video caption generation task to obtain text semantic information strongly related to the video content. In order to handle the task of ever-changing user queries, video caption generation generates text information according to multiple dimensions such as time, location, character relationship, behavior, spatial relationship and video content summary. The specific implementation steps are as follows:

[0203] Set the prompt template for video caption generation, and generate captions from multiple dimensions such as time, location, character relationship, spatial relationship, behavior, etc. mentioned above. The specific setting method of the prompt template is:

[0204] # Character

[0205] You are an expert in video caption generation. Please complete the tasks with the following requirements and output accurate video descriptions.

[0206] # Task description:

[0207] You will receive a video and need to generate a descriptive caption according to the video content. Please ensure that the caption contains information in the following dimensions: time, location, character relationship, behavior, spatial relationship and video content summary. The generated caption should be strongly related to the video content and accurately reflect the key events and scenes in the video.

[0208] **Input**:

[0209] A video clip.

[0210] **Output**:

[0211] A subtitle text containing the following dimensional information:

[0212] Time: The time when the event occurs in the video, such as a specific time point or chronological order.

[0213] Location: The location or environment where the event occurs in the video.

[0214] Character relationship: The relationship between the characters appearing in the video.

[0215] Action: The actions or movements of the characters in the video.

[0216] Spatial relationship: The spatial position relationship between the objects in the video.

[0217] Summary of video content: A concise summary of the overall video content.

[0218] # Requirements

[0219] The subtitles should be concise, coherent, and accurately reflect the video content.

[0220] The length of the subtitles should be appropriate and not too long.

[0221] Please ensure that the information in the subtitles truly reflects the video content and avoid fictional or misleading descriptions.

[0222] Input the video in combination with the prompt template and call the Qwen2-VL 7B model for video subtitle generation; the specific implementation process of video subtitle generation is as Figure 4 shown.

[0223] Speech recognition

[0224] Automatic Speech Recognition (ASR) is a technology that converts human speech into text. It involves knowledge from multiple disciplines, including physiology, acoustics, signal processing, computer science, pattern recognition, linguistics, and psychology, etc. It is an important branch in the field of artificial intelligence, which involves converting human speech into machine-readable text data. The main task of speech recognition technology is to recognize and understand the words, phrases, and sentences in human speech. Its functions include: speech input, controlling devices or applications through voice commands; speech-to-text conversion, converting speech into text data for easy storage, editing, and searching; sentiment analysis, analyzing features such as pitch and speech rate in speech to identify the speaker's emotional state. There are mainly two research directions for speech recognition algorithms: traditional machine learning algorithms, such as Support Vector Machine (SVM), Hidden Markov Model (HMM), and neural networks, etc.; multi-modal large language models: having achieved remarkable success in large vocabulary continuous speech recognition, the method based on multi-modal large language has become the mainstream speech recognition modeling method. Currently, relatively advanced speech recognition large models include FunAudioLLM, DFSMN, Moonshine, and Whisper. In this collection, the Large version of the Whisper series model is used as the speech recognition model. The Whisper Large model is an advanced automatic speech recognition (ASR) and speech translation model developed by OpenAI.

[0225] This system first determines whether there is speech information in the video and whether the speech information is clear. If there is speech information, it then calls Whisper to recognize the speech information. Figure 5 The specific design process for speech recognition in the video.

[0226] Pedestrian detection and tracking:

[0227] This design mainly uses pedestrian detection algorithms in the field of deep learning. Pedestrian detection algorithms in deep learning mainly include the R-CNN series, YOLO series algorithms, and SSD series algorithms. Among them, the R-CNN series of object detection algorithms include R-CNN, Fast R-CNN, Faster R-CNN, etc. R-CNN extracts candidate regions through selective search, and then uses a convolutional neural network to extract features and classify them; Fast R-CNN improves R-CNN and realizes end-to-end training for object detection and feature extraction; Faster R-CNN further uses a region proposal network to replace selective search, greatly improving the detection speed. The YOLO series of object detection: transforms the object detection problem into a regression problem, and simultaneously predicts the category and location of the object through a single forward propagation, with the advantage of fast detection speed. For example, YOLOv5 performs well in scenarios with high real-time requirements and can quickly and accurately detect pedestrian targets. The SSD series of object detection algorithms: uses multi-scale feature maps to detect objects of different sizes and also has good results in pedestrian detection. It can better adapt to the scale changes of pedestrian targets by making predictions on feature maps at different levels. Among them, the YOLO series of algorithms have been widely used because of their advantages such as fast detection speed, efficient use of computing resources, accurate target localization and recognition, simple model structure, and strong flexibility and scalability. This design uses the advanced YOLOv9 algorithm model for video pedestrian detection.

[0228] DeepSORT is an algorithm that combines appearance features and motion information and can maintain continuous tracking of targets in complex scenarios. It extracts the appearance features of targets through a convolutional neural network, combines the Kalman filter to predict the motion state of targets, and then uses the Hungarian algorithm for data association to achieve accurate tracking of targets. For example, in security monitoring, it can stably track specific personnel in crowded areas with diverse target postures. The design uses YOLOv9+DeepSORT for pedestrian detection and tracking

[0229] Structurally express the relevant information output by the above video preprocessing module. Please refer to Figure 6 That is, according to the video segmentation results, structurally express the video classification information, video segments, subtitle information, speech recognition information, and object detection and tracking information.

[0230] Multi-modal encoding module:

[0231] The multi-modal encoding module is used to encode the results of video preprocessing into text and video for convenient subsequent storage and construction in a vector database. It is mainly divided into a text encoder and a video encoder.

[0232] A text encoder is a technology that converts text data into numerical vectors. These numerical vectors can capture the semantic information of the text, enabling the originally difficult-to-process text data to be effectively used in machine learning models. It can be used for information retrieval, semantic similarity measurement, clustering analysis, etc. In the embodiments of the present invention, the text embedding model BGE3 (BGE-M3-Embedding) is used for text encoding. The BGE3 model is a general semantic vector model that performs excellently in terms of multi-linguality, multi-functionality, and multi-granularity. The model aims to provide a unified model foundation for information retrieval (IR) applications in the real world, supporting semantic representation and retrieval tasks in more than 100 languages. In multiple evaluations, the BGE3 model is comprehensively ahead. Especially in multi-lingual retrieval tasks, its dense vector retrieval ability achieves competitive results in each language, and the overall level is significantly better than the previous baseline vector models. The text information obtained by preprocessing the video is vectorized. And a text embedding model Text-embedding-3-large provided by OpenAI is used as an alternative for text vectorization. The BGE3 model is used to encode the video captions and the content of speech recognition.

[0233] The video encoder in the multi-modal large language model converts video data into a numerical representation, i.e., an embedding vector, which captures the basic features and semantic information of the video content. This technology is crucial in various applications in the fields of machine learning and computer vision. It can convert video information into vectorized data that can be understood by machines. It is commonly used in tasks such as video content representation, video similarity measurement, and video retrieval. This design uses the video-text representation learning model ViCLIP based on ViT-L, and through contrastive learning on a large-scale multi-modal dataset InternVid centered on videos, achieves leading zero-shot action recognition and video retrieval performance. The ViCLIP is used to encode the content of video segments.

[0234] Vector database construction module:

[0235] A vector database is a database system specifically designed to store and query high-dimensional vector data. With the development of artificial intelligence and machine learning technologies, vector databases are playing an increasingly important role in processing complex data, achieving efficient retrieval, and supporting multimodal data analysis. Vector databases are mainly used for efficient similarity search, feature representation and embedding learning, and multimodal large-scale data storage. Commonly used vector databases include various frameworks such as Milvus, Qdrant, Weaviate, Vespa, Pinecone, FAISS (Facebook AI Similarity Search), Annoy (Approximate NearestNeighbors Oh Yeah), and HNSWlib. Among them, Milvus has been widely praised in the market for its high performance, high availability, and easy scalability. It is developed and maintained by Zilliz and later became one of the hosted projects of the LF AI&Data Foundation. It is commonly used to process large-scale unstructured data retrieval and has extensive applications in fields such as video retrieval, image retrieval, audio retrieval, and recommendation systems. Due to these characteristics, this system uses this vector database as the framework for data storage.

[0236] Design of Online User Video Retrieval System:

[0237] The online user video retrieval system mainly includes three parts: a user query intention recognition module, a query encoding module, and a vector database retrieval module;

[0238] Intention Recognition Module: First, analyze the user's input query information. Generally, it is impossible to know in advance what the user's input query is like. Therefore, in the first step, it is necessary to recognize the intention of the user's input query to identify the specific intention of the user's query for subsequent video retrieval and analysis. In this design, the user's intentions are classified according to the field, and the specific classification categories refer to the video classification categories mentioned above, with a total of 46 fields. By recognizing the intention, the specific field retrieved by the user can be obtained, thereby improving the retrieval accuracy and speed. The specific implementation method is to design an intention recognition prompt template and output the specific query field intention of the user through the Qwen2-VL7B algorithm model. The design of the prompt template refers to the video classification prompt template mentioned above, and the specific template is as follows:

[0239] #Role

[0240] You are an expert in user intention recognition in user video retrieval.

[0241] #Task Description

[0242] You need to identify the video content field that the user is querying based on the query information entered by the user. The specific video fields include the following 46:

[0243] 1. Movies, 2. TV series, 3. Variety shows, 4. Music videos (MVs), 5. Academic lectures, 6. Teaching courses, 7. Training tutorials, 8. Cultural popularization, 9. News reports, 10. Documentaries, 11. Interview shows, 12. Live broadcasts, 13. Live sports events, 14. Highlights of sports events, 15. Sports training tutorials, 16. Travel videos, 17. Food production, 18. Health and fitness, 19. Home decoration, 20. Product demonstrations, 21. Commercial advertisements, 22. Technology lectures, 23. Stock market analysis, 24. Game live broadcasts, 25. Game guides, 26. E-sports events, 27. Game reviews, 28. 2D animations, 29. 3D animations, 30. Animated movies, 31. Animated series, 32. Vlogs, 33. Short films, 34. Social media videos, 35. Personal sharing videos, 36. Corporate promotional videos, 37. Product introductions, 38. Marketing event videos, 39. Customer testimonials, 40. Traffic monitoring, 41. Public security monitoring, 42. Home security monitoring, 43. Dance performances, 44. Drama performances, 45. Art exhibitions, 46. Cultural festival activities.

[0244] **Input**:

[0245] The text information queried by the user.

[0246] **Output**:

[0247] The field queried by the user

[0248] # Requirements

[0249] Accurately identify the user's intention;

[0250] You cannot guess the user's intention. If you don't know, please output "Unable to identify the user's intention".

[0251] Please ensure that the information in the subtitles truly reflects the video content and avoid fictional or misleading descriptions.

[0252] Query encoding module: According to the multimodal encoder method of the video knowledge base construction module, the text encoder and video encoder use BGE3 and ViCLIP respectively. Therefore, the text content of the user query is encoded by the text encoder modules in the BGE3 and ViCLIP algorithms for subsequent retrieval. Note that the text information of the user query encoded here is the rewriting of the user query after intent recognition, that is, the user query content is rewritten into the form of: user query field + user query information. For example, the user query: Which athletes violated the rules in the game? This query content will be rewritten as: In the field of live sports events, which athletes violated the rules in the game? .

[0253] Retrieval module: The encoding vectors of the user query generated by the text encoder and the video encoder are respectively calculated by cosine similarity with the text encoding vectors and the video encoding vectors in the video knowledge base to obtain the retrieved topK video content. The retrieval module will first perform video retrieval in the field of specific video classification after intent recognition. If the similarity of the retrieval is high enough, the topK video content will be returned. If the cosine similarity is lower than 0.8, all the content in the video knowledge base will be retrieved. In addition, the similarity will be weighted by text-to-text (video subtitles, speech recognition content, target retrieval and tracking content) retrieval and text-to-video retrieval to obtain more accurate retrieval results.

[0254] In summary, through multimodal and multi-granular video retrieval design, many difficult problems of video retrieval can be greatly solved, and a powerful video retrieval system with strong adaptability to domain tasks, strong generalization ability, high retrieval efficiency and strong semantic understanding can be created.

[0255] Please refer to Figure 7 , Figure 7 A multi-granularity video retrieval device 110 based on a multi-modal large model provided in an embodiment of the present invention includes:

[0256] The acquisition module 1101 is used to acquire video query information input by a user; perform intent recognition on the video query information to obtain a query field corresponding to the video query information; perform query content rewriting on the video query information and the query field to obtain a video query vector;

[0257] The retrieval module 1102 is used to search in a preset retrieval video knowledge base according to the video query vector and the query field to obtain target retrieval video content.

[0258] It should be noted that the implementation principle of the multi-granularity video retrieval device 110 based on the multi-modal large model can refer to the implementation principle of the multi-granularity video retrieval method based on the multi-modal large model, which will not be elaborated here. It should be understood that the division of each module of the above device is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the multi-granularity video retrieval device 110 based on the multi-modal large model can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and the function of the multi-granularity video retrieval device 110 based on the multi-modal large model can be called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instruction in the form of software.

[0259] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as: one or more application-specific integrated circuits (ASICs), or, one or more microprocessors (digital signal processors, DSPs), or, one or more field programmable gate arrays (field programmable gate arrays, FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0260] An embodiment of the present invention provides a computer device 100. The computer device 100 includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned multi-granularity video retrieval device 110 based on the multi-modal large model. As Figure 8 shown, Figure 8The computer device 100 is a block diagram of a structure of a computer device 100 provided in an embodiment of the present invention. The computer device 100 comprises a multi-granularity video retrieval device 110 based on a multi-modal large model, a memory 111 , a processor 112 and a communication unit 113 .

[0261] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The multi-granularity video retrieval device 110 based on the multimodal large model includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the multi-granularity video retrieval device 110 based on the multimodal large model stored in the memory 111, such as the software function modules and computer programs included in the multi-granularity video retrieval device 110 based on the multimodal large model.

[0262] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned multi-granularity video retrieval device 110 based on a multi-modal large model.

[0263] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A multi-granularity video retrieval method based on a multi-modal large model, characterized in that Including: Obtain the video query information input by the user; Perform intent recognition on the video query information to obtain the query field corresponding to the video query information; Rewrite the query content of the video query information and the query field to obtain a video query vector; Retrieve in a preset retrieved video knowledge base according to the video query vector and the query field to obtain the target retrieved video content.

2. The method according to claim 1, wherein The rewriting the query content of the video query information and the query field to obtain a video query vector includes: Encode the video query information and the query field through a text encoder and a video encoder to obtain a to-be-query text encoding vector and a to-be-query video encoding vector corresponding to the video query information and the query field; Use the to-be-query text encoding vector and the to-be-query video encoding vector as the video query vector.

3. The method according to claim 2, wherein Retrieving in a preset retrieved video knowledge base according to the video query vector and the query field to obtain the target retrieved video content includes: Calculate the similarity between the to-be-query text encoding vector and the to-be-query video encoding vector and the candidate text encoding vector and the candidate video encoding vector corresponding to the query field stored in the preset retrieved video knowledge base respectively to obtain the candidate retrieved video content with the highest similarity; If the similarity corresponding to the candidate retrieved video content with the highest similarity exceeds a preset similarity threshold, use the candidate retrieved video content with the highest similarity as the target retrieved video content; If the similarity corresponding to the candidate retrieved video content with the highest similarity does not exceed the preset similarity threshold, perform a full-scale retrieval in the preset retrieved video knowledge base to obtain the target retrieved video content; the full-scale retrieval includes cross-retrieval between the to-be-query text encoding vector and the full-scale candidate video encoding vector and cross-retrieval between the to-be-query video encoding vector and the full-scale candidate text encoding vector.

4. The method according to claim 1, wherein The preset retrieved video knowledge base is constructed by the following method, including: Obtain the video to be processed; Classify the video to be processed in a zero-shot manner based on a multimodal large model to obtain the video category of the video to be processed; Select a segmentation strategy corresponding to the business scenario according to the video category to segment the video to be processed to obtain multiple video segments; For each video segment, perform operations of generating descriptive captions, speech recognition, and pedestrian detection respectively, and perform structured processing to obtain a structured expression corresponding to the video to be processed, where the structured expression includes the video classification information of the video to be processed, the time stamps, caption contents, speech recognition contents, and the identifiers and coordinates of target detection and tracking information corresponding to each video segment; Perform text and video encoding processing on the structured expression corresponding to the video to be processed through a multimodal encoder to obtain a to-be-processed text encoding vector and a to-be-processed video encoding vector corresponding to the video to be processed; Store the to-be-processed text encoding vector and the to-be-processed video encoding vector in an initial vector database to obtain the preset retrieved video knowledge base.

5. The method according to claim 4, wherein Select a segmentation strategy corresponding to the service scenario according to the video category, and segment the video to be processed according to the segmentation strategy corresponding to the service scenario to obtain a plurality of video segments, including: If there is a visual narrative structure in the service scenario corresponding to the video category, normalize the multiple video frames included in the video to be processed; Extract features from the multiple normalized video frames to obtain video frame features corresponding to each video frame; Calculate the similarity metric between consecutive video frames based on the video frame features; the similarity metric is obtained based on at least two similarity calculation methods; Determine a preset similarity threshold according to the at least two similarity calculation methods, and use the target frames whose similarity metric between consecutive video frames exceeds the preset similarity threshold as shot transition points; Cluster based on multiple shot transition points to obtain multiple clusters, and each cluster represents a shot; Classify based on the type corresponding to each cluster, output the start frame and end frame of each shot, and obtain the multiple video segments based on shot segmentation; If there is no visual narrative structure in the service scenario corresponding to the video category, evenly segment the video to be processed at a preset interval to obtain the multiple video segments.

6. The method according to claim 4, characterized in that, For each video segment, generate descriptive subtitles, including: Process each video segment through a preset visual large model to obtain the descriptive subtitles corresponding to each video segment, including event time, event location, character relationships, character actions, spatial relationships, and a summary of the video content.

7. The method according to claim 4, wherein For each video segment, perform pedestrian detection respectively, including: Perform pedestrian detection through YOLOv9 and use DeepSORT for pedestrian tracking.

8. A multi-granularity video retrieval device based on a multi-modal large model, characterized in that, Include: An acquisition module for acquiring video query information input by a user; Perform intent recognition on the video query information to obtain a query field corresponding to the video query information; Rewrite the video query information and the query field to obtain a video query vector; A retrieval module for retrieving in a preset retrieval video knowledge base according to the video query vector and the query field to obtain target retrieval video content.

9. A computer device, characterized in that, The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the method according to any one of claims 1-7.

Images