Multi-granularity semantic enhanced video retrieval method based on multi-modal large language model

Abstract

The application discloses a multi-granularity semantic enhancement video retrieval method based on a multi-modal large language model, and is specifically implemented according to the following steps: step 1, pre-processing an input reference video; step 2, obtaining a multi-granularity semantic description according to the pre-processed reference video and corresponding modification text; step 3, generating first-stage global query features according to the pre-processed reference video and the multi-granularity semantic description; step 4, selecting a first-stage candidate video set from a candidate video set according to the first-stage global query features; step 5, generating second-stage fine-granularity query features according to the modification text corresponding to the reference video and the multi-granularity semantic description; and step 6, selecting a target video from the first-stage candidate video set according to the second-stage fine-granularity query features. The application solves the problem that the prior art does not fully consider the fine-granularity semantic relationship between a reference video and modification text, thereby limiting retrieval accuracy.

Description

Technical Field

[0001] This invention belongs to the technical field of multimodal information processing and video retrieval methods, and relates to a multi-granularity semantic enhancement video retrieval method based on a multimodal large language model. Background Technology

[0002] Combined video retrieval is a crucial task in computer vision and multimodal understanding, widely applied in smart education, security monitoring, and video content management. With the increasing prevalence of surveillance equipment and the continuous growth in the scale of various video data, accurately retrieving target videos that meet user needs from massive amounts of video data has become a significant research challenge.

[0003] Existing video retrieval methods mainly include visual content-based video retrieval, text description-based video retrieval, and cross-modal video-text retrieval methods. These methods typically rely on the similarity of video features, text features, or both in a unified semantic space for matching, which can meet general retrieval needs. However, in combined video retrieval tasks, users use reference videos combined with modified text to express their target video requirements. The model needs to understand both the global semantics in the reference video and the local semantic changes in the modified text.

[0004] Existing combined video retrieval methods often employ direct concatenation, weighted fusion, or mapping matching of reference video features and modified text features, making it difficult to fully model the fine-grained semantic relationships between the reference video and the modified text. On the one hand, modified text is usually short and cannot fully express the overall scene and context of the target video; on the other hand, single-stage retrieval methods are prone to retrieving videos that are generally similar but do not meet the local modification criteria, thus limiting retrieval accuracy.

[0005] In recent years, multimodal large language models have demonstrated strong capabilities in visual understanding and cross-modal reasoning, providing new ideas for complex video retrieval tasks. However, existing technologies still lack an effective method that can combine reference videos, modified text, and multi-granular semantic descriptions, and utilize a staged retrieval mechanism to improve retrieval accuracy. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-granular semantically enhanced video retrieval method based on a multimodal large language model, which solves the problem in the prior art that the fine-grained semantic relationship between the reference video and the modified text is not fully considered, resulting in limited retrieval accuracy.

[0007] The technical solution adopted in this invention is a multi-granularity semantic enhancement video retrieval method based on a multimodal large language model, which is implemented according to the following steps: Step 1: Preprocess the input reference video; Step 2: Obtain multi-granular semantic descriptions based on the preprocessed reference video and corresponding modified text; Step 3: Generate the first-stage global query features based on the preprocessed reference video and multi-granular semantic description; Step 4: Select the first-stage candidate video set from the candidate video set based on the first-stage global query features; Step 5: Generate second-stage fine-grained query features based on the modified text and multi-granular semantic descriptions corresponding to the reference video; Step 6: Select the target video from the candidate video set in the first stage based on the fine-grained query features of the second stage.

[0008] Furthermore, the reference video in step 1 is represented as follows:

[0009] in, This refers to the first video in the reference video. frame, Indicates the total number of frames in the reference video; Reference video The preprocessing specifically involves: processing the reference video. Keyframe extraction and scaling are performed sequentially.

[0010] Furthermore, in step 1, the reference video... Keyframe extraction is performed using uniform frame extraction, specifically: extracting frames from a reference video. Frame keyframe, Then the sampled reference video :

[0011] in, Indicates the number of frames drawn. This represents the frame index that is uniformly sampled from the original video;

[0012] in, This indicates the rounding operation; The scaling in step 1 refers to scaling the sampled reference video. Each frame image in the image is scaled while maintaining its aspect ratio, specifically as follows: Set the reference video after sampling The resolution of each frame image in the image is Let the maximum side length threshold be... Then the scaling factor Represented as:

[0013] After scaling, the size of each image frame is 1. ; Obtain the preprocessed reference video frame sequence .

[0014] Furthermore, step 2 specifically involves: Step 2.1, Constructing Joint Information : ; in, For reference video The corresponding modified text, Provides prompts for multimodal chain thinking; Step 2.2, combine the information Input into a multimodal large language model to generate focused modification descriptions. :

[0015] in, This represents the mapping process by which a multimodal large language model generates focused modification descriptions under the guidance of multimodal chained thinking prompts; Step 2.3, combine the information Input into a multimodal large language model to generate a focused global description. :

[0016] in, This represents the mapping process by which a multimodal large language model generates a focused global description under the guidance of multimodal chained thinking prompts; Step 2.4 will focus on modifying the description. and focus on global description As a multi-granularity semantic description output, we get:

[0017] in, This represents a set of multi-granular semantic descriptions generated from the reference video under modified text constraints.

[0018] Furthermore, step 3 specifically involves: Step 3.1, the preprocessed reference video frame sequence Input video encoder Representation learning is performed to obtain the global visual embedding features of the reference video:

[0019] in, Used to characterize the overall visual content, action process, and scene context semantics of the reference video; Step 3.2, generate the focused global description Input text encoder Semantic encoding is performed to obtain the text embedding features corresponding to the focused global description. :

[0020] in, Used to characterize the semantic information of the target video in its overall action flow, event semantics, and modified global state; Step 3.3: Embed global visual features Text embedding features corresponding to the focus on global description We perform weighted fusion to obtain the first-stage global query features. :

[0021] in, For fusion weighting coefficients.

[0022] Furthermore, step 4 specifically involves: Step 4.1, let the video candidate library be... ,in, This indicates the total number of candidate videos. Indicates the first video candidate in the video candidate library 1 candidate video; process each candidate video using the uniform frame extraction method in step 1. Video preprocessing is performed to obtain the corresponding sampled keyframe sequence. ; Step 4.2, keyframe sequence Input video encoder In this process, the visual embedding features of the corresponding candidate videos are obtained:

[0023] in, Candidate Videos The corresponding visual embedding features represent candidate videos. Visual representation in a unified cross-modal semantic space; Step 4.3, Calculate the global query features for the first stage. With each candidate video Visual embedding features The cosine similarity is used to obtain candidate videos. First stage retrieval score :

[0024] in, Represents the cosine similarity function; Step 4.4: Calculate the first-stage retrieval scores for all candidate videos. Sort in descending order and select the top scorers. Candidate videos, This constitutes the first phase of the candidate video set. .

[0025] Furthermore, step 5 specifically includes: Step 5.1, modify the text Input text encoder Encode the modified text to obtain its text embedding features. :

[0026] Among them, modifying the text embedding features corresponding to the text. Used to characterize the semantics of local modification constraints given by the user; Step 5.2, generate the focus modification description Input text encoder Encode the text to obtain the text embedding features corresponding to the focused modification description. :

[0027] Among them, the focus is on modifying the text embedding features corresponding to the description. Used to characterize the semantics of local action changes, object changes, attribute changes, or interaction changes corresponding to the modified text under the constraints of the reference video context; Step 5.3: Modify the text embedding features corresponding to the text. Text embedding features corresponding to the focus modification description Weighted fusion is performed to obtain the second-stage fine-grained query features. :

[0028] in, These are the weighting coefficients.

[0029] Furthermore, step 6 specifically includes: Step 6.1: Extract the first-stage candidate video set from the candidate video visual embedding features obtained in Step 4.2. Visual embedding features corresponding to each candidate video: Let the candidate video set for the first stage be:

[0030] in, The first phase of candidate video collection The i-th candidate video in the list, ; The corresponding set of visual embedding features is:

[0031] in, For step 4.2, via video encoder The obtained candidate videos The visual embedding features, and these visual embedding features and the second-stage fine-grained query features are located in a unified cross-modal semantic space; Step 6.2: Calculate the correspondence representation of the candidate video in the text matching space and the fine-grained query features of the second stage. The cosine similarity is used to obtain the second-stage fine-grained query similarity score. :

[0032] in, Represents the cosine similarity function. Candidate Videos Visual embedding features in a unified cross-modal semantic space; Step 6.3: Query the similarity score based on the second-stage fine-grained query. For the first phase of candidate video set The candidate videos are sorted in descending order to obtain the final target video ranking result, and the ranking result is then used to sort the candidate videos. The highest-ranking candidate video is selected as the target video. ; Furthermore, the fusion weighting coefficients and weighting coefficients The value is .

[0033] The beneficial effects of this invention are: This invention addresses the problem that existing combined video retrieval methods struggle to simultaneously consider the overall semantic information of the reference video and the constraints of local changes in the modified text, often resulting in videos that are generally similar but have mismatched modifications. It proposes a multi-granularity semantically enhanced video re-ranking method based on a multimodal large language model. This method generates focused global and focused modification descriptions by jointly using the reference video and the modified text. A two-stage retrieval mechanism is then employed: in the first stage, reference video features and focused global description features are fused for candidate video recall; in the second stage, modified text features and focused modification description features are fused to re-rank the candidate videos. This approach more accurately models the overall event semantics and local change semantics of the target video, improving the understanding of modified text and the accuracy of target video retrieval. It also reduces false positives for videos that are generally similar but have mismatched details, thereby enhancing the recall and ranking accuracy of combined video retrieval results. Attached Figure Description

[0034] Figure 1 This is a flowchart of the multi-granularity semantic enhancement video retrieval method based on a multimodal large language model, as described in this invention. Detailed Implementation

[0035] The following detailed description is provided in conjunction with specific implementation methods.

[0036] Example 1 This invention relates to a multi-granularity semantically enhanced video retrieval method based on a multimodal large language model, the process of which is as follows: Figure 1 As shown, the specific steps are as follows: Step 1: Preprocess the input reference video; Step 2: Obtain multi-granular semantic descriptions based on the preprocessed reference video and corresponding modified text; Step 3: Generate the first-stage global query features based on the preprocessed reference video and multi-granular semantic description; Step 4: Select the first-stage candidate video set from the candidate video set based on the first-stage global query features; Step 5: Generate second-stage fine-grained query features based on the modified text and multi-granular semantic descriptions corresponding to the reference video; Step 6: Select the target video from the candidate video set in the first stage based on the fine-grained query features of the second stage.

[0037] Example 2 Based on Example 1, the reference video in step 1 is represented as follows:

[0038] in, This refers to the first video in the reference video. frame, Indicates the total number of frames in the reference video; Reference video The preprocessing specifically involves: processing the reference video. Keyframes are extracted and scaled sequentially to obtain a frame sequence suitable for multimodal model input.

[0039] In step 1, refer to the video. Keyframe extraction is performed using uniform frame extraction, specifically: extracting frames from a reference video. Frame keyframe, Then the sampled reference video :

[0040] in, Indicates the number of frames drawn. This represents the frame index that is uniformly sampled from the original video;

[0041] in, This indicates the rounding operation; To reduce model input redundancy and control computational overhead, the extracted frame images are scaled while maintaining aspect ratio. The scaling in step 1 involves scaling the sampled reference video... Each frame image in the image is scaled while maintaining its aspect ratio, specifically as follows: Set the reference video after sampling The resolution of each frame image in the image is Let the maximum side length threshold be... Then the scaling factor Represented as:

[0042] After scaling, the size of each image frame is 1. ; Obtain the preprocessed reference video frame sequence .

[0043] Example 3 Based on Example 2, step 2 specifically includes: Step 2.1, Constructing Joint Information : ; in, For reference video The corresponding edited text describes the changes that have occurred in the target video relative to the reference video. These are multimodal chain-like thinking prompts used to guide multimodal large language models. ; Step 2.2, combine the information Input into a multimodal large language model to generate focused modification descriptions. :

[0044] in, This represents the mapping process by which a multimodal large language model generates focused modification descriptions under the guidance of multimodal chained thinking prompts; Used to characterize the local action changes, object changes, or attribute changes corresponding to the modified text, the generated focused modification description mainly expresses the modified action fragment itself and does not contain redundant background information unrelated to the modification. Step 2.3, combine the information Input into a multimodal large language model to generate a focused global description. :

[0045] in, This represents the mapping process by which a multimodal large language model generates a focused global description under the guidance of multimodal chained thinking prompts; Used to integrate the modified content into the overall action flow, representing the overall semantics of the target video; the focus is on the global description, which includes not only the modified content, but also the semantics of the actions before and after the modification, to reflect the complete video event process; Step 2.4 will focus on modifying the description. and focus on global description As a multi-granularity semantic description output, we get:

[0046] in, This represents a set of multi-granular semantic descriptions generated from the reference video under modified text constraints.

[0047] This step guides a multimodal large language model to perform joint reasoning on the reference video and modified text through a multidimensional chain-like thinking prompt template, generating textual description information with different semantic granularities. This solves the problems of the original modified text being too short, lacking semantic information, and being difficult to fully align with the visual content.

[0048] Example 4 Based on Example 3, step 3 specifically includes: Step 3.1, the preprocessed reference video frame sequence Input video encoder Representation learning is performed to obtain the global visual embedding features of the reference video:

[0049] in, Used to characterize the overall visual content, action process, and scene context semantics of the reference video; Step 3.2, generate the focused global description Input text encoder Semantic encoding is performed to obtain the text embedding features corresponding to the focused global description. :

[0050] in, Used to characterize the semantic information of the target video in its overall action flow, event semantics, and modified global state; Step 3.3: Embed global visual features Text embedding features corresponding to the focus on global description We perform weighted fusion to obtain the first-stage global query features. :

[0051] in, To integrate the weighting coefficients, This is used to adjust the contribution ratio of reference video visual information and focus global semantic information in the first stage of retrieval, so as to achieve a balanced fusion of the original visual content of the reference video and the modified overall semantic information. Step 4 specifically involves: Step 4.1, let the video candidate library be... ,in, This indicates the total number of candidate videos. Indicates the first video candidate in the video candidate library 1 candidate video; process each candidate video using the uniform frame extraction method in step 1. Video preprocessing is performed to obtain the corresponding sampled keyframe sequence. ; Step 4.2, keyframe sequence Input video encoder In this process, the visual embedding features of the corresponding candidate videos are obtained:

[0052] in, Candidate Videos The corresponding visual embedding features represent candidate videos. Visual representation in a unified cross-modal semantic space; Step 4.3, Calculate the global query features for the first stage. With each candidate video Visual embedding features The cosine similarity is used to obtain candidate videos. First stage retrieval score :

[0053] in, The cosine similarity function is used to perform preliminary screening of candidate videos by comprehensively utilizing the original visual information of the reference video and the modified overall semantic information provided by the global focus description.

[0054] Step 4.4: Calculate the first-stage retrieval scores for all candidate videos. Sort in descending order and select the top scorers. Candidate videos, This constitutes the first phase of the candidate video set. .

[0055] After the first stage of retrieval, candidate videos that are similar to the overall content of the reference video and are relatively consistent with the modified global semantics can be quickly recalled from the video candidate library, thus narrowing the search scope for subsequent re-ranking while ensuring recall capability.

[0056] By fusing the visual features of the reference video with the focused global descriptive features in steps 3 and 4, the first-stage retrieval process simultaneously considers the original video content and the overall event semantics of the target video, thereby improving the candidate recall quality of the target video.

[0057] Example 5 Based on Example 4, step 5 specifically includes: Step 5.1, modify the text Input text encoder Encode the modified text to obtain its text embedding features. :

[0058] Among them, modifying the text embedding features corresponding to the text. Used to characterize the semantics of local modification constraints given by the user; Step 5.2, generate the focus modification description Input text encoder Encode the text to obtain the text embedding features corresponding to the focused modification description. :

[0059] Among them, the focus is on modifying the text embedding features corresponding to the description. Used to characterize the semantics of local action changes, attribute changes, or interaction changes corresponding to the modified text under the constraints of the reference video context; Step 5.3: Modify the text embedding features corresponding to the text. Text embedding features corresponding to the focus modification description Weighted fusion is performed to obtain the second-stage fine-grained query features. :

[0060] in, These are the weighting coefficients. .

[0061] Step 6 specifically involves: Step 6.1: Extract the first-stage candidate video set from the candidate video visual embedding features obtained in Step 4.2. Visual embedding features corresponding to each candidate video: Let the candidate video set for the first stage be:

[0062] in, The first phase of candidate video collection The i-th candidate video in the list, ; The corresponding set of visual embedding features is:

[0063] in, For step 4.2, via video encoder The obtained candidate videos The visual embedding features, and these visual embedding features and the second-stage fine-grained query features are located in a unified cross-modal semantic space; Step 6.2: Calculate the correspondence representation of the candidate video in the text matching space and the fine-grained query features of the second stage. The cosine similarity is used to obtain the second-stage fine-grained query similarity score. :

[0064] in, Represents the cosine similarity function. Candidate Videos Visual embedding features in a unified cross-modal semantic space.

[0065] Step 6.3: Query the similarity score based on the second-stage fine-grained query. For the first phase of candidate video set The candidate videos are sorted in descending order to obtain the final target video ranking result, and the ranking result is then used to sort the candidate videos. The highest-ranking candidate video is selected as the target video. .

[0066] By introducing focused modification descriptions to semantically enhance the modified text, the second-stage re-ranking process can more accurately match local change conditions, reduce the probability of videos that are only similar in overall content but do not meet the modification requirements being ranked higher, and improve the accuracy of the final video retrieval results.

[0067] Example 6 Based on Example 5, in order to verify the effectiveness of the multi-granularity semantic enhancement video retrieval method based on a multimodal large language model proposed in this invention, a practical video retrieval experiment was conducted on a combined video retrieval dataset.

[0068] The experiment used reference videos and corresponding modified text as query conditions, and constructed a video candidate library using candidate videos from the dataset. Seven different combinations of video encoders and text encoders were constructed using cross-modal coding models such as LanguageBind, EgoVLPv2, CLIP, and BLIP to verify the applicability of the proposed method under different feature coding models.

[0069] The baseline method directly performs combined video retrieval based on the reference video and modified text, without incorporating the focused global description and focused modification description generated by the multimodal large language model. The method of this invention uses GPT-4, GPT-4o, Video-R1, and GPT-5.5 as multimodal large language models, respectively, to generate focused global descriptions and focused modification descriptions based on the reference video and modified text, and performs first-stage candidate video recall and second-stage fine-grained re-ranking according to steps 3 to 6. Except for the different combinations of multimodal large language models and video / text encoders, all other parameters and the candidate video set remain consistent.

[0070] The experiment used recall rates Recall@1, Recall@2, and Recall@3 as evaluation metrics. This indicates that the target video appears before the search results. The proportion of query samples in each candidate video to the total number of query samples is calculated as follows:

[0071] in, This indicates the total number of samples queried. This indicates that the target video appears before the search results. Number of query samples per bit. A higher value indicates better video retrieval performance. The experimental results are shown in Table 1. In Table 1, the three data points in the baseline method and this method columns represent the calculated recall rates Recall@1, Recall@2, and Recall@3, respectively. Table 1

[0072] As shown in Table 1, the method of this invention achieved superior retrieval results compared to benchmark methods under different combinations of video encoders, text encoders, and multimodal large language models, indicating that the method has good cross-model applicability. Experimental results demonstrate that by generating focused global and focused modification descriptions, and combining them with a two-stage retrieval mechanism, the semantic information in the original modified text can be effectively supplemented, enhancing the model's understanding of the overall content and local changes of the target video, thereby improving the recall performance and ranking accuracy of combined video retrieval.

Claims

1. A multi-granularity semantically enhanced video retrieval method based on a multimodal large language model, characterized in that, The specific steps are as follows: Step 1: Preprocess the input reference video; Step 2: Obtain multi-granular semantic descriptions based on the preprocessed reference video and corresponding modified text; Step 3: Generate the first-stage global query features based on the preprocessed reference video and multi-granular semantic description; Step 4: Select the first-stage candidate video set from the candidate video set based on the first-stage global query features; Step 5: Generate second-stage fine-grained query features based on the modified text and multi-granular semantic descriptions corresponding to the reference video; Step 6: Select the target video from the candidate video set in the first stage based on the fine-grained query features of the second stage.

2. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 1, characterized in that, The reference video in step 1 is represented as follows: in, This refers to the first video in the reference video. frame, Indicates the total number of frames in the reference video; Reference video The preprocessing specifically involves: processing the reference video. Keyframe extraction and scaling are performed sequentially.

3. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 2, characterized in that, In step 1, the reference video Keyframe extraction is performed using uniform frame extraction, specifically: extracting frames from a reference video. Frame keyframe, Then the sampled reference video : in, Indicates the number of frames drawn. This represents the frame index that is uniformly sampled from the original video; in, This indicates the rounding operation; The scaling in step 1 refers to: scaling the sampled reference video. Each frame image in the image is scaled while maintaining its aspect ratio, specifically as follows: Set the reference video after sampling The resolution of each frame image in the image is Let the maximum side length threshold be... Then the scaling factor Represented as: After scaling, the size of each image frame is 1. ; Obtain the preprocessed reference video frame sequence .

4. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 3, characterized in that, Step 2 specifically involves: Step 2.1, Constructing Joint Information : ； in, For reference video The corresponding modified text, Provides prompts for multimodal chain thinking; Step 2.2, combine the information Input into a multimodal large language model to generate focused modification descriptions. : in, This represents the mapping process by which a multimodal large language model generates focused modification descriptions under the guidance of multimodal chained thinking prompts; Step 2.3, combine the information Input into a multimodal large language model to generate a focused global description. : in, This represents the mapping process by which a multimodal large language model generates a focused global description under the guidance of multimodal chained thinking prompts; Step 2.4, modify the focus description. and focus on global description As a multi-granularity semantic description output, we get: in, This represents a set of multi-granular semantic descriptions generated from the reference video under modified text constraints.

5. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 4, characterized in that, Step 3 specifically involves: Step 3.1, the preprocessed reference video frame sequence Input video encoder Representation learning is performed to obtain the global visual embedding features of the reference video: in, Used to characterize the overall visual content, action process, and scene context semantics of the reference video; Step 3.2, generate the focused global description Input text encoder Semantic encoding is performed to obtain the text embedding features corresponding to the focused global description: in, To focus on the text embedding features corresponding to the global description; Step 3.3: Embed global visual features Text embedding features corresponding to the focus on global description We perform weighted fusion to obtain the first-stage global query features. : in, For fusion weighting coefficients.

6. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 5, characterized in that, Step 4 specifically involves: Step 4.1, let the video candidate library be... ,in, This indicates the total number of candidate videos. Indicates the first video candidate in the video candidate library 1 candidate video; process each candidate video using the uniform frame extraction method in step 1. Video preprocessing is performed to obtain the corresponding sampled keyframe sequence. ; Step 4.2, keyframe sequence Input video encoder In this process, the visual embedding features of the corresponding candidate videos are obtained: in, Candidate Videos Corresponding visual embedding features; Step 4.3, Calculate the global query features for the first stage. With each candidate video Visual embedding features The cosine similarity is used to obtain candidate videos. First stage retrieval score : in, Represents the cosine similarity function; Step 4.4: Calculate the first-stage retrieval scores for all candidate videos. Sort in descending order and select the top scorers. Candidate videos, This constitutes the first phase of the candidate video set. .

7. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 6, characterized in that, Step 5 specifically involves: Step 5.1, modify the text Input text encoder Encode the modified text to obtain its text embedding features: in, To modify the text embedding features corresponding to the text; Step 5.2, generate the focus modification description Input text encoder Encode the text to obtain the text embedding features corresponding to the focused modification description: in, To focus on modifying the text embedding features corresponding to the description; Step 5.3: Modify the text embedding features corresponding to the text. Text embedding features corresponding to the focus modification description Weighted fusion is performed to obtain the second-stage fine-grained query features. : in, These are the weighting coefficients.

8. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 7, characterized in that, Step 6 specifically involves: Step 6.1: Extract the first-stage candidate video set from the candidate video visual embedding features obtained in Step 4.

2. Visual embedding features corresponding to each candidate video: set up The first phase of candidate videos is as follows: in, The first phase of candidate video collection The i-th candidate video in the list, ; The corresponding set of visual embedding features is: in, For step 4.2, via video encoder The obtained candidate videos The visual embedding features, and these visual embedding features and the second-stage fine-grained query features are located in a unified cross-modal semantic space; Step 6.2: Calculate the correspondence representation of the candidate video in the text matching space and the fine-grained query features of the second stage. The cosine similarity is used to obtain the second-stage fine-grained query similarity score. : in, Represents the cosine similarity function. Candidate Videos Visual embedding features in a unified cross-modal semantic space; Step 6.3: Query the similarity score based on the second-stage fine-grained query. For the first phase of candidate video set The candidate videos are sorted in descending order to obtain the final target video ranking result, and the ranking result is then used to... The highest-ranking candidate video is selected as the target video. .

9. The multi-granularity semantic enhancement video retrieval method based on a multimodal large language model according to claim 8, characterized in that, The fusion weight coefficient and weighting coefficients The value is .

Images