A video shot recommendation method based on a multi-modal large model

Abstract

The application discloses a video shot recommendation method based on a multimodal large model, S1: video shots are split; S2: the split shots are labeled and information is identified; S3: the shot information is converted into a vector form and stored in an ES database; S4: an intermediate frame vector is processed through a CLIP model and a text information vector is processed through a BERT model; S5: keywords are generated according to user requirements, the keywords are vectorized, and then searched; S6: the searched shot content is reordered, and a recommended shot is output. Through deep analysis of video images, audio, text and other multi-source information, video semantic connotation is mined through semantic search, keywords are generated according to user requirements, the keywords are vectorized, the searched shot content is reordered, and a recommended shot is output, so that full-link intelligentization from data processing to accurate recommendation is realized, and the search result is highly matched with the user requirements while meeting the requirements of a fuzzy search scene.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video shot recommendation method based on a multimodal large model. Background Technology

[0002] With the explosive growth of video data, how to quickly and accurately retrieve the desired content from massive amounts of video has become an urgent problem to be solved. Traditional video retrieval methods are usually based on keyword matching or simple visual feature extraction, which cannot fully understand the semantic information of video content. For example, when dealing with complex scenes and events, traditional methods struggle to accurately identify people, behaviors, and related details in the scene, resulting in low accuracy and relevance of search results. In addition, existing video retrieval systems lack a deep understanding of user needs and personalized recommendation capabilities when recommending shots, failing to meet users' needs for efficient video production. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a video shot recommendation method based on a multimodal large model.

[0004] The objective of this invention is achieved through the following technical solution: a video shot recommendation method based on a multimodal large model, comprising the following steps:

[0005] S1: Segment the video footage;

[0006] S2: Label the segmented shots and perform information recognition;

[0007] S3: Convert the lens information into vector form and store it in the ES database;

[0008] S4: Process intermediate frame vectors using the CLIP model and text information vectors using the BERT model;

[0009] S5: Generate keywords based on user needs, vectorize the keywords, and then perform a search;

[0010] S6: Reorder the retrieved footage and output recommended footage.

[0011] Preferably, step S1 further includes the following step:

[0012] S11: Split the input video into a continuous frame sequence according to the time series. ;

[0013] S12: Extract hue, saturation, brightness, and edge features;

[0014] S13: Perform weighted summation and normalization on the four-dimensional difference values ​​extracted in step S12, and calculate the difference values ​​between adjacent frames. and Difference score ,

[0015] ;

[0016] in, This represents the weighting coefficient for hue differences. For the first Frame and the Differences in pixel tone values ​​between frames The weighting coefficients for saturation differences. For the first Frame and the Differences in pixel saturation between frames The weighting coefficient for brightness difference. For the first Frame and the Differences in pixel brightness values ​​between frames These are the weighting coefficients for the differences in edge features. For the first Frame and the Differences in edge features between frames.

[0017] Preferably, step S2 further includes the following step:

[0018] S21: Independent shot sequence obtained after video shot segmentation As input data, they are sequentially fed into the large model processing module;

[0019] S22: The large model processing module performs preliminary semantic analysis on each shot, generating a basic tag set for each shot. ;

[0020] S23: Extract corresponding information from the shot using a classifier and regression model.

[0021] Preferably, in step S3, the image information is mapped into a vector with semantic representation through the CLIP model, and the text field and GLM_description field in the shot are vectorized through the natural language processing model to generate the corresponding text_vector and GLM_description_vector, and an index named footage_vector is created in the ES database.

[0022] Preferably, in step S4, the specific steps for the CLIP model to process the intermediate frame vector are as follows:

[0023] S41: Using bilinear interpolation algorithm to... The pixels are uniformly scaled to 224×224 pixels and then normalized.

[0024] ;

[0025] in, The pixel mean of the dataset. is the standard deviation of pixels in the dataset;

[0026] S42: Transfer the preprocessed intermediate frames The visual encoder input to the CLIP model Image features are extracted through a multi-layer Transformer architecture.

[0027] ;

[0028] in, For visual feature vectors, For vector dimensions;

[0029] S43: Visual feature vectors Perform linear projection to generate the final intermediate frame semantic vector. ,

[0030] ;

[0031] in, Let be the projection matrix. For bias vectors, This is an L2 normalization operation.

[0032] Preferably, in step S4, the specific steps for the BERT model to process text information vectors are as follows:

[0033] A41: Facing the camera Scene content description and event description After text cleaning, the WordPiece tokenizer is used to convert the text into a sequence of subwords.

[0034] ;

[0035] ;

[0036] Add marker and Construct the input sequence,

[0037] ;

[0038] A42: The preprocessed input sequence Input into BERT model Context-dependent word vector representations are extracted through a multi-layer Transformer architecture.

[0039] ;

[0040] in, The hidden layer dimension of the BERT model;

[0041] A43: Adopt The vector corresponding to the tag As a semantic representation of the entire text, the final text vector is generated through pooling layers and linear transformations.

[0042] ;

[0043] in, Let be the projection matrix. For bias vectors, This is the activation function.

[0044] Preferably, step S5 further includes the following step:

[0045] S51: Extract the core words input by the user, expand their semantics, and classify them according to a five-dimensional structure of scene / action / emotion / style / technology;

[0046] S52: Keyword set generated for the large model Perform cleaning and normalization processes;

[0047] S53: Based on the Skip-Gram architecture, it learns word vector representations by optimizing the objective function.

[0048] ;

[0049] in, The current keyword is For the word vectors of the context words, Keywords Word vectors, These are model parameters;

[0050] S54: Store the vectorized keyword vectors in the ES vector database and create an index for each vector;

[0051] S55: When it is necessary to retrieve shot information related to keywords, the similarity between the keyword vector to be retrieved and all vectors in the ES vector database is calculated.

[0052] ;

[0053] in, For vector dot product, For vectors norm, For vectors norm,

[0054] The vectors are sorted from highest to lowest based on the calculated similarity scores, and those with similarity scores higher than a preset threshold are selected. The vector is used to obtain the corresponding lens information.

[0055] Preferably, step S6 further includes the following step:

[0056] S61: Use the CLIP model to perform vector transformation on the middle frame of the shot and the search terms respectively;

[0057] S62: Calculate the initial similarity between the retrieval word vector and the intermediate frame vectors of each shot.

[0058] ;

[0059] Obtain the initial similarity score set.

[0060] ;

[0061] S63: Constructing the Lens Content Feature Matrix Based on this matrix, the cosine similarity between each pair of shots is calculated to generate a shot content similarity matrix. ,

[0062] ;

[0063] S64: Based on the similarity of content between shots Similarity to the initial For each shot Perform weighted sorting.

[0064] ;

[0065] in, These are the weight parameters.

[0066] This invention has the following advantages: By segmenting video shots and converting shot information into vector form, this invention processes intermediate frame vectors using the CLIP model and text information vectors using the BERT model, deeply analyzing multi-source information such as video images, audio, and text. Then, it mines the semantic connotation of the video through semantic retrieval, generates keywords based on user needs, vectorizes the keywords, reorders the retrieved shot content, and outputs recommended shots. This achieves full-link intelligence from data processing to accurate recommendation, ensuring that the search results highly match user needs and simultaneously meet the requirements of fuzzy search scenarios. Attached Figure Description

[0067] Figure 1 A schematic diagram of the process for recommending video shots. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0069] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0070] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0071] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0072] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0073] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] In this embodiment, as Figure 1 As shown, a video shot recommendation method based on a multimodal large model includes the following steps:

[0075] S1: Segment the video footage;

[0076] S2: Label the segmented shots and perform information recognition;

[0077] S3: Convert the lens information into vector form and store it in the ES database;

[0078] S4: Process intermediate frame vectors using the CLIP model and text information vectors using the BERT model;

[0079] S5: Generate keywords based on user needs, vectorize the keywords, and then perform a search;

[0080] S6: Reorder the retrieved footage and output recommended footage. By segmenting video footage and converting the footage information into vector form, the CLIP model processes the intermediate frame vectors, and the BERT model processes the text information vectors. This allows for deep analysis of multi-source information such as video images, audio, and text. Semantic retrieval is then used to mine the semantic connotation of the video, generate keywords based on user needs, and vectorize the keywords. The retrieved footage is then reordered to output recommended footage, achieving end-to-end intelligent processing from data processing to accurate recommendation. This ensures that the search results highly match user needs and simultaneously meets the requirements of fuzzy search scenarios.

[0081] Furthermore, step S1 also includes the following steps:

[0082] S11: Split the input video into a continuous frame sequence according to the time series. ;

[0083] S12: Extract hue, saturation, brightness, and edge features;

[0084] S13: Perform weighted summation and normalization on the four-dimensional difference values ​​extracted in step S12, and calculate the difference values ​​between adjacent frames. and Difference score ,

[0085] ;

[0086] in, This represents the weighting coefficient for hue differences. For the first Frame and the Differences in pixel tone values ​​between frames The weighting coefficients for saturation differences. For the first Frame and the Differences in pixel saturation between frames The weighting coefficient for brightness difference. For the first Frame and the Differences in pixel brightness values ​​between frames These are the weighting coefficients for the differences in edge features. For the first Frame and the Differences in edge features between frames. Specifically, It is obtained by calculating the average absolute difference of pixel tone values ​​between two frames and is used to reflect changes in the color type of the image. For example, when switching from a red scene to a blue scene, this value will increase significantly. This is determined by comparing statistical differences in pixel saturation, reflecting changes in the intensity of colors in the image. For example, this value changes when switching from a high-saturation scene to a low-saturation scene. It is a value calculated based on the difference in pixel brightness values, used to measure changes in the brightness of an image. For example, this value will increase significantly when switching from a daytime scene to a nighttime scene. By extracting frame edge contours using algorithms such as Canny edge detection and calculating contour feature differences, changes in the shape and structure of objects in the scene can be reflected. For example, abrupt changes in the object contour during camera transitions can significantly increase this value. The value ranges from 0 to 255.0. The higher the score, the greater the difference in content between the current frame and the previous frame, and the more likely it is to be a shot transition point. Used to adjust the importance of tonal differences in the overall score, increasing... This means that when calculating the difference score, more attention will be paid to the tonal changes between adjacent frames; Used to control the contribution of saturation difference to the overall score, if A larger value indicates that when judging inter-frame differences, more emphasis is placed on changes in color vibrancy; The proportion of brightness difference in the overall score is determined when At higher levels, inter-frame brightness variations have a greater impact on the difference score; This is used to adjust the influence of edge feature differences on the overall score. Since edge feature difference values ​​are usually large, they may need to be appropriately reduced in practical applications. To avoid false detections. This is the sum of all weight coefficients, used to normalize the weighted sum of the numerator. By dividing by the total weight sum, we can obtain the result under different weight configurations. To maintain a consistent score range and avoid the influence of the total weighted average on the score, ensuring score comparability and stability, the weighted parameters can be flexibly adjusted according to the video type. For example, for animated videos, the weighted average can be increased. To highlight the impact of tonal variations on shot composition; for documentaries, this can be appropriately increased. This emphasizes the role of changes in scene structure.

[0087] Furthermore, step S2 also includes the following steps:

[0088] S21: Independent shot sequence obtained after video shot segmentation As input data, they are sequentially fed into the large model processing module;

[0089] S22: The large model processing module performs preliminary semantic analysis on each shot, generating a basic tag set for each shot. ,in, For the first The first shot A basic tag;

[0090] S23: Extracting corresponding information from the shots using classifiers and regression models. Specifically, the shot data is input into a multimodal large model, which integrates visual, language, and audio processing capabilities. It achieves cross-modal information fusion through a Transformer architecture. In visual modality processing, convolutional neural networks (CNNs) are used to extract visual features of the shot images, including scene composition, object shapes, and colors. In language modality processing, natural language processing (NLP) techniques are used to parse text information such as subtitles and narration corresponding to the shots. Simultaneously, an audio analysis module is used to identify audio features such as speech and ambient sounds in the shots. The multimodal large model deeply fuses the above multi-dimensional information, adaptively allocating modal weights through an attention mechanism to achieve a comprehensive understanding of the shot content. During information extraction, a scene classification model is used to identify specific scenes in the shots (such as indoors, outdoors, cities, forests, etc.); object detection algorithms are used to locate and identify key objects such as people and objects in the shots; an event understanding model is used to analyze the events and actions occurring in the shots; and a recognition model, combining image pixel features and machine learning algorithms, is used to determine information about people in the shots. Finally, the extracted information is integrated with the basic tag set to form a complete tag system containing detailed information such as scenes, people, scene content, and events, providing rich semantic data support for subsequent vector storage and retrieval recommendations.

[0091] In this embodiment, in step S3, the image information is mapped into a vector with semantic representation using the CLIP model. A natural language processing model is used to vectorize the text and GLM_description fields in the shot, generating corresponding text_vectors and GLM_description_vectors. An index named footage_vector is then created in the ES database. Specifically, for each video shot segmented and whose information recognition is completed, the CLIP model is used to process the intermediate frames of the shot. The CLIP model can map image information into a vector with semantic representation. The generated vector is denoted as image_vector (the visual semantic vector generated after processing the intermediate frame of the shot by the CLIP model). This vector contains the visual semantic information of the intermediate frame of the shot. Simultaneously, a natural language processing model (such as BERT, GLM, etc.) is used to vectorize the content of the text and GLM_description fields in the shot, generating corresponding text_vectors (the semantic vector corresponding to the information matched by speech recognition, generated by the natural language processing model) and GLM_description_vectors (the semantic vector corresponding to the information generated by the large model). These vectors extract the deep semantic features of the text.

[0092] Furthermore, in step S4, the specific steps of the CLIP model in processing the intermediate frame vectors are as follows:

[0093] S41: Using bilinear interpolation algorithm to... The pixels are uniformly scaled to 224×224 pixels and then normalized.

[0094] ;

[0095] in, The pixel mean of the dataset. is the standard deviation of pixels in the dataset;

[0096] S42: Transfer the preprocessed intermediate frames The visual encoder input to the CLIP model Image features are extracted through a multi-layer Transformer architecture.

[0097] ;

[0098] in, For visual feature vectors, The vector dimension is 768 or 1024.

[0099] S43: Visual feature vectors Perform linear projection to generate the final intermediate frame semantic vector. ,

[0100] ;

[0101] in, Let be the projection matrix. For bias vectors, This is an L2 normalization operation to ensure that the vector magnitude is 1, which facilitates subsequent similarity calculation.

[0102] Furthermore, in step S4, the specific steps by which the BERT model processes text information vectors are as follows:

[0103] A41: Facing the camera Scene content description and event description After text cleaning, the WordPiece tokenizer is used to convert the text into a sequence of subwords.

[0104] ;

[0105] ;

[0106] Add marker and Construct the input sequence,

[0107] ;

[0108] A42: Input the preprocessed input sequence into the BERT model , and extract the context-related word vector representation through a multi-layer Transformer architecture,

[0109] ;

[0110] Among them, is the hidden layer dimension of the BERT model, which is 768 or 1024;

[0111] A43: Adopt the vector corresponding to the token as the semantic representation of the entire text, and generate the final text vector through a pooling layer and a linear transformation,

[0112] ;

[0113] Among them, is the projection matrix, is the bias vector, is the activation function to ensure that the vector element values are in the interval [-1, 1]. Specifically, the main role of text cleaning is to remove special characters and perform lowercase conversion.

[0114] In this embodiment, in step S5, the following steps are further included:

[0115] S51: Extract the core words input by the user, perform semantic expansion, and classify them according to the five-dimensional structure of scenario / action / sentiment / style / technology;

[0116] S52: Clean and normalize the keyword set generated by the large model; Specifically, the specific role of this step is to remove stop words (such as "de", "le", "zai", etc.), special symbols in the keywords, and uniformly convert all keywords to lowercase form.<000039​​​​​​​​​​​​​​​​​​These are the model parameters; the keyword set is obtained by training the model. Transform into the corresponding vector set , This represents the number of keywords after deduplication.

[0120] S54: Store the vectorized keyword vectors in the ES vector database and create an index for each vector; specifically, in the ES vector database, use the HNSW algorithm to build the index structure, which can efficiently organize high-dimensional vector data and reduce retrieval time complexity.

[0121] S55: When it is necessary to retrieve shot information related to keywords, the similarity between the keyword vector to be retrieved and all vectors in the ES vector database is calculated.

[0122] ;

[0123] in, For vector dot product, For vectors norm, For vectors norm,

[0124] The vectors are sorted from highest to lowest based on the calculated similarity scores, and those with similarity scores higher than a preset threshold are selected. The vector is used to obtain the corresponding lens information, for example, if a preset threshold is used. Then all will be filtered out. vector And its associated shot information. Specifically, the core elements of the user input information are first broken down, that is, the large model needs to prioritize the identification of the following five core dimensions: the theme and core content of the video (such as "sci-fi movie fight scene"); detailed scene description (covering time, place, environment, props such as "cars in a rainy scene at the beach at dusk"); specific actions and plot (such as "the protagonist running" and "the crowd cheering"); emotional tone and artistic style (such as "suspense atmosphere" or "cyberpunk style"); and technical requirements (including shot type, shooting technique, and duration requirements such as "close-up aerial shot of 5 seconds"). Secondly, the large model analyzes video requirements through a three-step keyword generation method: First, it extracts the core words input by the user (such as "cherry blossom tree" and "spaceship explosion"); then it expands semantically—abstract concepts ("walking" extends to "romance"), associative similarities ("cherry blossom" is associated with "spring flowers"), and supplements technical information ("aerial photography" corresponds to "drone perspective"); finally, it is structurally classified according to five dimensions: scene / action / emotion / style / technology (such as breaking down "running in the ruins at dusk" into scene + action categories), forming a tag system for an efficient matching lens retrieval system. For example, for the user's creative needs of "urban white-collar workers returning home late at night after working overtime", the large model first generates basic keywords (urban white-collar workers / overtime / subway / streetlights / street scene); then, it supplements the emotional dimension (loneliness / fatigue) and scene details (empty streets / neon lights / back view) through semantic expansion; finally, it outputs technical keywords in a structured manner: using night scene shooting and handheld lenses to enhance the documentary feel, using cool tones with long exposure to render the atmosphere, using close-up shots to capture facial fatigue, and systematically presenting the core elements of the theme.

[0125] In this embodiment, step S6 further includes the following step:

[0126] S61: Use the CLIP model to perform vector transformation on the middle frame of the shot and the search terms respectively;

[0127] S62: Calculate the initial similarity between the retrieval word vector and the intermediate frame vectors of each shot.

[0128] ;

[0129] Obtain the initial similarity score set.

[0130] ;

[0131] S63: Constructing the Lens Content Feature Matrix Based on this matrix, the cosine similarity between each pair of shots is calculated to generate a shot content similarity matrix. ,

[0132] ;

[0133] S64: Based on the similarity of content between shots Similarity to the initial For each shot Perform weighted sorting.

[0134] ;

[0135] in, For weight parameters, This is used to balance the impact of direct similarity between search terms and shots, and the content relevance between shots. Specifically, the intermediate frame processing procedure is as follows: for each retrieved shot, its intermediate frame image is extracted. , The number of retrieved shots is then input into the CLIP model's image encoder. The feature vector of the intermediate frame is obtained. The search term processing procedure is as follows: The user-input search terms are processed... After text preprocessing (such as stop word removal and formatting), the text encoder is input into the CLIP model. The feature vector of the search term is obtained. .

[0136] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video shot recommendation method based on a multimodal large model, characterized in that: Includes the following steps: S1: Segment the video footage; S2: Label the segmented shots and perform information recognition; S3: Convert the lens information into vector form and store it in the ES database; S4: Process the intermediate frame vectors using the CLIP model and the text information vectors using the BERT model. The text information includes shot details. Scene content description and event description ; S5: Generate keywords based on user needs, vectorize the keywords, and then perform a search; S6: Reorder the retrieved footage and output recommended footage; Step S6 further includes the following steps: S61: Use the CLIP model to perform vector transformation on the middle frame of the shot and the search terms respectively; S62: Calculate the initial similarity between the retrieval word vector and the intermediate frame vectors of each shot. ； Obtain the initial similarity score set. ； S63: Constructing the Lens Content Feature Matrix Based on this matrix, the cosine similarity between each pair of shots is calculated to generate a shot content similarity matrix. , ； S64: Based on the similarity of content between shots Similarity to the initial For each shot Perform weighted sorting. ； in, These are the weight parameters.

2. The video shot recommendation method based on a multimodal large model according to claim 1, characterized in that: Step S1 further includes the following steps: S11: Split the input video into a continuous frame sequence according to the time series. ; S12: Extract hue, saturation, brightness, and edge features; S13: Perform weighted summation and normalization on the four-dimensional difference values ​​extracted in step S12, and calculate the difference values ​​between adjacent frames. and Difference score , ； in, This represents the weighting coefficient for hue differences. For the first Frame and the Differences in pixel tone values ​​between frames The weighting coefficients for saturation differences. For the first Frame and the Differences in pixel saturation between frames The weighting coefficient for brightness difference. For the first Frame and the Differences in pixel brightness values ​​between frames These are the weighting coefficients for the differences in edge features. For the first Frame and the Differences in edge features between frames.

3. The video shot recommendation method based on a multimodal large model according to claim 2, characterized in that: Step S2 further includes the following steps: S21: Independent shot sequence obtained after video shot segmentation As input data, they are sequentially fed into the large model processing module; S22: The large model processing module performs preliminary semantic analysis on each shot, generating a basic tag set for each shot. ; S23: Extract corresponding information from the shot using a classifier and regression model.

4. The video shot recommendation method based on a multimodal large model according to claim 3, characterized in that: In step S3, the image information is mapped into a vector with semantic representation through the CLIP model. The text field and GLM_description field in the shot are vectorized through the natural language processing model to generate the corresponding text_vector and GLM_description_vector. An index named footage_vector is created in the ES database.

5. The video shot recommendation method based on a multimodal large model according to claim 4, characterized in that: In step S4, the specific steps of the CLIP model in processing the intermediate frame vector are as follows: S41: Using bilinear interpolation algorithm to... The pixels are uniformly scaled to 224×224 pixels and then normalized. ； in, The pixel mean of the dataset. is the standard deviation of pixels in the dataset; S42: Transfer the preprocessed intermediate frames The visual encoder input to the CLIP model Image features are extracted through a multi-layer Transformer architecture. ； in, For visual feature vectors, For vector dimensions; S43: Visual feature vectors Perform linear projection to generate the final intermediate frame semantic vector. , ； in, Let be the projection matrix. For bias vectors, This is an L2 normalization operation.

6. The video shot recommendation method based on a multimodal large model according to claim 5, characterized in that: In step S4, the specific steps of the BERT model in processing text information vectors are as follows: A41: Facing the camera Scene content description and event description After text cleaning, the WordPiece tokenizer is used to convert the text into a sequence of subwords. ； ； Add marker and Construct the input sequence, ； A42: The preprocessed input sequence Input into BERT model Context-dependent word vector representations are extracted through a multi-layer Transformer architecture. ； in, The hidden layer dimension of the BERT model; A43: Adopt The vector corresponding to the tag As a semantic representation of the entire text, the final text vector is generated through pooling layers and linear transformations. ； in, For the projection matrix, For bias vectors, This is the activation function.

7. The video shot recommendation method based on a multimodal large model according to claim 6, characterized in that: Step S5 also includes the following steps: S51: Extract the core words input by the user, expand their semantics, and classify them according to a five-dimensional structure of scene / action / emotion / style / technology; S52: Keyword set generated for the large model Perform cleaning and normalization processes; S53: Based on the Skip-Gram architecture, it learns word vector representations by optimizing the objective function. ； in, The current keyword is For the word vectors of the context words, Keywords Word vectors, These are model parameters; S54: Store the vectorized keyword vectors in the ES vector database and create an index for each vector; S55: When it is necessary to retrieve shot information related to keywords, the similarity between the keyword vector to be retrieved and all vectors in the ES vector database is calculated. ； in, For vector dot product, For vectors norm, For vectors norm, The vectors are sorted from highest to lowest based on the calculated similarity scores, and those with similarity scores higher than a preset threshold are selected. The vector is used to obtain the corresponding lens information.

Images