Video segment retrieval method and system based on large model, terminal and medium

Abstract

The invention belongs to the field of video retrieval, and particularly discloses a video segment retrieval method and system based on a large model, a terminal and a medium, and the method comprises the steps: segmenting each piece of original video data into a plurality of video segments; the video segments are input into a pre-trained multi-modal large model, feature descriptions and target video frames of the video segments are extracted, and the feature descriptions comprise natural language feature descriptions and multi-modal feature vectors; combining the natural language feature description of each video segment, the multi-modal feature vector and the target video frame to obtain video segment feature representation; storing the video segment feature representation in a database; and obtaining retrieval information input by a user, comparing the retrieval information with each video segment feature representation in the database, screening out video data matched with the retrieval information, and feeding back the video data to the user. According to the method, the accuracy of video segment retrieval is improved by fusing the natural language feature description and the multi-modal feature vector and accurately extracting the target video frame.

Description

Technical Field

[0001] The present invention belongs to the field of video retrieval, and in particular relates to a video segment retrieval method, system, terminal and medium based on a large model. Background Art

[0002] With the explosive growth of multimedia data, video retrieval technology has become an important research direction in the field of information retrieval. Traditional video retrieval methods mainly rely on keyword matching, content-based retrieval (such as color, texture, motion features) or metadata search. However, these methods have limitations in dealing with complex video content and users' diverse query needs, especially when dealing with large-scale video data, it is often difficult to balance efficiency and accuracy.

[0003] In recent years, the development of deep learning technology, especially the rise of large-scale pre-trained models (big models), has brought new opportunities for video retrieval. Big models have achieved remarkable results in image recognition, natural language processing and other fields due to their powerful representation learning and cross-modal fusion capabilities. In the field of video retrieval, some studies have attempted to use deep learning models to extract video features. However, most methods are still limited to the extraction of single modality features (such as visual features), and the internal structure and key frames of video clips are not fully utilized, which affects the accuracy of video retrieval. Summary of the invention

[0004] To solve the above problems, the present invention provides a video segment retrieval method, system, terminal and medium based on a large model, which improves the accuracy of video segment retrieval by fusing natural language feature description and multimodal feature vectors and accurately extracting target video frames.

[0005] In a first aspect, the technical solution of the present invention provides a video segment retrieval method based on a large model, comprising the following steps: Acquire a number of original video data, and divide each original video data into a plurality of video segments; Input each video segment into a pre-trained multimodal large model, extract feature descriptions of each video segment and target video frames, wherein the feature descriptions include natural language feature descriptions and multimodal feature vectors; Combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Storing the video segment feature representation in a database; The search information input by the user is obtained, the search information is compared with the feature representations of each video segment in the database, and the video data matching the search information is screened out and fed back to the user.

[0006] In an optional implementation, each video segment is input into a pre-trained multimodal large model, and a feature description of each video segment and a target video frame are extracted, specifically including: Extract natural language features of video segments through optical character recognition technology, speech recognition technology and natural language technology; The video features and audio features of the video segment are extracted through the neural network model and audio classification technology, and the video features and audio features are fused to obtain a multimodal feature vector; The video segments are processed through a neural network model to extract the target video frames.

[0007] In an optional implementation, the natural language features of the video segment are extracted by using optical character recognition technology, speech recognition technology and natural language technology, specifically including: Using optical character recognition technology, extract subtitles from the video segment and record them as first text; Extract the speech signal from the video segment using speech recognition technology and pre-process the speech signal; Extracting spectral feature information related to speech recognition from the preprocessed speech signal; The spectral feature information is processed according to the pre-trained neural network model to obtain the acoustic characteristics; Decode the acoustic features to obtain a text sequence, which is recorded as the second text; Preprocessing the first text and the second text; The natural language processing model performs semantic understanding on the preprocessed first text and second text, and extracts key information and contextual relationships in the text; Based on the results of semantic understanding, the corresponding natural language feature description is generated.

[0008] In an optional implementation, video features and audio features are extracted from the video segment by using a neural network model and audio classification technology, and the video features and audio features are fused to obtain a multimodal feature vector, specifically including: Processing the video segment according to the neural network model to obtain video features of the video segment; Extracting features of the audio track of the video segment according to the audio classification technology to obtain the audio features of the video segment; The visual features and audio features are concatenated and fused to obtain a multimodal feature vector.

[0009] In an optional implementation, after each original video data is divided into a plurality of video segments, the following steps are further included: Add metadata to individual video segments, including timestamps, locations, and scene descriptions.

[0010] In an optional implementation, after combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation, the following steps are further included: Each video segment representation is mapped and associated with metadata of the corresponding video segment.

[0011] In an optional implementation, the search information is compared with the feature representations of each video segment in the database, and video data matching the search information is screened out and fed back to the user, specifically including: The cosine similarity is used to calculate the distance between the retrieved information and the representation of each video segment in the database; Matching similarity scores based on the calculated distances; Sort the feature representations of each video segment in descending order of similarity scores; The video segments with the highest similarity scores in the screening and sorting are fed back to the user as the final video data.

[0012] In a second aspect, the technical solution of the present invention provides a video segment retrieval system based on a large model, comprising: Video segmentation module: used to obtain a number of original video data and segment each original video data into multiple video segments; Video feature extraction module: used to input each video segment into the pre-trained multimodal large model, extract the feature description of each video segment and the target video frame, where the feature description includes natural language feature description and multimodal feature vector; Video feature representation module: used to combine the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Video feature storage module: used to store the video segment feature representation in the database; User retrieval module: used to obtain the retrieval information input by the user, compare the retrieval information with the feature representations of each video segment in the database, and filter out the video data that matches the retrieval information and feed it back to the user.

[0013] In a third aspect, the technical solution of the present invention provides a terminal, including: A memory for storing a video segment retrieval program based on a large model; A processor is used to implement the steps of any of the above-mentioned large-model-based video segment retrieval methods when executing the large-model-based video segment retrieval program.

[0014] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which a large model-based video segment retrieval program is stored. When the large model-based video segment retrieval program is executed by a processor, the steps of the large model-based video segment retrieval method as described in any one of the above items are implemented.

[0015] The present invention provides a video segment retrieval method, system, terminal and medium based on a large model, which have the following beneficial effects compared with the prior art: first, a number of original video data are segmented into multiple video segments, and then the natural language feature description, multimodal feature vector and target video frame of each video segment are extracted based on the multimodal large model, and the natural language feature description, multimodal feature vector and target video frame are combined to obtain the video segment feature representation, and after the user inputs the search information, the search information is compared with the feature representation of each video segment to screen out the matching video segment. By fusing the natural language feature description and the multimodal feature vector, and accurately extracting the target video frame, the present invention can more accurately capture the semantic and visual features of the video content, thereby improving the relevance of the search results and improving the accuracy and efficiency of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flowchart of a video segment retrieval method based on a large model provided by an embodiment of the present invention.

[0017] Figure 2 The present invention provides a schematic block diagram of a video segment retrieval system based on a large model.

[0018] Figure 3 It is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0021] Figure 1 : is a flow chart of a video segment retrieval method based on a large model provided by an embodiment of the present invention. Figure 1 The execution subject may be a video segment retrieval system based on a large model. The video segment retrieval method based on a large model provided in the embodiment of the present invention is executed by a computer device, and accordingly, the video segment retrieval system based on a large model runs in the computer device. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0022] like Figure 1 As shown, the method includes the following steps.

[0023] S1, obtaining a number of original video data, and dividing each original video data into a plurality of video segments.

[0024] The original video data is the basis for user retrieval, and users filter out target videos from the original video data. Video data containing various scenes and character actions are collected as original video data, including normal and abnormal behaviors (such as falling, fighting, running away, etc.).

[0025] Each long video is split into multiple shorter video segments, exemplarily each segment contains a few seconds of content, for ease of processing and analysis. To facilitate subsequent analysis and retrieval optimization, metadata such as timestamp, location, and scene description are added to each video segment.

[0026] S2, input each video segment into a pre-trained multimodal large model, extract the feature description of each video segment and the target video frame, wherein the feature description includes a natural language feature description and a multimodal feature vector.

[0027] Select a pipeline that can process visual and textual data to identify key frames in the video, extract visual features (such as people, actions, and environments), and understand the semantic information of the video content. The specific steps include the following.

[0028] S2.1, extract natural language features of the video segment through optical character recognition technology, speech recognition technology and natural language technology.

[0029] This embodiment uses the text generation capability of the large model to generate a natural language summary for each video segment, describing the main events and character actions in the video.

[0030] S2.1.1, using optical character recognition technology, extract subtitles from the video segment and record them as the first text.

[0031] This embodiment first extracts text from the subtitles in the video segment, which can be achieved using optical character recognition technology.

[0032] S2.1.2, extract the speech signal in the video segment using speech recognition technology, and pre-process the speech signal.

[0033] S2.1.3, extracting spectral feature information related to speech recognition from the preprocessed speech signal.

[0034] S2.1.4, processing the spectral feature information according to the pre-trained neural network model to obtain the acoustic characteristics.

[0035] S2.1.5, decode the acoustic features to obtain a text sequence, which is recorded as the second text.

[0036] Steps S2.1.2 to S2.1.5 are used to extract voice information from the video segment, extract the voice signal from the video, and perform preprocessing, such as noise reduction, echo removal, endpoint detection, etc., to improve the accuracy of voice recognition. The most relevant information for voice recognition is extracted from the preprocessed voice signal. In this embodiment, the spectral features are selected as the relevant information, and the extracted features are modeled using a deep neural network DNN to predict the acoustic characteristics of the voice signal, and then the voice signal is decoded to obtain the recognized text sequence.

[0037] S2.1.6, preprocess the first text and the second text.

[0038] Preprocess the recognized text sequence, such as removing punctuation, word segmentation, and part-of-speech tagging.

[0039] S2.1.7, the natural language processing model performs semantic understanding on the preprocessed first text and the second text, and extracts key information and contextual relationships in the text.

[0040] S2.1.8, based on the results of semantic understanding, generate corresponding natural language feature descriptions.

[0041] Use the natural language processing model to perform semantic understanding on the preprocessed text, extract key information and contextual relationships in the text, and then generate corresponding natural language feature descriptions based on the results of semantic understanding, including text summarization, keyword extraction, and sentiment analysis in various forms.

[0042] S2.2, extract video features and audio features from the video segment through a neural network model and audio classification technology, and fuse the video features and audio features to obtain a multimodal feature vector.

[0043] S2.2.1, processing the video segment according to the neural network model to obtain video features of the video segment.

[0044] S2.2.2, extract features of the audio track of the video segment according to the audio classification technology to obtain audio features of the video segment.

[0045] S2.2.3, concatenate and fuse the visual features and audio features to obtain a multimodal feature vector.

[0046] The neural network model of this embodiment processes the video segment to obtain the video features of the video segment, which may include visual details and dynamic information in the video. At the same time, the audio features of the video are extracted, and the visual features and audio features are spliced ​​and fused to obtain a multimodal feature vector. S2.3, the video segment is processed through a neural network model to extract the target video frame.

[0047] In this embodiment, the prediction ability of the large model is used to identify and extract key frames containing abnormal actions. These frames usually have significant visual features (such as abnormal character postures, sudden changes in motion speed) or semantic importance (such as the occurrence of sudden events). The frame numbers of these frames are recorded as part of the video clip features.

[0048] S3, combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation.

[0049] Combining natural language feature descriptions, multimodal feature vectors, and target video frames and their frame numbers to construct a complete feature representation of the video clip can be achieved by concatenating feature vectors.

[0050] In this embodiment, each video segment representation is mapped and associated with metadata of the corresponding video segment to provide additional context information.

[0051] S4, storing the video segment feature representation in the database.

[0052] S5, obtaining the search information input by the user, comparing the search information with the feature representations of each video segment in the database, screening out the video data matching the search information and feeding it back to the user.

[0053] S5.1, use cosine similarity to calculate the distance between the retrieved information and the representation of each video segment in the database.

[0054] S5.2, matching similarity scores according to the calculated distances.

[0055] S5.3, sort the feature representations of each video segment in descending order of similarity score.

[0056] S5.4, several video segments with the highest similarity scores in the screening and sorting are fed back to the user as the final video data.

[0057] In this embodiment, the user inputs the search information, which can be a text description (such as "find a falling clip in the video"), an image (such as a picture depicting a falling scene) or a short video clip (such as a clip containing a falling action). Similar feature extraction and representation processing is performed on the input search information to generate a feature vector or description comparable to the feature representation of the video clip. The distance between the search information and the representation of each video segment in the database is calculated using cosine similarity, and the similarity score is matched according to the calculated distance. This can be achieved by pre-configuring the similarity score corresponding to each distance range, and then sorting the video clips according to the similarity score, and returning the video clips with the highest scores as the search results.

[0058] The large model-based video segment retrieval method provided in this embodiment can more accurately capture the semantic and visual features of the video content by fusing natural language feature descriptions and multimodal feature vectors and accurately extracting target video frames, thereby improving the relevance of retrieval results and improving the accuracy and efficiency of video retrieval.

[0059] An embodiment of a large model-based video segment retrieval method is described in detail above. Based on the large model-based video segment retrieval method described in the above embodiment, an embodiment of the present invention also provides a large model-based video segment retrieval system corresponding to the method.

[0060] Figure 2 is a schematic block diagram of the structure of a video segment retrieval system based on a large model provided by an embodiment of the present invention. In this embodiment, the video segment retrieval system 200 based on a large model can be divided into multiple functional modules according to the functions performed by the system, such as Figure 2 The functional modules may include: a video segmentation module 210, a video feature extraction module 220, a video feature representation module 230, a video feature storage module 240, and a user retrieval module 250. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, which are stored in a memory.

[0061] The video segmentation module 210 is used to obtain a number of original video data and segment each of the original video data into a plurality of video segments.

[0062] Video feature extraction module 220: used to input each video segment into a pre-trained multimodal large model, extract feature descriptions of each video segment and target video frames, wherein the feature descriptions include natural language feature descriptions and multimodal feature vectors.

[0063] The video feature representation module 230 is used to combine the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation.

[0064] Video feature storage module 240: used to store the video segment feature representation in a database.

[0065] User retrieval module 250: used to obtain retrieval information input by the user, compare the retrieval information with the feature representations of each video segment in the database, filter out video data matching the retrieval information and feed it back to the user.

[0066] The big model-based video segment retrieval system of this embodiment is used to implement the aforementioned big model-based video segment retrieval method. Therefore, the specific implementation method of the system can be seen in the embodiment part of the big model-based video segment retrieval method in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding parts of the embodiments, which will not be elaborated here.

[0067] In addition, since the large model-based video segment retrieval system of this embodiment is used to implement the aforementioned large model-based video segment retrieval method, its function corresponds to that of the above method and will not be described in detail here.

[0068] Figure 3 A schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention includes: a processor 310, a memory 320 and a communication unit 330. The processor 310 is used to implement the following steps when implementing the video segment retrieval program based on the large model stored in the memory 320: Acquire a number of original video data, and divide each original video data into a plurality of video segments; Input each video segment into a pre-trained multimodal large model, extract feature descriptions of each video segment and target video frames, wherein the feature descriptions include natural language feature descriptions and multimodal feature vectors; Combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Storing the video segment feature representation in a database; The search information input by the user is obtained, the search information is compared with the feature representations of each video segment in the database, and the video data matching the search information is screened out and fed back to the user. The terminal 300 includes a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. It can be understood by those skilled in the art that the structure of the server shown in the figure does not constitute a limitation of the present invention, and it can be a bus structure or a star structure, and can also include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0069] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.

[0070] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0071] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0072] The present invention also provides a computer storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0073] The computer storage medium stores a large model-based video segment retrieval program, and when the large model-based video segment retrieval program is executed by a processor, the following steps are implemented: Acquire a number of original video data, and divide each original video data into a plurality of video segments; Input each video segment into a pre-trained multimodal large model, extract feature descriptions of each video segment and target video frames, wherein the feature descriptions include natural language feature descriptions and multimodal feature vectors; Combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Storing the video segment feature representation in a database; The search information input by the user is obtained, the search information is compared with the feature representations of each video segment in the database, and the video data matching the search information is screened out and fed back to the user.

[0074] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.

[0075] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0076] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0077] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0078] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A video segment retrieval method based on a large model, characterized in that: The following steps are involved: Acquire a number of original video data, and divide each original video data into a plurality of video segments; Input each video segment into a pre-trained multimodal large model, extract feature descriptions of each video segment and target video frames, wherein the feature descriptions include natural language feature descriptions and multimodal feature vectors; Combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Storing the video segment feature representation in a database; The search information input by the user is obtained, the search information is compared with the feature representations of each video segment in the database, and the video data matching the search information is screened out and fed back to the user.

2. The video segment retrieval method based on a large model according to claim 1, characterized in that: Each video segment is input into the pre-trained multimodal large model to extract the feature description of each video segment and the target video frame, including: Extract natural language features of video segments through optical character recognition technology, speech recognition technology and natural language technology; The video features and audio features of the video segment are extracted through the neural network model and audio classification technology, and the video features and audio features are fused to obtain a multimodal feature vector; The video segments are processed through a neural network model to extract the target video frames.

3. The video segment retrieval method based on a large model according to claim 2, characterized in that: The natural language features of the video segment are extracted through optical character recognition technology, speech recognition technology and natural language technology, including: Using optical character recognition technology, extract subtitles from the video segment and record them as the first text; Extract the speech signal from the video segment using speech recognition technology and pre-process the speech signal; Extracting spectral feature information related to speech recognition from the preprocessed speech signal; The spectral feature information is processed according to the pre-trained neural network model to obtain the acoustic characteristics; Decode the acoustic features to obtain a text sequence, which is recorded as the second text; Preprocessing the first text and the second text; The natural language processing model performs semantic understanding on the preprocessed first text and second text, and extracts key information and contextual relationships in the text; Based on the results of semantic understanding, the corresponding natural language feature description is generated.

4. The video segment retrieval method based on a large model according to claim 2, characterized in that: The video features and audio features of the video segment are extracted through the neural network model and audio classification technology, and the video features and audio features are fused to obtain a multimodal feature vector, including: Processing the video segment according to the neural network model to obtain video features of the video segment; Extracting features of the audio track of the video segment according to the audio classification technology to obtain the audio features of the video segment; The visual features and audio features are concatenated and fused to obtain a multimodal feature vector.

5. The video segment retrieval method based on a large model according to any one of claims 1 to 4, characterized in that: After each original video data is divided into a plurality of video segments, the following steps are also included: Add metadata to individual video segments, including timestamps, locations, and scene descriptions.

6. The video segment retrieval method based on a large model according to claim 5, characterized in that: After combining the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation, the following steps are also included: Each video segment representation is mapped and associated with metadata of the corresponding video segment.

7. The video segment retrieval method based on a large model according to claim 6, characterized in that: Compare the search information with the feature representations of each video segment in the database, filter out the video data that matches the search information and feed it back to the user, including: The cosine similarity is used to calculate the distance between the retrieved information and the representation of each video segment in the database; Matching similarity scores based on the calculated distances; Sort the feature representations of each video segment in descending order of similarity scores; The video segments with the highest similarity scores in the screening and sorting are fed back to the user as the final video data.

8. A video segment retrieval system based on a large model, characterized in that: include, Video segmentation module: used to obtain a number of original video data and segment each original video data into multiple video segments; Video feature extraction module: used to input each video segment into the pre-trained multimodal large model, extract the feature description of each video segment and the target video frame, where the feature description includes natural language feature description and multimodal feature vector; Video feature representation module: used to combine the natural language feature description of each video segment, the multimodal feature vector and the target video frame to obtain the video segment feature representation; Video feature storage module: used to store the video segment feature representation in the database; User retrieval module: used to obtain the retrieval information input by the user, compare the retrieval information with the feature representations of each video segment in the database, and filter out the video data that matches the retrieval information and feed it back to the user.

9. A terminal, characterized in that: include: A memory for storing a video segment retrieval program based on a large model; A processor, used to implement the steps of the video segment retrieval method based on a large model as described in any one of claims 1 to 7 when executing the video segment retrieval program based on a large model.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a large model-based video segment retrieval program, which, when executed by a processor, implements the steps of the large model-based video segment retrieval method as described in any one of claims 1 to 7.

Images