Movie and television play video highlight fragment extraction method and device and computer readable medium

Abstract

The embodiment of the invention provides a movie and television play video highlight segment extraction method and device and a computer readable medium, and the scheme writes an extracted video frame picture into a database in a feature vector form, and carries out matching retrieval by using the feature vector of prompt information, so that the highlight segment extraction efficiency can be improved through a multi-mode retrieval mode. According to the method, the video content is extracted in a large scale, and the target video frame picture suitable for the highlight clip of the specific type is found from all the video frame pictures to be used for synthesizing the final highlight clip, so that the video content which does not conform to the requirement can be avoided, the highlight clip extraction accuracy is improved, and the user experience is better.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method, device and computer-readable medium for extracting highlight segments from a film or TV series video. Background Art

[0002] With the development of the Internet, watching TV series and movies through video playback platforms has become a common form of entertainment. When choosing TV series and movies, users often want to know the wonderful content of the TV series and movies in advance so that they can choose the TV series and movies that suit their interests. For a large number of mid-level TV series and movies, they also need to play the highlight clips to drive the playback and subscription of the main film.

[0003] To this end, most video playback platforms provide a highlight clip preview function, which extracts the wonderful content in the film and television video in advance and provides it to users as a highlight clip for preview, so that users can understand the wonderful content in the film and television video in advance. However, the existing technology for extracting highlight clips in film and television videos mainly uses trained models to achieve this, so its accuracy is also completely dependent on the accuracy of the model. When the model is not suitable for the current extraction scenario, it is easy to lead to insufficient accuracy in the extraction of highlight clips, and may extract highlight clips that are significantly different from the content of the film and television video, resulting in a poor user experience. Summary of the invention

[0004] One purpose of the present application is to provide a method, device and computer-readable medium for extracting highlight segments from film and television drama videos, so as to solve the problem that the solutions in the prior art are insufficient in accuracy and easily lead to poor user experience.

[0005] To achieve the above-mentioned purpose, the present application provides a method for extracting highlight segments from a film or TV series video, the method comprising:

[0006] Perform frame extraction processing on the film and television drama video to obtain a video frame image of the film and television drama video, perform feature extraction processing on the video frame image using a multimodal model, obtain a first feature vector of the video frame image, and write it into a database;

[0007] Determining prompt information for retrieval according to the type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information;

[0008] Retrieving a matching target video frame picture from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector;

[0009] The target video frame images are merged to generate highlight segments.

[0010] Further, performing frame extraction processing on the film and television drama video to obtain video frame images of the film and television drama video includes:

[0011] According to a preset sampling period, frame extraction processing is performed on the film and television drama video, and a video frame image of the film and television drama video is extracted in each preset sampling period.

[0012] Furthermore, after performing frame extraction processing on the film and television drama video and obtaining the video frame pictures of the film and television drama video, it also includes:

[0013] The video frame image is stored in the object storage, and the program code of the film and television drama video is used as the folder name when storing, and the time information of the video frame image in the film and television drama video is used as the file name when storing.

[0014] Furthermore, the database is a vector database;

[0015] Performing feature extraction processing on the video frame picture by using a multimodal model, obtaining a first feature vector about the video frame picture, and writing it into a database, including:

[0016] Using an image encoder of a multimodal model, performing feature extraction processing on the video frame picture to obtain a first feature vector of the video frame picture;

[0017] The file address of the video frame image in the object storage and the first feature vector are written into the vector database, wherein the file address at least includes a folder name and a file name.

[0018] Furthermore, the prompt information includes prompt words;

[0019] Determining prompt information for retrieval according to type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information, including:

[0020] Determine the prompt words used for retrieval according to the type information of the highlight segment;

[0021] The text encoder of the multimodal model is used to perform feature extraction processing on the prompt word to obtain a second feature vector related to the prompt word.

[0022] Furthermore, the prompt information also includes a prompt picture;

[0023] Determining prompt information for retrieval according to type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information, including:

[0024] According to the type information of the highlight segment, the prompt words and prompt pictures used for retrieval are determined;

[0025] The text encoder of the multimodal model is used to perform feature extraction processing on the prompt word, and the image encoder of the multimodal model is used to perform feature extraction processing on the prompt picture to obtain a second feature vector about the prompt information.

[0026] Further, according to the similarity between the second feature vector and the first feature vector, retrieving a matching target video frame picture from the video frame pictures in the database includes:

[0027] Retrieving matching candidate video frame images from the video frame images in the database according to the similarity between the second feature vector and the first feature vector;

[0028] The candidate video frame pictures are sorted according to the similarities, and according to the sorting result, a target video frame picture with a higher similarity is determined from the candidate video frame pictures.

[0029] Furthermore, the target video frame pictures are merged to generate highlight segments, including:

[0030] The target video frame images are merged according to the time information of the target video frame images, and the target video frame images with similar time information are merged into a highlight segment.

[0031] Some embodiments of the present application also provide a device for extracting highlight clips from a film or TV series video, wherein the device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned method for extracting highlight clips from a film or TV series video.

[0032] Other embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, and the computer program instructions can be executed by a processor to implement the method for extracting highlight segments from a film or TV series video.

[0033] Compared with the prior art, in a solution for extracting highlight clips from film and television drama videos provided in an embodiment of the present application, frame extraction processing is first performed on the film and television drama video to obtain video frame images of the film and television drama video, and feature extraction processing is performed on the video frame images using a multimodal model to obtain a first feature vector of the video frame images and write them into a database. Then, based on the type information of the highlight clip, prompt information for retrieval is determined, and feature extraction processing is performed on the prompt information using the multimodal model to obtain a second feature vector of the prompt information. Then, based on the similarity between the second feature vector and the first feature vector, a matching target video frame image is retrieved from the video frame images in the database, and the target video frame images are merged to generate a highlight clip. In this solution, the extracted video frame images are written into the database in the form of feature vectors, and the feature vectors of the prompt information are used for matching and retrieval. In this way, large-scale extraction of video content can be completed through multimodal retrieval, and target video frame images suitable for specific types of highlight segments can be found from all video frame images to synthesize the final highlight segments. This can avoid the appearance of video content that does not meet the requirements, improve the accuracy of highlight segment extraction, and provide a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 A schematic diagram of a processing flow of a method for extracting highlight segments from a film or TV series video provided in an embodiment of the present application;

[0036] Figure 2 A processing flow chart for extracting highlight fragments using the solution provided in an embodiment of the present application;

[0037] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0038] The present application is described in further detail below in conjunction with the accompanying drawings.

[0039] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0040] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0041] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, program devices or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0042] An embodiment of the present application provides a method for extracting highlight clips from film and television drama videos. The method writes the extracted video frame images into a database in the form of feature vectors, and uses the feature vectors of prompt information for matching and retrieval. This allows large-scale extraction of video content to be completed through multimodal retrieval, and target video frame images suitable for specific types of highlight clips are found from all video frame images to synthesize the final highlight clips. This avoids the appearance of video content that does not meet the requirements, improves the accuracy of highlight clip extraction, and provides a better user experience.

[0043] In actual scenarios, the execution subject of the method can be a user device, a network device, or a device formed by integrating a user device and a network device through a network, or it can also be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, and tablet computers; the network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0044] Figure 1 The processing flow of a method for extracting highlight segments from a film or TV series video provided in an embodiment of the present application is shown. The method is used to extract highlight segments from a film or TV series video, and at least includes the following processing steps:

[0045] Step S101, perform frame extraction processing on the film and television drama video, obtain video frame pictures about the film and television drama video, use a multimodal model to perform feature extraction processing on the video frame pictures, obtain the first feature vector about the video frame pictures, and write it into the database.

[0046] Among them, when performing frame extraction processing on a film and television drama video and obtaining a video frame image about the film and television drama video, the frame extraction processing can be performed on the film and television drama video according to a preset sampling period, and a video frame image about the film and television drama video can be extracted in each preset sampling period. In actual scenarios, the preset sampling period can be set as needed. For example, in this embodiment, the preset sampling period can be set to 1 second, that is, a video frame image is extracted from the film and television drama video every 1 second. Therefore, taking a film and television drama video with a length of 1 hour as an example, 3600 video frame images can be extracted from it.

[0047] In some embodiments of the present application, after performing frame extraction processing on a film or TV series video and obtaining video frame images of the film or TV series video, the video frame images can also be stored in an object storage. For example, in an actual scenario, object storage services provided by various cloud service providers can be used, such as OSS (Object Storage Service), COS (Cloud Object Storage) or other public cloud object storage services. After the video frame images of the film or TV series video are stored in the object storage, these video frame images can have an HTTP (Hyper Text Transfer Protocol) access link address so that they can be quickly accessed through the access link address in subsequent processing.

[0048] When storing the video frame image in the object storage, the program code of the film and television video can be used as the folder name when storing, and the time information of the video frame image in the film and television video can be used as the file name when storing. Among them, the program code is the identification information of the film and television video, which is used to distinguish different film and television videos. The time information corresponds to the specific time of the video frame image in the film and television video to which it belongs. Taking the above-mentioned scenario of frame extraction processing according to the preset sampling period of 1 second as an example, the time information of the video frame image can be Tx, where x=1, 2, 3, 4, ..., 3600, respectively representing the time information of the video frame image extracted from every 1 second in the film and television video.

[0049] Take a TV series video video1 as an example. If its program code is code123, the time information of the video frame picture at the first second is T1. Therefore, it can be determined that the folder name when the video frame picture is stored is code123, the file name is T1, and the corresponding file address can be determined as \code123\T1. Furthermore, if the object storage access entry address storing the video frame picture is adr1, the video frame picture access link address can be determined to be adr1\code123\T1. Since the folder name and file name have marked the relationship between the video frame picture and the TV series video, after storing the video frame picture in the object storage in this way, there is no need to establish an additional relational database to manage the TV series video and the address of each video frame picture, thereby saving the resource overhead of storing related data and simplifying the difficulty of data management.

[0050] The database used in the scheme of this embodiment can adopt a vector database, such as the milvus vector database, etc., and can also be built-in with a variety of ANN (Approximate Nearest Neighbor, approximate nearest neighbor search algorithm) algorithms such as Faiss, HNSW, Annoy, etc. When performing feature extraction processing on the video frame picture, the image encoder (image encoder) of the multimodal model can be used to extract the feature vector. For example, the multimodal model used in this embodiment can adopt the Chinese CLIP (Contrastive Language-Image Pre-training, contrast language-image pre-training) model. At the same time, the specific content about the video frame picture written in the vector database may include the file address of the video frame picture in the object storage and the first feature vector, wherein the file address includes at least the folder name and the file name, so as to quickly locate the specific video frame picture from the storage object and achieve fast access. Therefore, when the scheme of this embodiment uses the multimodal model to perform feature extraction processing on the video frame image, obtains the first feature vector about the video frame image, and writes it into the database, it can use the image encoder of the multimodal model to perform feature extraction processing on the video frame image, obtain the first feature vector about the video frame image, and then write the file address of the video frame image in the object storage and the first feature vector into the vector database.

[0051] Step S102, according to the type information of the highlight segment, determine the prompt information for retrieval, and use the multimodal model to perform feature extraction processing on the prompt information to obtain a second feature vector about the prompt information. The type information refers to the specific type of the film and television drama video for which the highlight segment needs to be generated, such as action movies, romance movies, science fiction movies, etc. For each different type of information, different prompt information for retrieval can be provided, so as to retrieve different video frame pictures to generate highlight segments that meet the actual specific type.

[0052] In some embodiments of the present application, the prompt information may include at least a prompt word. Therefore, when determining the prompt word, the prompt word used for retrieval may be determined according to the type information of the highlight segment. Specifically, in this embodiment, different prompt words may be pre-set for different type information. For example, if the type information is a romance film, prompt words such as hug, kiss, dance, send flowers, go shopping, watch a movie, propose, quarrel, etc. may be provided. For other type information, other prompt words corresponding to the type information may be provided to facilitate retrieval of a suitable target video frame image.

[0053] After determining the prompt word, the text encoder of the multimodal model can be used to perform feature extraction processing on the prompt word to obtain a second feature vector about the prompt word. Among them, the multimodal model used in this step can also use the Chinese CLIP model. Since the CLIP model learns to encode images and texts into unified feature vectors through comparative learning during the training process, this enables the CLIP model to understand the correspondence between images and texts in natural language and image vision. Thus, the first feature vector and the second feature vector obtained by the image encoder and the text encoder of the same CLIP model can represent the semantic association between the prompt word and the video frame picture, thereby accurately realizing the retrieval of the video frame picture, thereby obtaining a suitable target video frame picture.

[0054] In some other embodiments of the present application, the prompt information may also include a prompt picture. Therefore, when determining the prompt information for retrieval based on the type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information, the prompt words and prompt pictures for retrieval may be determined based on the type information of the highlight segment, and then the text encoder of the multimodal model may be used to perform feature extraction processing on the prompt words, and the image encoder of the multimodal model may be used to perform feature extraction processing on the prompt picture to obtain a second feature vector about the prompt information.

[0055] In actual scenarios, prompt information in the form of images can be used to achieve retrieval for specific purposes, such as finding video frame images with specific screen content to batch produce video covers of highlight clips, or using the headshots of the leading actors in film and television drama videos as prompt images, combined with relevant prompt words, so that they can be used to retrieve video frame images containing these leading actors, thereby generating highlight clips about the leading actors. Therefore, by using prompt information in the form of text combined with images, the required target video frame images can be retrieved more flexibly, thereby generating highlight clips that better meet user needs, improving the flexibility of the solution.

[0056] Step S103: Retrieve a matching target video frame picture from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector.

[0057] Since the second feature vector represents the characteristics of the prompt information, and the first feature vector represents the characteristics of the video frame image extracted from the film and television drama video, the similarity between the two can reflect the degree of matching between the prompt information and the video frame image. Therefore, through this step, the target video frame image that matches the prompt information can be accurately retrieved from the video frame images in the database.

[0058] In order to further improve the accuracy of the search, the search results returned initially will first enter the Rerank stage of post-search processing, and the final search results will be determined based on the processing results of the Rerank stage. Rerank is to re-rank the multiple search results so that the ranking results are more closely matched to the search conditions, so that more relevant and accurate search results are ranked higher, thereby improving the output quality of the search results.

[0059] Specifically, in the scheme of this embodiment, when retrieving a matching target video frame picture from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector, the matching candidate video frame pictures can be first retrieved from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector, and then the candidate video frame pictures are sorted according to the similarity, and according to the sorting result, the target video frame picture with the highest similarity is determined from the candidate video frame pictures. Among them, the target video frame picture with the highest similarity can refer to the target video frame picture whose similarity is in the top N in the sorting result, or it can also be the target video frame picture whose similarity is within the previous preset ratio, thereby determining the candidate video frame picture with higher similarity as the final target video frame picture, so that the highlight clip finally generated can more accurately express the wonderful content of the film and television drama video.

[0060] Step S104: merge the target video frame images to generate highlight segments.

[0061] In some embodiments of the present application, when generating highlight segments, the target video frame images can be merged according to the time information of the target video frame images, and the target video frame images with similar time information can be merged into one highlight segment. In actual scenarios, target video frame images with a time difference within a preset duration can be judged as having similar time information. For example, the preset duration can be set to 10 seconds. If it is determined that the time difference between two target video frame images is within 10 seconds based on the time information of the two target video frame images, the two target video frame images can be merged. When actually merging highlight segments, the time information corresponding to all target video frames can be traversed and compared and judged, so as to merge multiple target video frame images into one highlight segment and realize the generation of highlight segments.

[0062] Since the extracted video frame images are written into the database in the form of feature vectors in this solution, and the feature vectors of the prompt information are used for matching and retrieval, a large number of video contents can be extracted through multimodal retrieval, and target video frame images suitable for specific types of highlight segments can be found from all video frame images to synthesize the final highlight segments. This can avoid the appearance of video content that does not meet the requirements, improve the accuracy of highlight segment extraction, and provide a better user experience.

[0063] Figure 2 The processing flow for extracting highlight fragments using the solution provided in the embodiment of the present application is shown, and may include the following processing steps:

[0064] Step S201, write to object storage. Extract video frames from TV dramas by second, and then store these video frames in object storage. The goal of this step is to give these video frames an HTTP access link address so that they can be quickly accessed and feature vectors of the video frames can be extracted. At the same time, when storing, the program code is used as the folder name, and the time information of the video frame is used as the file name, so there is no need to establish an additional relational database to manage the correspondence between the video frame and the file address. Among them, object storage can use OSS, COS or other public cloud object storage services.

[0065] Step S202, vector extraction and writing. Call the multimodal model to perform feature extraction processing on the video frame image, obtain the first feature vector of the video frame image, and then write the file address and the first feature vector of the video frame image into the vector database through the vector tool. Among them, the vector database can use the milvus vector database, etc., and can also be built-in with various ANN algorithms such as Faiss, HNSW, Annoy, etc. The multimodal model used when extracting features from the video frame image can be the Chinese CLIP model.

[0066] Step S203, database query. According to the type information of the highlight clips to be produced, the prompt information such as prompt words and prompt pictures required for retrieval is determined. For example, for a romance film, prompt words such as hug, kiss, dance, send flowers, go shopping, watch movies, propose, quarrel, etc. can be provided. Then, the multimodal model is used to perform feature extraction processing on the prompt information to obtain a second feature vector about the prompt information. The second feature vector is used to query the vector database, and the matching query results are retrieved from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector. Among them, the multimodal model used in this step can adopt the same Chinese CLIP model.

[0067] Step S204, merge highlight segments. According to the returned query results, enter the Rerank stage of post-retrieval processing. Reorder the multiple candidate video frame images retrieved so that the ranking results are more in line with the retrieval conditions, so that more relevant and accurate video frame images are ranked higher, thereby improving the quality of the retrieval results. Based on the query results after Rerank, select the video frame images with a higher degree of match with the prompt information as the target video frame images for video frame merging, thereby producing the corresponding highlight segments. Specifically, when merging, video frame images with similar time information can be merged into the same highlight segment. For example, the query results include n target video frame images, and the time information corresponding to these target video frame images is different, but for those with similar time information, such as within 10 seconds, they can be considered to be the same highlight scene, so they can be merged into the same highlight segment.

[0068] In addition, an embodiment of the present application also provides a device for extracting highlight clips from film and television drama videos, which includes a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned method for extracting highlight clips from film and television drama videos.

[0069] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the processing unit, the above functions defined in the method of the present application are executed.

[0070] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0071] In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0072] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0073] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] As another aspect, the present application further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the above multiple embodiments of the present application.

[0075] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0076] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for extracting highlight segments from film and television drama videos. It is characterized in that The method comprises: Perform frame extraction processing on the film and television drama video to obtain a video frame image of the film and television drama video, perform feature extraction processing on the video frame image using a multimodal model, obtain a first feature vector of the video frame image, and write it into a database; Determining prompt information for retrieval according to the type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information; Retrieving a matching target video frame picture from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector; The target video frame images are merged to generate highlight segments.

2. The method according to claim 1, It is characterized in that Performing frame extraction processing on a film or TV series video to obtain a video frame image of the film or TV series video includes: According to a preset sampling period, frame extraction processing is performed on the film and television drama video, and a video frame image of the film and television drama video is extracted in each preset sampling period.

3. The method according to claim 1, It is characterized in that After performing frame extraction processing on the film and television drama video and obtaining video frame images of the film and television drama video, the method further includes: The video frame image is stored in the object storage, and the program code of the film and television drama video is used as the folder name when storing, and the time information of the video frame image in the film and television drama video is used as the file name when storing.

4. The method according to claim 3, It is characterized in that The database is a vector database; Performing feature extraction processing on the video frame picture by using a multimodal model, obtaining a first feature vector about the video frame picture, and writing it into a database, including: Using an image encoder of a multimodal model, performing feature extraction processing on the video frame picture to obtain a first feature vector of the video frame picture; The file address of the video frame image in the object storage and the first feature vector are written into the vector database, wherein the file address at least includes a folder name and a file name.

5. The method according to claim 1, It is characterized in that The prompt information includes prompt words; Determining prompt information for retrieval according to type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information, including: Determine the prompt words used for retrieval according to the type information of the highlight segment; The text encoder of the multimodal model is used to perform feature extraction processing on the prompt word to obtain a second feature vector related to the prompt word.

6. The method according to claim 5, It is characterized in that The prompt information also includes a prompt picture; Determining prompt information for retrieval according to type information of the highlight segment, and performing feature extraction processing on the prompt information using the multimodal model to obtain a second feature vector about the prompt information, including: According to the type information of the highlight segment, the prompt words and prompt pictures used for retrieval are determined; The text encoder of the multimodal model is used to perform feature extraction processing on the prompt word, and the image encoder of the multimodal model is used to perform feature extraction processing on the prompt picture to obtain a second feature vector about the prompt information.

7. The method according to claim 1, It is characterized in that Retrieving a matching target video frame picture from the video frame pictures in the database according to the similarity between the second feature vector and the first feature vector, comprising: Retrieving matching candidate video frame images from the video frame images in the database according to the similarity between the second feature vector and the first feature vector; The candidate video frame pictures are sorted according to the similarities, and according to the sorting result, a target video frame picture with a higher similarity is determined from the candidate video frame pictures.

8. The method according to claim 1, It is characterized in that The target video frame images are merged to generate highlight segments, including: The target video frame images are merged according to the time information of the target video frame images, and the target video frame images with similar time information are merged into a highlight segment.

9. A device for extracting highlight clips from film and television drama videos. in, The device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 7.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images