Method, device, equipment and storage medium for aligning video and plot text

By dividing the plot text into sentences and matching features with the video clips, and filtering the time intervals with the dynamic time bending algorithm, the problem of low efficiency in existing video production is solved, efficient alignment of video and plot text is achieved, and video is improved Production efficiency.

CN114860994BActive Publication Date: 2024-11-22BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210425148.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-21
Publication Date
2024-11-22
Estimated Expiration
2042-04-21

AI Technical Summary

Technical Problem

The existing video production methods are complex, time-consuming and inefficient, resulting in low video production efficiency.

Method used

By obtaining plot text and video clips, using semantic segmentation model to divide plot text into sentences, and selecting matching target video clips from video clips according to feature similarity, and filtering time-matched video intervals with dynamic time bending algorithms, and storing Correlationships to achieve alignment of video and plot text.

Benefits of technology

It improves the alignment accuracy of video and plot text, realizes fully automatic video summary production, reduces manual operations, and improves video production efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114860994B_ABST
    Figure CN114860994B_ABST
Patent Text Reader

Abstract

The present application relates to a method, device, equipment and storage medium for aligning a video and a plot text; wherein the method comprises: obtaining the plot text and obtaining at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier; dividing the plot text into at least one sentence; wherein each sentence reflects at least one video plot; for each sentence, from the at least one video clip, selecting a video clip with the greatest similarity to the sentence in the same feature space as a target video clip that matches the sentence; for each sentence, from the target video clip, selecting a target video interval that matches the sentence in time, and storing the corresponding relationship between the sentence and the target video interval. The present application is used to solve the technical problem of low video production efficiency in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of multimedia intelligent processing technology, and in particular to a method, device, equipment and storage medium for aligning video and plot text. Background Art

[0002] Currently, video editing is generally done by experienced video editors, who operate relevant production software or tools to manually complete all text proofreading and video timeline alignment work line by line.

[0003] Therefore, the current traditional video production method is complicated to operate, wastes manpower, and consumes time, resulting in low video production efficiency. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for aligning a video and a plot text, so as to solve the technical problem of low efficiency in video production in the prior art.

[0005] In a first aspect, the present application provides a method for aligning a video and a plot text, comprising:

[0006] Acquire the plot text and acquire at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier;

[0007] Divide the plot text into at least one sentence; wherein each sentence reflects at least one segment of the video plot;

[0008] For each of the sentences, selecting, from the at least one video segment, a video segment having the greatest similarity to the sentence in the same feature space as a target video segment matching the sentence;

[0009] For each of the sentences, a target video interval that matches the sentence in time is screened out from the target video segment, and a corresponding relationship between the sentence and the target video interval is stored.

[0010] Optionally, obtaining at least one video clip includes:

[0011] Obtain the complete video corresponding to the plot text;

[0012] The complete video is divided into the at least one video segment according to a preset duration.

[0013] Optionally, dividing the plot text into at least one sentence includes:

[0014] Get the semantic segmentation model;

[0015] The plot text is semantically segmented to obtain the at least one sentence.

[0016] Optionally, selecting, from the at least one video segment, a video segment having the greatest similarity with the sentence in the same feature space as a target video segment matching the sentence includes:

[0017] extracting a first feature of the sentence;

[0018] For each of the video segments, extract a second feature of the video segment; calculate the similarity between the first feature and the second feature in the same feature space;

[0019] A maximum similarity is determined from the similarities; and a video segment corresponding to the maximum similarity is used as a target video segment matching the sentence.

[0020] Optionally, the step of selecting, from the target video segment, a target video interval that temporally matches the sentence includes:

[0021] Acquire at least one time interval corresponding to the target video segment to obtain a time interval set;

[0022] Using a dynamic time warping algorithm, extracting a target time interval that matches the sentence in time from the set of time intervals;

[0023] A video interval corresponding to the target time interval is extracted from the target video segment as the target video interval.

[0024] Optionally, after storing the correspondence between the sentence and the target video interval, the method further includes:

[0025] Based on the corresponding relationship, reading the sentence corresponding to the target video interval;

[0026] The sentence is displayed on a screen of the target video section.

[0027] In a second aspect, the present application provides a device for aligning a video and a plot text, comprising:

[0028] An acquisition module, used to acquire the plot text and at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier;

[0029] A sentence module, used to divide the plot text into at least one sentence; wherein each sentence reflects at least one segment of video content;

[0030] A retrieval module, configured to select, for each of the sentences, from the at least one video segment, a video segment having the greatest similarity with the sentence in the same feature space as a target video segment matching the sentence;

[0031] The screening module is used to screen out, for each of the sentences, a target video interval that matches the sentence in time from the target video segment, and store the corresponding relationship between the sentence and the target video interval.

[0032] Optionally, a retrieval module is used to extract the first feature of the sentence; for each of the video clips, extract the second feature of the video clip; calculate the similarity between the first feature and the second feature in the same feature space; determine the maximum similarity from each of the similarities; and use the video clip corresponding to the maximum similarity as the target video clip that matches the sentence.

[0033] In a third aspect, the present application provides an electronic device comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to implement the method for aligning the video and plot text described in the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for aligning a video and a plot text described in the first aspect is implemented.

[0035] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application takes each sentence in the plot text as the granularity, and for each sentence, selects the video clip with the greatest similarity to the sentence in the same feature space from at least one video clip as the target video clip that matches the sentence, and selects the target video interval that matches each sentence in time from the target video clip, and stores the correspondence between the sentence and the target video interval. Through this method, the video and the plot text can be effectively aligned, the alignment accuracy can be improved, the usability is strong, and it can be used to automatically produce a video summary of the plot explanation, thereby improving the efficiency of video production; according to the plot text, the target video clip and the target video interval can be retrieved from a large number of video clips, providing materials to video editors, thereby improving the efficiency of video production. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] Figure 1 A flowchart of a method for aligning a video and a plot text provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the structure of a device for aligning a video and a plot text provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0042] In order to solve the technical problem of low video production efficiency in the prior art, the embodiment of the present application provides a method for aligning a video and a plot text, such as Figure 1 As shown, the method for aligning a video and a plot text provided in an embodiment of the present application specifically includes the following steps:

[0043] Step 101, obtaining a plot text and obtaining at least one video clip; wherein the plot text and the at least one video clip correspond to the same multimedia content identifier;

[0044] In the embodiment of the present application, the plot text can be a plot summary or a more detailed plot description. For example, the plot description can be an ordered collection of scene environment, character psychology, narration, character language, etc.

[0045] The at least one video segment may be at least one video segment pre-acquired from the Internet, or may be at least one video segment divided from a complete video. The plot text and the video segment are for the same multimedia content identifier, for example, they are for the same TV series.

[0046] Among them, dividing a complete video into at least one video segment can be achieved through the following steps: first, obtaining the complete video corresponding to the plot text; and dividing the complete video into at least one video segment according to a preset duration.

[0047] For ease of understanding, the method of dividing a complete video into at least one video segment is described by way of example. For example, if the complete video is an episode, the duration of an episode is 45 minutes, and the preset duration is set to 5 minutes, then the complete video is evenly divided into 9 5-minute segments according to the preset duration.

[0048] In the specific implementation, the preset duration can be set according to actual needs, for example: 5 minutes, of course, it can also be other preset durations.

[0049] Step 102, dividing the plot text into at least one sentence; wherein each sentence reflects at least one segment of the video plot;

[0050] The plot text may be semantically segmented to obtain at least one sentence. In a specific embodiment, the semantic segmentation model may be used to implement the method. Specifically, the plot text is divided into at least one sentence, including: obtaining a semantic segmentation model; inputting the plot text into the semantic segmentation model to obtain at least one sentence.

[0051] The semantic segmentation model can be a Bert (Bidirectional Encoder Representation from Transformers) model; of course, other semantic segmentation models can also be used. The semantic segmentation model is used to segment the plot text into short sentences, i.e., clauses. Each short sentence can reflect a certain video interval. The short sentence is the smallest unit of alignment, i.e., each short sentence has a corresponding target video interval.

[0052] In the specific implementation, take the plot synopsis of a whole episode as an example. On the one hand, because the punctuation granularity of some plot synopsis is relatively coarse, if the full stop in the plot synopsis is used directly to divide the sentences, the sentence will be too long, resulting in a long video interval that is matched, and the granularity is too coarse. On the other hand, if the division is based solely on punctuation marks, the short sentences obtained may not reflect certain video content. Therefore, the plot synopsis should be re-sentenced using a trained fine-grained semantic segmentation model, so that each short sentence is as short as possible while still reflecting certain video content.

[0053] Specifically, when using a semantic segmentation model to segment a plot text, the sentence segmentation results of a pre-annotated plot summary can be used to train the semantic segmentation model. When annotating the sentence segmentation results of the pre-annotated plot summary, the sentence segmentation results are combined with the video content so that the annotated sentence segmentation results are as short as possible and can still reflect certain video content.

[0054] For example, the original sentence is: Liu Qingluan is worried that she is a new face in the palace and will be easily discovered, so she casts an illusion to make everyone in the palace feel that Liu Qingluan is familiar. When the two walk in the palace, everyone they meet is indeed familiar with them. After segmentation by the semantic segmentation model, at least one sentence is obtained:

[0055] Liu Qingluan was worried that she would be easily discovered as a new face in the palace.

[0056] Wuxin cast an illusion to make everyone in the palace feel that Liu Qingluan looked familiar.

[0057] The two walked in the palace, and everyone they met was indeed familiar to them.

[0058] Step 103, for each sentence, select from at least one video segment a video segment with the greatest similarity to the sentence in the same feature space as a target video segment matching the sentence;

[0059] Specifically, selecting a video segment having the greatest similarity with the sentence in the same feature space from at least one video segment as a target video segment matching the sentence includes:

[0060] Extract the first feature of the sentence; for each video segment, extract the second feature of the video segment; calculate the similarity between the first feature and the second feature in the same feature space; from each similarity, determine the maximum similarity; use the video segment corresponding to the maximum similarity as the target video segment matching the sentence.

[0061] For step 103, in the specific implementation, when selecting the target video segment that matches the sentence, the video segment retrieval model can be used to calculate the target video segment corresponding to each sentence; wherein, the video segment retrieval model can be a multimodal model, and the multimodal model performs well in image-text matching.

[0062] In the specific implementation, we must first train the multimodal model. The process of training the multimodal model is mainly divided into two parts: the first part is to pre-train the initial multimodal model using multiple sets of pre-training data to obtain a pre-trained multimodal model; the second part is to fine-tune the pre-trained multimodal model using labeled data to finally obtain a multimodal model.

[0063] In the past two years, multimodal models based on the Transformer structure have emerged. They are pre-trained with massive unlabeled data and then fine-tuned using a small amount of labeled data.

[0064] The process of training a multimodal model specifically includes:

[0065] First, multiple groups of pre-training data are used to train the initial multimodal model to obtain a pre-trained multimodal model; each group of pre-training data includes: a first text and a first video corresponding to the first text; the first text and the first video have a scene correspondence relationship; multiple second texts and second videos corresponding to each of the second texts are obtained; the second videos are annotated with annotated time intervals corresponding to the second texts; the following operations are performed on the second video corresponding to each of the second texts: the pre-trained multimodal model is trained using the second text, the second video corresponding to the second text, and the annotated time interval to obtain a multimodal model.

[0066] At present, the mainstream multimodal model is the Bert-based multimodal model, such as the VL-Bert (Visual-Linguistic BERT, VL-BERT for short) model, which treats inputs of different modalities such as images and texts equally; of course, other multimodal models can also be used. In order for the multimodal model to better align the two modalities of text and video, it is necessary to use CLIP to extract features. The model structure of CLIP adopts the classic dual-tower structure, with different image encoders (Image Encoder) and text encoders (Text Encoder) for the image domain and text domain. After obtaining the two features of text and video, the corresponding relationship between them is used to train the initial multimodal model to obtain a pre-trained multimodal model.

[0067] After the pre-training is completed, the pre-trained multimodal model is trained using the annotated data (using the second text, the second video corresponding to the second text, and the annotated time interval) to obtain a multimodal model.

[0068] After step 102, at least one sentence is obtained. In step 103, the purpose is to match each sentence to a target video segment and a corresponding time interval (also called a candidate time interval) in the target video segment.

[0069] For ease of understanding, here is an example. After step 102, 3 short sentences are obtained; the video range to be retrieved is: a complete video of 45 minutes is divided into 9 5-minute segments, and the multimodal model predicts the target video frequency band corresponding to each short sentence, as well as the corresponding time interval set in the target video segment. For example: a short sentence is: Liu Qingluan is worried that she is a new face in the palace and will be easily discovered; the 9 5-minute video segments that have been divided are segment 1, segment 2..., segment 9; using the multimodal model, the target video segment corresponding to the short sentence is learned as segment 1, and the corresponding time interval set in segment 1 is: 273s-284s, 271s-284s.

[0070] Step 104: for each sentence, select a target video interval that matches the sentence in time from the target video segment, and store the sentence and the target video interval correspondingly.

[0071] For obtaining the target video interval, in the specific implementation, at least one time interval corresponding to the target video segment is obtained to obtain a time interval set; a dynamic time warping algorithm can be used to extract a target time interval that matches the sentence in time from the time interval set; and a video interval corresponding to the target time interval is extracted from the target video segment as the target video interval.

[0072] In a specific implementation, the target video segment can be divided into multiple sub-segments according to a preset duration. For each sub-segment, the third feature of each sub-segment is extracted, the third feature and the first feature are mapped to the same feature space, and feature fusion is performed to obtain fused features; classification is performed based on each fused feature to obtain sub-segments corresponding to the sentence, and the time interval of each sub-segment is determined to obtain at least one time interval corresponding to the target video segment, that is, a time interval set. In a specific implementation, the sentence and the target video segment can be input into a multimodal model to identify the time interval set.

[0073] The purpose of the dynamic time warping algorithm (also called dynamic time warping algorithm, Dynamic Time Warping) is to find the time correspondence between two different templates (plot text and complete video). In the embodiment of the present application, the time correspondence between the plot text and the complete video is found, or in other words, the correspondence between a certain sentence and the target video interval. The dynamic time completion algorithm is based on an important assumption: the text order of the plot text and the time axis of the video should generally correspond, that is to say: the first sentence of the plot text generally corresponds to the beginning of the video, the last sentence of the plot text generally corresponds to the end of the video, and the sentences after a sentence are generally in the latter order on the video.

[0074] For the above-obtained time interval set, a dynamic time warping algorithm is used to filter out the time intervals that do not match the sequence, and obtain the best matching time interval, that is, the target time interval.

[0075] Among them, the corresponding relationship between the stored sentences and the target video intervals can be stored in a local database or in a cloud server to provide a material basis for video production.

[0076] In a specific embodiment, after storing the correspondence between the sentence and the target video interval, the method further includes: reading the sentence corresponding to the target video interval based on the correspondence; and displaying the sentence on a screen corresponding to the target video interval.

[0077] When making a video file, the corresponding relationship can be read from a local database or a cloud server, and the sentence corresponding to the target video interval can be found based on the corresponding relationship, and the sentence can be displayed on the screen of the target video interval. This can not only help the computer or user understand the video better and faster, but also quickly complete the video production.

[0078] In the embodiment of the present application, each sentence in the plot text is used as the granularity. For each sentence, a video clip with the greatest similarity to the sentence in the same feature space is selected from at least one video clip as the target video clip that matches the sentence, and a target video interval that matches each sentence in time is screened out from the target video clip, and the correspondence between the sentence and the target video interval is stored. Through this method, the video and the plot text can be effectively aligned, the alignment accuracy can be improved, the usability is strong, and it can be used to automatically produce a video summary of the plot explanation, thereby improving the efficiency of video production; according to the plot text, the target video clip and the target video interval can also be retrieved from a large number of video clips to provide materials for video editors, thereby improving the efficiency of video production.

[0079] Based on the same concept, a video and plot text alignment device is provided in the embodiment of the present application. The specific implementation of the device can be found in the description of the method embodiment part, and the repeated parts will not be repeated. Figure 2 As shown, the device mainly includes:

[0080] The acquisition module 201 is used to acquire the plot text and at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier;

[0081] A sentence division module 202, used to divide the plot text into at least one sentence; wherein each sentence at least reflects a segment of video content;

[0082] A retrieval module 203 is configured to select, for each of the sentences, from the at least one video segment, a video segment having the greatest similarity with the sentence in the same feature space as a target video segment matching the sentence;

[0083] The screening module 204 is used to screen out, for each of the sentences, a target video interval that temporally matches the sentence from the target video segment, and store the corresponding relationship between the sentence and the target video interval.

[0084] In a specific embodiment, the acquisition module 201 is used to acquire the complete video corresponding to the plot text; and divide the complete video into the at least one video segment according to a preset duration.

[0085] In a specific embodiment, the sentence segmentation module 202 is used to perform semantic segmentation on the plot text to obtain the at least one sentence.

[0086] In a specific embodiment, the retrieval module 203 is used to extract the first feature of the sentence; for each of the video clips, extract the second feature of the video clip; calculate the similarity between the first feature and the second feature in the same feature space; determine the maximum similarity from each of the similarities; and use the video clip corresponding to the maximum similarity as the target video clip that matches the sentence.

[0087] In a specific embodiment, the screening module 204 is used to obtain at least one time interval corresponding to the target video segment to obtain a time interval set; use a dynamic time warping algorithm to extract a target time interval that matches the sentence in time from the time interval set; and extract a video interval corresponding to the target time interval from the target video segment as the target video interval.

[0088] In a specific embodiment, the video and plot text alignment device provided in the embodiment of the present application also includes: a production module, which is used to read the sentence corresponding to the target video interval based on the corresponding relationship after storing the corresponding relationship between the sentence and the target video interval; and display the sentence on the screen of the target video interval.

[0089] In the embodiment of the present application, each sentence in the plot text is used as the granularity, and at least one target video segment that matches each sentence is retrieved from at least one video segment, and the target video interval that matches each sentence in time is screened from the target video segment, and the correspondence between the sentence and the target video interval is stored. Through this method, the video and the plot text can be effectively aligned, the alignment accuracy can be improved, the usability is strong, and it can be used to automatically produce a video summary of the plot explanation, thereby improving the efficiency of video production; according to the plot text, the target video segment and the target video interval can also be retrieved from a large number of video segments, providing materials for video editors, thereby improving the efficiency of video production.

[0090] Based on the same concept, an electronic device is also provided in the embodiment of the present application, such as Figure 3 As shown, the electronic device mainly includes: a processor 301, a memory 302 and a communication bus 303, wherein the processor 301 and the memory 302 communicate with each other through the communication bus 303. The memory 302 stores a program executable by the processor 301, and the processor 301 executes the program stored in the memory 302 to implement the following steps:

[0091] Acquire the plot text and acquire at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier;

[0092] Divide the plot text into at least one sentence; wherein each sentence reflects at least one segment of the video plot;

[0093] For each of the sentences, selecting, from the at least one video segment, a video segment having the greatest similarity to the sentence in the same feature space as a target video segment matching the sentence;

[0094] For each of the sentences, a target video interval that matches the sentence in time is screened out from the target video segment, and a corresponding relationship between the sentence and the target video interval is stored.

[0095] The communication bus 303 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 303 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] The memory 302 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the processor 301 .

[0097] The above-mentioned processor 301 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0098] In another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed on a computer, the computer executes a method for aligning a video and a plot text described in the above embodiment.

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction is transmitted from a website site, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk), etc.

[0100] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0101] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for aligning a video and a plot text, characterized in that: include: Acquire the plot text and acquire at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier; Divide the plot text into at least one sentence; wherein each sentence reflects at least one segment of the video plot; For each of the sentences, selecting, from the at least one video segment, a video segment having the greatest similarity to the sentence in the same feature space as a target video segment matching the sentence; For each of the sentences, a target video interval that matches the sentence in time is selected from the target video segment, and a corresponding relationship between the sentence and the target video interval is stored, including: Predicting a set of time intervals in the target video segment corresponding to each of the sentences through a multimodal model; Extracting a target time interval that matches the sentence in time from the time interval set; extracting a video interval corresponding to the target time interval from the target video clip as the target video interval; The time interval set is determined in the following manner: dividing the target video segment into a plurality of sub-segments according to a preset duration, extracting the third feature of each sub-segment, mapping the third feature and the first feature of the sentence to the same feature space, and performing feature fusion to obtain fused features; performing classification based on the fused features to obtain sub-segments corresponding to the sentence, determining the time interval of each sub-segment, obtaining at least one time interval corresponding to the target video segment, and obtaining the time interval set.

2. The method for aligning a video and a plot text according to claim 1, characterized in that: The obtaining of at least one video clip comprises: Obtain the complete video corresponding to the plot text; The complete video is divided into the at least one video segment according to a preset duration.

3. The method for aligning a video and a plot text according to claim 1, characterized in that: The step of dividing the plot text into at least one sentence comprises: The plot text is semantically segmented to obtain the at least one sentence.

4. The method for aligning a video and a plot text according to claim 1, characterized in that: The step of selecting, from the at least one video segment, a video segment having the greatest similarity to the sentence in the same feature space as a target video segment matching the sentence includes: extracting a first feature of the sentence; For each of the video segments, extract a second feature of the video segment; calculate the similarity between the first feature and the second feature in the same feature space; A maximum similarity is determined from the similarities; and a video segment corresponding to the maximum similarity is used as a target video segment matching the sentence.

5. The method for aligning a video and a plot text according to claim 4, characterized in that: The step of selecting a target video segment that temporally matches the sentence from the target video segment includes: Acquire at least one time interval corresponding to the target video segment to obtain a time interval set; Using a dynamic time warping algorithm, extracting a target time interval that matches the sentence in time from the set of time intervals; A video interval corresponding to the target time interval is extracted from the target video segment as the target video interval.

6. The method for aligning a video and a plot text according to any one of claims 1 to 5, characterized in that: After storing the correspondence between the sentence and the target video interval, the method further includes: Based on the corresponding relationship, reading the sentence corresponding to the target video interval; The sentence is displayed on a screen of the target video section.

7. A device for aligning video and plot text, characterized in that: include: An acquisition module, used to acquire the plot text and at least one video clip; wherein the plot text and at least one video clip correspond to the same multimedia content identifier; A sentence module, used to divide the plot text into at least one sentence; wherein each sentence reflects at least one segment of video content; A retrieval module, configured to select, for each of the sentences, from the at least one video segment, a video segment having the greatest similarity with the sentence in the same feature space as a target video segment matching the sentence; A screening module, configured to screen out, for each of the sentences, a target video interval that temporally matches the sentence from the target video segment, and store a corresponding relationship between the sentence and the target video interval; The screening module is specifically used to predict a set of time intervals in the target video segment corresponding to each of the sentences through a multimodal model; Extracting a target time interval that matches the sentence in time from the time interval set; extracting a video interval corresponding to the target time interval from the target video clip as the target video interval; The time interval set is determined in the following manner: dividing the target video segment into a plurality of sub-segments according to a preset duration, extracting the third feature of each sub-segment, mapping the third feature and the first feature of the sentence to the same feature space, and performing feature fusion to obtain fused features; performing classification based on the fused features to obtain sub-segments corresponding to the sentence, determining the time interval of each sub-segment, obtaining at least one time interval corresponding to the target video segment, and obtaining the time interval set.

8. The video and plot text alignment device according to claim 7, characterized in that: A retrieval module is used to extract the first feature of the sentence; for each of the video clips, extract the second feature of the video clip; calculate the similarity between the first feature and the second feature in the same feature space; determine the maximum similarity from each of the similarities; and use the video clip corresponding to the maximum similarity as the target video clip matching the sentence.

9. An electronic device, comprising: A processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to implement the method for aligning video and plot text as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for aligning a video and a plot text as described in any one of claims 1 to 6 is implemented.