Method for automatically understanding, editing and explaining movie and television play based on large model

Abstract

The invention discloses an automatic movie and television play understanding, editing and explaining method based on a large model, and the method comprises the steps: obtaining a complete audio in a to-be-explained video based on an audio extraction tool and a multi-modal model, so as to obtain a timestamp of an audio clip with a speaker, and further obtaining a script description of the to-be-explained video; the story abstract of the video to be explained is obtained; based on a text embedding model and an image-text feature extraction model, obtaining a video clip corresponding to the sentence-by-sentence picture description; and finally, acquiring the voice dubbing of the sentence-by-sentence picture description based on a voice-to-text conversion model, acquiring a complete video interpretation, and completing understanding, editing and interpretation of the video to be interpreted. According to the invention, the manual intervention is obviously reduced, the processing efficiency of video explanation editing is improved, and the labor cost is reduced. By automatically generating the script description, the story abstract and the explanation, the user can obtain the core content of the video more quickly and conveniently, and the user experience is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of video production, and in particular to a method for automatic understanding, editing and interpretation of film and television dramas based on a large model. Background Art

[0002] With the rapid development of the film and television industry, the number and types of film and television works are increasing, and the audience's demand for content is becoming more and more diversified, but watching a complete film and television series will consume a lot of time. With the rapid development of short video platforms in recent years, video commentary introduces the main content of the video in a relatively short time, saving the audience a lot of time, and gradually being accepted by more audiences. Traditional film and television editing and commentary work relies on manual operation. First, it is necessary to fully understand the video content, then add a plot description, and finally edit and merge through editing tools. The whole process is very dependent on the editor's editing experience and deep understanding of the video. It is inefficient and requires a lot of manpower and time. Summary of the invention

[0003] The present invention discloses a method for automatic understanding, editing and interpretation of film and television dramas based on a large model to overcome the above technical problems.

[0004] In order to achieve the above object, the technical solution of the present invention is:

[0005] A method for automatic understanding, editing and interpretation of film and television dramas based on a large model, comprising the following steps:

[0006] S1: Based on the audio extraction tool and the multimodal model, the complete audio in the video to be explained is obtained to obtain the timestamp of the audio segment with the speaker, and then the script description of the video to be explained is obtained;

[0007] S2: obtaining a plot summary of the video to be explained according to the script description of the video to be explained;

[0008] S3: obtaining a sentence-by-sentence picture description according to the plot summary of the video to be explained, and obtaining a video clip corresponding to the sentence-by-sentence picture description based on a text embedding model and a graphic feature extraction model;

[0009] S4: obtaining voice dubbing of the sentence-by-sentence picture description based on a speech-to-text conversion model, so as to obtain a complete video explanation according to the video clips corresponding to the sentence-by-sentence picture description, so as to complete the understanding, editing and explanation of the video to be explained.

[0010] Furthermore, in S1, the method for obtaining the script description of the video to be explained is as follows:

[0011] S11: obtaining the complete audio in the video to be explained based on an audio extraction tool;

[0012] S12: According to the complete audio, obtain the index of the audio segment with the speaker and the start time and end time of the audio segment with the speaker; obtain the timestamp of the audio segment with the speaker; and then obtain the partial audio corresponding to the timestamp;

[0013] S13: acquiring an identifier of a speaker corresponding to the timestamp according to the voice library and the local audio corresponding to the timestamp;

[0014] S14: Obtain a video segment corresponding to a timestamp in the video to be explained; and obtain a subtitle text in the video segment;

[0015] S15: Acquire a text description of the video segment based on the multimodal model according to the video segment corresponding to the timestamp;

[0016] S16: Obtain a script description of the video to be narrated according to the timestamp, the speaker identifier corresponding to the timestamp, the video segment corresponding to the timestamp, the subtitle text in the video segment, and the text description of the video segment.

[0017] Furthermore, in S2, the method for obtaining the plot summary of the video to be explained is as follows:

[0018] S21: using a shot segmentation algorithm to obtain a preliminarily segmented scene in the video to be explained; and based on a deep learning model, obtaining actual scenes in multiple videos to be explained, so as to obtain a start time and an end time of the actual scene in the video to be explained;

[0019] S22: acquiring a plot description of the actual scene in the video to be explained according to the start time and the end time of the actual scene in the video to be explained and the script description of the video to be explained;

[0020] S23: According to the plot description of the actual scene in the video to be explained, based on the large language model, a plot summary of the video to be explained is obtained.

[0021] Furthermore, in S3, the method for obtaining the video clip corresponding to the sentence-by-sentence picture description is as follows:

[0022] S31: Obtaining a sentence-by-sentence picture description according to a plot summary of the video to be explained;

[0023] S32: Based on the text embedding model, obtain a text feature vector of the subtitle text in the x-th video segment and a text feature vector of the picture description sentence by sentence to obtain a semantic similarity between the subtitle text and the picture description sentence by sentence in the x-th video segment, where x represents an index number of the video segment;

[0024] S33: based on the image-text feature extraction model, obtaining the image-text similarity between the x-th video clip and the sentence-by-sentence picture description;

[0025] S34: According to the semantic similarity between the subtitle text and the sentence-by-sentence picture description in the x-th video segment and the graphic similarity between the x-th video segment and the sentence-by-sentence picture description, obtain the maximum similarity score between the sentence-by-sentence picture description and the video segment to obtain the video segment corresponding to the sentence-by-sentence picture description.

[0026] Furthermore, in S34, the formula used to obtain the video clip corresponding to the sentence-by-sentence picture description is as follows:

[0027] sim=max{0.5*a x +0.5*w x}

[0028] Where: sim represents the maximum similarity score between the sentence-by-sentence description and the video clip; a x represents the semantic similarity between the sentence-by-sentence description and the subtitle text in the x-th video clip; w x Indicates the graphic similarity between the sentence-by-sentence picture description and the video segment corresponding to the subtitle text in the x-th video segment; x represents the index number of the video segment.

[0029] Beneficial effects: The method of automatic understanding, editing and interpretation of film and television dramas based on a large model of the present invention obtains the complete audio in the video to be interpreted based on an audio extraction tool and a multimodal model, so as to obtain the timestamp of the audio segment with the speaker, and then obtains the script description of the video to be interpreted, so as to obtain the plot summary of the video to be interpreted; then obtains the picture description sentence by sentence, and obtains the video segment corresponding to the picture description sentence by sentence based on the text embedding model and the graphic feature extraction model; finally, the voice dubbing of the picture description sentence by sentence is obtained based on the voice-to-text conversion model, so as to combine the video segment corresponding to the picture description sentence by sentence, obtain a complete video interpretation, and complete the understanding, editing and interpretation of the video to be interpreted. The present invention can automatically understand, edit and interpret the video based on the large model, significantly reduce manual intervention, improve the processing efficiency of video interpretation editing, and reduce labor costs. By automatically generating script descriptions, plot summaries and interpretations, users can obtain the core content of the video more quickly and conveniently, and enhance the user experience. At the same time, it can provide new ideas and tools for the secondary creation of film and television drama content, which is helpful to promote content innovation and diversified development. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 A flow chart of the method for automatic understanding, editing and interpretation of film and television dramas based on a large model of the present invention;

[0032] Figure 2 This is a schematic diagram of the overall structure of automatic understanding, editing and interpretation of film and television dramas based on a large model in an embodiment of the present invention;

[0033] Figure 3 It is an overall flow chart of the automatic video understanding module in an embodiment of the present invention;

[0034] Figure 4 It is an overall flow chart of the video summarization module in an embodiment of the present invention;

[0035] Figure 5 is a flow chart of a method for obtaining a video clip corresponding to a sentence-by-sentence picture description in an embodiment of the present invention;

[0036] Figure 6 The present invention is a flowchart of a method for obtaining a complete video description in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] This embodiment introduces a method for automatic understanding, editing and interpretation of film and television dramas based on a large model. Figure 1 , Figure 2 As shown, including:

[0039] S1: Based on the audio extraction tool and the multimodal model, the complete audio in the video to be explained is obtained to obtain the timestamp of the audio segment with the speaker, and then the speaker identifier corresponding to the timestamp, the video segment corresponding to the timestamp, the subtitle text in the video segment, and the text description of the video segment are obtained to obtain the script description of the video to be explained;

[0040] Specifically, this embodiment uses the video automatic understanding module to obtain the script description of the video to be explained, such as Figure 3 As shown, the specific method is as follows:

[0041] S11: Acquire complete audio in the video according to the video to be explained;

[0042] S12: According to the complete audio, obtain the index of the audio segment with the speaker and the start time and end time of the audio segment with the speaker; obtain the timestamp of the audio segment with the speaker; and then obtain the partial audio corresponding to the timestamp;

[0043] Specifically, this embodiment first uses an audio extraction tool (such as FFmpeg) to extract the complete audio in the video to be explained. Then, the Voice Activity Detection (VAD) model is used to analyze the extracted complete audio, identify all audio segments with speakers in the complete audio, and extract the corresponding timestamps. The output result of this part is a timestamp list, marking each time period in the video to be explained that contains the speaker's voice. Then, an audio extraction tool (such as FFmpeg) is used to obtain the local audio corresponding to the timestamp of the audio segment with the speaker;

[0044] S13: acquiring an identifier of a speaker corresponding to the timestamp according to the voice library and the local audio corresponding to the timestamp;

[0045] Specifically, in this embodiment, the existing speaker confirmation tool is used to compare these partial audios with the voice library input by the user, and the identity of the speaker in the partial audio corresponding to the timestamp of the audio segment with the speaker is identified. A dictionary format data is output, such as: {time period 1: speaker X, time period 2: speaker Y...}, which clearly indicates the identity of the speaker in each time period.

[0046] S14: according to the video to be explained, obtaining a video segment corresponding to a timestamp in the video to be explained; and obtaining a subtitle text in the video segment;

[0047] Specifically, according to the extracted timestamp list, FFmpeg is used to split the video to generate fragmented video clip materials. These materials will be used for subsequent analysis and processing to ensure that each video clip corresponds to the corresponding voice time period.

[0048] After the video clips are split into video clips corresponding to the timestamps, the text recognition tool (Optical Character Recognition, OCR) is used to extract the subtitles in each video clip. The specific method is to first convert the video clips into image frames and perform text recognition on each frame. In this process, in order to avoid the influence of the text that may be included in the image of some videos on the subtitle recognition, it is necessary to record the text information of each frame and its text box position, and take the text in the text box area with the largest confidence interval as the subtitle of the video segment, while other text is considered as interference text and needs to be removed from the subtitle.

[0049] S15: Acquire a text description of the video segment based on the multimodal model according to the video segment corresponding to the timestamp;

[0050] Specifically, this embodiment obtains the key video frame sequence of the video segment, and uses a multimodal model (such as gpt-4v, InternVL, etc.) to describe the screen content and obtain a text description of the video segment; specifically, first extract the image frame of the video segment, and use the method of equally divided sampling points to extract multiple frames from the video to obtain a representative video frame sequence. The extracted video frames are then preprocessed, their image sizes are adjusted and converted into tensors, and the tensors of multiple images are spliced ​​and input into the multimodal model to generate a short summary text description for each video segment, that is, the text description of the video segment. The multimodal model is a conventional technology in the field and is not described in detail here.

[0051] S16: Obtaining a script description of the video to be explained according to the timestamp, the speaker identifier corresponding to the timestamp, the video segment corresponding to the timestamp, the subtitle text in the video segment, and the text description of the video segment;

[0052] Specifically, this embodiment obtains the script description of the video to be explained based on the timestamp, the identification of the speaker corresponding to the timestamp, the video clip corresponding to the timestamp, the subtitle text in the video clip, and the text description of the video clip. The method adopted is the existing technology in the field and is not described in detail here. Among them, the script description is composed of the text descriptions of each video clip. The generated script description of the video to be explained basically covers the main information of the video, thereby realizing the automatic understanding of the video for subsequent further processing. This embodiment performs multi-dimensional analysis on the video to be explained, including video splitting, character feature information (voice library) comparison, screen description and other methods, so as to realize automatic understanding of the video content and generate a corresponding script description for the video.

[0053] S2: obtaining a plot summary of the video to be explained according to the script description of the video to be explained;

[0054] Specifically, the plot summary of the video to be explained in this embodiment is obtained through the video summary module, such as Figure 4 As shown, the specific method is as follows:

[0055] S21: using a shot segmentation algorithm to obtain a preliminarily segmented scene in the video to be explained; and based on a deep learning model, obtaining actual scenes in multiple videos to be explained, so as to obtain a start time and an end time of the actual scene in the video to be explained;

[0056] S22: acquiring a plot description of the actual scene in the video to be explained according to the start time and the end time of the actual scene in the video to be explained and the script description of the video to be explained;

[0057] Specifically, each video in a film or TV work usually contains multiple actual scenes. In order to describe the plot more accurately, it is necessary to first distinguish each actual scene. In this embodiment, a shot segmentation algorithm is first used to identify each shot switching point by analyzing the changes in the color space of the picture. The shot switching points obtained by this algorithm will divide each actual scene into multiple scenes due to the changes in light and darkness. In other words, each actual scene is divided into multiple preliminary segmented scenes. Therefore, this embodiment uses a deep learning model (such as iic / cv_resnet50-bert_video-scene-segmentation_movienet) to aggregate and combine the multiple preliminary segmented scenes in each actual scene, thereby obtaining the timeline of each actual scene, that is, the start time and end time of the actual scene in the video to be explained, and the script description of the video to be explained is corresponded to the start time and end time of the actual scene in the video to be explained, so as to obtain the plot content in each scene.

[0058] S23: obtaining a plot summary of the video to be explained based on the large language model according to the plot description of the actual scene in the video to be explained;

[0059] Specifically, the plot events in each actual scene are analyzed through a large language model (such as ChatGPT, Kimi, etc.), and the key plots in the video are identified. At the same time, the key scenes in the video are identified to describe the key scene information in detail and the non-key information in a simple manner, thereby converting the script into a plot summary. Among them, the method of obtaining the plot summary of the video based on the script description of each scene in the video based on the large language model is a prior art in the field and will not be described in detail here.

[0060] S3: obtaining a sentence-by-sentence picture description according to the plot summary of the video to be explained, and obtaining a video clip corresponding to the sentence-by-sentence picture description based on a text embedding model and a graphic feature extraction model;

[0061] S31: Obtaining a sentence-by-sentence picture description according to a plot summary of the video to be explained;

[0062] Specifically, this embodiment uses regular matching to segment the plot summary document according to the punctuation marks in the plot summary, including commas, semicolons, periods, exclamation marks, question marks, quotation marks, line breaks, etc. Through this segmentation method, the continuous plot summary is segmented into sentence-by-sentence picture descriptions to ensure that each sentence can accurately correspond to a certain video clip in the original video to be explained, such as Figure 4 , Figure 5 shown.

[0063] S32: Based on the text embedding model, obtain the text feature vector of the subtitle text in the x-th video segment and the text feature vector of the picture description sentence by sentence to obtain the semantic similarity between the subtitle text in the video segment and the picture description sentence by sentence,

[0064] Specifically, each sentence of the plot summary is matched with the corresponding video clip. The specific method is to first extract the text feature vector of each sentence-by-sentence picture description and the subtitle text in the video clip through a text embedding model (such as BAAI / bge-large-zh); and then calculate it through cosine similarity. This process can obtain the similarity between the subtitle text in the video clip and the sentence-by-sentence picture description, which is recorded as a1, a2, a3, a4….

[0065] S33: Based on the image-text feature extraction model, obtain the image-text similarity between the x-th video segment and the sentence-by-sentence screen description; wherein x represents the index number of the video segment; S34: According to the semantic similarity between the subtitle text in the x-th video segment and the sentence-by-sentence screen description and the image-text similarity between the x-th video segment and the sentence-by-sentence screen description, obtain the maximum similarity score between the sentence-by-sentence screen description and the video segment, so as to obtain the video segment corresponding to the sentence-by-sentence screen description;

[0066] Preferably, the formula used to obtain the video clip corresponding to the sentence-by-sentence picture description is as follows:

[0067] sim=max{0.5*a x +0.5*w x}

[0068] Where: sim represents the maximum similarity score between the sentence-by-sentence picture description and the video clip, and the video clip corresponding to this score is the video clip corresponding to the sentence-by-sentence picture description; a x represents the semantic similarity between the sentence-by-sentence description and the subtitle text in the x-th video clip; w x represents the graphic similarity between the sentence-by-sentence picture description and the video segment corresponding to the subtitle text in the x-th video segment; x represents the index number of the video segment;

[0069] Specifically, this embodiment uses a graphic feature extraction model (such as CLIP) to perform feature extraction on the segmented video segments corresponding to the subtitle text in the video segments and the sentence-by-sentence screen descriptions. The specific method is to extract video frames, and it is possible to choose to take fixed frames at equal intervals or take frames at time intervals. The CLIP model is used to extract image feature vectors from the extracted frames. For multi-frame image features, the feature vector of the video segment is obtained through average pooling processing to ensure accurate representation of the video content. Then, the cosine similarity is calculated. This process can obtain the visual similarity between the sentence-by-sentence screen description and the video segment corresponding to the subtitle text in the video segment, which is denoted as w1, w2, w3, w4... Therefore, the similarity between each sentence-by-sentence screen description and the video segment is expressed as sim=0.5*a x +0.5*w x , the maximum value of sim is the video screen that best matches the text segment. This process matches both text and vision at the same time, ensuring the precise association between the sentence-by-sentence picture description and the video clip, and achieving high-quality multimodal video content retrieval.

[0070] S4: obtaining voice dubbing of the sentence-by-sentence picture description based on a speech-to-text conversion model, so as to obtain a complete video explanation according to the video clips corresponding to the sentence-by-sentence picture description, so as to complete the understanding, editing and explanation of the video to be explained.

[0071] Specifically, this embodiment uses a text-to-speech (tts) model to dub each sentence-by-speech description to generate a commentary, and then uses tools such as ffmpeg to synthesize each video segment corresponding to the sentence-by-speech description, the voice dubbing of the sentence-by-speech description, and the sentence-by-speech description to obtain a video segment with dubbing and subtitles. Finally, the various video segments are spliced ​​to generate a complete video commentary, such as Figure 6 shown.

[0072] The method for automatic understanding, editing and interpretation of film and television dramas based on a large model in this embodiment has the following beneficial effects:

[0073] (1) Video content can be automatically understood: Based on a multimodal large model, this embodiment combines audio and video processing, speaker recognition, speaker confirmation, subtitle extraction, screen description and other technologies to comprehensively analyze video content from multiple dimensions (such as voice, screen, subtitles, etc.) and generate a script description of the corresponding video. This achieves automatic and in-depth understanding of video content.

[0074] (2) Video content can be accurately summarized: Faced with complex plots, how to quickly extract key plots and generate concise and clear summaries is a challenge. This embodiment is based on a large language model and uses technologies such as automatic scene segmentation and key plot identification to achieve key plot summary and summary generation.

[0075] (3) Video and text can be accurately matched: This embodiment achieves accurate matching of text and video content through multimodal video retrieval technology.

[0076] (4) Efficient video synthesis and interpretation: This embodiment uses text-to-speech synthesis (TTS) and audio and video splicing technology to process and splice video segments, thereby achieving efficient and high-quality video synthesis and interpretation.

[0077] The video understanding, editing and interpretation method of the present invention has significant technical advantages and is not only applicable to the field of film and television dramas, but can also be expanded to multiple fields such as news, documentaries, short videos, etc., and has broad application prospects.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic understanding, editing and interpretation of film and television dramas based on a large model, characterized in that: The steps include: S1: Based on the audio extraction tool and the multimodal model, the complete audio in the video to be explained is obtained to obtain the timestamp of the audio segment with the speaker, and then the script description of the video to be explained is obtained; S2: obtaining a plot summary of the video to be explained according to the script description of the video to be explained; S3: obtaining a sentence-by-sentence picture description according to the plot summary of the video to be explained, and obtaining a video clip corresponding to the sentence-by-sentence picture description based on a text embedding model and a graphic feature extraction model; S4: obtaining voice dubbing of the sentence-by-sentence picture description based on a speech-to-text conversion model, so as to obtain a complete video explanation according to the video clips corresponding to the sentence-by-sentence picture description, so as to complete the understanding, editing and explanation of the video to be explained.

2. The method for automatic understanding, editing and interpretation of film and television dramas based on a large model according to claim 1, characterized in that: In S1, the method used to obtain the script description of the video to be explained is as follows: S11: obtaining the complete audio in the video to be explained based on an audio extraction tool; S12: According to the complete audio, obtain the index of the audio segment with the speaker and the start time and end time of the audio segment with the speaker; obtain the timestamp of the audio segment with the speaker; and then obtain the partial audio corresponding to the timestamp; S13: acquiring an identifier of a speaker corresponding to the timestamp according to the voice library and the local audio corresponding to the timestamp; S14: Obtain a video segment corresponding to a timestamp in the video to be explained; and obtain a subtitle text in the video segment; S15: Acquire a text description of the video segment based on the multimodal model according to the video segment corresponding to the timestamp; S16: Obtain a script description of the video to be narrated according to the timestamp, the speaker identifier corresponding to the timestamp, the video segment corresponding to the timestamp, the subtitle text in the video segment, and the text description of the video segment.

3. The method for automatic understanding, editing and interpretation of film and television dramas based on a large model according to claim 1, characterized in that: In S2, the method for obtaining the plot summary of the video to be explained is as follows: S21: using a shot segmentation algorithm to obtain a preliminarily segmented scene in the video to be explained; and based on a deep learning model, obtaining actual scenes in multiple videos to be explained, so as to obtain a start time and an end time of the actual scene in the video to be explained; S22: acquiring a plot description of the actual scene in the video to be explained according to the start time and the end time of the actual scene in the video to be explained and the script description of the video to be explained; S23: According to the plot description of the actual scene in the video to be explained, based on the large language model, a plot summary of the video to be explained is obtained.

4. The method for automatic understanding, editing and interpretation of film and television dramas based on a large model according to claim 1, characterized in that: In S3, the method used to obtain the video clip corresponding to the sentence-by-sentence picture description is as follows: S31: Obtaining a sentence-by-sentence picture description according to a plot summary of the video to be explained; S32: Based on the text embedding model, obtain a text feature vector of the subtitle text in the x-th video segment and a text feature vector of the picture description sentence by sentence to obtain a semantic similarity between the subtitle text and the picture description sentence by sentence in the x-th video segment, where x represents an index number of the video segment; S33: based on the image-text feature extraction model, obtaining the image-text similarity between the x-th video clip and the sentence-by-sentence picture description; S34: According to the semantic similarity between the subtitle text and the sentence-by-sentence picture description in the x-th video segment and the graphic similarity between the x-th video segment and the sentence-by-sentence picture description, obtain the maximum similarity score between the sentence-by-sentence picture description and the video segment to obtain the video segment corresponding to the sentence-by-sentence picture description.

5. The method for automatic understanding, editing and interpretation of film and television dramas based on a large model according to claim 4 is characterized in that: In S34, the formula used to obtain the video segment corresponding to the sentence-by-sentence picture description is as follows: sim=max{0.5*a x +0.5*w x } Where: sim represents the maximum similarity score between the sentence-by-sentence description and the video clip; a x represents the semantic similarity between the sentence-by-sentence description and the subtitle text in the x-th video clip; w x Indicates the graphic similarity between the sentence-by-sentence picture description and the video segment corresponding to the subtitle text in the x-th video segment; x represents the index number of the video segment.

Images