A method for intelligent reduction of film and television dramas based on large models

Abstract

The present application discloses a method for intelligent reduction of film and television dramas based on a large model, including: S1: dividing the film and television drama video and the plot introduction document, completing the segment division of the film and television drama video, and the plot segment division of the plot introduction document; S2: using the large language model to complete the matching between the video segment and the plot segment; S3: using the large language model to generate a reduction script according to the plot introduction document; S4: the large language model completes the selection of the video segment in step S1 based on the reduction script obtained in step S3 and the matching relationship between the plot segment and the video segment obtained in step S2; S5: using the reduced storyline content, cross-modal retrieval is performed within the range of the video segment selected in step S4 to obtain the video screen. The present application realizes an automated editing process by intelligently generating a reduction script, reduces manual operations, and improves editing efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent editing, and in particular to an intelligent reduction method for film and television dramas based on a large model. Background Art

[0002] At present, the editing of film and television dramas mainly relies on manual editing, which faces many difficulties. First of all, manual editing is not only time-consuming and laborious, but also requires professional editors to analyze each shot frame by frame to ensure the accurate expression of the plot, characters and emotions. This highly manpower-dependent working method not only leads to the extension of the production cycle, but also increases labor costs.

[0003] Secondly, existing automated editing tools often seem incapable of handling complex plots, character relationships, and emotional changes. These tools usually rely on cross-modal models to intelligently select pictures based on picture semantics, but these models have difficulty in deeply understanding the content of film and television drama pictures and their role in the plot narrative, resulting in the inability to accurately locate the appropriate video pictures. In addition, existing tools are also insufficient in ensuring the timing of editing. They cannot limit the selection of pictures according to the chronological order of the plot development, which may lead to time sequence confusion.

[0004] The following technical problems exist in the current film and television drama downsizing process:

[0005] 1. Time-consuming and labor-intensive manual editing: From determining the plot to selecting the corresponding pictures for editing, the entire process requires a lot of time and human resources.

[0006] 2. Existing intelligent editing tools have difficulty understanding and processing film and television drama videos: Existing intelligent editing tools mostly rely on cross-modal models to intelligently select images from the semantic level of the images, but this model cannot fully understand the content of film and television drama images and the plot they are telling, making it difficult to accurately find suitable video images.

[0007] 3. Existing intelligent editing tools often ignore the time sequence constraints of plot development: Existing automated tools usually select pictures from the semantic level within the entire film and television video, but fail to limit the selection range according to specific plots, which may lead to time sequence confusion.

[0008] For example, when the plot is "fireworks during the Spring Festival in 2000", the cross-modal model can only select the scene of "fireworks during the Spring Festival" from a semantic point of view, but cannot distinguish whether this scene occurred in 2000. If there are multiple scenes of fireworks during the Spring Festival in a film or TV series, the cross-modal model cannot accurately judge and select the scenes that meet the timing requirements, which may cause the editing results to violate the timing logic of the original plot. Summary of the invention

[0009] The purpose of this application is to overcome the problems of the prior art and disclose a method for intelligent reduction of film and television dramas based on a large model. By introducing a plot introduction document as a bridge, based on the natural language processing and image understanding capabilities of the large model, and combining artificial intelligence technology in the field of computer vision, the complete film and television drama video is divided into multiple video clips and accurately located at each different plot node. In addition, the method of this application realizes an automated editing process by intelligently generating reduction scripts, reducing manual operations and improving editing efficiency.

[0010] The purpose of this application is achieved through the following technical solutions:

[0011] A method for intelligently compiling a film or TV series based on a large model, the method comprising:

[0012] S1: The division and processing of the film and television drama videos and the plot introduction documents are completed, and the segment division of the film and television drama videos and the plot paragraph division of the plot introduction documents are completed;

[0013] S2: Use a large language model to match video clips with plot segments;

[0014] S3: Generate an abridged script based on the plot introduction document using a large language model;

[0015] S4: The large language model completes the selection of the video clip in step S1 based on the abridged script obtained in step S3 and the matching relationship between the plot paragraph and the video clip obtained in step S2;

[0016] S5: Using the condensed storyline content, perform cross-modal retrieval within the video clip selected in step S4 to obtain video images.

[0017] According to a preferred embodiment, step S1 further includes obtaining feature information of each video clip, wherein the feature information includes: picture description information, information about characters appearing in the video, and dialogue information.

[0018] According to a preferred embodiment, step S2 includes: taking a video clip as a unit, the large language model selects a plot segment matching the video clip from a list of plot segments to be selected based on the first prompt word, and finally obtains a set of video clips matching each plot segment;

[0019] The first prompt word includes characteristic information of the video clip and a list of plot segments to be selected.

[0020] According to a preferred embodiment, step S3 includes: the large language model generates a shortened script of the plot introduction document based on the second prompt word, wherein the second prompt word includes instructions for generating the shortened script and the story outline content;

[0021] The generated abbreviated script contains several pieces of data, each piece of data includes the abbreviated storyline content and its corresponding plot paragraph information.

[0022] According to a preferred embodiment, step S4 includes: taking a data in the abbreviated script as a unit, reading a set of video clips matching the plot segment from step S2 according to its corresponding plot segment information, and then reading information of each video clip in the set of video clips from the result of step S1 to form a list of videos to be selected;

[0023] The large language model selects a video clip that matches the shortened storyline from the list of candidate videos based on the third prompt word;

[0024] The third prompt word includes: video clip selection instructions, condensed storyline content and to-be-selected video list information.

[0025] According to a preferred embodiment, in step S1, the complete film and television drama video is divided into several video clips according to the scene by using a video scene segmentation algorithm.

[0026]

[0027] in, represents the video scene segmentation algorithm, For complete videos of movies and TV series, are the n video clips obtained after algorithm processing;

[0028] The picture description information is obtained through a multimodal large model:

[0029]

[0030] in, represents a multimodal large model, Indicates prompt words, To represent a video clip, To obtain On the screen description information, enter For video or by The resulting image sequence;

[0031] The information of the people appearing in the scene is obtained through the face recognition algorithm and the pedestrian re-identification algorithm:

[0032]

[0033] in, represents the face recognition algorithm and pedestrian re-identification algorithm, Indicates that the match belongs to a video segment As a result, To get in Information about the people appearing on the screen;

[0034] The lines information is obtained through a speech recognition algorithm:

[0035]

[0036] in, represents the speech recognition algorithm, is the one obtained by matching Line information;

[0037] If the TV series or film has subtitles, the OCR algorithm can be used to obtain the lines.

[0038]

[0039] in, Indicates the OCR algorithm.

[0040] According to a preferred embodiment, in step S1, the plot introduction document is divided into a number of plot sections according to semantics by using a large language model.

[0041]

[0042] in, represents a large language model, Indicates the corresponding prompt word, Complete video for TV series The corresponding plot introduction document, The plot is divided into sections.

[0043] According to a preferred embodiment, step S2 comprises:

[0044]

[0045] in, Indicates the corresponding prompt word, Represents a list of plot segments to be selected, for each plot segment or a subset thereof, Represents a large language model from Select the video clip from Matching plot passages;

[0046] After completing the matching of all video clips, we finally get a set of video clips that match each plot segment, such as

[0047]

[0048] Among them, the collection Representation and plot The set of matching video clips includes all the ones selected by the large language model in this step of .

[0049] According to a preferred embodiment, the abbreviated script generated in step S3 is:

[0050]

[0051] in, Indicates the corresponding prompt word, Represents the abridged script generated by the large language model.

[0052] According to a preferred embodiment, in step S4, the set of video clips matching the paragraph 1 read from step S2 is ,m≤n,

[0053] Based on the picture description information, character information and dialogue information of the corresponding video clip read in step S1, then:

[0054]

[0055] in, Indicates a list of video clips to be selected;

[0056] Then, using a large language model, Selected from Matching video clips :

[0057]

[0058] in, Indicates the corresponding prompt word, Represents the storyline content information in the abridged script.

[0059] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0060] Beneficial effects of this application:

[0061] 1. Improve editing accuracy and efficiency: This invention improves the analysis accuracy of film and television plots and emotions by applying large model technology, combining natural language processing and image understanding capabilities, and optimizes the expression of plots, characters and emotions during the editing process, thereby improving the quality and efficiency of editing.

[0062] 2. Reduce labor costs and production cycles: By automating the editing process, the present invention reduces dependence on professional editors and saves time for frame-by-frame analysis of footage, thereby shortening the production cycle and reducing labor costs.

[0063] 3. Enhance plot understanding and timing accuracy: By introducing the plot introduction document as a bridge, the present invention can more accurately understand and disassemble the film and television drama video, accurately locate different plot nodes, ensure the accuracy of the editing timing, and avoid the problem of timing disorder.

[0064] 4. Realize the automation and intelligent editing process of film and television drama reduction: The present invention reduces manual operations, improves editing efficiency, and realizes the automation and intelligence of editing work by intelligently generating reduction scripts and automated editing processes.

[0065] 5. Promote innovation in the short video production industry: This invention not only saves the time and cost of film and television drama production, but also brings innovative solutions to short video production, which is expected to promote the development of the entire industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a schematic diagram of the process of the intelligent reduction method of film and television dramas in this application;

[0067] Figure 2 This is a flow chart corresponding to step S1 of the method of this application;

[0068] Figure 3 This is a flow chart corresponding to step S2 of the method of this application;

[0069] Figure 4 This is a flow chart corresponding to step S3 of the method of this application;

[0070] Figure 5 This is a flow chart corresponding to step S4 of the method of this application;

[0071] Figure 6 It is a flow chart corresponding to step S5 of the method of this application. DETAILED DESCRIPTION

[0072] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0073] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. The terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0074] In addition, the present application would like to point out that, in the present application, unless the specific structure, connection relationship, positional relationship, power source relationship, etc. are specifically written out, the structure, connection relationship, positional relationship, power source relationship, etc. involved in the present application are all known by those skilled in the art on the basis of the prior art without creative work.

[0075] Example

[0076] refer to Figures 1 to 6 As shown, the present application discloses a method for intelligent reduction of film and television dramas based on a large model, and the specific implementation process is as follows.

[0077] Step 1: Segment the TV series video and segment the plot introduction document. Specifically, for TV series videos, the complete TV series video can be divided into multiple video clips according to the scene through the video scene segmentation algorithm.

[0078]

[0079] in, represents the video scene segmentation algorithm, For a complete video of a film or TV series, These are the video clips obtained after algorithm processing.

[0080] Then, for each video clip, the multimodal large model can be used to obtain the picture description information.

[0081]

[0082] in, represents a multimodal large model, Indicates the prompt word used here, For a video clip, To obtain The input here is the description of the screen. It can be a video directly or by The resulting image sequence.

[0083] The information of the people appearing in the scene can be obtained through the face recognition algorithm and the pedestrian re-identification algorithm.

[0084]

[0085] in, represents the face recognition algorithm and pedestrian re-identification algorithm, Indicates that the match belongs to a video segment As a result, To get in In addition, the extraction of the information of the characters in the scene can also be directly Go up and do it, .

[0086] The dialogue information can be obtained through speech recognition algorithm or optical character recognition (OCR) algorithm. If the audio is subjected to speech recognition algorithm:

[0087]

[0088] in, represents the speech recognition algorithm, is the one obtained by matching Line information;

[0089] If the film or TV series has subtitles, you can also use the OCR algorithm:

[0090]

[0091] in, In other words, the extraction of lines can also be done directly in Go up and do it, or .

[0092] For the plot introduction document, the large language model can be used to divide it into multiple plot sections according to semantics. If the story synopsis has already divided the plot sections, this process can be omitted. The semantic segmentation task is explained in the prompt word, and the complete story synopsis is provided in the prompt word. The large language model outputs the divided plot sections.

[0093]

[0094] in, represents a large language model, Indicates the prompt word used here, Complete video for TV series The corresponding plot introduction document, It is divided into multiple plot sections.

[0095] Step 2: Use the large language model to complete the matching between the video clip and the plot segment. Take a video clip as a unit, explain the matching task in the prompt word, and provide the information on the video clip obtained in step 1 and the list of plot segments to be selected in the prompt word. The large language model selects the plot segment that matches the video clip from the list of plot segments to be selected.

[0096]

[0097] in, Indicates the prompt word used here, Represents a list of plot segments to be selected, which is or a subset thereof. Represents a large language model from Select the video clip from Matching plot segments. Complete all Finally, we get a set of video clips that match each plot segment, such as:

[0098]

[0099] Among them, the collection Representation and plot The set of matching video clips includes all the ones selected by the large language model in this step of .

[0100] Step 3: Use the large language model to generate an abridged script based on the plot introduction document. In the prompt, explain the task of generating an abridged script and provide the story outline in the prompt. The large language model generates the abridged script.

[0101] The abridged script contains multiple pieces of data, each piece of data contains the abridged storyline content and the corresponding plot paragraph information from which it is derived.

[0102]

[0103] in, Indicates the prompt word used here, Represents the abbreviated script generated by the large language model. In the example, there are multiple data, each of which is as follows: {"plot content":"xxxxxxx","source":"paragraph 1"}, "plot content":"xxxxxxx" is the abridged plot content, "source":"paragraph 1" is the plot paragraph information from which it is derived, and "paragraph 1" is the .

[0104] Step 4: The large language model completes the selection of the video clips in step 1 based on the abridged script obtained in step 3 and the matching relationship between the plot segments and the video clips obtained in step 2.

[0105] Taking a data in the abridged script as a unit, a set of video clips matching the plot segment is read from step 2 according to its corresponding plot segment information, and then the information of each video clip in the video clip set is read from the result of step 1 to form a list of videos to be selected; the large language model selects video clips that match the abridged plot content from the list of videos to be selected based on corresponding prompt words; the prompt words include: an indication of selecting a video clip, the abridged plot content and information of the list of videos to be selected.

[0106] For each piece of data in the abbreviated script, take {"plot content": "Xiao Ming happily goes to school", "source": "paragraph 1"} as an example. The abbreviated plot content is "Xiao Ming happily goes to school", and the plot paragraph information it comes from is paragraph 1. The set of video clips matching paragraph 1 is read from the result of step 2. Assume that , it is also necessary to read the picture description information, the information of the characters appearing in the video clip, and the information of the lines from the result of step 1. Then

[0107]

[0108] in, Indicates a list of video clips to be selected;

[0109] Then use the large language model from Select the video clips that match "Xiao Ming goes to school happily" :

[0110] v z = LLM ( prompt step4 ("Xiao Ming went to school happily", V list ))

[0111] in, Indicates the prompt word used here. Each data in the abbreviated script is carried out in this way.

[0112] Step 5: According to the abbreviated script, for each piece of data, use the abbreviated storyline content as the search condition on the video clip selected in step 4, and obtain the video screen through cross-modal search.

[0113] Continuing with {"Plot content": "Xiao Ming goes to school happily", "Source": "Paragraph 1"} as an example, the video clip selected from this data in step 4 is , then in In the above example, we use "Xiao Ming happily goes to school" as the search condition, and through cross-modal search, we take the first search result as the video screen.

[0114] After completing all the data in the shortened script, the corresponding pictures for each shortened storyline will be obtained, completing the intelligent shortening process.

[0115] This application is based on a large model and introduces plot introduction documents to improve the understanding and analysis of film and television drama videos: using the natural language processing and image understanding capabilities of the large model to improve the accuracy of plot and emotion analysis. At the same time, by using large model technology and artificial intelligence technology in fields such as computer vision, the plot introduction document is introduced as a bridge to achieve the segmentation of complete film and television drama videos into several video clips and locate different plots. In addition, this application uses a large model to analyze film and television drama videos and plots, generate editing scripts and select corresponding materials, reducing manual operations and improving editing efficiency.

[0116] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for intelligent reduction of film and television dramas based on a large model, characterized in that: The intelligent shortening method for film and television dramas comprises: S1: The division and processing of the film and television drama videos and the plot introduction documents are completed, and the segment division of the film and television drama videos and the plot paragraph division of the plot introduction documents are completed; S2: Use a large language model to match video clips with plot segments; S3: Generate an abridged script based on the plot introduction document using a large language model; Step S3 includes: the large language model generates a shortened script of the plot introduction document based on the second prompt word, wherein the second prompt word includes instructions for generating the shortened script and a story outline; The generated abridged script contains several pieces of data, each piece of data including the abridged storyline content and its corresponding plot paragraph information; S4: The large language model completes the selection of the video clip in step S1 based on the abridged script obtained in step S3 and the matching relationship between the plot paragraph and the video clip obtained in step S2; Step S4 includes: taking a data in the abbreviated script as a unit, reading a video clip set matching the plot segment from step S2 according to its corresponding plot segment information, and then reading information of each video clip in the video clip set from the result of step S1 to form a list of videos to be selected; The large language model selects a video clip that matches the shortened storyline from the list of candidate videos based on the third prompt word; The third prompt word includes: video clip selection instructions, abridged storyline content and a list of videos to be selected; S5: Using the condensed storyline content, perform cross-modal retrieval within the video clip selected in step S4 to obtain video images.

2. The intelligent shortening method for film and television dramas according to claim 1, characterized in that: Step S1 also includes obtaining feature information of each video clip, wherein the feature information includes: picture description information, character information and dialogue information.

3. The intelligent shortening method for film and television dramas as claimed in claim 2, characterized in that: Step S2 includes: taking a video clip as a unit, the large language model selects a plot segment matching the video clip from a list of plot segments to be selected based on the first prompt word, and finally obtains a set of video clips matching each plot segment; The first prompt word includes characteristic information of the video clip and a list of plot segments to be selected.

4. The intelligent shortening method for film and television dramas according to claim 2, characterized in that: In step S1, the complete film and television drama video is divided into several video clips according to the scene by using a video scene segmentation algorithm. in, represents the video scene segmentation algorithm, For complete videos of movies and TV series, are the n video clips obtained after algorithm processing; The picture description information is obtained through a multimodal large model: in, represents a multimodal large model, Indicates prompt words, Represents a video clip. To obtain On the screen description information, enter For video or by The resulting image sequence; The information of the people appearing in the scene is obtained through the face recognition algorithm and the pedestrian re-identification algorithm: in, represents the face recognition algorithm and pedestrian re-identification algorithm, Indicates that the match belongs to a video segment As a result, To get in Information about the people appearing on the screen; The lines information is obtained through a speech recognition algorithm: in, represents the speech recognition algorithm, is the one obtained by matching Line information; If the TV series or film has subtitles, the OCR algorithm can be used to obtain the lines. in, Indicates the OCR algorithm.

5. The intelligent shortening method for film and television dramas as claimed in claim 4, characterized in that: In step S1, the plot introduction document is divided into several plot sections according to semantics through a large language model. in, represents a large language model, Indicates the corresponding prompt word, Complete video for TV series The corresponding plot introduction document, The plot is divided into sections.

6. The intelligent shortening method for TV series and movies as claimed in claim 4, characterized in that: Step S2 includes: in, Indicates the corresponding prompt word, Represents a list of plot segments to be selected, for each plot segment or a subset thereof, Represents a large language model from Select the video clip from Matching plot passages; After completing the matching of all video clips, we finally get a set of video clips that match each plot segment, such as: Among them, the collection Representation and plot The set of matching video clips includes all the video clips selected by the large language model as matching the plot paragraphs in this step. Matching video clips .

7. The intelligent shortening method for film and television dramas according to claim 6, characterized in that: The abbreviated script generated in step S3 is: in, Indicates the corresponding prompt word, Represents the abridged script generated by the large language model.

8. The intelligent shortening method for TV series and movies as claimed in claim 7, characterized in that: In step S4, the set of video clips matching the first paragraph in the list of selected plot paragraphs read from step S2 is ,m≤n, Based on the picture description information, character information and dialogue information of the corresponding video clip read in step S1, then: in, Indicates a list of video clips to be selected; Then, using a large language model, Selected from Matching video clips : in, Indicates the corresponding prompt word, Represents the storyline content information in the abridged script.

Images