Automatic second creation method and system based on reconstructed story script

By building a multi-modal annotation library and using large language models to generate scripts, combining visual, audio and text annotation information for material selection and post-processing, the technical problem that the existing technology is difficult to realize secondary creation of film and television dramas is solved, and automated Video material retrieval and splicing reduce the difficulty of creation and improve the adaptability of generated content.

CN118916802APending Publication Date: 2024-11-08WUHAN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410851527.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing video analysis technology is difficult to meet the needs of secondary creation of film and television dramas, especially when the script text is created, it is impossible to realize automated video material retrieval and splicing.

Method used

By building a multimodal annotation library, using a large language model to generate scripts, and combining visual, audio and text annotation information for material selection and post-processing, one-to-one mapping from script lines to shots and audio synchronous generation.

Benefits of technology

The second creation of film and television cameras has been realized, which has reduced the difficulty of creation, improved the fit and quality of the generated content, and can automatically complete the production of second creation content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916802A_ABST
    Figure CN118916802A_ABST
Patent Text Reader

Abstract

The invention provides an automatic second creation method and system based on a reconstructed story script. According to the method, firstly, multi-modal information labeling is carried out on movie and television play shots, visual labeling comprises figure identities, figure emotions, facial orientations and garment slices, audio labeling constructs an identity audio library through voiceprint recognition, text labeling fits the relation between line duration and length, and line length distribution is counted. And then entering a creation process, firstly performing script creation with the help of a large language model, secondly screening the materials based on lines under the proposed two-stage material selection algorithm, and finally performing two post-processing of audio generation and lip shape synchronization on the materials. According to the automatic second creation method based on the reconstructed story script, movie and television play characteristics are fully structured, the creation threshold of second creation works is greatly reduced, and the whole process can be automated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a video analysis and processing problem in the field of computer vision, and belongs to an automated secondary creation method and system based on reconstructing a story script. Background Art

[0002] Secondary creation (i.e., re-editing videos from stock footage, i.e., creators use materials in the database to collage clips with reorganized lens language, such as intercepting key words of characters and recombining them into new stories) is currently a very popular type of video content on various video platforms, and has become an important tool for producers to promote popularity. The development of secondary creation automation methods has lowered the threshold for the creation of high-quality secondary content, which is conducive to the promotion of film and television dramas and can promote the development of the film and television industry.

[0003] The core difficulty of automated secondary creation lies in the retrieval of video materials:

[0004] (1) AI content generation

[0005] Visual content generation is the hottest topic in the field of artificial intelligence in recent years. With the continuous development of algorithms and computing power, the quality of generated content has shown amazing capabilities in terms of realism and creative diversity. The relevant technologies are currently mainly divided into two categories: generative adversarial networks and diffusion models.

[0006] Generative adversarial networks are mainly composed of two parts: the generator and the discriminator. The generator generates fake data from random noise (usually a multidimensional normal distribution or uniform distribution), and the discriminator is used to distinguish between real data and generated data. The training process includes alternating training of the generator and the discriminator. First, the generator generates fake samples from random noise; then, the discriminator tries to distinguish between real data and fake data. The loss of the discriminator is calculated by the accuracy of its judgment, and its parameters are updated through back propagation. Next, the generator tries to deceive the discriminator into thinking that the fake data is real. The loss of the generator is calculated based on the discriminator's judgment of the fake samples, and the parameters of the generator are updated through back propagation. This process is repeated until the samples generated by the generator are so realistic that the discriminator cannot distinguish them.

[0007] The diffusion model is trained through a series of processes of gradually adding noise and reverse denoising to generate data. Its workflow is as follows: forward diffusion process: starting from the original data, gradually adding Gaussian noise to generate multiple noisy images until it eventually becomes pure noise. This process defines the transition from data distribution to noise distribution; reverse generation process: by training a neural network model, learn each step of denoising operation from the noisy image to the original image. The model predicts the denoised version of the current noisy image at each step and gradually removes the noise; sampling process: starting from pure noise, applying the trained denoising model, gradually removing the noise, and reversely restoring the data.

[0008] (2) Video Retrieval

[0009] The task of video retrieval is mainly to complete the matching from text to video, which mainly includes semantic retrieval and descriptive text retrieval. Semantic retrieval is mainly through the composite retrieval of multiple simple semantics, such as people, places, and actions. The relevant methods mainly focus on completing the retrieval of a single semantic first and then fusing the retrieval results of the single semantic. Descriptive text retrieval is to find the video clip that best matches the text through a whole sentence of text. The main method is to encode the corresponding features through the encoders of the two modal information of text and vision, and then calculate the similarity, and take the video with the highest similarity as the retrieval result.

[0010] However, both of the above methods are based on the subsequent generation or retrieval of descriptive texts, which is far from the characteristics of secondary creation based on the script text, and cannot meet the requirements of secondary creation. Summary of the invention

[0011] The purpose of the present invention is to attempt to deconstruct and annotate the multimodal content in film and television drama shots, and to perform automated secondary creation in three stages: self-writing, self-directing, and self-acting based on the annotated labels.

[0012] The technical solution of the present invention is an automated secondary creation method based on reconstructing a story script, comprising the following steps:

[0013] Step 1: Build a material library, including a visual annotation library, an audio annotation library, and a text annotation library;

[0014] Step 2: Reconstruct script generation. The creator inputs the storyline to be created, uses the pre-designed prompt word format to guide the large language model to generate the corresponding script, and uses the text annotation data to limit the length of the generated lines. Finally, the duration of each line is predicted. The script includes characters, locations, lines and corresponding emotions.

[0015] Step 3: Based on the selection of script lines, for each line of the script generated in step 2, the corresponding matching shot is selected according to the annotation information obtained in step 1, so as to achieve a one-to-one mapping from lines to shots;

[0016] Step 4, post-processing the material selected in step 3, including audio generation and lip synchronization based on the lines;

[0017] Step 5, stitch all the materials obtained in step 4 into a film.

[0018] Furthermore, in step 1, for visual annotation: SCRFD and ArcFace are used to perform face detection and face recognition frame by frame on the video to obtain the character identity c, and key point annotation and emotion recognition are used on the basis of the face frame to obtain the facial orientation o and the character emotion e. Then, Faster RCNN is used for tracking detection, and the position information of the tracking frame and the face frame is combined to obtain the slice image u of the character's clothing. If a series is divided into n shots, the visual annotation is expressed as V = {v1,v2,…,v n}; where v n Represents the annotation information of the nth video.

[0019] Furthermore, in step 1, for audio annotation: first use the UVR5 tool to denoise the original audio, and then combine the identity information to obtain the identity audio library A through multiple rounds of voiceprint recognition and filtering outliers. idt ={a1,a2,…,a m}, where a m Indicates the mth audio in the audio library.

[0020] Furthermore, in step 1, for text annotation: firstly, the duration of the lines is fitted to obtain a fitting function, the formula of which is as follows:

[0021]

[0022] Where i is the character identity, j is the character emotion, f is the line length, α and β are fitting function parameters, and the length distribution of the lines is statistically analyzed to obtain the distribution D(i, j), which means the statistical value of the line length distribution for each character i under emotion j.

[0023] Furthermore, the specific implementation of step 3 includes the following sub-steps:

[0024] Step 3.1, hard constraints, use the predicted duration, character identity and location information to filter the available materials and obtain the candidate material library corresponding to each line

[0025] Step 3.2, soft sorting, uses facial orientation and clothing color similarity to sort the availability of the candidate library. For facial orientation, based on the selected shots, the characters of the same identity are prioritized according to the three groups of same orientation, middle orientation, and opposite orientation, while the characters of different identities are prioritized according to the three groups of opposite orientation, middle orientation, and same orientation. For clothing color similarity, a similarity function is used to evaluate the similarity, and the characters are sorted from large to small according to the similarity score in each facial orientation group to obtain the sorted library.

[0026] Further, the alternative material library The calculation formula is as follows:

[0027]

[0028] )

[0029] The above formula means to remove all shots that do not contain character c i , the duration is less than the predicted duration And the shots that are not in L, is an indicator function, which takes the value 1 when the condition is true and 0 when it is false. j is the label of the jth shot, V[·] is the corresponding field using the shot label, t is the duration field of the video, and l is the location field of the video.

[0030] Furthermore, the similarity function D in step 3.2 col The calculation formula is as follows:

[0031]

[0032] Where λ1, λ2 are the color vectors of the lab space of clothing 1 and clothing 2, i indicates that there are three components in total, where l is the lightness, a represents the component from green to red, and b represents the component from blue to yellow.

[0033] Furthermore, in step 4, for audio generation, GPTSoVits is used for generation. First, the pre-trained GPTSoVits model is fine-tuned according to the identity audio library obtained in step 1, and then the lines in the reconstructed script are input to obtain audio a gen ,Then the generated audio and the corresponding lens are input into the Wav2Lip model for lip synchronization to obtain the final video.

[0034] The present invention also provides an automated secondary creation system based on reconstructing story scripts, comprising the following modules:

[0035] The material library construction module is used to construct the material library, including the visual annotation library, the audio annotation library and the text annotation library;

[0036] The script generation module is used to reconstruct script generation. The creator inputs the storyline to be created, uses the pre-designed prompt word format to guide the large language model to generate the corresponding script, and uses the text annotation data to limit the length of the generated lines. Finally, the duration of each line is predicted. The script includes characters, locations, lines and corresponding emotions.

[0037] The material selection module is used to select materials based on the script lines. For each line of the script generated by the script generation module, the corresponding matching lens is selected according to the annotation information obtained by the material library construction module to achieve a one-to-one mapping from the lines to the lens;

[0038] A post-processing module, used for post-processing the material selected by the material selection module, including audio generation and lip synchronization based on the lines;

[0039] The finished product splicing module is used to splice all the materials obtained by the post-processing module into a film.

[0040] The present invention can automatically complete the production of secondary creative content, and has the following advantages: 1. Detailed annotation of multimodal content of film and television drama shots. Multimodal information tags can help us better filter materials in the subsequent selection link; 2. Introducing a large language model for script creation. Script writing often requires creators to receive some professional training, and the addition of a large language model greatly reduces the difficulty of this link; 3. A two-stage material selection method is designed for the self-directed stage, so that the selected material is more in line with the lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart for the implementation of the present invention;

[0042] Figure 2 This is a structural diagram of the lens annotation in the implementation of the present invention.

[0043] Figure 3 It is a schematic diagram of the overall method in the implementation of the present invention. Specific implementation methods

[0045] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0046] The present invention provides an automated secondary creation method based on reconstructing story scripts. It is a method proposed for the field of secondary creation of film and television dramas that can automatically complete script writing, material retrieval and post-processing. First, after obtaining a material library by annotating the multimodal information of the shots, a large language model is used to assist the group creators in completing the script creation. Finally, the materials in the material library are retrieved based on the script lines. Finally, the retrieved materials are post-processed to obtain the final film. Computer software can be used in the specific implementation to support the process operation. See Figure 1 The process of the embodiment includes the following steps:

[0047] Step 1: Build a library. For visual annotation: Use SCRFD and ArcFace to perform face detection and face recognition frame by frame on the video to obtain the person identity c, and use key point annotation and emotion recognition based on the face frame to obtain the facial orientation o and the person emotion e. Then use Faster RCNN for tracking detection, and combine the position information of the tracking frame and the face frame to obtain the slice image u of the person's clothing. If a drama is divided into n shots, then this visual annotation can be expressed as V = {v1,v2,…,v n}; where v n Represents the annotation information of the nth video. Other face detection and tracking algorithms can also be used in specific operations. For audio annotation: First, use the UVR5 tool to denoise the original audio, and then combine the identity information to obtain the identity audio library A through multiple rounds of voiceprint recognition and filtering outliers. idt ={a1,a2,…,a m}; where a m It represents the mth audio in the audio library. For text annotation, the duration of the lines is fitted to obtain the fitting function, and its formula is as follows:

[0048]

[0049] Where i is the character identity, j is the character emotion, f is the line length, α and β are fitting function parameters, and the line length distribution is statistically analyzed to obtain distribution D(i, j), which means the statistical value of the line length distribution for each character i under emotion j.

[0050] Figure 2 This is the architecture diagram for shot annotation. After inputting a shot video, it goes through modules such as face recognition, emotion recognition, and facial orientation recognition to obtain the visual annotation v of the shot. The script is input to perform line length statistics and duration fitting. The input audio is filtered and voiceprint recognized to obtain the identity audio library of each character.

[0051] Step 2: Reconstruct script generation. The creator inputs the plot outline to be created, and the background will automatically use the pre-designed prompt word format to guide the large language model to generate the corresponding script, which includes characters, locations, lines and corresponding emotions, and use text annotation data to limit the length of the generated lines. Finally, the duration t corresponding to each line is predicted. pre ;

[0052] Step 3, based on the selection of script lines, for each line of the script generated in step 2, according to the annotation information obtained in step 1, select the corresponding matching shot to achieve a one-to-one mapping from lines to shots, which specifically includes the following sub-steps:

[0053] Step 3.1, hard constraint, use prediction time t pre , character identity c and location information L are used to filter the available materials, and the candidate material library corresponding to each line is obtained. The formula is as follows:

[0054]

[0055]

[0056] The above formula means to remove all shots that do not contain character c i , the duration is less than the predicted duration And the shots that are not in L, is an indicator function, which takes the value 1 when the condition is true and 0 when it is false. j is the annotation of the jth shot, and V[·] is the corresponding field of the shot annotation.

[0057] Step 3.2, soft sorting, uses facial orientation and clothing color similarity to sort the availability of the candidate material library. For facial orientation, based on the selected shots, we prioritize the characters of the same identity according to the same orientation, middle orientation, and opposite orientation, while the characters of different identities are prioritized according to the opposite orientation, middle orientation, and the same orientation. For clothing color similarity, we use the function D col (·) is used to evaluate the similarity, and the formula is as follows:

[0058]

[0059] Where λ1 and λ2 are color vectors in the lab space, i indicates that it has three components, where l is lightness, a represents the component from green to red, and b represents the component from blue to yellow. At the same time, in each facial orientation group, the similarity scores are sorted from large to small to obtain a sorted material library.

[0060] Step 4: Post-process the selected material in step 3, including audio generation and lip sync based on the lines. We use GPTSoVits to generate the audio. First, we fine-tune the pre-trained GPTSoVits model with the identity audio library obtained in step 1, and then input the lines in the reconstructed script to obtain audio a gen , and then the generated audio is compared with the corresponding shot V i Input the Wav2Lip model for lip sync to get the final video.

[0061] Figure 3 This is the architecture diagram of the overall method. First, creators can input the plot outline of the story they want to create, generate the corresponding script through the large language model, and search the material library based on the length of the lines, character identity, location and emotion. The available materials retrieved are sorted from large to small according to the scores of the two indicators of facial orientation and clothing color similarity, and the creators can interactively select them. Finally, audio is generated according to the script lines and lip-synced with the corresponding video to obtain the final video.

[0062] Step 5, stitch all the materials obtained in step 4 into a film.

[0063] On the other hand, the embodiment of the present invention also provides an automated secondary creation system based on reconstructing a story script, including the following modules:

[0064] The material library construction module is used to construct the material library, including the visual annotation library, the audio annotation library and the text annotation library;

[0065] The script generation module is used to reconstruct script generation. The creator inputs the storyline to be created, uses the pre-designed prompt word format to guide the large language model to generate the corresponding script, and uses the text annotation data to limit the length of the generated lines. Finally, the duration of each line is predicted. The script includes characters, locations, lines and corresponding emotions.

[0066] The material selection module is used to select materials based on the script lines. For each line of the script generated by the script generation module, the corresponding matching lens is selected according to the annotation information obtained by the material library construction module to achieve a one-to-one mapping from the lines to the lens;

[0067] A post-processing module, used for post-processing the material selected by the material selection module, including audio generation and lip synchronization based on the lines;

[0068] The finished product splicing module is used to splice all the materials obtained by the post-processing module into a film.

[0069] The specific implementation of each module refers to the steps of the method flow, which will not be described in detail in the present invention.

[0070] The specific embodiments described herein are merely examples of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. An automated secondary creation method based on reconstructing story scripts, characterized in that: The following steps are involved: Step 1: Build a material library, including a visual annotation library, an audio annotation library, and a text annotation library; Step 2: Reconstruct script generation. The creator inputs the storyline to be created, uses the pre-designed prompt word format to guide the large language model to generate the corresponding script, and uses the text annotation data to limit the length of the generated lines. Finally, the duration of each line is predicted. The script includes characters, locations, lines and corresponding emotions. Step 3, based on the selection of script lines, for each line of the script generated in step 2, according to the annotation information obtained in step 1, select the corresponding matching shot to achieve a one-to-one mapping from lines to shots; Step 4, post-processing the material selected in step 3, including audio generation and lip synchronization based on the lines; Step 5, stitch all the materials obtained in step 4 together to form a film.

2. The automated secondary creation method based on reconstructing story scripts according to claim 1, characterized in that: In step 1, for visual annotation: SCRFD and ArcFace are used to perform face detection and face recognition frame by frame on the video to obtain the person identity c, and key point annotation and emotion recognition are used on the basis of the face frame to obtain the facial orientation o and the person emotion e. Then, Faster RCNN is used for tracking detection, and the position information of the tracking frame and the face frame is combined to obtain the slice image u of the person's clothing. If a drama is segmented into n shots, the visual annotation is expressed as V = {v1,v2,…,v n }; where v n Represents the annotation information of the nth video.

3. The automated secondary creation method based on reconstructing story scripts according to claim 1, characterized in that: In step 1, for audio annotation: first use the UVR5 tool to denoise the original audio, then combine the identity information to obtain the identity audio library through multiple rounds of voiceprint recognition to filter outliers A idt ={a1,a2,…,a m }, where a m Indicates the mth audio in the audio library.

4. The automated secondary creation method based on reconstructing story scripts according to claim 1, characterized in that: In step 1, for text annotation: first, the duration of the lines is fitted to obtain the fitting function, the formula of which is as follows: Where i is the character identity, j is the character emotion, f is the line length, α and β are fitting function parameters, and the length distribution of the lines is statistically analyzed to obtain the distribution D(i, j), which means the statistical value of the line length distribution for each character i under emotion j.

5. The automated secondary creation method based on reconstructing story scripts according to claim 1, characterized in that: The specific implementation of step 3 includes the following sub-steps: Step 3.1, hard constraints, use the predicted duration, character identity and location information to filter the available materials and obtain the candidate material library corresponding to each line Step 3.2, soft sorting, uses facial orientation and clothing color similarity to sort the availability of the candidate material library. For facial orientation, based on the selected shots, the characters of the same identity are prioritized according to the three groups of same orientation, middle orientation, and opposite orientation, while the characters of different identities are prioritized according to the three groups of opposite orientation, middle orientation, and same orientation; For clothing color similarity, a similarity function is used to evaluate the similarity, and the facial orientation groups are sorted from large to small according to the similarity scores to obtain a sorted material library.

6. The automated secondary creation method based on reconstructing story scripts according to claim 5, characterized in that: Alternative material library The calculation formula is as follows: The above formula means to remove all shots that do not contain character c i , the duration is less than the predicted duration And the shots that are not in L, is an indicator function, which takes the value 1 when the condition is true and 0 when it is false. j is the label of the jth shot, V[·] is the corresponding field using the shot label, t is the duration field of the video, and l is the location field of the video.

7. The automated secondary creation method based on reconstructing story scripts according to claim 5, characterized in that: Similarity function D in step 3.2 col The calculation formula is as follows: Where λ1, λ2 are the color vectors of the lab space of clothing 1 and clothing 2, i indicates that there are three components in total, where l is the lightness, a represents the component from green to red, and b represents the component from blue to yellow.

8. The automated secondary creation method based on reconstructing story scripts according to claim 3, characterized in that: In step 4, for audio generation, GPTSoVits is used for generation. First, the pre-trained GPTSoVits model is fine-tuned according to the identity audio library obtained in step 1, and then the lines in the reconstructed script are input to obtain audio a gen ,Then the generated audio and the corresponding lens are input into the Wav2Lip model for lip synchronization to obtain the final video.

9. The automated secondary creation system based on reconstructed story scripts is characterized by: Includes the following modules: The material library construction module is used to construct the material library, including the visual annotation library, the audio annotation library and the text annotation library; The script generation module is used to reconstruct script generation. The creator inputs the storyline to be created, uses the pre-designed prompt word format to guide the large language model to generate the corresponding script, and uses the text annotation data to limit the length of the generated lines. Finally, the duration of each line is predicted. The script includes characters, locations, lines and corresponding emotions. The material selection module is used to select materials based on the script lines. For each line of the script generated by the script generation module, the corresponding matching lens is selected according to the annotation information obtained by the material library construction module to achieve a one-to-one mapping from the lines to the lens; A post-processing module, used for post-processing the material selected by the material selection module, including audio generation and lip synchronization based on the lines; The finished product splicing module is used to splice all the materials obtained by the post-processing module into a film.

Citation Information

Patent Citations

  • Video generation method and device

    CN111711855A

  • Intelligent cartoon generation system and method

    CN113505212A

  • Video translation method fused with lip language generation

    CN117496993A

  • Artificial intelligence-based script creation method, apparatus and device, and chip

    CN117521628A

  • Large language model-based script content generation method and device

    CN117992568A