Movie and television role and scene graph generation method and system based on AIGC

Abstract

The invention discloses a film and television role and scene graph generation method and system based on AIGC, and relates to artificial intelligence. Obtaining script text content input by a user, and extracting role description and scene description; and converting the role description and the scene description into Prompt in a standard format through a large language model, inputting the Prompt into Stable Diffusion, and generating a reference role graph or a reference scene graph. For role consistency, according to an IP-Adapter identity keeping module, inputting a benchmark reference graph and text description to generate different states of the same role. For scene consistency, edge contour information is extracted from the reference scene graph, the contour information and text description are used as input of controlnet, and different weather and illumination states are generated. The role consistency and scene consistency generated by AIGC are improved, different states of the same role in the same scene are efficiently generated, and a reference is provided for role pass generation and scene exploration and selection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating film and television characters and scene graphs based on AIGC. Background Art

[0002] In the process of film and television production, the design of characters and scenes is an important task in the early stage. Existing film and television character and scene design usually relies on professional art teams for creation. First, the characters and scenes are tailored for specific scripts, which is a complicated and time-consuming process; second, the ready-made templates provided for ordinary users often lack personalization and creativity, and it is difficult to meet diverse needs. Existing methods rely on manual operation, which is time-consuming and labor-intensive, and it is difficult to respond quickly to script modifications. Secondly, for ordinary users, if they lack professional artistic skills and experience, it takes a lot of work to create the characters and scenes required by the script.

[0003] In recent years, with the rapid development of AIGC technology, new solutions have been developed for the design of film and television characters and scenes. By using large language models and image generation models, it is possible to parse and extract the text content of the script, and then generate character and scene images that meet the needs of the script.

[0004] How to provide a method and system for generating film and television characters and scene graphs based on AIGC to achieve efficient generation of personalized character and scene images, improve generation quality, and reduce creation costs has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for generating film and television characters and scene graphs based on AIGC in order to address the above-mentioned technical problems existing in the prior art, so as to generate character and scene images that meet the requirements of the script, improve the generation quality, and reduce the creation cost.

[0006] In a first aspect, the present invention provides a method for generating film and television characters and scene graphs based on AIGC, comprising the following steps:

[0007] Step S10, obtaining the script text content input by the user as input data for subsequent image generation, ensuring that the system can receive and process the information provided by the user;

[0008] Step S20, extracting character descriptions and scene descriptions from the script text content input by the user, so as to provide necessary basic data for subsequent generation;

[0009] Step S30: using a large language model, convert the extracted role description and scene description into a prompt in a standard format;

[0010] Step S40, inputting the Prompt in the standard format into the deep learning model Stable Diffusion to generate character and scene images;

[0011] Step S50: Use IP-Adapter and Controlnet to generate controllable generation that keeps identity features consistent in different states from the role and scene reference graph.

[0012] Furthermore, in step S10, the script text content input by the user is obtained, and the user inputs relevant text information of the script into the system through the system interface. The relevant text information of the script includes character description, scene background, emotional atmosphere, etc.

[0013] Furthermore, in step S20, the character description and the scene description are extracted from the script text content input by the user, and the character description and the scene description are extracted from the script text content input by the user through the language big model, and the text description is adjusted and optimized to ensure the accuracy and detail completeness of the Prompt generated later, so as to make it more suitable for image generation; the character description includes descriptions of the appearance features, clothing style, personality characteristics, emotional state, etc. of each character in the script; the scene description includes descriptions of the scene background, environmental atmosphere, weather conditions, lighting effects, etc. in the script.

[0014] Furthermore, in step S30, the extracted role description and scene description are converted into a prompt in a standard format through a large language model. The system standardizes the extracted role description and scene description through a large language model to generate data in a unified format, ensuring an accurate match between the role ID and the scene ID so that the role and scene can be correctly identified and associated in subsequent processing.

[0015] Furthermore, in step S40, the character and scene images are generated, specifically: based on the Prompt optimized in step S30, the standard format Prompt is input into the locally deployed Stable Diffusion, and the corresponding generation model is called to generate images. By calling the optimized Stable Diffusion, the system can efficiently convert the description into high-quality visual content and realize the automatic generation of characters and scenes.

[0016] Further, in step S50, the multi-state consistency is generated, specifically: for role consistency, according to the IP-Adapter identity maintenance module, different states of the same role can be generated by inputting the reference reference image and text description (clothing features, emotional state, etc.). For scene consistency: contour information is extracted from the reference scene graph, and the contour information and text description are used as input of controlnet to generate different weather and light states.

[0017] In a second aspect, the present invention provides a film and television character and scene graph generation system based on AIGC, comprising the following modules:

[0018] The script parsing module is used to obtain the script text content input by the user and extract the character description and scene description from it to ensure that the extracted information can accurately reflect the emotions and situations of the script;

[0019] The prompt standardization module is used to standardize the extracted character descriptions and scene descriptions and generate prompts in a unified format to ensure the precise matching of the character ID and the scene ID, laying the foundation for subsequent image generation;

[0020] The character and scene generation module is used to input the prompt in the standard format into Stable Diffusion to realize the automatic generation of character and scene images, and quickly convert text information into high-quality visual content;

[0021] The consistency generation module uses IP-Adapter and Controlnet to generate controllable generation that keeps identity features consistent in different states from the role and scene reference graphs.

[0022] Furthermore, the script parsing module is specifically:

[0023] Get the script text content input by the user, analyze the text structure, extract the role description and scene information, and prepare for subsequent processing.

[0024] Furthermore, the Prompt standardization module is specifically:

[0025] Perform natural language processing on the extracted role descriptions and scene descriptions to generate prompts in a standard format to improve the generation quality.

[0026] Furthermore, the role and scene generation module is specifically:

[0027] The role and scene generation module specifically includes: inputting the prompt in the standard format into the generation model, and obtaining the image output corresponding to each role and scene through the Stable Diffusion image generation model.

[0028] Furthermore, the consistency generation module is specifically as follows: for character consistency: according to the IP-Adapter identity maintenance module, the base image plus the states of various costumes and the text description of the states of joy, anger, sorrow and happiness can be used as two inputs to generate different states of the same character. For scene consistency: the pre-trained model is used to extract contour information from the base scene image, and the contour information and text description are used as inputs of the controlnet to generate states of different weather and lighting.

[0029] The advantages of the present invention are:

[0030] By parsing the script text and extracting character and scene information, natural language processing is used to generate prompts in a standard format to ensure the consistency of character IDs and scene IDs. Then, personalized character and scene images are generated through the deep learning model Stable Diffusion, and the character's clothing status and emotional expression, as well as the weather and lighting effects of the scene are further fine-tuned. The present invention greatly improves the character consistency and scene consistency generated by AIGC, efficiently generates different states of the same character in the same scene, and provides a reference for the film and television industry to generate character biographies and scout locations. It effectively overcomes the complexity and high cost of traditional design, while being able to meet diverse creative needs, improve the efficiency and quality of character and scene generation, and reduce overall creative costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart of a method for generating film and television characters and scene graphs based on AIGC in the field of artificial intelligence.

[0032] Figure 2 It is a structural schematic diagram of a film and television character and scene graph generation system based on AIGC in the field of artificial intelligence of the present invention.

[0033] Figure 3 The Little Prince benchmark reference image generated for the present embodiment.

[0034] Figure 4 A fox benchmark reference image generated for the present embodiment.

[0035] Figure 5 This is a benchmark reference image under the apple tree generated for the present embodiment.

[0036] Figure 6 This is a reference diagram of the different emotional states of the Little Prince generated by this embodiment.

[0037] Figure 7 A reference image of the Little Prince in the universe generated for this embodiment.

[0038] Figure 8 Reference image of the Little Prince at sunset generated for this embodiment.

[0039] Fig. 9 A picture of an apple tree in the rain generated for the present embodiment.

[0040] Fig.10 This is a composite picture of people and scenes generated by the present embodiment. DETAILED DESCRIPTION

[0041] The technical solution in the embodiment of this application has the following general idea: deep learning and AIGC technology are applied to the automatic generation of film and television characters and scene graphs, overcoming the complexity and high cost of traditional design, and ensuring the consistency of characters and scenes in different scripts and emotional states. This method effectively solves the problem that characters and scenes cannot flexibly adapt to script changes, improves generation efficiency and quality, reduces creation costs, and ultimately provides a more efficient and economical solution for film and television production.

[0042] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a method for generating a film and television character and scene graph based on AIGC of the present invention comprises the following steps:

[0043] Step S10, obtaining the script text content input by the user; the user inputs the relevant text information of the script into the system for subsequent analysis and processing.

[0044] Specifically, the user inputs relevant text information of the script into the system through the system interface. The relevant text information of the script includes dialogues and actions of characters in the script story.

[0045] In this embodiment: the user inputs a dialogue segment between the little prince and the fox in "The Little Prince".

[0046] Step S20, extracting character descriptions and scene descriptions from the script text content input by the user; inputting the character descriptions and scene descriptions input by the user into the large language model for processing, adjusting, enriching and optimizing the text description, and ensuring that the generated results can effectively reflect the characteristics of the characters and scenes in the script.

[0047] Specifically: through the large language model, extract the role description and scene description from the script text content input by the user, and adjust and optimize the text description, including the role description, scene background, and emotional atmosphere, as the basic data for subsequent parsing, processing, and generation of prompts and images. For example, after the user enters the script, the large language model extracts the role description as the age, appearance, background information, personality characteristics, and other information of a certain character. Ensure the accuracy and completeness of the details of the later generated prompts, making them more suitable for image generation. The role description includes descriptions of the appearance characteristics, clothing style, personality characteristics, emotional state, etc. of each character in the script; the scene description includes descriptions of the scene background, environmental atmosphere, weather conditions, lighting effects, etc. in the script.

[0048] In this example, the large language model helps the user extract the characters: the little prince, the fox. The scene: under the apple tree. And refine the description of each character one by one. For example: "The little prince is characterized as a young boy, possibly aged between 7 to 10 years old. He has a slim, delicate frame thatembodies his innocence and purity. His facial expressions often carry a sense of curiosity and wonderment, reflecting his constant desire to learn about the world around him. He is almost always seen in his prince attire, including a green jumpsuit with a yellow scarf tied around his neck, enhancing his royal and regal appearance. The most distinctive feature of the little prince is his hair, described as the color of gold, which captures the optimism and radiance in his personality."

[0049] Step S30: Convert the role description and the scene description into a prompt in a standard format through a large language model;

[0050] Specifically, the system generates relevant rules, examples, and result formats for the large language model input prompt, so that it can convert the extracted character descriptions and scene descriptions into standard format prompts. For example, add a suffix after the positive prompt word to ensure the best generation quality, translate the above character description into English, and append the commonly used positive prompt words to the end.

[0051] Here is an example of a commonly used prompt word:

[0052] (masterpiece:1,2),bestquality,masterpiece,highres,original,extremelydetailedwallpaper,perfec- tlighting,(extremelydetailedCG:1.2),drawing,paintbrush.

[0053] Step S40: input the Prompt in the standard format into Stable Diffusion to generate corresponding character and scene images; use the deep learning algorithm to quickly convert the text information into high-quality visual content, thereby realizing automatic image generation.

[0054] Specifically, based on the optimized Prompt, the system automatically generates film and television characters and scene graphs by calling the locally deployed Stable Diffusion. According to different character characteristics or scene settings, the system can generate image effects of various styles. For example, the character graph generation process can automatically generate character images that meet the description by providing a detailed character description (including appearance, clothing, emotions, etc.). The scene graph generation is based on the scene background information and key details provided by the user. In this embodiment, the reference images of the little prince, the fox, and the apple tree are generated ( Figures 3-5 ).

[0055] Step S50: Use IP-Adapter and Controlnet to generate controllable images that keep identity features consistent in different states from the role and scene reference images. IP-Adapter and Controlnet are plug-and-play fine-tuning modules that can ensure the consistency of generated images.

[0056] For the consistency of the role, according to the IP-Adapter identity maintenance module, the reference image and text description (clothing features, emotional state, etc.) can be input to generate different states of the same role. In this embodiment, according to the reference image, pictures with different backgrounds that maintain the facial features of the little prince can be generated, and pictures of the emotional states of happiness, anger, sorrow, and joy ( Figure 6). You can generate pictures of the Little Prince in different scenes and weather conditions ( Figure 7 and 8 ).

[0057] For scene consistency: a pre-trained model is used to extract contour information from the reference scene graph, and the contour information and text description are used as inputs of the controlnet to generate different weather and lighting conditions. In this embodiment, based on the reference graph under the apple tree, pictures of apple trees under different weather and lighting conditions can be generated, such as pictures of apple trees in the rain ( Fig. 9 ).

[0058] Finally, the system not only realizes the diversified generation from character graphs to scene graphs, but also ensures the diversity and accuracy of image generation, and meets the image requirements of different links in film and television production ( Fig.10 ).

[0059] The present invention provides a film and television character and scene graph generation system based on AIGC, comprising the following modules:

[0060] The script parsing module is used to obtain the script text content input by the user and extract the character description and scene description from it to ensure that the extracted information can accurately reflect the emotions and situations of the script;

[0061] The prompt standardization module is used to standardize the extracted character descriptions and scene descriptions and generate prompts in a unified format to ensure the precise matching of the character ID and the scene ID, laying the foundation for subsequent image generation;

[0062] The character and scene generation module is used to input the prompt in the standard format into Stable Diffusion to realize the automatic generation of character and scene images and quickly convert the text information into a reference image;

[0063] The consistency generation module uses IP-Adapter and Controlnet to generate controllable generation that keeps identity features consistent in different states from the role and scene reference graphs.

[0064] Combined with the text description entered by the user, the system automatically generates film and television characters and scene graphs by calling the locally deployed Stable Diffusion. According to different character characteristics or scene settings, the system can generate image effects with various styles. For example, the character graph generation process can automatically generate a character image that meets the description by providing a detailed character description (including appearance, clothing, emotions, etc.). The scene graph generation is based on the scene background information and key details provided by the user.

[0065] In addition, the system integrates a fine-tuning function to further enhance the image's detail performance and artistic effect based on the initial image generation. By fine-tuning the generated results, users can adjust the detail elements in the image to meet specific needs.

[0066] Ultimately, the system not only achieves diversified generation from character graphs to scene graphs, but also ensures the diversity and accuracy of image generation through efficient deep learning technology, meeting the image requirements of different links in film and television production.

[0067] The script parsing module is specifically:

[0068] The script text content entered by the user is obtained, and character descriptions and scene descriptions are extracted from it through natural language processing technology. This module ensures that the extracted information can accurately reflect the emotions and situations of the script, so as to provide necessary data support for subsequent image generation.

[0069] The Prompt standardization module is specifically:

[0070] The extracted character descriptions and scene descriptions are standardized, including removing redundant information, unifying the format, and labeling the character ID and scene ID. Through these steps, a standardized prompt is generated, laying a solid foundation for subsequent image generation.

[0071] The role and scene generation module is specifically:

[0072] Input the prompt in standard format into the locally deployed Stable Diffusion API, call the corresponding generation model, and realize the automatic generation of character and scene images. This module can quickly convert text information into high-quality visual content to meet user needs.

[0073] Specifically: For role consistency, according to the IP-Adapter identity preservation module, different states of the same role can be generated by inputting the reference reference image and text description (clothing features, emotional state, etc.). For scene consistency: contour information is extracted from the reference scene graph, and the contour information and text description are used as input to the controlnet to generate different weather and lighting states.

[0074] In summary, the advantages of the present invention are:

[0075] By parsing the script text input by the user, accurate role descriptions and scene descriptions are extracted, and standardized processing is used to generate prompts in a unified format, thereby ensuring the precise match of role IDs and scene IDs. Next, the prompts in the standard format are input into Stable Diffusion to achieve rapid generation of role and scene images, greatly improving the efficiency of conversion from text information to high-quality visual content. At the same time, the consistency generation module uses IP-Adapter and Controlnet to generate controllable generation that maintains consistent identity features under different states from role and scene benchmark reference images. This series of processes effectively overcomes the complexity and high cost problems in traditional film and television production, and ultimately greatly improves the efficiency and quality of film and television role and scene graph generation, reducing overall production costs.

[0076] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for generating film and television characters and scene graphs based on AIGC, characterized in that: The steps include: Step S10, obtaining the script text content input by the user as input data for subsequent image generation, ensuring that the system can receive and process the information provided by the user; Step S20, extracting character descriptions and scene descriptions from the script text content input by the user, so as to provide necessary basic data for subsequent generation; Step S30: using a large language model, convert the extracted role description and scene description into a prompt in a standard format; Step S40, inputting the Prompt in the standard format into the deep learning model Stable Diffusion to generate character and scene images; Step S50: Use IP-Adapter and Controlnet to generate controllable generation that keeps identity features consistent in different states from the role and scene reference graph.

2. The method for generating film and television characters and scene graphs based on AIGC as claimed in claim 1, characterized in that: In step S10, the script text content input by the user is obtained. The user inputs relevant text information of the script into the system through the system interface. The relevant text information of the script includes character description, scene background, and emotional atmosphere.

3. The method for generating film and television characters and scene graphs based on AIGC as claimed in claim 1, characterized in that: In step S20, the character description and scene description are extracted from the script text content input by the user, and the character description and scene description are extracted from the script text content input by the user through a large language model, and the text description is adjusted and optimized to ensure the accuracy and detail completeness of the Prompt generated later, so that it is more suitable for image generation; the character description includes the appearance features, clothing style, personality characteristics, and emotional state description of each character in the script; the scene description includes the scene background, environmental atmosphere, weather conditions, and lighting effects description in the script.

4. The method for generating film and television characters and scene graphs based on AIGC as claimed in claim 1, characterized in that: The step S30 specifically includes: standardizing the extracted role description and scene description to generate prompts in a unified format.

5. The method for generating film and television characters and scene graphs based on AIGC as claimed in claim 1, characterized in that: In step S40, the character and scene images are generated, specifically: based on the Prompt optimized in step S30, the Prompt in a standard format is input into the locally deployed Stable Diffusion, and the corresponding generation model is called to generate the image.

6. The method for generating film and television characters and scene graphs based on AIGC as claimed in claim 1, characterized in that: In step S50, multi-state consistency is generated. Specifically, for role consistency, different states of the same role are generated by inputting a reference reference graph and a text description according to the IP-Adapter identity maintenance module; for scene consistency, contour information is extracted from the reference scene graph, and the contour information and text description are used as inputs of the controlnet to generate states of different weather and lighting.

7. A film and television character and scene graph generation system based on AIGC, characterized by: Includes the following modules: The script parsing module is used to obtain the script text content input by the user and extract the character description and scene description from it to ensure that the extracted information can accurately reflect the emotions and situations of the script; The prompt standardization module is used to standardize the extracted character descriptions and scene descriptions and generate prompts in a unified format to ensure the precise matching of the character ID and the scene ID, laying the foundation for subsequent image generation; The character and scene generation module is used to input the prompt in the standard format into Stable Diffusion to realize the automatic generation of character and scene images and quickly convert the text information into a reference image; The consistency generation module uses IP-Adapter and Controlnet to generate controllable generation that keeps identity features consistent in different states from the role and scene reference graphs.

8. The AIGC-based film and television character and scene graph generation system as claimed in claim 7, characterized in that: The script parsing module specifically obtains the script text content input by the user, analyzes the text structure, and extracts the role description and scene information therein for subsequent processing.

9. The AIGC-based film and television character and scene graph generation system as claimed in claim 7, characterized in that: The prompt standardization module specifically performs natural language processing on the extracted role description and scene description to generate prompts in a standard format to improve the generation quality; The role and scene generation module is specifically as follows: a prompt in a standard format is input into the generation model, and the image output corresponding to each role and scene is obtained through the StableDiffusion image generation model.

10. The AIGC-based film and television character and scene graph generation system as claimed in claim 7, characterized in that: The consistency generation module is specifically as follows: for role consistency, according to the IP-Adapter identity maintenance module, different states of the same role are generated by inputting a reference reference graph and a text description; for scene consistency: contour information is extracted from the reference scene graph, and the contour information and text description are used as inputs of the controlnet to generate states of different weather and lighting.

Images