A movie generation method and apparatus

Abstract

A method and apparatus for movie generation. The method includes obtaining a script of a movie; performing a natural language processing procedure (NLP) on the script to generate an action list, the action list including a plurality of actions with attributes, the attributes of each action including a subject, a predicate, and a location of the action; rendering three-dimensional (3D) data in a 3D scene of the movie according to the action list, the 3D data reflecting the action performed by the subject at the location in the corresponding 3D scene; determining a camera sequence of cameras for shooting two-dimensional (2D) frames in the 3D scene by performing cinematography optimization on the movie; generating a 2D video of the movie by combining 2D frames shot by the cameras of the determined camera sequence based on the 2D frames.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and more specifically, to a method and apparatus for generating movies using a computing device. Background Technology

[0002] In the traditional film industry, screenwriting and filmmaking are completely separate processes. As Rabiger et al. emphasize in "Directing Film Techniques and Aesthetics," "The writer is the writer, not the director." In other words, it states, "The screenwriter hands the script to the director, leaving room for other creative talents in the team (director, actors, cinematographer, art director, editor, etc.) to do their work." Due to historical and technological reasons, the writing and filmmaking processes are separated in the film industry, and there are many differences between them.

[0003] Writing and filmmaking are two separate processes. It is entirely feasible for a well-educated person to write a story or a script (if following a certain format), a task virtually impossible in filmmaking. With the recent advancements in Artificial Intelligence (AI) technology, some aspects of the filmmaking process can be accomplished by computers. However, few attempts have been made to combine the writing and filmmaking processes. The methods and systems disclosed in this invention aim to address one or more of the aforementioned problems, as well as others. Summary of the Invention

[0004] One aspect of the present invention provides a method for implementing Write-A-Movie technology. The method is a way of generating a movie, comprising: obtaining a movie script; performing Natural Language Processing (NLP) on the script to generate an action list, the action list including multiple actions with attributes, each action's attributes including a subject, a predicate, and the position of the action; rendering 3D data in a three-dimensional (3D) scene of the movie based on the action list, the 3D data reflecting the actions performed by the subject at the corresponding position in the 3D scene; determining a camera sequence for shooting two-dimensional (2D) frames in the 3D scene by optimizing the movie's cinematography; shooting 2D frames based on the determined camera sequence; and generating a 2D video of the movie by combining the 2D frames.

[0005] One aspect of the present invention provides an apparatus for implementing filmmaking techniques. The apparatus includes a memory and a processor coupled to the memory. The processor is configured to perform: obtaining a script for a film; generating an action list based on the script by performing NLP on the script, the action list including multiple actions with attributes, each action's attributes including a subject, a predicate, and the position of the action; rendering 3D data in a 3D scene of the film based on the action list, the 3D data reflecting the actions performed by the subject at the corresponding position in the 3D scene; determining a camera sequence for shooting 2D frames in the 3D scene by performing cinematographic optimization on the film; shooting 2D frames based on the determined camera sequence; and generating a 2D video of the film by combining the 2D frames.

[0006] Optionally, the processor is further configured to execute:

[0007] Obtain a plan view of the 3D scene of the movie, as well as animation resources of the characters and objects in the movie; and

[0008] The 3D data is rendered in the 3D scene based on the plan view and the animation resources.

[0009] Optionally, the plan includes marked location indicators; and

[0010] The method further includes: identifying phrases in the script by performing the natural language processing procedure to determine the location of an action, the phrases corresponding to a marked location indicator.

[0011] Optionally, obtaining the film's script includes:

[0012] Obtain a script editing tool that allows users to add and modify multiple elements of the script; and

[0013] Obtain the script generated by the script editing tool.

[0014] Optionally, the natural language processing process includes:

[0015] The sentences in the script are converted into a series of ordered actions.

[0016] Optionally, the processor is further configured to execute:

[0017] The action browsing interface displays the attributes of the action and options for modifying the attributes of the action. The modifiable attributes of the action include at least one of predicate, position, and duration.

[0018] The modified attributes of the action are obtained based on the user's input on the action browsing interface; and

[0019] A modified action list is generated based on the modified attributes, and the 3D data is rendered based on the modified action list.

[0020] Optionally, the processor is further configured to execute:

[0021] Receive a first user selection, where the first user selection is the location of the action entered in the action browsing interface;

[0022] In response to the first user selection, a floor plan corresponding to the action is displayed;

[0023] A second user selection is obtained at the designated location of the action described in the plan view; and

[0024] Update the position attribute of the action based on the second user's selection.

[0025] Optionally, the processor is further configured to execute:

[0026] Based on the planar view of the three-dimensional scene, the actions corresponding to the three-dimensional scene, and the action sequence corresponding to the three-dimensional scene, a movement trajectory diagram of the scene is displayed, which includes the movement trajectory and direction of movement of the characters or objects in the scene.

[0027] Optionally, the processor is further configured to execute:

[0028] Play the generated two-dimensional video;

[0029] After playing the generated two-dimensional video, the script editing tool is displayed so that the user can modify elements in the script; and

[0030] An updated 2D video is generated based on the revised script.

[0031] Optionally, the processor is further configured to execute:

[0032] Play the generated two-dimensional video;

[0033] After playing the generated two-dimensional video, the action browsing interface is displayed so that the user can modify the action list; and

[0034] An updated 2D video is generated based on the modified action list.

[0035] Another aspect of the present invention provides an apparatus for generating video in a three-dimensional animation environment, including a memory and a processor coupled to the memory.

[0036] Other aspects of the invention can be understood by those skilled in the art based on the specification, claims, and drawings. Attached Figure Description

[0037] The following figures are based on embodiments described in this invention and are merely illustrative examples, and are not intended to limit the scope of the invention.

[0038] Figure 1 This is a block diagram of an exemplary computing system described in the embodiments of the present invention;

[0039] Figure 2 This is an exemplary system architecture 200 of the system for creating film technology described in the embodiments of the present invention;

[0040] Figure 3 This is a flowchart of an exemplary filmmaking process 300 as described in an embodiment of the present invention;

[0041] Figure 4 The graphical user interface (GUI) of the exemplary editing tool described in the embodiments of the present invention is shown;

[0042] Figure 5 An exemplary action browsing interface 500 for presenting an action list is shown in an embodiment of the present invention;

[0043] Figure 6A A top view of an exemplary 3D scene described in an embodiment of the present invention is shown;

[0044] Figure 6B It shows Figure 6A A floor plan of an exemplary 3D scene shown;

[0045] Figure 7 It shows Figure 6B The traces of character movement shown in the floor plan; and

[0046] Figure 8 An exemplary prompt-based automatic cinematography frame 800 as described in an embodiment of the present invention is shown. Detailed Implementation

[0047] Exemplary embodiments of the invention will now be described in detail with reference to the accompanying drawings. Hereinafter, embodiments consistent with the invention will be described with reference to the accompanying drawings. Throughout the drawings, the same reference numerals will be used to refer to the same or similar parts. It is obvious that what is described are some, but not all, embodiments of the invention. Based on the disclosed embodiments, those skilled in the art can derive other embodiments consistent with the invention, all of which are within the scope of the invention.

[0048] This invention provides a Write-A-Movie technique that unifies writing and filming, and relates to a method and apparatus for automatically (or semi-automatically) generating movies based on scripts and customizable features.

[0049] There are several differences between the screenwriting and filmmaking processes. First, the talent requirements differ: screenwriting focuses on creating / editing the text, while filmmaking requires a much broader range of skills. Even in animated filmmaking, expertise in 3D modeling, character animation and rendering, camera handling, post-processing, and other areas is needed. Second, resource requirements differ: screenwriting only requires text editing tools, while filmmaking demands a wider range of resources. Even for animated filmmaking, essential resources such as 3D and post-production software, scene models, character and object models, voice acting, and many other elements are required to get started. Third, the execution timeline differs: current filmmaking begins with a completed script; therefore, the characters and scenes are uncertain until the script is finalized, meaning filmmaking cannot begin until the script is finalized. Fourth, the economic value differs: filmmaking typically costs significantly more than screenwriting, with screenwriters usually receiving only 2% to 3% of the filmmaking budget. Finally, scarcity makes a difference: only a very small percentage of screenplays make it to the film production stage. Reports indicate that approximately 50,000 screenplays are registered with the Writers Guild of America each year. Hollywood studios release about 150 films annually, suggesting a 99.7% failure rate for writing such a project.

[0050] With the latest developments in artificial intelligence (AI) technology, many tasks mentioned in the filmmaking process can be completed by computers. For example, deepfake technology can implant the image of a celebrity into a video in which he / she did not act, and computer photography can be used to automatically generate camera sequences that follow film rules or conventions.

[0051] The existing technological assumption that combines the writing process with the video production process can utilize online video resources to montage multiple video shots, matching these shots with the input text, and then combining the selected video shots by optimizing cinematography rules to generate the final output video. However, this method cannot be directly applied to the film production process of a script because the merged online shots can only vaguely represent the text of the script, which is difficult to meet the standards of film.

[0052] Users have a strong desire to use video as an effective storytelling element in writing and editing text, i.e., to generate films based on storytelling text. This invention provides a film creation technology that enables ordinary users to experience a process of unifying story writing and filmmaking, ultimately generating a film.

[0053] Specifically, "Creative Film Technology" enables screenwriters to pre-play their scripts in animated video format during the scriptwriting stage, allowing for on-the-spot revisions. This technology can be applied in many contexts, such as interactive films. The difference between interactive and traditional films lies in the fact that interactive films are based on a hyperstory framework with multiple branches, while traditional films tell a single story. The hyperstory structure comprises numerous stories (i.e., branches) sharing the same main narrative. Therefore, animated assets already established within available story branches, including scene and character animations, can be reused in subsequent stories. As the hyperstory and its associated assets grow, Creative Film Technology provides a "playing video while creating" experience for creators who can utilize existing resources. Clearly, this tool aims to bridge the gap between writing and filmmaking, significantly lowering the barrier to entry for ordinary users who wish to create films by simply writing stories and revising text based on generated, real-time animated videos. Using this Creative Film Technology can significantly reduce the disparities between writing and filmmaking in terms of talent, resources, execution time, and scarcity. The aforementioned film creation technology can promote the spirit of sharing animation resources in content creation communities, making the entire process more efficient.

[0054] The film creation technology combines techniques from various fields, including NLP and understanding technologies, automatic object staging, automatic motion arrangement, and computer-generated imagery. With minimal user interaction, the system utilizes an automated workflow to export the film script in production into corresponding animated films.

[0055] The feasibility of a system for creating film technology is based on the following assumptions: that the floor plans of all scenes the author plans to use in his script are available before the writing process; and that the animation assets the author plans to use in his script, including animation assets for characters and objects, are available before the writing process.

[0056] Clearly, these assumptions impose constraints on authors, and the hyperstory structure can satisfy these conditions, as can authors who compromise (reuse available resources) to achieve the experience of "playing video in creation."

[0057] To the inventor's knowledge, this is the world's first achievement in this technology. The technology to realize this experience requires other technologies such as natural language processing and understanding, automatic object staging, automatic motion arrangement, and computer-generated imagery. Unifying the writing and filmmaking processes can significantly increase productivity and have a major impact on the film industry. This technology can help promote the trend of sharing and reusing vast amounts of 3D resources and can accelerate the process of creating excellent screenplays.

[0058] Figure 1 A block diagram of an exemplary computing system 100 is shown, which is capable of implementing the aforementioned methods for creating film techniques. Figure 1 As shown, the computing system 100 may include a processor 102, a storage medium 104, a display 106, a communication module 108, a database 110, and peripheral devices 112, as well as one or more buses 114 that couple the above devices together. Some devices may be omitted, and others may be included.

[0059] The processor 102 may include any suitable processor. Furthermore, the processor 102 may include multiple cores for multi-threaded or parallel processing. The processor 102 may execute sequences of computer program instructions or sequences of program modules to perform various processing procedures, such as receiving user input in a script editing tool to generate a script, automatically generating an action list based on the script, rendering a scene based on the action list, and generating animated videos by performing editing and optimization processes. The storage medium 104 may include memory modules such as read-only memory (ROM), random access memory (RAM), flash memory modules, and erasable and rewritable memory, as well as mass storage devices such as CD-ROMs, USB flash drives, and hard disks. When the processor 102 executes computer program instructions or program modules, the storage medium 104 may store the computer program instructions or program modules, and may store scripts, images, animation materials, video data used to generate motion pictures, and / or movies generated according to the disclosed methods.

[0060] Furthermore, the communication module 108 may include a network device for establishing connections via a communication network. The database 110 may include one or more databases for storing data (e.g., images, videos, floor plans used in movie scenes, animation materials such as animation assets of characters and objects) and for performing operations on the stored data, including retrieving data from the database.

[0061] The display 106 may include any suitable type of computer display device or electronic display device (e.g., CRT (cathode ray tube) or LCD (liquid crystal display) based device, touch screen, LED (light emission diode) display). The peripheral device 112 may include various sensors and other I / O (input / output) devices, such as speakers, cameras, motion sensors, keyboards, mice, etc.

[0062] During operation, the system 100 can perform a series of actions to implement the disclosed filmmaking techniques and framework. The filmmaking techniques and apparatus can also be referred to as motion picture generation methods and apparatus. The system 100 can be implemented on a terminal or a server, or a combination of both. As described herein, a terminal can refer to any suitable user terminal with specific computing capabilities, including collecting user-edited scripts, displaying action commentary interfaces, rendering 3D scenes based on action lists, performing editing and optimization processes on videos, and displaying edited videos. For example, a terminal can be a personal computer (PC), a workstation computer, a server computer, a handheld computing device (tablet), a mobile terminal (mobile phone or smartphone), or any other user-end computing device. As described herein, a server can refer to one or more server computers configured to provide specific server functions, such as performing natural language processing on scripts to generate action lists, determining camera configurations for shooting animated videos, generating animated videos based on said camera configurations, and editing said animated videos by finding a path with a minimum cost function in a graphical model. The server may also include one or more processors to execute computer programs in parallel. The terminal and / or the server can be configured to provide structure and functionality for the actions and operations described above. In some embodiments, certain parts of the actions can be performed on the server, while other parts of the actions can be performed on the terminal.

[0063] The film is generated based on a script or screenplay. The terms script and screenplay are used interchangeably in this invention, and they refer to the text in the film that describes character dialogue, performance instructions, scene descriptions, and directorial instructions. The terms user, writer, and author are used interchangeably in this invention, and they refer to the creator of the screenplay and the film. A film consists of multiple scenes arranged to describe a series of actions / events. The action list described herein can refer to a list of annotations based on the film's script / screenplay, noting what character or object is doing what event at what time. The various scenes in the film can be filmed by cameras (e.g., virtual cameras placed in a 3D environment). Obviously, different camera settings can be used to film the same scene. Embodiments of the invention use a scene as the basic unit of the screenplay. However, it is understood that the system described herein can also be applied to units smaller than a scene. The camera sequence of the film described herein can refer to a configuration sequence of cameras arranged chronologically for filming scenes in the film. For example, the camera sequence can specify where to place the camera to capture a scene, specify what settings to use when capturing the scene (such as lens size and angle), specify how long the shot will last (such as how many units of time), and so on.

[0064] Figure 2 An exemplary system architecture 200 of a system for creating film techniques according to a partial embodiment of the present invention is shown. Figure 3 A flowchart illustrating an exemplary filmmaking process 300 according to a partial embodiment of the present invention is provided. The exemplary filmmaking process 300 and the exemplary system architecture 200 may be implemented by a computing system 100 (e.g., configuring one or more processors 102 to execute programs to implement the aforementioned functionality).

[0065] Reference Figure 2 and Figure 3 The system architecture 200 for creating film technology allows a scene script (i.e., a scene script) to be automatically converted into a list of actions (i.e., an action list) by an NLP engine 202. Optionally, an action correction component 204 is introduced to allow the user to modify the attributes of the actions and / or correct errors that may be introduced due to insufficient script understanding capabilities of the NLP engine 202. Upon receiving the user-verified action list, the 3D rendering engine 206 can generate 3D data reflecting the performance of characters in the 3D scene. The optimization engine 208 is configured to select a camera for each output 2D frame from all possible options in an automatic camera optimization process (e.g., finding an optimized camera sequence). The 2D frames output by the optimization engine are combined to form a 2D video, which is the film generated by the film creation technology system.

[0066] Specifically, system architecture 200 obtains the film script written by the user (S302). In some embodiments, the film creation technology system provides the user with editing tools for creating the script. These editing tools are feasible for both professional and non-professional scriptwriters, and ordinary users can master them in a short time.

[0067] In some embodiments, the system architecture 200 may also acquire floor plans of scenes from the movie and animation resources of characters and objects in the movie. Customizable candidate options provided by the editing tool can be generated based on the floor plan, the characters, and the objects. For example, the editing tool may provide a menu that includes optional candidate locations for scenes in the movie. The candidate locations correspond to the floor plan. As another example, the editing tool may provide a menu that includes optional candidate characters / objects appearing in the movie. The names of the characters / objects can be obtained from the animation resources of the characters and objects.

[0068] Figure 4 The graphical user interface (GUI) of an exemplary editing tool in some embodiments of the present invention is shown. For example... Figure 4 As shown, the GUI may include a designated area for the script editor 402 (e.g., the left column in the figure) and a designated preview area 404 for presenting the generated script in a professional format (e.g., the right column in the figure).

[0069] The script editor 402 may include multiple selectable labels 4022, each corresponding to an element that can be added to the script, such as fade-in, fade-out, headings (e.g., a scene title describing the location and date of the scene), characters and dialogue (e.g., characters appearing in the scene and their lines), and descriptions (e.g., a narrative description of the scene's events). For example, when the "+Fade In" label is selected first, the element "Fade In" is added to the script. When the "+Heading" label is selected, optional controls related to the scene title are added to the editing area 4024. These optional controls may include drop-down menus that allow the user to select options such as location, character, and the time of the current scene. When the "+Characters and Dialogue" label is selected, it is configured to add character options (e.g., a drop-down menu listing available characters), add character actions and / or add attitude / emotion options, and add a text box in the editing area 4024 for inputting the selected character's voice content. When the “+Description” label is selected, the “+Description” label is configured to add a text box in the editing area 4024 for the user to enter a narrative description of the scene.

[0070] Each element shown in the editing area 4024 may have a corresponding position adjustment icon and a delete icon. When the position adjustment icon is dragged up and down in the editing area 4024, the corresponding element can be moved up and down in the structure of the script.

[0071] The editing tool is configured to provide customization features, allowing users to add and modify script elements by simply clicking buttons and filling in content. The results of these additions and modifications can be viewed in a professional format, such as fonts, formatting, page layout, and margins consistent with the professional requirements of a script.

[0072] Reference Figure 3 NLP can be performed on the script to generate an action list. The action list includes multiple ordered actions with attributes, each action's attributes including a subject, predicate, and position (S304). For example, the output of a script editing tool (i.e., a user-generated scene script) can be input into the NLP engine 202 to automatically extract and convert it into a list of actions. The NLP engine 202 is configured to use NLP technology to automatically analyze sentences in the scene script and generate an action list. The action list includes a list of actions and their associated attributes.

[0073] An action includes one or more of the following components: Subject: Typically a character in the script, or a physical object that may trigger the action (e.g., a book, a tree, etc.). Predicate: The action itself, such as walking, sitting, speaking, etc. Object: The recipient of the action, such as a character or physical object. Emotion: The emotional expression of the subject when performing the action. Location: The target location of the action (e.g., walking to a place) or the location where the action occurs. Duration: The duration of the action, such as the duration of a dialogue sentence.

[0074] In some embodiments, the floor plan provided by the user to the system architecture 200 may include marked location indicators. The NLP engine 202 is configured to determine the location of actions by recognizing phrases in a script, each phrase corresponding to one of the marked location indicators.

[0075] The NLP engine 202 is configured to break down script paragraphs into sentences and process each sentence to convert it into one or more actions. These actions are executed in a specific order. In some embodiments, some actions can operate in parallel, and the system architecture 200 provides the user with the ability to specify relationships between these actions (e.g., by using keywords such as "simultaneously" in the script recognizable by the NLP engine 202, and / or by specifying that such actions operate at the same time through the action correction component 204). In some embodiments, if the NLP engine 202 fails to successfully map the subject and / or object of an action to a recognizable character or object, user assistance helps in correcting this using the action correction component 204. The system architecture 200 for filmmaking technology should be intelligent enough to minimize unnecessary user intervention. That is, the user should only need to provide minimal assistance to correct the action list and related attributes.

[0076] The motion correction component 204 can support a motion browsing interface to allow users to browse and revise the motion list generated by the NLP engine 202. According to some embodiments of the present invention, Figure 5 An exemplary action browsing interface 500 is shown, which is used to display a list of actions. For example... Figure 5 As shown, the script browsing area 502 can display the current part of the script corresponding to the action list, which is displayed in the first action display area 504 and the second action display area 506.

[0077] Each action display area can show one or more action units corresponding to elements in the script, with each action unit corresponding to one action. For example, the first action display area 504 includes one action unit corresponding to a first element shown in the script browsing area 502 (i.e., character elements and dialogue elements in Aunt Mary's speech). The second action display area 506 includes multiple action units corresponding to a second element shown in the script browsing area 502 (i.e., narrative descriptions of multiple characters such as Lizzie and Aunt Mary). Each action unit identifies the subject from the script (e.g., what the subject says in dialogue or the original text describing the subject), the subject's predicates (e.g., say, stand, cry, walk), and the text about the subject.

[0078] Furthermore, users are provided with the ability to modify actions and related attributes (such as location and time). Specifically, the attributes of the action and options for modifying those attributes are displayed in the action browsing interface. In some embodiments, the modifiable attributes of an action include at least one of the action's predicate verb, location, or duration. Revised attributes of actions can be generated on the action browsing interface based on user input, and a revised list of actions can be generated based on these revised attributes.

[0079] In some embodiments, the motion correction component 204 is configured to allow a user to correct actions and related attributes automatically extracted from a script by the NLP engine 202. This can be based on user input (e.g., through...). Figure 5 (Select other options from the drop-down menu shown) to adjust the predicate of the subject. In some embodiments, for example in Figure 5 The grayed-out subject and / or text shown is immutable within the motion correction unit. The motion unit may include a delete icon for completely removing the motion unit, and / or an add icon for adding another motion unit. The motion unit may also include other adjustable attributes, such as the object of the dilemma, the character's emotion, duration (e.g., specifying seconds as the time unit), and / or location (e.g., specifying coordinates in a 3D scene or specifying 2D planar coordinates corresponding to the 3D scene).

[0080] The action unit may further include a positioning icon 508, which is configured to present a 3D scene corresponding to the action or a planar view of the 3D scene when selected. Character entrances are an important task in filmmaking. This invention provides an automated entrance method to accelerate the entrance process and reduce manual intervention and effort. In some embodiments, the NLP engine 202 is configured to perform automated entrances with the assistance of a planar view of the scene. For example, one or more planar views with marked locations can be input into the NLP engine 202. The NLP engine 202 can identify the planar view corresponding to the scene based on location keywords extracted from sentences in the script, and / or identify specific marked locations in the planar view corresponding to the scene.

[0081] Figure 6A An exemplary 3D scene top view of some embodiments of the present invention is shown. Figure 6B It shows Figure 6A A floor plan of an exemplary 3D scene. (e.g.) Figure 6BAs shown, a floor plan of the current scene, marking all locations, can help authors / users describe the location of actions during script creation. In some embodiments, when the location indicated in the action unit of the action browsing interface 500 is blank (i.e., the NLP engine does not recognize the location of the action) or an unnecessary location, the user can simply select the positioning icon 508 to browse the floor plan and click on the displayed floor plan to assign a location to the action. The action correction component 204 can record the location clicked by the user and convert it into corresponding coordinates within the 3D scene. In other words, a first user selection (e.g., selecting the positioning icon 508 in the action unit) can be received in the action browsing interface to input the action location (e.g., modifying or specifying a location). In response to the first user selection, a floor plan corresponding to the action can be displayed. A second user selection at a specified location in the floor plan can be obtained. The location attributes of the action can be updated based on the second user selection (e.g., the location clicked by the user on the floor plan).

[0082] Furthermore, the orientation of a character / object during an action can be calculated based on the direction of movement to the next position (e.g., the direction of movement of the subject from the current position to the next position). The trajectory of character movement in the scene can serve as a reference for users / writers to review whether the actions are properly arranged. In some embodiments, after viewing and editing the actions and attributes of the current scene, the action browsing interface can generate a movement trajectory diagram of the characters appearing in the current scene based on the attributes of the actions in the current scene and the order / sequence of the actions. In other words, the movement trajectory diagram of the scene can be displayed based on the plan view of the scene, the actions corresponding to the scene, and the sequence of actions corresponding to the scene. The movement trajectory diagram includes the movement trajectory and direction of movement of characters or objects in the scene. Users can browse the movement trajectory diagram, and if they find any unwanted movement or position of certain characters / objects, the system can allow the user to return to the action browsing interface to make the necessary modifications.

[0083] Figure 7 It shows Figure 6B The character's movement trajectory is shown in the plan view. Figure 7As shown, the movement trajectories of two characters are illustrated in a plan view. Each dot or square represents the position of an action, while arrows indicate the sequence between the two actions. Taking the character Mary as an example, Mary's action at the second position 704 immediately follows her action at the first position 702; therefore, arrow 706 starts from the first position 702 and points to the second position 704. Users can view the motion trajectory diagram and adjust the position of actions as needed, and / or adjust the sequence of actions (e.g., by interacting with the labels / lines / arrows displayed on the motion trajectory diagram and / or modifying the attributes of actions in the action browsing interface).

[0084] The output of the NLP engine 202 and the optional motion correction component 204 is a motion list, which includes actions and attributes arranged in sequence, the sequence being consistent with the script.

[0085] Reference Figure 3 The 3D rendering engine 206 can render 3D data in a 3D scene of a movie based on an action list (e.g., an action list output by the NLP engine 202 or the motion correction component 204) (S306). An action is the basic operational unit of the 3D rendering engine 206. Based on the action list, the 3D rendering engine 206 is configured to utilize virtual 3D models to depict the story described in the script. The 3D rendering engine 206 can generate 3D data, including at least one of the following: a 3D scene, 3D characters and objects appearing in the scene, and the motion of the 3D characters and objects in the 3D scene.

[0086] In some embodiments, the duration of an action can be specified by the author / user in the action browsing interface. If no duration is specified, the 3D rendering engine 306 can be configured to determine the duration. In this way, each character / object maintains an action timeline, and certain actions of certain characters / objects can operate in parallel. If all the corresponding time information for each action is calculated or specified, a map of the actions of all characters / objects on the timeline can be generated.

[0087] To convert a 3D scene into 2D video, an automatic cinematography module (e.g., the optimization engine 208) dynamically selects cameras for each frame of the output video. In other words, the camera sequence for shooting two-dimensional (2D) frames in the 3D scene can be determined by performing an automatic cinematography optimization process (S308). In some embodiments, the automatic cinematography employs a cue-based optimization process that ensures the author's preferences can be considered during the final video generation process. This is effective for allowing the author to influence the final video result.

[0088] In filmmaking techniques, any writer / user can become a director, deciding what is important (e.g., who is the protagonist whose actions have a substantial impact, which object reflects an undeniable clue, etc.), the atmosphere of the scene (e.g., the intensity of audience perception, conflict and emotional intensity, etc.), the focus model (e.g., the intensity of character fear, the intensity of the villain, the intensity of handheld attacks, etc.), and many other factors that may directly affect audience perception.

[0089] Figure 8 An exemplary cue-based automatic cinematography framework 800 is illustrated in some embodiments of the present invention. The framework 800, used in filmmaking techniques, allows the filmmaker to provide cues to guide the camera optimization process. Utilizing the concept of "magic space," an infinite search space of 7 degrees of freedom (7-DOF) is minimized to countably discrete camera configurations, with camera placement conforming to the grammar of cinematic language. The cue-based optimization process determines how to select the camera for each frame of the output video. This is based on a cost function defined in a mapping process, also known as a cue translation process, which converts the filmmaker's cue into a mathematical expression and incorporates that mathematical expression into the cost function as a constraint.

[0090] like Figure 8As shown, the framework 800 includes two main blocks: a cue translation process block 800-1 and an optimization process block 800-2. Script decomposition 802 is implemented by the NLP engine 202 and / or the motion correction component 204 to generate an action list. The system of the present invention includes a cue translator configured to implement the functions in the cue translation process block, including obtaining the protagonist 804 based on magic space 806 and automatically trimming pre-shot content 812, cue translation 814, director cue stylization and emotion-related cue translation 816, user-added camera interface 822, and optimized data preparation 824. For example, a trimmed action list can be obtained after time-adaptive pre-shot content trimming (automatic pre-shot content trimming 812). A content preview 8122 can then be presented to the user on the user interface, and the user can adjust it on the user interface based on the importance and relevance of the actions 8124 (e.g., through a user interface depicting an action importance curve). After the camera sequence of the video preview 820 optimization algorithm, the user can repeatedly adjust the output of the automatic camera sequence. Users can indirectly change the arrangement of camera sequences at different time periods through an intuitive prompt translator, and can also add user-defined free cameras in the 3D animation environment. Optimization process block 800-2 includes a default camera position 808, a rendering element 810 that renders based on the camera configuration and position in the camera search space, a prompt-based optimizer 818, and lens pass-through detection (not shown in the figure). As described in this invention, director's prompts or director's hints refer to information that conveys or translates the user's expectations for cinematography. The system of this invention can obtain various types of director's prompts, including protagonist prompts, content trimming-related prompts, clue prompts, atmosphere-related prompts, directorial style-related prompts, and / or at least one of user-added free cameras. Users of the system of this invention can decide whether to provide these director's prompts and input them into the system to influence filmmaking. The director's prompts may affect the automatic filmmaking process at different stages, such as when determining which camera to use to shoot a scene, which camera configuration to use for the scene, and whether to include certain parts of the scene in the final film.

[0091] During operation, based on the framework 800, the automatic film shooting process may include acquiring and translating director's cues for creating a 3D animated film based on user input; determining camera configurations in the 3D environment based on the director's cues; constructing a camera search space, which includes multiple candidate cameras that will be used at different timestamps to shoot one or more scenes of the film according to the camera configuration; after establishing the camera search space, performing a cues-based editing optimization process based on the camera search space and the director's cues, such as finding suitable camera positions in different scenes (specifying the camera's lens parameters, orientation, and focal length for each time unit of the film). When an optimized camera sequence is found, the edited output video 828 is obtained based on the rendering result 826. The editing optimization process is formalized as finding the lowest-cost path in a graphical model, where each path describes a candidate camera sequence for making the film, and at least some of the director's cues are converted into a cost function of the graphical model.

[0092] Reference Figure 3 The 2D video of the movie is generated by combining 2D frames captured by a camera, which is selected based on a determined camera sequence (S310).

[0093] In some embodiments, 2D video can be played, and users can view the 2D video and determine if further editing is needed, such as script editing, editing of character / object action attributes, and direct editing of camera configuration and position. The system architecture 200 of this invention integrates all the tools disclosed above, such as script editing tools, motion browsing and correction interfaces, and prompt-based automatic camera tools, into a single software, allowing users to view and modify the results at any stage of the filmmaking process (including scriptwriting and filmmaking). For example, when or after playing a 2D video, the system of this invention can provide the user with an option to select from multiple editing tools to complete the desired revisions. The candidate selections may include script editing tools, motion browsing and correction interfaces, and prompt-based automatic camera tools.

[0094] In some embodiments, after playing the generated 2D video, a script editing tool can be presented to the user for modification of script elements. Alternatively, while playing the generated 2D video, if the user pauses at a specific scene and selects a script editing tool from among several editing tools to modify elements in the script corresponding to the paused scene, an updated 2D video based on the modified script can be generated.

[0095] In some embodiments, after playing the generated 2D video, an action browsing interface can be displayed to the user to modify the action list (e.g., modifying the positional attributes of actions based on a floor plan). Alternatively, when playing the generated 2D video, if the user pauses at a specific scene and selects the action browsing interface from among several editing tools to modify the attributes of the actions in the action list corresponding to the paused scene, an updated 2D video based on the modified action list can be generated.

[0096] This invention discloses a new technology called Write-A-Movie, which merges the previously separate scriptwriting and filmmaking processes, allowing ordinary people to easily write stories and make movies. To the inventor's knowledge, this is the world's first product to use a scriptwriting tool to drive the filmmaking process. This invention also discloses a novel framework that integrates script understanding, action list generation, action correction, automatic (or semi-automatic) staging, 3D rendering, and prompt-based automatic cinematography into a unified process, thereby promoting film production based on simple story creation and user preferences.

[0097] Creative filmmaking technology enables a unique "video playback during creation" experience. This technology will have a long-term impact on the media and display industries by inspiring a broader pool of potential writers and media contributors, allowing them to bring their ideas and stories to the screen. This vision would have been impossible before the invention and realization of this technology.

[0098] Other embodiments of the invention will be apparent to those skilled in the art upon consideration of the specification and practice disclosed herein. The specification and embodiments are merely exemplary, and the true scope of the invention is indicated by the claims.

Claims

1. A method for generating a movie, characterized in that, The method includes: Obtain the film script; The script is subjected to natural language processing to obtain an action list, which includes multiple actions with attributes. The attributes of each action include one or more of the following components: Subject: A character in the script, or the physical object that triggers the action; Predicate: The action itself; Object: The recipient of the action; Emotion: The emotional expression of the subject when performing the action; Location of the action: The target location of the action or the location where the action occurs; Duration: The duration of the action. According to the action list, three-dimensional data is rendered in the three-dimensional scene of the movie, and the three-dimensional data reflects the actions performed by the subject at the corresponding position in the three-dimensional scene; The film is optimized for photography by determining the camera sequence for shooting two-dimensional frames in the three-dimensional scene; and The two-dimensional video of the movie is generated by combining the two-dimensional frames, and the two-dimensional frames are captured based on the determined camera sequence; The generated action list includes: Divide the paragraphs of the script into sentences; Each sentence is processed automatically, and each sentence is converted into one or more actions through a natural language processing procedure; Sort the one or more actions according to the order of the sentences; The step of optimizing the film's photography and determining the camera sequence for shooting two-dimensional frames in the three-dimensional scene includes: Obtain and translate director's cues for making 3D animated films; Based on the director's instructions, the camera configuration in the 3D environment is determined, and a camera search space is constructed. The camera search space includes multiple candidate cameras, which will be used at different timestamps to shoot one or more scenes of the movie according to the camera configuration. After establishing the camera search space, an editing and optimization process based on the camera search space and the director's prompts is performed to find suitable camera positions in different scenes; The editing optimization process is formalized as finding the lowest-cost path in a graphical model, where each path describes a candidate camera sequence for making the film, and at least some director's cues are converted into a cost function of the graphical model.

2. The movie generation method according to claim 1, characterized in that, Also includes: Obtain a plan view of the 3D scene of the movie, as well as animation resources of the characters and objects in the movie; and The 3D data is rendered in the 3D scene based on the plan view and the animation resources.

3. The movie generation method according to claim 2, characterized in that: The plan includes marked location indicators; as well as The method further includes: identifying phrases in the script by performing the natural language processing procedure to determine the location of an action, the phrases corresponding to a marked location indicator.

4. The movie generation method according to claim 1, characterized in that, The obtained film script includes: Obtain a script editing tool that allows users to add and modify multiple elements of the script; and Obtain the script generated by the script editing tool.

5. The movie generation method according to claim 1, characterized in that, Also includes: The action browsing interface displays the attributes of the action and options for modifying the attributes of the action. The modifiable attributes of the action include at least one of predicate, position, and duration. The modified attributes of the action are obtained based on the user's input on the action browsing interface; as well as A modified action list is generated based on the modified attributes, and the 3D data is rendered based on the modified action list.

6. The film generation method according to claim 5, characterized in that, Also includes: Receive a first user selection, where the first user selection is the location of the action entered in the action browsing interface; In response to the first user selection, a floor plan corresponding to the action is displayed; A second user selection is obtained at the designated location of the action described in the plan view; and Update the position attribute of the action based on the second user's selection.

7. The movie generation method according to claim 2, characterized in that, Also includes: Based on the planar view of the three-dimensional scene, the actions corresponding to the three-dimensional scene, and the action sequence corresponding to the three-dimensional scene, a movement trajectory diagram of the scene is displayed, which includes the movement trajectory and direction of movement of the characters or objects in the scene.

8. The movie generation method according to claim 4, characterized in that, Also includes: Play the generated two-dimensional video; After the generated two-dimensional video is played, the script editing tool is displayed so that the user can modify elements in the script; and An updated 2D video is generated based on the revised script.

9. The film generation method according to claim 5, characterized in that, Also includes: Play the generated two-dimensional video; After the generated two-dimensional video is played, the action browsing interface is displayed so that the user can modify the action list; and An updated 2D video is generated based on the modified action list.

10. A film generation apparatus, characterized in that, include: The memory and the processor coupled to the memory, wherein the processor is configured to perform the following operations: Obtain the film script; The script is subjected to natural language processing to obtain an action list, which includes multiple actions with attributes. The attributes of each action include one or more of the following components: Subject: The character in the script, or the physical object that triggers the action; Predicate: The action itself; Object: The receiver of the action. Emotion: The emotional expression of the subject when performing the action; Location of the action: The target location of the action or the location where the action occurs; Duration: The duration of the action; Based on the action list, render three-dimensional data in the three-dimensional scene of the movie, the three-dimensional data reflecting the action performed by the subject at the corresponding location in the three-dimensional scene; The film is optimized for photography to determine the camera sequence of cameras that capture two-dimensional frames in the three-dimensional scene; and The two-dimensional video of the movie is generated by combining the two-dimensional frames, wherein the two-dimensional frames are captured by the camera based on the determined camera sequence; The generated action list includes: Divide the paragraphs of the script into sentences; Each sentence is processed automatically, and each sentence is converted into one or more actions through a natural language processing procedure; Sort the one or more actions according to the order of the sentences; The step of optimizing the film's photography and determining the camera sequence for shooting two-dimensional frames in the three-dimensional scene includes: Obtain and translate director's cues for making 3D animated films; Based on the director's instructions, the camera configuration in the 3D environment is determined, and a camera search space is constructed. The camera search space includes multiple candidate cameras, which will be used at different timestamps to shoot one or more scenes of the movie according to the camera configuration. After establishing the camera search space, an editing and optimization process based on the camera search space and the director's prompts is performed to find suitable camera positions in different scenes; The editing optimization process is formalized as finding the lowest-cost path in a graphical model, where each path describes a candidate camera sequence for making the film, and at least some director's cues are converted into a cost function of the graphical model.

Images