A script-to-film generation method and system

Abstract

A script-to-movie generation method and system, the method for a computing device, comprising: obtaining a movie script; generating a list of actions from the movie script; generating a stage performance based on each action in the list of actions; extracting observation information from the stage performance; selecting a camera based on the observation information using a camera agent trained using a reinforcement learning algorithm, wherein the camera comprises camera settings that define a position of the camera relative to a character filmed by the camera; capturing a video of the stage performance using the selected camera; and outputting the video.

Description

[0001] priority

[0002] This invention claims priority to U.S. Patent Application No. 17 / 858,067, filed July 6, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of computer graphics technology, and more particularly to a novel automated film shooting system using reinforcement learning. Background Technology

[0004] Automated filmmaking has proven to be a highly effective means of shortening and reducing the cost of film production by incorporating numerous rules of camera movement into the optimization process. Many renowned directors have developed their own camera language styles, sometimes breaking conventional rules. Integrating these directors' styles into an automated filmmaking framework is difficult for two reasons: (1) it is difficult to capture these styles and translate them into a rule base; and (2) the data on directorial styles collected from existing films is often insufficient and inaccurate for learning purposes. Therefore, films produced by robot directors based on common camera language and conventional rules may not meet the expectations of human filmmakers. Summary of the Invention

[0005] One aspect of the present invention provides a script-to-video generation method for a computing device. The method includes: obtaining a video script; generating a list of actions based on the video script; generating a stage performance based on each action in the list of actions; extracting observation information from the stage performance; selecting a camera based on the observation information using a camera agent trained with a reinforcement learning algorithm, wherein the camera includes camera settings that define the position of the camera relative to the character being filmed by the camera; capturing video of the stage performance using the selected camera; and outputting the video.

[0006] Another aspect of the present invention provides a script-to-movie generation system. The system includes a memory storing program instructions and a processor coupled to the memory and configured to execute the program instructions to: obtain a movie script; generate a list of actions based on the movie script; generate a stage performance based on each action in the list of actions; extract observation information from the stage performance; select a camera based on the observation information using a camera agent trained with a reinforcement learning algorithm, wherein the camera includes camera settings that define the position of the camera relative to the character being filmed by the camera; capture video of the stage performance using the selected camera; and output the video. Attached Figure Description

[0007] Based on the various embodiments disclosed, the following figures are merely examples for illustrative purposes and are not intended to limit the scope of the invention.

[0008] Figure 1 A schematic functional diagram of an exemplary script-to-video generation system according to some embodiments of the present invention is shown;

[0009] Figure 2 A schematic structural diagram of an exemplary script-to-video generation system according to some embodiments of the present invention is shown;

[0010] Figure 3 A flowchart illustrating an exemplary script-to-video generation method according to a partial embodiment of the present invention is shown;

[0011] Figure 4 Exemplary frames generated by different models are shown;

[0012] Figure 5 A schematic diagram illustrating the relationship between camera placement and acceptance threshold according to a partial embodiment of the present invention;

[0013] Figure 6 Exemplary frame-building techniques generated by the aesthetic model (left) and the RT2A model (right) are shown.

[0014] Figure 7 The diagram shows exemplary dialogue frames generated by the aesthetic model (left) and the RT2A model (right); and

[0015] Figure 8 This shows an exemplary single motion frame generated by the aesthetic model (left) and the RT2A model (right). Detailed Implementation

[0016] Exemplary embodiments of the invention will now be described in detail with reference to the accompanying drawings. Hereinafter, embodiments consistent with the invention will be described with reference to the drawings. Throughout the drawings, the same reference numerals are used wherever possible to refer to the same or similar parts. It will be apparent that the described embodiments are part, but not all, of the embodiments of the invention. Based on the disclosed embodiments, other embodiments consistent with the invention can be derived by those skilled in the art, all of which are within the scope of the invention.

[0017] The script-to-movie generation technology is an adaptive, self-reinforcing framework for automatically generating videos based on movie scripts. This invention provides a script-to-movie generation system that leverages recent advances in natural language processing, computational filmmaking, and video understanding. The automated workflow of the script-to-movie generation system significantly reduces the time and knowledge required for the process. By introducing a novel hybrid objective evaluation mechanism that considers both the comprehensibility of the movie script's visual presentation and its compliance with filmmaking guidelines, the video generation process is mapped to an optimization problem aimed at producing higher-quality videos. Dynamic programming has been recognized as a method for solving optimization problems and obtaining optimal solutions with the most efficient computational complexity. Therefore, the script-to-movie generation system consistent with this invention significantly accelerates the movie production process.

[0018] In the traditional film industry, scriptwriting and filmmaking are completely separate processes. With recent advancements in artificial intelligence, a significant portion of the filmmaking process can now be automated. Linking scriptwriting with filmmaking offers direct benefits to all participants. Scriptwriting allows for visualization and editing of the final product before submission. Producers can sift through scripts by viewing pre-visualized versions, rather than reading hundreds of pages. The script-to-film generation system needs to meet two quality requirements. First, the output film must maintain a reasonable level of fidelity to the script. Second, the output film must adhere to cinematic aesthetics and rules.

[0019] Therefore, a mechanism is needed to evaluate the fidelity of the generated video to the corresponding film script and, if the evaluated fidelity falls below an acceptable standard, to provide feedback to the animation and filmmaking process for further improvement. In this way, the computational filmmaking process needs to consider not only aesthetics but also perception from the perspective of film script fidelity. While current state-of-the-art video understanding capabilities are not yet sufficient to accurately assess the fidelity of generated videos to the film script, they are adequate for evaluating certain types of films where scene complexity and motion recognition are less challenging.

[0020] The script-to-video generation system automatically converts video scripts into videos (e.g., animated videos). This system includes an arbitration mechanism supported by video understanding and natural language understanding technologies. The video understanding technology converts the generated video into a list of actions to be performed, while the natural language understanding technology converts the video script into a list of expected actions, allowing for the determination of the comprehensibility of the generated video and the fidelity to the video script. The evaluation results are then fed back to the stage performance to improve the quality of the generated video. Furthermore, aesthetic and fidelity requirements are combined within a unified evaluation framework, and the video quality improvement process is mapped to an optimization problem of adjusting camera settings and character motion settings to achieve the desired video quality. The optimization problem is designed to be solvable through dynamic programming to reduce computational complexity.

[0021] However, films generated based on aesthetic and fidelity requirements have limited quality compared to those produced by human directors. Human directors often break aesthetic rules while pursuing artistry and creativity. Capturing the behavior of human directors and incorporating it into automated film shooting systems can improve the quality of films generated by such systems.

[0022] This invention provides a novel automated film shooting system using reinforcement learning. The system is a unified framework for reinforcement learning-based text-to-animation (RT2A), which integrates reinforcement learning into the automated film shooting system. In the RT2A framework, human directors' decisions regarding camera shot settings are recorded and used to train a machine learning model. The RT2A framework includes a reward function to guide the optimization process (i.e., the optimization algorithm) to identify desired rules and mimic the decision-making process of a human director for camera selection in each scene. Experimental results demonstrate that the RT2A framework can efficiently mimic the director's use of shot language patterns. Compared to a reference algorithm, the RT2A framework achieves up to a 50% gain in camera placement acceptance and up to an 80% gain in mimicking camera transition rhythm.

[0023] Filmmaking is an art of selecting shot types and angles to capture action images. In the filmmaking industry, camera placement is an effective way to showcase artistic appeal. Applying filmmaking to the content creation process requires extensive training and knowledge. In the computer age, the possibility of using robot directors has been explored, and this challenging task has been delegated to them. This is also known as automated filmmaking. Some successful efforts have partially achieved this by translating rules from standard filmmaking guidelines into numerous cost functions and applying them during the optimization process. Such aesthetically rule-based robot directors can provide valuable references for novice artists with limited filmmaking experience. This approach is further improved by replacing the aesthetic model with a hybrid model that also considers the fidelity level of the output video by comparing it with the textual content in the original script. Simply applying these expressive rules in filmmaking is far from satisfactory. In the filmmaking industry, human directors are experts at breaking the rules of filmmaking, and they develop their own unique cinematic language to express human creativity and imagination. Therefore, creating robot directors based on shared cinematic language and summarized rules is feasible, but it is difficult to reach the level of human filmmaking artists.

[0024] Neural networks have been used to extract cinematic language from existing films to mimic the behavior of human directors in filmmaking. However, this effort has encountered problems, such as the data extracted from existing films often being inaccurate and incomplete because the data is extracted from two-dimensional frames, and information from the other dimension is essentially lost. Therefore, this method can only be applied to limited scenarios and cannot fully mimic the cinematic language of human directors. This is a behavior mimicry problem with limited training data and can be solved using reinforcement learning (RL) techniques, which generate rewards from feedback and annotations from human directors. To develop robotic directors that can truly learn the cinematic language of human directors and universally use the learned cinematic language, it is necessary not only to accurately collect data during actual filming processes but also to iteratively improve the neural network based on feedback from human directors.

[0025] In the RT2A framework, a human director modifies the camera settings calculated by the automatic film shooting system. These changes are recorded as training data for a reinforcement learning algorithm used by the automatic film shooting system to learn the human director's cinematic language. This cinematic language is captured in a neural network used to select the camera settings. Each time the human director changes the camera settings, the neural network iteratively improves through the reinforcement learning algorithm.

[0026] In this invention, reinforcement learning algorithms are used to improve script-to-film generation methods and systems. When a human director uses the script-to-film generation system to create a film, the data required by the reinforcement learning algorithm can be collected during the film production process without additional effort. The text-to-animation (T2A) film production framework is reused. A reinforcement learning functional module for automated film shooting replaces the camera optimization process in T2A. The improved framework is called the reinforcement learning-based text-to-animation framework (RT2A). The RT2A framework also includes a director feedback functional module for human directors to modify or correct camera placement calculated by a camera agent; this is a functional module for calculating camera settings. The feedback or changes made by the human director are input into the reinforcement learning functional module in the camera agent, allowing the camera agent to learn and mimic the human director's shot language.

[0027] In practical applications, feedback from human directors can come from stored training data collected from human directors during their daily filmmaking work using script-to-video generation systems. With sufficient data and feedback, the RT2A framework can effectively learn to support such robotic directors, potentially replacing real human directors and creating animations using a cinematic language similar to that of human directors.

[0028] The use of cinematic language is a key factor in determining the quality of the generated film. Using shots that go beyond basic rules requires a keen artistic sense and extensive knowledge of filmmaking. Using automated filmmaking techniques to reduce film production costs involves two approaches. One approach is based on filmmaking guidelines. By defining multiple constraints according to various rules and formulating them into corresponding loss functions, the optimal shot settings for the current scene can be calculated by minimizing the total loss. Different aesthetic constraints can be applied to different scenes, such as dialogue scenes, cooking scenes, and outdoor activities. However, using only aesthetic models makes it difficult to determine whether the use of cinematic language is adequate. Fidelity models consider the original script during optimization and ensure that the produced film is consistent with the original script. Rule-based methods ensure that the produced film follows pre-established constraints and that no rule violations occur. However, the artist's freedom and creativity are substantially limited.

[0029] Another approach is based on a behavioral model of the camera settings extracted from existing footage. This approach is data-driven rather than rule-driven. There is no need to create rules or constraints to formulate the camera agent. Instead, the camera agent is obtained through reinforcement learning algorithms such as deep Q-learning (DQN), trust region method (TRPO), and policy optimization (PPO). Once adequately trained, the camera agent can achieve the desired performance. Therefore, collecting accurate and extensive training data is necessary.

[0030] Figure 1 A schematic functional diagram of an exemplary script-to-video generation system according to some embodiments of the present invention is shown. Figure 1 As shown, the system includes an action list generation module, a stage performance module, an observation extractor module, an automatic film shooting module, a director adjustment module, and a video generation module. The action list generation module takes a text script as input, performs natural language processing (NLP) to analyze the text script, and generates a corresponding action list. The automatic film shooting module also includes a policy submodule, an RL algorithm submodule, a reward function submodule, and a training data submodule. The policy submodule and the RL algorithm submodule are collectively referred to as the camera agent. The reward function submodule and the training data submodule are collectively referred to as the automatic film shooting environment.

[0031] An action list is a list of action objects arranged in chronological order (a t This can be considered a special format used to represent the content of text scripts. Each a t It contains the necessary information for one or more virtual characters to create the corresponding performance in the stage performance module. t Performance sequences p that follow the order in the action list can be used in the stage performance module. t To perform the story from the text script. At this point, one or more virtual characters appearing in the scene are bound to multiple cameras, and each camera can be identified by a unique index. Frames captured by the cameras become p t Part of it. The observation extractor module uses p t To generate corresponding observations defined in an automated film shooting environment. t So that the camera agent can select camera C i t (i.e., camera placement), where i is the camera index. c is calculated by the camera agent in the automatic film shooting module when there is insufficient training data initially. it The quality is low and requires manual adjustment by a human director or user. i t To ensure the quality of the produced video, adjustments are made in the director's adjustment module. The corrected c i t Marked as ground-based live camera GC t And with the corresponding o t Add them together to the training data. The video generation module can use p t and GC t This is used to generate videos. As the number of high-quality videos generated using the RT2A framework increases, so does the size of the training data. When enough training data has been accumulated, the automatic film shooting module can correctly train the camera agent and update the policy using RL algorithms. Therefore, once the camera agent is trained with enough training data, the workload of human directors or users adjusting camera placement is significantly reduced.

[0032] like Figure 1 As shown, the automatic film shooting module is configured to train a camera agent. The observation generator module uses the training data to generate observation o at time t. t The camera agent observes o t Processing to select camera c i t The reward function submodule selects the camera c by applying the camera agent. i t With the camera GC chosen by the human director t To calculate the reward r t Reward r t Used to update camera agents, for example, by updating the policy submodule from the RL algorithm submodule.

[0033] In an automated film shooting environment, the observation at time t is... t Includes camera agent policy-based camera selection c i t Required information. Observe o t This includes character visibility, camera configuration, left-to-right order (LRO), action type, action start time and duration, dialogue start time and duration, or combinations thereof.

[0034] Character visibility is determined by the size of each character in a frame relative to the total frame size, the weights of different characters, and the camera setup. Since each character is attached to more than one camera, some cameras may be obscured by obstacles, and some characters may become blurry from the view of some cameras.

[0035] When switching cameras, the configuration of the previously selected camera can influence the choice of subsequent cameras. For example, in dialogue scenes, the lens reversal rule is often used. In this case, the configuration of the previously selected camera needs to be included in the observation... t middle.

[0036] LRO (Local Relationship) is a rule used to show the positional relationship between two or more characters in a frame. Including the positional relationship between two or more characters in a frame enforces the 180-degree rule used in the filmmaking industry. However, the LRO may differ depending on the camera used at point t.

[0037] The action type refers to the character's action type. For example, the action type could be "idle," which does not affect camera selection. In another example, the action type is "facial," which does affect camera selection. In this case, it is likely that the camera will be selected to shoot a close-up of the character's face. In this invention, action types include, for example, facial, upper limb, lower limb, full-body, and idle character movements.

[0038] The start time and duration of an action can influence camera selection. Generally, camera lens c i The transition occurs at the beginning of the action. Actions with long durations (e.g., more than 10 seconds) require a combination of different camera shots. In other words, multiple cameras need to be selected sequentially.

[0039] The start and duration of a dialogue can influence camera selection. Actions within a dialogue scene require special handling. In long dialogue scenes, it's necessary to switch camera angles back and forth between the speakers, and over-the-shoulder cameras are often used.

[0040] In an automated film shooting environment, each camera includes a camera setting that defines the position of the camera relative to the corresponding character being filmed. Multiple default cameras are positioned at different distances and angles to film the character. These default cameras cover most camera placements required by film shooting guidelines, and each default camera is assigned a unique index that can be selected by a camera agent. The camera setting also includes: a shot type d(c) representing the distance between the camera and the corresponding character, a camera pan angle h(c), and a camera pitch angle.

[0041] The shot type d(c) is selected from extreme close shot (ECU), close shot (CU), medium shot (MS), full body shot (FS), and long shot (LS), and is quantified as a number from 0 to 4. When calculating the difference between two camera settings, the number corresponding to the distance is subtracted. The camera pan angle h(c) is between 0° and 360°. The camera pitch angle p(c) is between -15° and +15°.

[0042] In an automated film shooting environment, the camera c is selected by the camera agent. i t Camera setup and camera GC chosen by human directors t The difference between camera settings determines the reward function. The smaller the difference, the more similar the camera settings, and the higher the reward r. t The higher.

[0043] The reward function for lens type d(c) is defined as:

[0044]

[0045] The reward function for the camera's translation angle is defined as:

[0046]

[0047] The reward function for the camera's pitch angle is defined as:

[0048]

[0049] When the difference between the selected camera and the actual ground situation is less than a predefined threshold δ, an additional reward is added to accelerate the learning process. The larger δ is, the more allowable c becomes. i t With GC t The greater the deviation, the better. The additional reward function is defined as:

[0050]

[0051] The overall reward function is defined as the sum of all previous rewards:

[0052]

[0053] The camera agent starts with observation o0, selects a camera according to policy π(o), and maximizes reward R.

[0054] R is defined as a plot with T steps:

[0055]

[0056] Where r t γ is the reward at time t, and γ is the reward r. t A discount factor relative to the importance of future rewards, where 0 ≤ γ ≤ 1. The higher the value of γ, the more important the future rewards.

[0057] The goal of the RL algorithm is to find the optimal π that maximizes R. * . Optimal π * Defined as:

[0058]

[0059] in It indicates the plot.

[0060] The process involves adjusting the policy π according to the loss function. θ The parameters are iteratively updated to complete the loss function, which is based on the current policy π. current The calculated reward is the same as that calculated by the previous strategy π. old The error between the calculated rewards is measured. A proximal policy optimization (PPO) algorithm is used to solve the problem in equation (7), which is a variant of the advantage actor-critic (A2C) algorithm. The PPO algorithm combines a policy-based RL algorithm with a value-based RL algorithm. The actor neural network model acquires the state (or observation) and outputs the action according to the policy π(). The critic neural network model maps each state to the corresponding quality value of that state (i.e., the expected future cumulative discounted reward). (or discounted return) is used to indicate how good the camera choice is compared to the average camera choice for a particular observation. The advantage at time t is... Defined as:

[0061]

[0062] Where V is the learned state-value function, and rt t It is the strategy π at step t. current With strategy π old The ratio of parameter changes between them.

[0063] During training, the PPO algorithm updates the parameters of both the actor neural network and the critic neural network using backpropagation of two loss functions. Each iteration of the policy π is to maximize the total reward (i.e., Maximize policy π. However, it is necessary to avoid decrementing policy π in a single update. θ Things have changed.

[0064] The actor loss function L actor Defined as:

[0065]

[0066] The clip(·) function modifies the surrogate target by clipping the probability ratio, which eliminates the excitation that shifts rt out of the interval [1-, 1+]. Regardless of the camera selection, c i t Regardless of the obtained positive feedback value, the PPO algorithm updates the policy only based on the results within the interval. Therefore, it can utilize the desired parameters to adjust the policy π. θ Incremental updates are performed. However, there are no limits to penalties based on negative rewards.

[0067] The critic loss function L critic Defined as:

[0068]

[0069] This invention provides a script-to-video generation system. Figure 2 A schematic structural diagram of an exemplary script-to-video generation system according to some embodiments of the present invention is shown. Figure 2 As shown, computing device 200 may include processor 202, storage medium 204, display 206, communication module 208, database 210, peripheral devices 212, and one or more buses 214 coupling the devices together. Some devices may be omitted and other devices may be included.

[0070] Processor 202 may include any suitable one or more processors. Furthermore, processor 202 may include multiple cores for multithreaded or parallel processing. Processor 202 may execute sequences of computer program instructions or sequences of program modules to perform various processes, such as requesting user input with guidance prompts on a graphical user interface, generating / presenting animated videos, translating guidance prompts for editing and optimizing animated videos, etc. Storage medium 204 may include memory modules (such as ROM, RAM, flash memory modules, and erasable and rewritable memory) and mass storage devices (such as CD-ROMs, USB flash drives, and hard disks). When executed by processor 202, storage medium 204 may store computer program instructions or program modules for implementing various processes.

[0071] In addition, communication module 208 may include network devices for establishing connections via a communication network. Database 210 may include one or more databases for storing certain data (e.g., images, videos, animation materials) and for performing certain operations on the stored data (such as database searches and data retrieval).

[0072] Display 206 may include any suitable type of computer display device or electronic device display (e.g., CRT or LCD based device, touch screen, LED display). Peripheral devices 212 may include various sensors and other I / O devices, such as speakers, cameras, motion sensors, keyboards, mice, etc.

[0073] In operation, computing device 200 can perform a series of actions to implement the disclosed automatic film shooting method and framework. Computing device 200 can implement a terminal, a server, or a combination of both. As used herein, a terminal can refer to any suitable user terminal with some computing capabilities, including, for example, collecting user input prompts, displaying preview videos, and performing editing optimizations on videos. For example, a terminal can be a personal computer (PC), a workstation computer, a server computer, a handheld computing device (tablet computer), a mobile terminal (mobile phone or smartphone), or any other user-side computing device. As used herein, a server can refer to one or more server computers configured to provide certain server functions, such as determining camera configurations for shooting animated video, generating animated video based on the camera configuration, and editing animated video by finding the path with the minimum cost function in a graph model. A server may also include one or more processors executing computer programs in parallel. Terminals and / or servers can be configured to provide the structure and functionality for such actions and operations. In some embodiments, some portions of the actions may be performed on the server, while other portions of the actions may be performed on the terminal.

[0074] This invention also provides a method for generating a script into a video. Figure 3 A flowchart illustrating an exemplary script-to-video generation method according to a partial embodiment of the present invention is shown. The method can be performed by a computing device (such as...) Figure 2 The device 200 shown is used for implementation. Figure 3 As shown, the method includes the following procedures.

[0075] At S302, obtain the movie script.

[0076] Specifically, a movie script is a text script used to generate a video that corresponds to the movie script.

[0077] At S304, a list of actions is generated based on the movie script.

[0078] In some embodiments, natural language processing (NLP) is performed to analyze the movie script and generate a list of actions. The list of actions is a chronologically ordered list of actions, including actions designed to visually represent the movie script.

[0079] At S306, a stage performance is generated based on each action in the action list.

[0080] In some embodiments, by {p t The stage performance represented by |t=1,2,…,} is generated based on each action in the action list, where p t This refers to the character's stage performance at time t, where T is the total performance time. In some embodiments, this is related to each action a. t The corresponding stage performance is determined by The diagram shows that... It is a i The duration of the action, and as well as These are fixed values ​​derived from the action list.

[0081] In some embodiments, one or more cameras are positioned from a stage performance {p t Capture candidate videos {f} in |t=1,2,…,} t |t=1,2,…,}. The one or more cameras are tactically deployed and attached to each character in the stage performance.

[0082] At S308, observation information is extracted from the stage performance.

[0083] In some embodiments, observation information includes one or more of the following: character visibility, camera configuration, left-to-right order (LRO), action type, action start time and duration, dialogue start time and duration, or combinations thereof. In the specification, observation information and observation are used interchangeably.

[0084] Character visibility is determined by the size of each character in a frame relative to the total frame size, the weights of different characters, and the camera setup. Since each character is attached to more than one camera, some cameras may be obstructed by obstacles, and some characters may become blurry from the view of some cameras.

[0085] When switching cameras, the configuration of the previously selected camera can influence the choice of subsequent cameras. For example, in dialogue scenes, the lens reversal rule is often used. In this case, the configuration of the previously selected camera needs to be included in the observation... t middle.

[0086] LRO (Local Relationship) is a rule used to show the positional relationship between two or more characters in a frame. Including the positional relationship between two or more characters in a frame enforces the 180-degree rule used in the filmmaking industry. However, the LRO may differ depending on the camera used at point t.

[0087] Action type refers to the character's action type. For example, the action type could be "idle," which does not affect camera selection. In another example, the action type is "facial," which does affect camera selection. In this case, it is likely that the camera will be selected to shoot a close-up of the character's face. In this invention, action type includes character movement, such as facial, upper limb, lower limb, full body, and idle.

[0088] The start time and duration of an action can influence camera selection. Generally, camera lens c i The transition occurs at the beginning of the action. Actions with long durations (e.g., more than 10 seconds) require a combination of different camera shots. In other words, multiple cameras need to be selected sequentially.

[0089] The start and duration of a dialogue can influence camera selection. Actions within a dialogue scene require special handling. In long dialogue scenes, it's necessary to switch camera angles back and forth between the speakers, and over-the-shoulder cameras are often used.

[0090] At S310, a camera agent trained with a reinforcement learning algorithm is used to select a camera based on observation information.

[0091] In some embodiments, the selected camera includes camera settings that define the position of the camera relative to the character being filmed. Camera selection may include determining camera settings for one or more cameras to be used in one or more scenes and / or one or more plot points as needed. When selecting a camera based on observation information, a camera agent is used to select the camera based on the observation information and a camera selection strategy. The camera selection strategy is pre-configured at the beginning and iteratively updated using a reinforcement learning algorithm.

[0092] In some embodiments, after selecting a camera, the method further includes evaluating the quality of the selected camera in response to the quality of the selected camera being lower than or equal to a pre-configured threshold, changing the selected camera based on camera selection input provided by a human director or user through a human director interface or graphical user interface, and using the changed camera as ground-based live data to train a camera agent and update the camera selection strategy.

[0093] In some embodiments, when updating the camera selection strategy, the method uses a reward function to calculate a reward based on the selected camera and the changed camera, and uses the reward to update the camera selection strategy.

[0094] In some embodiments, the camera settings include: lens type d(c), which represents the distance between the camera and the character being filmed, where d(c)∈[0,1,2,3,4] represents an extremely close-up shot, a medium shot, a full-body shot, and a long shot; h(c), indicating a translation angle ranging from 0° to 360°; and p(c), indicating a pitch angle ranging from -15° to 15°.

[0095] In some embodiments, the reward function includes a lens-type reward function, a translation angle reward function, a pitch angle reward function, and additional rewards.

[0096]

[0097]

[0098]

[0099] The camera selected at time t, i is the camera index, and gc t δ is the camera that changes at time t, and δ is a pre-configured threshold.

[0100] In some embodiments, for a system with T steps The fragments t = 0, 1, ..., T use the aforementioned reward, where r t γ is the reward at time t, and γ is the reward r. t A discount factor that limits the importance of future rewards, and 0 ≤ γ ≤ 1.

[0101] In some embodiments, the method further includes: finding the optimal camera selection strategy π that maximizes the reward R for a plot with T steps. * : Where π represents the camera selection strategy.

[0102] In some embodiments, the optimal camera selection strategy is determined. * This includes iteratively updating the parameters of the camera selection strategy π based on a loss function L, which is applied using the Approximate Policy Optimization (PPO) algorithm with the current camera selection strategy π. current The calculated reward estimate is compared with the previous camera selection strategy π. previousThe error between the calculated reward estimates is measured.

[0103] In some embodiments, the current camera selection strategy π is evaluated by calculating the advantage at time t. current Compared to previous camera selection strategy π previovs Improvements between: Where V is the learned state-value function, o t It is the observation information at time t and rt t The current camera selection strategy π at time t current Compared to previous camera selection strategy π previous The ratio of parameter changes between them.

[0104] In some embodiments, the PPO algorithm updates the parameters of both the actor neural network and the critic neural network via backpropagation based on the following two loss functions: as well as Where θ indicates whether it is used before the camera selects a policy or previously the camera selects a policy, clip(·) modifies the alternative target by clipping the probability ratio of removing the stimuli used to move rt outside the interval [1-,1+].

[0105] At S312, use the selected camera to capture video of the stage performance.

[0106] In some embodiments, the selected camera may be a camera agent or a camera selected by a human director or user through a human director interface or graphical user interface. The selected camera is used to capture video of the stage performance.

[0107] Output the video at S314.

[0108] In some embodiments, the output is a video of a captured stage performance. The videos of the stage performances, based on a motion list generated from a movie script, are combined into a movie.

[0109] In embodiments of the invention, the camera agent is trained by capturing the behavior of a human director using an automated film shooting system. After training the camera agent with a sufficient amount of training data using a reinforcement learning algorithm, the camera agent is able to select cameras similar to those used by a human director, and improves the quality of the produced film compared to methods based solely on aesthetic and fidelity models.

[0110] The following examples illustrate the advantages of the RT2A framework.

[0111] In some embodiments, experimental setup and execution details of the disclosed RL automatic cinematography method are shown. The automatic cinematography environment and camera agent are implemented using OpenAI Gym. The training process based on an NVIDIA GTX 2080TI GPU takes approximately 38 hours. To evaluate the advantages of the RL automatic cinematography model, comparisons are made with the RT2A method, the aesthetic model method, and the aesthetic model + fidelity model method. Performance is evaluated in two aspects: 1) the comparison between the camera agent generated by the RT2A method and the camera agent generated by the reference algorithm. Camera placement 1) Compare the visual quality of the videos generated by RT2A and the reference algorithm; 2) Compare the visual quality of the videos generated by RT2A and the reference algorithm.

[0112] The differences in camera placement in the experimental environment were shown by sampling cameras generated by different methods every second. The performance of the algorithm was measured by the percentage of accepted cameras, referred to as the acceptance rate, by comparing the physical distance between the camera placement of each algorithm and the actual ground situation (placement manually input by a human director) and multiple acceptance rate thresholds (i.e., accepting the placement if the physical distance is less than the acceptance rate threshold). The physical distance was calculated based on equations (1), (2), and (3). The results are as follows: Figure 5 As shown in the diagram. The RT2A algorithm consistently outperforms the reference algorithm, and the gain in acceptance rate is significant. Figure 4 As shown in the image, a visual comparison of frames generated by various algorithms demonstrates that the RT2A algorithm is more similar to the camera selection of a human director than the reference algorithm. Therefore, the RT2A algorithm is able to effectively mimic the behavior of a human director's camera selection model after the camera agent has been trained with a sufficient amount of training data.

[0113] A complete shot is viewed continuously and uninterruptedly through a single camera. The number of shots used in a single scene (i.e., average shot length) is also an important indicator of the director's shooting style. Table 1 below shows the number of shots used by the RT2A automatic cinematography method compared to the reference method in several different scenes selected from the test dataset. The results show that the number of shots used by the RT2A method in each test scene is closer to real-world conditions than the reference method. Visual results also support this conclusion when comparing frames generated from the aesthetic model method with those generated from the RT2A method. Although rule-based methods (such as the aesthetic model method) can be optimized by configuring multiple rules to mimic human directors, the challenge of tuning the weights of various parameters and cost functions is much more complex than that of the RT2A method.

[0114] Table 1 below shows the differences in the number of shots between the RT2A method and the reference method compared to the camera placement chosen by the human director. The results show the number of shots used by the different methods in a single scene (and the percentage difference compared to the shots chosen by the human director).

[0115] Table 1

[0116] script aesthetics Aesthetics + Fidelity RT2A Human Director 1 32(33％) 35(46％) 27(13％) 24 2 50(150％) 43(115％) 28(40％) 20 3 45(95％) 40(74％) 31(40％) 23 4 37(146％) 39(160％) 21(35％) 15 5 15(150％) 13(116％) 8(33％) 6

[0117] The following sections demonstrate how the reference model can achieve similar visual results by adding a corresponding cost function or adjusting the weights. Data-driven RT2A methods offer advantages over manually creating numerous rules.

[0118] Typically, every new scene begins with a wide-angle shot. The function becomes clear: to establish an overall perspective before cutting into specific parts. The wide-angle shot, as an initiating shot, is used to set a specific tone and mood for what the viewer is about to see. Figure 6 This shows an exemplary frame generated by the aesthetic model (left) and the RT2A model (right). Figure 6 As shown, compared to the reference method, the RT2A method learns the shot language better and uses a build-up shot at the beginning of the scene. This is possible if the reference algorithm expects to achieve similar results (i.e., a new cost function used to determine whether the first few frames need to be included in the optimization framework) and if the characters captured by the LS shot need to occupy the least amount of space in the frame compared to other shot types. Therefore, it is necessary to minimize the weight of character visibility in the cost function.

[0119] Over-the-shoulder shots are widely used in dialogue scenes to help the audience understand the relationships between characters and to convey dramatic tension. They are also used to assemble reverse shots within dialogue scenes. Figure 7 This shows an example dialogue frame generated by the aesthetic model (left) and the RT2A model (right). Figure 7 As shown, the RT2A method successfully learned camera language and used over-the-shoulder shots in dialogue scenes. To achieve similar results using the reference method, a new cost function needs to be added to determine whether dialogue actions have sufficient duration for switching between shots. Additionally, during the optimization process, the weights of the camera position used for over-the-shoulder shots need to be modified to relax the requirement of capturing character actions from behind.

[0120] Some actions of certain characters need to be filmed from close range and at appropriate angles so that the audience can better understand the content. Figure 8 This shows an exemplary single motion frame generated by the aesthetic model (left) and the RT2A model (right). Figure 8As shown, the frames captured by the camera in the reference model do not correctly depict the action of "inspecting the object." To achieve similar results with the aesthetic model approach, the weights of different camera configurations need to be manually modified for specific actions based on the interpretation of the story.

[0121] In embodiments of the invention, the creativity and imagination demonstrated through the use of cinematic language in filmmaking requires significant expertise and skill. If automated filmmaking algorithms could learn from professional directors and thus mimic the cinematic language used in successful films, it would benefit novice artists. However, a major obstacle to achieving this is the lack of a sufficient amount of accurate training data. The RT2A framework, with the support of human directors, generates accurate data for training automated filmmaking systems. By learning cinematic language from human directors, the RT2A method is able to select the correct shot types and camera angles for the automated filmmaking process.

[0122] Specific examples are used in this specification to illustrate the principles and embodiments of the invention. The description of the embodiments is intended to aid in understanding the method and core inventive concept of the invention. Furthermore, those skilled in the art can modify or alter the specific embodiments and scope of this application based on the embodiments of the invention. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A script-to-video generation method, the method being used in a computing device, characterized in that, include: Obtain the film script; Generate an action list based on the video script; A stage performance is generated based on each action in the action list; Extract observation information from the stage performance; A camera agent trained with a reinforcement learning algorithm is used to select a camera based on the observation information, wherein the camera includes camera settings that limit the position of the camera relative to the character being filmed by the camera; Use the selected camera to capture video of the stage performance; as well as Output the video; The camera selection using the camera agent is also based on a camera selection strategy, which is pre-configured at the beginning and iteratively updated by the reinforcement learning algorithm. The updated camera selection strategy includes: A reward function is used to calculate the reward based on the selected camera and the modified camera; and Use the rewards to update the camera selection strategy; The camera settings of the camera include: Lens type The lens type indicates the distance between the camera and the subject being filmed, wherein Indicates camera, and The lens type of the camera is selected from ultra-close-up lens, close-up lens, medium shot lens, full-body lens, and long shot lens; Translation angle The translation angle is within the range of 0° to 360°; and Pitch angle The pitch angle is within the range of -15° to 15°; The reward function is determined based on the difference between the camera settings of the camera selected by the camera agent and the camera settings of the camera selected by the human director, and when calculating the difference, a number corresponding to the distance is subtracted.

2. The method according to claim 1, characterized in that, After selecting the camera, the following are also included: Evaluate the quality of the selected camera; In response to the selected camera's quality being lower than or equal to a pre-configured threshold, the selected camera is changed based on camera selection input provided by the user via a graphical user interface; and The camera agent is trained and the camera selection strategy is updated using the modified camera as ground real-time data.

3. The method according to claim 1, characterized in that, in: The reward function includes a reward function for the lens type, a reward function for the translation angle, a reward function for the pitch angle, and additional rewards; The reward function for the lens type is: ; The reward function for the translation angle is: ; The reward function for the pitch angle is: ; The additional reward is ;and It is in time The selected camera, It is the index of the camera. gc t It is in time The camera was changed, and δ It is the pre-configured threshold.

4. The method according to claim 1, characterized in that, The use of the rewards to update the camera selection strategy includes: For a process with T steps The segment uses the reward to update the camera selection strategy, wherein It is time The rewards mentioned above, The reward A discount factor that is limited by the importance of future rewards, and .

5. The method according to claim 4, characterized in that, Also includes: Find the reward for a scenario with T steps. Maximizing the best camera selection strategy .

6. The method according to claim 5, characterized in that, in: Find the optimal camera selection strategy Including based on the loss function Camera selection strategy The parameters are iteratively updated, and the loss function is optimized using an approximate policy selection algorithm with the current camera. The calculated reward estimate compared to using the previous camera selection strategy The error between the calculated reward estimates is measured.

7. The method according to claim 6, characterized in that, in: By calculating time Advantages of location To evaluate the current camera selection strategy Compared with the previous camera selection strategy Improvements between: ， in It is the state-value function of learning. It is time The observation information at the location and It is time The current camera selection strategy described at the location Compared with the previous camera selection strategy The ratio of parameter changes between them.

8. The method according to claim 7, characterized in that, in: The approximation strategy optimization algorithm updates the parameters of both the actor neural network and the critic neural network via backpropagation based on the following two loss functions: ； as well as ； in This indicates whether it is used for the current camera selection pre-strategy or the previous camera selection strategy. By removing the used to Move to interval The probability of external stimuli is compared to the pruning of alternative targets.

9. The method according to claim 1, characterized in that, in: The observation information includes character visibility, camera configuration, left-to-right order, action type, action start time and duration, dialogue start time and duration, or combinations thereof.

10. A script-to-video generation system, characterized in that, include: The memory stores program instructions; as well as A processor, coupled to the memory and configured to execute the program instructions to: Obtain the film script; Generate an action list based on the video script; A stage performance is generated based on each action in the action list; Extract observation information from the stage performance; A camera agent trained with a reinforcement learning algorithm is used to select a camera based on the observation information, wherein the camera includes camera settings that limit the position of the camera relative to the character being filmed by the camera; Use the selected camera to capture video of the stage performance; as well as Output the video; The camera selection using the camera agent is also based on a camera selection strategy, which is pre-configured at the beginning and iteratively updated by the reinforcement learning algorithm. The updated camera selection strategy includes: A reward function is used to calculate the reward based on the selected camera and the modified camera; and Use the rewards to update the camera selection strategy; The camera settings of the camera include: Lens type The lens type indicates the distance between the camera and the subject being filmed, wherein Indicates camera, and The lens type of the camera is selected from ultra-close-up lens, close-up lens, medium shot lens, full-body lens, and long shot lens; Translation angle The translation angle is within the range of 0° to 360°; and Pitch angle The pitch angle is within the range of -15° to 15°; The reward function is determined based on the difference between the camera settings of the camera selected by the camera agent and the camera settings of the camera selected by the human director, and when calculating the difference, a number corresponding to the distance is subtracted.

11. The system according to claim 10, characterized in that, After selecting the camera, the processor is further configured to: Evaluate the quality of the selected camera; In response to the quality of the selected camera being lower than or equal to a pre-configured threshold, the selected camera is changed according to the camera selection input provided by the user through the graphical user interface; as well as The camera agent is trained and the camera selection strategy is updated using the modified camera as ground real-time data.

12. The system according to claim 10, characterized in that, in: The reward function includes a reward function for the lens type, a reward function for the translation angle, a reward function for the pitch angle, and additional rewards; The reward function for the lens type is: ; The reward function for the translation angle is: ; The reward function for the pitch angle is: ; The additional reward is ; and It is in time The selected camera, It is the index of the camera. gc t It is in time The camera was changed, and δ It is the pre-configured threshold.

13. The system according to claim 10, characterized in that, When using the reward to update the camera selection strategy, the processor is further configured to: For a process with T steps The segment uses the reward to update the camera selection strategy, wherein It is time The rewards mentioned above, The reward A discount factor that is limited by the importance of future rewards, and .

14. The system according to claim 13, characterized in that, The processor is further configured to: Find the reward for a plot with T steps. Maximizing the best camera selection strategy .

Images