Method and device for generating movies from scripts

Abstract

The invention discloses a method and a device for generating a movie from a script. The method comprises the following steps: acquiring a movie script, generating a video according to the movie script, optimizing the generated video until a passing condition is met, and outputting the generated video.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and more particularly to a method and apparatus for generating movies from scripts. Background Technology

[0002] In the traditional film industry, scriptwriting (i.e., creating a film script) and filmmaking are completely separate processes. The emerging Write-A-Movie technology automatically generates video from the film script, significantly improving filmmaking efficiency. However, automatically generated video often fails to fully reflect the content of the film script.

[0003] This invention provides a script-to-movie generation method that incorporates a novel evaluation mechanism that combines the comprehensibility of the visual presentation of the movie script with compliance with cinematography guidelines. Therefore, the script-to-movie generation process is mapped as an optimization problem to improve the quality of automatically generated videos. Furthermore, dynamic programming is incorporated into solving this optimization problem to reduce computational complexity and shorten film production time. Summary of the Invention

[0004] One aspect of the present invention provides a method for generating a movie from a script on a computer device. The method includes: acquiring a movie script; generating a video based on the movie script; optimizing the generated video until a passing condition is met; and outputting the optimized video.

[0005] Another aspect of the present invention provides an apparatus for generating a movie from a script. The apparatus includes a memory storing program instructions and a processor coupled to the memory, the processor being configured to execute the program instructions to: acquire a movie script; generate a video based on the movie script; optimize the generated video until a passing condition is met; and output the optimized video.

[0006] The aforementioned apparatus, wherein the processor is further configured to:

[0007] A first action list is generated based on the movie script;

[0008] Generate a stage performance based on the actions in the first action list; and

[0009] The stage performance was filmed using a camera.

[0010] The aforementioned apparatus, wherein the processor is further configured to:

[0011] Evaluate the total aesthetic distortion value of the video, which was captured by a camera from the stage performance;

[0012] A second list of actions is generated based on the video, which is captured by a camera from the stage performance;

[0013] Determine the fidelity error E between the first action list and the second action list; and

[0014] The camera settings and character performances are iteratively optimized to minimize the total aesthetic distortion D, thereby satisfying the pass condition, wherein the pass condition includes satisfying that the fidelity error E is less than or equal to a pre-set fidelity error threshold Th. E Or minimize the number of iterations until a pre-set counting threshold is reached.

[0015] The aforementioned apparatus, wherein:

[0016] Each action in the first action list and the second action list has attributes, including subject, action, object, action duration, subject start position, subject end position, subject emotion, and action style.

[0017] The aforementioned apparatus, wherein:

[0018] The first action list is represented by an action list arranged in chronological order {a i |i=1,2,…,N}; and

[0019] The second action list consists of an action list {a′} arranged in chronological order. i |i=1,2,…,N} represents;

[0020] Among them, a i This represents the i-th action object, which includes information about one or more virtual characters in a stage performance scene; a′ i It is the i-th action object, which includes information about one or more virtual characters in the stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance.

[0021] The aforementioned apparatus, wherein:

[0022] The stage performance is used {p t |t=1,2,…,T} represents, where p t The stage performance of the character at time t, where T is the total performance time; and

[0023] Corresponding to a i The stage performance mentioned above is by It means that among them It is action a i Duration, and From the action list {ai The fixed values ​​derived from |i=1,2,…,N}.

[0024] Other aspects of the invention include what those skilled in the art would understand from the specification, claims and drawings. Attached Figure Description

[0025] The following figures are embodiments based on the present invention and are examples for illustrative purposes only, and are not intended to limit the scope of the invention.

[0026] Figure 1 A functional schematic diagram of a script-to-movie generation device according to an embodiment of the present invention is shown.

[0027] Figure 2 A schematic diagram of a script-to-movie generation device according to an embodiment of the present invention is shown.

[0028] Figure 3 A flowchart illustrating a method for generating a movie from a script according to an embodiment of the present invention is shown;

[0029] Figure 4A and Figure 4B A schematic diagram showing the camera placement position according to an embodiment of the present invention is provided;

[0030] Figure 5 A functional schematic diagram of another script-to-movie generation device according to an embodiment of the present invention is shown. Detailed Implementation

[0031] Reference will now be made in detail to the embodiments of the invention illustrated in the accompanying drawings. Hereinafter, embodiments of the invention will be described with reference to the accompanying drawings. Where possible, the same reference numerals will be used in the drawings to refer to the same or similar parts. Obviously, the described embodiments are some, but not all, embodiments of the invention. Based on the disclosed embodiments, those skilled in the art can derive other embodiments consistent with the invention, all of which are within the scope of protection of the invention.

[0032] The "Write-A-Movie" technique is an adaptive, self-reinforcing framework for automatically generating video from a movie script. This invention provides a script-to-movie generation apparatus that leverages recent advances in natural language processing, computer cinematography, and video understanding. The automated workflow of this script-to-movie generation apparatus significantly reduces the time and knowledge required for the script-to-movie generation process. By incorporating a novel hybrid objective evaluation mechanism, the video generation process is mapped to an optimization problem aimed at generating better quality videos, simultaneously considering the comprehensibility of the movie script's visual presentation and compliance with cinematography guidelines. Dynamic programming can solve the optimization problem and serve as the optimal solution with the most efficient computational complexity. Therefore, the script-to-movie generation apparatus of this invention significantly accelerates the filmmaking process.

[0033] In the traditional film industry, scriptwriting and filmmaking are completely separate processes. With the recent advancements in artificial intelligence, a significant portion of the filmmaking process can now be accomplished by computers. Combining scriptwriting and filmmaking can bring direct benefits to all stakeholders. Scriptwriters can visualize and edit their work before submission. Producers can screen scripts by reviewing pre-visualized versions, eliminating the need to read hundreds of pages. The process of transforming a script into a film needs to meet two quality requirements. First, the output film must maintain a reasonable fidelity to the script. Second, the output film must adhere to cinematic aesthetics and principles.

[0034] Therefore, a mechanism is needed to evaluate the fidelity of the generated video to the corresponding film script, and to provide feedback to the animation and cinematography processes for further improvement when the evaluated fidelity falls below acceptable standards. Thus, the computer cinematography process needs to consider not only aesthetics but also the ability to perceive fidelity to the film script. While today's state-of-the-art video understanding capabilities are not yet sufficient for accurately evaluating the fidelity of the generated video to the film script, they are adequate for evaluating certain types of films with lower scene complexity and less demanding motion recognition.

[0035] In embodiments of the invention, the script-to-movie generation device automatically converts a movie script into a movie, such as an animated film. The script-to-movie generation device includes an arbitration mechanism supported by video understanding and natural language understanding technologies. The video understanding technology converts the generated video into a list of executed actions, and the natural language understanding technology converts the movie script into a list of expected actions, thus determining whether the generated video can be understood and the fidelity of the movie script. The evaluation results are then fed back to the stage performance to improve the quality of the generated video. Furthermore, aesthetic and fidelity requirements are combined within a unified evaluation framework, and the video quality improvement process is mapped to an optimization problem: adjusting camera settings and character motion settings to achieve the desired video quality. The optimization problem is designed to be solvable through dynamic programming to reduce computational complexity.

[0036] Figure 1 This is a functional diagram of a script-to-movie generation device according to an embodiment of the present invention. Figure 1 As shown, the script (i.e., the movie script) is input into the action list generation process to generate a chronologically ordered action list. The action list includes actions composed of {a} i |i=1,2,…,N} represents the list of expected actions, where a i This represents the i-th action object, which includes information about one or more virtual characters in a stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance. The action list {a i |i=1,2,…,N} is a set of action objects used to generate character performances during a stage performance, arranged in chronological order and without overlap. For example, the characters are virtual characters in an animated film. In some embodiments, multiple characters execute action objects simultaneously, allowing a single action object to contain multiple characters within the same scene. For example, two people are fighting, or a mother is hugging her daughter.

[0037] In some embodiments, the action list {a i Each action in |i=1,2,…,N} includes attributes such as subject, action, object, action duration, subject start position, subject end position, subject mood, and action style. The subject start position is the subject's position at the start of the action. The subject end position is the subject's position at the end of the action. The default value for subject mood is neutral. The default value for action style is neutral (i.e., no preferred style). Users can select an action style from the following: self-action (when the camera is focused on the subject), multiple actions (when the camera is focused on both the subject and the object simultaneously), and environmental actions (when the camera is focused on the environment surrounding the subject, such as the view around the action).

[0038] refer to Figure 1 The action list {a i The input list of actions {a, i = 1, 2, ..., N} is used to generate the video during the stage performance. i |i=1,2,…,N} is converted into corresponding stage performance data, and the stage performance data is represented by {p t |t=1,2,…,T} represents, where p t This represents the character's stage performance at time t, where T is the total performance time. Stage performance data in the 3D scene {p t The sequence |t = 1, 2, ..., T} is continuous. However, due to computational limitations, continuous information is converted into discrete information for camera optimization. Stage performance data is recorded as p for each time unit (e.g., half a second). t One of the elements. In this specification, stage performance, stage performance data, and character performance are used interchangeably.

[0039] For the action list {a i For each action in |i=1,2,…,N}, the corresponding performance data is provided by… It means that among them It is action a i Duration, and From the action list {a i |i=1,2,…,N} is a fixed value derived from the sequence. In some embodiments, different action objects overlap each other. For example, two events occur simultaneously and both need to be shown to the audience. In various scenes, all cameras capture all views of all characters from all angles. Then, a camera optimization process calculates the optimal camera path to capture the character performance. The camera optimization process uses the stage performance data {p t |t=

[0040] Using {c1, 2, ..., T} as input, calculate the optimal camera settings for each time t, where the optimal camera settings are determined by {c1, 2, ..., T}. t |t=1,2,…,T} represents the camera setup. In some embodiments, the camera setup includes at least one of a camera path and camera parameters. A camera optimization process is performed on discrete data from time t to T. The camera setup {c t |t=1,2,…,T} represents all allowed camera selections for each time slot from time t to T, and for each time slot, only one camera can be selected during the camera optimization process.

[0041] In some embodiments, the camera optimization process identifies the camera path with the minimum distortion D. The distortion D is calculated based on a cost function derived from the cinematography guidelines. Then, based on the optimized camera settings, stage performance data {p} is generated. t The video corresponding to |t=1,2,…,T}, the video is composed of {f t |t=1,2,…,T} represents.

[0042] Because the camera optimization process minimizes errors solely from an aesthetic perspective, the script-to-movie generation device of this invention also considers the fidelity of the generated video to the movie script. On one hand, this fidelity needs to be evaluated in objective measurements. On the other hand, the measurement of this fidelity needs to be incorporated into the camera optimization process to minimize aesthetic distortion. Therefore, the generated video is evaluated and output after meeting a pass condition, which ensures the quality of the output video. If the aesthetics or fidelity of the generated video is determined to be unacceptable, the camera optimization process or the stage performance process is iterated once or multiple times to generate another video with adjusted camera settings and / or adjusted character performance.

[0043] In some embodiments, when a camera is identified as a cause of the generated video failing to meet the pass criteria, the corresponding cost associated with the identified camera is maximized as a subsequent iteration of the camera optimization process or the stage performance process. In other words, the identified camera is removed from the cameras filming the stage performance.

[0044] In some embodiments, the video understanding process will select candidate videos {f} t |t=1,2,…,T} is used as input to generate another action list, which includes a list of actions that have been performed. The action list identified by the video understanding process is composed of {a′ i |i=1,2,…,N} represents, where a′ i Here, is the i-th action object, which includes information about one or more virtual characters in a stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance. Then, the arbitration process compares the action list {a} i |i=1,2,…,N} and the action list {a′ i The fidelity error E is obtained by calculating the action similarity between |i=1,2,…,N}. The fidelity error E is used to quantify the consistency between visual perception and textual meaning, where visual perception refers to the visual perception of the generated video, and textual meaning refers to the textual meaning of the movie script. Simultaneously, in evaluating candidate videos {f... tWhen |t=1,2,…,T}, the arbitration process also considers the total aesthetic distortion D. If the candidate video {f t When the total aesthetic distortion D and fidelity error E given by |t=1,2,…,T} are unacceptable, a wider range of acceptable camera settings and acceptable settings for character action performance will be considered for recalculation and re-arbitration. This process is repeated iteratively until the candidate video {f} is found to be acceptable. t |t=1,2,…,T} is qualified or the iteration count reaches the preset counting threshold.

[0045] In some embodiments, after comparing the action list {a i |i=1,2,…,N} and the action list {a′ i After determining the action similarity among |i=1,2,…,N}, the action list {a′ i All actions in the group |i=1,2,…,N} are sorted according to their similarity. When optimization of the stage performance is needed, the action with the highest similarity is selected from the sorted list and remade.

[0046] Figure 5 A functional schematic diagram of another script-to-movie generation device according to an embodiment of the present invention is shown. Figure 5 The shown are the generation devices from script to movie and Figure 1 The devices shown are similar to those used for generating movies from scripts. The difference lies in whether the video understanding and arbitration processes are omitted. Specific implementation methods can be found in the previous description and will not be repeated here.

[0047] In embodiments of the present invention, the script-to-movie generation device leverages recent advances in natural language processing, computational cinematography, and video understanding to significantly reduce the time and knowledge required for the script-to-movie generation process. By incorporating a novel hybrid objective evaluation mechanism, the video generation process is mapped to an optimization problem aimed at generating higher-quality videos, which simultaneously considers the comprehensibility of the film script's visual presentation and compliance with cinematography guidelines. Dynamic programming can solve the optimization problem and serve as the optimal solution with the most efficient computational complexity. Therefore, the script-to-movie generation device of the present invention greatly accelerates the filmmaking process.

[0048] Figure 2 A schematic diagram of the structure of a script-to-movie generation device according to some embodiments of the present invention is shown. For example... Figure 2 As shown, computing device 200 includes a processor 202, storage medium 204, display 206, communication module 208, database 210, and peripheral devices 212, as well as one or more buses 214 coupling the devices together. Some devices may be omitted and others may be included.

[0049] The processor 202 can be any suitable processor or multiple processors. Furthermore, the processor 202 can include multi-core processors for multi-threaded or parallel processing. The processor 202 can execute sequences of computer program instructions or program modules to perform various processes, such as requesting director's prompts from the user on a graphical user interface, generating / rendering animated videos, translating director's prompts for editing and optimizing animated videos, etc. The storage medium 204 can include memory modules such as read-only memory (ROM), random access memory (RAM), flash memory modules, erasable and rewritable memory, and mass storage devices such as CD-ROM, USB flash drives, hard disks, etc. When the processor 202 runs the storage medium 204, the storage medium 204 can store computer program instructions or program modules for implementing various processes.

[0050] Furthermore, the communication module 208 may include a network device for establishing connections via a communication network. The database 210 may include one or more databases for storing data (e.g., images, videos, animation materials, etc.) and performing operations on the stored data, such as retrieving data from the database and retrieving data from the database.

[0051] The display 206 may include any suitable type of computer display device or electronic device display (e.g., cathode ray tube (CRT) or liquid crystal (LCD) based device, touch screen, light-emitting diode (LED) display, etc.). The peripheral device 212 may include various sensors and other input / output (I / O) devices, such as speakers, cameras, motion sensors, keyboards, mice, etc.

[0052] In operation, the computing device 200 can perform a series of actions to implement the disclosed automatic cinematography method and framework. The computing device 200 can run a terminal or a server, or a combination of both. The terminal used herein can refer to any suitable user terminal with certain computing capabilities, including collecting director's prompts from user input, displaying preview videos, and editing and optimizing videos. For example, the terminal can be a personal computer (PC), a workstation computer, a server computer, a handheld computing device (tablet), a mobile terminal (phone or smartphone), or any other user-end computing device. The server referred to herein can refer to one or more server computers configured to provide certain server functions, including determining camera settings for shooting animated video, generating the animated video based on the camera settings, and editing the animated video by finding the path with the minimum cost function in an image model. The server may also include one or more processors to execute computer programs in parallel. The terminal and / or the server can be configured to provide structure and functionality for the above actions and operations. In some embodiments, some actions can be performed on the server, while other actions can be performed on the terminal.

[0053] The present invention also provides a method for generating a movie from a script. Figure 3 A flowchart illustrating a method for generating a movie from a script, as described in some embodiments of the present invention, is shown. The method can be performed by, for example... Figure 2 The computing device 200 shown is implemented as follows. Figure 3 As shown, the method includes the following steps.

[0054] S302, Get the movie script.

[0055] Specifically, the movie script is used to generate a video corresponding to the movie script.

[0056] S304, Generate a video based on the movie script.

[0057] Specifically, generating video based on the movie script includes generating a first list of actions based on the movie script, generating a stage performance based on each action in the first list of actions, and filming the stage performance using one or more cameras. In some embodiments, the first list of actions is a chronologically ordered list of actions, including actions designed to demonstrate a visual presentation of the movie script. The first list of actions consists of {a i |i=1,2,…,N} represents, where a i The i-th action object is represented, which includes information about one or more virtual characters in a stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance.

[0058] In some embodiments, the stage performance uses {p t |t=1,2,…,T} indicates that the stage performance is based on the first action list {a i Each action in |i=1,2,…,N} generates p, where p t Let t be the stage performance of the character at time t, where T is the total performance time. In some embodiments, this corresponds to each action a. i stage performance It means that among them It is action a i Duration, and From the first action list {a i The fixed values ​​derived from |i=1,2,…,N}.

[0059] In some embodiments, one or more cameras capture images from the stage performance. t |t=

[0060] Candidate videos {f} of {1,2,…,T} t |t=1,2,…,T}。 In the stage performance, one or more cameras are deployed as planned and attached to each character.

[0061] S306, Optimize the generated video until the passing conditions are met. The optimization process can be performed based on the aesthetic evaluation and fidelity evaluation of the video.

[0062] Specifically, optimizing the generated video until a passing condition is met includes evaluating the total aesthetic distortion D of the video, which was filmed by one or more of the cameras from the stage performance; generating a second action list based on the video, which was also filmed by one or more of the cameras from the stage performance; determining the fidelity error E between the first action list and the second action list; and iteratively optimizing the camera settings and character performance to minimize the total aesthetic distortion D, thereby meeting the passing condition. The passing condition includes ensuring that the fidelity error E is less than or equal to a pre-configured fidelity error threshold Th. E Alternatively, minimize the number of iterations to reach a pre-configured counting threshold.

[0063] In some embodiments, the candidate video {f} is evaluated. t The total aesthetic distortion D of |t=1,2,…,T}, the candidate video being captured by one or more cameras from the stage performance {p t |t=1,2,…,T} to take photos.

[0064] In some embodiments, the total aesthetic distortion D includes camera settings {ct The visibility of role k in |t=1,2,…,T} is V(c t The visibility of the role V(c) t ) through calculation To evaluate, r k It is the ratio of the size of the character k in the current video frame to the total size of the current video frame. k This indicates how easily viewers notice character k in a video frame. When multiple characters appear in front of camera c... t When the camera is in the field of view, the camera c t The character associated with it is considered the most important character. I(c t k) by the camera c t And the role k determines that different combinations of different roles and different cameras are assigned different weights, I(c t (k) represents the camera c t The correlation between I(c) and the aforementioned role k. t A low value for k indicates that the role k is relative to the camera c. t The more important role, namely I(c) t The lower the value of k, the better the relationship between character k and camera c. t Its importance is even greater.

[0065] In some embodiments, the total aesthetic distortion D also includes character motion A(c) t The character action A(c) t This describes whether character k is acting at time t. Viewers are more likely to notice a character in motion. If character k is acting at time t, the camera c associated with that character k is more likely to be selected. t .

[0066] For example, A(c) t It can be obtained from the following formula:

[0067]

[0068] In some embodiments, the total aesthetic distortion D also includes the distortion for camera c. t Camera settings The camera setup Through calculation To evaluate, among which Represents the camera position. Represents the shooting direction. The action object at time t, and φ C () represents the distortion cost function set by the camera.

[0069] Different camera setups serve different purposes in filmmaking. For example, medium shots are most commonly used when a character is performing general actions. When a character is performing specific actions, such as panoramic shots, long shots, shots of the surrounding environment, and shots of the character, are often better choices. On the other hand, different actions may require the camera to be shot from different directions. For example, walking and running actions can be shot from the front and back of the character with minimal distortion. However, speaking actions may have more severe distortion when shot from the back of the character compared to shooting from the front and side. Therefore, the distortion of the camera setup depends on the time t from the action object (i.e., a). i The exported action type, and the action type from the camera c t The exported camera position p and shooting direction d.

[0070] In some embodiments, the total aesthetic distortion D also includes screen continuity S(c t ,c t-1 The screen continuity S(c) t ,c t-1 This includes a summary of the positional changes of each individual character in the current video frame. The screen continuity S(c) t ,c t-1 ) is calculated To evaluate, where p(k,c) t ) indicates that it is produced by camera c t The position of character k in the current video frame, p(k,c) t-1 ) indicates that it is produced by camera c t-1 The position of character k in the current video frame being captured; if character k appears in the camera c t In the view, v(k,c t ) = 1, otherwise v(k,c t ) = 0; φ S () is the penalty for changes in character position, which is related to the character position p(k,c) t ) and character position p(k,c t-1 The nonlinear function of the distance between ).

[0071] Visual-spatial continuity in video can prevent viewers from perceiving distortion. For example, cinematography guidelines include the 180-degree rule. The minimum penalty for changes in character position is 0, and as the character position p(k,c) changes... t ) and character position p(k,c t-1 The penalty increases with the distance between them. When character k appears in only one video frame, the maximum penalty of 1 is applied.

[0072] In some embodiments, the total aesthetic distortion D also includes motion continuity M(c t ,c t-1 The mobility continuity M(c) t ,c t-1 This includes changes in the character's movement direction, which are caused by changes in the camera's movement direction. t The movement continuity M(c) is caused by character actions before or after a view change. t ,c t-1 ) is calculated To evaluate, where m(k,c) t ) indicates that it is produced by camera c t The motion direction vector of the character in the current video frame, m(k,c) t-1 ) indicates that it is produced by camera c t-1 The direction vector of character motion in the current video frame being captured; φ M () represents the penalty for changes in the character's movement direction, which is related to the character's movement direction vector m(k,c). t ) and the character's movement direction vector m(k,c t-1 The penalty is a non-linear function of the difference between the motion direction vectors. As the angle between the motion direction vectors increases, the penalty increases. A maximum penalty of 1 is applied when character k appears in only one video frame.

[0073] In some embodiments, the total aesthetic distortion D also includes shot duration distortion. The lens duration distortion Through calculation To evaluate, among which φ is the average shot duration set for each scene, q is the maximum allowed shot duration, and φ is the maximum allowed shot duration. U () is a penalty for the duration of a video frame that changes the camera within the range [tq,…,t].

[0074] Shot duration is closely related to the level of viewer attention. Generally, the shorter the shot duration, the more intense the content in the video, and the easier it is to attract the viewer's attention. In some embodiments, an average shot duration is assigned to each scene in the shot duration distribution. In some other embodiments, shot duration configurations learned from existing films are used for various scenes in shot duration distributions.

[0075] After summarizing various degrees of distortion, the total aesthetic distortion is... ω0, ω1, ω2, ω3 and ω4 are values ​​between 0 and 1, and are the weights of each distortion component.

[0076] In some embodiments, a second list of actions is generated based on the stage performance. Specifically, one or more cameras capture the stage performance {p t Candidate videos {f} are captured in |t=1,2,…,T} t |t=1,2,…,T}. Then, based on the candidate video {f t The second action list is generated from |t=1,2,…,T}. The second action list is an action list arranged in chronological order, including a list of executed actions. The second action list is generated from {a′ i |i=1,2,…,N} represents, where a′ i It is the i-th action object, which includes information about one or more virtual characters in the stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance.

[0077] In some embodiments, a fidelity error E is determined between the first action list and the second action list, and camera settings and character performances are optimized to minimize the total aesthetic distortion D, such that the pass condition is met (e.g., the fidelity error E is less than or equal to a pre-set fidelity error threshold Th). E ).

[0078] Specifically, the similarity of actions between the first action list and the second action list is compared to obtain a fidelity error E. The fidelity error E is used to quantify the consistency between visual perception and textual meaning, where visual perception refers to the visual perception of the generated video, and textual meaning refers to the textual meaning of the movie script. Simultaneously, in evaluating the candidate video {f t When |t=1,2,…,T}, the total aesthetic distortion D is also considered. This applies when the fidelity error E is less than or equal to a pre-set fidelity error threshold Th. E At that time, the candidate video {f t |t=1,2,…,T} is qualified. If the candidate video {f t When the total aesthetic distortion D and fidelity error E given by |t=1,2,…,T} are unacceptable, a wider range of acceptable camera settings and acceptable settings for character action performance will be considered for recalculation and optimization. Then, the fidelity error E will be recalculated. This iteration is repeated until the candidate video {f} is found to be acceptable. t |t=1,2,…,T} is qualified or the iteration count reaches the preset counting threshold.

[0079] In some embodiments, the fidelity error E between the generated video and the movie script can be approximated and evaluated by the difference between a first action list and a second action list, wherein the second action list is derived from the candidate video through a video understanding process. The video understanding process reads the candidate video and outputs a list of actions identified from the candidate video. Generally, the video understanding process can perform as well as a human, and the action list generation process can thoroughly understand the movie script. Therefore, it is feasible to approximate the fidelity error E using action list difference comparison. The fidelity error is primarily caused by character performance or camera shooting processes. In the former case, the character performance does not trigger natural human intuition to reflect specific actions in the movie script. In the latter case, a view matching a specific meaning in the movie script is missing. In practical applications, errors may occur in the video understanding process and the action list generation process. However, in embodiments of the present invention, the possibility of errors in the video understanding process and the action list generation process is not considered.

[0080] In some embodiments, the action difference d i Used to represent two related actions a i and a′ i The difference lies in the textual descriptions. The arbitration process generates two vector forms of descriptions using the GloVe (Global Vectors for Word Representation) word embedding model, and then calculates the difference between the two vectors. Where G() is the GloVe word embedding model. Therefore, the fidelity error E can be expressed by the equation... Description. The function W() is defined as: when time t equals a i The start time, W(t) = d t Otherwise, W(t) = 0, then the above equation can be transformed into

[0081] In some embodiments, the camera settings are optimized to minimize overall aesthetic distortion D. Specifically, the camera placement is optimized for different lens sizes, different profile angles, and different camera heights. Multiple virtual cameras are placed around each bound character. Each camera maintains a relative position to the bound character.

[0082] Positioning a camera in three-dimensional space to capture video that satisfies two-dimensional constraints is a 7-degree-of-freedom problem, involving the camera's position, orientation, and focal length (i.e., lens size). In practical applications, 7-dimensional optimization can consume a significant amount of computational power. To simplify the problem without loss of generality, based on the camera positions in classic films, the 7-degree-of-freedom infinite search space is reduced to a countable number of discrete camera settings.

[0083] In some embodiments, only cameras with a maximum of two characters are considered, because shots with more characters in the view can often be replaced by multiple single-character shots. Consider shots with two characters in a toroidal model. Figure 4A and Figure 4B A schematic diagram illustrating the camera placement in some embodiments of the present invention is shown. A camera that maintains a relative position to the bound character during a stage performance is called a Point of View (POV) camera. The POV camera follows the head movement of the bound character.

[0084] In some embodiments, such as Figure 4A As shown, during the stage performance, each character was paired with 34 cameras. Each camera was labeled with an index number. The 34 cameras included 1 POV camera (index 0), 3 Close-up Shot (CS) cameras (indexes 1-3), 20 Medium Shot (MS) cameras (indexes 4-23), 2 Environment Medium Shot (MS-S) cameras (indexes 24-25), 4 Full Shot (FS) cameras (indexes 26-29), and 4 Long Shot (LS) cameras (indexes 30-33). The profile angle (i.e., shooting direction) of each camera was... Figure 4A The middle is indicated by a separate dashed arrow. Of the 34 cameras, 8 MS cameras (index 4-11) and 2 MS-S cameras are deployed at the character's line-of-sight height (e.g., ...). Figure 4B As shown), the relative positions are as follows: Figure 4A As shown (i.e., indicated by the dashed arrows at 0°, 60°, 90°, 120°, 180°, -120°, -90°, -60°); 6 MS cameras (index 12-17) are deployed at high angles to the character (e.g., Figure 4B (as described above), relative positions as follows Figure 4A As shown (i.e., indicated by the dashed arrows at 60°, 90°, 120°, -120°, -90°, -60°); another 6 MS cameras (index 18-23) are deployed at low angles to the character (e.g. Figure 4B(as described above), relative positions as follows Figure 4A As shown (i.e., indicated by the dashed arrows at 0°, 60°, 90°, 120°, 180°, -120°, -90°, -60°). Two MS-S cameras set to observe the environment in front of the character have the following... Figure 4A The outline angle shown.

[0085] In some embodiments, the Lagrange multiple method is used to relax the error constraints of identification, allowing the shortest path algorithm to be used to solve the problem. The Lagrange cost function is J. λ (c t ,a t )=D+λ·E, where λ is the Lagrange multiplier. If there exists λ * Make And E = Th E So {c * t ,a * t} is an equation The optimal solution is such that Therefore, solving the above equation is transformed into a simpler task: finding the Lagrange cost function J that minimizes the Lagrange cost function. λ (c t ,a t Find the optimal solution to the unconstrained problem and select appropriate Lagrange multipliers to satisfy the constraints.

[0086] In some embodiments, z k =(c k ,a k and cost function G T (z T-q ,…,z T The value is defined as the minimum fidelity error E and total aesthetic distortion D, up to the k-th video frame, where z k-q ,…,z t It is the decision vector from the (kq)th video frame to the kth video frame. Therefore, G T (z T-q ,…,z T ) represents the minimum sum of the fidelity error E and the total aesthetic distortion D of all video frames, therefore

[0087] In some embodiments, a key observation for deriving an effective algorithm is the fact that, given q+1 decision vectors z from the (kq-1)th video frame to the (k-1)th video frame...k-q-1 ,…,G k-1 Given the cost function z k-1 (z k-q-1 ,…,z k-1 For the next decision vector z k The choice is independent of the previous decision vectors z1, z2, ..., z k-q-2 The choice of . This means that the cost function can be recursively expressed as

[0088]

[0089] S308 outputs the optimized video.

[0090] Specifically, after performing the optimization process described in this invention on the video, the quality of the optimized video is improved. The optimized video is then output to conclude the process from script to movie.

[0091] The recursive expression of the cost function described above makes future steps in the optimization process independent of their past steps, which is the basis of dynamic programming. This problem can be transformed into a graph theory problem of finding the shortest path in a directed acyclic graph (DAG). The computational complexity of the algorithm is O(T×|Z|). q+1 (where Z is the list of actions {a}) i All actions described in |i=1,2,…,N} in the stage performance {p t The total number of available performances in |t=1,2,…,T}, where |Z| is the cardinality of Z), and the computational complexity directly depends on the value of q. In most cases, q is a small number, so the algorithm is much more efficient than exhaustive search algorithms with exponential computational complexity.

[0092] In embodiments of the present invention, the script-to-movie generation method leverages recent advances in natural language processing, computational cinematography, and video understanding to significantly reduce the time and knowledge required for the script-to-movie generation process. By incorporating a novel hybrid objective evaluation mechanism, the video generation process is mapped to an optimization problem aimed at generating better quality videos, which simultaneously considers the comprehensibility of the film script's visual presentation and compliance with cinematography guidelines. Dynamic programming can solve the optimization problem and serve as the optimal solution with the most efficient computational complexity. Therefore, the script-to-movie generation method of the present invention greatly accelerates the filmmaking process.

[0093] The principles and implementation of the invention are illustrated in the specification with specific examples. The descriptions of the embodiments are intended to aid in understanding the methods and core inventive concepts of the invention. Furthermore, those skilled in the art can make changes or modifications to the specific implementations and scope of this application based on the embodiments of the invention. Therefore, the content of this specification should not be construed as a limitation of the invention.

Claims

1. A video generation method, characterized in that, include: Get the video generated from the movie script; Based on the total aesthetic distortion D of the video, the video is adjusted to obtain the adjusted video.

2. The method according to claim 1, characterized in that, Based on the total aesthetic distortion D of the video, the video is adjusted to obtain the adjusted video, including: The camera settings and the stage performance corresponding to the film script are iteratively adjusted to minimize the total aesthetic distortion D, thereby obtaining a video that meets the passing conditions.

3. The method according to claim 1, characterized in that, The total aesthetic distortion D includes shot duration distortion. and / or The total aesthetic distortion D includes the motion continuity M(c) t ,c t-1 ); and / or The total aesthetic distortion D includes screen continuity S(c) t ,c t-1 ); and / or The total aesthetic distortion D includes character movement A(c) t ); and / or The total aesthetic distortion D includes the distortion for camera c. t Camera settings and / or The total aesthetic distortion D includes the camera settings {c t The visibility of role k in |t=1,2,…,T} is V(c t ); The lens duration distortion Through calculation To evaluate, among which φ is the average shot duration set for each scene, q is the maximum allowed shot duration, and φ is the maximum allowed shot duration. U () is a penalty for the duration of the shot of a video frame that changes the camera in the range [tq,…,t]. The mobility continuity M(c) t ,c t-1 This includes changes in the character's movement direction; The screen continuity S(c) t ,c t-1 This includes a summary of the positional changes of each individual character in the current video frame; The character action A(c) t Describe whether character k takes action at time t; The camera setup Through calculation To evaluate, among which Represents the camera position. Represents the shooting direction. The action object at time t, and φ C () represents the distortion cost function set by the camera.

4. The method according to claim 2, characterized in that, The method further includes: A first action list is generated based on the movie script, and a second action list is generated based on the video; Determine the fidelity error E between the first action list and the second action list; and The iterative adjustment of camera settings and the stage performance corresponding to the film script to minimize the total aesthetic distortion D, thereby obtaining a video that meets the passing conditions, includes: Iterative adjustments are made to camera settings and stage performance to minimize the total aesthetic distortion D, thereby obtaining a video that meets the passing conditions, wherein the passing conditions include ensuring that the fidelity error E is less than or equal to a pre-set fidelity error threshold Th. E ;or, Minimize the number of iterations until the count reaches a pre-set counting threshold.

5. The method according to claim 4, characterized in that, The step of generating a second action list based on the video includes: One or more cameras are used to capture the stage performance corresponding to the video. t Candidate videos {f} are captured in |t=1,2,…,T} t |t=1,2,…,T}; According to the candidate video {f t The second action list is generated by |t=1,2,…,T}.

6. The method according to claim 4, characterized in that: The fidelity error E is used to quantify the consistency between visual perception and text meaning, where visual perception refers to the visual perception of the generated video and text meaning refers to the text meaning of the movie script. Each action in the first action list and the second action list has attributes, including at least one of subject, action, object, action duration, subject start position, subject end position, subject emotion, and action style; The subject's starting position is the subject's position at the beginning of the action; the subject's ending position is the subject's position at the end of the action; the default value for the subject's emotion is neutral; the default value for the action style is neutral; the action style includes at least one of self-action, multiple actions, and environmental actions; The first action list consists of an action list {a} arranged in chronological order. i |i=1,2,…,N} represents; The second action list consists of an action list {a} arranged in chronological order. ′ i |i=1,2,…,N} represents; Among them, a i This represents the i-th action object, which includes information about one or more virtual characters in a stage performance scene; a ′ i It is the i-th action object, which includes information about one or more virtual characters in the stage performance scene; N is the total number of action objects performed by multiple characters in multiple scenes of the stage performance.

7. The method according to claim 4, characterized in that, Determining the fidelity error E between the first action list and the second action list includes: Determine that there are two related actions a i and a ′ i Differences in textual descriptions of action differences d i ; use Calculate the fidelity error E.

8. The method according to claim 7, characterized in that, The determination refers to two related actions a i and a ′ i Differences in textual descriptions of action differences d i ,include: calculate To determine the action differences between the text descriptions of the first action list and the second action list, where G() is a GloVe word embedding model; as well as, Define a function W(t) when time t equals a i At the start time, W(t) = d t Otherwise W(t) = 0; and calculate Where T represents the total performance time.

9. The method according to claim 4, characterized in that, The stage performance is used {p t |t=1,2,…,T} represents, where p t The stage performance of the character at time t, where T is the total performance time; and Corresponding to a i The stage performance mentioned above is by It means that, among them It is action a i Duration, and From the action list {a i The fixed values ​​derived from |i=1,2,…,N}; as well as, For {p t |t=1,2,…,T}, the adjusted camera settings are determined by {c t |t=1,2,…,T} represents; and Corresponding to {p t The video of |t=1,2,…,T} is generated by {f t |t=1,2,…,T} represents.

10. The method according to claim 9, characterized in that, The method for calculating the total aesthetic distortion D includes: Character k in the camera setting {c t Role visibility V(c) under |t=1,2,…,T} t ) through calculation To evaluate, r k It is the ratio of the size of the character k in the current video frame to the total size of the current video frame, I(c t ,k) represents camera c t The correlation between I(c) and the aforementioned role k t The lower the value of k, the better the relationship between character k and camera c. t Its importance is even greater; If with the camera c t If the bound character k takes an action at time t, the character's action A(c) will be applied. t The evaluation is 0 if the condition is not met, otherwise it is 1. Through calculation To evaluate the camera c t Camera settings in Represents the camera position. Represents the shooting direction. The action object at time t, and φ C () represents the distortion cost function set for the camera; Through calculation To evaluate screen continuity S(c) t ,c t-1 The screen continuity S(c) t ,c t-1 This includes a summary of the positional changes of each individual character in the current video frame, where p(k,c) t ) indicates that the camera c t The position of character k in the current video frame, p(k,c) t-1 ) indicates that it is produced by camera c t-1 The position of character k in the current video frame being captured; if character k appears in the camera c t In the view, v(k,c t ) = 1, otherwise v(k,c t ) = 0; φ S () is the penalty for changes in character position, which is related to the character position p(k,c) t ) and character position p(k,c t-1 A nonlinear function of the distance between (). Through calculation To evaluate the continuity of movement M(c) t ,c t-1 The mobility continuity M(c) t ,c t-1 This includes changes in the character's movement direction, which are caused by changes in the direction of movement of the character's movement caused by the camera c. t The change in view is caused by the character's actions before or after the change, where m(k,c) t ) indicates that the camera c t The motion direction vector of the character in the current video frame, m(k,c) t-1 ) indicates that the camera c t-1 The character's motion direction vector in the current video frame being captured; φ M () represents the penalty for changes in the character's movement direction, which is related to the character's movement direction vector m(k,c). t ) and the character's movement direction vector m(k,c t-1 The nonlinear function of the difference between ) and ; and Through calculation To evaluate lens duration distortion in φ is the average shot duration set for each scene, q is the maximum allowed shot duration, and φ is the maximum allowed shot duration. U () is a penalty for the duration of the shot of a video frame that changes the camera in the range [tq,…,t]. The total aesthetic distortion ω0, ω1, ω2, ω3 and ω4 are values ​​between 0 and 1, and are the weights of each distortion component.

11. The method according to claim 10, characterized in that, The minimization of total aesthetic distortion D includes: as well as, Define J λ (c t ,a t ) = D + λ·E, where λ is the Lagrange multiplier; and Will Make Simplified to min J λ (c t ,a t ); as well as, Define z k =(c k ,a k ); Define the cost function G T (z T-q ,…,z T Let represent the minimum sum of the fidelity error E and the total aesthetic distortion D of all video frames; and as well as, as well as in: The future steps of the adjustment process are independent of the past steps of those future steps; The adjustment process is transformed into a graph theory problem of finding the shortest path in a directed acyclic graph; and The computational complexity of the adjustment process is O(T×|Z|). q+1 It is more efficient than exhaustive search algorithms, which have exponential computational complexity.

12. The method according to claim 1, characterized in that, The video generated from the movie script was obtained based on the following method: A first action list is generated based on the movie script; Generate a stage performance based on the actions in the first action list; and The stage performance was filmed using a camera.

13. A video generation apparatus, characterized in that, include: Memory that stores program instructions; and A processor coupled to memory, the processor being configured to execute program instructions to: Get the video generated from the movie script; Based on the total aesthetic distortion D of the video, the video is adjusted to obtain the adjusted video.

Images