A method and apparatus for one-touch video production

Abstract

A method and apparatus for one-touch filmmaking are provided. The method includes obtaining a script from a user, detecting a single user operation, obtaining a plurality of shots in response to the single user operation and estimating information of the plurality of shots based on the script, and automatically generating a film based on an automatic movie shooting algorithm and the estimated information of the plurality of shots. The estimated information of one of the plurality of shots includes at least one of a character of the shot, a scene of the shot, one or more positions of the character in the shot, a duration of the shot, or a shot type.

Description

[0001] priority

[0002] This invention claims priority to U.S. Provisional Patent Application No. 63 / 302,686, filed January 25, 2022, and U.S. Patent Application No. 17 / 732,167, filed April 28, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of computer graphics processing technology, and more particularly to methods and apparatus for one-click video production. Background Technology

[0004] Making the user experience as simple and effective as possible has always been an interesting research topic. Amazon's invention of one-click purchasing is legendary and has been widely used over the past two decades. Based on the assumption that television (TV) is a leaning experience requiring as little user interaction as possible, the frequency of user interaction and / or clicks on remote control buttons is considered a fundamental metric for evaluating TV performance. An extremely simple experience called Binary TV has been proposed, completely eliminating user interaction with a complex remote control, where the user only needs to make a binary (yes or no) decision immediately when a query is received from the TV. The user can express their intention (explicitly or implicitly) while watching TV, and the story will then follow their intention accordingly.

[0005] The "write-as-you-go" experience allows ordinary users to go through a unified story writing and filmmaking process, ultimately generating a movie. However, these early efforts did not focus on detailed steps and how to automatically extract data to achieve one-click filmmaking. The disclosed methods and systems aim to solve one or more of the above-mentioned problems, as well as others, and are useful for non-professionals or professionals who cannot handle the entire filmmaking process alone. Summary of the Invention

[0006] One aspect of the present invention provides a method for one-click video production. The method includes: obtaining a script from a user; detecting a single user action; obtaining multiple shots in response to the single user action and estimating information about the multiple shots based on the script; and automatically generating a video based on an automatic film shooting algorithm and the estimated information about the multiple shots. The estimated information about one of the multiple shots includes at least one of the following: the character in the shot, the scene of the shot, one or more positions of the character in the shot, the duration of the shot, or the shot type.

[0007] Another aspect of the invention provides an apparatus for one-click video production, the apparatus including a memory and a processor coupled to the memory. The processor is configured to perform a plurality of operations, the plurality of operations including: obtaining a script from a user; detecting a single user action; obtaining a plurality of shots in response to the single user action and estimating information of the plurality of shots based on the script; and automatically generating a video based on an automatic film shooting algorithm and the estimated information of the plurality of shots. The estimated information of one of the plurality of shots includes at least one of the following: the character in the shot, the scene of the shot, one or more positions of the character in the shot, the duration of the shot, or the shot type. Attached Figure Description

[0008] Based on the various embodiments disclosed, the following figures are merely examples for illustrative purposes and are not intended to limit the scope of the invention.

[0009] Figure 1 This is a block diagram of an exemplary computing system according to some embodiments of the present invention.

[0010] Figure 2 An exemplary one-click video production process 200 is shown according to a partial embodiment of the present invention.

[0011] Figure 3 A diagram illustrating script comprehension is shown.

[0012] Figure 4 A schematic diagram showing the estimated roles is provided.

[0013] Figure 5 A schematic diagram of the estimated scenario is shown.

[0014] Figure 6 A diagram showing the character's movement path.

[0015] Figure 7 An exemplary window is shown for displaying information as a lens estimate to the user. Detailed Implementation

[0016] Exemplary embodiments of the invention will now be described in detail with reference to the accompanying drawings. Hereinafter, embodiments consistent with the invention will be described with reference to the drawings. Throughout the drawings, the same reference numerals are used wherever possible to refer to the same or similar parts. It will be apparent that the described embodiments are part, but not all, of the embodiments of the invention. Based on the disclosed embodiments, other embodiments consistent with the invention can be derived by those skilled in the art, all of which are within the scope of the invention.

[0017] This invention provides a method and apparatus for one-click video production. The disclosed method and / or apparatus can be applied to any suitable occasion requiring video production.

[0018] Figure 1 This is a block diagram of an exemplary computing system / device capable of implementing the disclosed one-click video production method according to certain embodiments of the present invention. Figure 1 As shown, the computing system 100 may include a processor 102 and a storage medium 104. According to some embodiments, the computing system 100 may also include a display 106, a communication module 108, peripheral devices 112, and at least one bus 114 coupling these devices together. Some devices may be omitted and other devices may be included.

[0019] Processor 102 may include any suitable processor. In some embodiments, processor 102 may include multiple cores and / or graphics processing units (GPUs) for multi-threaded or parallel processing. Processor 102 may execute sequences of computer program instructions to perform various processes, such as a one-click movie creation program. Storage medium 104 may be a non-transitory computer-readable storage medium and may include memory modules (such as ROM, RAM, flash memory modules, and erasable and rewritable memory) and mass storage devices (such as CD-ROMs, USB flash drives, and hard disks). When executed by processor 102, storage medium 104 may store computer programs for implementing various processes. Storage medium 104 may also include one or more databases for storing certain data (such as text scripts, library data, training datasets), and certain operations (such as database searches and data retrieval) may be performed on the stored data.

[0020] Communication module 108 may include network devices for establishing connections over a network. Display 106 may include any suitable type of computer display device or electronic device display (e.g., CRT or LCD-based devices, touch screens). Peripheral devices 112 may include additional I / O devices such as a keyboard, mouse, etc.

[0021] In operation, the processor 102 can be configured to execute instructions stored on the storage medium 104 and perform various operations related to the one-click video production method, as described in detail below.

[0022] It should be noted that the terms "film" and "film production" used in this invention can refer to any element included in media content, such as films, television programs, music videos, VR content, and any script-based media format.

[0023] Figure 2An exemplary one-click video creation process 200 according to a partial embodiment of the present invention is illustrated. Process 200 may be implemented by a one-click video creation device, which may be any suitable computing device / server having one or more processors and one or more memories, such as computing system 100 (e.g., processor 102).

[0024] like Figure 2 As shown, the one-click video production method consistent with the embodiments of the present invention includes the following process.

[0025] At S202, obtain the script from the user.

[0026] In some embodiments, the user uploads a script, writes a text script in the system, or inputs some text into the system by any means (e.g., using voice input), and the system uses automatic speech recognition (ASR) to convert the speech into text. That is, obtaining a script from the user includes at least one of the following: receiving a script uploaded by the user; receiving a text script input by the user; or receiving voice input from the user and converting the voice input into text using ASR.

[0027] At S204, individual user actions are detected.

[0028] In some embodiments, a single user action includes a click. In some embodiments, the user clicks a button (or uses other means to perform this single action) to begin a one-click video creation process.

[0029] At S206, multiple shots are acquired in response to a single user action, and information about the multiple shots is estimated based on a script. The estimated information for each shot includes at least one of the following: the character in the shot, the scene of the shot, one or more positions of the character in the shot, the duration of the shot, or the shot type.

[0030] In some embodiments, natural language processing (NLP) technology is used to analyze the script: the script is divided into multiple segments, such as... Figure 3 As shown, the shots contain themes and actions corresponding to the original script. For example, the script is divided into paragraphs. Each paragraph is then broken down into sentences. Each sentence is further divided into shots, each shot having a theme and action.

[0031] In some embodiments, estimating information for multiple shots includes estimating the character in each shot based on a script and estimating the scene in each shot based on a script.

[0032] In some embodiments, estimating the character of a shot based on the script includes finding candidate characters from a library that match the description of the character in the shot in the script. In some embodiments, based on the character description in the script, the system may find the best-matching male / female actor in the library based on the matching of tags associated with each candidate, such as... Figure 4 As shown in the image.

[0033] In some embodiments, estimating the scene based on the script includes finding candidate scenes from a library that match the scene description for the shot in the script. In some embodiments, based on the scene description in the script, the system can find the best-matching scene in the library based on the matching of tags associated with each candidate, such as... Figure 5 As shown in the image.

[0034] In some embodiments, estimating information from multiple shots includes estimating one or more locations of a character within a scene based on a script. For each scene in the library, multiple landmarks (e.g., doors, bathrooms, hallways) are pre-defined, so the system can follow the script's description to guess the possible locations of characters within the scene, such as... Figure 6 As shown in the image. If the clues are insufficient, the system will make the best guess at the location.

[0035] In some embodiments, estimating information for multiple shots includes estimating the duration of a shot based on a script.

[0036] In some embodiments, estimating the duration of a shot based on a script involves deriving the shot duration based on at least one of the length of dialogue in the script, the distance a character moves, or the duration of an action in the script. In some embodiments, the duration of a performance can typically be derived from the length of dialogue, the distance moved, or the typical length of an action. When no clue is found, the system uses a default value for this field.

[0037] In some embodiments, estimating information for multiple lenses includes automatically configuring the lens type for each lens based on a deep learning algorithm.

[0038] The shot type (also known as shot size) in a film refers to the elements that make the film entertaining. For example, shot types can include panoramic shots, ultra-wide-angle shots, medium shots, and close-ups.

[0039] In some embodiments, using deep learning algorithms, the system can be trained on existing lens type settings based on past experience and generate the best guess for the current camera lens type.

[0040] At S208, a film is automatically generated based on an automatic movie shooting algorithm and estimated information from multiple shots.

[0041] In some embodiments, automatically generating a film based on an automatic film shooting algorithm includes selecting a camera and selecting a camera path. The automatic film shooting algorithm automatically generates a film by generating a camera sequence that follows cinematic rules, based on estimated information from multiple shots.

[0042] To support automated video generation based on information extracted by the system and confirmed by the user, the system provides an automated movie shooting algorithm that automatically determines camera selection and camera path for each frame. The generated video may include multiple video frames. Each video frame is generated by a virtual camera at a certain location in the scene, using a shooting angle. In some embodiments, the virtual camera for each frame of the video is determined based on the automated movie shooting algorithm. In some embodiments, an optimization algorithm that minimizes both fidelity error and aesthetic distortion can be used to select the camera path.

[0043] In some embodiments, the one-click video creation method consistent with the embodiments of the present invention also includes obtaining the user's personal information and preferences.

[0044] In some embodiments, the system is required to manage user information and preferences such that such information can be set once but can be used repeatedly in the future without change, similar to a one-click purchase experience.

[0045] In some embodiments, before automatically generating the video, the one-click video production method consistent with the embodiments of the present invention further includes displaying the estimated information of each shot to the user and receiving the user's confirmation of the estimated information.

[0046] In some embodiments, such as Figure 7 As shown, the system pop-up window allows the user to confirm the following information, including but not limited to:

[0047] a. Character Setting: Mapping between characters in the script and available male / female actors (digital model format) in the library;

[0048] b. Scene setting: The mapping between the scene in the script and the 3D scenes available in the library;

[0049] c. Shot Segments: The script is broken down into several shots, and within each shot, the subject (i.e., the character) and his / her actions (completing the mapping between actions in the script and performance action models in the library), as well as the shot type (e.g., wide shot, medium shot, close-up, etc.), are all specified. For speaking actions, the subject's dialogue is extracted from the script and associated with the shot;

[0050] d. Position of a moving character: These positions are interpreted from the script based on context (e.g., "near the door"), and may sometimes be inaccurate due to a lack of information;

[0051] e. Duration of the performance: the duration of an action over time, such as the duration of a conversational sentence;

[0052] f. User personal information related to this transaction: This may include the user's email address, preferred method of receiving the generated video, payment information, etc.

[0053] In some embodiments, before receiving user confirmation for the estimated information, the one-click video production method according to embodiments of the present invention further includes receiving user adjustment operations for the estimated information.

[0054] In some embodiments, the user views the information displayed in the window, edits or adjusts some settings until the user is satisfied. Then, the user confirms that all information is available and can continue.

[0055] In some embodiments, if the user has preset "skip confirmation" in the system settings, the estimated information for each shot can be skipped from being displayed to the user and from receiving the user's confirmation of that information.

[0056] In some embodiments, for virtual users who want to fully trust the system's intelligence, the three-step process (entering a text script, selecting one-click video creation, and then confirming / editing the information automatically generated from the script) can be simplified to a two-step process, simply by having the user enable "skip confirmation steps" in the system settings: enter the script and select one-click video creation. In other words, the system will no longer require confirmation / editing in the future.

[0057] In some embodiments, after automatically generating the video, the one-click video creation method consistent with the present invention further includes requesting a user selection regarding whether to play the generated video and playing the video to the user in response to the decision to play the video.

[0058] In some embodiments, after automatically generating the video, the one-click video production method consistent with embodiments of the present invention further includes requesting a user selection regarding whether the generated video is approved; in response to determining that the video is not approved, displaying estimated information of one of the plurality of shots to the user; receiving user adjustment operations based on the estimated information; and automatically generating an updated video based on the adjusted information and an automatic movie shooting algorithm.

[0059] In some embodiments, displaying estimated information from one of multiple shots to the user includes highlighting a portion of the information estimated with low confidence. As technology advances and algorithms improve, the user's confirmation / review / editing workload is significantly reduced. For example, the system may be able to highlight low-confidence portions of information extracted / generated for confirmation, which speeds up the user's confirmation process and makes the one-click video creation experience as simple as the widely available one-click purchasing experience.

[0060] The one-click video creation method, consistent with embodiments of the present invention, is useful for non-professionals or professionals who cannot handle the entire video creation process independently. Users simply follow three steps (exactly the same steps indicated in Amazon's one-click purchase experience): enter a text script, select one-click video creation, and confirm / edit the information automatically generated from the script; a video in video format will then be automatically generated for the user. This simple process does require a significant amount of behind-the-scenes technical support, including script understanding, automated stage setup, 3D rendering, machine learning, and automated filming.

[0061] On the other hand, the system also allows (but does not require) users to perform more iterations to improve the video by returning to the confirmation / editing step and adjusting the detailed settings as much as possible.

[0062] Other embodiments of the invention will be apparent to those skilled in the art from the description and practice of the invention disclosed herein. The description and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

Claims

1. A method for one-click video production, the method being applied to a computing device, characterized in that, The method includes: Obtain the script from the user; Detect individual user actions; In response to the single user action, multiple shots are obtained and information about the multiple shots is estimated based on the script. The estimated information for one of the multiple shots includes at least one of the following: the character in the shot, the scene of the shot, one or more positions of the character in the shot, the duration of the shot, or the shot type. The film is automatically generated based on the automatic film shooting algorithm and the estimated information from the multiple shots; The automatic film generation based on the automatic film shooting algorithm includes selecting a camera and selecting a camera path; the automatic film shooting algorithm is used to automatically generate a film by generating a camera sequence that follows film rules based on estimated information from multiple shots.

2. The method according to claim 1, characterized in that, The estimation of the information of the plurality of lenses includes automatically configuring the lens type of one of the plurality of lenses based on a deep learning algorithm.

3. The method according to claim 1, characterized in that, Before automatically generating the video, the following is also included: Displaying the estimated information from one of the plurality of lenses to the user; and The user confirms the information received in response to the estimation.

4. The method according to claim 3, characterized in that, Before receiving the user confirmation action for the estimated information, the method further includes: Adjust the operation based on the estimated information receiving user.

5. The method according to claim 1, characterized in that, After the video is automatically generated, the following is also included: The request asks for a user choice regarding whether to play the generated video; and The video is played to the user in response to a decision to play the video.

6. The method according to claim 1, characterized in that, After the video is automatically generated, the following is also included: Requesting a user's choice regarding whether to approve the generated video; In response to determining that the video has not been approved, the estimated information of one of the plurality of shots is displayed to the user; Adjustments to the estimated information receiving user; and The updated film is automatically generated based on the adjusted information and the automatic film shooting algorithm.

7. The method according to claim 1, characterized in that, The script obtained from the user includes at least one of the following: Receive scripts uploaded by the user; Receive a text script input by the user; or Receive voice input from the user and convert the voice input into text using automatic speech recognition.

8. The method according to claim 1, characterized in that, The script described therein was analyzed using natural language processing techniques.

9. The method according to claim 1, characterized in that, The estimation of the information from the plurality of shots includes performing the following operations for each of the plurality of shots: Estimate the character of the shot based on the script, including finding candidate characters from a library that match the description of the character of the shot in the script; and Estimating the scene of the shot based on the script includes finding candidate scenes from the library that match the description of the scene of the shot in the script.

10. The method according to claim 1, characterized in that, The information for estimating the plurality of shots includes estimating the duration of one of the plurality of shots based on the script, including deriving the duration of the shot based on at least one of the length of dialogue in the script, the distance the character moves, or the duration of an action in the script.

11. A device for one-click video production, characterized in that, include: Memory; as well as A processor, coupled to the memory and configured to perform a plurality of operations, including: Obtain the script from the user; Detect individual user actions; In response to the single user action, multiple shots are obtained and information about the multiple shots is estimated based on the script. The estimated information for one of the multiple shots includes at least one of the following: the character in the shot, the scene of the shot, one or more positions of the character in the shot, the duration of the shot, or the shot type. The film is automatically generated based on the automatic film shooting algorithm and the estimated information from the multiple shots; The automatic film generation based on the automatic film shooting algorithm includes selecting a camera and selecting a camera path; the automatic film shooting algorithm is used to automatically generate a film by generating a camera sequence that follows film rules based on estimated information from multiple shots.

12. The device according to claim 11, characterized in that, The information used to estimate the plurality of lenses includes: The lens type of one of the multiple lenses is automatically configured based on a deep learning algorithm.

13. The device according to claim 11, characterized in that, The plurality of operations performed by the processor before automatically generating the video also include: Displaying the estimated information from one of the plurality of lenses to the user; and The user confirms the information received in response to the estimation.

14. The device according to claim 13, characterized in that, The plurality of operations performed by the processor before receiving the user confirmation operation for the estimated information further include: Adjust the operation based on the estimated information receiving user.

15. The device according to claim 11, characterized in that, The plurality of operations performed by the processor after the automatic generation of the video also include: The request asks for a user choice regarding whether to play the generated video; and The video is played to the user in response to a decision to play the video.

16. The device according to claim 11, characterized in that, The plurality of operations performed by the processor after the automatic generation of the video also include: Requesting a user's choice regarding whether to approve the generated video; In response to determining that the video has not been approved, the estimated information of one of the plurality of shots is displayed to the user; Adjustments to the estimated information receiving user; and The updated film is automatically generated based on the adjusted information and the automatic film shooting algorithm.

17. The device according to claim 11, characterized in that, The script obtained from the user includes at least one of the following: Receive scripts uploaded by the user; Receive a text script input by the user; or Receive voice input from the user and convert the voice input into text using automatic speech recognition.

18. The device according to claim 11, characterized in that, The script described therein was analyzed using natural language processing techniques.

19. The device according to claim 11, characterized in that, The estimation of the information from the plurality of shots includes performing the following operations for each of the plurality of shots: Estimating the character for the shot based on the script includes finding candidate characters from a library that match the description of the character for the shot in the script; and Estimating the scene of the shot based on the script includes finding candidate scenes from the library that match the description of the scene of the shot in the script.

20. The device according to claim 11, characterized in that, The information for estimating the plurality of shots includes estimating the duration of one of the plurality of shots based on the script, including deriving the duration of the shot based on at least one of the length of dialogue in the script, the distance the character moves, or the duration of an action in the script.

Images