System and method for creating a 2D film from immersive content

Abstract

The system, method, and non-transitory computer-readable medium may obtain data associated with a computer-based experience. The computer-based experience may be based on interactive real-time technology. At least one virtual camera may be configured within the computer-based experience in an animation creation application. A plurality of shots corresponding to two-dimensional content may be generated in the animation creation application from edited cuts of content captured by the at least one virtual camera. Data associated with a two-dimensional version of the computer-based experience may be generated in a real-time engine based on the plurality of shots. The two-dimensional version may be rendered based on the generated data.

Description

Technical Field

[0001] The present technology relates to the field of digital animation. More specifically, the present technology relates to a technology for generating 2D content from AR / VR content.

Background Art

[0002] Virtual Reality (VR) and Augmented Reality (AR) are new media for entertainment and storytelling that enable content creators to immerse viewers in ways not possible in other media. VR and AR are powerful immersive platforms for telling engaging stories using characters with whom the audience can interact and empathize. The user (e.g., viewer) is intuitively connected to the world around them. The user can be immersed, have agency, and see any location. The user can also have a role to play and be triggered to act. The characters can be aware of the user's presence in their world and can react in real time to user actions. In contrast, 2D content (e.g., movies, films, TV shows) is a passive, movie-like medium that can induce empathy using characters, but of course there is no interaction.

Summary of the Invention

[0003] Various embodiments of the present technology may include a system, a method, and a non-transitory computer-readable medium configured to obtain data associated with a computer-based experience. The computer-based experience may be based on interactive real-time technology. At least one virtual camera may be configured within the computer-based experience in a real-time engine. Data associated with an edit cut of the computer-based experience may be obtained based on content captured by at least one virtual camera. A plurality of shots corresponding to two-dimensional content may be generated from the edit cut of the computer-based experience in a real-time engine. Data associated with a two-dimensional version of the computer-based experience may be generated in the real-time engine based on the plurality of shots. The two-dimensional version may be rendered based on the generated data.

[0004] In one embodiment, the computer-based experience is based on immersive real-time technology.

[0005] In one embodiment, the data associated with the two-dimensional version is interactive 2D content.

[0006] In one embodiment, obtaining the data associated with the edit cut further includes importing data explaining a set of edits reflected in the edit cut from non-linear video editing software and into the real-time engine.

[0007] In one embodiment, generating a plurality of shots corresponding to two-dimensional content includes, using a real-time engine, applying additional set decorations and layout data to one or more frames associated with at least one shot included in the plurality of shots, and applying writing and one or more media effects to one or more frames, wherein the writing and one or more media effects are added on top of the writing and one or more media effects applied to the computer-based experience in the real-time engine.

[0008] In one embodiment, generating data associated with a two-dimensional version of a computer-based experience includes, in an animation creation application, generating at least one new shot for the two-dimensional version or applying one or more animation fixes to at least one shot for the two-dimensional version.

[0009] In one embodiment, the system, method, and non-transitory computer-readable medium are configured to create a copy of at least one shot prior to the application of one or more animation fixes, and the one or more animation fixes are applied to the copy.

[0010] In one embodiment, generating data associated with a two-dimensional version further includes rendering the two-dimensional version using a real-time engine.

[0011] In one embodiment, the rendering is performed in-editor through the real-time engine.

[0012] In one embodiment, a system, method, and non-transitory computer-readable medium determine a timeline associated with a 2D version, determine an area marked to be rendered in the timeline, and are configured to render a portion of the 2D version corresponding to the area using a real-time engine.

[0013] Various embodiments of the present technology may include a system, method, and non-transitory computer-readable medium configured to obtain data associated with a computer-based experience. The computer-based experience may be based on interactive real-time technology. At least one virtual camera may be configured within the computer-based experience in an animation creation application. A plurality of shots corresponding to 2D content may be generated from edit cuts of content captured by at least one virtual camera in an animation creation application. Data associated with the 2D version of the computer-based experience may be generated in a real-time engine based on the plurality of shots. The 2D version may be rendered based on the generated data.

[0014] In one embodiment, a system, method, and non-transitory computer-readable medium are configured to render a 2D version in a real-time engine.

[0015] In one embodiment, generating a plurality of shots corresponding to 2D content further includes obtaining from non-linear video editing software data associated with edit cuts of a computer-based experience.

[0016] In one embodiment, generating a plurality of shots corresponding to 2D content further includes importing into a real-time engine data associated with a plurality of shots generated in an animation creation application.

[0017] In one embodiment, a system, method, and non-transitory computer-readable medium are configured to create a first animation layer associated with a two-dimensional version separated from a second animation layer associated with a computer-based experience, and adjustments made to a shot in a plurality of shots are applied to the first animation layer.

[0018] In one embodiment, a system, method, and non-transitory computer-readable medium are configured to create a first animation layer associated with a two-dimensional version separated from a second animation layer associated with a timeline corresponding to a computer-based experience, and adjustments made to a timeline corresponding to the two-dimensional version are applied to the first animation layer.

[0019] In one embodiment, generating a plurality of shots corresponding to two-dimensional content further includes, in an animation creation application, applying one or more animation fixes to at least one shot for the two-dimensional version.

[0020] In one embodiment, applying one or more animation fixes to at least one shot for the two-dimensional version in an animation creation application further includes creating a copy of at least one shot prior to the application of the one or more animation fixes, and the one or more animation fixes are applied to the copy.

[0021] In one embodiment, a system, method, and non-transitory computer-readable medium are configured to apply writings and one or more media effects to frames associated with a computer-based experience in a real-time engine, and to apply writings and one or more media effects to frames associated with a two-dimensional version in the real-time engine, and the writings and one or more media effects for the frames associated with the two-dimensional version are added on top of the writings and one or more media effects applied to the computer-based experience.

[0022] In one embodiment, a system, method, and non-transitory computer-readable medium are configured to apply an additional set of decoration and layout data to one or more frames associated with at least one shot included in a plurality of shots in a real-time engine.

Brief Description of the Drawings

[0023]

Figure 1A

Figure 1B

[0024]

Figure 2

[0025]

Figure 3A

Figure 3B

[0026]

Figure 4

[0027]

Figure 5A

Figure 5B

[0028]

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

[0029]

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

[0030]

Figure 25

[0031] These figures illustrate various embodiments of the disclosed technology for illustrative purposes only, and in these figures, like reference numerals are used to identify like elements. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown in the figures can be utilized without departing from the principles of the disclosed technology described herein.

DETAILED DESCRIPTION OF THE INVENTION

[0032] [Creation of 2D Films from Immersive Content] Virtual reality (VR) and augmented reality (AR) are new media for entertainment and storytelling that enable content creators to immerse viewers in ways not possible with other media. VR and AR are powerful real-time immersive platforms for telling engaging stories using characters with whom the audience can interact and empathize. The user (e.g., viewer) is intuitively connected to the world around them. The user can become immersed, have agency, and see any location. The user can also have a role to play and be triggered to act. The characters can be aware of the user's presence in their world and react in real time to user actions. In contrast, 2D content (e.g., movies, films, TV shows) is a passive and cinematic medium that can induce empathy using characters but generally has no interaction. More recently, interactive 2D content (e.g., interactive TV shows) has enabled the user to make choices at specific moments in a story that branches into alternative 2D storylines and alternative 2D content for each branch of the subsequent story.

[0033] A storytelling project may rely on a traditional computer animated movie pipeline (or process) to produce 2D content for non-interactive media such as movies or television. Additionally, a storytelling project may rely on an interactive real-time pipeline (or process) to produce AR / VR content for interactive media such as computer animated real-time experiences based on VR technology or AR technology. A storytelling project may rely on an interactive real-time pipeline (or process) to produce interactive game content such as computer animated real-time experiences based on mobile, console, or PC technology. In some cases, a storytelling project may need content to be produced for both interactive and non-interactive media. For example, a storytelling project may need an interactive version of a story for viewers who prefer immersive experiences based on VR technology or AR technology, and a non-interactive version of the story for viewers who prefer traditional 2D movie experiences. Under conventional approaches, as described in connection with FIGS. 1A and 1B, the AR / VR version is created based on an interactive real-time process and the 2D version is created separately based on a traditional computer animated movie process.

[0034] Figure 1A shows an exemplary conventional interactive real-time pipeline 100 for creating AR / VR content for a story. As shown, the interactive real-time pipeline 100 begins with concept development using various two-dimensional (2D) tools 102. Initially, at block 104, a story and script can be created for interactive content such as an interactive story or game. The story and script may consider the role of the user (or viewer), interactive mechanics, and non-linearity. For example, there may be alternative dialogue lines based on user selection. At block 106, as the story becomes more refined, concept artwork can be created. Environmental concept art and character designs can be created to explore the world and characters of the story. At block 108, the way the user interacts with the characters, story, and environment can be designed. For example, this may involve designing core mechanics, interactive elements, and non-linear story flows. At block 110, the script can be converted into 2D storyboard diagrams that visually realize the scenes. The 2D storyboard is first presented starkly to the director, iterated, and then edited into a video story reel. The 2D storyboard provides a quick and inexpensive way to view all of the film without performing costly production work. At block 114, a real-time engine 112 generates a 3D storyboard based on the 2D storyboard produced by the 2D tools 102. The storyboard can be created in 3D so that it can be placed and viewed in a three-dimensional environment. The 3D storyboard can be created in the real-time engine 112, other 3D rendering / storyboard tools, or VR rendering / storyboard tools as 2D sprites that move around in 3D space or are directly drawn in 3D. Next, at block 116, a preview can be performed in the real-time engine 112 that visualizes the scenes before proceeding to animation production.In block 118, the editorial adds dialogues, music, audio, and edits the shot sequence for timing. In some cases, movie editing may not be generated because the content changes based on user input. In block 120, set decoration, the final environmental model, and the final characters can be incorporated into the scene. In block 124, the animation creation tool 122 is used to create all character assets and environmental assets. In block 126, the scene and shots can be animated. In interactive games and AR / VR content, generally one shot represents one character action (referred to as an animation cycle or cycle animation). In block 128, the real-time engine 112 can combine and / or procedurally generate animations through a non-linear animation framework, an AI system, and other procedural methods. In block 130, the real-time engine 112 can create FX and simulation elements. In block 132, the real-time engine 112 can apply lighting effects to the animation in real time. This is often achieved through a little dynamic lighting calculated in real time for the dynamic and non-static elements (i.e., characters) of the scene, combined with pre-existing global lighting calculated for the static elements (i.e., the non-moving parts of the environment) in the scene. In block 134, the real-time engine 112 outputs an interactive application. The interactive application may be a computer animation real-time experience based on VR technology or AR technology, or an interactive game.

[0035] Figure 1B shows an exemplary conventional computer animated movie pipeline 150 for creating non-interactive 2D content (e.g., 2D computer animated movie). As shown, the computer animated movie pipeline 150 starts with concept development using various two-dimensional (2D) tools 152. At block 154, a story can be created for the 2D computer animated movie. The story can be used to create a script or screenplay. At block 156, as the story becomes refined, concept artwork can be created to define the look of the movie. Environmental concept art and character design can be created to explore the world and characters of the story. At block 158, the script can be converted into 2D storyboard diagrams that visually realize the scenes. The 2D storyboard is first presented starkly to the director, iterated, and then edited into a video story reel. The 2D storyboard provides a quick and inexpensive way to view all of the film without performing costly production work. At block 160, a previs artist converts the storyboard reel into a 3D preview visualization of the sequence. The previs artist creates mock-ups of environmental models and character models. Then, the artist can place virtual cameras in the scene to establish cinematography and provide edited video for cutting the shots together. At block 162, an editor can create edit cuts in non-linear video editing software that combines the camera shots into a single sequence movie. This edit cut can be reviewed by the director, which often requires adjustments to the existing cameras by the previs artist and / or requires additional cameras to be created. At block 164, the characters and environment can be converted into final 3D models with textures. By animators, the character assets are equipped as digital puppets so that these characters can be moved realistically and made to express emotions. At block 166, the final shots are created, the camera cinematography is refined, and the set decoration is adjusted for each shot.In block 168, the animation can be split into a series of shots or scenes. For a given shot, the animator can either animate the character manually over time or create it by interpolating from keyframes, or the animation can be motion captured from the performers of a live performance. In block 170, FX elements (e.g., water, fire, fog, etc.) can be added to the shot. Additionally, clothing, hair, and other dynamic components are simulated. These elements are rarely applied in real time. In block 172, the lighting artist may apply visual lighting to the shot to create the final image of the movie. Master lighting is often first done to establish the overall look for the entire scene and / or for a smaller number of key shots. Shot lighting can then be applied to fine-tune the lighting on the character and the environment for each shot. In block 174, the scene data can be rendered offline using a non-real-time renderer. The rendering can occur on the artist's desk or on a rendering farm machine on-premises or in the cloud. The rendering often generates multiple render layer passes (e.g., depth pass, matte pass, etc.) for each frame for the compositing package. In block 176, the final combined rendered frame can be generated using compositing software to produce non-interactive 2D content.

[0036] Therefore, under the conventional approach, an entity needs to use an interactive real-time process, such as the process shown in FIG. 1A, to create an AR / VR version of a story. Further, under the conventional approach, an entity needs to use a separate computer animation process, such as the process shown in FIG. 1B, to create a 2D version of the story. The need for different processes to generate interactive and non-interactive versions of the same story can cause duplicated work, increased costs, and increased production complexity.

[0037] An improved approach rooted in computer technology overcomes the aforementioned and other drawbacks associated with the conventional approach, which occurs particularly in the field of computer technology. The present technology provides the ability to create interactive and non-interactive versions of a story based on a single process (or pipeline). By the present technology, it becomes possible to generate non-interactive 2D content for a story from an existing interactive AR / VR version of the story. For example, by the present technology, it becomes possible to generate a 2D computer animation movie from an existing computer animation real-time experience based on VR technology or AR technology. As a result, the present technology may provide a comprehensive toolset that enables a movie maker to meticulously create a 2D computer animation movie without sacrificing features available in the conventional process. The present technology also enables the simultaneous development of both 2D-based projects and AR / VR-based projects, where changes to the AR / VR-based project can be reflected in the 2D-based project and vice versa. The present technology may also utilize AR / VR as a content creation medium for developing cinematography and search for 2D projects. The present technology can also handle different pipeline architectures. More details related to the present technology are provided below.

[0038] FIG. 2 shows an exemplary system 200 according to an embodiment of the present technology. The exemplary system 200 may include an animation creation module 202, a real-time engine module (or real-time engine) 212, and a 2D editing module 232. The exemplary system 200 may be implemented to generate non-interactive 2D content from AR / VR content. The exemplary system 200 may be implemented to generate non-interactive 2D content from interactive game content developed for mobile, console, and PC technologies. For example, the exemplary system 200 may generate a non-interactive 2D computer-animated film or television program from an interactive computer animation real-time experience based on VR technology or AR technology. In another example, the exemplary system 200 may generate a first-person 2D computer animation film or television program from a character-based computer animation real-time experience based on VR technology or AR technology. The character-based computer animation real-time experience may involve an immersive experience in which the viewer can participate in the story as a character. In another example, the exemplary system 200 may generate an interactive 2D computer animation film or television program from a character-based computer animation real-time experience based on VR technology or AR technology. The exemplary system 200 generates a set of 2D content paths representing different branches of the story. Many variations are possible. In contrast to the exemplary system 400 of FIG. 4 described in more detail herein, the exemplary system 200 allows camera placement and shot creation to occur in relation to the real-time engine module 212 instead of the animation creation module 202. The exemplary system 200 may be well suited for interactive AR / VR content in which the final real-time content is procedurally generated from a set of animation clips created based on the animation creation module 202.The animation creation module 202, the real-time engine module 212, and the 2D editing module 232 may be implemented in one or more software applications operating on one or more computing devices. The components (e.g., modules, elements, etc.) shown in this figure and all figures in this specification are merely exemplary, and in other implementations, may include additional components, fewer components, integrated components, or different components. Some components may not be shown so as not to obscure relevant details. In various embodiments, one or more of the functions described in connection with the animation creation module 202, the real-time engine module 212, and the 2D editing module 232 may be implemented in a suitable order and combination.

[0039] In some embodiments, the various modules and / or applications described herein may be implemented, in part or in whole, as software, hardware, or any combination thereof. Generally, as described herein, modules and / or applications may be associated with software, hardware, or any combination thereof. In some implementations, the operation of one or more functions, tasks, and / or modules and / or applications may be performed or executed by software routines, software processes, hardware, and / or any combination thereof. In some cases, the various modules and / or applications described herein may be implemented, in part or in whole, as software operating on one or more computing devices or systems, such as on a user computing device or client computing device or on a server. For example, one or more of the modules and / or applications described herein, or at least a portion thereof, may be implemented as, or within, an application (e.g., app), program, or applet, etc., operating on a user computing device or client computing system. In another example, one or more modules and / or applications, or at least a portion thereof, may be implemented using one or more computing devices or systems that include one or more servers, such as a network server or cloud server. It should be understood that there may be many variations or other possibilities. In one exemplary embodiment, the animation creation module 202 may be implemented in or using animation creation software such as Autodesk Maya, and the real-time engine module 212 may be implemented in or using a real-time engine such as the Unity game engine.

[0040] The animation creation module 202 may include a VR animation module 204 and a shot fix module 206.

[0041] The VR animation module 204 may be configured to animate an interactive version of a story. For example, the interactive version of the story may be a computer animated real-time experience based on VR technology or AR technology. The interactive version of the story may be composed of a sequence of shots. A shot may represent several frames captured by a virtual camera located within a computer animated real-time experience in three-dimensional space. In some embodiments, a shot may represent a single character action such as an animation cycle or cycle animation. Further, the sequence of shots may correspond to several related shots. For example, the sequence may include shots captured at a specific location within the computer animated real-time experience. The VR animation module 204 may store data (such as shots, sequences of shots, etc.) associated with the interactive version of the story in a data store 208. In one embodiment, the data store 208 resides on a remote server. In another embodiment, a web-based application interfaces with various modules in the animation creation module 202, the real-time engine module 212, and the 2D editing module 232 to synchronize the data in the data store 208.

[0042] The shot fix module 206 can be configured to apply various fixes to shots associated with a non-interactive 2D version of a story generated from an interactive version of the story. For example, the shot fix module 206 can apply a fix to a shot generated by the real-time engine module 212 for the non-interactive 2D version of the story. Generally, shots associated with the non-interactive 2D version of the story can be derived from shots associated with the interactive version of the story. As a result, updates to shots associated with the interactive version of the story may be reflected in the corresponding shots associated with the non-interactive 2D version of the story. In some examples, specific adjustments that are only applied to shots associated with the non-interactive 2D version of the story may be required. For example, the character eye lines represented in a shot may be correct in the interactive version of the story, but may appear to be looking in the wrong direction in shots associated with the non-interactive 2D version of the story. In such examples, the shot fix module 206 can be configured to branch shots that require specific adjustments for the non-interactive 2D version of the story. When a shot is branched, a separate copy of the shot is maintained for the interactive version and the non-interactive 2D version. As a result, adjustments to the shot can be made to the non-interactive 2D version of the story without affecting the interactive version of the story. In the foregoing example, adjustments can be made to the shot character eye lines associated with the non-interactive 2D version of the story without affecting the character eye lines in the corresponding shot associated with the interactive version of the story. In some examples, the non-interactive version of the story may require that new shots be created.In one example, one shot in the interactive version of the story is split into several smaller and potentially overlapping shots in the non-interactive 2D version that correspond to different camera angles (e.g., a close-up, a medium shot, and for establishing the shot). These new shots may correspond to computer animations created specifically for the non-interactive 2D version of the story. For example, the look associated with a character may not be acceptable in the associated shots from the non-interactive 2D version of the story. In another example, a shot associated with the interactive version of the story may involve some interaction by the viewer. Such an interaction may be acceptable in the interactive version of the story, but the interaction is not suitable for duplication in the non-interactive 2D version of the story. In such an example, the shot fix module 206 may permit the creation of new shots for the non-interactive 2D version of the story that correct or replace existing shots associated with the interactive version of the story. The shot fix module 206 may store data associated with the shots fixed or added in relation to the non-interactive 2D version of the story in the data store 208.

[0043] The real-time engine module 212 may include a VR animation module 214, a VR writing and FX module 216, a 2D previz module 218, a 2D shot creation module 220, a 2D layout module 222, and a 2D writing and FX module 224.

[0044] The VR animation module 214 may be configured to combine and / or procedurally generate animations using a non-linear animation framework, an artificial intelligence (AI) system, and other generally known procedural methods.

[0045] The VR writing and FX module 216 may be configured to apply writing and media effect (FX) elements used in an interactive version of the story. The writing and FX elements may generally be created using known techniques.

[0046] The 2D preview module 218 can be configured to arrange a set of virtual cameras in an interactive version of the story as animated by the VR animation module 204. For example, an artist may instruct the 2D preview module 218 to arrange a set of virtual cameras in an interactive version of the story based on the 2D film script 210. For example, the 2D film script 210 can be written based on a script associated with the interactive version of the story. The set of virtual cameras can be placed within the interactive version of the story to image the animation scene from various different camera angles. In some embodiments, the set of virtual cameras and associated parameters can be created using AR / VR technology. For example, an artist can place and manipulate the virtual camera by moving a 6DOF hand controller with their hand in VR. The artist can then record video from the original VR / AR experience using this virtual camera in VR. This can be used to create the effect of a handheld camera. In one embodiment, multiple artists can use VR / AR technology to place and record different virtual cameras, thus creating separate sets of camera data. The 2D preview module 218 can also be configured to export a 2D "playblast" feed (or movie) for each virtual camera. In some embodiments, as shown in the exemplary FIG. 6, the 2D preview module 218 can insert reticle information and slate information into each frame of the exported 2D "playblast" feed. The reticle information and slate information can be provided as a frame overlay. In some embodiments, the reticle information and slate information can include reticle data (e.g., camera boundaries), sequence data, file names, camera lenses, camera / shot names, timestamps, take numbers or repeat counts, and animation frame numbers.

[0047] The 2D shot creation module 220 can be configured to create new shots and update existing shots. Generally, the 2D shot creation module 220 can create shots from edit cuts. Edit cuts can be produced by the 2D editing module 232 as described below. In one embodiment, any virtual camera represents a single shot. In another embodiment, multiple shots may be constructed from a single virtual camera. The 2D shot creation module 220 can store each shot and its associated metadata information in the data store 208. An example of shot metadata information associated with a virtual (or animated) camera is shown in FIG. 7. For example, the shot metadata information can include a shot name, duration, animation file, frame range, characters included in the shot, and a publish version. Many variations are possible. In one embodiment, the 2D shot creation module 220 can update a shared file that stores shot data. In one embodiment, the 2D shot creation module 220 can update the data store 208. In another embodiment, the 2D shot creation module 220 updates the data store 208 through a web application API (application programming interface).

[0048] The 2D layout module 222 can be configured to enhance the shots created by the 2D shot creation module 220. For example, the 2D layout module 222 can apply additional set decoration and layout to each shot, generally using known approaches.

[0049] The 2D writing and FX module 224 can be configured to provide various options for applying features to a shot. For example, the 2D writing and FX module 224 can apply lights, shadows, contact shadows, FX elements, and post-processing effects such as subject blur and depth of field. For example, additional features can be applied based on instructions by a writing artist and an FX artist. Applying such features can extend the appearance of each shot.

[0050] The 2D composition unit 242 can generate a final rendering frame based on generally known approaches. For example, the 2D composition unit 242 can generate a final rendering frame based on the final frame and layer passes when being rendered by the real-time engine module 212. The 2D composition unit 242 can also generate a non-interactive 2D version of a story, such as a 2D computer animated movie 244, based on the final rendering frame.

[0051] The 2D editing module 232 may provide information describing editing cuts that combine various camera shots into a single sequence movie. The editing cuts may be produced using non-linear video editing software. Information describing the editing cuts, and related editing timing and track information, may be imported into the real-time engine module 212. The real-time engine module 212 permits further adjustment to existing virtual cameras and the addition of new virtual cameras as needed to correct the editing cuts. In some embodiments, the 2D editing module 232 may read editing selections created by non-linear video editing software in relation to the editing cuts. The 2D editing module 232 may reproduce (or provide) the editing selections in the real-time engine module 212. In one embodiment, the 2D editing module 232 may read the following data below a non-linear editing software project: the positions of source video and audio clips, the timecodes of the clips from the video and audio tracks, and effects such as cross-fades and audio level curves that may be applied to the video and audio clips. In one embodiment, the 2D editing module 232 may use such data to identify assets in the real-time engine module 212 corresponding to the video clips and audio clips edited in the non-linear video editing software. In one embodiment, the 2D editing module 232 may use such data to create animation tracks and audio tracks using the identified assets in the real-time engine module 212. In one embodiment, the 2D editing module 232 may use such data to apply editing decisions to the assets in the real-time engine module 212. In one embodiment, the 2D editing module 232 may create a single timeline representing all the shots, whereby the user may move back and forth between the shots in the real-time engine module 212 to refine them. The single timeline may help facilitate real-time editing using the context of the surrounding shots.Furthermore, the 2D editing module 232 supports updating existing camera and shot information already present in the real-time engine module 212. In one embodiment, camera / shot tracks from non-linear video editing software are associated with tracks in the real-time engine module 212 by convention in naming. In another embodiment, the tracks can be associated with each other by metadata information that is passed to the non-linear video editing software and then shuttled back and forth to the real-time engine module 212. In another embodiment, the tracks can be associated by using an open-source file format such as OpenTimeline. In yet another embodiment, the tracks can be associated with each other using data comparison heuristics. Many variations are possible.

[0052] FIG. 3A shows an exemplary method 300 according to one embodiment of the present technology. In some embodiments, method 300 may be executed by system 200 of FIG. 2. At block 302, sequences and shots may be animated for AR / VR animation or experience as described above with reference to the VR animation module 204. At block 304, the animation may be combined and / or procedurally generated through a non-linear animation framework as described above with reference to the VR animation module 214. At block 306, lighting and FX elements may be created for AR / VR animation or experience as described above with reference to the VR lighting and FX module 216. At block 308, the virtual camera may be placed within the real-time engine as described above with reference to the 2D preview module 218. At block 310, edit cuts may be produced in non-linear video editing software as described above with reference to the 2D editing module 232. At block 312, once the edit cuts are approved, 2D shots are created from the camera and timing metadata as described above with reference to the 2D shot creation module 220. At block 314, set decoration and layout may be applied to the shots as described above with reference to the 2D layout module 222. At block 316, additional lights, shadows, contact shadows, FX elements, and post-processing effects may be applied as described above with reference to the 2D lighting and FX module 224. At block 318, fixes may be applied to the 2D shots and additional 2D shots may be created as described above with reference to the shot fix module 206.

[0053] Many variations to the exemplary method are possible. It should be understood that within the scope of the various embodiments described herein, there may be additional steps, fewer steps, or alternative steps that are performed in the same or alternative order, or in parallel, unless otherwise specified.

[0054] Figure 3B shows an exemplary method 350 according to an embodiment of the present technology. For example, method 350 may be executed by system 200 of FIG. 2. At block 352, data associated with a computer-based experience may be obtained. The computer-based experience may be based on interactive real-time technology. For example, the interactive real-time technology may apply augmented reality (AR) and / or virtual reality (VR) to provide interactive content. At block 354, at least one virtual camera may be configured within the computer-based experience in a real-time engine. At block 356, data associated with an edit cut of the computer-based experience may be obtained based on content captured by at least one virtual camera. At block 358, a plurality of shots corresponding to 2D content may be generated from the edit cut of the computer-based experience in a real-time engine. At block 360, data associated with a 2D version of the computer-based experience may be generated in the real-time engine based on the plurality of shots. The 2D version may be rendered based on the generated data.

[0055] Many variations to the exemplary method are possible. It should be understood that within the scope of the various embodiments described herein, additional steps, fewer steps, or alternative steps may be performed in the same or alternative order, or in parallel, unless otherwise specified.

[0056] Figure 4 shows an exemplary system 400 according to one embodiment of the present technology. The exemplary system 400 may include an animation creation module 402 and a real-time engine module (or real-time engine) 422. The exemplary system 400 may be implemented to generate a non-interactive 2D version of a story from an interactive version of the story. For example, the exemplary system 400 may generate a non-interactive 2D computer-animated film or television program from an interactive computer animation real-time experience based on VR technology or AR technology. In another example, the exemplary system 400 may generate a first-person 2D computer animation film or television program from a character-based computer animation real-time experience based on VR technology or AR technology. The character-based computer animation real-time experience may involve an immersive experience in which the viewer can participate in the story as a character. Many variations are possible. In contrast to the exemplary system 200 of FIG. 2, the exemplary system 400 allows camera placement and shot creation to occur in relation to the animation creation module 402 instead of the real-time engine module 422. Thus, the exemplary system 400 may be well-suited for content that is primarily linear in nature, where the traditional animation in the animation creation module 402 matches the animation in the real-time engine module 422. The animation creation module 402 and the real-time engine module 422 may be implemented in one or more software applications operating on one or more computing devices. The components (e.g., modules, elements, etc.) shown in this figure and all figures in this specification are exemplary only, and in other implementations, additional components, fewer components, integrated components, or different components may be included. Some components may not be shown so as not to obscure the relevant details.In various embodiments, one or more of the features described in connection with the animation creation module 402 and the real-time engine module 422 may be implemented in any suitable order and combination.

[0057] In some embodiments, the various modules and / or applications described herein may be implemented, in part or in whole, as software, hardware, or any combination thereof. Generally, as described herein, a module and / or application may be associated with software, hardware, or any combination thereof. In some implementations, one or more functions, tasks, and / or operations of a module and / or application may be performed or executed by software routines, software processes, hardware, and / or any combination thereof. In some cases, the various modules and / or applications described herein may be implemented, in part or in whole, as software operating on one or more computing devices or systems such as on a user computing device or client computing device or on a server. For example, one or more of the modules and / or applications described herein, or at least a portion thereof, may be implemented as, or within, an application (e.g., app), program, or applet, etc. operating on a user computing device or client computing system. In another example, one or more modules and / or applications, or at least a portion thereof, may be implemented using one or more computing devices or systems including one or more servers such as a network server or cloud server. It should be understood that there may be many variations or other possibilities. In one exemplary embodiment, the animation creation module 402 may be implemented in or using animation creation software such as Autodesk® Maya, and the real-time engine module 422 may be implemented in or using a real-time engine such as the Unity® game engine.

[0058] The animation creation module 402 may include a VR animation module 404, a 2D preview module 406, a 2D shot creation module 408, and a shot fix module 410.

[0059] The VR animation module 404 may be configured to animate an interactive version of a story. For example, the interactive version of the story may be a computer animation real-time experience based on VR technology or AR technology. The interactive version of the story may be composed of a sequence of shots. A shot may represent several frames captured by a virtual camera located within a computer animation real-time experience in three-dimensional space. In some embodiments, a shot may represent a single character action such as an animation cycle or a cycle animation. Further, the sequence of shots may correspond to several related shots. For example, the sequence may include shots captured at a specific location within the computer animation real-time experience.

[0060] The 2D preview module 406 can be configured to arrange a set of virtual cameras in an interactive version of the story as animated by the VR animation module 404. For example, an artist may instruct the 2D preview module 406 to arrange a set of virtual cameras in an interactive version of the story based on the 2D film script 412. For example, the 2D film script 412 can be written based on a script associated with the interactive version of the story. The set of virtual cameras can be placed within the interactive version of the story to image the animation scene from various different camera angles. The 2D preview module 406 can also be configured to export a 2D "playblast" feed (or movie) for each virtual camera. For example, the 2D preview module 406 can provide a camera recording interface that provides options for recording the data captured by each camera. In some embodiments, as shown in the exemplary FIG. 6, the 2D preview module 406 can insert reticle information and slate information into each frame of the exported 2D "playblast" feed. The reticle information and slate information can be provided as a frame overlay. In some embodiments, the reticle information and slate information can include reticle data (e.g., camera boundaries), sequence data, file names, camera lenses, camera / shot names, timestamps, take numbers or iteration numbers, and animation frame numbers. The 2D preview module 406 can interact with the 2D editing process 414. For example, the 2D editing process 414 can provide edit cuts that combine various camera shots from the exported "playblast" feed into a single sequence movie. The edit cuts can be produced using non-linear video editing software. The edit cuts can be reviewed, and if necessary, new virtual cameras can be introduced and / or further adjustments can be made to existing virtual cameras. In one embodiment, each virtual camera represents a single shot.In another embodiment, multiple shots may be constructed from a single camera.

[0061] The 2D shot creation module 408 may be configured to create new shots and update existing shots. Generally, the 2D shot creation module 408 may create shots from the edited cuts provided by the 2D editing process 414. The 2D shot creation module 408 may store each shot and its associated metadata information in the data store 416. An example of shot metadata information associated with a virtual (or animation) camera is shown in exemplary FIG. 7. For example, the shot metadata information may include a start frame, an end frame, a production name, a sequence, a version, and a rendition. Many variations are possible. In one embodiment, the 2D shot creation module 408 may update a shared file storing the shot data. For example, the 2D shot creation module 408 may export the shot data associated with one or more identified virtual cameras to a specific output location such as the data store 416. In one embodiment, the data store 416 resides on a remote server. In another embodiment, a web-based application interfaces with various modules in the animation creation module 402, the real-time engine module 422, and the 2D editing module 414 to synchronize the data in the data store 416. FIG. 8 shows an exemplary export interface for exporting shot data. In another embodiment, the 2D shot creation module 408 may update a show database accessible through the data store 416. The 2D shot creation module 408 may also export the virtual camera information to, for example, a file system or a cloud-based storage system.

[0062] The shot fix module 410 may be configured to apply various fixes to shots associated with a non-interactive version of a story generated from an interactive version of the story. For example, the shot fix module 410 may apply a fix to a shot generated by the real-time engine module 422 for a non-interactive version of the story. Generally, shots associated with a non-interactive version of a story may be derived from shots associated with an interactive version of the story. As a result, updates to shots associated with an interactive version of the story may be reflected in corresponding shots associated with a non-interactive version of the story. In some examples, specific adjustments may be required that are only applied to shots associated with a non-interactive version of the story. For example, the character eyelines represented in a shot may be correct in an interactive version of the story, but may appear to be looking in the wrong direction in a shot associated with a non-interactive version of the story. In such an example, the shot fix module 410 may be configured to branch shots that require specific adjustments for a non-interactive version of the story. When a shot is branched, a separate copy of the shot is maintained for both the interactive and non-interactive versions of the story. Adjustments to the shot may be made to the non-interactive version of the story without affecting the interactive version of the story. In the foregoing example, adjustments may be made to the character eyelines of a shot associated with a non-interactive version of the story without affecting the character eyelines of the corresponding shot associated with the interactive version of the story. In some examples, a non-interactive version of the story may require that new shots be created.In one example, one shot in an interactive version of a story is split into several smaller and potentially overlapping shots in a non-interactive 2D version corresponding to different camera angles (e.g., a close-up, a medium shot, and for establishing the shot). These new shots may correspond to computer animations created especially for the non-interactive version of the story. For example, the look associated with a character for the interactive version of the story may not be acceptable in the associated shots from the non-interactive version of the story. In another example, a shot associated with an interactive version of a story may involve some interaction by the viewer. Such interaction may be allowed in the interactive version of the story, but the interaction is not suitable for duplication in the non-interactive version of the story. In such an example, the shot fix module 410 may permit the creation of new shots for the non-interactive version of the story that correct or replace existing shots associated with the interactive version of the story. The shot fix module 410 may store data associated with the shots fixed or added in relation to the non-interactive version of the story in the data store 416.

[0063] The real-time engine module 422 may include a VR writing and FX module 424, a 2D shot import module 426, a 2D layout module 428, and a 2D writing and FX module 430.

[0064] The VR writing and FX module 424 may be configured to apply writing and media effect (FX) elements for the interactive version of the story. The writing and FX elements may generally be created using known techniques.

[0065] The 2D shot import module 426 may be configured to import shot data into the real-time engine module 422. For example, once a shot and camera are added to the data store 416 (e.g., a show database), the 2D shot import module 426 may provide an interface that provides options for importing virtual camera feeds and 2D shot data into the real-time engine module 422, as shown in the example of FIG. 9. A new shot timeline may be set up for each 2D shot for which camera data is imported. For example, the directory storing the 2D shot data (e.g., virtual camera feed) may be identified through the interface. The interface also provides options for viewing shot information (e.g., frame range, path, etc.) for a given shot, re-importing the shot, and deleting the shot. When importing a shot, the 2D shot import module 426 may create additional 2D data for each shot. The additional 2D data may create a layer on top of the existing AR / VR content. This enables additional per-shot adjustments to be made while using the existing AR / VR content as a basis. In one embodiment, the 2D data may be represented as an additional timeline that is adjusted per department and applied on top of the AR / VR content timeline.

[0066] The 2D layout module 428 may be configured to enhance the shots. For example, the 2D layout module 222 may apply additional set dressing and layouts for each shot.

[0067] The 2D writing and FX module 430 can be configured to provide options for adding additional features to a shot. For example, the 2D writing and FX module 430 can apply lights, shadows, contact shadows, FX elements, and post - processing effects such as subject blur and depth of field. For example, additional features can be applied as directed by a writing artist and an FX artist. Applying additional features can enhance the appearance of each shot. In one embodiment, the 2D writing and FX module 430 represents VR content as a set of master timelines representing character animations, writing, and FX elements. In this embodiment, the content operates mainly linearly. In another embodiment, the content is non - linear and uses additional systems in addition to timelines such as finite state machines, blend trees, and custom AI systems.

[0068] The 2D composition unit 432 can generate a final rendering frame based on generally known approaches. For example, the 2D composition unit 432 can generate a final rendering frame based on the final frame and layer passes when rendered by the real - time engine module 422. The 2D composition unit 432 can also generate a non - interactive version of the story, such as a 2D computer - animated movie 434, based on the final rendering frame.

[0069] FIG. 5A shows an exemplary method 500 according to an embodiment of the present technology. For example, method 500 may be executed by system 400 of FIG. 4. At block 502, sequences and shots may be animated for AR / VR animations or experiences in an animation creation tool as described above with reference to VR animation module 404. At block 504, a virtual camera may be placed within the animation creation tool as described above with reference to 2D preview module 406. At block 506, edit cuts may be produced in non-linear video editing software as described above with reference to 2D editing process 414. At block 508, once an edit cut is approved, 2D shots are created from the camera and timing metadata as described above with reference to 2D shot creation module 408. At block 510, fixes may be applied to the 2D shots and additional 2D shots may be created as described above with reference to shot fix module 410. At block 512, lighting and FX elements may be created for AR / VR animations or experiences as described above with reference to VR lighting and FX module 424. At block 514, 2D shot data may be imported into the real-time engine as described above with reference to 2D shot import module 426. At block 516, set decoration and layout may be applied to the shots as described above with reference to 2D layout module 428. At block 518, additional lights, shadows, contact shadows, FX elements, and post-processing effects may be applied as described above with reference to 2D lighting and FX module 430.

[0070] Many variations to the exemplary method are possible. It should be understood that within the scope of the various embodiments described herein, additional steps, fewer steps, or alternative steps may be performed in the same or alternative order, or in parallel, unless otherwise specified.

[0071] FIG. 5B shows an exemplary method 550 according to an embodiment of the present technology. For example, method 550 may be executed by system 400 of FIG. 4. At block 552, data associated with a computer-based experience may be acquired. The computer-based experience may be based on interactive real-time technology. For example, the interactive real-time technology may apply augmented reality (AR) and / or virtual reality (VR) technology to provide interactive content. At block 554, at least one virtual camera may be configured within the computer-based experience in an animation creation application. At block 556, a plurality of shots corresponding to 2D content may be generated from edit cuts of content captured by at least one virtual camera in an animation creation application. At block 558, data associated with a 2D version of the computer-based experience may be generated in a real-time engine based on the plurality of shots. The 2D version may be rendered based on the generated data.

[0072] Many variations to the exemplary method are possible. It should be understood that within the scope of the various embodiments described herein, there may be additional steps, fewer steps, or alternative steps that are performed in a similar or alternative order, or in parallel, unless otherwise specified.

[0073] FIG. 10 shows an exemplary shot picker interface that can be used to select shots within a real-time engine (e.g., real-time engine module 212, real-time engine module 422) according to one embodiment of the present technology. The shot picker interface may be provided by a 2D shot creation module 220 or a 2D shot creation module 408. The interface may provide a list of all shots available for a given scene. Once a shot selection is made, the scene may be configured to use the corresponding shot timeline. As a result, an artist can refine the shot within the timeline window of the real-time engine. Additionally, the shot picker interface may provide useful workflow objects such as components to hide geometry per shot, and options to access game objects where additional 2D lights are present. Adjustments made to lighting in a 2D shot may be additive on top of AR / VR lighting. Further, any changes to AR / VR lighting are reflected in the corresponding 2D shot. In some embodiments, the lighting manager interface may provide an option by which an artist can break the link between the AR / VR scene and the 2D shot. Once the link is broken, changes to AR / VR lighting are not reflected in the corresponding 2D shot.

[0074] FIG. 11 shows an exemplary camera recording interface that may provide options for recording shots captured by a virtual camera according to one embodiment of the present technology. For example, the camera recording interface may be provided by a 2D shot creation module 220 or a 2D shot creation module 408. The camera recording interface may be customized to record various contents with different configurations in a single batch. Further, reticle information and slate information may be applied to the frame as described above. In one embodiment, the following information is included: reticle (camera boundary), sequence, file name, camera lens, camera / shot name, date, and animation frame number.

[0075] Figure 12 shows a real-time shot renderer interface according to an embodiment of the present technology. For example, the real-time shot renderer interface may be provided by the real-time engine module 212 or the real-time engine module 422. The real-time shot renderer interface may provide options for rendering the final frame using the real-time engine module 212 and the real-time engine module 422. For example, the real-time shot renderer interface may provide options for rendering previews or intermediate frames. In some embodiments, the real-time shot renderer interface may be used in relation to the camera recording interface. In such embodiments, the camera recording interface may be used to identify the virtual camera and slate information to be included in the frame. And the real-time shot renderer interface may be used to render the frame using the slate information. The real-time shot renderer interface may provide options for selecting different resolutions for rendering. The real-time shot renderer interface may also provide options for creating an EXR and producing multiple render layer passes (e.g., depth, matte, and beauty) for the 2D compositing section. In various embodiments, the frame may be rendered using the real-time engine module 212 or the real-time engine module 422. In one embodiment, the rendering software may activate the selected shots and render them one by one. In one embodiment, the rendering software may render the shots in parallel on one or more remote machines. The rendering may be performed in the editor. That is, instead of rendering the shots from the build, the rendering software puts the real-time engine module 212 or the real-time engine module 422 into "playback mode" and then proceeds to gather the rendering textures of the shot camera from the shot camera for each frame.In one embodiment, the rendering software may operate in relation to a recording system built into the real-time engine module 212 or the real-time engine module 422. Rendering may be performed in the in-editor. In this embodiment, the user may identify recorder clips on a timeline that defines the area to be rendered. These clips also define which optional output variables (AOVs) are to be rendered. An AOV or rendering pass provides a way to render different images with any arbitrary shading network component. For example, an artist may render separate depth, matte, and beauty passes and later recombine them in a compositing process. The selected AOVs function as additional rendering passes for the forward rendering pipeline. In particular, these passes render in accordance with user-defined rendering textures instead of the camera's rendered textures. The recorder for these passes then writes these user-defined rendering textures to disk. To facilitate the compositing process, the AOV passes for a single frame are written into one EXR file, for example, using the SimpleImageIO library.

[0076] FIG. 13 shows an exemplary rendered frame. The frame may be rendered using the real-time engine module 212 or the real-time engine module 422. The frame may include metadata information such as the animation name, sequence, file name, time stamp, identification number associated with the frame in the animation creation application, and identification number associated with the frame in the real-time engine.

[0077] FIG. 14 shows a timeline interface that can be used to create an additional 2D shot timeline that creates layers on top of the VR master timeline. For example, the timeline interface may be provided by the 2D shot import module 426.

[0078] Figure 15 shows another timeline interface that enables a writing artist to make shot-specific additional improvements without the need to modify the underlying VR writing. In some embodiments, making these additional changes does not require re-running computationally expensive global illumination calculations.

[0079] Figure 16 shows a combined writing interface that incorporates the aforementioned writing interface. The combined writing interface may be provided by the real-time engine module 212 or the real-time engine module 422.

[0080] Figure 17 shows an exemplary contact sheet that may be generated based on the present technology. The contact sheet may include frames associated with one or more shots.

[0081] Figures 18 to 24 show various improvements that can be made based on the present technology. For example, the present technology can improve shot writing in 2D content. Generally, the writing for interactive projects and VR projects often appears to be accurate, but this writing may not produce the intended effect when configured for a specific shot angle. For example, Figure 18 shows 2D content with writing from an AR / VR version. In this example, the crow character 1802 is intended to be in focus. However, the writing from the AR / VR version does not show the crow character 1802 in focus. As described above, with the present technology, an artist can create layers on shot writing in 2D content to emphasize the crow character 1802 in focus as shown in the example of Figure 19. With the present technology, an artist can add a contact shadow to the turtle character 2002 as shown in the example of Figure 20. Further, with the present technology, an artist can apply post-processing (e.g., depth of field) to focus the viewer's attention and make the shot cinematic as shown in the example of Figure 21. Also with the present technology, an artist can add additional FX elements to 2D content that is not suitable for AR / VR content. For example, Figure 22 shows the original shot 2202 from AR / VR content. Figure 22 also shows an improved 2D shot 2212 derived from the original shot 2202. In the improved 2D shot 2212, an FX element corresponding to snow footprints is added to place the turtle character on the ground in the environment. Also with the present technology, an artist can apply animation fixes to improve 2D content. For example, Figure 23 shows a frame from AR / VR content where characters are looking at a viewer who is playing a certain character in a computer animation real-time experience. In Figure 23, the virtual camera associated with the viewer who is playing a certain character in a computer animation real-time experience is being looked at by the characters.While acceptable for computer animated real-time experiences, the character actions reflected in Figure 23 would result in the character looking in the wrong direction and thus are not converted to 2D content. To address this, new animations are created by this technology and can be used as 2D content. For example, Figure 24 shows a new animation where the crow character 2402 is looking at the ant character 2404 while the animation of the ant character 2404 is smoothly connected to the previous shot in Figure 23. The new animation reflected in Figure 24 can be better adapted compared to the animation corresponding to the AR / VR content in Figure 23 for viewing as 2D content. Many variations are possible. [Hardware Implementation Forms]

[0082] The processes and features described above can be implemented in a variety of machines and computer system architectures in a variety of network and computing environments. Figure 25 shows an exemplary machine 2500 in which a set of instructions can be executed to cause the machine to execute one or more of the embodiments described herein, according to one embodiment of the present technology. The embodiments can relate to one or more systems, methods, or computer-readable media. The machine may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) distribution network environment.

[0083] Computer system 2500 includes a processor 2502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 2504, and a non-volatile memory 2506 (e.g., volatile RAM and non-volatile RAM, respectively), which communicate with each other via a bus 2508. The processor 2502 can be implemented in any suitable form such as a parallel processing system. In some cases, the exemplary machine 2500 can correspond to, include, or be included within a computing device or system. For example, in some embodiments, the machine 2500 can be a desktop computer, a laptop computer, a personal digital assistant (PDA), a household appliance, a wearable device, a camera, a tablet, or a mobile phone, etc. In one embodiment, the computer system 2500 also includes a video display 2510, an alphanumeric input device 2512 (e.g., a keyboard), a cursor control device 2514 (e.g., a mouse), a drive unit 2516, a signal generating device 2518 (e.g., a speaker), and a network interface device 2520.

[0084] In one embodiment, the video display 2510 includes a touch sensor screen for user input. In one embodiment, a touch sensor screen is used instead of a keyboard and a mouse. The disk drive unit 2516 includes a machine-readable medium 2522 in which one or more sets of instructions 2524 (e.g., software) that implement any one or more of the methodologies or functions described herein are stored. The instructions 2524 can also reside completely or at least partially within the main memory 2504 and / or within the processor 2502 during their execution by the computer system 2500. The instructions 2524 can further be transmitted or received over a network 2540 via the network interface device 2520. In some embodiments, the machine-readable medium 2522 also includes a database 2525.

[0085] The volatile RAM may be implemented as dynamic RAM (DRAM) that continuously requires power to refresh or maintain the data in the memory. Non-volatile memory generally includes magnetic hard drives, magneto-optical drives, optical drives (e.g., DVD RAM), or other types of memory systems that maintain data even after power is removed from the system. The non-volatile memory 2506 may also be random access memory. The non-volatile memory 2506 can be a local device directly coupled to the remaining components in the computer system 2500. Non-volatile memory that is remote from the system, such as a network storage device coupled to any of the computer systems described herein through a network interface such as a modem or Ethernet (registered trademark) interface, can also be used.

[0086] Although the machine-readable medium 2522 is shown as a single medium in one exemplary embodiment, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be taken to include any medium that is capable of storing, encoding, or holding a set of instructions for machine execution and that causes a machine to perform any one or more of the methodologies of the present technology. The term "machine-readable medium" should accordingly be taken to include, but not be limited to, solid-state memory, optical and magnetic media, and carrier signals. The term "storage module" as used herein may be implemented using a machine-readable medium.

[0087] Generally, routines executed to implement embodiments of the present invention may be implemented as part of an operating system or as a module or sequence of instructions called a particular application, component, program, object, "program" or "application". For example, one or more programs or applications may be used to execute any or all of the functions, techniques, and processes described herein. A program or application is generally a set of one or more instruction sets set at various times in various memories and storage devices in a machine, which, when read and executed by one or more processors, causes a computing system 2500 to perform operations that involve elements of various aspects of the embodiments described herein.

[0088] Executable routines and data may be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache memory. Some of these routines and / or data may be stored in any of the storage devices. Further, the routines and data may be obtained from a centralized server or a peer-to-peer network. Different portions of the routines and data may be obtained from different centralized servers and / or peer-to-peer networks at different times and different communication sessions, or in the same communication session. The routines and data may be obtained in their entirety prior to the execution of the application. Alternatively, some of the routines and data may be obtained dynamically just-in-time when needed for execution. Thus, the routines and data need not be entirely present on a machine-readable medium at a particular instance in time.

[0089] Although embodiments have been described entirely in the context of a computing system, those skilled in the art will understand that various embodiments can be distributed as program products in a variety of forms and that the embodiments described herein are equally applicable regardless of the particular type of machine or computer-readable medium used to actually achieve distribution. Examples of machine-readable media include, but are not limited to, among others, volatile memory devices and non-volatile memory devices, floppy disks and other removable disks, hard disk drives, and recordable types of media such as optical disks (e.g., compact disk read only memory (CD-ROM), digital versatile disk (DVD), etc.), as well as transmission-type media such as digital and analog communication links.

[0090] Alternatively, or in combination, the embodiments described herein can be implemented using dedicated circuits either with or without software instructions, such as using application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). Embodiments can be implemented using wiring circuitry either without software instructions or in combination with software instructions. Thus, the technology is not limited to any particular combination of hardware circuitry and software, or to any particular source for the instructions executed in a data processing system.

[0091] For purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the description. However, it will be apparent to one of ordinary skill in the art that the embodiments of the present disclosure may be practiced without these specific details. In some instances, modules, structures, processes, features, and devices are shown in block diagram form or described herein to avoid obscuring the description. In other instances, functional block diagrams and flow diagrams are shown to represent data and logical flows. The components of the block diagrams and flow diagrams (e.g., modules, engines, blocks, structures, devices, features, etc.) may be combined, separated, removed, reordered, and replaced in various ways different from those explicitly described and shown herein.

[0092] References in this specification to "one embodiment", "an embodiment", "another embodiment", "alternative embodiment", or "various embodiments" mean that a particular feature, design, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. For example, the appearances of the phrases "in one embodiment", "in an embodiment", "in one embodiment", "in various embodiments", or "in an alternative embodiment" in various places in this specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive alternative embodiments. Moreover, various features may be described that may be combined or included in some embodiments but omitted in other embodiments, regardless of whether there is an explicit reference to "embodiment" or the like. Similarly, various features may be described that may be preferences or requirements for some embodiments but not for others.

[0093] Although the embodiments have been described with reference to specific exemplary embodiments, it will be apparent that various modifications and variations can be made to these embodiments. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The foregoing specification provides an explanation with reference to specific exemplary embodiments. It will be apparent that various modifications can be made to this without departing from the broader spirit and scope as set forth in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0094] Some of the drawings show several operations or method steps in a particular order, but steps that are not affected by the order may be rearranged, and other steps may be combined or omitted. Although some rearrangements or other groupings are specifically mentioned, others will be apparent to those skilled in the art, so a comprehensive list of alternative forms is not presented. Moreover, it should be recognized that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0095] It should also be understood that various changes may be made without departing from the gist of the present invention. Such changes are also implicitly included in this description. They are still included within the scope of the present invention. It should be understood that this disclosure is intended to result in patents that encompass many aspects of the present invention, both independently and as an overall system, and in both method and apparatus modes.

[0096] Furthermore, each of the various elements of the present invention and the claims may also be achieved in various ways. This disclosure is to be understood as encompassing each such variation, whether it be a variation of an embodiment of any device, an embodiment of a method or process, or even simply a variation of any of these elements.

[0097] Furthermore, the use of the transitional phrase "comprising" is used to maintain the "open-ended" scope of the claims herein with respect to the interpretation of conventional claim scope. Thus, unless specifically required by the context, the term "comprise", or variations such as "comprises" or "comprising", are intended to imply the inclusion of the stated elements or steps, or groups of elements or steps, and are not intended to exclude any other elements or steps, or groups of elements or steps. Such terms should be construed in their broadest form so as to afford the applicant the broadest legally permissible scope for the claims that follow.

[0098] The terms used herein are mainly selected for readability and for the purpose of instruction and may not have been selected to detail or limit the subject matter of the invention. Thus, the scope of the invention is intended to be limited not by this detailed description but rather by any claims that may arise from an application based on this specification. Thus, the disclosure of embodiments of the invention is intended, but not limited to, to be illustrative of the scope of the invention as set forth in the following claims.

Claims

1. A computing system obtains data associated with a computer-based experience, wherein the computer-based experience is based on interactive real-time technology; the computing system configures at least one first virtual camera associated with a first timeline within the computer-based experience in an animation creation application; the computing system generates, for the first timeline, a first plurality of shots corresponding to two-dimensional content from an edit cut of content including at least a first object imaged by the at least one first virtual camera in the animation creation application; the computing system imports a second timeline different from the first timeline of two-dimensional content generated based on a second plurality of shots including at least a second object imaged by at least one second virtual camera; the computing system generates data associated with a two-dimensional version of the computer-based experience in a real-time engine based on the first plurality of shots and the second plurality of shots, wherein the two-dimensional version can be rendered based on the generated data; A method executed by a computer, comprising the above steps.

2. The method according to claim 1, further comprising the step of the computing system rendering the two-dimensional version in the real-time engine.

3. The method according to claim 1 or 2, wherein the step of generating the first plurality of shots corresponding to the two-dimensional content further comprises: the computing system obtains data associated with the edit cut of the computer-based experience from non-linear video editing software.

4. The method according to claim 1 or 2, wherein the step of generating the first plurality of shots corresponding to the two-dimensional content further comprises: ​ ​ The computing system imports data associated with the first plurality of shots generated in the animation creation application into the real-time engine The method executed by a computer according to any one of claims 1 to 3, further comprising: **Claim 5** The computing system creates a first animation layer associated with the two-dimensional version separated from a second animation layer associated with the computer-based experience, wherein adjustments made to the shots in the first plurality of shots are applied to the first animation layer The method executed by a computer according to claim 4, further comprising: **Claim 6** The computing system creates a first animation layer associated with a first timeline associated with the two-dimensional version separated from a second animation layer associated with the computer-based experience, wherein adjustments made to the first timeline corresponding to the two-dimensional version are applied to the first animation layer The method executed by a computer according to claim 4, further comprising: **Claim 7** The step of generating the first plurality of shots corresponding to the two-dimensional content The computing system applies one or more animation fixes to at least one shot for the two-dimensional version in the animation creation application The method executed by a computer according to any one of claims 1 to 6, further comprising: **Claim 8** In the animation creation application, the step of applying the one or more animation fixes to at least one shot for the two-dimensional version The computing system creates a copy of the at least one shot prior to the application of the one or more animation fixes, and the one or more animation fixes are applied to the copy The method executed by a computer according to claim 7, further comprising: **Claim 9** The computing system, in the real-time engine, applies writing and one or more media effects to a frame associated with the computer-based experience; The computing system, in the real-time engine, applies writing and one or more media effects to a frame associated with the two-dimensional version, wherein the writing and the one or more media effects for the frame associated with the two-dimensional version are added on top of the writing and the one or more media effects applied to the computer-based experience; The method executed by a computer according to any one of claims 1 to 8, further comprising.

10. The computer-based experience is based on immersive real-time technology, or The data associated with the two-dimensional version is interactive 2D content, The method executed by a computer according to any one of claims 1 to 9.

11. A system comprising: At least one processor; and When executed by the at least one processor, causing the system to: Obtain data associated with a computer-based experience, wherein the computer-based experience is based on interactive real-time technology; In an animation creation application, configure at least one first virtual camera associated with a first timeline within the computer-based experience; In the animation creation application, generate a first plurality of shots corresponding to two-dimensional content for the first timeline from an edit cut of content including at least a first object imaged by the at least one first virtual camera; Import a second timeline different from the first timeline of two-dimensional content generated based on a second plurality of shots including at least a second object imaged by at least one second virtual camera; and Based on the first plurality of shots and the second plurality of shots, in a real-time engine, generating data associated with a two-dimensional version of the computer-based experience, wherein the two-dimensional version can be rendered based on the generated data A memory storing instructions for causing a method having the above to be executed A system comprising the above **Claim 12** The instructions cause the system to Render the two-dimensional version in the real-time engine The system according to claim 11, further comprising the above **Claim 13** The step of generating the first plurality of shots corresponding to the two-dimensional content further comprises Obtaining data associated with the editing cuts of the computer-based experience from non-linear video editing software The system according to claim 11 or 12, further comprising the above **Claim 14** The step of generating the first plurality of shots corresponding to the two-dimensional content further comprises Importing data associated with the first plurality of shots generated in the animation creation application into the real-time engine The system according to any one of claims 11 to 13, further comprising the above **Claim 15** The instructions cause the system to Create a first animation layer associated with the two-dimensional version separated from a second animation layer associated with the computer-based experience, wherein adjustments made to the shots in the first plurality of shots are applied to the first animation layer The system according to any one of claims 11 to 14, further comprising the above **Claim 16** In a computing system A procedure for obtaining data associated with a computer-based experience, wherein the computer-based experience is based on interactive real-time technology In an animation creation application, a procedure for configuring at least one first virtual camera associated with a first timeline within the computer-based experience In the animation creation application, for the first timeline, a procedure for generating a first plurality of shots corresponding to two-dimensional content from an edit cut of content including at least a first object imaged by the at least one first virtual camera; A procedure for importing a second timeline different from the first timeline of two-dimensional content generated based on a second plurality of shots including at least a second object imaged by at least one second virtual camera; A procedure for generating data associated with a two-dimensional version of the computer-based experience in a real-time engine based on the first plurality of shots and the second plurality of shots, wherein the two-dimensional version can be rendered based on the generated data; A computer program for causing execution.

17. In the computing system, A procedure for rendering the two-dimensional version in the real-time engine The computer program according to claim 16, further causing execution.

18. The procedure for generating the first plurality of shots corresponding to the two-dimensional content A procedure for obtaining data associated with the edit cut of the computer-based experience from non-linear video editing software The computer program according to claim 16 or 17, further comprising.

19. The procedure for generating the first plurality of shots corresponding to the two-dimensional content A procedure for importing data associated with the first plurality of shots generated in the animation creation application into the real-time engine The computer program according to any one of claims 16 to 18, further comprising.

20. In the computing system, A procedure for creating a first animation layer associated with the two-dimensional version separated from a second animation layer associated with the computer-based experience, wherein adjustments made to the shots in the first plurality of shots are applied to the first animation layer; The computer program according to any one of claims 16 to 19, further causing execution.

Images