Multimedia editing system and method

Abstract

A method, including: generating, a graphical user interface including an interactive timeline region on a touchscreen display; displaying within the interactive timeline region a view of at least a portion of a timeline of a multimedia project, the timeline having graphical representations of items of media content from which a multimedia production is to be generated, at least some of the representations are displayed at respective vertical positions within the timeline representing respective compositing layers, and a horizontal position of each representation represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; responsive to dual-axis zoom touch gestures to concurrently change vertical and horizontal zoom levels of the displayed view, re-generating and rendering the view on the display, in real-time during the touch gesture to dynamically update the displayed view in a continuous manner during and in accordance with the corresponding touch gesture.

Description

[0001] MULTIMEDIA EDITING SYSTEM AND METHOD

[0002] TECHNICAL FIELD

[0003] The present invention relates to methods, systems and applications for creating and editing digital multimedia productions such as movies, film clips and animations from items of media content.

[0004] BACKGROUND

[0005] Multimedia productions such as movies, film clips, motion graphics and animations are created and / or edited using media compositing and editing systems and applications. There are many such systems and applications available, with varying capabilities and tailored to different categories of end users, from inexperienced casual users to industry professionals. These systems and applications typically provide at least a subset of functions such as raw content creation, motion design, digital compositing, colour correction, visual effects, and other postproduction functions. Examples of such applications include ToonBoom, DaVinci Resolve, iMovie and Final Cut Pro from Apple Inc, and Adobe Premiere Pro. These are all examples of timeline-based non-linear video editing applications that allow a multimedia or video project to be created by importing and editing separate constituent components such as audio tracks, video clips and still images, arranging them individually with respect to a common temporal scale or 'timeline', and finally combining them to generate output video productions.

[0006] The usability and capabilities of such applications varies dramatically, dependent in part on whether the applications have been designed for casual consumers or industry professionals. Regardless of category, the applicability, usability and efficiency of a given application is critically dependent upon its user interface. Although some applications are extremely powerful in their capabilities, they can be difficult to learn, and difficult or inconvenient to use even after learning.

[0007] It is desired to overcome or alleviate one or more difficulties of the prior art, or to at least provide a useful alternative. SUMMARY

[0008] In accordance with some embodiments of the present invention, there is provided a computer-implemented method, including the steps of: generating, with a computer system, a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline region; displaying within the interactive timeline region and on the touchscreen display a view of at least a portion of a timeline of a multimedia project, the timeline being a visual representation of the multimedia project with graphical representations of items of media content from which a multimedia production is to be generated, wherein at least some of the graphical representations of the items of media content are displayed at respective vertical positions within the timeline representing respective compositing layers, and a horizontal position of each of the graphical representations within the timeline represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; responsive to touch gestures by a user on the interactive timeline region of the touchscreen display, changing the view displayed within the interactive timeline region, the touch gestures including zoom touch gestures to change zoom levels of the displayed view, and translation touch gestures to pan the displayed view; wherein the step of changing the view includes, responsive to at least the zoom touch gestures, dynamically re-generating the view, and causing the regenerated view to be rendered to the touchscreen display, in real-time during the corresponding touch gesture to dynamically update the displayed view in a continuous manner during and in accordance with the corresponding touch gesture; wherein the zoom touch gestures include dual-axis zoom touch gestures to concurrently change vertical and horizontal zoom levels of the displayed view.

[0009] In some embodiments, the step of dynamically re-generating the view includes: storing in a cache objects representing respective items of media content that are visible in the view to be displayed, the objects being instances of a packed format data structure; determining that the objects stored in the cache do not include one or more objects representing respective items of media content that are visible in the view to be displayed; responsive to the determining, retrieving, from non-volatile storage, media data representing the one or more items of media content that are visible in the view to be displayed; processing the retrieved media data to generate one or more corresponding objects as respective instances of the packed format data structure representing the one or more items of media content that are visible in the view to be displayed; storing the generated objects in the cache; and culling from the cache objects representing respective items of media content that are not visible in the view to be displayed.

[0010] In some embodiments, the media data is stored on the non-volatile storage as B-trees in an ordered key value store.

[0011] In some embodiments, the step of dynamically re-generating the view includes processing the objects stored as instances of the packed format data structure in the cache to generate corresponding timeline objects in a timeline format based on a translation and scale of the view to determine absolute screen coordinates for the objects. In some embodiments, the timeline objects are generated only if they represent interactive items.

[0012] In some embodiments, the method includes storing the timeline objects in linear arrays for efficient access. The method may include using the timeline objects to effect user interaction with items of media content.

[0013] In some embodiments, the step of dynamically re-generating the view includes processing the timeline objects to generate corresponding render geometry data in the form of low-level shader commands of a graphics processing unit (GPU) and associated data, and sending the render geometry data to the GPU via a low-level interface to cause the GPU to efficiently render the view on the touchscreen display.

[0014] In some embodiments, the method includes determining visual stylings from properties of the objects in timeline format, and generating corresponding render geometry data for the determined visual stylings.

[0015] In some embodiments, the render geometry data represents a plurality of rendering layers, including a track layer to render track backgrounds, a content layer to render rectangles containing content over the track layer, and an adornment layer to render content colour tags over the content layer.

[0016] In some embodiments, the method further includes storing the multimedia project on a non-volatile storage medium as B-trees in an ordered key value store.

[0017] In some embodiments, the method further includes, responsive to a non-uniform dualaxis zoom touch gesture among the dual-axis zoom touch gestures, concurrently changing the vertical and horizontal zoom levels of the displayed view by corresponding different amounts during and as indicated by the non-uniform dual-axis zoom touch gesture.

[0018] In some embodiments, the re-generating includes dynamically changing one or more of the graphical representations of items of media of the displayed view in accordance with the zoom levels.

[0019] In some embodiments, the method further includes, responsive to a vertical zoom touch gesture among the zoom touch gestures, changing a zoom level of the displayed view in the vertical dimension only. In some embodiments, the method further includes, responsive to a horizontal zoom touch gesture among the zoom touch gestures, changing a zoom level of the displayed view in the horizontal dimension only.

[0020] In some embodiments, the method further includes, responsive to a composite touch gesture among the zoom touch gestures, concurrently changing at least one zoom level of the displayed view as indicated by a zoom gesture component of the composite touch gesture, and panning the displayed view in any direction indicated by a translate gesture component of the composite touch gesture.

[0021] In some embodiments, the method further includes, responsive to a translate touch gesture among the touch gestures, panning the displayed view along any path indicated by the translate touch gesture. In some embodiments, the displayed view is panned in dynamic dependence on translation speeds of the translation touch gestures on the touchscreen display such that a fast translation of a translation touch gesture causes the displayed view to be panned further than the same translation touch gesture performed slowly. In some embodiments, the method further includes, responsive to a double tap touch gesture at a location within an item of content, increasing a zoom level of the displayed view of the timeline to display individual frames of the item of content, and the displayed view to be spatially translated such that a frame of the item of content corresponding to a temporal location of the double tap touch gesture is centered in the timeline region.

[0022] In some embodiments, the double tap touch gesture is a first double tap touch gesture and was performed when the displayed view was at a first zoom level, and, responsive to a subsequent double tap touch gesture, decreasing the zoom level of the displayed view to the first zoom level, and panning the displayed view to centre a location of the subsequent double tap touch gesture in the timeline region.

[0023] In some embodiments, the method further includes, responsive to a flick touch gesture on the timeline region, causing playback of the multimedia project.

[0024] In some embodiments, the method further includes, responsive to a selection gesture on the interactive timeline region of the touchscreen display, selecting a corresponding plurality of displayed items of media content indicated by the selection gesture so that one or more actions can be collectively performed on the selected items of media. In some embodiments, the method further includes, responsive to a deselection gesture on the interactive timeline region of the touchscreen display, deselecting from the selected items of media content one or more corresponding items of the media content indicated by the deselection gesture.

[0025] In some embodiments, the method further includes, defining a group whose members are the selected items of media, and displaying the group in the timeline region with thumbnail previews of content of the group; and, responsive to corresponding user input, toggling a display state of a group between an open state and a closed state; and displaying a group in the open state together with its member items of content displayed vertically and aligned horizontally with one another.

[0026] In some embodiments, the method further includes, responsive to a split touch gesture on the interactive timeline region of the touchscreen display, splitting each of one or more corresponding items of media content into corresponding separate portions.

[0027] In some embodiments, the method further includes, responsive to a composite split gesture on the interactive timeline region of the touchscreen display, the composite split gesture having a split gesture component and a subsequent split adjust gesture component, splitting each of one or more corresponding items of media content into corresponding separate portions indicated by the split gesture component, and dynamically adjusting the temporal location at which the one or more items of media are to be split into separate portions as indicated by the split adjust gesture component.

[0028] In some embodiments, the timeline region of the graphical user interface includes an interactive playhead component whose location within the timeline region represents an active temporal location within an active item of content, the interactive playhead component being responsive to corresponding touch gestures to allow selection of a desired action from a list of available actions appropriate for the content type of the item of content.

[0029] In some embodiments, the method includes displaying looped playback of a portion of the multimedia production whose starting and ending times within the multimedia production correspond to the leftmost and rightmost edges of the displayed view of the timeline region.

[0030] In some embodiments, the graphical user interface includes a media display region to display a preview of the multimedia production or an item of media content, and the method further includes, responsive to a corresponding touch gesture on the graphical user interface, expanding the media display region to facilitate playback viewing, and replacing the timeline region with a corresponding reduced-size timeline window floating over the expanded media display region.

[0031] In some embodiments, the method includes, responsive to corresponding touch gestures on the reduced-size timeline window, previewing a corresponding portion of an animation in the media display region by jumping back and forth between corresponding frames of the animation.

[0032] In some embodiments, the method includes, responsive to corresponding touch gestures on the reduced-size timeline window, previewing a corresponding portion of an animation in the media display region by skipping forward and backward in a sequence of animation frames to the next frame that is different from the current frame. In some embodiments, the method includes, responsive to corresponding touch gestures on the timeline region, tagging items of media with corresponding display colours to visually identify and differentiate different subsets of items of media content.

[0033] In some embodiments, the method includes determining that a zoom level of the timeline region is greater than a threshold at a first time, and, in response, displaying the names of colour tagged items of content in a shape filled with the corresponding tag colour, and determining that the zoom level of the timeline region is not greater than a threshold at a second time and, in response, displaying colour tagged items of content as corresponding coloured horizontal lines representing time locations and durations of the colour tagged items of content.

[0034] In some embodiments, an item of keyframed content is displayed in the timeline region with temporally spaced thumbnail images representing keyframes of the item of keyframed content, and the method further includes, responsive to a corresponding touch gesture on the item of keyframed content, displaying graphical representations of one or more keyframe parameters of the item of keyframed content at respective vertical locations in the timeline region.

[0035] In some embodiments, the method includes, responsive to the corresponding touch gesture on the item of keyframed content, displaying graphical representations of only those one or more of the keyframe parameters that change between keyframes, and responsive to a further touch gesture on the item of keyframed content, displaying graphical representations of all of the keyframe parameters of the item of keyframed content at respective vertical locations in the timeline region.

[0036] In accordance with some embodiments of the present invention, there is provided a tangible, non-transitory, computer-readable storage medium having stored thereon a multimedia editing application that, when executed by at least one processor of a computing device or system, cause the at least one processor to perform the method of any one of the above methods.

[0037] In accordance with some embodiments of the present invention, there is provided a system including: a touchscreen display; a memory; and at least one processor configured to execute the method of any one of the above methods.

[0038] In accordance with some embodiments of the present invention, there is provided a tangible, non-transitory, computer-readable storage medium having stored thereon a multimedia editing application that, when executed by at least one processor of a computing device or system, cause the at least one processor to perform a method, including the steps of: generating, with a computer system, a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline region; displaying within the interactive timeline region and on the touchscreen display a view of at least a portion of a timeline of a multimedia project, the timeline being a visual representation of the multimedia project with graphical representations of items of media content from which a multimedia production is to be generated, wherein at least some of the graphical representations of the items of media content are displayed at respective vertical positions within the timeline representing respective compositing layers, and a horizontal position of each of the graphical representations within the timeline represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; responsive to touch gestures by a user on the interactive timeline region of the touchscreen display, changing the view displayed within the interactive timeline region, the touch gestures including zoom touch gestures to change zoom levels of the displayed view, and translation touch gestures to pan the displayed view; wherein the step of changing the view includes, responsive to at least the zoom touch gestures, dynamically re-generating the view, and causing the regenerated view to be rendered to the touchscreen display, in real-time during the corresponding touch gesture to dynamically update the displayed view in a continuous manner during and in accordance with the corresponding touch gesture; and wherein the zoom touch gestures include dual-axis zoom touch gestures to concurrently change vertical and horizontal zoom levels of the displayed view. Also described herein is a computer-implemented multimedia editing process for generating multimedia productions from items of media of a multimedia project, the process including the steps of: generating a graphical user interface for display on a touchscreen display, the graphical user interface including: an interactive timeline region for displaying views of graphical representations of items of media from which a multimedia production is to be generated, wherein at least some of the graphical representations of the items of media content are displayed at respective vertical positions within the timeline region representing respective compositing layers, and a horizontal position of each of the graphical representations within the timeline region represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; wherein the interactive timeline region of the graphical user interface is operable such that a user can perform dual-axis zoom touch gestures on the touchscreen display to simultaneously change a zoom level of a displayed view of the timeline region in vertical and horizontal dimensions; and dynamically re-generating and rendering the displayed view of the interactive timeline region in real-time during the dual-axis zoom touch gestures to reflect the dynamically changing zoom level.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Some embodiments of the present invention are hereinafter described, by way of example only, with reference to the accompanying drawings, in which:

[0041] Figures 1 to 46 are representative screenshots of the user interface ("UI") generated by the multimedia editing system and process in accordance with the described embodiments of the present invention, with example animation and movie projects loaded to illustrate interactive features of the UI and system, and wherein:

[0042] Figure 1 illustrates major components of the UI, in particular a media display region to preview media content in the upper part of the display, and a timeline region in the lower part of the display, showing graphic representations of items of media content from which a multimedia production is to be generated, separated by a toolbar with interactive controls;

[0043] Figure 2 shows an example of a global zoom-in touch gesture of the UI, in the form of a pinch-in gesture, which causes the system to concurrently increase the zoom levels of the displayed view of the timeline region in vertical and horizontal dimensions;

[0044] Figure 3 shows the results of the global zoom-in touch gesture of Figure 2, and also an example of a translate touch gesture to pan the displayed view;

[0045] Figure 4 shows the result of the translate touch gesture of Figure 3, and also a second pinch-in gesture to further zoom-in the displayed view;

[0046] Figure 5 shows the result of the second zoom-in gesture, providing a zoom level that allows access to individual frames of an animation, and also a global zoom-out touch gesture of the UI, in the form of a pinch-out gesture, which causes the system to simultaneously decrease the zoom level of the displayed view of the timeline region in vertical and horizontal dimensions;

[0047] Figure 6 shows the UI during editing of a different example animation project, with an example of a horizontal zoom-in touch gesture, in the form of a three-finger swipe right gesture, that causes the system to increase the zoom level of the displayed view of the timeline region in the horizontal dimension only; Figure 7 shows the result of the horizontal zoom-in touch gesture of Figure 6, providing a zoom level that allows access to individual keyframes; because the system play head was displayed when the horizontal zoom-in touch gesture was invoked, the zoom is anchored to the horizontal position of the playhead;

[0048] Figure 8 shows a horizontal zoom-out touch gesture in the form of a three-finger swipe left gesture that causes the system to decrease the zoom level of the displayed view of the timeline region in the horizontal dimension only;

[0049] Figure 9 shows the result of the horizontal zoom-out touch gesture of Figure 8, and also a further horizontal zoom-in touch gesture, but invoked without the playhead displayed, causing the system to zoom-in the display but anchored to the location at which the gesture was invoked;

[0050] Figure 10 shows the results of the horizontal zoom-out touch gesture of Figure 9;

[0051] Figure 11 shows an example of a vertical zoom-in touch gesture, in the form of a three-finger swipe up gesture, that causes the system to increase the zoom level of the displayed view of the timeline region in the vertical dimension only;

[0052] Figure 12 shows the result of the vertical zoom-in touch gesture of Figure 11, allowing thumbnail previews to be more clearly seen without changing the horizontal time scale;

[0053] Figure 13 shows an example of a vertical zoom-out touch gesture, in the form of a three-finger swipe down gesture, that causes the system to decrease the zoom level of the displayed view of the timeline region in the vertical dimension only;

[0054] Figure 14 shows the result of the vertical zoom-out touch gesture of Figure 13;

[0055] Figure 15 shows an example of a pan touch gesture, in the form of a singlefinger swipe gesture, that allows the displayed view to be panned ( / .e.^ spatially translated) in an arbitrary direction;

[0056] Figure 16 shows the result of the pan touch gesture of Figure 15;

[0057] Figure 17 shows an example of a double-tap gesture performed on an item of media content, which causes the system to dynamically zoom-in the display in both horizontal and vertical dimensions to clearly display a frame of the item of media content corresponding to the location of the double-tap gesture; Figure 18 shows the results of the double-tap gesture of Figure 17, and a second double-tap gesture performed on the item of media content, which causes the system to dynamically zoom-out the display in both horizontal and vertical dimensions, anchored to the location of the second double-tap gesture;

[0058] Figure 19 shows the results of the second double-tap gesture of Figure 18;

[0059] Figures 20 and 21 show an example of a quick-flick gesture on the playhead that causes the system to begin playback from the beginning of the item of media content under the playhead;

[0060] Figures 22 to 28 show examples of selection touch gestures that cause the system to select corresponding items of media content, allowing the system to collectively apply actions to the selected items;

[0061] Figures 29 and 31 show examples of cut touch gestures that cause the system to cut individual items of media into separate portions, and allowing the user to precisely adjust the temporal location of the cut;

[0062] Figures 32 and 33 show examples of deletion touch gesture is the cause the system to delete corresponding items of media content from a multimedia project;

[0063] Figure 34 illustrates the use of grouping to group together individual items of content so that they can be treated as though they were a single item of content for actions such as moving or applying effects;

[0064] Figures 35 to 37 show an example of using the UI playhead to apply an effect to an item of media content, in this case being a Gaussian blur effect;

[0065] Figure 38 shows an example of a "loop to screen" playback mode, whereby only the displayed portion of a selected item of media content is repeatedly played back on the media display region until interrupted by the user;

[0066] Figures 39 and 40 show an example of a compact timeline mode of the UI, invoked by a single-digit swipe down touch gesture such as the gesture shown in Figure 39, which causes the system to increase the size of the media display region by displaying only a compact representation of the timeline region with limited functionality in a floating window over the media display region, the compact timeline providing a flipbook function that allows the user to flick backwards and forwards between successive frames of an animation;

[0067] Figures 41 and 42 show an example of the UI's colour tags that facilitate locating, identifying, and grouping different items of media content by tagging them with different colours, the tags being displayed as background fill colours of content names when the zoom level is sufficient to enable the content names to be clearly seen, as shown in Figure 41, and as coloured horizontal lines when the zoom level is too low for the content names to be meaningfully displayed, as shown in Figure 42;

[0068] Figures 43 to 46 illustrate how keyframes can be displayed at different levels of detail, depending on the user's needs;

[0069] Figure 47 is a block diagram of the multimedia editing system in accordance with the described embodiments of the present invention;

[0070] Figure 48 is a schematic diagram of the architecture of the multimedia editing system of Figure 47 in accordance with the described embodiments of the present invention;

[0071] Figure 49 is a schematic diagram illustrating the sequence of transformations between data formats involved when rendering the timeline region in real-time; and

[0072] Figure 50 is a schematic diagram illustrating data objects of the system and their relationships.

[0073] DETAILED DESCRIPTION

[0074] In work leading up to the invention, the inventors identified difficulties with existing multimedia editing systems and applications. In particular, the inventors identified that existing timeline-based media editing systems and applications are hampered by a user interface paradigm that constrains the navigation of the timeline in various ways, making it inefficient and awkward for users, and limiting visibility and access to the constituent media assets of multimedia projects.

[0075] In addition, the inventors also recognised that existing multimedia editing systems and applications for video and animation have been developed for desktop computers with large monitors, which facilitates navigating large projects with many constituent media assets and of long duration (such as a feature-length film production, for example). However, the inventors believed that it should be possible to devise a graphical user interface (UI) for a portable touchscreen computing device with limited display size and computing power (such as an Apple iPad™) that would nevertheless facilitate the production of large multimedia projects. The inventors identified that a common limitation of existing video editing and animation applications is that they are generally only able to display around two to five tracks of media content at a time, which introduces issues when editing projects with many (e.g., ten or more) tracks because the user loses all context of content items outside of their limited visibility on the display, requiring the user to manually translate or zoom separately to regain context of the items in relation to other content items and to the entire project. This is a significant limitation, because projects with ten or more tracks are common. For example, a single video frame of a movie or animation may be composed using multiple techniques and effects, sound effects, music, animation elements, video footage, lighting, foreground elements, background elements, etc., each of which can involve multiple tracks or layers of content (displayed in a vertical stack) at any moment of time in a video editing or animation project. Because tracks are displayed as vertical stacks to create layered compositions, and arranged horizontally over the total duration of the project, the requirement to navigate large projects on an iPad™ screen as small as 8.3 inches in a convenient manner posed a significant and potentially insoluble challenge, recognising that streamlined interactions and navigation are critical to positive user experiences.

[0076] In this regard, the inventors determined that existing video editing applications limit project navigation to panning vertically between tracks and horizontally along the timeline, and adjusting the view zoom level between a small number of predetermined and fixed percentage values; for example, 25%, 50%, 75%, 100%, etc. These limitations severely impact user experience by making standard and frequent interactions such as context-based timing and adjustments cumbersome. The inventors identified that more flexible and powerful methods of navigation and scaling were needed.

[0077] Once the limitations of existing multimedia editing systems and applications had been identified, the inventors spent several years researching, developing and prototyping possible solutions until ultimately devising a technical solution that resolved all of the identified shortcomings of the prior art, and arriving at the present invention. The major requirements identified by the inventors and consequential practical issues arising from them are discussed further at the end of the description.

[0078] As described herein, embodiments of the present invention address difficulties of the prior art by providing a timeline-based multimedia editing system and application having a graphical user interface that represents a new paradigm in timeline navigation, and substantially simplifies the interactions required for producing multimedia productions such as movies, film clips, animations, and the like. By simplifying the user interactions required to navigate the timeline to locate and access specific items of media content, and assemble them into multimedia projects and productions, embodiments of the present invention not only greatly improve efficiency and usability for users, but also improve efficiency of the underlying system at both the software and hardware level.

[0079] Embodiments of the present invention are described herein in the context of a touchscreen display device in which a user interacts with the multimedia editing system or application by contacting or touching the touchscreen, often at multiple locations simultaneously, with respective parts of the user's body (typically, but not necessarily, being one or more of the user's fingers and / or thumb) and / or a stylus device such as an Apple Pencil™, and in some cases moving the contact locations, also referred to herein as 'touch points' (i.e., moving the contacting parts of the user's body or stylus device while maintaining contact with the touchscreen surface) to define an interaction 'touch gesture' that represents a corresponding user input to the multimedia editing system.

[0080] In the described embodiments, the touchscreen display device is the touchscreen display of an Apple iPad tablet computing device, as shown in Figure 47, and the multimedia editing processes described herein are implemented as programming instructions of a software application 4800 installed on the iPad device. However, it will be apparent to those skilled in the art that other types of computing device, including a desktop computer with a touchscreen display, may be used in other embodiments, and also that the processes described herein can be alternatively implemented in other forms, including, for example, as firmware, or as configuration data of a field- programmable gate array (FPGA), or as an application-specific integrated circuit (ASIC) of a dedicated multimedia production system, or as any practical combination of these forms.

[0081] As described above, a timeline-based multimedia editing system or application provides a graphical user interface (also referred to in the art as a "GUI" or "UI") that allows a user to interact with the system or application in order to import, edit, arrange and combine individual media items or components into a multimedia project and multimedia production. The UI is displayed on a touchscreen display so that the user's input is provided by various 'gestures' in relation to whatever is displayed on the touchscreen display at that time. These gestures are generally defined by contacting the screen at one or more corresponding locations (also referred to herein as 'touch points'), in some cases also dynamically moving one or more touch points along paths over the touchscreen surface, and in some cases also the velocity of that movement.

[0082] Overview of the User Interface

[0083] An embodiment of the multimedia editing system will now be described in the context of example multimedia projects with, for the sake of simplicity, only a moderate number of individual items of video and audio content. However, in practice the number or items and the number of layers is primarily constrained by the hardware. As a guide, current models of iPad with Apple silicon (M2 or M4) chips are able to display up to 200 tracks of content.

[0084] Figure 1 is a representative screenshot showing the UI of the system during editing of an example animation project. As is typical of timeline-based multimedia editing applications, the UI provides two major content regions: a media display region 102 in the upper part of the display, and a timeline region (also referred to herein for brevity as simply "the timeline") 104 in a lower part of the display, separated by a toolbar 106. The timeline region 104 displays graphical representations (e.g., keyframes, thumbnail previews) of items of media content 108 that are to be combined in order to generate an output multimedia production such as a movie, film clip, or animation. The items of content include digital assets such as videos, drawings, animations, audio, images, or text.

[0085] The graphical representations of the items of multimedia content are arranged as a vertical stack of horizontal rows referred to herein as layers or "tracks" 110, where the horizontal position of each item of content corresponds to the temporal position of that item of content in the project / production, as indicated by a timeline ruler or axis 111 displayed across the top of the timeline region 104, with ticks and time value labels to indicate those temporal positions. A single track can contain multiple items of content, as shown.

[0086] The order of tracks in the vertical stack represents a hierarchical layering, such that tracks higher in the stack will be rendered over tracks that are lower in the stack when composited ( / .e., combined together). Accordingly, simply put, the final output production can be thought of as being generated by collapsing or flattening the layers of tracks ( / .e., the individual content item rows) of the project into a single track in which the horizontal dimension represents elapsed time from the beginning of the project / production.

[0087] To improve efficiency, the system allows a user to group together multiple tracks so that they can be processed collectively, allowing one or more actions (e.g., moving or applying an effect or keyframe, etc.) to be simultaneously performed on all of the tracks of the group. For example, Figure 1 shows two groups of content 112, 114. The group 112 to the left is shown in an expanded or open state to reveal the members of the group, as indicated by the downward orientation of the arrow icon to the right of the Group label, and the members of the group being displayed as a stack of layered tracks contained within a (light grey) bounding box with rounded corners.

[0088] To reduce visual clutter, a group can also be displayed in a minimised or closed state, as shown by the group 114 to the right in Figure 1. The minimised state of this group 114 is indicated by the rightward orientation of the arrow icon to the right of the Group label, and also by the absence of tracks displayed below the Group track within a bounding box. A minimised group is displayed as a single "Group" item containing representative preview bitmaps, thus reducing visual clutter.

[0089] The toolbar 106 between the media display region 102 and the timeline region 104 extends horizontally across the display. A set of icons displayed to the right-hand side of the toolbar 106 provides access to respective controls for selecting respective operating modes of the system, including : a playback mode, a perform mode, an edit mode, and a paint mode. A plus sign ("+") icon at the extreme right of the toolbar 106 can be selected to add content to the project. The project title ("Untitled Movie" in this example) is displayed to the left of the toolbar 106.

[0090] The media display region 102 in the upper part of the display is for displaying selected items of media content or tracks during editing, and previewing output. As shown in Figure 2, the media display region 102 has two distinct parts, respectively referred to as "the Stage" 202 and "the Backstage" 204. Selected content can be displayed on either part, but content that will be exported ( / .e., will be included in the output production) is only displayed on the Stage 202. Content displayed on the Backstage 204 will not be exported. The Stage 202 can be resized and moved by the user within the bounds of the media display region 102, and when the Stage 202 is not maximised, the portion or portions of the media display region 102 not occupied by the Stage 202 reveal the Backstage 204. Figure 1 shows the Stage 202 maximised to occupy the entirety of the media display region 102, and the Backstage 204 is not visible, whereas Figure 2 shows the Stage 202 in a reduced size to occupy only a portion of the media display region 102, the Backstage 204 being the other (empty in Figure 2) region of the media display region 102 outside (or 'behind') the Stage 202.

[0091] Returning to Figure 1, the highlighted border 116 around the minimised group 114 to the right indicates that the group 114 is currently selected, and an animation frame from the selected group (not shown) is displayed on the Stage 202. Selected or active content and the current position in time within the selected or active content is indicated by the location of a visual feature referred to herein as 'the playhead' 118, which in this mode and in the described embodiment is displayed as an icon 120 in the form of a (red coloured) lozenge-shape containing a traditional film production clapper board (facilitating visual focus for the user) immediately below the content, and an accompanying vertical white line 122 overlapping the content (to accurately indicate the current playback position therein). The current position in time of the playhead (in the editing context of Figure 1, relative to the selected item of content) is shown as text 124 in the lower left-hand corner of the media display region 102, formatted as "hh:mm:ss.ff", where "ff" represents frame number. Thus in Figure 1, the time position text "00:00:07.08" indicates that the Stage 202 is displaying the eighth frame of the seventh second in the project.

[0092] Timeline Navigation

[0093] Except for the simplest projects generated from only a very small number of individual items or rows, most projects have many rows / layers (e.g., at least ten and often hundreds) that cannot be displayed, or at least not clearly displayed, in a single view, on any touchscreen, and particularly not on the small screen of a portable computing device such as an Apple iPad. Consequently, the user will typically only be able to view and interact with a small subset of the media items and tracks of a project at any given time. Similarly, except for projects of very short temporal duration, the user can only clearly view a short temporal segment of the entire project at any given time. In other words, the user will nearly always view and interact with only a small portion or window of the complete timeline and tracks of a project at any given time. Thus the timeline region constitutes a viewport for displaying (typically) only a selected portion of the entire timeline, or set of contents, of a project at any time. Consequently, the ability to navigate the timeline to locate and interact with individual items of content of interest is extremely important for useability and efficiency.

[0094] As described above, prior art video editing applications require multiple zoom and horizontal and vertical scrolling or pan actions to access a specific item of content and at a specific point on the project timeline. For a complex project, this can require dozens of discrete zoom actions and pan actions, which is obviously inefficient and inconvenient, and renders such applications barely usable for large projects.

[0095] To address the limitations of prior art timeline-based multimedia editing systems, embodiments of the present invention provide a touchscreen-based user interface with a timeline region that displays a view of the timeline, and typically only a portion of the timeline. The displayed view can be dynamically, continuously, and fully zoomed and panned in both spatial (corresponding to time and layer) dimensions independently, concurrently, and in real-time by simple touch gestures. That is, the user can dynamically and continuously (and thus smoothly) navigate by zooming and / or panning to view any desired portion of the project timeline in the horizontal time dimension and the vertical track dimension and at any practical zoom level, in a manner broadly similar to zooming and panning a digital image in an image viewer application. However, with at least the zoom touch gestures of the multimedia editing system described herein, and unlike the viewing of a static image, the system responds by continually and dynamically re-generating the view, including its layout, and causing the regenerated view to be rendered to the touchscreen, all in real-time and in accordance with the dynamically changing zoom level as the gesture is performed, effectively generating in real-time a live animation of the changing view, providing instant feedback to the user, and further facilitating locating and accessing a particular item or portion of the timeline region of interest to the user.

[0096] The view is dynamically regenerated in this manner because, unlike the zooming of a static image, a change of zoom level of the timeline region changes not only the zoom level / magnification, but also the actual layout of the view, and in some cases the objects that are displayed. By way of example, some visual stylings do not scale with zoom level (e.g., line dash intervals), whereas others do (e.g., line widths, text size etc), and the scaling of some stylings is subject to corresponding maximum and minimum thresholds. Once text becomes too small with decreasing zoom levels, it is drawn as single lines. A single axis zoom does not change the aspect ratios of text and other objects visible in the displayed view, and some objects might not be displayed at all at some zoom levels. For example, decreasing the horizontal (time axis) zoom level (whether or not in combination with a concurrent change of vertical zoom level) results in views wherein the thumbnail images of content items and keyframes are correspondingly brought closer together along the temporal (horizontal) dimension of the timeline. However, as the horizontal zoom level is decreased, some thumbnail images may be omitted from the layout to reduce visual clutter, while maintaining the remaining thumbnail images at approximately the same (and visually pleasing) size. Similarly, although keyframe tracks will always be displayed in some form, the keyframe icons in a keyframe track will not be displayed if the zoom level falls below a threshold. Similarly, each audio track is represented as a bar chart of 'samples'. As the horizontal zoom level is reduced, the number of displayed samples is corresponding reduced so as to maintain a consistent sample bar width (in screen space). These and other dynamic changes of layout require the view to be dynamically re-generated and displayed in real-time as the user performs at least the zoom touch gestures on the timeline region.

[0097] These abilities constitute a radical departure from prior art timeline-based multimedia editing systems and the conventional navigation paradigm that constrains navigation of the timeline and user interaction with multimedia projects. As described below, the dynamic capabilities of the multimedia editing system presented numerous technical challenges that had to be solved before the system could be practically implemented. For example, Figure 2 is a screenshot of the system touchscreen, showing the timeline region 104 in a fully zoomed out state so that the user can view all of the content items of an animation project. However, finer details such as individual keyframes and thumbnail previews of the project can be difficult to differentiate or interact with in this global view. To facilitate the latter, the UI allows the user to zoom-in the timeline region 104 to focus on a particular item of content 206 by performing a simple pinch-out gesture familiar to users of touchscreen devices such as smart phones when viewing static images.

[0098] In the Figures, solid circles are used to represent 'touch point' locations (e.g., the locations of the user's fingers or stylus) on the touchscreen, and arrows extending from these solid circles represent movement of the touch point locations as the user performs a 'touch gesture'. Thus in Figure 2, an example of the pinch-out zoom gesture described above is represented by opposing and outwardly directed arrows 208 pointing generally south-west and north-east, indicating that the touch points (typically, but not necessarily, corresponding to the user's finger and thumb contacting the touchscreen), initially at the locations represented as solid circles, are moved away from each other along those directions to zoom in on a desired portion of the timeline region bounded by those two locations in a manner familiar to users of touchscreen computing devices. For convenience of description, a reference in this specification to a user's finger or thumb is to be understood broadly as encompassing a stylus or any part of the user's body, typically a user's finger or thumb, unless context indicates otherwise.

[0099] This form of zooming, where both the horizontal and vertical directions are concurrently and dynamically zoomed, is referred to herein as 'uniform zooming' because the horizontal and vertical directions are zoomed together by the same amount. However, in some embodiments the system also supports non-uniform zooming, where the horizontal and vertical dimensions are concurrently zoomed by different amounts.

[0100] As a result of the pinch-out zoom gesture 208 described above, the corresponding portion of the timeline region is displayed at a higher magnification or zoom level, as shown in Figure 3, so that only a corresponding portion of the entire timeline region is displayed. The displayed timeline portion is anchored to a location from which the user initiated the pinch-out zoom gesture 208; specifically, the mid-point between the touch locations. This is important because it allows the user to retrieve more details from a specific desired location quickly and in a single gesture. In prior art applications, this change of viewpoint would take multiple gestures to achieve. Moreover, in a large project of long duration with many tracks and items of content, it can take dozens of individual zoom actions and vertical and horizontal pan actions to arrive at a similar view, noting that prior art video editing applications perform zooms from a fixed central location of the display, whereas the multimedia editing system described herein performs zooms from the gesture location, which is simple and intuitive.

[0101] After performing the pinch-out zoom gesture 208, the user can continue touching the screen and use a two-finger dragging gesture 302, as shown in Figure 3, to dynamically pan in real-time the displayed view to a different part of the timeline region at the same magnification / zoom level. Thus without removing their fingers from the screen, the user can adjust the zoom level and the position of the displayed portion in one smooth interaction or composite gesture.

[0102] Figure 4 shows the result of performing the composite gesture described above, combining the pinch-out zoom gesture 208 and the pan gesture 302. The resulting high magnification or zoom level makes it easier to interact with individual keyframes and thumbnail previews, in the corresponding track 402. At this zoom level, individual frames of content are not directly accessible and are collectively represented by previews, wherein a single preview represents multiple individual frames. In order to interact with individual frames, a higher magnification view is required.

[0103] Accordingly, another pinch-out gesture 404, as shown by the touch points and arrows in Figure 4, can be performed to zoom in even further to enable viewing and interaction with individual frames of the desired item of content 402, as shown in Figure 5. At this very high magnification or zoom level, the user can add and / or interact with individual frames 502 of a frame-by-frame animation. For example, selection of an individual frame 504 by a single-finger tap gesture results in the selected frame being highlighted by a border, as shown in Figure 5, with the playhead 118 displayed under the selected frame to indicate the current playback position. (In the context of individual frames, the vertical white line 122 of the playhead 118 (indicating playback position within an item of content) is not displayed because it has no role.) A circular button containing a plus sign ("+") 506 is displayed over the upper right corner of the frame border, and can be selected to insert a new frame at the position of the selected frame 504. The user can also perform pinch gestures in the reverse direction, pinching in instead of pinching out, to perform a zoom-out pinch gesture 508, as shown in Figure 5. As with the pinch-in gestures, a pinch-out gesture 508 is also anchored to a location from which the user initiated the gesture. However, if the pinch-out gesture 508 is performed quickly, the magnification or zoom-level is immediately reset to its lowest level to display the entire project, in this example returning to the view shown in Figure 2.

[0104] Timeline Navigation: Horizontal (temporal) axis zooming

[0105] Figure 6 shows the timeline region with a single content group 602. This view shows the timeline ruler 111 labelled in units of seconds, so that the user can see the overall temporal location and duration of the group 602 in one view. However, at this zoom level they are unable to see or edit finer details, such as individual keyframes. To facilitate these actions, the user can use a horizontal zoom gesture to expand or contract the timeline region view in the horizontal time axis only. In the described embodiment, horizontal zoom gestures are horizontal drag or swipe zoom gestures performed by simultaneously moving three touch points (typically, three fingers) to the right, although this need not be the case in other embodiments.

[0106] Thus in Figure 6, a horizontal three-finger zoom-in gesture 604 is represented by three touch locations with respective arrows directed to the right, representing three of the user's fingers contacting the touchscreen at the locations shown and then moving them horizontally to the right. This causes the system to dynamically zoom-in the displayed view of the timeline region with respect to the horizontal time scale dimension only, resulting in an expanded view such as the view shown in Figure 7, in which the scale of the timeline ruler 111 has been correspondingly updated to display units of frames rather than seconds. At this horizontal zoom level, the user can clearly view and interact with individual keyframes 606. Comparison of Figures 6 and 7 confirm that the vertical magnification or scale remains unaffected by the gesture. If the playhead 118 is displayed when the gesture 604 is initiated, the resulting displayed portion of the timeline is anchored to the location of the playhead 118, rather than the location of the gesture.

[0107] The user can also reverse the direction of the horizontal three-finger swipe zoom gesture, swiping left instead of swiping right, to perform a horizontal three-finger swipe zoom-out gesture 802, as shown by the three touch points and respective arrows in Figure 8. The timeline view and the timescale of the timeline ruler 111 dynamically adjust in real-time as the user changes the horizontal zoom level during the gesture. As shown in Figure 9, the resulting view allows more content to be shown along the horizontal axis, but without losing or reducing information along the vertical dimension.

[0108] If the playhead 118 is not visible in the current view, the horizontal zoom-in and zoom- out gestures are instead anchored to the location from which the user initiated the gesture. For example, performing the horizontal three-finger zoom-in gesture 902 from the view shown in Figure 9, in which the playhead 118 is not visible, results in the timeline view shown in Figure 10.

[0109] Timeline Navigation: Vertical axis zooming

[0110] Changing the magnification ( / .e., zooming in or out) of the vertical (content track or layer) dimension only can similarly be achieved by performing vertical zoom gestures. In the described embodiment, the vertical zoom gestures are performed by simultaneously moving three touch points (typically, three fingers) upwards or downwards along the touchscreen, although this need not be the case in other embodiments. For example, Figure 11 shows a view of the timeline region with a single content group 1102. This view shows the timeline ruler 111 labelled in units of seconds so that the user can see the overall duration of the group 1102 in one view. The user can use a vertical three-finger swipe zoom-in gesture 1104 to expand the vertical scale of the group without affecting the horizontal time scale.

[0111] In Figure 11, the vertical three-finger swipe zoom-in gesture 1104 described above is represented by three upwards-facing arrows, indicating that three touch points, typically three of the user's fingers contacting the touchscreen, at the locations represented as solid circles, are moved in the same generally vertical direction to expand the vertical scale of the timeline region, resulting in the vertically expanded view shown in Figure 12. The content group 1102 is expanded vertically without changing the horizontal (time) scale, allowing the visible keyframe thumbnails 1202 to be displayed at a correspondingly higher magnification, allowing them to be more clearly seen by the user. The aspect ratio of the keyframe thumbnails 1202 remains unchanged so that they are not distorted.

[0112] Conversely, by performing a vertical zoom gesture in the opposite direction, swiping down instead of swiping up, a vertical zoom-out gesture 1302, as shown in Figure 13. can be performed to reduce the vertical scale, resulting in the view shown in Figure 14. As with the other zoom gestures described herein, the UI responds to the gesture by updating the displayed view of the timeline region dynamically and in real-time during the gesture, allowing the user to dynamically tailor their gesture as it is being performed to quickly locate and focus on a desired item of content and at a desired point of time along the project timeline.

[0113] Timeline Navigation: Single finger navigation

[0114] When working on multimedia projects, users often need to simply scroll or move horizontally along the timeline or vertically to locate and view different portions of a project. To address this need, the multimedia editing system described herein provides a single-finger pan gesture to dynamically pan (i.e., spatially translate) the displayed view along any desired direction, dynamically adjusting the displayed view accordingly in real-time and without changing the zoom level or magnification. For example, Figure 15 shows an example single-finger pan gesture 1502 wherein the user's finger or stylus contacts the screen while moving in a generally south-west direction, resulting in the translated view shown in Figure 16. However, the system respects the acceleration of the pan gesture, panning the view rapidly or slowly in relation to the user's gesture speed. Unlike prior art multimedia editing systems, panning is not limited to either the horizontal or the vertical direction, but rather can be performed in any direction, allowing for seamless movement along the horizontal and vertical directions simultaneously.

[0115] Timeline Navigation: Double Tap to Zoom

[0116] Figure 17 shows the timeline region in a fully zoomed out state so that the user can view the entire timeline region and all the contents of the project. However, finer details such as individual keyframes and thumbnail previews are difficult to differentiate or interact with at this zoom level. To address this need, the user can perform a double tap gesture to zoom-in the timeline view and quickly focus on a particular item of content at the location of the double tap gesture.

[0117] For example, the concentric circles shown in Figure 17 represent a double tap gesture 1702 performed at a corresponding desired time point of a desired item of content 1704, causing the system to display the view shown in Figure 18. The corresponding portion of the timeline at the gesture touch point is displayed centred in the timeline and at a very high zoom level, enabling the user to view and interact with individual frames of, for example, a frame-by-frame animation. The user can subsequently perform another double tap gesture 1802, as shown in Figure 18, to zoom back out to the previous zoom level. However, this gesture is also anchored to the touch point location, so will zoom out with the touch point position centred in the displayed view of the timeline, as shown in Figure 19, which may be different to the original view (of Figure 17 in this example).

[0118] Timeline Navigation: Flick backwards to play

[0119] Figure 20 shows the Timeline region in a fully zoomed-in state, displaying only a portion of the project, with the playhead 118 positioned near the middle of the project. To quickly begin playback of the project from its beginning, the user can perform a rapid 'flick backwards' gesture, as represented schematically by the two overlapping hand positions, by touching the playhead 118 and quickly moving or 'flicking' it to the left (towards the start of the project).

[0120] Thus, in Figure 20, a flick backwards gesture 2002 is represented by the two overlapping hand outlines shown rotated relative to one another. The lower opacity hand icon with a finger pointing upwards represents the location where the user first contacts the screen, while the higher opacity hand icon represents the directional movement of the flick gesture. Thus the user touches and quickly 'flicks' the playhead to the left, towards the start of the project.

[0121] Responsive to the flick gesture 2002 described above, the multimedia editing system moves the playhead 118 to the very beginning of the project timeline, as shown in Figure 21, and automatically commences playback of the project from its beginning, respecting the user's playback settings. If the view was zoomed in at the time that the gesture 2002 was initiated, then the view also zooms out to display the entire project in the timeline region. Timeline selection

[0122] Selection

[0123] Figure 22 shows the timeline region displaying various items of content of a multimedia project. In some situations, a user may wish to perform one or more actions to multiple items of content simultaneously, for efficiency. To facilitate these actions, the user can invoke a multi-selection input mode of the system by selecting (touching) the edit mode icon 2202, and then use their stylus to perform a selection gesture to freely select desired items of content displayed in the timeline region. When the system is in this mode, a selection toolbar 2204 is displayed across the top of the touchscreen, with the mode name shown to the left, and interactive controls to the right, providing access to corresponding settings and options.

[0124] In the Figures, a pencil icon 2206 represents the location of the user's stylus contacting the device touchscreen, and a dashed line 2208 represents movement of the stylus touch point over the touchscreen as the user performs a stylus gesture.

[0125] Figure 22 shows a selection gesture 2210 as described above, represented by a dashed line crossing through three items of content 2212, 2214, 2216, indicating that those items 2212, 2214, 2216 are to be included in the user's new selection. In general, the user can select each of multiple items of content by dragging their stylus through the item or by encircling it (partially or fully).

[0126] As a result of the selection gesture 2210 described above, the three items of content 2212, 2214, 2216 are selected (as indicated by highlighting the border of each selected item), allowing them to have batch actions performed on them, such as the actions listed in the pop-up menu 2302 shown in Figure 23.

[0127] Selection gestures can also be performed on tracks and keyframes 2402, in addition to items of content, as shown in Figure 24. The user can determine the type of asset that is to be selected by a subsequent selection gesture via a selection type pop-up menu 2502, as shown in Figure 25. Deselection

[0128] Similarly, one or more items of content can be removed from an active selection by performing a deselection gesture to freely remove one or more items of content from the active selection. For example, Figure 26 shows the timeline region with two items of content 2602, 2604 included in an active selection. While in selection mode, the user can use their stylus to 'strike through' one of the selected items of content, as represented by a dotted line 2606 running through that item of content 2602, indicating that the item 2602 is to be removed from the active selection. Alternatively, the user can encircle (fully or partially) 2702 a selected item of content 2602 to deselect it, as shown in Figure 27. Regardless of which form of deselection gesture is performed, the result is that the corresponding item of content is removed from the active selection, as shown in Figure 28, and consequently will not be affected by any batch actions subsequently performed on the active selection.

[0129] Split

[0130] In some situations, a user may wish to split ( / .e., divide) multiple items of content in succession. To facilitate these actions, the user can enter the selection mode by touching the selection tool icon 2202 as described above, and then use their stylus to perform a split gesture to split some content. In the described embodiments, the split gesture is performed by moving the touch point along a vertical line through each item of content to be split, emulating the physical motion of a slice or cut.

[0131] Throughout this specification, unless context indicates otherwise, references to directions of movement such as vertical, horizontal, and the like should be understood as encompassing approximations to these directions, as can be expected from a user manually performing such movements on a touchscreen display. In many instances, the multimedia editing system uses context and shape detection to identify the intended gesture from approximations of the intended direction of movement.

[0132] For example, Figure 29 shows a view of the timeline region with various items of content, and a split gesture 2902 to simultaneously split two items of content 2904, 2906. In the Figure, the clock icon 2908 shown adjacent to the pencil icon indicates that the user is required to continue to hold the stylus down at this position of the gesture to invoke the next step of the interaction. In Figure 29, the split gesture 2902 is represented by a dotted line 2910 drawn approximately vertically through the items of content 2904, 2906, indicating the temporal position at which the items of content 2904, 2906 are to be split. After performing the split gesture 2902, the user can continue holding their stylus on the screen and add a dragging gesture 3002, as shown in Figure 30, to adjust the precise temporal location of the split. Thus without removing their stylus from the screen, the user can both initiate the split and fine tune its temporal position by moving their stylus vertically (2902) and then horizontally (3002) along the temporal (horizontal) direction in one smooth composite interaction.

[0133] As a result of the split gestures 2902 and 3002 described above, the items of content 2904, 2906 are each split or divided into two corresponding portions 3102, 3104, 3106, 3108, as shown in Figure 31. Multiple split gesture can be performed sequentially while in selection mode, allowing for quick editing.

[0134] Delete

[0135] In some situations, a user may wish to delete multiple items of content in succession. To facilitate multiple deletions, the user can enter selection mode and use their stylus to perform a delete gesture to freely delete multiple items of content. In the described embodiments, the delete gesture is in the form of a wavy or zigzag motion, emulating the traditional motion of 'scribbling out' or erasing on paper.

[0136] For example, Figure 32 shows a delete gesture 3202 being used to delete various items of content 3204, the delete gesture 3202 being represented by a dotted line 3206 drawn in a zig-zag pattern through the items of content that are to be deleted. As a result of the delete gesture 3202 described above, the deleted content items 3204 are no longer displayed, as shown in Figure 33. Multiple delete gestures can be performed in succession while selection mode is active, allowing for quick editing.

[0137] Timeline Grouping

[0138] As described above, in some situations, a user may wish to group multiple items of content so that they can be treated collectively, as though they were a single content item. To group items, the user can enter selection mode, use their stylus to freely select items of content, and then group the selected items via a popup context menu. For example, Figure 34 shows a timeline view with a group 3402 of tracks in its expanded or open state, showing the parent track 3404 providing the name of the group (in this example of an unnamed group, being simply the text "Group") 3406 and shaded borders, below which the group member (child) tracks 3408 are displayed in a vertical stack that is flush ( / .e., horizontally aligned) with the parent track 3404 of the group 3402, since the horizontal axis represents timing. Groups in the closed or minimised state are displayed with a full thumbnail previewing the contents of the group, while open groups show a smaller thumbnail preview, helping to identify immediately which groups content is nested into.

[0139] Advanced playhead

[0140] Figure 35 shows a view of the timeline region with various items of content and the playhead 118. The playhead 118 acts as a cursor, and its icon 120 is displayed just below the active item of content 3502, at the corresponding temporal position within that item 3502, as indicated more precisely by the horizontal alignment of the playhead's vertical white line 122 (displayed over the item) with the timeline ruler 111, and by highlighting the closest tick mark on the timeline ruler 111. The user can interact with the playhead 118 to perform actions to the corresponding item of content 3502 at the corresponding time position within it.

[0141] In the Figures, the outline of a hand 3504 represents a touch location of the user's stylus or finger. Tapping the playhead 118 (e.g., with a finger or stylus) causes the system to open a context menu 3602 on the timeline, as shown in figure 36, from which the user can instruct the system to perform actions pertaining to that type of content item, such as adding motion keyframes 3604 and filter keyframes 3606, and editing the content 3608. Adding keyframes via the playhead 118, for example by applying Gaussian blur 3702 as shown in Figure 37, also updates the timeline to display a corresponding keyframe 3704. This allows for quick and precise keyframing, animating, and editing. Loop to screen playback

[0142] In some situations, a user may wish to repeatedly view playback of a particular portion of the timeline. The system allows this to be done when the "loop to screen" playback setting is enabled in the app settings (which it is by default).

[0143] While this setting is active, and the system is in Playback mode, as shown in Figure 38, only the (temporal) portion of the project being displayed in the timeline region at that time is played on the Stage 202, and as a repeating loop. Thus the starting and ending time locations of the playback are those shown on the timeline at the left-hand 3802 and right-hand 3804 edges of the touchscreen. Moreover, the playback dynamically updates in real-time to reflect any changes in the timeline region. For example, if the user uses timeline navigation gestures as described above to display a different temporal portion of the project during looped playback, the playback automatically updates to play only the content displayed in that different temporal portion of the timeline region 104, without pausing playback. Playback continues to loop, or play on repeat, based on the boundaries of the project displayed in the timeline region until the user stops playback. This provides users with a quick and easy way to check the outcomes of their changes to a small or large temporal portion of their project, to change focus easily and without needing to reset their playback settings between changes.

[0144] Painting Timeline view

[0145] Figure 39 is a screenshot showing the standard default appearance of the GUI when editing a project. Because the user can both composite and edit animations in the timeline region as well as display individual frames on the Stage, there is often a need to provide more room for painting and drawing. In this case, the user can use a handle 3902 on the main toolbar 106 to enter a Painting Timeline view.

[0146] To use the handle 3902, the user performs a drag down gesture 3904 by touching the screen at the position of the handle 3902 and dragging the touch point downwards from that position. If a velocity threshold is reached during the gesture 3904, the timeline region snaps or transforms into a compact timeline window 4002, allowing the Stage 202 to expand to occupy nearly all of the touchscreen, with the compact timeline window 4002 floating above it, as shown in Figure 40. A Draw & Paint toolbar 4004 is displayed at the top of the screen, including tool controls at the right-hand end to invoke different drawing and painting tools, and controls at the left-hand end to invoke different drawing and painting modes. Further controls labelled "Cancel" and "Done" are provided to allow the user to cancel or complete an action, respectively.

[0147] The painting timeline view provides an analogue to traditional cell animation in which the user can flip quickly between drawings to see the transition between them. To perform this flip action, the user performs swiping gestures as shown in Figure 40, swiping left 4006 on the compact timeline window to advance forward in time, and swiping right 4008 on the compact timeline window to advance backwards in time.

[0148] In many cases, the user will have 'held frames' where a single frame of an animation is extended ('held') for longer than one frame. In these cases, the user can perform a double tap gesture 4010, 4012 on the compact timeline 4002 to advance to the next different frame ( / .e., the next frame in sequence that is different to the current frame). As shown in Figure 40, the user can perform this action on the left side 4010 of the compact timeline to quickly proceed to the previous different frame ( / .e., backwards in time), or on the right side 4012 of the compact timeline 4002 to proceed to the next different frame ( / .e., forwards in time).

[0149] This action can also be performed by rapidly performing a 'flick' swipe gesture 4014, 4015 in either direction, also shown in Figure 40, to proceed to the next different keyframe. These gestures are invoked if the touch point velocity as the user lifts away from the screen reaches or exceeds a threshold value. The user can perform a flick gesture, flicking right 4014 on the right side of the compact timeline 4002 to advance backwards in time to the next different keyframe, and flicking left 4012 on the left side of the compact timeline 4002 to advance forward in time to the next different frame.

[0150] Otherwise, the compact timeline window 4002 provides only a reduced set of functionalities for any track in the timeline to ensure that the size of the compact timeline window 4002 is as small as possible. These limited functionalities include inserting frames (via control 4016) and deleting frames via a frame context menu.

[0151] To return to the full-sized timeline region 104, the user performs the same drag-down gesture described earlier from the handle 4018 near the top of the compact timeline window 4002. Timeline colour tags

[0152] To facilitate the identification of different items of content on the timeline, the multimedia editing system provides the ability to use colour tags on any item of content so that it can be easily identified and visually differentiated from other items of content.

[0153] Figure 41 shows an example project with colour tags in the timeline region. At this zoom level, the name of each colour tagged item of content in the timeline region is displayed as text in a shape 4102 with a solid fill in the corresponding tag colour. The colour tagging thus highlights the name of the item of content, which visually differentiates items without adding visual clutter. When the user zooms out to a sufficiently low zoom level, to view their entire project for example, the names can no longer be clearly displayed, so colour tagged items of content are displayed as corresponding coloured horizontal lines 4202 representing their respective time locations and durations. Thus each of the color tags visually highlights the entire item of content on the timeline, as shown in Figure 42. Without colour tags, it can be difficult to identify different items of content at this zoom level of a complicated project, particularly when effects and transparency are utilized. With the addition of colour tags, the user can identify different items of content with ease, allowing them to choose their own organizational colour scheme.

[0154] Keyframe Tracks

[0155] Figure 43 shows the timeline region with an item of keyframed content 4302. The user can tell that this item of content 4302 has keyframes on it because they are represented by mutually spaced thumbnails 4304, 4306 displayed at respective different temporal positions along the horizontal timeline direction to identify this. In this view, the user can easily view and delete their keyframes, but they may also need to edit keyframes that they have previously created. To do this, the user simply taps on a corresponding keyframe; for example, the rightmost keyframe 4306.

[0156] When a keyframe is tapped, all relevant keyframe tracks expand out below the corresponding item of content, as shown in Figure 44. In the example shown in Figure 44, there is only a single keyframe track 4402, involving one or more changes in the set of parameters consisting of hue, saturation and brightness, collectively represented by the label "HSB". The starting and ending values of these parameters can be viewed and changed by selecting the keyframe icons 4404 displayed in the keyframe track at temporal positions aligned with those of the corresponding starting and ending keyframes 4304, 4306. Specifically, the user can adjust the keyframe parameter of the starting or ending keyframe by tapping on the corresponding one of the keyframe icons 4404, 4406.

[0157] If the user wants to view or access all of the available parameter values for that keyframe type, they simply tap the Keyframe Track name, in this example being the text "HSB" 4408. Once tapped, the Keyframe track is further expanded vertically to show every editable parameter value for this keyframe type, as shown in Figure 45. Thus in the example shown, the HSB keyframe track has the three parameters of hue, saturation, and brightness, but is currently configured to transition (in this example, by easing out) only the hue parameter values between the endpoint keyframes.

[0158] The expanded view of Figure 45 can be exited by tapping elsewhere in the timeline 4504, or by tapping 4506 the keyframe thumbnail 4304, 4306 on the corresponding content item 302, returning to the view of Figure 44.

[0159] The hierarchy of Keyframe Tracks allows the user to use different levels of expansion for different purposes. The first level (Figure 44) displays a summary of the keyframes on the user's track. As there is less on the screen at this level, it makes is easier to snap to or compare to other items of content. This makes it easy for users to find what they are looking for, and to perform basic actions such as retiming keyframes and editing single values. Once expanded to the second level of the hierarchy (Figure 45), the user can edit or adjust all individual keyframe values related to that keyframe type. This allows them to perform precise adjustments (for example, editing the translate-x value of a Move & Scale keyframe in isolation).

[0160] A user may also wish to adjust the easing of their keyframes 4304, 4306 so that the animation transitions differently between keyframes. To do this, they tap on the easing title 4502 between the two keyframes, as shown in Figure 45, causing the system to display an easing popover menu 4602, as shown in figure 46, enabling the user to adjust the Easing values. Technical Challenges

[0161] While the ability to dynamically and simultaneously zoom and update in real-time the displayed view of the interactive timeline region in both spatial dimensions and pan in any direction by (and during) touch gestures might appear conceptually straightforward in hindsight, its practical implementation required significant technical problems to be addressed.

[0162] In particular, the simultaneous requirement for high quality vector graphics that maintain fixed screen space scaling of line / stroke stylings during zooming precludes the use of image based caching to accelerate the rendering of the timeline (by either caching tiles or individual timeline elements). As a result, image caching is restricted to thumbnails that present preview images of content items in the timeline. Allowing vertical or dual axis zooming also requires that image thumbnails (presenting previews of content items) can be of varying sizes that change dynamically with zoom level, such that the thumbnail previews are of similar pixel density to the screen at all times.

[0163] Heavy use of caching as a performance strategy also does not work well in the worst case, when the cache is empty, usually on first load of the timeline or during major changes. Worst case performance is an important performance property to optimise for interactive systems. The system provides a high performance interactive timeline that only uses caching of project media data ( / .e., content items and tracks), not of the rendered timeline.

[0164] The described system provides a high performance pipeline that renders the timeline from a project's semantic definition (in contrast to derived layout / raster information) through to the screen within l / 120th of a second ( / .e., 8.3 ms) if supported by the hardware, or otherwise l / 120th of a second ( / .e., 16.6 ms). As described below, such high performance is enabled by memory efficient data formats, simple linear data structures, visibility culling of objects, efficient batch generation of render data (in layers), and generating the render data in data formats friendly to GPU rendering.

[0165] Figure 47 is a block diagram of the system of the described embodiment 4700, in which the multimedia editing processes described herein are implemented as programming instructions of a software application 4800 installed on an Apple iPad™ touchscreen computing device. Figure 48 is a schematic diagram showing the application architecture of the described embodiment. The User Interface 4802 represents what is displayed to the user and is interacted with directly, including the states of the interface elements, and receives inputs from a user of the system. A logic component 4804 processes user interactions that change the project. A Real-time Engine 4806 determines the state of the project and renders it for the user, within real-time requirements. An Animation Core 4808 processes, interprets and computes a multimedia production at any moment in time. A Storage component 4810 provides transactional disk-backed storage of each project as a file that can potentially include the entire history of the project, and all of its required data in a single file, via the host operating system's file system API 4812. A Resource Scheduler 4814 manages and distributes the limited memory resources to other components of the system. An audio engine 4816 decodes and processes audio for both real-time playback and exporting offline. A Video Engine 4818 decodes and processes video for input and output. A Rendering Engine 4820 provides image manipulation as sparsely stored image content and renders it via a Render Graph / Compositor component 4824. A Painting component 4822 renders drawn strokes in real-time from user inputs, using complex brush parameters. The Compositor / Render Graph component 4824 interprets a multimedia project constructed by the user, and effectively converts it to a format that is suitable for rendering purposes. A Metal API 4826 is the host system's low-level interface to the GPU used for GPU computation and displaying graphics to the screen. A Metal Abstraction Layer 4828 provides a low level wrapper over the Metal API 4826 that is more convenient than the latter, and is used by other components of the system. The Video Toolbox API 4830 is the host system's low-level interface for hardware acceleration for decompression and compression of video. The Audio Toolbox API is the host system's low-level interface to the Audio hardware.

[0166] With an Apple iPad as the hardware platform, the display refresh rate is 120 Hz, meaning that it needs to be re-rendered within 8.3 ms. Multimedia projects such as animations are typically large, and potentially include both huge resources and also large volumes of smaller data, keyframes, values and details of content to render and when they should be rendered. The inventors determined that rendering the timeline region as described herein requires real-time composition and rendering of both vector graphics (icons, outlines, gradients, etc.) and bitmap graphics (thumbnail imagery for items of content) on the GPU. Because the timeline region is also an interactive user interface, it needs to be rendered at interactive frame rates (ideally 120Hz on modern hardware) in order to provide a satisfactory user experience.

[0167] Moreover, the requirement for both single-axis (horizontal and vertical) dynamic zooming and uniform dual-axis dynamic zooming presents unique technical challenges and pressures on timeline rendering because it precludes the ability to use image caching of the composition during periods of interactivity as a method of achieving high performance.

[0168] The requirement for high quality vector graphics (lines, strokes etc.) also limits the ability to use composited image caching. The use of 1 pixel stroked shapes requires rerendering in real-time as the user zooms in or out on the timeline region.

[0169] In order to satisfy the requirements described above without being able to cache any final image of the timeline region, the timeline region needs to be generated dynamically each frame. The inventors determined that this places significant constraints on data formats, data transformations, and caching strategies (even though the final image output cannot be cached, caching remains a viable strategy for intermediary formats).

[0170] As shown in Figure 49, the data formats are used :

[0171] (i) Storage (e.g., Disk) Format 4902 (Cached);

[0172] (ii) Packed Format 4904 (Cached);

[0173] (iii) Timeline Object Format 4906; and

[0174] (iv) Timeline Render format 4908 (this is the final form of geometry for sending to the graphics driver).

[0175] The data sent to the rendering engine 4820 for rendering the timeline region includes property data pertaining to which entities are being displayed and at what temporal points in the project (e.g., exposure / timing data, entity 5 is present in an animation between frames 10 and 25), whether they have an image and / or sound, whether they are transformed (e.g., entity 5 includes an image and a translation), and keyframe data for their properties (e.g., there are keyframes at frame 1, 4, 5, for “translate x” for entity #5).

[0176] Storage Format 4902

[0177] Storing data on disk is not well suited for direct use in compositing either the multimedia production itself, or the timeline region. The system uses a storage format 4902 that prioritises both efficient disk usage (small file size) and rapid storage / retrieval. However, if data was stored in a form ideal for rendering (packed and dense), then any change to the data would cause heavily amplified changes to the disk, because the stored information would need to be repacked. For example, a trivial change (e.g., updating a single keyframe) would require repacking all of the keyframes for the changed entity, and potentially all keyframe data. To avoid this difficulty, the system uses a storage format that allows the user to make small changes to their project while requiring only small changes to the data stored on disk.

[0178] Specifically, the multimedia editing system stores data in a storage friendly form as B- trees in an ordered key value store. In the described embodiment, the data is stored in a standard key-value database, although in other embodiments the data could be stored in SQLite or any other appropriate in-memory database or order key value store such as RocksDB / LMDB, for example. Figure 49 is a high level flow diagram of the rendering process, illustrating the sequence of data formats from disk to the rendering engine 4820.

[0179] Figure 50 is a schematic diagram illustrating the different types of data object created by the multimedia editing system and their relationships. The data objects represent movie items 5002, content items 5004, media (image 5006, audio 5008, and video 5010) items, keyframe tracks 5012, and tracks 5016. As represented by the opposing arrows linking content items 5004 and tracks 5014, tracks 5014 and content items 5004 have a mutually recursive relationship, whereby tracks 5014 can have content 5004, and content 5004 can have tracks 5014. This allows an item of content 5004 to also perform a role as a grouping structure. A content item 5004 is bound / exposed in a multimedia project for a period of time, and can have animation property data, media data, and child entities 5004 that are bound. As this data is stored on disk, the host operating system (iOS in the described embodiment) will generally provide automatic caching of disk data.

[0180] Packed Format 4904 (cached)

[0181] The packed format 4904 is designed to be compact and as fast to read as possible. Within a project such as an animation project, typically an artist will only change small quantities of the data at a time. Consequently, caching of animation data is still a viable strategy.

[0182] The system stores packed forms on a per data object basis. The term "packed" means that, within the scope of a cached object, data is stored linearly / consecutively in memory, which is a necessary requirement to satisfy the performance goals of the realtime interactive timeline region.

[0183] The packed format 4904 is different to common storage friendly formats, and thus requires translation between formats 4902, 4904 unless the storage format 4902 is also packed. Packing the storage format, and thus removing the need for two formats, is a viable method if modifications to the storage format 4904 do not need to be fast or efficient.

[0184] The code below, in the open source Swift programming language (originally developed by Apple, Inc), illustrates the packed data structures used for track and content objects. struct Generational lndex : Hashable , Equatable { var id : UInt 64 var generation : UInt 64

[0185] } typealias Storagecontent Id = Generational lndex typealias StorageTrackld = Generational lndex typealias MediaRef erenceld = UInt 64 typealias MovieFrame = UInt 64 struct MovieFrameRange { var start : MovieFrame var end : MovieFrame } struct PackedStorageContent { var name : String var color : Color var mediaRef erenceld : MediaRef erenceld? var tracks : [ StorageTrackld] } struct PackedStorageTrack { var boundcontent : [ (MovieFrameRange , Storagecontent Id) ] }

[0186] The MovieFrameRange for each bound item of content represents the period of time in which a content item is exposed in the animation. This example uses absolute time, although time local to the parent content can also be used. Absolute time is simpler from an animation perspective, but using local time usually requires fewer changes for storage modifications.

[0187] The timeline information is stored in packed format 4904 in a cache, and is as fast as possible for retrieval and usage by the timeline layout process 4912.

[0188] The elements of the packed cache format are stored by identifier (e.g., a simple UInt64 per record) in a size bounded hash map based cache using a standard eviction method such as LRU to keep the cache size bounded. Any of a variety of methods known to those skilled in the art can be used to ensure that only the latest version of data is cached. For example, cache invalidation can be achieved by attaching the cache to the storage system (a push methodology then allows the storage layer to invalidate elements of the cache when the underlying storage version is changed), or alternatively the identifier can be supplemented by a version number (e.g. (id: Int, version: Int)) that is incremented by the storage system when it makes changes, so that cache misses occur naturally when data changes in the storage layer. For example, the StorageCache structure can be defined as follows: struct Generational lndex : Hashable , Equatable { var id : UInt 64 var generation : UInt 64

[0189] } typealias Storagecontent Id = Generational lndex typealias StorageTrackld = Generational lndex struct PackedStorageCache { var rootContent Id : Storagecontent Id var content : [ Storagecontent Id : PackedS tor ageContent ] var tracks : [ StorageTrackld : PackedStorageTrack] } where "RootContentld" represents the animation itself, which is represented by an instance of the PackedStorageContent structure defined above, "mediaReferenceld" is a notional reference representing that timeline content will have underlying media that is displayed in the output animation. This could include, but is not limited to, bitmap images, vector graphics, video or audio.

[0190] The cache is built and keyed on the identity of data objects. In the described embodiment, the keys are a combination of two unsigned 64 bit integers: one for identity, and another for the “generation” ( / .e., version) of the object. In response to modification by the user, the generation of the corresponding data object(s) is incremented. This causes cache invalidation to occur on read, because the cache cannot supply the latest generation of the object. The system thus reads the object from storage, and updates the cache accordingly (a pull based approach).

[0191] As mentioned above, generational indices is only one suitable cache invalidation strategy; other embodiments could use other pull based approaches (e.g., content addressing), or push based methods (only using an identifier), or invalidation of all caches as modifications are changed. All such methods involve tradeoffs, but the inventors chose a pull based method because they are typically easier to execute with correctness. Keyframe Track objects 5012 store individual keyframes (values) and associated easing data (e.g., Linear or Cubic In Out), stored in time order. Media objects 5006, 5008, 5010 store image data 5006, video data 5010, and audio blob data 5008.

[0192] The storage format of these objects 5002 to 5012 is generic, and supports both generating the rendered timeline, and the composition of the animation itself. The objects 5002 to 5012 are stored in a hash map keyed on their object identity, with each object’s information (value) being stored linearly.

[0193] Timeline Object Format 4906

[0194] Timeline objects are stored in a timeline object format 4906 that covers Ul / visual / layout information for data objects so that the timeline can be both rendered (requiring all dynamic information necessary to draw a visual depiction of the timeline) and interactive (requiring information that lets users interact with the timeline, largely layout, identity and metadata information).

[0195] Another requirement of the timeline region is animation of the timeline region itself in response to user actions (e.g., adding a keyframe, adding or deleting an entity). The data representation 4906 of the timeline object is the format that visual animations are performed upon.

[0196] The properties of a content object 5004 include:

[0197] (i) Identity (id and generation, matching the storage cache format);

[0198] (ii) Bound / exposed range (e.g., from frame 5 to frame 20);

[0199] (iii) Layout information: Relative / absolute bounding rectangle; and

[0200] (iv) Name (user specified).

[0201] All timeline object data is stored by object type, where each type is stored linearly, as follows:

[0202] (i) all content items 5004 are stored in a single array and are referenced by index;

[0203] (ii) all tracks 5012 are stored in a single array and referenced by index; and

[0204] (iii) all keyframes are stored in a single array and are referenced by index. For example, in the described embodiment, timeline objects are stored as content objects and track objects of an Object class, as follows: typealias Objectld = Int struct Contentobject { var storageld: Generationallndex var name: String var frame: Rect var color: Color var timeRange: MovieFrameRange var parentTrack: Objectld var visible: Bool var mediaRef erenceld : MediaRef erenceld? } struct Trackobject { var storageld: Generationallndex var frame: Rect var visible: Bool var parentEntity : Objectld

[0205] } class Objects { var content: [Contentobject] var tracks: [Trackobject] internal init() { self. content = [] self. tracks = [] } func add (content c: Contentobject) -> Objectld { let id = content . count content . append ( c) return id

[0206] } func add(track t: Trackobject) -> Objectld { let id = tracks. count tracks . append ( t ) return id

[0207] } func track (_ objectld: Objectld) -> Trackobject { tracks [objectld] } func content (_ obj ectld : Obj ectld) -> Contentobj ect { content [ obj ectld] } }

[0208] Timeline Render Format 4908

[0209] Modern GPUs, including those used in Apple iPads, perform rendering to the hardware frame buffers via programmable shaders. Consequently, the rendering is achieved by generating shader commands or instructions that specify which GPU shader to use and its configuration, data bindings for each shader invocation, and the raw data to use (the majority of which is geometry data). For vector graphics that are not GPU friendly, the system offloads the preparation of vector graphics artefacts to the CPU. These are sliced up and cached as square textures of pre-rendered vector graphics that have distinct gradients / strokes / icons. These are then used as texture-based inputs to traditional texture mapped GPU rasterization. Vector rendering is performed at a variety of scales, which are accessed through mipmaps in order to render accurate vector graphics regardless of scale. The render data is stored per vertex.

[0210] The timeline render format 4908 needs to include vector graphic primitives, including text and bitmap images. The format 4908 is processed by the GPU's vertex shaders, and requires vertex data in addition to descriptors that allow the vertex / fragment shader to interpret the geometry to achieve the desired effect, such as applying appropriate colours, and the like.

[0211] For example, the timeline render format 4908 can be defined as follows: struct TimelineRender { var layers : [ TimelineRenderLayer ] = [ ]

[0212] } struct TimelineRenderLayer { var text : [ Text ] var image : [ Image ] var roundedRectangles : [RoundedRectangle ] } struct RoundedRectangle { var area : Rect var cornerRadius : CGFloat var stroke : (width : Double , color : Color ) ? var fill : Color?

[0213] } struct Image { var area : Rect var uv : Rect var media : MediaRef erenceld

[0214] } struct Text { var point : Point var fontsi ze : CGFloat var color : Color var content : String

[0215] }

[0216] The above render format is indicative, since the actual render format is specific to the combination of hardware platform, programming language, and graphics library. For the Metal API 4826, it includes separate buffers of per vertex data in addition to separate buffers of "per object" descriptors.

[0217] Building Steps

[0218] The rendering of the timeline region involves a sequence of transformations between the data formats 4902 to 4908 described above, as shown in the flow diagram of Figure 49, and described below.

[0219] Disk Format 4902 -> Packed Storage Format 4904

[0220] In order to display the timeline region, the system needs to generate a layout for the current viewport ( / .e., the visible rectangular portion of the complete timeline) at step 4912. If the data object for an object that is to be displayed is not already in the packed format cache ( / .e., packed cache miss), at step 4910 the relevant data for that object is retrieved from storage (e.g., a key value store or database), converted from the storage format 4902 to the packed format 4904, and stored in the packed format cache. In order to cache with bounded memory, an eviction strategy is necessary, and an LRU (least recently used) strategy is used in the described embodiment, on a pull per object basis.

[0221] Generating the layout at step 4912 involves generating a timeline representation of each object in the current viewport (in timeline object format 4906) from its packed data representation ( / .e., in packed format 4904). In addition to being an intermediate format for rendering, objects in timeline object format 4906 are also used for user interaction at step 4918.

[0222] Packed Storage Format 4904 -> Timeline Object Format 4906

[0223] The timeline region is rendered using the following information, starting with the root entity 5002 that represents the entire multimedia project:

[0224] (i) project (e.g., movie / animation) 5002; this determines the data in the Packed

[0225] Storage Format 4904; and

[0226] (ii) for a given viewport ( / .e., the viewed portion of the timeline, as determined by the user's pan and zoom interactions).

[0227] In some embodiments, the rendering can also use the following additional information:

[0228] (iii) in the context of the previously rendered collection of timeline objects (the state of the previous rendering is used as a source to generate visual animations of timeline objects);

[0229] (iv) with UI state; and

[0230] (v) which entity the user is looking at: a. the location of the playhead within the timeline region; and b. which entities 5004 are expanded (entities also serve as groups).

[0231] The layout process uses a "Transform" data structure that contains a 2D translation and a 2D scale that together define an affine transform that is used to view at least a portion of the timeline in the current viewport, the latter being a rectangle that represents the visible portion of the timeline that is being rendered to the touchscreen. In the described embodiment, the Transform data structure is defined as follows: struct Transform { var offset: Point var horizontalScalePointsPerFrame: Double var verticalscale: Double func map (point: Point) -> Point {

[0232] (point * Point (x: horizontalScalePointsPerFrame, y: verticalscale) ) + offset

[0233] } func map (range: MovieFrameRange, yStart: Double, yEnd: Double) -> Rect { let startX = Double ( range . start ) * horizontalScalePointsPerFrame let endX = Double ( range . end) * horizontalScalePointsPerFrame let startY = yStart * verticalscale let endY = yEnd * verticalscale let rect = Rect (start: Point (x: startX, y: startY) , end: Point (x: endX, y: endY) ) return rect . of f set (by : offset)

[0234] } func unmap (point: Point) -> Point {

[0235] (point - offset) / Point (x: horizontalScalePointsPerFrame, y: verticalscale) }

[0236] }

[0237] With this Transform data structure, the layout process is defined as follows: func layout ( storage : PackedStorageCache, transform: Transform, viewport: Rect) -> Objects { let objects = Objects () let root = storage . contentFor ( id : storage . rootcontent Id) let rootBinding = MovieFrameRange ( start : 0, end: 1000)

[0238] > = layout ( storage : storage, binding: rootBinding, content: root, transform: transform, objects: objects, yOffset: 0, viewport: viewport, drawContent: false, parentTrack: Obj ect Id . null ) return objects

[0239] } func layout ( storage : PackedStorageCache, binding: MovieFrameRange, content: PackedStorageContent , transform: Transform, objects: Objects, yOffset: Double, viewport: Rect, drawContent: Bool, parentTrack: Objectld) -> Double { var y = yOffset let paddingBetweenTracks = 0.3 let contentArea = transform. map (range : binding, yStart: y, yEnd: y + 1.0) let contentld = ob j ects . add ( content :

[0240] ContentOb j ect (name : content . name, frame: contentArea, color: content . color , timeRange: binding, parentTrack: parentTrack, visible: drawContent && contentArea . intersects (other: viewport) ) ) y += 1.0 + paddingBetweenTracks for trackld in content . tracks { let track = storage . trackFor ( id : trackld) let area = transform. map (range : binding, yStart: y, yEnd: y + 1.0) if area . isBelow ( rect : viewport) { return y } let trackld = obj ects . add ( track :

[0241] TrackObj ect ( frame : area, visible: area . intersects (other : viewport) , parentEntity : contentld) ) let trackStartY = y y += 1.0 + paddingBetweenTracks for ( childBinding, childContent Id) in track . boundcontent { let childContent = storage . cont ent For ( id : childContent Id) let contentMaxY = layout ( storage : storage, binding: childBinding, content: childContent, transform: transform, objects: objects, yOffset: trackStartY, viewport: viewport, drawContent: true, parentTrack: trackld) y = max ( contentMaxY, y)

[0242] }

[0243] } return y }

[0244] The layout process 4912 generates object data in timeline object format 4906 from the corresponding packed format data in the packed cache. The layout process 4912 executes recursively from the root content item, and is performed from top to bottom. All layout decisions are made during this time, and culling is used to reduce the quantity of work to be done. Timeline objects are added to linear arrays as a simple approach to provide high performance. Culling is used to track which objects need to be drawn, and allows early termination and to avoid rendering any tracks (and their content) that are not needed for layout or display. However, due to the hierarchical layout approach and the fact that laying out tracks requires working out the maximum height of their descendent content, it is often necessary to calculate layout for tracks and content that are not visible on the screen.

[0245] Strings (such as content names) are efficiently stored in one additional append only array. Performance can be improved by allocating these linear arrays with the same size they were in the previous frame, relying on the temporal similarity of timelines rendered in a real-time application.

[0246] The Transform object stores the translation and scale of the interface, and is used to position all content and tracks at absolute screen space coordinates. Other coordinate systems can be used, but working in screen space is necessary when stylings also have their dimensions in screen space.

[0247] Animation is achieved by retaining the previous rendered timeline, comparing contents of the current and previous rendered timelines on a per object, per property level, and generating animations when their contents differ. If animation of positional elements is desired, layout should first be performed in the space local to their parent content, and animations executed in local space before generating the absolute layout (animations performed in local space are generally preferred for aesthetic reasons).

[0248] Any visual stylings that can be derived from the Timeline Object Format 4906 are not calculated at this point, but rather during the render geometry generation process 4914 described below. These visual stylings include aesthetic stylings such as dashed line spacings, the corner rounding of rectangles, colours and the like. Since these are either constant or can be derived from properties, they do not need to be stored separately, thereby reducing the storage size of timeline objects.

[0249] The layout process 4912 and subsequent rendering steps 4914, 4916 are executed in response to a user touching the touchscreen to interact with the system, and in response to changes to the project data that could cause the visible portion of the timeline to change (including animations and storage changes).

[0250] Render Geometry Generation 4914

[0251] Once the layout has been updated at step 4912, corresponding render data for the GPU is generated at step 4914, effectively involving the conversion of object data in timeline object format 4906 to timeline render format 4908.

[0252] As described above, the layout process 4912 generates linear timeline object arrays. The render process 4914 involves converting the macro layout of these timeline objects to geometry data that can be rendered by the GPU shaders. Additionally, the render process 4914 also involves generating a layout of any visual elements that do not have interactive behaviour (anything that requires interaction being in Timeline Object Format 4906).

[0253] A simple layering strategy is used to determine the rendering order of elements. The described embodiment uses the following three layers:

[0254] (i) a tracklayer which displays track backgrounds;

[0255] (ii) a contentLayer which displays rectangles displaying content; and

[0256] (iii) a contentAdornmentsLayer which displays content colour tags on top.

[0257] This layering strategy allows separation of rendering order from iteration order, if needed. This is important because, in order to render efficiently, the storage centric data is processed iteratively but only once, in the order of the natural hierarchy of the animation. When rendering the timeline, however, efficient implementation of features such as Drag and Drop (rendering a dragged item of content on top of all others), requires a rendering order (determining which items of content are rendered on top of others) that is different from the natural order of the animation.

[0258] The render generation process 4914 is implemented as a prepare function, as follows: func prepare ( obj ects : Objects, viewport: Rect, transform: Transform) -> TimelineRender { let trackFill = color (white: 0.2) let contentFill = color (white: 0.5) let contentstroke = color (white: 0.8) let textColor = color (white: 1.0) let fontsize = 5.0 let trackLayer = TimelineRenderLayer ( ) let contentLayer = TimelineRenderLayer ( ) let contentAdornmentsLayer =

[0259] TimelineRenderLayer ( ) for track in ob ects . tracks where track . visible { trackLayer . roundedRectangles . append (RoundedRect anglefarea: track. frame, cornerRadius : 0.0, fill: trackFill) )

[0260] } let colorTagSize = Point(x: 10.0, y: 10.0) * transform. verticalscale * 0.025 let colorTagCornerRadius = colorTagSize . y * 0.5 for content in obj ects . content where content . visible { contentLayer . roundedRectangles . append (RoundedRe ctangle ( area : content . frame, cornerRadius: 4.0, stroke: (1.0, contentstroke) , fill: contentFill) ) let colorTag = Rect (centre: content . frame . topLeft , size: colorTagSize) contentAdornmentsLayer . roundedRectangles . append (RoundedRectangle (area : colorTag, cornerRadius: colorTagCornerRadius, fill: content . color ) ) contentLayer . text . append ( Text (point : content . frame . start , fontsize: fontsize, color: textColor, content: content . name) )

[0261] } return TimelineRender ( layers : [ trackLayer, contentLayer , contentAdornmentsLayer ] ) }

[0262] As described above, modern GPU rendering uses programmable shaders, so the rendering data includes GPU instructions indicating which shader to use and its configuration, data bindings for a shader invocation, and the raw data to use (the majority of which is geometry data).

[0263] The GPU instructions are generated as described above for each layer and for each form of geometry (rounded rectangles, text, image). However, other embodiments could use essentially any modern high performance method of rendering vector graphics. This could involve purely GPU preparation of graphics, or a combined CPU / GPU approach involving pre-prepa ration rendering of vector graphics artefacts on the CPU, stored in textures that are then composited together on the GPU.

[0264] User Interaction 4918

[0265] To allow interactive navigation of the timeline using a touch screen, the system supports touch based gestures to effect translation ( / .e., panning), single axis scaling and dual axis scaling of the timeline, as follows: func updateGestures ( touches : [ Point ] , trans form : inout Transform) { if touches . count >= 1 { var gesture = self . gesture ? ? Gesture (maxTouches : touches . count , originalTransf orm : transform, oneFingerTouchS tart Cent reScreen Space : touches . centre ) gesture . maxTouches = max ( gesture . maxTouches , touches . count ) if gesture . maxTouches == touches . count { if touches . count >= 3 { singleAxis ZoomGesture (touches : touches , gesture : &gesture , trans form : ^transform, bounds : bounds )

[0266] } else if touches . count >= 2 { dualAxis ZoomGesture (touches : touches , gesture : &gesture , trans form : ^transform)

[0267] } else { translateGesture ( touches : touches , gesture : gesture , transform : ^trans form)

[0268] }

[0269] } self . gesture = gesture

[0270] } else { self . gesture = nil

[0271] }

[0272] }

[0273] The mutually exclusive gesture selection based on gesture. maxTouches ensures that only one specific gesture is actioned at a time, and to prevent switching back to a different gesture if, for example, the user lifts a finger mid-gesture.

[0274] As a touch begins, moves, ends, or is cancelled, the touch locations are stored in the touches array and the gestures are updated. After updating a gesture, the timeline again is rendered again using the updated Transform structure.

[0275] Simple translation of the timeline can be effected by a translate gesture. In the described embodiment, the translate gesture is a gesture wherein a single touch location (e.g., by the user touching the screen with only one finger) that is translated across the screen, as follows: func trans lateGesture (touches : [ Point ] , gesture :

[0276] Gesture , transform : inout Trans form) { transform . of f set = gesture . originalTrans form . of f set + ( touches . centre

[0277] - gesture . one Finger Touchstart Cent reScreenSpa ce ) }

[0278] Simultaneous translation and dual axis zoom of the timeline can be effected by a dualAxis ZoomGesture, invoked by a user touching the screen at two locations, and moving those touch locations as described above: func dualAxis ZoomGesture ( touches : [ Point ] , gesture : inout Gesture , transform : inout Trans form) { gesture . twoFingerTouchStartAverageDistanceToCen tre = gesture . twoFingerTouchStartAverageDistanceToCentre

[0279] ? ? touches . averageDistanceToCenter let anchor = gesture . twoFinger Anchor Time line Space ? ? transform . unmap (point : touches . centre ) gesture . twoFingerAnchorTimelineSpace = anchor let relativescale = touches . averageDistanceToCenter / gesture . twoFingerTouchStartAverageDistanceToCentre ! transform . horizontalScalePoints Per Frame = gesture . or iginalTrans form . hor i zont al Seal ePoint s Per F rame * relativescale transform . verticalscale = gesture . or iginalTrans form . vert i cal Scale * relativescale let untranslatedPoint = trans form . map (point : anchor ) transform . of f set += touches . centre - untranslatedPoint }

[0280] The average location of the touches when the gesture is started is converted to "timeline space", defining an anchor point on the timeline that is maintained at the average location of the touches during the gesture. Zoom is applied uniformly, in proportion to the average distance between the touches. However, it will be apparent that in some embodiments a dual axis zoom can also be applied non-uniformly to zoom horizontally and vertically by different amounts.

[0281] A single-axis zoom gesture is initiated when three touch locations have moved a minimum length of 30 typography points, and a single axis scale direction is determined from that movement. Then a relative scaling in the determined dimension is performed relative to an anchor point at the bottom left of the timeline. func singleAxisZoomGesture (touches : [Point] , gesture: inout Gesture, transform: inout Transform, bounds: Rect) { let scaleAnchor = gesture . threeFingerScaleAnchor ?? touches . centre let displacement = touches . centre - scaleAnchor if displacement . length > 30 && gesture . threeFingerScaleDirection == nil { gesture . three Finger Seal eDirecti on = abs (displacement . x) > abs (displacement . y) ? .horizontal : .vertical

[0282] } let reference = Point (x: 0.0, y: bounds . height ) let a = Point(x: 0.000001, y:

[0283] 0 .000001) .max ( (scaleAnchor - ref erence) . abs ) let b = Point(x: 0.000001, y:

[0284] 0 .000001 ) .max ( (touches . centre - ref erence ) . abs ) let scale = (b / a) .pow (Point (x : 2.0, y: 2.0) ) gesture . threeFingerScaleAnchor = scaleAnchor switch gesture . threeFingerScaleDirection { case .horizontal: transform. horizontalScalePoints Per Frame = gesture . or iginalTrans form . hor i zont al Seal ePoint s Per F rame * scale. x case .vertical: transform . verticalscale = gesture . originalTransform. verticalscale * scale. y case none:

[0285] }

[0286] }

[0287] As described herein, the multimedia editing system and process addresses the described limitations of the prior art and also many associated difficulties that result from the ability to zoom the timeline region UI as described. For example, in work leading up the invention, the inventors developed and tested several prototypes that adjusted relative heights or spacings on the display, which allowed for clearer visual cues of the project structure, but were found to cause motion sickness and loss of spatial context as content moved around. As described herein, the multimedia editing system and process provides many advantages, as follows.

[0288] The system is able to handle projects that are large in vertical and horizontal dimensions (e.g., many tracks, many content items, long duration), even on touchscreen devices with extremely limited dimensions such as a small iPad™ (including the iPad Mini™), or even a smartphone.

[0289] The system allows efficient navigation to any point in the timeline

[0290] Prior art editing applications generally require multiple (even dozens) of zoom gestures to access a specific item of content and at a specific point on the timeline. For a complex project, this can require dozens of discrete zooms. The combination of zooming horizontally and vertically while also panning the timeline region overcomes this technical limitation. Also, existing timelines anchor the zoom gesture around a static needle in the centre of the timeline. In contrast, the described system anchors zooms around an initial gesture touch point location, allowing users to target effectively using a single gesture.

[0291] Users feel in control of the navigation

[0292] Early prototypes allowed the user to navigate freely, allowing them to end up with no content in the timeline region, or to zoom in or out too far to be useful. A lot of experimentation was required to determine the best constraints on movement and zooming that allow the user to locate specific items content quickly anywhere in the timeline region, and without feeling lost or motion sick. Early prototypes of the system allowed non-uniform zooming, such as scaling on the Y axis and decimating the X axis. These felt unpredictable and difficult to control.

[0293] The timeline region is not visually overwhelming

[0294] Being able to see a large project all at once can introduce issues of legibility. In order for the user to find what they want, they need distinct but simplified visuals. This has proven an ongoing task of removal and reintroduction of information at different zoom levels, involving hiding elements associated with items of content, such as visibility checkboxes and expand buttons. Although text descriptions are useful for identifying keyframe tracks, to show them all at once is counterproductive. In order to not overwhelm the user with unnecessary information, the system limits the details shown prior to the user making a selection.

[0295] Content or information cannot be offset in horizontal space

[0296] A general constraint of the system is that, because the horizontal axis represents time, nothing can be displayed horizontally between items of content. The ruler represents time consistently for every track.

[0297] Vertical offsets are used sparingly

[0298] Similarly, when assembling a project, the context of other tracks is very important to create transitions or create actions based on the timing of music or sound effects. If information or padding is introduced at the top or bottom of tracks, it reduces the number of tracks that can be displayed on-screen at once and adversely affects the context.

[0299] Another consideration is not requiring each item of content to have its own track. Many existing multimedia editing applications take this approach to simplify and consolidate possible actions on each track, but this is undesirable on mobile touchscreen devices with a limited amount of vertical display space. Stacking content horizontally as much as possible, allows projects to be displayed in a more compact manner. Accordingly, the system allows audio, video, and groups with different hierarchy to all be placed on the same track.

[0300] Groups can have multiple levels of hierarchy

[0301] As described above, the system allows users to group multiple items of content. For example, consider an animation of a bee that has several frames to show its wings flapping, which can be grouped together so they can be moved together as one object. The bee can also be in another group of foreground objects that receive the same effects. Groups can be nested.

[0302] Due to the constraints around horizontal space, indenting cannot be used to represent hierarchy, and as discussed above, vertical padding is also avoided. Prior art timelinebased editing systems introduce a side panel laid over the main timeline. However, is not suitable for the system because it cannot be synchronised with the timeline uniform zoom. The system provides significant visual cues such as naming, borders, backgrounds, with minimal vertical padding.

[0303] Groups need context outside of the group

[0304] One method the inventors explored to reduce vertical complexity and represent nesting in the timeline was to require opening new views for each group as the user traversed down into the group hierarchy. Opening a group would replace the displayed timeline region with only the content within that group, which breaks up the timeline region into more manageable chunks.

[0305] However, it is common to need to adjust timing in relation to other tracks, which may be outside of the group. For example, the user might have an animation of a flower whose different parts move differently, but are placed into a group so that effects can be collectively performed on the flower as a whole. The petals of the flower might need to bend when the bee, from the previous example, lands on it. So the user needs to see the keyframes of the movement of that bee, that are in another group and on a different track. Therefore the inventors determined that the content of groups needs to be visible to any other content.

[0306] This system allows context from grouped content to always be available while reducing the vertical space they take up. For example, the collapsing and expanding of groups introduces additional vertical space only when needed. Keyframes can take up a lot of vertical space

[0307] In work leading up to the invention, various approaches to displaying keyframes in the timeline region were prototyped. In particular, all tracks related to the keyframed parameter were shown below the content on the Timeline when expanded. For most keyframe types, this meant that at least 3 keyframe tracks were shown beneath the content for any additional keyframe added. In some cases, such as warp, expanding keyframe tracks would show a minimum of 16 keyframe tracks, one for each node of the warp that could be adjusted. While editing values on the Timeline, users can generally see 3-6 keyframe tracks at a time, so having a separate track for each value required valuable screen real estate on the Timeline, and required additional effort to navigate. In some prototypes, keyframe track titles did not appear on the keyframe track, and unless the keyframe track icon was significantly distinct, the user would need to select the individual keyframe track icon to see the title.

[0308] The described system overcomes these difficulties as follows. When a keyframe is expanded, each keyframed parameter relevant at that moment of the Timeline is displayed below the Content. Each keyframe track icon can be tapped to edit all relevant values in a popover which gives access to all parameters of a keyframe track. For example, if a Move and Scale keyframe track icon is selected, the popover allows the user to edit Translate X, Translate Y, Scale X, Scale Y and Rotate without having to expand those parameters into individual entities on the Timeline. This allows users to quickly and easily edit everything they need without adding vertical complexity to the Timeline. In some cases, the user may wish to edit the parameters as individual entities, and in this case, they can expand them by tapping the name of the keyframe track which is displayed between keyframes on the track. Selecting the name hides all other tracks, and displays the individual tracks for each of that keyframe types parameters. For example, if that moment in time had a Move and Scale keyframe, and a Noise keyframe, selecting the Move and Scale keyframe to expand its tracks would hide the Noise keyframe track from view. This allows users to have individual control of all relevant keyframe track parameters, without any other keyframe types to distract them or add visual clutter. Keyframes are easily identifiable and do not obscure content

[0309] To reduce vertical space, the inventors explored placing the keyframe icons over the content thumbnails. However, this created an issue where the icons covered the thumbnails to the point where they could not be used to identify specific content, especially when zoomed out. To address this difficulty, the described system represents the keyframe as a render of the content at that moment in time. For example, given two keyframes that transition from 0% blur to 10% blur, the first keyframe will show the content without the blur and the second keyframe shows the content with the blur. This visual information makes it much easier to find specific keyframes without also displaying their specific properties. This also provides an interface element that can be moved around, deleted etc. Using the content at that moment in time creates multiple visual representations of the content on the timeline. The keyframes not only do not obscure, but also provide even more context.

[0310] All keyframes are usually displayed as a single icon: most commonly as diamonds with text descriptions to the left of them. However, when zoomed out, these litter the timeline with small dots that do not assist rapid navigation. To address this difficulty when advanced keyframing, the system uses a specific icon for each animatable property, allowing them to easily be identified at lower zoom levels. They also assist in finding a specific track of interest to facilitate adding more keyframes of the same type.

[0311] Keyframes can use multiple easing types

[0312] Allowing users to keyframe multiple properties on the parent content introduces issues for customising easing. Although the keyframes can be represented as one object on the parent content, those properties might have different easings, such as linear or ease-in, for example. The system solves this problem by making the easings only available on keyframe tracks between the relevant keyframes. A popover becomes available when the user places the playhead between two keyframes. The user does not need to access these tracks to animate, it will use a default easing, but the customisation is there if required.

[0313] Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.

Claims

CLAIMS:

1. A computer-implemented method, including the steps of: generating, with a computer system, a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline region; displaying within the interactive timeline region and on the touchscreen display a view of at least a portion of a timeline of a multimedia project, the timeline being a visual representation of the multimedia project with graphical representations of items of media content from which a multimedia production is to be generated, wherein at least some of the graphical representations of the items of media content are displayed at respective vertical positions within the timeline representing respective compositing layers, and a horizontal position of each of the graphical representations within the timeline represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; responsive to touch gestures by a user on the interactive timeline region of the touchscreen display, changing the view displayed within the interactive timeline region, the touch gestures including zoom touch gestures to change zoom levels of the displayed view, and translation touch gestures to pan the displayed view; wherein the step of changing the view includes, responsive to at least the zoom touch gestures, dynamically re-generating the view, and causing the regenerated view to be rendered to the touchscreen display, in real-time during the corresponding touch gesture to dynamically update the displayed view in a continuous manner during and in accordance with the corresponding touch gesture; and wherein the zoom touch gestures include dual-axis zoom touch gestures to concurrently change vertical and horizontal zoom levels of the displayed view.

2. The method of claim 1, wherein the step of dynamically re-generating the view includes: storing in a cache objects representing respective items of media content that are visible in the view to be displayed, the objects being instances of a packed format data structure; determining that the objects stored in the cache do not include one or more objects representing respective items of media content that are visible in the view to be displayed; responsive to the determining, retrieving, from non-volatile storage, media data representing the one or more items of media content that are visible in the view to be displayed; processing the retrieved media data to generate one or more corresponding objects as respective instances of the packed format data structure representing the one or more items of media content that are visible in the view to be displayed; storing the generated objects in the cache; and culling from the cache objects representing respective items of media content that are not visible in the view to be displayed.

3. The method of claim 2, wherein the media data is stored on the non-volatile storage as B-trees in an ordered key value store.

4. The method of claim 2 or 3, wherein the step of dynamically re-generating the view includes processing the objects stored as instances of the packed format data structure in the cache to generate corresponding timeline objects in a timeline format based on a translation and scale of the view to determine absolute screen coordinates for the objects.

5. The method of claim 4, wherein the timeline objects are generated only if they represent interactive items.

6. The method of any one of claims 3 to 5, including storing the timeline objects in linear arrays for efficient access.

7. The method of any one of claims 3 to 6, including using the timeline objects to effect user interaction with items of media content.

8. The method of any one of claims 3 to 7, wherein the step of dynamically regenerating the view includes processing the timeline objects to generate corresponding render geometry data in the form of low-level shader commands of a graphics processing unit (GPU) and associated data, and sending the render geometry data to the GPU via a low-level interface to cause the GPU to efficiently render the view on the touchscreen display.

9. The method of claim 8, including determining visual stylings from properties of the objects in timeline format, and generating corresponding render geometry data for the determined visual stylings.

10. The method of claim 8 or 9, wherein the render geometry data represents a plurality of rendering layers, including a track layer to render track backgrounds, a content layer to render rectangles containing content over the track layer, and an adornment layer to render content colour tags over the content layer.

11. The method of any one of claims 1 to 10, including storing the multimedia project on a non-volatile storage medium as B-trees in an ordered key value store.

12. The method of any one of claims 1 to 11, including, responsive to a non- uniform dual-axis zoom touch gesture among the dual-axis zoom touch gestures, concurrently changing the vertical and horizontal zoom levels of the displayed view by corresponding different amounts during and as indicated by the non-uniform dual-axis zoom touch gesture.

13. The method of any one of claims 1 to 12, wherein the re-generating includes dynamically changing one or more of the graphical representations of items of media of the displayed view in accordance with the zoom levels.

14. The method of any one of claims 1 to 13, further including, responsive to a vertical zoom touch gesture among the zoom touch gestures, changing a zoom level of the displayed view in the vertical dimension only.

15. The method of any one of claims 1 to 14, further including, responsive to a horizontal zoom touch gesture among the zoom touch gestures, changing a zoom level of the displayed view in the horizontal dimension only.

16. The method of any one of claims 1 to 15, further including, responsive to a composite touch gesture among the zoom touch gestures, concurrently changing at least one zoom level of the displayed view as indicated by a zoom gesture component of the composite touch gesture, and panning the displayed view in any direction indicated by a translate gesture component of the composite touch gesture.

17. The method of any one of claims 1 to 16, further including, responsive to a translate touch gesture among the touch gestures, panning the displayed view along any path indicated by the translate touch gesture.

18. The method of claim 17, wherein the displayed view is panned in dynamic dependence on translation speeds of the translation touch gestures on the touchscreen display such that a fast translation of a translation touch gesture causes the displayed view to be panned further than the same translation touch gesture performed slowly.

19. The method of any one of claims 1 to 18, further including, responsive to a double tap touch gesture at a location within an item of content, increasing a zoom level of the displayed view of the timeline to display individual frames of the item of content, and the displayed view to be spatially translated such that a frame of the item of content corresponding to a temporal location of the double tap touch gesture is centered in the timeline region.

20. The method of claim 19, wherein the double tap touch gesture is a first double tap touch gesture and was performed when the displayed view was at a first zoom level, and, responsive to a subsequent double tap touch gesture, decreasing the zoom level of the displayed view to the first zoom level, and panning the displayed view to centre a location of the subsequent double tap touch gesture in the timeline region.

21. The method of any one of claims 1 to 20, further including, responsive to a flick touch gesture on the timeline region, causing playback of the multimedia project.

22. The method of any one of claims 1 to 21, further including, responsive to a selection gesture on the interactive timeline region of the touchscreen display, selecting a corresponding plurality of displayed items of media content indicated by the selection gesture so that one or more actions can be collectively performed on the selected items of media.

23. The method of claim 22, further including, responsive to a deselection gesture on the interactive timeline region of the touchscreen display, deselecting from the selected items of media content one or more corresponding items of the media content indicated by the deselection gesture.

24. The method of claim 22 or 23, further including defining a group whose members are the selected items of media, and displaying the group in the timeline region with thumbnail previews of content of the group; and, responsive to corresponding user input, toggling a display state of a group between an open state and a closed state; and displaying a group in the open state together with its member items of content displayed vertically and aligned horizontally with one another.

25. The method of any one of claims 1 to 24, including, responsive to a split touch gesture on the interactive timeline region of the touchscreen display, splitting each of one or more corresponding items of media content into corresponding separate portions.

26. The method of claim 25, including, responsive to a composite split gesture on the interactive timeline region of the touchscreen display, the composite splitgesture having a split gesture component and a subsequent split adjust gesture component, splitting each of one or more corresponding items of media content into corresponding separate portions indicated by the split gesture component, and dynamically adjusting the temporal location at which the one or more items of media are to be split into separate portions as indicated by the split adjust gesture component.

27. The method of any one of claims 1 to 26, wherein the timeline region of the graphical user interface includes an interactive playhead component whose location within the timeline region represents an active temporal location within an active item of content, and the method further includes, responsive to a corresponding touch gesture on the interactive playhead component, displaying a list of available actions appropriate for the content type of the item of content to allow selection by the user of a desired action from the displayed list.

28. The method of any one of claims 1 to 27, including displaying looped playback of a portion of the multimedia production whose starting and ending times within the multimedia production correspond to the leftmost and rightmost edges of the displayed view of the timeline region.

29. The method of any one of claims 1 to 28, wherein the graphical user interface includes a media display region to display a preview of the multimedia production or an item of media content, and the method further includes, responsive to a corresponding touch gesture on the graphical user interface, expanding the media display region to facilitate playback viewing, and replacing the timeline region with a corresponding reduced-size timeline window floating over the expanded media display region.

30. The method of claim 29, further including, responsive to corresponding touch gestures on the reduced-size timeline window, previewing a corresponding portion of an animation in the media display region by jumping back and forth between corresponding frames of the animation.

31. The method of claim 30, further including, responsive to corresponding touch gestures on the reduced-size timeline window, previewing a corresponding portion of an animation in the media display region by skipping forward andbackward in a sequence of animation frames to the next frame that is different from the current frame.

32. The method of any one of claims 1 to 31, further including, responsive to corresponding touch gestures on the timeline region, tagging items of media with corresponding display colours to visually identify and differentiate different subsets of items of media content.

33. The method of claim 32, including determining that a zoom level of the timeline region is greater than a threshold at a first time, and, in response, displaying the names of colour tagged items of content in a shape filled with the corresponding tag colour, and determining that the zoom level of the timeline region is not greater than a threshold at a second time and, in response, displaying colour tagged items of content as corresponding coloured horizontal lines representing time locations and durations of the colour tagged items of content.

34. The method of any one of claims 1 to 33, wherein an item of keyframed content is displayed in the timeline region with temporally spaced thumbnail images representing keyframes of the item of keyframed content, and the method further includes, responsive to a corresponding touch gesture on the item of keyframed content, displaying graphical representations of one or more keyframe parameters of the item of keyframed content at respective vertical locations in the timeline region.

35. The method of claim 34, further including, responsive to the corresponding touch gesture on the item of keyframed content, displaying graphical representations of only those one or more of the keyframe parameters that change between keyframes, and responsive to a further touch gesture on the item of keyframed content, displaying graphical representations of all of the keyframe parameters of the item of keyframed content at respective vertical locations in the timeline region.

36. A tangible, non-transitory, computer-readable storage medium having stored thereon a multimedia editing application that, when executed by at least oneprocessor of a computing device or system, cause the at least one processor to perform the method of any one of claims 1 to 35.

37. A system including: a touchscreen display; a memory; and at least one processor configured to execute the method of any one of claims 1 to 35.

38. A tangible, non-transitory, computer-readable storage medium having stored thereon a multimedia editing application that, when executed by at least one processor of a computing device or system, cause the at least one processor to perform a method, including the steps of: generating, with a computer system, a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline region; displaying within the interactive timeline region and on the touchscreen display a view of at least a portion of a timeline of a multimedia project, the timeline being a visual representation of the multimedia project with graphical representations of items of media content from which a multimedia production is to be generated, wherein at least some of the graphical representations of the items of media content are displayed at respective vertical positions within the timeline representing respective compositing layers, and a horizontal position of each of the graphical representations within the timeline represents a corresponding temporal position of the corresponding item of media in relation to the multimedia production; responsive to touch gestures by a user on the interactive timeline region of the touchscreen display, changing the view displayed within the interactive timeline region, the touch gestures including zoom touch gestures to change zoom levels of the displayed view, and translation touch gestures to pan the displayed view; wherein the step of changing the view includes, responsive to at least the zoom touch gestures, dynamically re-generating the view, and causing the regenerated view to be rendered to the touchscreen display, in real-time during the corresponding touch gesture to dynamically update the displayed view in acontinuous manner during and in accordance with the corresponding touch gesture; and wherein the zoom touch gestures include dual-axis zoom touch gestures to concurrently change vertical and horizontal zoom levels of the displayed view.

Images