Multimedia editing system and method

Abstract

A method includes: generating a graphical user interface including an interactive timeline area on a touchscreen display; displaying a view of at least a portion of a timeline of a multimedia project within the interactive timeline area, the timeline having graphical representations of media content items from which a multimedia work is to be generated, at least some of these representations being displayed at corresponding vertical positions of corresponding composition layers within the timeline, and the horizontal position of each representation representing a corresponding media item relative to a corresponding time position of the multimedia work; and regenerating the view in real time and rendering the view on the display during a dual-axis zoom touch gesture for concurrently changing the vertical and horizontal zoom levels of the displayed view, so as to dynamically update the displayed view in a continuous manner according to the corresponding touch gesture during the corresponding touch gesture.

Description

Technical Field

[0001] This invention relates to methods, systems, and applications for creating and editing digital multimedia works such as movies, film clips, and animations from media content items. Background Technology Multimedia works such as films, video clips, motion graphics, and animations are created and / or edited using media compositing and editing systems and applications. Many such systems and applications are available, varying in functionality and tailored to different categories of end users, from inexperienced general users to industry professionals. These systems and applications typically offer at least a subset of features such as original content creation, motion design, digital compositing, color correction, visual effects, and other post-production capabilities. Examples of such applications include ToonBoom, DaVinciResolve, iMovie, and Final Cut Pro (from Apple Inc.), as well as Adobe Premiere Pro. These are examples of timeline-based, non-linear video editing applications that allow multimedia or video projects to be created by: importing and editing individual components such as audio tracks, video clips, and still images; arranging these components one by one according to a common timescale or 'timeline'; and finally combining these components to generate an output video work.

[0003] Such applications vary greatly in ease of use and functionality, depending in part on whether they are designed for ordinary consumers or industry professionals. Regardless of the category, the suitability, ease of use, and efficiency of a given application depend heavily on its user interface. While some applications are extremely powerful, they may be difficult to learn, and even if learned, difficult or inconvenient to use.

[0004] The aim is to overcome or mitigate one or more difficulties of the existing technology, or at least provide a useful alternative. Summary of the Invention According to some embodiments of the present invention, a computer-implemented method is provided, the method comprising the following steps: A computer system is used to generate a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline area; Within the interactive timeline area, a view of at least a portion of the timeline of a multimedia project is displayed on the touchscreen display. The timeline is a visual representation of the multimedia project and has graphical representations of media content items from which a multimedia work is to be generated. At least some of the graphical representations of the media content items are displayed within the timeline at corresponding vertical positions representing corresponding composition layers, and the horizontal position of each graphical representation within the timeline represents the corresponding time position of the corresponding media item relative to the multimedia work. In response to a user's touch gesture on the interactive timeline area of ​​the touchscreen display, the view displayed in the interactive timeline area is changed, the touch gesture including a zoom touch gesture for changing the zoom level of the displayed view and a pan touch gesture for dragging the displayed view; The step of changing the view includes: at least in response to the zoom touch gesture, dynamically regenerating the view in real time during the corresponding touch gesture and rendering the regenerated view onto the touch screen display, so as to dynamically update the displayed view in a continuous manner according to the corresponding touch gesture during the corresponding touch gesture; The zoom touch gestures include dual-axis zoom touch gestures for concurrently changing the vertical zoom level and the horizontal zoom level of the displayed view.

[0006] In some embodiments, the step of dynamically regenerating the view includes: The cache stores objects representing the corresponding media content items visible in the view to be displayed; these objects are instances of a packaged format data structure. It is determined that the objects stored in the cache do not include one or more objects representing corresponding media content items that are visible in the view to be displayed; In response to the determination, media data representing the one or more media content items visible in the view to be displayed is obtained from a non-volatile storage device; The acquired media data is processed to generate one or more corresponding objects as representations of the packaged format data structure, which are corresponding instances of the one or more media content items visible in the view to be displayed; The generated object is stored in the cache; and Remove objects from the cache that represent corresponding media content items that are not visible in the view to be displayed.

[0007] In some embodiments, the media data is stored as a B-tree in an ordered key-value store on the non-volatile storage device.

[0008] In some embodiments, the step of dynamically regenerating the view includes: processing the object stored in the cache as an instance of the packaged format data structure to generate a corresponding timeline object in a timeline format based on the panning and scaling of the view, thereby determining the absolute screen coordinates of the object. In some embodiments, the timeline object is generated only when it represents an interactive item.

[0009] In some embodiments, the method includes storing the timeline object in a linear array for efficient access. The method may also include using the timeline object to enable user interaction with media content items.

[0010] In some embodiments, the step of dynamically regenerating the view includes: processing the timeline object to generate corresponding rendering geometry data in the form of low-level shader commands and associated data of a graphics processing unit (GPU); and sending the rendering geometry data to the GPU via a low-level interface to enable the GPU to efficiently render the view on the touchscreen display.

[0011] In some embodiments, the method includes: determining a visual style based on the attributes of the timeline-formatted object, and generating corresponding rendering geometry data for the determined visual style.

[0012] In some embodiments, the rendering geometry data represents a plurality of rendering layers, the plurality of rendering layers including a track layer for rendering a track background, a content layer on the track layer for rendering a rectangle containing content, and a decorative layer on the content layer for rendering content color markers.

[0013] In some embodiments, the method further includes: storing the multimedia item as a B-tree in an ordered key-value store on a non-volatile storage medium.

[0014] In some embodiments, the method further includes: in response to a non-uniform dual-axis scaling touch gesture in the dual-axis scaling touch gesture, concurrently changing the vertical scaling level and horizontal scaling level of the displayed view by different amounts as indicated by the non-uniform dual-axis scaling touch gesture during the non-uniform dual-axis scaling touch gesture.

[0015] In some embodiments, the regeneration includes one or more graphical representations of the media items in the displayed view that are dynamically changed according to the zoom level.

[0016] In some embodiments, the method further includes: in response to a vertical zoom touch gesture in the zoom touch gesture, changing only the zoom level of the displayed view in the vertical dimension. In some embodiments, the method further includes: in response to a horizontal zoom touch gesture in the zoom touch gesture, changing only the zoom level of the displayed view in the horizontal dimension.

[0017] In some embodiments, the method further includes: in response to a composite touch gesture in the zoom touch gesture, changing at least one zoom level of the displayed view as indicated by the zoom gesture portion of the composite touch gesture, while simultaneously dragging the displayed view in any direction as indicated by the pan gesture portion of the composite touch gesture.

[0018] In some embodiments, the method further includes: in response to a panning touch gesture in the touch gesture, dragging the displayed view along any path as indicated by the panning touch gesture. In some embodiments, the displayed view is dynamically dragged according to the panning speed of the panning touch gesture on the touchscreen display, such that a rapid panning of the panning touch gesture causes the displayed view to be dragged further than a slowly executed identical panning touch gesture.

[0019] In some embodiments, the method further includes: in response to a double-touch gesture at a location within a content item, increasing the zoom level of the displayed view of the timeline to display an independent frame of the content item, and spatially translating the displayed view such that the frame of the content item corresponding to the time position of the double-touch gesture is centered in the timeline region.

[0020] In some embodiments, the double-touch gesture is a first double-touch gesture and is performed when the displayed view is at a first zoom level, and in response to a subsequent double-touch gesture, the zoom level of the displayed view is reduced to the first zoom level, and the displayed view is dragged so that the position of the subsequent double-touch gesture is centered in the timeline region.

[0021] In some embodiments, the method further includes: responding to a flick touch gesture on the timeline region to cause playback of the multimedia item.

[0022] In some embodiments, the method further includes: in response to a selection gesture on the interactive timeline area of ​​the touchscreen display, selecting corresponding to a plurality of displayed media content items as indicated by the selection gesture, such that one or more actions can be uniformly performed on the selected media items. In some embodiments, the method further includes: in response to a deselection gesture on the interactive timeline area of ​​the touchscreen display, deselecting one or more corresponding items of the media content from the selected media content items as indicated by the deselection gesture.

[0023] In some embodiments, the method further includes: defining a group whose members are selected media items, and displaying the group in the timeline area as a thumbnail preview of the group's content; and switching the display state of the group between an open state and a closed state in response to a corresponding user input; and displaying the group in the open state, while the member content items of the group are displayed vertically and horizontally aligned with each other.

[0024] In some embodiments, the method further includes: in response to a split touch gesture on the interactive timeline area of ​​the touchscreen display, splitting each of one or more corresponding media content items into corresponding separate parts.

[0025] In some embodiments, the method further includes: responding to a composite split gesture on the interactive timeline area of ​​the touchscreen display, the composite split gesture having a split gesture portion and a subsequent split adjustment gesture portion, splitting each of one or more corresponding media content items into corresponding individual parts as indicated by the split gesture portion, and dynamically adjusting the timing of splitting the one or more media items into individual parts as indicated by the split adjustment gesture portion.

[0026] In some embodiments, the timeline area of ​​the graphical user interface includes an interactive playhead portion, the position of which within the timeline area indicates the time position of an active content item, and the interactive playhead portion responds to a corresponding touch gesture to allow selection of a desired action from a list of available actions suitable for the content type of the content item.

[0027] In some embodiments, the method includes: displaying a loop of a portion of the multimedia work, wherein the start and end times of the portion within the multimedia work correspond to the leftmost and rightmost edges of the displayed view of the timeline region.

[0028] In some embodiments, the graphical user interface includes a media display area for displaying a preview of the multimedia work or media content item, and the method further includes: in response to a corresponding touch gesture on the graphical user interface, expanding the media display area to facilitate playback viewing, and replacing the timeline area with a corresponding, smaller-sized timeline window that floats on the expanded media display area.

[0029] In some embodiments, the method includes: in response to a corresponding touch gesture on the reduced-size timeline window, previewing a corresponding portion of the animation in the media display area by jumping back and forth between corresponding frames of the animation.

[0030] In some embodiments, the method includes: in response to a corresponding touch gesture on the reduced-size timeline window, previewing a corresponding portion of the animation in the media display area by jumping forward and backward in the animation frame sequence to the next frame different from the current frame.

[0031] In some embodiments, the method includes: in response to a corresponding touch gesture on the timeline region, marking media items with a corresponding display color to visually identify and distinguish different subsets of media content items.

[0032] In some embodiments, the method includes: determining that the scaling level of the timeline region is greater than a threshold at a first time, and in response, displaying the name of a color-marked content item in a shape filled with the corresponding marker color; and determining that the scaling level of the timeline region is not greater than the threshold at a second time, and in response, displaying the color-marked content item as a corresponding colored horizontal line representing the time position and duration of the color-marked content item.

[0033] In some embodiments, the timeline region displays content items with keyframes, wherein time-spaced thumbnail images represent keyframes of the content items with keyframes, and the method further includes: in response to a corresponding touch gesture on the content item with keyframes, displaying a graphical representation of one or more keyframe parameters of the content item with keyframes at a corresponding vertical position in the timeline region.

[0034] In some embodiments, the method includes: in response to a corresponding touch gesture on the content item with keyframes, displaying a graphical representation of only one or more keyframe parameters that vary between keyframes; and in response to another touch gesture on the content item with keyframes, displaying a graphical representation of all keyframe parameters of the keyframe parameters of the content item with keyframes at a corresponding vertical position in the timeline region.

[0035] According to some embodiments of the present invention, a tangible non-transitory computer-readable storage medium is provided, wherein a multimedia editing application is stored on the computer-readable storage medium, and the multimedia editing application, when executed by at least one processor of a computing device or system, causes the at least one processor to perform any of the methods described above.

[0036] According to some embodiments of the present invention, a system is provided, the system comprising: Touchscreen display; Memory; and At least one processor, wherein the at least one processor is configured to perform any of the methods described above.

[0037] According to some embodiments of the present invention, a tangible non-transitory computer-readable storage medium is provided, wherein a multimedia editing application is stored on the computer-readable storage medium, and the multimedia editing application, when executed by at least one processor of a computing device or system, causes the at least one processor to perform a method, the method comprising the following steps: A computer system is used to generate a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline area; Within the interactive timeline area, a view of at least a portion of the timeline of a multimedia project is displayed on the touchscreen display. The timeline is a visual representation of the multimedia project and has graphical representations of media content items from which a multimedia work is to be generated. At least some of the graphical representations of the media content items are displayed within the timeline at corresponding vertical positions representing corresponding composition layers, and the horizontal position of each graphical representation within the timeline represents the corresponding time position of the corresponding media item relative to the multimedia work. In response to a user's touch gesture on the interactive timeline area of ​​the touchscreen display, the view displayed in the interactive timeline area is changed, the touch gesture including a zoom touch gesture for changing the zoom level of the displayed view and a pan touch gesture for dragging the displayed view; The step of changing the view includes: at least in response to the zoom touch gesture, dynamically regenerating the view in real time during the corresponding touch gesture and rendering the regenerated view to the touchscreen display, so as to dynamically update the displayed view continuously according to the corresponding touch gesture during the corresponding touch gesture; and The zoom touch gestures include dual-axis zoom touch gestures for concurrently changing the vertical zoom level and the horizontal zoom level of the displayed view.

[0038] This article also describes a computer-implemented multimedia editing process for generating multimedia works from media items of a multimedia project, the process comprising the following steps: Generate a graphical user interface for display on a touchscreen display, the graphical user interface comprising: An interactive timeline area is used to display a view of graphical representations of media items from which a multimedia work is to be generated, wherein at least some of the graphical representations of the media content items are displayed in the timeline area at corresponding vertical positions representing corresponding composition layers, and the horizontal position of each graphical representation in the timeline area represents the corresponding time position of the corresponding media item relative to the multimedia work. The interactive timeline area of ​​the graphical user interface is operable to allow a user to perform a dual-axis zoom touch gesture on the touchscreen display, thereby simultaneously changing the zoom level of the view displayed in the timeline area in both the vertical and horizontal dimensions; and During the dual-axis zoom touch gesture, the displayed view of the interactive timeline area is dynamically regenerated and rendered in real time to reflect the dynamically changing zoom level. Attached Figure Description The following description, by way of example only, refers to the accompanying drawings, in which: Figures 1 to 46 These are representative screenshots of a user interface (“UI”) generated by a multimedia editing system and process according to embodiments described in this invention, wherein example animations and movie projects are loaded to illustrate the interactive features of the UI and system, and wherein: Figure 1 The illustration shows the main parts of the UI, particularly the media display area (used to preview media content) in the upper part of the display and the timeline area (showing a graphical representation of the media content items from which a multimedia work will be generated) in the lower part of the display, which are separated by a toolbar with interactive controls. Figure 2 An example of a global zoom-in touch gesture in the UI, which takes the form of a pinch-to-zoom gesture, is shown. This global zoom-in touch gesture causes the system to simultaneously increase the zoom level of the displayed view in the timeline area in both the vertical and horizontal dimensions. Figure 3 It shows Figure 2 The results of the global zoom touch gesture are shown, and an example of a pan touch gesture for dragging the displayed view is also shown; Figure 4 It shows Figure 3 The results of the panning touch gesture are shown, and a second pinch zoom-out gesture is also shown for further zooming in on the displayed view; Figure 5 The results of the second zoom gesture are shown (which provides access to the zoom levels of individual frames of the animation), and a global zoom touch gesture in the form of a pinch zoom gesture is also shown, which causes the system to simultaneously reduce the zoom level of the displayed view in the timeline area in both the vertical and horizontal dimensions. Figure 6 The UI is shown during the editing of different sample animation projects, as well as an example of a horizontal zoom touch gesture in the form of a three-finger right swipe gesture, which causes the system to increase the zoom level of the displayed view in the horizontal dimension only in the timeline area; Figure 7 It shows Figure 6 The result of the horizontal zoom touch gesture provides zoom levels that allow access to individual keyframes; because the system playhead is displayed when the horizontal zoom touch gesture is invoked, the zoom is anchored to the horizontal position of the playhead. Figure 8 A horizontal zoom-out touch gesture in the form of a three-finger swipe left is shown, which causes the system to reduce the zoom level of the displayed view in the horizontal dimension only in the timeline area; Figure 9 It shows Figure 8 The results of the horizontal zoom-out touch gesture are shown, and another horizontal zoom-in touch gesture is also shown, but this horizontal zoom-in touch gesture is invoked without the playhead being displayed, thereby causing the system to zoom in on the display, but this horizontal zoom-in touch gesture is anchored to the location where the gesture was invoked; Figure 10 It shows Figure 9 The result of a horizontally shrinking touch gesture; Figure 11 An example of a vertical zoom touch gesture is shown, which uses a three-finger swipe up gesture to increase the zoom level of the displayed view in the vertical dimension only in the timeline area; Figure 12 It shows Figure 11 The result of vertically magnifying touch gestures allows for a clearer view of thumbnail previews without altering the horizontal time scale; Figure 13 An example of a vertical zoom-out touch gesture in the form of a three-finger swipe down is shown, which causes the system to reduce the zoom level of the displayed view in the vertical dimension only in the timeline area; Figure 14 It shows Figure 13 The result of a vertically shrinking touch gesture; Figure 15 An example of a drag-and-drop touch gesture in the form of a single-finger swipe gesture is shown, which allows the displayed view to be dragged in any direction (i.e., translated in space); Figure 16 It shows Figure 15 The result of a drag-and-drop touch gesture; Figure 17 An example of a double-tap gesture performed on a media content item is shown, which causes the system to dynamically zoom in on the display in both the horizontal and vertical dimensions to clearly show the frame of the media content item corresponding to the position of the double-tap gesture; Figure 18 It shows Figure 17 The result of the double-click gesture is shown, and a second double-click gesture performed on a media content item is shown, which causes the system to dynamically shrink the display in both the horizontal and vertical dimensions, and the second double-click gesture is anchored to the position of the second double-click gesture; Figure 19 It shows Figure 18 The result of the second double-tap gesture; Figure 20 and Figure 21 An example of a quick flick gesture on the playhead is shown, which causes the system to start playback from the beginning of the media content item below the playhead; Figures 22 to 28 Examples of selected touch gestures are shown, which enable the system to select the corresponding media content item, thereby allowing the system to uniformly apply actions to these selected items; Figure 29 and Figure 31 Examples of split touch gestures are shown, which allow the system to divide individual media items into separate parts and allow users to precisely adjust the timing of the splits; Figure 32 and Figure 33 Examples of delete touch gestures are shown, which cause the system to remove the corresponding media content item from a multimedia project; Figure 34 The illustration shows how to use grouping to group individual content items together, so that these content items can be treated as a single content item when performing actions such as moving or applying effects. Figures 35 to 37 This example demonstrates applying an effect to a media content item using a UI playhead; in this case, the effect applied is a Gaussian blur. Figure 38 An example of the "Loop to Screen" playback mode is shown, in which only the display portion of the selected media content item is repeatedly played on the media display area until it is interrupted by the user; Figure 39 and Figure 40 An example of a compact timeline mode for the UI is shown, which is achieved through a single-finger swipe down touch gesture (such as...). Figure 39 The gesture shown in the image is invoked, and the single-finger swipe down touch gesture enables the system to increase the size of the media display area by displaying a compact representation of a limited timeline area in a floating window on the media display area. This compact timeline provides a page-turning function that allows users to flick backward and forward between consecutive frames of the animation. Figure 41 and Figure 42 This example illustrates color-coded UI elements that facilitate the location, identification, and grouping of media content items by using different colors to mark them. When the zoom level is sufficient to clearly see the content names, these markers are displayed as the background fill color of the content names (e.g., ...). Figure 41 As shown), and when the zoom level is too low to meaningfully display these content names, these markers are displayed as colored horizontal lines (such as...). Figure 42 (as shown) Figures 43 to 46 The diagram illustrates how keyframes can be displayed at different levels of detail based on user needs. Figure 47 This is a block diagram of a multimedia editing system according to an embodiment of the present invention; Figure 48 yes Figure 47 A schematic diagram of the architecture of a multimedia editing system according to an embodiment of the present invention; Figure 49 This is a schematic diagram illustrating the sequence of data format transformations involved in real-time rendering of the timeline region; and Figure 50 It is a schematic diagram illustrating the data objects of the system and their relationships. Detailed Implementation In the process of developing this invention, the inventors discovered difficulties inherent in existing multimedia editing systems and applications. In particular, the inventors found that existing timeline-based media editing systems and applications are constrained by a user interface paradigm that restricts timeline navigation in various ways, resulting in decreased user efficiency and inconvenience, and limiting the visibility and access to the constituent media assets of a multimedia project.

[0041] Furthermore, the inventors recognized that existing multimedia editing systems and applications for video and animation were developed for desktop computers with large monitors to facilitate navigation of large projects (e.g., feature-length films) containing numerous constituent media assets and lasting for extended periods. However, the inventors believed that a graphical user interface (UI) could be designed for portable touchscreen computing devices with limited display size and computing power (such as the Apple iPad™) that would still facilitate the production of large multimedia projects.

[0042] The inventors discovered that a common limitation of existing video editing and animation applications is that these applications typically only display about two to five media content tracks at a time. This poses a problem when editing projects containing a large number of tracks (e.g., ten or more), as users lose all context beyond the limited visibility of content items on the display, requiring manual panning or individual zooming to regain context regarding the relationship between these items and other content items and the project as a whole. This is a significant limitation because projects containing ten or more tracks are common. For example, a single video frame of a film or animation can be composed of various techniques and effects, sound effects, music, animation elements, video clips, lighting, foreground elements, background elements, etc., each of which may involve multiple tracks or content layers (displayed vertically stacked) at any point in a video editing or animation project. Because tracks are displayed as vertically stacked to create layered works and arranged horizontally over the total duration of the project, the need for convenient navigation of large projects on a small 8.3-inch iPad™ screen presents a significant and potentially unsolvable challenge, given the recognition that simplified interaction and navigation are crucial for a good user experience.

[0043] In this regard, the inventors determined that existing video editing applications limit project navigation to vertical dragging between tracks and horizontal dragging along the timeline, with adjustments to view zoom levels between a few predetermined and fixed percentage values, such as 25%, 50%, 75%, 100%, etc. These limitations make standard and frequent interactions, such as context-based timing and adjustments, cumbersome, thus severely impacting the user experience. The inventors found a need for more flexible and robust navigation and zoom methods.

[0044] Having identified the limitations of existing multimedia editing systems and applications, the inventors spent several years researching, developing, and prototyping possible solutions until they finally devised a technical solution that resolved all the identified shortcomings of the prior art, thus forming this invention. The inventors' key requirements and the resulting practical problems are further discussed at the end of this specification.

[0045] As described herein, embodiments of the present invention address the difficulties of the prior art by providing a timeline-based multimedia editing system and application with a graphical user interface that represents a new paradigm in timeline navigation, and substantially simplifies the interactions required for creating multimedia works such as movies, film clips, and animations. By simplifying the user's interaction in navigating the timeline to locate and access specific media content items and integrating these items into multimedia projects and works, embodiments of the present invention not only significantly improve user efficiency and ease of use, but also enhance the efficiency of the underlying system at both the software and hardware levels.

[0046] This document describes embodiments of the invention in the context of touchscreen display devices in which a user interacts with a multimedia editing system or application by contacting or touching the touchscreen using appropriate parts of the user's body (typically, but not necessarily, one or more fingers and / or the thumb) and / or a stylus device (such as the Apple Pencil™), typically simultaneously at multiple locations. In some cases, this interaction is defined by moving the contact location (also referred to herein as a 'touch point' (i.e., moving the contact part of the user's body or stylus device while maintaining contact with the touchscreen surface)) to define an interactive 'touch gesture' representing corresponding input from the user to the multimedia editing system or application.

[0047] In the described embodiments, the touchscreen display device is the touchscreen display of an Apple iPad tablet computing device (such as...). Figure 47 As shown), the multimedia editing process described herein is implemented as programming instructions for a software application 4800 installed on an iPad device. However, it will be apparent to those skilled in the art that other types of computing devices (including desktop computers with touchscreen displays) may be used in other embodiments, and the process described herein may also be implemented alternatively in other forms, including, for example, as firmware, or configuration data for a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC) for a dedicated multimedia production system, or any practical combination of these forms.

[0048] As described above, timeline-based multimedia editing systems or applications provide a graphical user interface (also referred to in the art as a "GUI" or "UI") that allows users to interact with the system or application to import, edit, arrange, and combine individual media items or portions into multimedia projects and multimedia works. The UI is displayed on a touchscreen monitor, allowing user input to be provided through various 'gestures' related to the content currently displayed on the touchscreen monitor. These gestures are typically defined by touching one or more corresponding locations on the screen (also referred to herein as 'touch points'), in some cases by dynamically moving one or more touch points along a path on the touchscreen surface, and in some cases by the speed of that movement.

[0049] Overview of the user interface An embodiment of the multimedia editing system will now be described in the context of example multimedia projects, which for simplicity will only have a modest number of individual video and audio content items. However, in practice, the number or items and the number of layers are primarily constrained by the hardware. For reference, current iPad models equipped with Apple's silicon (M2 or M4) chips can display up to 200 content tracks.

[0050] Figure 1 This is a representative screenshot showing the system's UI during the editing of a sample animation project. As is typical of timeline-based multimedia editing applications, the UI provides two main content areas: a media display area 102 in the upper portion of the display and a timeline area (also referred to herein as the "timeline") 104 in the lower portion of the display, separated by a toolbar 106. The timeline area 104 displays graphical representations (e.g., keyframes, thumbnail previews) of media content items 108, which are to be combined to generate output multimedia works such as movies, movie clips, or animations. Content items include digital assets such as video, drawings, animations, audio, images, or text.

[0051] The graphical representation of multimedia content items is arranged in vertically stacked horizontal rows (referred to herein as layers or “tracks”) 110, where the horizontal position of each content item corresponds to its temporal position within the project / work (as indicated by a timeline ruler or axis 111 displayed at the top of the entire timeline area 104), which has scales and time value labels to indicate these temporal positions. A single track may contain multiple content items, as shown in the figure.

[0052] The order of tracks in a vertical stack represents a hierarchical layer, so that higher tracks in the stack will be rendered on lower tracks during compositing (i.e., combining them together). Therefore, simply put, the final output can be considered as generated by folding or flattening the track layers of an item (i.e., individual content item rows) into a single track, where the horizontal dimension represents the time elapsed since the beginning of the item / work.

[0053] To improve efficiency, the system allows users to group multiple tracks together so that these tracks can be processed uniformly, thus allowing one or more actions (such as movement, applying effects, or keyframes) to be performed simultaneously on all tracks in that group. For example, Figure 1 Two content groups 112 and 114 are shown. Group 112 on the left is shown in an expanded or open state to display group members (as indicated by the downward orientation of the arrow icon to the right of the group label), and the group members are shown as a stack of layered tracks contained within rounded (light gray) bounding boxes.

[0054] To reduce visual clutter, you can also display groups that are minimized or closed, such as... Figure 1 Group 114 is shown on the right. The minimized state of group 114 is indicated by the rightward orientation of the arrow icon to the right of the group label, and also by the absence of a track below the group track within the bounding box. The minimized group is displayed as a single "group" item containing a representative preview bitmap, thus reducing visual clutter.

[0055] A toolbar 106 extends horizontally across the monitor between the media display area 102 and the timeline area 104. A set of icons displayed on the right side of the toolbar 106 provides access to corresponding controls for selecting the appropriate operating modes of the system, including playback mode, execution mode, editing mode, and painting mode. The plus sign ("+") icon on the far right of the toolbar 106 can be selected to add content to the project. The project title ("Untitled Movie" in this example) is displayed on the left side of the toolbar 106.

[0056] The media display area 102 in the upper part of the monitor is used to display the selected media content item or track during editing and to preview the output. For example... Figure 2As shown, the media display area 102 has two distinct sections, referred to as the "Stage" 202 and the "Backstage" 204. Selected content can be displayed on either section, but content to be exported (i.e., included in the output work) is displayed only on the Stage 202. Content displayed on the Backstage 204 will not be exported. Users can resize and move the Stage 202 within the boundaries of the media display area 102, and when the Stage 202 is not maximized, one or more portions of the media display area 102 not occupied by the Stage 202 display the Backstage 204. Figure 1 The stage 202 is shown maximized to occupy the entire media display area 102, while the background 204 is not visible. Figure 2 The diagram shows a stage 202 reduced in size to occupy only a portion of the media display area 102, with the background 204 being another area outside (or "behind") the media display area 102. Figure 2 (The area is empty).

[0057] Return to Figure 1 A highlighted border 116 around the minimized group 114 on the right indicates that group 114 is currently selected, and animation frames (not shown) from the selected group are displayed on stage 202. The selected content or active content and its current time position within the selected content or active content are indicated by the position of a visual feature referred to herein as 'playhead' 118, which in this mode and in the described embodiment is displayed as an icon 120 in the form of a (red) diamond shape, which includes a traditional filmmaking clapperboard immediately below the content (to facilitate user visual focus) and is accompanied by a vertical white line 122 overlapping the content (to accurately indicate the current playback position therein). The current time position of the playhead (in Figure 1 In the editing context, the selected content item is displayed as text 124 located in the lower left corner of the media display area 102, formatted as "hh:mm:ss.ff", where "ff" represents the frame number. Therefore, in Figure 1 In the middle, the time position text "00:00:07.08" indicates that stage 202 is displaying the eighth frame at the seventh second of the project.

[0058] Timeline Navigation Apart from the simplest projects, which consist of only a very small number of individual items or rows, most projects have many (e.g., at least ten and often hundreds) rows / layers that cannot be displayed in a single view on any touchscreen, or at least not clearly, and especially not on the small screens of portable computing devices like the Apple iPad. Therefore, users will typically only be able to view and interact with a small subset of the project's media items and tracks at any given time. Similarly, except for projects with very short durations, users can only clearly view a short segment of the entire project at any given time. In other words, users will almost always only be able to view and interact with a small portion or window of the project's complete timeline and tracks at any given time. Therefore, timeline areas constitute viewports for displaying only a selected portion of the entire timeline or content set of the project at any given time. Thus, the ability to navigate the timeline to locate and interact with individual items of interest is extremely important for ease of use and efficiency.

[0059] As mentioned above, existing video editing applications require multiple zoom actions, as well as horizontal and vertical scrolling or dragging actions, to access specific content items at a specific point on the project timeline. For complex projects, this can potentially involve dozens of discrete zoom and drag actions, which is clearly inefficient and inconvenient, making such applications virtually unusable for large projects.

[0060] To address the limitations of existing timeline-based multimedia editing systems, embodiments of the present invention provide a touchscreen-based user interface with a view displaying a timeline, typically showing only a portion of the timeline. The displayed view can be dynamically, continuously, and fully zoomed and dragged individually, concurrently, and in real-time across two spatial dimensions (corresponding to time and layers) via simple touch gestures. That is, the user can dynamically and continuously (and therefore smoothly) navigate by zooming and / or dragging to view any desired portion of the project timeline in a manner largely similar to zooming and dragging a digital image in an image viewer application, both in the horizontal time dimension and the vertical track dimension, and at any actual zoom level. However, at least for the zoom touch gestures of the multimedia editing system described herein, unlike viewing a static image, the system responds by continuously and dynamically regenerating the view (including its layout) and rendering the regenerated view onto the touchscreen (all of which are performed in real-time according to the dynamically changing zoom level as the gesture is executed), thereby effectively generating a real-time animation of the constantly changing view, providing immediate feedback to the user, and further facilitating the location and access to specific items or portions of the timeline area of ​​interest to the user.

[0061] The view is dynamically regenerated in this way because, unlike scaling static images, changes in the scaling level of the timeline region not only alter the scaling level / magnification but also change the actual layout of the view and, in some cases, the objects displayed. For example, some visual styles (e.g., dashed line spacing) do not scale with the scaling level, while others (e.g., line width, text size, etc.) do, and some styles are affected by corresponding maximum and minimum thresholds. When text becomes too small as the scaling level decreases, it is drawn as a single line. Single-axis scaling does not change the aspect ratio of visible text and other objects in the displayed view, and at some scaling levels, some objects may not be visible at all. For example, decreasing the horizontal (timeline) scaling level (with or without concurrently changing the vertical scaling level) causes the thumbnail images of content items and keyframes in the view to be pulled closer along the time (horizontal) dimension of the timeline. However, as the horizontal scaling level decreases, some thumbnail images can be omitted from the layout to reduce visual clutter while maintaining the remaining thumbnail images at roughly the same size (and visually pleasing). Similarly, while keyframe tracks will always be displayed in some form, keyframe icons within those tracks will not appear if the zoom level drops below a threshold. Likewise, each audio track is represented as a bar chart of 'samples'. As the horizontal zoom level decreases, the number of samples displayed decreases accordingly to maintain a consistent sample bar width (in screen space). These and other dynamic changes to the layout require the view to be dynamically regenerated and displayed in real-time whenever the user performs at least a zoom touch gesture on the timeline area.

[0062] These capabilities represent a fundamental breakthrough from existing timeline-based multimedia editing systems and the conventional navigation paradigms that constrain timeline navigation and user interaction with multimedia projects. As described below, the dynamic capabilities of multimedia editing systems present numerous technical challenges that must be addressed before the system can be implemented.

[0063] For example, Figure 2 This is a screenshot of the system touchscreen, showing the timeline area 104 in a fully zoomed-out state, allowing the user to view all content items of the animation project. However, in this global view, it may be difficult to distinguish or interact with finer details such as individual keyframes and thumbnail previews of the project. To facilitate interaction with these details, the UI allows the user to zoom in on the timeline area 104 to focus on a specific content item 206 by performing a simple pinch-to-zoom gesture familiar to users of touchscreen devices (such as smartphones) when viewing static images.

[0064] In the accompanying drawings, solid circles represent the locations of 'touch points' on the touchscreen (e.g., the location of a user's finger or stylus), and arrows extending from these solid circles indicate the movement of the touch point location as the user performs a 'touch gesture'. Therefore, in Figure 2 In the text, examples of the pinch-to-zoom gestures described above are represented by opposing, outward-pointing arrows 208, generally pointing southwest and northeast. These arrows indicate that the touch points, initially located at positions indicated by solid circles (typically, but not necessarily, corresponding to the positions where the user's fingers and thumbs touch the touchscreen), move away from each other in these directions to zoom in on the desired portion of the timeline area defined by these two positions in a manner familiar to the user of the touchscreen computing device. For ease of description, unless the context otherwise indicates, references to the user's fingers or thumbs in this specification should be broadly understood to encompass any part of the stylus or the user's body (typically the user's fingers or thumbs).

[0065] This form of scaling (where both the horizontal and vertical directions are scaled concurrently and dynamically) is referred to herein as 'uniform scaling' because the horizontal and vertical directions are scaled by the same amount. However, in some embodiments, the system also supports non-uniform scaling, where the horizontal and vertical dimensions are scaled by different amounts concurrently.

[0066] As a result of the pinch-to-zoom gesture 208 described above, the corresponding portion of the timeline area is displayed at a higher magnification or zoom level (e.g., ...). Figure 3 As shown), this allows only the corresponding portion of the entire timeline area to be displayed. The displayed timeline portion is anchored to the location where the user initiates the pinch-to-zoom gesture 208; specifically, anchored to the midpoint between the touch locations. This is important because it allows the user to quickly obtain more detail from a specific desired location with a single gesture.

[0067] In existing technology applications, achieving such a perspective change requires multiple gestures. Furthermore, in large projects with extended durations and numerous tracks and content items, obtaining a similar view may require dozens of individual zoom actions as well as vertical and horizontal dragging actions. It's worth noting that existing video editing applications perform zooming from a fixed center position on the display, while the multimedia editing system described in this paper performs zooming from the gesture position—a simpler and more intuitive approach.

[0068] After performing the pinch-to-zoom gesture 208, the user can continue to touch the screen and use the two-finger drag gesture 302 (such as...). Figure 3 (As shown), this allows the displayed view to be dynamically dragged in real-time to different parts of the timeline area at the same magnification / zoom level. Therefore, users can adjust the zoom level and the position of the displayed portion through a smooth interaction or a combination of gestures without taking their fingers off the screen.

[0069] Figure 4 The result of executing the composite gesture described above, combining the pinch-to-zoom gesture 208 and the drag gesture 302, is shown. The resulting high magnification or zoom level makes it easier to interact with individual keyframes and thumbnail previews in the corresponding track 402. At this zoom level, individual frames of content are not directly accessible but are represented uniformly through previews, where a single preview represents multiple individual frames. A higher magnification view is required to interact with individual frames.

[0070] Therefore, another pinch-to-amplify gesture 404 can be performed (such as...) Figure 4 (As shown by the touch point and arrow in the image) to zoom in further, thereby enabling viewing and interacting with individual frames of the desired content item 402, such as... Figure 5 As shown. At this very high magnification or scaling level, users can add individual frames 502 for frame-by-frame animation and / or interact with individual frames. For example, selecting an individual frame 504 via a single-finger tap gesture causes the selected frame to be highlighted with a border (as shown). Figure 5 As shown), the playhead 118 is displayed below the selected frame to indicate the current playback position. (In the context of an independent frame, the vertical white line 122 of the playhead 118 (indicating the playback position within the content item) is not displayed because it has no function.) A circular button 506 containing a plus sign ("+") is displayed in the upper right corner of the frame border and can be selected to insert a new frame at the position of the selected frame 504.

[0071] Users can also perform the opposite pinch gesture (pinch to shrink instead of pinch to enlarge) to perform the shrink pinch gesture 508, such as... Figure 5 As shown. Similar to the pinch-to-zoom gesture, the pinch-to-zoom gesture 508 is also anchored to the location where the user initiates the gesture. However, if the pinch-to-zoom gesture 508 is executed quickly, the magnification or zoom level is immediately reset to its lowest level to display the entire item; in this example, it returns to the previous zoom level. Figure 2 The view shown.

[0072] Timeline navigation: Horizontal (time) axis zoom Figure 6A timeline area with a single content group 602 is shown. This view displays a timeline ruler 111 marked in seconds, allowing the user to see the overall time position and duration of group 602 in a single view. However, at this zoom level, they cannot see or edit finer details, such as individual keyframes. To facilitate these actions, the user can use a horizontal zoom gesture to expand or shrink the timeline area view only on the horizontal timeline. In the described embodiment, the horizontal zoom gesture is a horizontal drag or swipe zoom gesture performed by simultaneously moving three touch points (typically three fingers) to the right, but this is not necessarily the case in other embodiments.

[0073] Therefore, in Figure 6 In the image, the horizontal three-finger zoom gesture 604 is represented by three touch positions, each with a corresponding arrow pointing to the right. This indicates that three fingers of the user's hand touch the indicated positions on the touchscreen and then move them horizontally to the right. This allows the system to dynamically zoom in on the displayed view of the timeline region only relative to the horizontal time scale dimension, thus producing an expanded view (e.g., ...). Figure 7 The view shown has its timeline scale 111 updated accordingly to display units of frames instead of seconds. At this horizontal zoom level, users can clearly view and interact with the individual keyframes 606. Figure 6 and Figure 7 The comparison confirms that vertical magnification or scale retention is unaffected by gestures. If the playhead 118 is displayed when gesture 604 is initiated, the resulting timeline display portion is anchored to the position of the playhead 118, not the position of the gesture.

[0074] Users can also reverse the direction of the horizontal three-finger swipe zoom gesture (swipe left instead of right) to perform the horizontal three-finger swipe zoom gesture 802, such as... Figure 8 The three touch points and corresponding arrows are shown in the image. As the user changes the horizontal zoom level during a gesture, the timeline view and the time scale of the timeline ruler 111 dynamically adjust in real time. Figure 9 As shown, the resulting view allows more content to be shown along the horizontal axis, but without losing or reducing information along the vertical dimension.

[0075] If the playhead 118 is not visible in the current view, the horizontal zoom in and zoom out gestures are instead anchored to the location where the user initiated the gesture. For example, from Figure 9 The view shown (in which the playhead 118 is not visible) will produce a horizontal three-finger zoom gesture 902. Figure 10 The timeline view shown.

[0076] Timeline navigation: Vertical axis zoom Changing only the magnification (i.e., zooming in or out) of the vertical (content track or layer) dimension can be similarly achieved by performing a vertical zoom gesture. In the described embodiment, the vertical zoom gesture is performed by simultaneously moving three touch points (typically three fingers) up or down along the touchscreen, but this is not necessarily the case in other embodiments. For example, Figure 11 A view of the timeline region with a single content group 1102 is shown. This view displays a timeline ruler 111 marked in seconds, allowing the user to see the overall duration of group 1102 in a single view. The user can expand the vertical scale of the group using a vertical three-finger swipe zoom gesture 1104 without affecting the horizontal time scale.

[0077] exist Figure 11 In the above-described vertical three-finger swipe zoom gesture 1104, it is represented by three upward-facing arrows. These arrows indicate that three touch points (typically the locations where three fingers of the user's finger touch the touchscreen) at locations represented as solid circles move in the same approximately vertical direction to expand the vertical scale of the timeline area, thereby producing... Figure 12 The vertically expanded view is shown. Content group 1102 expands vertically without changing the horizontal (time) scale to allow visible keyframe thumbnails 1202 to be displayed at a correspondingly higher magnification, thus allowing the user to see these keyframe thumbnails more clearly. The aspect ratio of the keyframe thumbnails 1202 remains unchanged, so that these keyframe thumbnails are not distorted.

[0078] Conversely, by performing a vertical zoom gesture in the opposite direction (swiping down instead of up), you can perform actions such as... Figure 13 The vertical shrinking gesture 1302 shown is used to reduce the vertical scale, thereby producing Figure 14 The view shown. Similar to other zoom gestures described in this article, the UI responds to gestures by dynamically and in real-time updating the displayed view of the timeline area during the gesture, allowing users to dynamically adjust their gestures as they perform them to quickly locate and focus on the desired content item at the desired point in time along the project timeline.

[0079] Timeline Navigation: Single Navigation When working with multimedia projects, users typically only need to scroll or move horizontally or vertically along the timeline to locate and view different parts of the project. To address this need, the multimedia editing system described in this paper provides a single-finger drag gesture to dynamically drag (i.e., translate in space) the displayed view in any desired direction, thereby dynamically adjusting the displayed view in real time without changing the zoom level or magnification. For example, Figure 15An example single-finger drag gesture 1502 is shown, in which the user's finger or stylus moves in a roughly southwest direction while touching the screen, thereby producing... Figure 16 The system shows a panning view. However, it follows the acceleration of the dragging gesture, moving the view quickly or slowly depending on the speed of the user's gesture. Unlike existing multimedia editing systems, dragging is not limited to the horizontal or vertical direction but can be performed in any direction, allowing for seamless movement in both horizontal and vertical directions simultaneously.

[0080] Timeline navigation: Double-click to zoom Figure 17 The timeline area is shown in a fully zoomed-out state, allowing users to view the entire timeline area and all content within the project. However, at this zoom level, it's difficult to distinguish or interact with finer details such as individual keyframes and thumbnail previews. To address this, users can perform a double-tap gesture to zoom in on the timeline view and quickly focus on a specific content item at the location of the double-tap.

[0081] For example, Figure 17 The concentric circles shown represent a double-click gesture 1702 performed at the corresponding desired time point of the desired content item 1704, thereby causing the system to display... Figure 18 The view shown. The corresponding portion of the timeline at the gesture touch point is centered in the timeline at a very high zoom level, allowing the user to view and interact with individual frames, such as frame-by-frame animation. Subsequently, the user can perform another double-tap gesture 1802 (as shown). Figure 18 As shown), to zoom out to the previous zoom level. However, the gesture is also anchored to the touch point location, so it will zoom out where the touch point location is centered in the displayed view of the timeline (as shown). Figure 19 (as shown), this may differ from (in this example) Figure 17 (The original view.)

[0082] Timeline navigation: Swipe back to play Figure 20 The timeline area is shown in full zoom, displaying only a portion of the item, with the playhead 118 positioned near the center of the item. To quickly resume playback from the beginning of the item, the user can perform a quick 'flick backward' gesture (as illustrated by the positions of two overlapping hands) by touching the playhead 118 and quickly moving it to the left (towards the beginning of the item) or by 'flicking' the playhead.

[0083] Therefore, in Figure 20In the image, the backward flick gesture 2002 is represented by two overlapping hand outlines shown rotating relative to each other. The hand icon with lower opacity (finger pointing upward) indicates the user's initial touch position on the screen, while the hand icon with higher opacity indicates the directional movement of the flick gesture. Thus, the user touches the playhead and quickly flicks it to the left, towards the beginning of the item.

[0084] In response to the flick gesture 2002 described above, the multimedia editing system moves the playhead 118 to the very beginning of the project timeline (e.g., Figure 21 (As shown), and automatically starts replaying the project from the beginning, following the user's playback settings. If the view is zoomed in when gesture 2002 is initiated, the view will also zoom out to display the entire project in the timeline area.

[0085] Timeline selection choose Figure 22 A timeline area displaying various content items of a multimedia project is shown. In some cases, for efficiency, users may wish to perform one or more actions on multiple content items simultaneously. To facilitate these actions, users can invoke the system's multi-select input mode by selecting (touching) the edit mode icon 2202, and then use their stylus to perform selection gestures to freely select the desired content item displayed in the timeline area. When the system is in this mode, a selection toolbar 2204 is displayed across the top of the touchscreen, with the mode name shown on the left and interactive controls on the right, providing access to the corresponding settings and options.

[0086] In the accompanying diagram, pencil icon 2206 indicates the position where the user's stylus touches the device's touchscreen, and dashed line 2208 indicates the movement of the stylus touch point on the touchscreen when the user performs a stylus gesture.

[0087] Figure 22 The selection gesture 2210, as described above, is shown. This gesture is represented by a dotted line passing through three content items 2212, 2214, and 2216, indicating that these items 2212, 2214, and 2216 will be included in the user's new selection. Typically, a user can select a content item by dragging their stylus through each of the multiple content items or by circling (partially or completely) the content item.

[0088] As a result of the selection gesture 2210 described above, three content items 2212, 2214, and 2216 were selected (as indicated by highlighting the border of each selected item), thus allowing batch actions to be performed on these three content items, such as... Figure 23 The actions listed in the pop-up menu 2302 shown.

[0089] In addition to content items, selection gestures can also be performed on tracks and keyframes 2402, such as... Figure 24 As shown. Users can select a type via pop-up menu 2502 (e.g. Figure 25 (As shown) to determine the type of asset to be selected via subsequent selection gestures.

[0090] Cancel selection Similarly, one or more content items can be removed from the active selection by performing a deselect gesture to freely remove one or more content items from the active selection. For example, Figure 26 The timeline area shown includes two content items 2602 and 2604 in the active selection. In selection mode, the user can use their stylus to 'drag' over one of the selected content items, as indicated by a dotted line 2606 extending through content item 2602, to indicate that item 2602 should be removed from the active selection. Alternatively, the user can circle (fully or partially) the selected content item 2602 to deselect it, as shown in the image. Figure 27 As shown. Regardless of the type of deselection gesture performed, the result is always the removal of the corresponding item from the active selection (e.g., Figure 28 As shown in the figure, this content item will not be affected by any subsequent batch actions performed on the activity selection.

[0091] Split In some cases, users may wish to split (i.e., divide) multiple content items sequentially. To facilitate these actions, users can enter selection mode by touching the selection tool icon 2202 as described above, and then use their stylus to perform a splitting gesture to divide a piece of content. In the described embodiment, the splitting gesture is performed by moving the touch point along a vertical line through each content item to be split to simulate the physical movement of slicing or dividing.

[0092] Throughout this specification, unless the context otherwise indicates, references to directions of movement (such as vertical, horizontal, etc.) should be understood to cover approximations of these directions, as would be expected from a user manually performing such movement on a touchscreen display. In many instances, multimedia editing systems use context and shape detection to identify the intended gesture from an approximation of the intended direction of movement.

[0093] For example, Figure 29The diagram shows a view of a timeline area with various content items, and a splitting gesture 2902 for simultaneously splitting two content items 2904 and 2906. In this diagram, a clock icon 2908, shown adjacent to a pencil icon, indicates that the user needs to continue gripping the stylus downwards at that position of the gesture to invoke the next step of the interaction. Figure 29 In the diagram, the split gesture 2902 is represented by a roughly vertically drawn dotted line 2910 passing through content items 2904 and 2906, indicating the time and position at which content items 2904 and 2906 are to be split. After executing the split gesture 2902, the user can continue to hold their stylus on the screen and add drag gestures 3002 (such as...). Figure 30 As shown), the precise timing of the split can be adjusted. Thus, users can initiate a split and fine-tune the timing of the split in a smooth, composite interaction by moving their stylus vertically (2902) and then horizontally along the time (horizontal) direction without removing their stylus from the screen.

[0094] As a result of the splitting gestures 2902 and 3002 described above, content items 2904 and 2906 are split or divided into two corresponding parts 3102, 3104, 3106, and 3108, respectively, as follows: Figure 31 As shown. In selection mode, multiple split gestures can be executed sequentially for rapid editing.

[0095] delete In some situations, users may wish to delete multiple items consecutively. To facilitate multiple deletions, users can enter selection mode and perform deletion gestures using their stylus, thus freely deleting multiple items. In the described embodiment, the deletion gesture takes the form of a wavy or zigzag motion, mimicking the traditional motion of 'scribbling' or erasing on paper.

[0096] For example, Figure 32 A deletion gesture 3202 for deleting various content items 3204 is shown. The deletion gesture 3202 is represented by a dotted line 3206 drawn in a jagged pattern through the content item to be deleted. As a result of the deletion gesture 3202 described above, the deleted content item 3204 is no longer displayed, such as... Figure 33 As shown. When the selection mode is active, multiple delete gestures can be executed consecutively, enabling rapid editing.

[0097] Timeline grouping As mentioned above, in some cases, users may want to group multiple content items so that they can be processed uniformly as if they were single items. To group items, users can enter selection mode, freely select content items using their stylus, and then group the selected items via a pop-up context menu.

[0098] For example, Figure 34 The timeline view shows track group 3402 in its expanded or open state. This timeline view displays a parent track 3404 providing a group name (in this example, simply the text "Group") 3406 and a shaded border. Below this parent track, group member (child) tracks 3408 are displayed vertically stacked, level with (i.e., horizontally aligned) the parent track 3404 within group 3402, as the horizontal axis represents timing. Groups that are closed or minimized display a full thumbnail preview of the group's contents, while open groups show smaller thumbnail previews, which helps to immediately identify which groups the content is nested within.

[0099] Advanced Playback Header Figure 35 A view showing a timeline area with various content items and a playhead 118 is presented. The playhead 118 acts as a cursor, and a cursor icon 120 is displayed directly below the active content item 3502, at the corresponding time position within that item 3502, as indicated by the vertical white line 122 of the playhead (displayed on the item) being horizontally aligned with the timeline scale 111, and more precisely by highlighting the nearest tick mark on the timeline scale 111. The user can interact with the playhead 118 to perform actions on the content item at the corresponding time position within the corresponding content item 3502.

[0100] In the accompanying drawings, the outline of hand 3504 indicates the touch position of the user's stylus or finger. Tapping the playhead 118 (e.g., with a finger or stylus) causes the system to open the context menu 3602 on the timeline (e.g., ...). Figure 36 As shown), users can use this context menu to instruct the system to perform actions related to this type of content item, such as adding dynamic keyframes 3604 and filtering keyframes 3606, and editing content 3608. Keyframes are added via the playback head 118 (e.g., by applying an application such as...). Figure 37 The Gaussian blur (3702) shown will also update the timeline to display the corresponding keyframe (3704). This allows for quick and precise keyframe setup, animation creation, and editing.

[0101] Loop to screen playback In some situations, users may want to repeatedly view replays of specific sections of the timeline. This is allowed when the "Loop to Screen" playback setting is enabled in the application settings (this setting is enabled by default).

[0102] When this setting is active and the system is in playback mode (e.g.) Figure 38 As shown), only the (time) portion of the item currently displayed in the timeline area plays repeatedly on stage 202. Therefore, the start and end time positions of the playback are the time positions shown on the timeline at 3802 on the left side and 3804 on the right side of the touchscreen. Furthermore, the playback is dynamically updated in real time to reflect any changes in the timeline area. For example, if the user uses the timeline navigation gestures described above to display different time portions of an item during loop playback, the playback automatically updates to play only what is displayed in that different time portion of the timeline area 104 without pausing. Based on the item boundaries displayed in the timeline area, the playback will continue to loop or repeat until the user stops playback. This provides users with a quick and easy way to check the results of their changes to smaller or larger time portions of their items, easily change focus, and avoid resetting their playback settings between changes.

[0103] Drawing Timeline View Figure 39 This is a screenshot showing the standard default appearance of the GUI when editing a project. Since users can both composite and edit animations in the timeline area and display individual frames on the stage, more drawing and painting space is typically required. In this case, the user can use the handle 3902 on the main toolbar 106 to enter the drawing timeline view.

[0104] To use the controller 3902, the user performs a downward drag gesture 3904 by touching the position of the controller 3902 on the screen and dragging the touch point downward from that position. If a speed threshold is reached during gesture 3904, the timeline area rapidly enters or transforms into a compact timeline window 4002, thereby allowing the stage 202 to expand to occupy almost the entire touchscreen, with the compact timeline window 4002 hovering above the touchscreen, as... Figure 40 As shown. A drawing and painting toolbar 4004 is displayed at the top of the screen, including tool controls on the right for accessing different drawing and painting tools and controls on the left for accessing different drawing and painting modes. Additional controls labeled "Cancel" and "Done" are provided to allow users to cancel or complete actions respectively.

[0105] The drawing timeline view provides a simulation of traditional unit animation, where users can quickly flip between drawings to see the transitions between them. To perform this flip action, the user performs the following... Figure 40 The swipe gesture shown allows you to swipe left 4006 on the compact timeline window to advance in time, and swipe right 4008 on the compact timeline window to rewind in time.

[0106] In many cases, users will have 'hold frames,' where a single frame of the animation is extended ('held') for a period longer than one frame. In these cases, the user can perform a double-tap gesture 4010, 4012 on the compact timeline 4002 to advance to the next different frame (i.e., the next frame in sequence different from the current frame). Figure 40 As shown, the user can perform the action on the left side 4010 of the compact timeline to quickly advance to a previous different frame (i.e., rewind in time), or on the right side 4012 of the compact timeline 4002 to advance to the next different frame (i.e., advance in time).

[0107] This action can also be performed by quickly executing the 'flick' swipe gesture 4014, 4015 in either direction (also... Figure 40 (As shown in the diagram), to advance to the next different keyframe. These gestures are invoked when the touch point speed reaches or exceeds a threshold when the user removes it from the screen. The user can perform a flick gesture, flicking right 4014 on the right side of the compact timeline 4002 to rewind time to the next different keyframe, and flicking left 4012 on the left side of the compact timeline 4002 to advance time to the next different frame.

[0108] Otherwise, the compact timeline window 4002 provides only a reduced set of functions for any track in the timeline to ensure that the size of the compact timeline window 4002 is as small as possible. These limited functions include inserting frames (via control 4016) and deleting frames via the frame context menu.

[0109] To return to the full-size timeline area 104, the user performs the same downward drag gesture as previously described from the handle 4018 near the top of the compact timeline window 4002.

[0110] Timeline color markers To facilitate the identification of different content items on the timeline, the multimedia editing system provides the ability to use color markings on any content item, making it easy to identify the item and visually distinguish it from other content items.

[0111] Figure 41Example items using color-coded markers in the timeline area are shown. At this zoom level, the name of each color-coded content item in the timeline area is displayed as text as a shape 4102 filled with the corresponding marker color. Thus, the color-coded markers highlight the names of the content items, visually distinguishing them without adding visual clutter. When the user zooms out sufficiently (e.g., to view their entire item), the names are no longer clearly visible, so the color-coded content items are displayed as corresponding colored horizontal lines 4202 indicating their respective time positions and durations. Therefore, each color marker in the color-coded markers visually highlights the entire content item on the timeline, such as... Figure 42 As shown. Without color coding, it can be difficult to identify different content items at this zoom level in complex projects, especially when utilizing effects and transparency. By adding color coding, users can easily identify different content items, allowing them to choose their own organizational color scheme.

[0112] Keyframe Track Figure 43 The timeline area with content item 4302 containing keyframes is shown. Users can identify the keyframes on content item 4302 because they are represented by mutually spaced thumbnails 4304 and 4306 displayed at corresponding different time positions along the horizontal timeline. In this view, users can easily view and delete their keyframes, but they may also need to edit keyframes they have previously created. To do this, the user simply taps the corresponding keyframe; for example, tapping the rightmost keyframe 4306.

[0113] When a keyframe is tapped, all relevant keyframe tracks expand below the corresponding content item, such as... Figure 44 As shown. In Figure 44 In the example shown, only a single keyframe track 4402 exists, involving one or more variations in a set of parameters consisting of hue, saturation, and brightness (uniformly represented by the label "HSB"). The starting and ending values ​​of these parameters can be viewed and changed by selecting the keyframe icon 4404 displayed in the keyframe track at a time position aligned with the corresponding start keyframe 4304 and end keyframe 4306. Specifically, the user can adjust the keyframe parameters of the start or end keyframe by tapping the corresponding keyframe icon in keyframe icons 4404 or 4406.

[0114] If users want to view or access all available parameter values ​​for this keyframe type, they simply tap the keyframe track name, which in this example is the text "HSB" 4408. After tapping, the keyframe track expands further vertically to show each editable parameter value for that keyframe type, such as... Figure 45 As shown. Therefore, in the example shown, the HSB keyframe track has three parameters: hue, saturation, and brightness, but it is currently configured to only change the hue parameter value between endpoint keyframes (in this example, this is achieved by easing out).

[0115] Figure 45 The expanded view can be exited by tapping elsewhere in timeline 4504, or by tapping the keyframe thumbnails 4304 and 4306 on the corresponding content item 302 in 4506, returning to the previous view. Figure 44 The view.

[0116] The hierarchical structure of keyframe tracks allows users to achieve different purposes using different extension levels. The first level ( Figure 44 This displays a summary of keyframes on the user's track. Because less content is displayed on the screen at this level, it's easier to quickly navigate to or compare with other content items. This allows users to easily find what they're looking for and perform basic actions such as re-timing keyframes and editing individual values. In the second level, which expands into the hierarchy... Figure 45 After that, users can edit or adjust all individual keyframe values ​​associated with that keyframe type. This allows them to perform precise adjustments (e.g., editing the translation x value of a Move & Scale keyframe individually).

[0117] Users may also want to adjust the easing of their keyframes 4304 and 4306 so that the animation transitions differently between keyframes. To do this, they tap the easing title 4502 between the two keyframes (as shown in the image). Figure 45 As shown), the system displays a easing pop-up menu 4602 (as shown). Figure 46 As shown in the figure, this allows users to adjust the easing value.

[0118] Technical challenges While in hindsight the ability to dynamically and simultaneously zoom and update the displayed view of the interactive timeline area in two spatial dimensions in real time via touch gestures (and during touch gestures), as well as the ability to drag in any direction, may seem conceptually simple, its actual implementation required solving significant technical problems.

[0119] Specifically, the simultaneous requirement for high-quality vector graphics—namely, maintaining a fixed screen-space scaling of line / stroke styles during zooming—prevents the use of image-based caching to accelerate timeline rendering (whether by caching tiles or individual timeline elements). Therefore, image caching is limited to thumbnails of preview images of content items within the timeline. Allowing vertical or dual-axis zoom also requires that image thumbnails (previews of content items) have dynamically varying sizes that change with the zoom level, ensuring that the thumbnail previews always have a pixel density similar to the screen.

[0120] In the worst-case scenario (when the cache is empty, which typically occurs during the initial load of the timeline or during major changes), heavy caching as a performance strategy is not very effective. Worst-case performance is an important performance attribute when optimizing interactive systems. This system provides a high-performance interactive timeline that uses only caching of project media data (i.e., content items and tracks), rather than caching of the rendered timeline.

[0121] The described system provides a high-performance pipeline that, with hardware support, renders the timeline from the semantic definition of the project (rather than the obtained layout / raster information) to the screen in less than 1 / 120th of a second (i.e., 8.3 ms), or in less than 1 / 120th of a second (i.e., 16.6 ms) without hardware support. This high performance is achieved through an efficient memory data format, simple linear data structures, object visibility culling, efficient batch generation of render data (in layers), and generating render data in a GPU-friendly format.

[0122] Figure 47 This is a block diagram of the system of the described embodiment 4700, in which the multimedia editing process described herein is implemented as programming instructions for a software application 4800 installed on an Apple iPad™ touchscreen computing device.

[0123] Figure 48This is a schematic diagram illustrating the application architecture of the described embodiment. User interface 4802 represents content displayed to and directly interacted with by the user (including the state of interface elements) and receives input from the system user. Logic component 4804 handles user interactions that change items. Real-time engine 4806 determines the state of the item and renders that state for the user within real-time requirements. Animation core 4808 processes, interprets, and computes multimedia works at any given time. Storage component 4810 provides transactional disk-supported storage for each item via the host operating system's file system API 4812, storing it as a file that can potentially contain the entire history of the item and all its required data in a single file. Resource scheduler 4814 manages limited memory resources and allocates them to other components of the system. Audio engine 4816 decodes and processes audio for both real-time playback and offline export. Video engine 4818 decodes and processes video for input and output. Rendering engine 4820 provides image manipulation as image content stored in sparse memory and renders that image content via rendering graph / compositer component 4824. The drawing component 4822 uses complex brush parameters to render drawn strokes in real time from user input. The compositor / rendering component 4824 interprets multimedia projects built by the user and efficiently converts them into a format suitable for rendering purposes. The Metal API 4826 is a low-level interface from the host system to the GPU, used for GPU computation and displaying graphics on the screen. The Metal Abstraction Layer 4828 provides low-level wrappers on top of the Metal API 4826, which are more convenient and used by other components of the system. The Video Toolbox API 4830 is a low-level interface from the host system for hardware acceleration of video decompression and compression. The Audio Toolbox API is a low-level interface from the host system to audio hardware.

[0124] Using the Apple iPad as the hardware platform, with a display refresh rate of 120 Hz, means it needs to re-render every 8.3 ms. Multimedia projects (such as animations) are typically large and potentially involve a large amount of resources, as well as a large amount of smaller data, keyframes, values ​​and details of the content to be rendered, and the time required for rendering.

[0125] The inventors determined that rendering the timeline region as described herein requires real-time compositing and rendering of both vector graphics (icons, outlines, gradients, etc.) and bitmap graphics (thumbnail images of content items) on the GPU. Because the timeline region is also an interactive user interface, it needs to be rendered at an interactive frame rate (ideally 120 Hz on modern hardware) to provide a satisfactory user experience.

[0126] Furthermore, the requirements for both single-axis (horizontal and vertical) dynamic scaling and uniform dual-axis dynamic scaling present unique technical challenges and pressures to timeline rendering, as this makes it impossible to use synthesized image caching during interactive periods as a method to achieve high performance.

[0127] The requirement for high-quality vector graphics (lines, strokes, etc.) also limits the ability to use composite image caching. When a user zooms in or out on the timeline area, shapes with 1-pixel strokes need to be re-rendered in real time.

[0128] To meet the requirements described above without caching any final images of the timeline region, the timeline region needs to be dynamically generated each frame. The inventors determined that this imposes significant constraints on data format, data transformation, and caching strategies (caching remains a viable strategy for intermediate formats even if the final image output cannot be cached).

[0129] like Figure 49 As shown, the following data format is used: (i) Storage (e.g., disk) format 4902 (cache); (ii) Packaging format 4904 (cached); (iii) Timeline object format 4906; and (iv) Timeline rendering format 4908 (this is the final geometry used to send to the graphics driver).

[0130] The data sent to the rendering engine 4820 for rendering the timeline region includes attribute data related to: which entities are being displayed and at which points in the project (e.g., exposure / timing data, entity 5 exists in the animation between frames 10 and 25), whether these entities have images and / or sounds, whether these entities are being transformed (e.g., entity 5 includes images and translations), and keyframe data corresponding to the attributes of these entities (e.g., there are keyframes at frames 1, 4, and 5, entity #5 corresponds to "translation x").

[0131] Storage format 4902 Storing data on disk is not ideal for direct use in compositing multimedia works or timeline regions. This system uses storage format 4902, which prioritizes both efficient disk use (small file size) and fast storage / retrieval. However, if data is stored in a form ideally suited for rendering (a packed and dense format), any change to the data will result in a significant amplification of disk changes, as the stored information will need to be repacked. For example, minor changes (e.g., updating a single keyframe) will require repackaging all keyframes of the changed entity, and potentially all keyframe data. To avoid this difficulty, the system uses a storage format that allows users to make small changes to their projects while only requiring small changes to the data stored on disk.

[0132] Specifically, the multimedia editing system stores the data as a B-tree in a storage-friendly format within an ordered key-value store. In the described embodiment, the data is stored in a standard key-value database; however, in other embodiments, the data may also be stored in SQLite or any other suitable in-memory database or ordered key-value store (e.g., RocksDB / LMDB). Figure 49 It is a high-level flowchart of the rendering process, illustrating the sequence of data formats from disk to rendering engine 4820.

[0133] Figure 50 This diagram illustrates the different types of data objects created by a multimedia editing system and their relationships. Data objects represent movie item 5002, content item 5004, media (image 5006, audio 5008, and video 5010) items, keyframe track 5012, and track 5016. As indicated by the opposing arrows linking content item 5004 and track 5014, track 5014 and content item 5004 have a recursive relationship; that is, track 5014 can have content 5004, and content 5004 can also have track 5014. This allows content item 5004 to also function as a grouping structure.

[0134] Content item 5004 is bound / exposed in a multimedia project for a period of time and may have animation attribute data, media data, and bound child entities 5004. When this data is stored on disk, the host operating system (iOS in the described embodiment) will typically provide automatic caching of the disk data.

[0135] Packaging format 4904 (cached) Packaging format 4904 is designed to be compact and read as quickly as possible. In projects such as animation, artists typically only modify a small amount of data at a time. Therefore, caching animation data remains a viable strategy.

[0136] The system stores data in packaged form according to data objects. The term "package" means that data is stored linearly / contiguously in memory within the scope of the cached object, which is a necessary requirement to meet the performance targets of the real-time interactive timeline region.

[0137] Packaging format 4904 differs from common storage-friendly formats, and therefore requires conversion between formats 4902 and 4904 unless storage format 4902 is also packaged. If modifications to storage format 4904 do not require quick or efficient execution, then packaging the storage format and thus eliminating the need for both formats is a viable approach.

[0138] The code below—written in the open-source Swift programming language (originally developed by Apple)—illustrates the packaging data structure used for track and content objects.

[0139]

[0140] Each bound content item's MovieFrameRange represents the duration of the content item's exposure in the animation. This example uses absolute time, but the parent content's local time could also be used. Absolute time is simpler from an animation perspective, but using local time generally requires fewer changes to store modifications.

[0141] Timeline information is stored in the cache in package format 4904 and is made available for the timeline layout process 4912 as quickly as possible.

[0142] Elements in the packaged cache format are stored in a size-bounded hash-based cache using identifiers (e.g., each record corresponds to a simple UInt64), and the cache size is kept bounded using standard eviction methods such as LRU. Any of a variety of methods known to those skilled in the art can be used to ensure that only the latest version of the data is cached. For example, cache invalidation can be implemented by attaching the cache to the storage system (then a push method allows the storage layer to invalidate cached elements when the underlying storage version changes), or alternatively, the identifier can be supplemented by a version number (e.g., (id: Int, version: Int)) that is incremented by the storage system as changes are made, so that a cache miss naturally occurs when the data in the storage layer changes.

[0143] For example, the StorageCache structure can be defined as follows:

[0144] Here, "RootContentId" represents the animation itself, which is represented by an instance of the PackedStorageContent structure defined above, while "mediaReferenceId" is a conceptual reference indicating that the timeline content will have the underlying media displayed in the output animation. This can include, but is not limited to, bitmap images, vector graphics, video, or audio.

[0145] The cache is constructed and keyed based on the identity of the data objects. In the described embodiment, the key is a combination of two unsigned 64-bit integers: one integer is used for the identity, and the other integer is used for the object's "generation" (i.e., version). In response to user modifications, the generation of the corresponding data object(s) increments. This can lead to cache invalidation on reads because the cache cannot provide the latest generation of the object. Therefore, the system reads the object from storage and updates the cache accordingly (using a pull-based approach).

[0146] As mentioned above, proxy indexing is only one suitable cache invalidation strategy; other embodiments may use other pull-based methods (e.g., content addressing) or push-based methods (using only identifiers) or invalidate all caches when modifications occur. All such methods involve trade-offs, but the inventors chose pull-based methods because these methods are generally easier to implement correctly.

[0147] Keyframe track object 5012 stores individual keyframes (values) and associated easing data (e.g., linear or cubic easing in / ease out), stored in chronological order. Media objects 5006, 5008, and 5010 store image data 5006, video data 5010, and audio blob data 5008.

[0148] The storage format for these objects 5002 to 5012 is universal and supports both the generation of rendering timelines and the compositing of animations themselves. Objects 5002 to 5012 are stored in a hash graph keyed by their object identities, where the information (values) of each object is stored in a linear fashion.

[0149] Timeline object format 4906 Timeline objects are stored in Timeline Object Format 4906, which overlays UI / visual / layout information on data objects. This allows timelines to be both rendered (requiring all the dynamic information necessary to draw a visual representation of the timeline) and interactive (requiring information that allows users to interact with the timeline, primarily layout, identity, and metadata information).

[0150] Another requirement for timeline areas is to animate the timeline area itself in response to user actions (such as adding keyframes, adding or deleting entities). The data representation 4906 of a timeline object is the format for performing visual animation on it.

[0151] The properties of content object 5004 include: (i) Identity (id and code, matching the storage cache format); (ii) Binding / exposure range (e.g., from frame 5 to frame 20); (iii) Layout information: relative / absolute boundary rectangle; and (iv) Name (user-specified).

[0152] All timeline object data is stored by object type, with each type stored linearly, as shown below: (i) All items 5004 are stored in a single array and referenced by index; (ii) All tracks 5012 are stored in a single array and referenced by index; and (iii) All keyframes are stored in a single array and referenced by index.

[0153] For example, in the described embodiment, the timeline object is stored as a content object and a track object of an object class, as follows:

[0154] Timeline rendering format 4908 Modern GPUs (including those used in Apple's iPads) perform rendering to hardware frame buffers via programmable shaders. Therefore, rendering is achieved by generating shader commands or instructions that specify which GPU shader to use and its configuration, the data bindings for each shader call, and the raw data to use (mostly geometric data). For vector graphics that are not GPU-friendly, the system offloads the preparation work of vector graphics artifacts to the CPU. These vector graphics are sliced ​​and cached as square textures with different gradients / strokes / icons for pre-rendered vector graphics. These vector graphics are then used as texture-based input for GPU rasterization of traditional texture mapping. Vector rendering is performed at various scales, accessed via multi-level mipmaps, to render accurate vector graphics regardless of scale. Render data is stored vertex-wise.

[0155] The Timeline Rendering Format 4908 requires vector graphics primitives (including text and bitmap images). Format 4908 is processed by the GPU's vertex shader and requires vertex data in addition to descriptors that allow the vertex / fragment shaders to interpret geometry to achieve desired effects (such as applying appropriate colors).

[0156] For example, timeline rendering format 4908 can be defined as follows:

[0157] The rendering formats described above are indicative, as the actual rendering format is specific to a combination of hardware platform, programming language, and graphics library. For Metal API 4826, in addition to a separate buffer for "by object" descriptors, it also includes a separate buffer for vertex data.

[0158] Construction steps Rendering the timeline region involves a transformation sequence of data formats 4902 to 4908 described above, such as... Figure 49 The flowchart is shown and described below.

[0159] Disk format 4902 -> Packed storage format 4904 To display the timeline region, at step 4912, the system needs to generate the layout for the current viewport (i.e., the visible rectangular portion of the complete timeline). If the data object of the object to be displayed is not yet in the packing format cache (i.e., the packing cache is missed), then at step 4910, the relevant data of the object is retrieved from storage (e.g., a key-value store or database), converted from storage format 4902 to packing format 4904, and stored in the packing format cache. To utilize bounded memory for caching, an eviction policy is necessary, and in the described embodiment, an LRU (Least Recently Used) policy is used based on object-based fetching.

[0160] The layout generation at step 4912 involves generating a timeline representation (in timeline object format 4906) of each object from its packed data representation (i.e., in packed format 4904) in the current viewport. In addition to serving as an intermediate format for rendering, the objects in timeline object format 4906 are used for user interaction at step 4918.

[0161] Packaging storage format 4904 -> Timeline object format 4906 Starting with the root entity 5002, which represents the entire multimedia project, the timeline region is rendered using the following information: (i) Project (e.g., film / animation) 5002; this identifies data in packaged storage format 4904; and (ii) For a given viewport (i.e., the portion of the timeline being viewed, as determined by the user’s drag and zoom interactions).

[0162] In some embodiments, rendering may also use the following additional information: (iii) In the context of the previously rendered collection of timeline objects (the previously rendered state is used as a source to generate visual animations of the timeline objects). (iv) Has UI state; and (v) Which entity is the user viewing: a. The position of the playhead within the timeline area; and b. Which entities are expanded by 5004 (entities also act as groups).

[0163] The layout process uses a "Transform" data structure, which contains 2D translation and 2D scaling that together define an affine transformation for viewing at least a portion of the timeline in the current viewport, which is a rectangle representing the visible portion of the timeline being rendered to the touchscreen.

[0164] In the described embodiment, the transformation data structure is defined as follows:

[0165] Using this transformed data structure, the layout process is defined as follows:

[0166] The layout process 4912 generates object data in timeline object format 4906 from the corresponding packaged format data in the package cache. The layout process 4912 is executed recursively from the root content item and from top to bottom. All layout decisions are made during this process, and culling is used to reduce the amount of work to be done. Timeline objects are added to a linear array as a simple way to provide high performance. Culling is used to track objects that need to be drawn and allows for early termination, avoiding rendering any tracks (and their content) that are not needed for the layout or display. However, due to the hierarchical layout method and the fact that layout tracks need to calculate the maximum height of their descendant content, layout calculations are often required for tracks and content that are not visible on the screen.

[0167] Strings (such as content names) are efficiently stored in an append-only array. Performance can be improved by allocating these linear arrays to the same size as they were in previous frames (depending on the temporal similarity of the timeline rendered in a real-time application).

[0168] Transform objects store the translation and scaling of the interface and are used to position all content and tracks in absolute screen space coordinates. Other coordinate systems can be used, but working in screen space is essential when styles also have their own dimensions in screen space.

[0169] Animation is achieved by preserving the timeline of previous renders, comparing the content of the current render timeline with that of previous render timelines by object and by property level, and generating animations when their contents differ. If you want to animate a positional element, you should first perform layout in the local space of its parent content, and then perform animation in the local space before generating an absolute layout (animation performed in the local space is generally preferred for aesthetic reasons).

[0170] Any visual styles obtainable from the timeline object format 4906 are not calculated at this time, but are calculated during the rendering geometry generation process 4914 described below. These visual styles include aesthetic styles such as dashed line spacing, rectangle rounded corners, and colors. Since these visual styles are either constants or can be obtained from attributes, they do not need to be stored separately, thus reducing the storage size of the timeline object.

[0171] The layout process 4912 and the subsequent rendering steps 4914 and 4916 are performed in response to the user touching the touchscreen to interact with the system and in response to changes in project data (including animation and storage changes) that may cause changes to the visible parts of the timeline.

[0172] Rendering geometry generation 4914 After updating the layout at step 4912, the corresponding rendering data for the GPU is generated at step 4914, thereby effectively involving the conversion of object data in timeline object format 4906 to timeline rendering format 4908.

[0173] As described above, the layout process 4912 generates an array of linear timeline objects. The rendering process 4914 involves converting the macro layout of these timeline objects into geometric data that can be rendered by the GPU shader. In addition, the rendering process 4914 also involves generating the layout of any visual elements that do not have interactive behavior (any content that requires interaction is in timeline object format 4906).

[0174] A simple layering strategy is used to determine the rendering order of elements. The described embodiment uses the following three layers: (i) The orbit layer, which displays the orbit background; (ii) A content layer that displays rectangular content; and (iii) Content decoration layer, which displays content color markers on top.

[0175] This layered strategy allows rendering order to be separated from iteration order if needed. This is important because for efficient rendering, storage-centric data is iterated over in the order of the animation's natural hierarchy, but only once. However, when rendering the timeline, the rendering order required to efficiently implement features such as drag and drop (rendering the dragged content item on top of all other content items) differs from the animation's natural order.

[0176] The rendering generation process 4914 is implemented as a preparation function, as shown below:

[0177] As mentioned above, modern GPU rendering uses programmable shaders, so rendering data includes GPU instructions that specify which shader to use and its configuration, data bindings for shader calls, and raw data to use (most of which is geometry data).

[0178] As described above, GPU instructions are generated for each layer and each geometry (rounded rectangle, text, image). However, other embodiments can essentially use any modern high-performance vector graphics rendering method. This might involve pure GPU graphics preparation or a CPU / GPU combined approach (involving pre-preparing vector graphics artifacts for rendering on the CPU, storing these artifacts in textures, and then compositing these textures together on the GPU).

[0179] User interaction 4918 To allow interactive navigation of the timeline using a touchscreen, the system supports touch-based gestures for panning (i.e., dragging), single-axis zooming, and dual-axis zooming of the timeline, as shown below:

[0180] Mutually exclusive gesture selection based on gesture.maxTouches ensures that only one specific gesture is executed at a time and prevents switching back to a different gesture, for example, when the user lifts their finger in the middle of a gesture.

[0181] When a touch begins, moves, ends, or is canceled, the touch position is stored in the touch array, and the gesture is updated. After updating the gesture, the timeline is rendered again using the updated transform structure.

[0182] Simple panning of the timeline can be achieved using panning gestures. In the described embodiment, a panning gesture is a gesture that moves across the screen from a single touch location (e.g., by a user touching the screen with only one finger), as shown below:

[0183] Simultaneous panning and dual-axis zooming of the timeline can be achieved using dualAxisZoomGesture, a gesture invoked by the user touching two locations on the screen and moving those touch locations as described above:

[0184] The average position of the touch at the start of the gesture is converted into "timeline space," thereby defining an anchor point on the timeline that is maintained at the average position of the touch during the gesture. Scaling is applied uniformly in proportion to the average distance between touches. However, it will be apparent that in some embodiments, dual-axis scaling may also be applied non-uniformly to scale by different amounts in the horizontal and vertical directions.

[0185] A single-axis scaling gesture is initiated when the minimum length of movement of the three touch positions is 30 print dots, and the single-axis scaling direction is determined based on this movement. Then, relative scaling is performed in the determined dimension relative to the anchor point at the bottom left corner of the timeline.

[0186]

[0187] As described herein, the multimedia editing system and process address the aforementioned limitations of the prior art, and also resolve many associated difficulties arising from the ability to scale the timeline area UI as described. For example, in the work that led to this invention, the inventors developed and tested several prototypes that could adjust the relative height or spacing on the display to allow for clearer visual cues of the project structure, but found that these prototypes caused dizziness and loss of spatial context when content moved around.

[0188] As described in this article, multimedia editing systems and processes offer many advantages, as follows.

[0189] This system is capable of handling projects that are large in both vertical and horizontal dimensions. (For example, a large number of tracks, a large number of content items, and a long duration), even on touchscreen devices with extremely limited size (such as small iPad™ (including iPad Mini™)) or even smartphones.

[0190] This system allows for efficient navigation to any point in the timeline. Existing editing applications typically require multiple (even dozens) zoom gestures to access a specific content item at a particular point on the timeline. For complex projects, this can potentially involve dozens of discrete zooms. Combining horizontal and vertical zooming, while also dragging timeline regions, overcomes this limitation. Furthermore, existing timelines anchor zoom gestures around a static pin at the center of the timeline. In contrast, the described system anchors zooming around the initial gesture touch point, allowing the user to effectively aim using a single gesture.

[0191] User experience when controlling navigation Early prototypes allowed users free navigation, which allowed them to either end the timeline if there was no content in the area or zoom in or out to the point of becoming unusable. Extensive experimentation was required to determine the optimal constraints on movement and zooming to allow users to quickly locate specific items anywhere in the timeline without feeling lost or dizzy. The early prototypes also allowed uneven scaling, such as scaling on the Y-axis and thinning on the X-axis. These felt unpredictable and difficult to control.

[0192] The timeline area is not visually cluttered. Being able to view large projects at once can introduce readability issues. To help users find the content they want, they need unique yet simplified visuals. Removing and reintroducing information at different zoom levels has proven to be an ongoing task, involving hiding elements associated with content items, such as visibility checkboxes and expand buttons. While text descriptions are useful for identifying keyframe tracks, displaying them all at once is counterproductive. To prevent users from being overwhelmed by unnecessary information, the system limits the details shown before the user makes a selection.

[0193] Content or information cannot be offset in horizontal space. The general constraint of the system is that, because the horizontal axis represents time, no content can be displayed horizontally between content items. For each track, the scale consistently represents time.

[0194] Vertical offset is used less frequently. Similarly, when integrating projects, the context of other tracks is crucial for creating transitions or actions based on music effects or sound effects. Introducing information or padding at the top or bottom of tracks reduces the number of tracks that can be displayed on the screen at once and negatively impacts the context.

[0195] Another consideration is avoiding the requirement that each content item have its own track. Many existing multimedia editing applications use this approach to simplify and consolidate possible actions on each track, but this is undesirable on mobile touchscreen devices where vertical display space is limited. Stacking content horizontally as much as possible allows items to be displayed more compactly. Therefore, this system allows audio, video, and groups with different hierarchical structures to all be placed on the same track.

[0196] Groups can have multiple hierarchical levels. As mentioned above, the system allows users to group multiple content items. For example, consider an animation of a bee with several frames showing its wings flapping; these frames can be grouped together so they can move together as a single object. The bee can also be located within another group of foreground objects that receive the same effect. Groups can be nested.

[0197] Due to constraints around the horizontal space, indentation cannot be used to represent hierarchy, and as discussed above, vertical fill is also avoided. Existing timeline-based editing systems introduce side panels that overlay the main timeline. However, this is unsuitable for this system because it cannot synchronize with the uniform scaling of the timeline. This system provides important visual cues such as naming, borders, backgrounds, and minimal vertical fill.

[0198] Groups need context outside the group. One method the inventors explored for reducing vertical complexity and representing nesting in the timeline is to require opening a new view for each group as the user navigates down the group hierarchy. Opening a group replaces the displayed timeline area with only the content within that group, thus breaking the timeline area down into more manageable chunks.

[0199] However, timing often needs to be adjusted relative to other tracks that may be located outside the group. For example, a user might have an animation of a flower whose different parts move differently, but are grouped together so that the effect can be applied uniformly to the entire flower. In the previous example, when a bee lands on a flower, the flower petals might need to bend. Therefore, the user needs to see keyframes of the bee's movement, which are located in another group and on a different track. Thus, the inventors determined that the group content must be visible to any other content.

[0200] This system allows context from grouped content to always be available while reducing the vertical space they occupy. For example, collapsing and expanding groups only introduces additional vertical space when needed.

[0201] Keyframes may occupy a large amount of vertical space In the work that led to this invention, various methods for displaying keyframes in the timeline region were prototyped. Specifically, all tracks associated with parameters containing keyframes are displayed below the content on the timeline when expanded. For most keyframe types, this means that for any additional keyframes added, at least three keyframe tracks will be displayed below the content. In some cases (such as warps), expanding the keyframe tracks will display at least 16 keyframe tracks, one for each adjustable node of the warp. When editing values ​​on the timeline, users typically see three to six keyframe tracks at a time, so having a separate track for each value would take up valuable screen space on the timeline and require additional effort for navigation. In some prototypes, keyframe track titles do not appear on the keyframe tracks, and users need to select a separate keyframe track icon to see the title unless the keyframe track icon is clearly different.

[0202] The described system overcomes these difficulties as follows. When expanding keyframes, each parameter with a keyframe relevant to that moment in the timeline is displayed below the content. Each keyframe track icon can be tapped to edit all relevant values ​​in a popup, giving access to all parameters of the keyframe track. For example, if the Move and Zoom keyframe track icon is selected, the popup allows the user to edit Translation X, Translation Y, Zoom X, Zoom Y, and Rotation without expanding these parameters into separate entities on the timeline. This allows users to quickly and easily edit everything they need without increasing the vertical complexity of the timeline. In some cases, users may want to edit parameters as separate entities, and in such cases, they can expand these parameters by tapping the keyframe track name displayed between keyframes on the track. Selecting the name hides all other tracks and displays a separate track for each parameter in that keyframe type. For example, if the moment has Move and Zoom keyframes as well as a Noise keyframe, selecting Move and Zoom keyframes to expand its track will hide the Noise keyframe track from the view. This allows users to independently control all relevant keyframe track parameters without any other keyframe type to distract them or add visual clutter.

[0203] Keyframes are easy to identify and do not obscure the content. To save vertical space, the inventors explored placing keyframe icons above content thumbnails. However, this created a problem: the icons obscured the thumbnails, making it impossible to identify specific content using these thumbnails, especially when zoomed out. To address this difficulty, the described system represents keyframes as renderings of the content at that moment. For example, given two keyframes transitioning from 0% blur to 10% blur, the first keyframe would show content without blur, while the second keyframe would show content with blur. This visual information makes it easier to find specific keyframes without simultaneously displaying their specific attributes. It also provides interface elements that can be moved around, deleted, etc. Multiple visual representations of the content at that moment can be created on the timeline. Keyframes not only avoid blurring but also provide even more context.

[0204] All keyframes are typically displayed as a single icon: most commonly a diamond with a text description on its left. However, when zoomed out, these icons scatter as small dots on the timeline, hindering quick navigation. To address this difficulty when setting up advanced keyframes, the system uses a specific icon for each animable property, allowing these properties to be easily identified at lower zoom levels. These icons also help locate specific tracks of interest, facilitating the addition of more keyframes of the same type.

[0205] Keyframes can use various easing types Allowing users to set keyframes for multiple properties on the parent content introduces the issue of custom easing. While a keyframe can be represented as an object on the parent content, these properties may have different easing styles, such as linear easing or easing in. This system addresses this by making these easing styles available only on keyframe tracks between relevant keyframes. A popup becomes available when the user places the playhead between two keyframes. Users do not need access to these tracks to create animations; the system will use the default easing, but customization is possible if needed.

[0206] Many modifications will be apparent to those skilled in the art without departing from the scope of the invention.

Claims

1. A computer-implemented method comprising the following steps: A computer system is used to generate a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline area; Within the interactive timeline area, a view of at least a portion of the timeline of a multimedia project is displayed on the touchscreen display. The timeline is a visual representation of the multimedia project and has graphical representations of media content items from which a multimedia work is to be generated. At least some of the graphical representations of the media content items are displayed within the timeline at corresponding vertical positions representing corresponding composition layers, and the horizontal position of each graphical representation within the timeline represents the corresponding time position of the corresponding media item relative to the multimedia work. In response to a user's touch gesture on the interactive timeline area of ​​the touchscreen display, the view displayed in the interactive timeline area is changed, the touch gesture including a zoom touch gesture for changing the zoom level of the displayed view and a pan touch gesture for dragging the displayed view; The step of changing the view includes: at least in response to the zoom touch gesture, dynamically regenerating the view in real time during the corresponding touch gesture and rendering the regenerated view to the touchscreen display, so as to dynamically update the displayed view continuously according to the corresponding touch gesture during the corresponding touch gesture; and The zoom touch gestures include dual-axis zoom touch gestures for concurrently changing the vertical zoom level and the horizontal zoom level of the displayed view.

2. The computer-implemented method as described in claim 1, wherein, The steps for dynamically regenerating the view include: The cache stores objects representing the corresponding media content items visible in the view to be displayed; these objects are instances of a packaged format data structure. It is determined that the objects stored in the cache do not include one or more objects representing corresponding media content items that are visible in the view to be displayed; In response to the determination, media data representing the one or more media content items visible in the view to be displayed is obtained from a non-volatile storage device; The acquired media data is processed to generate one or more corresponding objects as representations of the packaged format data structure, which are corresponding instances of the one or more media content items visible in the view to be displayed; The generated object is stored in the cache; and Remove objects from the cache that represent corresponding media content items that are not visible in the view to be displayed.

3. The computer-implemented method as described in claim 2, wherein, The media data is stored as a B-tree in an ordered key-value store on the non-volatile storage device.

4. The computer-implemented method as described in claim 2 or 3, wherein, The step of dynamically regenerating the view includes: processing the object stored in the cache as an instance of the packaged format data structure to generate a corresponding timeline object in timeline format based on the panning and scaling of the view, thereby determining the absolute screen coordinates of the object.

5. The computer-implemented method as described in claim 4, wherein, The timeline object is generated only when it represents an interactive item.

6. The computer-implemented method of any one of claims 3 to 5, comprising storing the timeline object in a linear array for efficient access.

7. The computer-implemented method of any one of claims 3 to 6, comprising using the timeline object to enable user interaction with media content items.

8. The computer-implemented method according to any one of claims 3 to 7, wherein, The step of dynamically regenerating the view includes: processing the timeline object to generate corresponding rendering geometry data in the form of low-level shader commands and associated data of a graphics processing unit (GPU); and sending the rendering geometry data to the GPU via a low-level interface so that the GPU can efficiently render the view on the touchscreen display.

9. The computer-implemented method of claim 8, comprising determining a visual style based on the attributes of the timeline-formatted object, and generating corresponding rendering geometry data for the determined visual style.

10. The computer-implemented method as described in claim 8 or 9, wherein, The rendering geometry data represents multiple rendering layers, including a track layer for rendering the track background, a content layer above the track layer for rendering a rectangle containing content, and a decoration layer above the content layer for rendering content color markers.

11. The computer-implemented method according to any one of claims 1 to 10, comprising storing the multimedia item as a B-tree in an ordered key-value store on a non-volatile storage medium.

12. The computer-implemented method according to any one of claims 1 to 11, comprising: In response to a non-uniform dual-axis scaling touch gesture in the dual-axis scaling touch gesture, the vertical and horizontal scaling levels of the displayed view are concurrently changed by different amounts as indicated by the non-uniform dual-axis scaling touch gesture during the non-uniform dual-axis scaling touch gesture.

13. The computer-implemented method according to any one of claims 1 to 12, wherein, The regeneration includes one or more graphical representations of the media items in the displayed view that are dynamically changed according to the zoom level.

14. The computer-implemented method according to any one of claims 1 to 13, further comprising: In response to the vertical zoom touch gesture in the zoom touch gesture, only the zoom level of the displayed view in the vertical dimension is changed.

15. The computer-implemented method according to any one of claims 1 to 14, further comprising: In response to the horizontal zoom touch gesture in the zoom touch gesture, only the zoom level of the displayed view in the horizontal dimension is changed.

16. The computer-implemented method according to any one of claims 1 to 15, further comprising: In response to a composite touch gesture in the zoom touch gesture, at least one zoom level of the displayed view is changed as indicated by the zoom gesture portion of the composite touch gesture, while the displayed view is dragged in any direction as indicated by the pan gesture portion of the composite touch gesture.

17. The computer-implemented method according to any one of claims 1 to 16, further comprising: In response to a panning touch gesture, the displayed view can be dragged along any path as indicated by the panning touch gesture.

18. The computer-implemented method of claim 17, wherein, The displayed view is dynamically dragged according to the translation speed of the panning touch gesture on the touchscreen display, such that a fast panning touch gesture causes the displayed view to be dragged further than a slow panning touch gesture.

19. The computer-implemented method according to any one of claims 1 to 18, further comprising: In response to a double-touch gesture at a location within a content item, the zoom level of the displayed view of the timeline is increased to display an independent frame of the content item, and the displayed view is spatially translated so that the frame of the content item corresponding to the time position of the double-touch gesture is centered in the timeline region.

20. The computer-implemented method of claim 19, wherein, The double-touch gesture is a first double-touch gesture and is performed when the displayed view is at a first zoom level, and in response to a subsequent double-touch gesture, the zoom level of the displayed view is reduced to the first zoom level, and the displayed view is dragged so that the position of the subsequent double-touch gesture is centered in the timeline region.

21. The computer-implemented method according to any one of claims 1 to 20, further comprising: A tactile tap gesture on the timeline area causes the multimedia item to play back.

22. The computer-implemented method according to any one of claims 1 to 21, further comprising: In response to a selection gesture on the interactive timeline area of ​​the touchscreen display, a selection is made of multiple displayed media content items as indicated by the selection gesture, enabling one or more actions to be performed uniformly on the selected media items.

23. The computer-implemented method of claim 22, further comprising: In response to a deselect gesture on the interactive timeline area of ​​the touchscreen display, the selection of one or more corresponding items of the selected media content is deselected from the selected media content items, as indicated by the deselect gesture.

24. The computer-implemented method of claim 22 or 23, further comprising defining a group whose members are selected media items, and displaying the group in the timeline area as a thumbnail preview of the content of the group; and switching the display state of the group between an open state and a closed state in response to a corresponding user input; and displaying the group in the open state, while the member content items of the group are displayed vertically and horizontally aligned with each other.

25. The computer-implemented method according to any one of claims 1 to 24, comprising: In response to a split touch gesture on the interactive timeline area of ​​the touchscreen display, each media content item in one or more corresponding media content items is split into a corresponding individual part.

26. The computer-implemented method of claim 25, comprising: In response to a compound split gesture on the interactive timeline area of ​​the touchscreen display, the compound split gesture having a split gesture portion and a subsequent split adjustment gesture portion, splitting each of one or more corresponding media content items into corresponding individual parts as indicated by the split gesture portion, and dynamically adjusting the timing of splitting the one or more media items into individual parts as indicated by the split adjustment gesture portion.

27. The computer-implemented method according to any one of claims 1 to 26, wherein, The timeline area of ​​the graphical user interface includes an interactive playhead portion, the position of which in the timeline area indicates the time position of an active content item, and the method further includes: in response to a corresponding touch gesture on the interactive playhead portion, displaying a list of available actions suitable for the content type of the content item, so as to allow the user to select a desired action from the displayed list.

28. A computer-implemented method according to any one of claims 1 to 27, comprising displaying a portion of the multimedia work in a loop, wherein the start and end times of the portion within the multimedia work correspond to the leftmost and rightmost edges of a displayed view of the timeline region.

29. The computer-implemented method according to any one of claims 1 to 28, wherein, The graphical user interface includes a media display area for displaying a preview of the multimedia work or media content item, and the method further includes: in response to a corresponding touch gesture on the graphical user interface, expanding the media display area to facilitate playback viewing, and replacing the timeline area with a corresponding, smaller timeline window that floats on the expanded media display area.

30. The computer-implemented method of claim 29, further comprising: In response to a corresponding touch gesture on the reduced-size timeline window, the corresponding portion of the animation can be previewed in the media display area by jumping back and forth between corresponding frames of the animation.

31. The computer-implemented method of claim 30, further comprising: In response to a corresponding touch gesture on the reduced-size timeline window, the corresponding portion of the animation is previewed in the media display area by jumping forward and backward in the animation frame sequence to the next frame different from the current frame.

32. The computer-implemented method according to any one of claims 1 to 31, further comprising: In response to a corresponding touch gesture on the timeline area, media items are marked with corresponding display colors to visually identify and distinguish different subsets of media content items.

33. The computer-implemented method of claim 32, comprising determining that the zoom level of the timeline region is greater than a threshold at a first time, and in response, displaying the name of a color-marked content item in a shape filled with the corresponding marker color; and determining that the zoom level of the timeline region is not greater than the threshold at a second time, and in response, displaying the color-marked content item as a corresponding colored horizontal line representing the time position and duration of the color-marked content item.

34. The computer-implemented method according to any one of claims 1 to 33, wherein, The timeline region displays content items with keyframes, wherein time-spaced thumbnail images represent keyframes of the content items with keyframes, and the method further includes: in response to a corresponding touch gesture on the content item with keyframes, displaying a graphical representation of one or more keyframe parameters of the content item with keyframes at a corresponding vertical position in the timeline region.

35. The computer-implemented method of claim 34, further comprising: In response to a corresponding touch gesture on the content item with keyframes, only a graphical representation of the one or more keyframe parameters that change between keyframes is displayed. In response to a further touch gesture on the content item with the keyframe, a graphical representation of all keyframe parameters of the content item with the keyframe is displayed at a corresponding vertical position in the timeline region.

36. A tangible, non-transitory, computer-readable storage medium having a multimedia editing application stored thereon, the multimedia editing application causing the at least one processor of a computing device or system to perform the method as described in any one of claims 1 to 35 when executed by the at least one processor.

37. A system comprising: Touchscreen display; Memory; as well as At least one processor, the at least one processor being configured to perform the method as claimed in any one of claims 1 to 35.

38. A tangible, non-transitory, computer-readable storage medium having a multimedia editing application stored thereon, the multimedia editing application causing the at least one processor of a computing device or system to perform a method comprising the steps of: A computer system is used to generate a graphical user interface for display on a touchscreen display, the graphical user interface including an interactive timeline area; Within the interactive timeline area, a view of at least a portion of the timeline of a multimedia project is displayed on the touchscreen display. The timeline is a visual representation of the multimedia project and has graphical representations of media content items from which a multimedia work is to be generated. At least some of the graphical representations of the media content items are displayed within the timeline at corresponding vertical positions representing corresponding composition layers, and the horizontal position of each graphical representation within the timeline represents the corresponding time position of the corresponding media item relative to the multimedia work. In response to a user's touch gesture on the interactive timeline area of ​​the touchscreen display, the view displayed in the interactive timeline area is changed, the touch gesture including a zoom touch gesture for changing the zoom level of the displayed view and a pan touch gesture for dragging the displayed view; The step of changing the view includes: at least in response to the zoom touch gesture, dynamically regenerating the view in real time during the corresponding touch gesture and rendering the regenerated view to the touchscreen display, so as to dynamically update the displayed view continuously according to the corresponding touch gesture during the corresponding touch gesture; and The zoom touch gestures include dual-axis zoom touch gestures for concurrently changing the vertical zoom level and the horizontal zoom level of the displayed view.

Images