Systems and methods for generating dynamic media

Abstract

To provide a method for generating customized dynamic video.SOLUTION: A method includes: partitioning customized dynamic video's timeline, using a predefined partition scheme, into a series of chunk descriptors; and associating an identifier to each chunk descriptor based on the combination of a set of parameters that represent visual, temporal and other characteristics of the video chunk. The method includes: for each chunk descriptor, querying a cache of previously rendered and encoded video chunks for a video chunk indexed by the identifier matching that of the chunk descriptor; and in the case that a video chunk whose identifier matches the queried identifier is found, retrieving the video chunk from the cache; otherwise, rendering and encoding the video chunk, for the chunk descriptor's timeline interval, and saving the video chunk in the cache, indexed by the identifier.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to the field of dynamic video generation, and in particular to a method for generating personalized dynamic videos. [Background technology]

[0002] A personalized video is a video whose content can be tailored to the end viewer. Similar to creating an email where a template is taken and modified with information specific to the target reader, a personalized video is composed of one or more video template elements that can be combined and tailored to enhance the viewing experience for the intended viewer.

[0003] Personalized video is widely used in marketing, sales, customer care, and social media as a human-centric solution that offers unparalleled levels of customer control and execution. It is also used in industries such as telecommunications, power, banking, insurance, retail, and automotive to support customer acquisition and relationship management.

[0004] Video creation involves two fundamental operations: rendering, which determines the pixel colors for each output frame, and encoding, which converts the video data into a format suitable for storage, possibly compressing the data in the process. These operations are known to be expensive and therefore typically consume the most time during the video generation process.

[0005] Video may also include audio information, and the audio case is considered similarly, thus including a rendering process that determines the amplitude level of each audio data sample, and an encoding process that transforms and potentially compresses the audio data.

[0006] Current technology makes it possible to render and encode personal videos using parallel processing systems, in particular graphical processing units (GPUs) and other related technologies. Nevertheless, because rendering and encoding are fundamental operations, their calculations represent a significant portion of the processing pipeline, which translates into significant costs across a large number of videos. The present invention is directed to avoiding rendering and / or encoding operations, which is made possible by reusing results that have already been rendered and / or encoded.

[0007] In the following discussion, the terms "dynamic" and "personalized" are used interchangeably. The terms "hash," "UID," and "unique identifier" are likewise used interchangeably. Finally, the term "video" can be confusing. In our more technical discussion, "video" refers to visual data that can be distinguished from audio data and other forms of information. Meanwhile, in our higher-level discussion, we intend the more general use to refer to what the end viewer sees: the combination of video and audio (and subtitles, etc.) information taken together as a complete experience. In the latter context, "video" and "media" are used interchangeably. Summary of the Invention

[0008] The present invention provides a method for generating customized dynamic video executed by one or more processors operatively coupled to a non-transitory computer-readable storage device that, when executed, causes the one or more processors to: a. receiving a request to generate a customized dynamic video; b. dividing the timeline of the customized dynamic video into a series of chunk descriptors using a predefined division configuration, each chunk descriptor defined by an interval of the timeline and corresponding to a video chunk of the customized dynamic video; c. for each chunk descriptor, constructing an identifier based on a combination of a set of parameters representing visual, temporal and other characteristics of said video chunk, and associating said identifier with said chunk descriptor; d. for each chunk descriptor, querying a cache of previously rendered and encoded video chunks for video chunks indexed by the identifier that matches the identifier in the chunk descriptor; e. if a video chunk is found whose identifier matches the queried identifier, retrieving the video chunk from the cache; f. Otherwise, rendering and encoding the video chunks for the timeline interval of the chunk descriptor as specified by the request and storing the video chunks indexed by the identifiers in the cache; g. merging the video chunks to generate the customized dynamic video; and storing a module of command code for carrying out the method, the module including:

[0009] According to some embodiments of the present invention, the predefined splitting settings include splitting the video timeline at screen boundaries.

[0010] According to some embodiments of the present invention, the predefined splitting settings include splitting the video timeline at times when one or more dynamic objects enter or exit the video.

[0011] According to some embodiments of the present invention, the predefined division settings include dividing the video timeline into intervals of a fixed size.

[0012] According to some embodiments of the present invention, the predefined division settings include dividing the video timeline along segments of a certain color.

[0013] According to some embodiments of the present invention, the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

[0014] According to some embodiments of the invention, the identifier is a hash value and / or is based on the result of some hash algorithm.

[0015] According to some embodiments of the present invention, the hash calculation is performed by hash computing: a. An element for calculating the duration hash representing the duration of the interval associated with the chunk; b. A component that calculates the static hash that represents the configuration of static elements of the video included in the interval associated with the chunk; c. A component that calculates the dynamic hash that represents the configuration of dynamic elements in the video associated with the chunk; d. An element that calculates the encoder hash, which represents the encoder and its configuration for the video; The present invention includes at least one of the above or a combination thereof.

[0016] According to some embodiments of the invention, the chunks contain only parts of video frames.

[0017] According to some embodiments of the invention, the chunk descriptor stores intermediate values ​​used to construct the final identifier.

[0018] According to some embodiments of the present invention, static elements of two or more video chunks that are identified as identical based on static hash values ​​are generated only once and used to render the two or more video chunks.

[0019] According to some embodiments of the present invention, two or more dynamic videos are generated simultaneously, and the chunks with identical static elements belong to different output videos.

[0020] According to some embodiments of the present invention, the chunks with identical static elements belong to the same output video.

[0021] According to some embodiments of the present invention, the video timeline consists of single frames and the output video consists of still images.

[0022] The present invention provides a method for generating customized dynamic audio executed by one or more processors operatively coupled to a non-transitory computer-readable storage device that, when executed, causes the one or more processors to: e. receiving a request to generate customized dynamic audio; f. dividing the timeline of the customized dynamic audio into a series of chunk descriptors using a predefined division configuration, each chunk descriptor defined by an interval of the timeline and corresponding to an audio chunk of the customized dynamic audio; g. for each chunk descriptor, constructing an identifier based on a combination of a set of parameters representing visual, temporal and other characteristics of said video chunk, and associating said identifier with said chunk descriptor; h. for each chunk descriptor, querying a cache of previously rendered and encoded audio chunks for audio chunks indexed by an identifier that matches an identifier in said chunk descriptor; i. if an audio chunk is found whose identifier matches the queried identifier, retrieving the audio chunk from the cache; j. Otherwise, rendering and encoding the audio chunks for the timeline interval of the chunk descriptor as specified by the request and storing the audio chunks indexed by the identifiers in the cache; k. merging the audio chunks to generate the customized dynamic audio; and storing a module of command code for carrying out the method, the module including:

[0023] According to some embodiments of the present invention, the predefined splitting settings include splitting the audio timeline at screen boundaries.

[0024] According to some embodiments of the present invention, the predefined splitting configuration comprises splitting the audio timeline at times when one or more dynamic audio elements start or stop playing.

[0025] According to some embodiments of the present invention, the predefined division settings include dividing the audio timeline into intervals of a fixed size.

[0026] According to some embodiments of the present invention, the predefined splitting settings include splitting the audio timeline along silence segments.

[0027] According to some embodiments of the present invention, the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

[0028] According to some embodiments of the invention, the identifier is a hash value and / or is based on the result of some hash algorithm.

[0029] According to some embodiments of the present invention, the hash calculation is performed by hash computing: l. an element for calculating the duration hash representing the duration of the interval associated with the chunk; m. an element for calculating the static hash representing the configuration of static elements of the audio included in the interval associated with the chunk; n. An element that calculates the dynamic hash that represents the configuration of dynamic elements in the audio associated with the chunk; o. an element that calculates the encoder hash representing the encoder of the audio and its configuration; The present invention includes at least one of the above or some combination thereof.

[0030] According to some embodiments of the invention, the chunk descriptor stores intermediate values ​​used to construct the final identifier.

[0031] According to some embodiments of the present invention, static elements of two or more audio chunks that are identified as identical based on static hash values ​​are generated only once and used to render the two or more audio chunks.

[0032] According to some embodiments of the present invention, two or more dynamic audios are generated simultaneously, and the chunks with the same static elements belong to different output audios.

[0033] According to some embodiments of the present invention, the chunks with identical static elements belong to the same output audio.

[0034] The present invention provides a system for generating customized dynamic video executed by one or more processors operatively coupled to a non-transitory computer-readable storage device, the system comprising: p. a segmentation module that segments a timeline of the requested customized dynamic video into a series of chunk descriptors using a predefined segmentation configuration, the chunk descriptors being defined by intervals of the timeline and corresponding to video chunks of the customized dynamic video; q. constructing an identifier for each chunk descriptor based on a combination of a set of parameters representing visual, temporal and other characteristics of the video chunk, and associating the identifier with the chunk descriptor; for each chunk descriptor, querying a cache of previously rendered and encoded video chunks for video chunks indexed by the identifier that matches the identifier of the chunk descriptor; If a video chunk is found whose identifier matches the queried identifier, retrieving the video chunk from the cache; otherwise, a checking module that renders and encodes the video chunk for the timeline interval of the chunk descriptor as specified by the request and stores the video chunk indexed by the identifier in the cache; r. a merging module that merges the video chunks to generate the customized dynamic video; Includes.

[0035] According to some embodiments of the present invention, the predefined splitting settings include splitting the video timeline at screen boundaries.

[0036] According to some embodiments of the present invention, the predefined splitting settings include splitting the video timeline at times when one or more dynamic objects enter or exit the video.

[0037] According to some embodiments of the present invention, the predefined division settings include dividing the video timeline into intervals of a fixed size.

[0038] According to some embodiments of the present invention, the predefined division settings include dividing the video timeline along segments of a certain color.

[0039] According to some embodiments of the present invention, the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

[0040] According to some embodiments of the invention, the identifier is a hash value and / or is based on the result of some hash algorithm.

[0041] According to some embodiments of the present invention, the hash calculation is performed by hash computing: s. an element for calculating the duration hash representing the duration of the interval associated with the chunk; t. A component that calculates the static hash that represents the configuration of static elements of the video included in the interval associated with the chunk; u. An element that calculates the dynamic hash that represents the configuration of dynamic elements in the video associated with the chunk; v. An element that calculates the encoder hash, which represents the encoder of the video and its configuration; The present invention includes at least one of the above or some combination thereof.

[0042] According to some embodiments of the invention, the chunks contain only parts of video frames.

[0043] According to some embodiments of the invention, the chunk descriptor stores intermediate values ​​used to construct the final identifier.

[0044] According to some embodiments of the present invention, static elements of two or more video chunks that are identified as identical based on static hash values ​​are generated only once and used to render the two or more video chunks.

[0045] According to some embodiments of the present invention, two or more dynamic videos are generated simultaneously, and the chunks with identical static elements belong to different output videos.

[0046] According to some embodiments of the present invention, the chunks with identical static elements belong to the same output video.

[0047] According to some embodiments of the present invention, the video timeline consists of single frames and the output video consists of still images. [Brief explanation of the drawings]

[0048] The present invention will be readily understood from the detailed description of the embodiments thereof as well as the accompanying drawings, in which: [Figure 1] (Prior Art) Shows a dynamic movie timeline in current video rendering technology. The screen can be positioned at any point along the time axis, as well as any z-index (larger z-index corresponds to being closer to the viewer). [Figure 2A] (Prior Art) Represents two screens, each containing one or more videos, images, text, or other assets. Several properties, such as timing (start, offset) and z-index, control the composition of the screens and their constituent layers. Next to each is a sketch showing how the screens are composed of layers of static and dynamic elements. [Figure 2B] (Prior Art) Represents two screens, each containing one or more videos, images, text, or other assets. Several properties, such as timing (start, offset) and z-index, control the composition of the screens and their constituent layers. Next to each is a sketch showing how the screens are composed of layers of static and dynamic elements. [Figure 3] 1 is a block diagram of a dynamic video production and management system according to some embodiments of the present invention. [Figure 3A] FIG. 1 is a diagram of multiple dynamic video generation modules accessing a single cache database according to some embodiments of the present invention. [Figure 4A] FIG. 4 is a flow diagram of a module that divides a video timeline into video chunk descriptors in some embodiments of the present invention. [Figure 4B] FIG. 10 is a flow diagram of a module that divides an audio timeline into audio chunk descriptors according to some embodiments of the present invention. [Figure 5A] FIG. 2 is a diagram of a video chunk identification (hash) module according to some embodiments of the present invention. [Figure 5B] FIG. 2 is a diagram of an audio chunk identification (hashing) module according to some embodiments of the present invention. [Figure 6] FIG. 2 is a diagram of a chunk cache query module according to some embodiments of the present invention. [Figure 7A] FIG. 4 is a flow diagram of an exemplary non-optimized execution of a video rendering module according to some embodiments of the present invention. [Figure 7B] 1 illustrates rendering and encoding events over time for an unoptimized execution of a video rendering module according to some embodiments of the present invention. [Figure 7C] FIG. 10 is a flow diagram of the execution of a video rendering module when single static / multiple dynamic (SSMD) optimization can be utilized, according to some embodiments of the present invention. [Figure 7D] 1 illustrates rendering and encoding events over time for an SSMD optimized execution of a video rendering module according to some embodiments of the present invention. [Figure 8] 1 is a flow diagram of a merging module according to some embodiments of the present invention; [Figure 9A]1 is an example of dividing a timeline into chunk descriptors according to some embodiments of the present invention. [Figure 9B] 9B is another example of dividing a timeline into chunk descriptors, according to some embodiments of the present invention. This timeline is similar to FIG. 9A except for the additional screen in the first screen and the shift in time relative to the remaining screens. [Figure 9C] 9B is another example of dividing a timeline into chunk descriptors, according to some embodiments of the present invention. This timeline is similar to FIG. 9A except for an additional frame near the end and a shift in z-value relative to the other two frames. [Figure 10] 1 is an example of a chunk hashing algorithm according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of elements set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments and to be practiced or carried out in various ways. It is also understood that the terminology used herein is for the purpose of description and should not be regarded as limiting.

[0050] Dynamic Movie Generation: Prior Art. According to current dynamic movie generation technology, as shown in Figure 1 (Prior Art), a movie is composed of one or more views arranged in a collage-like configuration on a timeline, with each view occurring at a specific time, lasting for a specific duration, and located at a specific z-index ("distance" from the viewer). Views can be thought of as templated building blocks. They contain layers, which hold assets such as images, video, text, and audio. Both the containing view and its constituent layers may expose properties that serve as parameters for the creation of the video. For example, a view may expose timing properties such as start time and duration, which control when the view begins on the global timeline and the z-index that governs how much of the view is displayed, or the order in which views are displayed when multiple views appear simultaneously in the video. Similarly, constituent layers may expose properties that govern transparency, asset alignment properties relative to bounding boxes, and especially the assets themselves. By referencing the view building blocks and specifying values ​​for the exposed properties, a video is assembled and generated to be customized for the end viewer. Furthermore, this "video collage" may be specified by a high-level object description format such as JSON or XML.

[0051] Across thousands or even millions of movies, each intended for a different possible end audience, some assets and properties will vary significantly and some will remain the same. Those that remain the same from video to video are called static, and those that change are called dynamic.

[0052] In the example screen shown in Figures 2A and 2B (Prior Art), the background is static, and the text and dog image are dynamic. These dynamic elements are also called "placeholders." Thus, in Figures 2A and 2B (Prior Art), the text and foreground image are placeholders.

[0053] The configuration, layers, and placeholder definitions for each screen are described in a predefined format whose function acts as a template for that screen. The configuration for each unique movie is then specified in some format / method / notation that is convenient for specifying hierarchical data structures, such as JavaScript Object Notation (JSON). Specifically, many aspects of the final video are specified using this simplified format, such as which screens will appear in the video, when they will play in time, where they are located (in z-space), which assets will be used to fill which placeholders within each screen, other placeholder attributes (alignment, font, etc.), etc.

[0054] For a given video campaign, there are typically multiple segments that are identical across thousands or tens of thousands of videos: for example, portions that have only static content, and portions that are dynamic but have placeholder configurations that are identical across the videos. The present invention exploits this assumption to provide an improved and efficient method for generating dynamic video that avoids rendering and / or encoding frame portions of video that have already been rendered and / or encoded.

[0055] Dynamic Movie Generation with Chunk Caching. Figure 3 shows a block diagram of several requesting entities 200 and a dynamic media generator 130 associated with a chunk cache database 120, according to some embodiments of the present invention. A requesting entity sends a video request to the dynamic media generator 130, including a description of the desired composition of the movie. This description may be received in the form of instructions for building a timeline, an object-based description of the timeline (such as a JSON format), or any other format that adequately describes the visual and audio composition of the movie over time.

[0056] The dynamic media generator 130 is comprised of a segmentation module 1320A and / or 1320B, a chunk identification module 1340A and / or 1340B, a chunk cache query module 1360, a chunk rendering module 1380 (A and / or B...), and a merging module 1400. Additionally, the dynamic media generator 130 is associated with a chunk cache database 120. As shown in Figure 3A, multiple dynamic media generator modules 130 can use the same chunk cache database 200.

[0057] Movie Timeline Partitioning. The Partitioning module 1320 divides the timeline described in the request into individual partitions called chunk descriptors. The main information associated with a chunk descriptor is an identifier (see Naming Chunks below) and a reference to a time interval on the global movie timeline. The interval information may take the form of a start and end time, a start time and duration, or an equivalent format. Each chunk descriptor is then used to render and / or encode a chunk of the final video or audio stream.

[0058] The segmentation process operates according to predefined settings designed to optimize the probability of so-called chunk cache hits. A chunk cache hit occurs when an already rendered chunk is found (in the chunk cache database) whose structure is identical to that described by the chunk descriptor and is therefore required by the movie currently being generated. Because video rendering / encoding and audio rendering / encoding are separate processes, different segmentation settings are used for each. However, the same video and audio settings are used across all movies to optimize the probability of chunk cache hits.

[0059] 4A and 4B further detail the segmentation process for video and audio, respectively. In both cases, configuration involves segmenting into chunks of a certain duration, segmenting via certain characteristics of the timeline description such as screen or layer boundaries, or segmenting by other means. For video data, there are at least five options, one or more of which may be combined or combined into the final configuration.

[0060] Option 1: Split based on screen boundaries (step 1326A). Option 2: Splitting based on the interval during which the dynamic layer is visible, or more generally, the interval during which the dynamic layer affects the visible configuration of the chunk (step 1328A). Option 3: Division into intervals of fixed length (step 1340A). Option 4: Segmentation based on blank intervals (ie, intervals with video information of a constant color, eg, all black). Option 5: Interval-based splitting with no dynamic video content.

[0061] In addition to the splitting options, the splitting settings include certain constraints. Constraint 1: The length of the video chunk is greater than the minimum length. Constraint 2: The length of the video chunk is less than the maximum length.

[0062] Below is an example of an algorithm that implements video segmentation settings that incorporates multiple options and constraints. 1. Divide the timeline into intervals that start and end at screen boundaries (option 1). 2. Subdivide each generated interval into intervals of constant size (option 3). If the length of any interval generated in step 1 is not a multiple of the desired constant size, this step generates at least one interval of a smaller size. 3. If any intervals generated in step 2 are smaller than the minimum length, combine these intervals with one of their neighboring intervals.

[0063] In this and similar setups, the splits are invariant to the absolute timing of the movie. This property increases the chances of a cache hit, even if, for example, new screens are added to the beginning of the movie (see the toy example for a concrete example).

[0064] For audio data, there are at least five options, one or more of which may be combined or integrated into the final setting.

[0065] Option 1: Split based on screen boundaries (step 1326B). Option 2: Splitting based on the interval during which the dynamic layer is audible, or more generally, the interval during which the dynamic layer affects the audio composition of the chunk (step 1328B). Option 3: Division into intervals of fixed length (step 1340B). Option 4: Segmentation based on silent intervals (i.e., intervals of silence or very quiet audio information). Option 5: Interval-based splitting with no dynamic audio content.

[0066] In addition to audio splitting options, the splitting settings include certain constraints. Constraint 1: The length of the audio chunk is greater than the minimum length. Constraint 2: The length of the audio chunk is less than the maximum length.

[0067] An audio segmentation algorithm similar to the video segmentation algorithm described above is used to indicate the audio chunk segmentation settings.

[0068] Naming Chunks. The primary technique used in this invention allows for the rapid identification and matching of chunks and / or chunk descriptors based on some or all of their composition. This technique involves generating simple unique identifiers (UIDs) that can be quickly compared for equality. As long as UIDs are configurable, they can take any form. However, in typical embodiments, they are represented as hash values, and for this reason, "UID" and "hash" (or "hash value") are used interchangeably.

[0069] The chunk identification modules 1340 (A and B) calculate a UID for each chunk descriptor, typically in the form of a hash value, using techniques further detailed in Figure 5A (for video) and Figure 5B (for audio). The goal of the UID generation setup is to synthesize a value that is unique enough to distinguish between two similar but non-identical compositions, and generic enough to equate those compositions that appear to be identical.

[0070] A hash value is generated for each chunk. This hash is composed of one or more hash values ​​(or otherwise associated in a one-to-one relationship with a combination thereof), each of which represents some sub-element of the composition. To accurately represent the constituent sub-elements, the hash values ​​may be composed from values ​​found in a high-level movie timeline description (e.g., from a high-level JSON description), from values ​​found in screen descriptions, and / or from other data sources.

[0071] 5A illustrates this process through the burden of the video chunk identification module 1340A, according to some embodiments of the present invention. This figure identifies the calculation of at least one of a plurality of values.

[0072] Static Hash: A hash value representing all static elements and their configuration, optionally including their position on the timeline relative to the chunk's position, and optionally including their ordinal z value, either implicitly or explicitly (step 1344A). Dynamic Hash: A hash value representing all dynamic elements and their configuration, optionally including their position on the timeline relative to the chunk's position, and optionally including their ordinal z value, either implicitly or explicitly (step 1346A). Encoder Hash: A hash value representing the encoder and its configuration, optionally including values ​​such as output resolution, frames per second (fps), codec name (H.264, png, etc.), and codec configuration (step 1348A). Duration Hash: A hash value representing the duration of the timeline interval to which this chunk corresponds (Step 1350A) .

[0073] The ordinal z-values ​​(OZV) mentioned above are a way to encode only the order of pictures, without encoding their absolute or uniform relative positions. They are calculated by taking the index of a picture in an array of pictures sorted by z-value. For example, suppose we have three scenes with the following z-values: Screen A@z=3 Screen B@z=7 Screen C@z=1 The OZV is as follows: OZV(Screen A)=1 OZV(Screen B)=2 OZV(Screen C)=0 Screen C is the rearmost, so it follows screen A and then screen B. Admittedly, this is just one way of encoding the order of the screens, but in practice any such method may actually be used.

[0074] Some or all of the hash values ​​listed above are then combined to generate a "full hash" (1352A), which is then used to identify a chunk by distinguishing its overall composition based on its subcomponents.

[0075] 10 is an example of pseudocode for a video hashing algorithm in some embodiments of the present invention, where the hash() procedure performs an arbitrary hash operation on the input data, and the combine() procedure combines the input hash values ​​into a single value. The combine() procedure is based on a commutative XOR operator if it is invariant to the order of the input values, or a non-commutative operator if it is order-sensitive. Furthermore, "c.duration" is a C-style syntax that captures the "duration" term contained in a structure named "c."

[0076] The overall effect of the procedure in Figure 10 is to fill in the hash values ​​of the four subelements mentioned above and combine them into a complete hash. It is not necessary to store the hash values ​​of the subelements to identify the chunk; however, these values ​​can be used for other purposes; for example, the static hash can be used to perform the SSMD optimization described below.

[0077] The process described above is somewhat focused on identifying video chunks. The same is true for audio. In audio, there is no concept of z-space like there is in video. Therefore, the ordinal z values ​​of the screen are not included in the hash calculation for audio chunks.

[0078] Toy Examples: Having described the segmentation and naming process, some toy examples will help to show how the present invention allows to avoid rendering and encoding chunks of video or audio that have already been rendered and encoded. For clarity and simplicity, and without loss of generality, these examples will be described in terms of the video case.

[0079] 9A, 9B, and 9C show examples of chunk division according to some embodiments of the present invention. Assuming these movies are rendered in a given order, there are several cases where accessing the same chunk cache database can save rendering and encoding time. Chunk names ("C1," "C2," etc.) are maintained between samples to indicate chunk reuse; for example, chunk "C3" in FIG. 9A is identical to its named "C3" in FIGS. 9B and 9C.

[0080] Figure 9A shows the timeline of the first sample movie. It contains four screens located at two different z indices. In this case, the segmentation module divides the timeline into five intervals according to the screen boundaries. The table below identifies the screens associated with each chunk. Within each structure, each screen is associated with a so-called ordinal z value (OZV). OZV is used because, as explained above, only the z ordinal of the screens is important, not the z distance between them. Bold font identifies items that need to be rendered, in this case, all of them.

[0081] Figure 9B shows the timeline of the second sample movie. It is almost identical to the movie from Figure 9A, except that a new screen, S5, is found first; everything else has been time-shifted to start after S5. Chunks C1 through C5 have also been time-shifted. However, because we split at a screen boundary, the split point for the screen is invariant to such a time shift, and they do not need to be re-rendered. Therefore, C1 through C5 can be reused; only C6 needs to be rendered. The table below the timeline indicates in bold which chunks need to be rendered; everything else is found in cache.

[0082] Figure 9C shows the timeline of the third sample movie. Here, screens S2 and S4 are packed in z-space. In addition, there is screen S6, which begins halfway through the duration of S3. This movie looks very different from the previous one, but because we are recording ordinal z-values, not absolute or relative z-values, C2, C3, and C4 are found in the cache from the existing example. Only C7 and C8 need to be rendered.

[0083] Querying the Chunk Cache Database. The chunk cache query module 1360 queries the chunk cache database 120 for the chunk based on the UID calculated by the chunk identification modules 1340A and / or 1340B. If the chunk is not available in the cache (1366), it is rendered and encoded by the rendering module 1380 according to the chunk descriptor and recorded in the chunk cache database 120 with its UID serving as a search key (step 1368A). If the chunk is retrieved from the chunk cache database 120, the entire rendering and encoding step can be skipped (1364). This process is illustrated in FIG. 6.

[0084] Rendering and Encoding Chunks (If Necessary). If a chunk is not found in the chunk cache database during querying, the associated rendering module 1380 (A or B) is activated and instructed to render and encode the chunk based on the information in the chunk descriptor, typically the start and end times relative to the global timeline, along with the screen graph assembled during the initial processing of the request. In the non-optimized case, each chunk is rendered individually, with the rendering module 1380 rendering and encoding all frames or audio samples over the time interval specified in the chunk descriptor and writing everything into the final video or audio chunk. The overall process is shown in Figure 7A, and its execution flow is detailed in Figure 7B. If a chunk contains both static and dynamic data, the rendering process itself can be considered to include at least two sub-steps before encoding: rendering static data and rendering dynamic data.

[0085] Single Static / Multiple Dynamic (SSMD) Optimization. According to some embodiments of the present invention, two or more dynamic videos may be generated simultaneously. When this occurs, some static chunks, or static elements of chunks, from two different videos may be identified as identical by comparing their static hash values. In this case, these chunks or chunk elements are temporarily cached in hardware memory and ultimately generated only once for use with multiple videos.

[0086] In some embodiments, it may be advantageous to implement a so-called single static / multiple dynamic (SSMD) optimization, which takes advantage of the chunk and hash settings described above, but optimizes at the rendering level by only rendering static elements once for use by multiple chunks, instead of looking up rendered and encoded sections of the video in a cache.

[0087] More specifically, SSMD introduces chunk descriptors for chunks whose static elements are identical, even if their dynamic elements differ. These chunk descriptors can be easily compared and grouped by identifying which chunk descriptors have the same static hash value (1370B). Once grouped, they are rendered and encoded as follows: First, a file is created and opened for each output chunk; then, for each frame in the range, for each chunk in the group, the frame is rendered and encoded to the file corresponding to that chunk, and the file is closed.

[0088] Assuming we have a buffer for rendering intermediate static elements, running the rendering process with an outer loop iterating over frames and an inner loop iterating over chunks, the static element buffer does not need to be updated with each successive rendering call, preventing the rendering of static elements for all but the first chunk in a group (1374B). This results in a significant reduction in rendering time. This difference can be seen by comparing the execution flow of the unoptimized case (see Figure 7B) with the execution flow of the SSMD-optimized case (see Figure 7D). In the SSMD case, all but one of the "static" rendering runs between K chunks in a group is omitted per frame.

[0089] Merging All Elements into the Final Movie. Once all of the relevant chunks and metadata are available, the Merger Module 1400 begins generating the desired output. To do so, it starts with a description of all the elements needed to create the final output movie. This description can be obtained from the original request and / or data derived therefrom. It can include one or more lists of video and / or audio chunk descriptors, and / or descriptions of information streams such as subtitles and other metadata. A flow diagram of the operation of the Merger Module in some embodiments of the present invention is shown in FIG. 8.

[0090] The merging module 1400 then writes the output according to the output movie description. If the output movie contains visual information, i.e., for a given list of video chunk descriptors, this module collects or receives all of these chunks from either the chunk cache database 120 or the video rendering module 1380A and copies their contents to the associated stream of output, concatenating the chunks in the process (1402A). The chunks are easy to copy because they have already been encoded via the desired codec. Avoiding re-encoding allows for a very fast chunk merging process. For the audio case (1402B), the process is very similar, except that audio chunks are received or collected from either the chunk cache database 120 or the audio rendering module 1380B.

[0091] As mentioned above, in addition to audio and video data, the output movie description may specify that the final movie should include other streams of information, such as subtitles or other types of metadata (1404). If so, the merging module 1400 receives or collects the associated information and writes the associated streams to the output. The data for these auxiliary streams may come from the request itself or from external sources referenced by the request.

[0092] Once the merge module 1400 has finished formatting the output and writing all of the requested streams to it, the generated output media is complete, and the dynamic media generator 130 notifies the requesting entity of completion.

Claims

1. 1. A method for generating customized dynamic video executed by one or more processors operably coupled to a non-transitory computer-readable storage device, the storage device, when executed, causing the one or more processors to: a. receiving a request to generate a customized dynamic video; b. dividing the timeline of the customized dynamic video into a series of chunk descriptors using a predefined division configuration, each chunk descriptor defined by an interval of the timeline and corresponding to a video chunk of the customized dynamic video; c) for each chunk descriptor, constructing an identifier based on a combination of a set of parameters representing visual or temporal characteristics of said video chunk, and associating said identifier with said chunk descriptor; d. for each chunk descriptor, querying a cache of previously rendered and encoded video chunks for video chunks indexed by the identifier that matches the identifier in the chunk descriptor; e. if a video chunk is found whose identifier matches the queried identifier, retrieving the video chunk from the cache; f. Otherwise, rendering and encoding the video chunks for the timeline interval of the chunk descriptor as specified by the request and storing the video chunks indexed by the identifiers in the cache; g. Merging the video chunks to generate the customized dynamic video; and storing a module of command code for carrying out the method, the module including:

2. The method of claim 1 , wherein the predefined split settings include splitting the customized dynamic video timeline at screen boundaries.

3. The method of claim 1 , wherein the predefined segmentation settings include segmenting the timeline of the customized dynamic video at times when one or more dynamic objects enter or exit the video.

4. The method of claim 1 , wherein the predefined division settings include dividing the customized dynamic video timeline into intervals of a fixed size.

5. The method of claim 1 , wherein the predefined division settings include dividing the timeline of the customized dynamic video along segments of a fixed color.

6. The method of claim 1 , wherein the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

7. The method of claim 1 , wherein the identifier is a hash value and / or is based on the result of some hash algorithm.

8. The hash algorithm is a hash computer's an element that calculates a duration hash representing the duration of the timeline interval associated with the video chunk; b. A component that calculates a static hash that represents the configuration of static elements of the video chunk contained in the timeline interval associated with the video chunk; c. A component that computes a dynamic hash that represents the configuration of dynamic elements in the video associated with the video chunk; d. An element that computes an encoder hash that represents the encoder and its configuration for the video; 8. The method of claim 7, comprising at least one of the following or some combination thereof:

9. The method of claim 8 , wherein the video chunk comprises only a portion of a video frame.

10. The method of claim 1 , wherein the chunk descriptor stores intermediate values ​​used to construct the identifier.

11. 10. The method of claim 8, wherein static elements of two or more video chunks identified as identical based on static hash values ​​are generated only once and used to render the two or more video chunks.

12. The method of claim 11 , wherein two or more dynamic videos are generated simultaneously, and the video chunks having identical static elements belong to different output videos.

13. The method of claim 11 , wherein the video chunks with identical static elements belong to the same output video.

14. 1. A method for generating customized dynamic audio executed by one or more processors operably coupled to a non-transitory computer-readable storage device, the storage device, when executed, causing the one or more processors to: a. receiving a request to generate customized dynamic audio; b. dividing the timeline of the customized dynamic audio into a series of chunk descriptors using a predefined division configuration, each chunk descriptor defined by an interval of the timeline and corresponding to an audio chunk of the customized dynamic audio; c) for each chunk descriptor, constructing an identifier based on a combination of a set of parameters representing visual or temporal properties of said audio chunk, and associating said identifier with said chunk descriptor; d. for each chunk descriptor, querying a cache of previously rendered and encoded audio chunks for audio chunks indexed by the identifier that matches the identifier of the chunk descriptor; e. if an audio chunk is found whose identifier matches the queried identifier, retrieving the audio chunk from the cache; f. Otherwise, rendering and encoding the audio chunks for the timeline interval of the chunk descriptor as specified by the request and storing the audio chunks indexed by the identifiers in the cache; g. Merging the audio chunks to generate the customized dynamic audio; and storing a module of command code for carrying out the method, the module including:

15. The method of claim 14 , wherein the predefined split settings include splitting the timeline of customized dynamic audio at screen boundaries.

16. 15. The method of claim 14, wherein the predefined division settings include dividing the timeline of customized dynamic audio at times when one or more dynamic audio elements start or stop playing.

17. The method of claim 14 , wherein the predefined division settings include dividing the timeline of customized dynamic audio into intervals of a fixed size.

18. The method of claim 14 , wherein the predefined division settings include dividing the timeline of customized dynamic audio along segments of silence.

19. The method of claim 14 , wherein the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

20. The method of claim 14 , wherein the identifier is a hash value and / or is based on the result of some hash algorithm.

21. Hash calculation is hash computing, a. An element that calculates a duration hash that represents the duration of the interval associated with the video chunk; b. A component that calculates a static hash that represents the configuration of audio static elements contained in the timeline interval associated with the chunk; c. A component that computes a dynamic hash representing the configuration of dynamic elements in the audio associated with the chunk; d. An element that calculates an encoder hash that represents the encoder and its configuration of the audio; 21. The method of claim 20, comprising at least one of the following or some combination thereof:

22. 22. The method of claim 21, wherein the chunk descriptor stores intermediate values ​​used to construct the identifier.

23. 22. The method of claim 21, wherein static elements of two or more audio chunks that are identified as identical based on static hash values ​​are generated only once and used to render the two or more audio chunks.

24. 22. The method of claim 21, wherein two or more dynamic audios are generated simultaneously, and the chunks having the same static elements belong to different output audios.

25. The method of claim 21 , wherein the chunks having the same static elements belong to the same output audio.

26. 1. A system for generating customized dynamic video executed by one or more processors operably coupled to a non-transitory computer readable storage device, comprising: a. a segmentation module that segments a timeline of a requested customized dynamic video into a series of chunk descriptors using a predefined segmentation setting, the chunk descriptors being defined by intervals of the timeline and corresponding to video chunks of the customized dynamic video; b. constructing an identifier for each chunk descriptor based on a combination of a set of parameters representing visual or temporal characteristics of the video chunk, and associating the identifier with the chunk descriptor; for each chunk descriptor, querying a cache of previously rendered and encoded video chunks for video chunks indexed by the identifier that matches the identifier of the chunk descriptor; If a video chunk is found whose identifier matches the queried identifier, retrieving the video chunk from the cache; otherwise, a checking module that renders and encodes the video chunk for the timeline interval of the chunk descriptor as specified by the request and stores the video chunk indexed by the identifier in the cache; c. a merging module that merges the video chunks to generate the customized dynamic video; Includes.

27. 27. The system of claim 26, wherein the predefined split settings include splitting the customized dynamic video timeline at screen boundaries.

28. 27. The system of claim 26, wherein the predefined segmentation settings include segmenting the timeline of the customized dynamic video at times when one or more dynamic objects enter or exit the video.

29. 27. The system of claim 26, wherein the predefined division settings include dividing the customized dynamic video timeline into intervals of a fixed size.

30. 27. The system of claim 26, wherein the predefined division settings include dividing the customized dynamic video timeline along segments of a fixed color.

31. 27. The system of claim 26, wherein the partitions generated by the predefined partitioning settings are constrained by a predefined partition size.

32. 27. The system of claim 26, wherein the identifier is a hash value and / or is based on the result of some hash algorithm.

33. The hash algorithm is a hash computing algorithm. an element that calculates a duration hash representing the duration of the timeline interval associated with the video chunk; b. A component that calculates a static hash that represents the configuration of static elements of the video chunk contained in the timeline interval associated with the video chunk; c. A component that computes a dynamic hash that represents the configuration of dynamic elements in the video associated with the video chunk; d. An element that computes an encoder hash that represents the encoder and its configuration for the video; 33. The system of claim 32, comprising at least one of the following or some combination thereof:

34. 27. The system of claim 26, wherein the video chunk comprises only a portion of a video frame.

35. 27. The system of claim 26, wherein the chunk descriptor stores intermediate values ​​used to construct the identifier.

36. 36. The system of claim 35, wherein static elements of two or more video chunks identified as identical based on static hash values ​​are generated only once and used to render the two or more video chunks.

37. 36. The system of claim 35, wherein two or more dynamic videos are generated simultaneously, and the chunks having identical static elements belong to different output videos.

38. 36. The system of claim 35, wherein the chunks having identical static elements belong to the same output video.

Images