Precisely targeted micro-encoding of content

Abstract

To provide systems and methods for accurately targeted re-encoding of content.SOLUTION: In a method, a software code is executed to: receive content having a sequence of content segments, and marker data identifying a location within the sequence; identify, using the content and the marker data, segment boundaries of a content segment containing the location; determine, using the location and the segment boundaries, whether the location is situated within a predetermined interval of one of the segment boundaries; and re-encode a subsection of the sequence to produce a new segment boundary at the location. When the location is not situated within the predetermined interval, the subsection includes the content segment containing the location. When the location is situated within the predetermined interval, the subsection includes the content segment containing the location and a content segment adjoining the content segment containing the location.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] background Due to its near-universal popularity as a content medium, more audio-video (AV) content than ever before is being produced and made available to consumers. Additionally, as streaming platforms become increasingly important distribution hubs for AV content, adjustable bit-rate (ABR) AV content also becomes increasingly important due to variable network conditions and differences in data processing capabilities among the wide variety of consumer devices receiving the streaming content. As a result, the efficiencies that can be added to AV content, including ABR content, or enhanced post-production, becomes increasingly important to the producers, owners, or distributors of that content. [Background technology]

[0002] As an example, in some use cases, it may be advantageous or desirable to supplement the AV content, post-production, with the addition of interstitial content, such as advertisements. If the AV content was originally produced without predefined insertion points for interstitial content, introducing post-production into the content may result in an unpleasant or disruptive experience for consumers viewing the content during playback. For example, advertisements that are thoughtlessly inserted into the AV content may appear mid-scene or even mid-shot, thereby disrupting the consumer's viewing experience in a jarring and audible way. A traditional way to add interstitial content while avoiding the disruption and aesthetic discomfort for the viewer is to re-encode the content in its entirety, this time taking into account the splice points used to adjust segment boundaries for advertisements or other interstitial content. However, such a traditional approach is undesirably and in some cases impractical, expensive, both in terms of the computational time and quality control (QC) review required to re-encode the content. [Brief description of the drawings]

[0003] [Figure 1] 1 is a diagram of an exemplary system for precisely targeted micro-encoding of content according to one implementation. [Figure 2A] FIG. 13 is an illustrative diagram depicting four alternative regions of a content segment that may be occupied by insertion points for interstitial content, according to one implementation. [Figure 2B] 2B-2C depict respective strategies for precisely targeted micro-encoding of content for each of three of the alternative regions shown in FIG. 2A, according to one implementation. [Figure 2C] 1 is an exemplary diagram illustrating a strategy for precisely targeting micro-encoding content for alternative use cases, according to one implementation. [Diagram 3] FIG. 2 illustrates a more detailed example of precisely targeted micro-encoding of content according to one implementation. [Figure 4A] 1 is a flowchart outlining an exemplary method for precisely targeted micro-encoding of content, according to one implementation. [Figure 4B] 4B is a flowchart describing example operations for extending the method outlined in FIG. 4A according to one implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Detailed Description The following description includes specific information related to the implementation of the present invention. Those skilled in the art will recognize that the present invention can be implemented in ways other than those specifically described herein. The drawings in this application and the detailed description accompanying these drawings are directed to preferred implementations only. Unless otherwise noted, similar or corresponding elements among the drawings may be indicated by similar or corresponding reference numerals. Additionally, the drawings and illustrations in this application are generally not drawn to scale and are not intended to correspond to actual relative dimensions.

[0005] As mentioned above, when audio-video (AV) content is originally produced without predefined insertion points for interstitial content, introducing post-production into the content can result in an unpleasant or disruptive experience for consumers viewing the content during playback. For example, interstitial content in the form of advertisements that are thoughtlessly inserted into AV content can appear as mid-scenes or even mid-shots, thereby disrupting the consumer's viewing experience in a jarring and audible way. A traditional method of adding interstitial content that avoids the disruptive and aesthetic discomfort for the viewer is to re-encode the content in its entirety, this time taking into account the splice points used to adjust segment boundaries for advertisements or other interstitial content. However, as also mentioned above, such traditional methods are undesirably and in some cases impractical, expensive, both in terms of the computational time and quality control (QC) review required to re-encode the content.

[0006] This application discloses a system and method for precisely targeted micro-encoding of content. As defined herein, the term "micro-encoding" refers to the selective re-encoding of one or more sections of previously transcoded content, but less than the entire content, to allow for the introduction of new interstitial content, such as advertisements, without adversely affecting the aesthetic integrity of the original content's creative intent. The precisely targeted micro-encoding solution disclosed herein allows for the creation of new segment boundaries where none existed before, by advantageously taking into account new content segment boundary locations, existing content segment boundaries, constraints on minimum and maximum content segment durations, the possibility that subjective audio or video irregularities may be generated by the re-encoding, and the need to preserve original metadata and to change content playlists. Additionally, the inventive solution advances the state of the art described above, by significantly reducing the time required for re-encoding, as well as the QC time required for validation, since only precisely targeted small areas are re-encoded. Moreover, the precisely targeted micro-encoding solution of the present invention can be implemented using automated or substantially automated systems and methods.

[0007] As defined herein, "automated," "automated," and "automate" refer to systems and processes that do not require the involvement of a user, such as a system administrator. For example, in some implementations, a human editor or QC technician may review the performance of the systems and methods disclosed herein, although such human involvement is optional. Thus, in some implementations, the processes described herein may be performed under the control of hardware processing components of the disclosed systems.

[0008] Also, as defined herein, "content" may refer to a variety of different types and genres of AV content, as well as video without audio or audio without video. Examples of AV content include adjustable bit rate (ABR) content in the form of movies, television (TV) episodes or series, video games, and sporting events. Additionally or alternatively, in some implementations, "content" may be or include digital representations of, for example, people, fictional characters, places, objects, and identifiers such as brands and logos, which exist in virtual reality (VR), augmented reality (AR), or mixed reality (MR) environments. Furthermore, this content may represent a virtual world that any number of users can experience synchronously and persistently, while providing continuity of data such as people's identities, user histories, entitlements, ownership, payments, and the like. It should be noted that the concepts disclosed herein can also be applied to content that is a hybrid of traditional AV and fully immersive VR / AR / MR experiences such as interactive video.

[0009] A diagram of an exemplary system for precisely targeted micro-encoding of content according to one implementation is shown in Figure 1. System 100 includes a computing platform 102 having processing hardware 104 and a system memory 106 implemented as a computer-readable non-transitory storage medium. As shown in Figure 1, according to some implementations, system memory 106 stores micro-encoding software code 110 and optionally stores a content database 108.

[0010] As further shown in Figure 1, system 100 may be implemented in an environment of use that includes a content source 116 providing transcoded content 130 (hereinafter "content 130") and marker data 126 for content 130, a communications network 112, and a user 124 utilizing a user system 120 including a display 122. In addition, Figure 1 shows communications links 114 communicatively coupling content source 116 and user system 120 with system 100 via communications network 112. Figure 1 also shows re-encoded content 150 corresponding to content 130 and provided by micro-encoding software code 110.

[0011] It should be noted that system 100 receives content 130 from content source 116 via communications network 112 and network communications link 114, and in some implementations content source 116 may take the form of a content source integrated with computing platform 102 or may communicate directly with system 100, as indicated by dashed communications link 118.

[0012] It should be noted that, with respect to the representation of the system 100 shown in FIG. 1, the microencoding software code 110 and any content database 108 are depicted as being stored in the system memory 106 for conceptual clarity, but more generally, the system memory 106 can take the form of any computer-readable non-transitory storage medium. As used herein, the phrase "computer-readable non-transitory storage medium" refers to any medium, other than a carrier wave or other transitory signal, that provides instructions to the processing hardware of a computing platform, such as the processing hardware 104 of the computing platform 102. Thus, the computer-readable non-transitory storage medium can represent a variety of media, such as, for example, volatile and non-volatile media. Volatile media can include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory can include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical disks, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and flash memory.

[0013] 1 depicts the microencoding software code 110 and any content database 108 as being co-located with one another in the system memory 106, this representation is merely provided as an aid to conceptual clarity. More generally, the system 100 may include one or more computing platforms, e.g., computer servers, which may be co-located or may form an interoperably linked, but distributed, system, e.g., a cloud-based system. As a result, the processing hardware 104 and the system memory 106 may represent distributed processor and memory resources within the system 100. Thus, it will be apparent that the microencoding software code 110 and any content database 108 may be stored remotely with respect to one another within the distributed memory resources of the system 100.

[0014] The processing hardware 104 may include multiple processing units, such as one or more central processing units, one or more graphics processing units, one or more tensor processing units, one or more field programmable gate arrays (FPGAs), and application programming interface (API) servers. For purposes of definition, the terms "central processing unit (CPU)", "graphics processing unit (GPU)", and "tensor processing unit (TPU)" as used herein shall have their respective meanings commonly used in the state of the art. In other words, a CPU includes an arithmetic unit (ALU) for performing arithmetic and logical operations of the computing platform 102, and a control unit (CU) for retrieving programs, such as microencoding software code 110, from the system memory 106, while a GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. A TPU is an application specific integrated circuit (ASIC) specifically configured for artificial intelligence (AI) applications such as machine learning modeling.

[0015] Additionally, as defined herein, the term "machine learning model" or "ML model" may refer to a mathematical model for making future predictions based on patterns learned from samples of data or "training data." Various learning algorithms may be used to correlate input data with output data. These correlations form a mathematical model that may be used to make future predictions for new input data. Such predictive models may include one or more logistic regression models, Bayesian models, or neural networks (NNs). Additionally, "deep neural networks," in the context of deep learning, may refer to NNs that utilize multiple hidden layers between the input and output layers, which may enable learning based on features that are not explicitly defined in the raw data. In various implementations, NNs may be trained as classifiers and may be used to perform image processing or natural language processing.

[0016] In some implementations, the computing platform 102 may represent one or more web servers accessible over a packet-switched network, such as the Internet. Alternatively, the computing platform 102 may represent one or more computer servers supporting a private wide area network (WAN), a local area network (LAN), or included within another type of limited-delivery or private network. As yet another alternative, in some implementations, the system 100 may be implemented virtually, such as in a data center. For example, in some implementations, the system 100 may be implemented in software or as a virtual machine.

[0017] 1, user system 120 is shown as a desktop computer, but this representation is provided by way of example only. More generally, user system 120 can be any suitable mobile or stationary computing device or system that provides a user interface, supports connection to communications network 112, and provides sufficient data processing capabilities to implement the functionality ascribed to user system 120 herein. For example, in other implementations, user system 120 can take the form of a laptop computer, a tablet computer, or a smart phone.

[0018] With respect to the display 122 of the user system 120, the display 122 may be physically integrated with the user system 120 or may be communicatively coupled to, but physically separate from, the respective user system 120. For example, if the user system 120 is implemented as a smart phone, laptop computer, or tablet computer, the display 122 may generally be integrated with the user system 120. In contrast, if the user system 120 is implemented as a desktop computer, the display 122 may take the form of a monitor separate from the user system 120 in the form of a computer tower. Furthermore, the display 122 of the user system 120 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot (QD) display, or any other suitable display screen that performs a physical conversion of signals to light.

[0019] FIG. 2A shows an exemplary diagram depicting four alternative regions of a content segment that may be occupied by an insertion point for interstitial content, according to one implementation. As shown in FIG. 2A, content 230 includes a sequence of content segments 232, each defined by a segment boundary 234. Also shown in FIG. 2A are markers 238a, 238b, 238c, and 238d, and a minimum content segment duration 233, which identify respective desired locations for the insertion of interstitial content. In addition, FIG. 2A shows a number of shaded boxes, represented by exemplary shaded box 236, each of which simply represents a slug, i.e., one or more transition frames, in which the video / audio transitions to black / silence for a few seconds, as is known in the state of the art. Slugs are often used to frame commercial breaks or other portions of a program for interstitial content. The slugs enable a pleasant subjective viewing experience, for example, when transitioning from content to an advertisement and back to content again. As shown in FIG. 2A, it may be desirable to place each of the markers 238a, 238b, 238c, and 238d at the midpoint of the respective slug. Note that although the present application refers to the presence of a dark slug, it is the positions of the markers 238a, 238b, 238c, and 238d that are being addressed.

[0020] 1. Thus, content 130 may share any of the characteristics ascribed to content 230 in accordance with the present invention, and vice versa. In other words, like content 230, content 130 may include a sequence of content segments 232 characterized by a minimum content segment duration 233.

[0021] It should be noted that, with regard to the minimum content segment duration 233, limitations may be imposed on the minimum and maximum content segment duration for various reasons. In a practical sense, for example, it is undesirable to have content that contains only a single frame. As a result, it may be the case that a streaming service has a minimum content segment duration 233 or imposes a minimum content segment duration 233 on the content that is delivered. Furthermore, for some ABR protocols, the maximum content segment duration 233 may be predefined and its value may be included in the content description. For example, in Hypertext Transfer Protocol (HTTP) Live Streaming (HLS), the content playlist tag EXT-X-TARGETDURATION is required, which specifies the maximum content segment duration 233.

[0022] Thus, the position of the marker with respect to the existing segment boundary 234 can be described as being in one of four relative positions, identified in Figure 2A as case 0, case 1, case 2, and case 3. In case 0, the marker 238a is already perfectly aligned with the existing segment boundary. As a result, no further processing or re-encoding is required for case 0, since a segment boundary 234 already exists that allows interstitial content to be inserted at the position of the marker 238a.

[0023] In contrast, in case 1, the location of marker 238b is within the minimum content segment duration 233 from the segment boundary 234 at the beginning of the content segment. Thus, simply creating a new segment boundary at the location of marker 238b would violate the minimum content segment duration 233. Similarly, the location of marker 238d is within the minimum content segment duration 233 from the segment boundary 234 at the end of the content segment. Thus, simply creating a new segment boundary at the location of marker 238d would also violate the minimum content segment duration 233.

[0024] In case 2, in contrast to cases 0, 1, and 3, the position of marker 238c is not located within the minimum content segment duration 233 from a segment boundary 234 at the beginning or end of the content segment. As a result, creating a new segment boundary at the position of marker 238c does not violate the minimum content segment duration 233.

[0025] Figure 2B illustrates a diagram depicting respective strategies for precisely targeted micro-encoding of content, according to one implementation, for each of three of the alternative regions shown in Figure 2A, namely, Case 1, Case 2, and Case 3. Note that any features identified in Figure 2B by the same reference numbers used to identify the features in Figure 2A correspond to the respective features in Figure 2A and may share any of the above-mentioned properties ascribed to those corresponding features.

[0026] As shown in Figure 2B, content 230 includes a sequence of content segments 232a, 232b, 232c, 232d, 232e, 232f, and 232g (hereinafter "content segments 232a-232g"), and segment boundaries represented by exemplary segment boundaries 234a and 234b. Also shown in Figure 2B are markers 238b, 238c, and 238d, and their respective slugs.

[0027] For case 1, re-encoding cannot start at the beginning of content segment 232d that contains marker 238b, because starting re-encoding at the beginning of content segment 232d and generating a new segment boundary at marker 238b would violate the minimum content segment duration requirement described above with reference to Figure 2A. Therefore, re-encoding must start at the start time of the beginning of the preceding content segment, i.e., no later than the start time of the immediately preceding content segment 232c. For case 1, re-encoding can end at the end of content segment 232d that contains marker 238b.

[0028] Note that the smallest subsection of content 230 that must be re-encoded to generate a new segment boundary at the position of marker 238b ends at the end of content segment 232d, which contains the position of marker 238b, and begins at the beginning of content segment 232c, which immediately precedes content segment 232d, although other re-encoding durations are possible. For example, re-encoding for case 1 could begin prior to content segment 232c, such as at the beginning of content segment 232b, or at the beginning of any content segment preceding content segment 232b. Additionally or alternatively, re-encoding for case 1 could end at the end of a later content segment, such as the end of content segment 232e, or the end of a subsequent content segment.

[0029] For case 2, re-encoding can start at the beginning of content segment 232d, which contains marker 238c, and can end at the end of content segment 232d, because marker 238c is not located within the minimum content segment duration from either of segment boundaries 234a or 234b of content segment 232d. Note that the smallest subsection of content 230 that must be re-encoded to generate a new segment boundary at the position of marker 238c begins and ends at the segment boundary of content segment 232d, which contains the position of marker 238c, although other re-encoding durations are possible. For example, re-encoding for case 2 can start prior to content segment 232d, such as at the beginning of content segment 232c, or at the beginning of any content segment preceding content segment 232c. Additionally or alternatively, the re-encoding for case 2 may end at the end of a later content segment, such as the end of content segment 232e or the end of a subsequent content segment 232e.

[0030] For case 3, re-encoding can start at the beginning of the content segment 232d that contains the marker 238d, but cannot end at the end of the content segment 232d, because ending the re-encoding at the end of the content segment 232d and generating a new segment boundary at the marker 238d would violate the minimum content segment duration requirement described above with reference to FIG. 2A. Therefore, re-encoding must end after the end time of the end of the following content segment, i.e., the end time of the adjacent following content segment 232e. For case 3, re-encoding can start at the beginning of the content segment 232d that contains the marker 238d, as described above.

[0031] The smallest subsection of content 230 that must be re-encoded to generate a new segment boundary at the location of marker 238d begins at the beginning of content segment 232d, which contains the location of marker 238d, and ends at the end of content segment 232e, which immediately follows content segment 232d, although other re-encoding durations are possible. For example, re-encoding for case 3 can begin prior to content segment 232d, such as at the beginning of content segment 232c, or at the beginning of any content segment preceding content segment 232c. Additionally or alternatively, re-encoding for case 3 can end at the end of a later content segment, such as the end of content segment 232f, or the end of a subsequent content segment.

[0032] FIG. 2C illustrates an exemplary diagram depicting a strategy for precisely targeted micro-encoding of content for an alternative use case, namely, case 4, according to one implementation. Note that any features in FIG. 2C identified by the same reference numbers as appearing in FIG. 2B correspond to respective features in FIG. 2B and may share any of the above-mentioned characteristics ascribed to those corresponding features in accordance with this disclosure. As shown in FIG. 2C, in case 4, marker 238e is located within minimum content segment duration 233 between each of segment boundaries 234a and 234b of content segment 232d.

[0033] As with case 1 above, for case 4, re-encoding cannot start at the beginning of the content segment 232d that contains the marker 238e, because starting re-encoding at the beginning of the content segment 232d and generating a new segment boundary at the marker 238e would violate the minimum content segment duration requirement described above with reference to FIG. 2A. Therefore, re-encoding must start before the start time of the beginning of the preceding content segment, i.e., the start time of the adjacent preceding content segment 232c. However, as with case 3, for case 4, re-encoding cannot end at the end of the content segment 232d, because starting re-encoding at the end of the content segment 232d and generating a new segment boundary at the marker 238e would also violate the minimum content segment duration requirement described above with reference to FIG. 2A. Therefore, re-encoding must end after the end time of the following content segment, i.e., the end time of the adjacent following content segment 232e. Thus, for case 4, re-encoding can start at the beginning of content segment 232c and end at the end of content segment 232e.

[0034] The smallest subsection of content 230 that must be re-encoded to generate a new segment boundary at the location of marker 238e begins at the beginning of content segment 232c that immediately precedes content segment 232d that contains the location of marker 238e, and ends at the end of content segment 232e that immediately follows content segment 232d, although other re-encoding durations are possible. For example, re-encoding for case 4 can begin prior to content segment 232c, such as at the beginning of content segment 232b, or at the beginning of any content segment that precedes content segment 232b. Additionally or alternatively, re-encoding for case 4 can end at the end of a later content segment, such as the end of content segment 232f, or the end of a subsequent content segment.

[0035] Note that the re-encoding strategies for each of cases 1, 2, 3, and 4 identify a minimal subsection of content 230 to be re-encoded, characterized by a triplet of data including a minimal subsection start time, a minimal subsection end time, and a marker location. Furthermore, the re-encoding strategies applied to cases 1, 3, and 4 generate new segment boundaries at the locations of markers 238b, 238d, and 238e, respectively, by shifting one or more existing segment boundaries to new locations without adding additional content segments to content 230, whereas the strategy applied to case 2 partitions single content segment 232d into two content segments.

[0036] Thus, any subsection of content 230 that is re-encoded by the novel and inventive precisely targeted micro-encoding of the present invention may initially contain an integer number "N" of content segments, and after re-encoding, the re-encoded subsection may contain N content segments or N+1 content segments. Furthermore, for case 2, since the only requirement imposed on the re-encoding is that the beginning and end of content segment 232d and the position of marker 238c are respected, in some embodiments, a re-encoded subsection of content segment 230 that initially contains N content segments may be re-encoded into N content segments, N+1 content segments, or N+2 or more content segments. In other words, the original subsection of content 230 may contain at least one less content segment than the re-encoded subsection.

[0037] The start and end times of the minimum re-encoded sub-sections of the content 230 described above do not take into account characteristics of the underlying content that may result in start and / or end times that make the re-encoding less than optimal. For example, the content at the beginning, end, or both of the minimum re-encoded sub-sections may have certain attributes that may result in audible or visible audio artifacts, video artifacts, or both audio and video artifacts during playback of the content 230 if the re-encoding were to begin or end there. The process for identifying an optimal re-encoded sub-sequence of the content 230 including the minimum re-encoded sub-sections described above with reference to Case 1, Case 2, Case 3, and Case 4 may include various methods to reduce or eliminate such problems. For example, the process for determining optimal start and end times for re-encoding may take into account one or more of the content metadata, such as Dolby Vision shot metadata, scene change detection, complexity analysis of chunk boundaries, etc. The end result is similar: a triplet of data that identifies a start time for the re-encoded subsection, an end time for the re-encoded subsection, and the location of a marker contained within the re-encoded subsection. The determination of the optimal sub-section for re-encoding the content 230 seeks to minimize the impact of audio and video in replacing the original content with the re-encoded content.

[0038] Regardless of whether the smallest subsection of content 230 described above or an optimal subsection of content 230 that includes this smallest subsection is re-encoded, the precisely targeted micro-encoding of content subsections disclosed herein results in the following characteristics: 1) re-encoding a content subsection having N content segments results in a re-encoded subsection having N or N+1 or more content segments, 2) only the first content segment at the beginning of the re-encoded content subsection is aligned to the start time of the original content subsection, 3) a new segment boundary is generated that is aligned to the position of a marker, and 4) the final frame (in display order) of the last content segment of the re-encoded content subsection is the same as the final frame of the last content segment of the original content subsection.

[0039] A more detailed diagram of one implementation of precisely targeted micro-encoding of content is shown in Figure 3. Note that the example shown in Figure 3 corresponds to case 1 in Figures 2A and 2B. As shown in Figure 3, only the content segment 332d-1 of the content 330-1 that includes the position of marker 338b and the content segment 332c-1 preceding it are precisely targeted re-encoded to generate re-encoded content 350 with a new segment boundary 340 at the position of marker 338b.

[0040] Marker 338b, content 330, content segment 332c-1, content segment 332d-1, and re-encoded content 350 generally correspond to marker 238b, content 130 / 230, content segment 232c, content segment 232d, and re-encoded content 150, respectively, shown separately in Figures 1, 2A, and 2B. Thus, marker 238b, content 120 / 230, content segment 232c, content segment 232d, and re-encoded content 150 may share any of the characteristics ascribed to marker 338b, content 330, content segment 332c-1, content segment 332d-1, and re-encoded content 350 in accordance with the present invention, and vice versa.

[0041] The principles of re-encoding a subsection of content 330, including content segments 332c-1 and 332d-1, have been described above with reference to Figure 2B and case 1 and will not be repeated here. However, according to the example shown in Figure 3, re-encoding content segments 332c-1 and 332d-1 results in a re-encoded content segment 332c-2 that includes the entire original content segment 332c-1 and a portion of the original content segment 332d-1, whereas the re-encoded content segment 332d-2 does not include the entire original content segment 332d-1.

[0042] It should be noted that some media formats, such as Dolby Vision, include per-frame content metadata that is associated with or integrated with the underlying video frames. Such metadata may be the result of a linear process that begins at the beginning of the content 130 / 230 / 330 and continues all the way to the end of the content. In these use cases, the precisely targeted micro-encoding solution of the present invention may include extracting such metadata from existing content segments and appropriately placing this metadata within the re-encoded content segments. With reference to the exemplary use case illustrated in FIG. 3, content metadata associated with or included in content segment 332d-1 may be partitioned between re-encoded content segments 332c-2 and 332d-2 bounded by the new segment boundary 340 based on the location of the new segment boundary 340.

[0043] The functionality of system 100 is further described with reference to Figure 4A, which shows a flowchart 460A illustrating an exemplary method for precisely targeted micro-encoding of content according to one implementation. With respect to the method outlined in Figure 4A, certain details and features have been omitted from flowchart 460A so as not to obscure the description of features of the present invention.

[0044] 4A in combination with FIGS. 1 and 2A, a flowchart 460A begins with receiving content 130 / 230 including a sequence of content segments 232 and marker data 126 for the content, which identifies a location within the sequence (operation 461). Note that the location within the sequence of content segments 232 included in the content 130 / 230 identified by the marker data 126 may be an insertion point for inserting interstitial content within the sequence of content segments 232. The content 130 / 230 and marker data 126 may be received in operation 461 by microencoding software code 110 executed by processing hardware 104 of the system 100. For example, as shown in FIG. 1, in some implementations, the content 130 / 230 and marker data 126 may be received by the system 100 from a content source 116 either via a communications network 112 and network communications link 114 or directly via communications link 118.

[0045] As mentioned above, the content 130 / 230 can include a variety of different types and genres of AV content, as well as video without audio or audio without video. Examples of AV content include ABR content in the form of movies, TV episodes or series, video games, and sporting events. Additionally or alternatively, in some implementations, the content 130 / 230 can be or include digital representations of people, fictional characters, places, objects, and identifiers such as brands and logos, which exist in, for example, a VR, AR, or MR environment. Furthermore, the content 130 / 230 can represent a virtual world that any number of users can experience synchronously and persistently, while providing continuity of data such as person identities, user history, entitlements, ownership, payments, etc. Alternatively or additionally, the content 130 / 230 can be or include a hybrid of traditional AV and fully immersive VR / AR / MR experiences such as interactive video.

[0046] Flowchart 460A may further include using the content 130 / 230 and the marker data 126 to identify a segment boundary of a content segment that includes the location identified by the marker data 126 (operation 462). Identifying the segment boundary of a content segment that includes the location identified by the marker data 126 in operation 462 may be performed by the processing hardware 104 of the computing platform 102 executing the microencoding software code 110 using any suitable technique known in the state of the art.

[0047] Flowchart 460A further includes using the location identified by marker data 126 and the segment boundaries identified in act 462 to determine whether the location identified by marker data 126 is located within a predetermined interval from one of those segment boundaries (act 463). By way of example, as shown in Figures 2A and 2B, in some implementations such a predetermined interval may be the minimum content segment duration 233 in Figure 2A, and act 463 may correspond to determining whether the location identified by marker data 126 corresponds to case 1, case 2, case 3, or case 4 illustrated in Figures 2A, 2B, and 2C for content segment 232d in Figures 2B and 2C.

[0048] 2B and 2C, a minimum subsection for re-encoding the content 230 to align with the new segment boundaries may be identified based on the determination performed in operation 463. However, and as further described above, in some implementations it may be advantageous or desirable to re-encode an optimal subsection of the content 230, where the optimal subsection is longer than and includes the minimum subsection.

[0049] Thus, in some implementations, flowchart 460A may further include determining a subsection within the sequence that is optimal for re-encoding based on content 130 / 230, the optimal subsection including multiple content segments in addition to content segment 232d that includes the location (operation 464). Note that operation 464 is optional, and in some implementations may be omitted from the method outlined by flowchart 460A. However, in implementations in which operation 464 is included in the method outlined by flowchart 460A, operation 464 may be performed by microencoding software code 110 executed by processing hardware 104 of computing platform 102, as described above with reference to FIG. 2B. It should be noted that, as discussed above with reference to FIG. 2B, in various use cases, in addition to content segment 232d, one or more of the content segments included in the optimal subsection may precede content segment 232d or may follow 232d, or one or more of these content segments may precede content segment 232d and one or more of these content segments may follow content segment 232d.

[0050] 3 in combination with FIGS. 1, 2A, 2B, 2C, and 4A, flowchart 460A further includes re-encoding a subsection of the sequence to generate a new segment boundary 340 at a location identified by marker data 126. If, in operation 463, it is determined that the location is not within the predetermined distance from one of the segment boundaries of content segment 232d, then the re-encoded subsection includes at least the content segment 232d that includes the location identified by marker data 126, and may include at most only content segment 232d (operation 465-1). However, if it is determined that the location identified by the marker data 126 is located within the predetermined distance from one of the segment boundaries of the content segment 232d, the subsection to be re-encoded includes at least the content segment that includes the location and the content segment adjacent to the content segment 232d that includes the location, i.e., either the content segment 232c that precedes the content segment 232d or the content segment 232e that follows the content segment 232d, and may include at most only the content segment 232d and one of the content segments 232c or 232e (operation 465-2). As noted above, in some implementations, operations 464 and 465 may be omitted from the flowchart 460A. In these implementations, operation 463 may be directly followed by operation 466.

[0051] Whether the re-encoding of the subsection of the content 230, including the content segment 232d, is performed by operation 465-1 or operation 465-2, the re-encoding may be performed by micro-encoding software code executed by the processing hardware 104 of the computing platform 102. Note that in some use cases, the content 130 / 230 may have previously been transcoded, segmented, and potentially encrypted, and an ABR media playlist generated, and the resulting content 130 / 230 may simply be stored in its original location. In other words, the original transcoded version of the content 130 / 230 may not exist or may be unavailable. In these use cases, operation 465-1 or operation 465-2 may include obtaining the original and re-encoding the subsection of the original. However, in use cases where the original transcoded version of content 130 / 230 is available, operation 465-1 or operation 465-2 may include first decoding the subsection of content 230 to be re-encoded and then re-encoding this subsection.

[0052] In some implementations, the method outlined by flowchart 460A ends at operation 465-1 or operation 465-2, while in other implementations, the method may be expanded to include additional operations. Referring now to Figure 4B, Figure 4B shows a flowchart 460B describing example operations for expanding the method outlined in Figure 4A, according to one implementation. It should be noted that certain details and features of the operations described in Figure 4B have been omitted from flowchart 460B so as not to obscure the description of the features of the present invention.

[0053] In some use cases, as described above with reference to FIG. 3 and further with reference to FIG. 1, the content segment 332d-1 including the location identified by the marker data 126 may further correspond to content metadata associated with or included in the content segment 332d-1. In these use cases, the method outlined by flowcharts 460A and 460B may further include partitioning this content metadata between the re-encoded content segments 332c-2 and 332d-2 bounded by the new segment boundary 340 based on the new segment boundary 340 (operation 466). Note that operation 466 is optional and may be omitted in some implementations from the method extended by flowchart 460B. However, in implementations in which operation 466 is included in the method extended by flowchart 460B, operation 466 may be performed by microencoding software code 110 executed by processing hardware 104 of computing platform 102, as described above with reference to FIG. 3.

[0054] Flowchart 460B may further include packaging the subsection of the sequence re-encoded at act 465-1 or 465-2, or the subsection of the sequence re-encoded at act 465-1 or 465-2 and the content metadata partitioned at act 466, as a replacement for the subsection of the sequence that includes the content segment 232d that includes the location identified by marker data 126 (act 467). By way of example, the subsection of the sequence re-encoded at act 465-1 or 465-2, or the subsection of the sequence re-encoded at act 465-1 or 465-2 and the content metadata partitioned at act 466 may be packaged, or packaged and encrypted, into any desired packaging format, such as, for example, Common Metadata Application Format (CMAF). However, act 467 is optional and may be omitted from the method extended by flow chart 460B in some implementations. In implementations in which operation 467 is included in the method expanded by flowchart 460B, operation 467 may be performed by microencoding software code 110 executed by processing hardware 104 of computing platform 102.

[0055] Flowchart 460B may further include updating a playlist for content 130 / 230 based on the re-encoding performed in act 465-1 or 465-2 (act 468). Many ABR protocol specifications use a form of playlist that uses a Uniform Resource Identifier (URI), such as a Uniform Resource Locator (URL), for the discrete segment files. In these use cases, the existing playlist for content 130 / 230 may be based on act 465-1 or 465-2. Instead, the subsections re-encoded in act 465-1 or 465-2 may need to be represented by the given ABR protocol specification. For example, HLS uses the #EXT-X-DISCONTINUITY tag when the encoding parameters or encoding order are different from the original. Note that act 468 is optional and may be omitted from the method extended by flow chart 460B in some implementations. However, in implementations in which operation 468 is included in the method expanded by flowchart 460B, operation 468 may be performed by microencoding software code 110 executed by processing hardware 104 of computing platform 102.

[0056] It should be noted that with respect to the operations described by flowcharts 460A and 460B, operations 461, 462, and 463 (hereinafter "operations 461-463"), operations 461-463 and 465-1, operations 461-463 and 465-2, operations 461-463, 464 and 465-1, operations 461-463, 464 and 465-2, or any sequence of the operations described above, may be combined with one or more of operations 466, 467, and 468 to be performed as an automated process that can eliminate human involvement.

[0057] Accordingly, this application discloses a system and method for precisely targeted micro-encoding of content. The precisely targeted micro-encoding solution disclosed herein allows for the creation of new segment boundaries where none existed before by advantageously taking into account new content segment boundary locations, existing content segment boundaries, constraints on minimum and maximum content segment durations, subjective audio and video irregularities created by re-encoding, and the need to preserve original metadata and alter content playlists. Furthermore, the inventive solution advances the state of the art described above while significantly reducing the time required for re-encoding as well as the QC time required for validation, since only precisely targeted micro-regions are re-encoded.

[0058] From the foregoing description, it is apparent that various techniques may be used to implement the concepts described herein without departing from the scope of these concepts. Moreover, while these concepts have been described with reference to specific implementations, those of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of these concepts. The described implementations are therefore to be considered in all respects as illustrative and not restrictive. It is also apparent that the present application is not limited to the specific implementations described herein, and that numerous rearrangements, modifications, and substitutions are possible without departing from the scope of the invention.

Claims

1. 1. A system comprising a computing platform having processing hardware and a system memory storing software code, The processing hardware executes the software code to receiving content including a sequence of content segments and marker data for the content, the marker data identifying a position within the sequence; using the content and the marker data to identify a segment boundary of the content segment that includes the identified location; using the location and the segment boundaries to determine whether the location is within a predetermined distance of one of the segment boundaries; configured to re-encode a subsection of the sequence based on the determination to generate a new segment boundary at the location; When the location is determined to not be within the predetermined distance from one of the segment boundaries, the subsection in the column includes the content segment that includes the location, and when the location is determined to be within the predetermined distance from one of the segment boundaries, the subsection in the column includes the content segment that includes the location and the content segment adjacent to the content segment that includes the location.

2. The processing hardware executes the software code to 2. The system of claim 1, further configured to package the re-encoded subsection in the string as a replacement for the subsection in the string that includes the content segment that includes the location.

3. The processing hardware executes the software code to 2. The system of claim 1, further configured to, prior to the re-encoding, determine an optimal subsection within the sequence for the re-encoding based on the content, the optimal subsection including a plurality of the content segments in addition to the content segment that includes the position.

4. The system of claim 3 , wherein at least one of the plurality of content segments precedes the content segment that includes the location or follows the content segment that includes the location.

5. The content segment including the location includes content metadata, and the processing hardware executes the software code to The system of claim 1 , further configured to partition the content metadata between the content segments bounded by the new segment boundary based on the location of the new segment boundary.

6. 6. The system of claim 5, wherein the processing hardware is further configured to execute the software code to package the re-encoded subsection in the sequence and the partitioned content metadata as a replacement for the subsection in the sequence that includes the content segment that includes the location.

7. The processing hardware executes the software code to The system of claim 1 , further configured to update a playlist for the content based on the re-encoding.

8. The system of claim 1 , wherein the content comprises adaptive bit rate (ABR) content.

9. 2. The system of claim 1, wherein the location within the string identified by the marker data comprises an insertion point for inserting interstitial content within the string of the content segment.

10. 2. The system of claim 1, wherein the subsection in the sequence that includes the content segment that includes the location includes at least one fewer of the content segments than the content segments included in the re-encoded subsection in the sequence.

11. 1. A method for use by a system having a computing platform having processing hardware and a system memory storing software code, comprising: receiving content including a sequence of content segments and marker data for the content, the marker data identifying a position within the sequence; using the content and the marker data to identify a segment boundary of the content segment that includes the identified location by executing the software code with the processing hardware; using the location and the segment boundaries to determine whether the location is within a predetermined distance from one of the segment boundaries by the processing hardware executing the software code; and executing the software code with the processing hardware to re-encode a subsection of the sequence based on the determination to generate a new segment boundary at the location; When the location is determined to not be within the predetermined distance from one of the segment boundaries, the subsection in the column includes the content segment that includes the location, and when the location is determined to be within the predetermined distance from one of the segment boundaries, the subsection in the column includes the content segment that includes the location and the content segment adjacent to the content segment that includes the location.

12. 12. The method of claim 11, further comprising packaging the re-encoded subsection in the sequence as a replacement for the subsection in the sequence that includes the content segment that includes the location by the processing hardware executing the software code.

13. 12. The method of claim 11, further comprising, prior to the re-encoding, determining, based on the content, the subsection within the sequence that is optimal for the re-encoding by the processing hardware executing the software code, the optimal subsection including a plurality of the content segments in addition to the content segment that includes the position.

14. The method of claim 13 , wherein at least one of the plurality of content segments precedes the content segment that includes the location or follows the content segment that includes the location.

15. The content segment including the location includes content metadata, and the method further comprises:

12. The method of claim 11, further comprising partitioning the content metadata between the content segments bounded by the new segment boundary based on the location of the new segment boundary by the processing hardware executing the software code.

16. 16. The method of claim 15, wherein the processing hardware executes the software code to package the re-encoded subsection of the sequence and the partitioned content metadata as a replacement for the subsection of the sequence that includes the content segment that includes the location.

17. 12. The method of claim 11, further comprising updating a playlist for the content based on the re-encoding by the processing hardware executing the software code.

18. The method of claim 11 , wherein the content comprises adaptive bit rate (ABR) content.

19. The method of claim 11 , wherein the location within the string identified by the marker data comprises an insertion point for inserting interstitial content within the string of the content segment.

20. 12. The method of claim 11, wherein the subsection in the sequence that includes the content segment that includes the location includes at least one fewer of the content segments than the content segments included in the re-encoded subsection in the sequence.

Images