A technology that reuses part of the original encoded video when encoding localized video.

Abstract

In various embodiments, a localization video encoding application encodes a localization video. In operation, the localization video encoding application calculates a predicted frame based on a target frame of the localization video and at least a portion of a reference frame of a decoded original video. The localization video encoding application also calculates a residual frame based on the predicted frame and the target frame of the localization video. The localization video encoding application also performs one or more encoding operations on the residual frame to generate a frame of an encoded localization layer. Then, the frame of the encoded localization layer and at least one frame of the encoded original video are sent to another device for decoding.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Patent Application No. 17 / 834,399, filed June 7, 2022, the contents of which are incorporated herein by reference. [Technical Field]

[0002] Various embodiments of the present disclosure relate generally to computer science and streaming video technology, and more particularly to techniques for reusing portions of an encoded original video (hereinafter, "encoded original video") when encoding localized video (hereinafter, "localized video"). [Background technology]

[0003] In a process known as "video localization," original video is modified to create localized video that is more tailored to a target audience. For example, lip reanimation techniques can be used to modify the lip movements of speakers in an English-language film to match the film's French dubbed audio, creating a localized version of the film that is more tailored to French-speaking audiences. Typically, a localized video contains most of the video data that is "original" video data, unchanged from the corresponding original video, while other video data is "localized" video data that is customized for the target audience. Typical video streaming services do not distinguish between localized and original video when streaming video to end-user devices.

[0004] In some streaming implementations, a video encoder is used to encode video "chunks" to generate encoded video chunks that are smaller in size than the original video chunks. These encoded video chunks are then stored on an origin server and streamed on-demand to end-user devices via a content delivery network (CDN). In many implementations, to play a particular video, an end-user device runs a playback application, which sends a series of requests to an edge server in the CDN for the encoded video chunks associated with the particular video. The edge server is a server that is closer to the end-user device than the origin server. If the edge server has a copy of the requested encoded video chunk stored in its associated cache memory, the request results in a "cache hit," and the edge server sends the requested encoded video chunk to the playback application. Otherwise, a "cache miss" occurs, and the edge server retrieves the requested encoded video chunk from the origin server and sends it to the playback application. As the playback application receives the various encoded video chunks, the video decoder decodes these encoded video chunks to generate corresponding decoded video chunks that are then played back via the end user device.

[0005] Considering the process of streaming localized videos instead of original videos, one problem with encoding localized video chunks using conventional encoders is that the original video data duplicated in the localized video chunks is typically redundantly encoded. More specifically, the original video data held in a localized video is encoded once when generating the encoded video chunks for the original video, and then encoded again when generating the encoded video chunks for the localized video. As a result, the total size of conventionally encoded localized videos (hereinafter, "encoded localized videos") is comparable to the size of the corresponding encoded original videos, and is therefore excessively large compared to the amount of localized video data present in the encoded localized videos. As a result, the amount of memory used to store the encoded video chunks for a localized video is unnecessarily large. Furthermore, because the size of the cache memory used by edge servers is typically much smaller than the total size of all encoded video chunks associated with a streamable video library stored on an origin server, the ratio of cache hits to cache misses ("cache efficiency") in a CDN is often excessively low relative to the amount of localized video data present in the encoded localized video. Because reduced cache efficiency can result in increased transmission delays, reduced effective transmission speeds, and reduced transmission reliability during video streaming, reduced CDN cache efficiency can degrade end-user quality of experience (QoE).

[0006] To reduce the amount of original video data that is redundantly encoded when encoding localized video, some implementations simply do not encode localized video chunks that are identical to original video chunks. Instead, metadata is used to indicate that the corresponding encoded original video chunks should be reused as encoded localized video chunks. Meanwhile, localized video chunks that are not identical to original video chunks are encoded to generate corresponding encoded localized video chunks. However, this approach also suffers from the problem that many localized video chunks often have very slight differences from their corresponding original video chunks. In such cases, a relatively large amount of original video data is still redundantly encoded when encoding the localized video chunks. For example, if lip reanimation techniques are used to generate localized video chunks, almost all localized video chunks will have very slight modifications to the speaker's lips in at least one frame compared to the corresponding original video chunks. The data contained in the localized video chunks, other than the lip correction data, is usually identical to the data contained in the original video chunks. Therefore, when encoding the localized video chunks, most of the original video data is also duplicated, which results in the need to transmit and store the original video data redundantly. Summary of the Invention [Problem to be solved by the invention]

[0007] As can be seen from the above, there is a need in the art for more effective techniques for encoding localized video. [Means for solving the problem]

[0008] In one embodiment, a computer-implemented method for encoding localized video is defined, the method including: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localized video; performing one or more encoding operations on the residual frame to generate frames of an encoded localization layer; and transmitting the frames of the encoded localization layer (hereinafter “encoded localization layer”) and at least one frame of the encoded original video to another device for decoding.

[0009] One technical advantage of the technology of the present disclosure over the prior art is that the technology of the present disclosure at least reduces the amount of original video data that is over-encoded when encoding localized video. In this sense, each frame of the encoded localization layer can be specified to reuse any number of portions (including entire frames) of the corresponding frame of the decoded original video. This significantly reduces the amount of memory used to store the encoded localization layer chunks used to construct the encoded localization video chunks compared to the amount of memory required to store the encoded localization video chunks using prior art techniques. Another technical advantage of the technology of the present disclosure is that storing the encoded localization layer chunks rather than the encoded localization video chunks for streaming localized video over a CDN can improve the caching efficiency of the CDN and ultimately improve the QoE for end users. These technical advantages result in one or more technical improvements over prior art approaches. [Brief explanation of the drawings]

[0010] Having briefly summarized the concept of the present invention above, the concept of the present invention will now be more particularly described with reference to various embodiments, some of which are also illustrated in the accompanying drawings, so that the features of the various embodiments described above can be understood in detail. However, it should be noted that the accompanying drawings merely illustrate exemplary embodiments of the concept of the present invention and therefore should not be considered to limit the scope of the present disclosure in any way, as there may be other embodiments having similar effects. [Figure 1] FIG. 1 is a conceptual diagram of a system configured to implement one or more aspects of various embodiments. [Figure 2] 2 illustrates a more detailed view of one of the localization encoders shown in FIG. 1, according to various embodiments. [Figure 3] 1 is a flow diagram illustrating method steps for encoding localized video, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the concepts of the present invention may be practiced without one or more of these specific details.

[0012] Video localization involves modifying an original video to generate one or more localized videos that are more tailored to the target audience. For example, a movie made in English can be localized for 30 different languages ​​by modifying the lip movements of the speaker to match the movie's dubbed audio in 30 different languages ​​using lip re-animation technology. Typically, the majority of the video data contained in the localized video is original video data that is unchanged from the corresponding original video, while the remaining video data is localized video data customized for the target audience. Typical video streaming services do not distinguish between localized and original video when streaming videos to end-user devices.

[0013] In some streaming implementations, a video encoder is used to encode multiple chunks of video to generate multiple encoded video chunks corresponding to multiple pre-encoded versions of the video. Typically, the multiple pre-encoded versions of a video correspond to different combinations of average bitrate and resolution, or "bitrate-resolution pairs," and are associated with different average image quality levels. Storing multiple encoded video chunks for a video corresponding to multiple pre-encoded versions of the video increases the likelihood that the video can be streamed to an end-user device with good image quality and uninterrupted playback, regardless of the throughput available on the network connection over which the video is streamed.

[0014] In some embodiments, these encoded video chunks are stored on an origin server and available for on-demand streaming to an end-user device via a CDN. In many implementations, to play a given video, an end-user device runs a playback application that sequentially sends a series of requests to an edge server in the CDN for the encoded video chunks associated with the given video. The edge server is a server that is closer to the end-user device than the origin server. If the edge server has a copy of the encoded video chunks requested by each such request stored in its associated cache memory, a cache hit occurs for the request, and the edge server sends the requested encoded video chunks to the playback application. Otherwise, a cache miss occurs, and the edge server retrieves the requested encoded video chunks from the origin server (optionally, and also stores the requested encoded video chunks in its associated cache memory) and sends the requested encoded video chunks to the playback application. As the playback application receives the various encoded video chunks, the video decoder decodes these encoded video chunks to generate corresponding decoded video chunks that are then played back via the end user device.

[0015] One problem with encoding localized video chunks using conventional encoders is that the original video data duplicated in the localized video chunks is typically coded redundantly. More specifically, when generating multiple coded video chunks associated with a given bitrate and resolution combination, the original video data contained in localized videos associated with an original video is coded once for the original video and then coded again for each localized video associated with the original video. As a result, the size of the pre-coded version of each conventional localized video is comparable to the size of the pre-coded version of the corresponding original video, and is therefore excessively large relative to the amount of localized video data present in the pre-coded version of the localized video. This results in an unnecessarily large amount of memory being used to store all the coded video chunks for each localized video.

[0016] For example, if an original video and 30 localized videos derived from the original video are encoded at 10 different bitrate and resolution combinations, the original video data contained in the localized videos will be redundantly coded 300 times, and the amount of memory used to store all the coded video chunks associated with the 30 localized videos will be approximately 30 times larger than the amount of memory used to store all the coded video chunks associated with the original video.

[0017] Furthermore, because the size of the cache memory used by an edge server is typically much smaller than the total size of all encoded video chunks associated with a video library stored on an origin server, the efficiency of the CDN cache is often excessively reduced relative to the amount of localized video data present in the encoded localized video. This reduced efficiency of the CDN cache can lead to increased transmission delays, reduced effective transmission speeds, and reduced transmission reliability during video streaming, potentially resulting in a reduced QoE for end users.

[0018] To reduce the amount of original video data that is over-encoded when encoding localized video, some implementations simply do not over-encode localized video chunks that are identical to original video chunks. Instead, they use metadata to indicate that the corresponding encoded original video chunks should be reused as encoded localized video chunks. Meanwhile, localized video chunks that are not identical to original video chunks are encoded to generate corresponding encoded localized video chunks. However, this approach also suffers from the problem that many localized video chunks often have very slight differences from their corresponding original video chunks. In such cases, a relatively large amount of original video data is still over-encoded when encoding the localized video chunks.

[0019] For example, if lip reanimation technology is used to generate localized video chunks corresponding to playback times of 2 to 10 seconds, almost all of the localized video chunks will have very slight modifications to the speaker's lips in at least one frame of the corresponding original video chunk. The data contained in the localized video chunks other than the lip modifications are typically identical to the data contained in the original video chunks. Therefore, when encoding the localized video chunks, most of the original video data is still duplicated and encoded.

[0020] However, according to the techniques of this disclosure, when encoding localized video derived from original video, the localized video encoding application does not generate an encoded version of the localized video, but instead reuses some frames of the encoded version of the original video to generate an encoded localization layer. In some embodiments, the localized video encoding application decodes the encoded version of the original video to generate decoded original video (hereinafter "decoded original video"). The localized video encoding application uses the decoded original video as a baseline for generating the encoded localization layer. The localized video encoding application then compares each frame of the localized video with the corresponding frame of the decoded original video.

[0021] If the localized video encoding application determines that there is no discernible difference between a given frame of the localized video and the corresponding frame of the decoded original video, the localized video encoding application instructs the encoding localization layer to reuse the frame of the encoded version of the original video as is (without modification) as the corresponding frame of the encoded version of the localized video. Thus, in this case, the encoding localization layer indirectly instructs the corresponding frame of the decoded original video to be reused as is (without modification) as the corresponding frame of the decoded localized video (hereinafter "decoded localized video").

[0022] On the other hand, if the localized video encoding application determines that there is a discernible difference between a given frame of the localized video and a corresponding frame of the decoded original video, the localized video encoding application divides the frame of the localized video into multiple localized portions (hereinafter "localized portions") and divides the corresponding frame of the decoded original video into multiple original portions. For each localized portion, if the localized video encoding application determines that the corresponding original portion is the "best matched" portion of the localized portion, the localized video encoding application reuses the original portion at a position in the predicted frame that corresponds to the position of the localized portion in the frame of the localized video. The localized video encoding application subtracts the predicted frame from the frame of the localized video to generate a residual frame representing the prediction error. The localized video encoding application also encodes instructions for reconstructing the predicted frame and encodes the residual frame to generate an encoded localization layer frame corresponding to the frame of the localized video. The localized video coding application then adds the coded localization layer frames to the coded localization layer.

[0023] In some embodiments, the multiple encoded video chunks for the original video and the multiple encoded localization layer chunks for the localized video (“encoded localization layer chunks”) are stored on an origin server device. In the same or other embodiments, when streaming the localized video to an end user device, a video delivery application running on the origin server device or edge server device CDN delivers the sequence of encoded video chunks for the original video via one stream and the sequence of encoded localization layer chunks for the localized video via another stream to a playback application running on the end user device. The playback application then generates the sequence of encoded video chunks for the localized video based on the sequence of video chunks for the original video and the sequence of encoded localization layer chunks for the localized video. As the playback application generates various encoded video chunks for the localized video, the video decoder decodes these encoded video chunks to generate corresponding decoded video chunks for the localized video, which are then played back via the end user device.

[0024] One technical advantage of the technology of the present disclosure over the prior art is that it reduces at least the amount of original video data that a localized video encoding application duplicates when encoding localized video. In this sense, because each frame of the encoded localization layer can be specified to reuse any number of portions (including entire frames) of the corresponding frame of the decoded original video, the size of the encoded localization layer can be significantly reduced compared to the size of the encoded version of the localized video. Therefore, by storing encoded localization layer chunks instead of encoded localized video chunks for streaming localized video over a CDN, the total amount of memory used to store the encoded representation of the video library on the origin server device can be significantly reduced. Another technical advantage of the technology of the present disclosure over the prior art is that the total amount of memory used to store the encoded representation of the video library on the origin server device can be reduced, thereby improving CDN caching efficiency and ultimately improving end-user QoE. These technical advantages result in one or more technical improvements over prior art approaches.

[0025] System Overview 1 is a conceptual diagram of a system 100 configured to implement one or more aspects of various embodiments. As shown, in some embodiments, system 100 includes, but is not limited to, compute instances 110(1), ..., compute instance 110(4), a display device 102, and a cloud-based video service 104. For ease of description, compute instances 110(1), ..., compute instance 110(4) may be individually referred to herein as "compute instances 110" and collectively referred to herein as "compute instances 110." In some embodiments, system 100 may include, but is not limited to, any number of compute instances 110, any number of display devices, any number and / or types of cloud-based services, or any combination thereof. In the same or other embodiments, display device 102, cloud-based video service 104, or both, are omitted from system 100.

[0026] Any number of these components of system 100 may be distributed across multiple geographic locations or may be implemented in any combination in one or more cloud computing environments (i.e., encapsulated shared resources, software, data, etc.). In some embodiments, any number of compute instances 110 may be implemented in a cloud computing environment, as part of another distributed computing environment, or standalone.

[0027] As shown, in some embodiments, compute instance 110(1) includes, but is not limited to, processor 112(1) and memory 116(1). In the same or other embodiments, compute instance 110(2) includes, but is not limited to, processor 112(2) and memory 116(2). For ease of explanation, processor 112(1) and processor 112(2) may be individually referred to herein as “processor 112” or collectively referred to as “processors 112.” Memory 116(1) and memory 116(2) may be individually referred to herein as “memory 116” or collectively referred to as “memories 116.” While not shown, compute instance 110(3) and compute instance 110(4) may each include, but are not limited to, any number of processors and any number of memories.

[0028] Each processor 112 may be any instruction execution system, apparatus, or device capable of executing instructions. For example, each processor 112 may comprise a central processing unit, a graphics processing unit, a controller, a microcontroller, a state machine, or any combination thereof. The memory 116 of each compute instance 110 stores content such as software applications and data used by the processor 112 of that compute instance 110. In some embodiments, each compute instance 110 may include any number of processors 112 and any number of memories 116, in any combination. In particular, any number of compute instances 110 (including one) may provide any number of multiprocessing environments in any manner that is technically feasible.

[0029] Each memory 116 may be one or more local or remote digital storage devices and may take the form of any readily available memory (e.g., random access memory, read-only memory, floppy disk, hard disk, etc.). In some embodiments, storage devices (not shown) may supplement or replace any number of memories 116. Storage devices may comprise any number and / or type of external memory accessible to any number of processors 112. For example, without limitation, storage devices may include secure digital cards, external flash memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0030] As shown in italics in the figure, in some embodiments, compute instance 110(3) is an origin server device included in a set of one or more origin server devices (not shown). In some embodiments, the set of origin server devices collectively stores any number of pre-encoded copies of each video in a video library, with at least one pre-encoded copy stored for each video, for streaming to end-user devices directly or through one or more CDNs. In the same or other embodiments, each origin server device included in the set of origin server devices stores one or more pre-encoded chunks of each video in any portion of the video library. In some embodiments, each origin server device may be included in any number of CDNs (including zero).

[0031] Each video may contain any amount and / or type of video data, including, but not limited to, a feature film, a television program, a music video, a podcast, etc. In some embodiments, multiple pre-encoded versions of a video typically correspond to different combinations of average bitrate and resolution, or "bitrate-resolution combinations," and are associated with different average image quality levels. A set of different "bitrate-resolution combinations" for a video is referred to herein as the "encode rate ladder" for that video. Having multiple pre-encoded versions of a video available increases the likelihood that the video can be streamed to an end-user device with good image quality and uninterrupted playback, regardless of the throughput available on the network connection over which it is streamed. A "low-quality" encoded version is typically streamed to an end-user device when the network connection has a relatively low throughput, while a "high-quality" encoded version is typically streamed to an end-user device when the network connection has a relatively high throughput. In some embodiments, each pre-encoded version of a video may be referred to herein as an "encoded video."

[0032] In some embodiments, a video may include, but is not limited to, any number of non-contiguous fragments of video data. Such non-contiguous fragments may be referred to herein as "chunks of the video." In the same or other embodiments, a coded video may include, but is not limited to, any number of non-contiguous fragments of coded video data. Such non-contiguous fragments may be referred to herein as "chunks of the encoded video." In some embodiments, a coded version of a video may include, but is not limited to, different coded video chunks for each video chunk of the video.

[0033] As shown in italics in the figure, in some embodiments, compute instance 110(4) is one of multiple CDN edge server devices within the CDN. A CDN edge server device can retrieve encoded video chunks from one or more origin server devices and receive and respond to requests for encoded video chunks from end-user devices. In the same or other embodiments, each CDN edge server device receives and responds to requests from end-user devices located closer to the CDN edge server device than the origin server device. As noted above, in some embodiments, each CDN edge server device can temporarily store copies of a limited number of encoded video chunks in its associated cache memory. In some embodiments, the CDN also includes any number of other cache memories distributed at intermediate locations throughout the CDN, including, but not limited to, any number of other cache memories.

[0034] In some embodiments, when a CDN edge server device receives a request for an encoded video chunk from an end user device, the CDN edge server device responds by locating the closest copy of the encoded video chunk and sending it to the end user device. In some embodiments, if the CDN edge server device has a copy of the requested encoded video chunk stored in its own cache memory, the request results in a "cache hit," and the CDN edge server device sends the requested copy of the encoded video chunk. However, in the same or other embodiments, if the requested encoded video chunk is not stored in its own cache memory, a "cache miss" occurs. When a cache miss occurs, the CDN server device must retrieve the requested encoded video chunk from an intermediate cache memory or the origin server device before sending it to the end user device.

[0035] As indicated in italics in the figures, in some embodiments, compute instance 110(2) is an end-user device. In some embodiments, compute instance 110(4) can stream video to compute instance 110(2) over a network connection (not shown). In the same or other embodiments, compute instance 110(2) can display video via display device 102. In some embodiments, display device 102 can be any type of device that can be configured to display any amount and / or type of video data in any technically feasible manner. In the same or other embodiments, compute instance 110(2), zero or more other compute instances 110, display device 102, and zero or more other display devices are incorporated into a user device (not shown). Examples of user devices include, but are not limited to, desktop computers, laptops, smartphones, smart TVs, game consoles, tablets, etc.

[0036] In some embodiments, cloud-based video service 104 includes, but is not limited to, microservices, databases, and storage for streaming video service-related activity and content that is not allocated to an origin server device, a CDN, or an end-user device. Examples of functionality that may be provided by cloud-based video service 104 include, but are not limited to, login and billing, logging, personalized video title recommendations, video transcoding, server and connection health monitoring, and client-specific CDN guidance. In some embodiments, cloud-based video service 104 monitors the health of compute instance 110(4), compute instance 110(2), and their associated network connections.

[0037] Each compute instance 110 is configured to implement one or more software applications. For purposes of illustration, each software application is illustrated as residing in the memory 116 of a single compute instance 110 and executing on the processor 112 of that single compute instance 110. However, one skilled in the art will recognize that the functionality of each software application may be distributed across any number of other software applications, which may reside in the memory 116 of any number of compute instances 110 and execute on the processors 112 of any number of compute instances 110, in any combination. Additionally, the functionality of any number of software applications may be integrated into a single application or subsystem.

[0038] In particular, in some embodiments, the compute instance 110(1) is configured to encode original video and localized videos derived from the original video through a video localization process. Video localization involves modifying one or more copies of the original video to generate one or more localized videos, each of which is tailored to a different target audience. Typically, a localized video contains a majority of the original video data unchanged from the corresponding original video, with the remaining video data being localized video data customized for the target audience of the localized video. As used herein, the "target audience" of a localized video may refer to, but is not limited to, end users who are likely to select the localized video based on any number and / or type of criteria. For example, the target audience of a localized video that has been modified to accommodate a voice-over in a given language may include, but is not limited to, end users who are likely to select the given language from among multiple available languages. As detailed above, typical video streaming services do not distinguish between localized and original video when streaming video to end-user devices, especially when using conventional encoders to encode chunks of localized video to generate encoded video chunks of localized video.

[0039] One problem with encoding localized video chunks using conventional encoders is that the original video data duplicated in a chunk of a localized video is typically encoded once to generate the encoded video chunks for the original video and then encoded again to generate the encoded video chunks for the localized video. This results in the total size of a conventionally encoded localized video being comparable to the size of the corresponding encoded original video, and thus excessively large relative to the amount of localized video data present in the encoded localized video. This results in an unnecessarily large amount of memory being used to store the encoded video chunks for a localized video. Furthermore, because the size of the cache memory used by edge servers is typically significantly smaller than the total size of all encoded video chunks associated with a streamable video library stored on an origin server, the efficiency of the CDN cache, and therefore the end-user QoE, often degrades excessively relative to the amount of localized video data present in the encoded localized video.

[0040] To reduce the amount of original video data that is over-encoded when encoding localized video, some implementations simply do not encode localized video chunks that are identical to original video chunks. Instead, they use metadata to indicate that the corresponding encoded original video chunks should be reused as the encoded localized video chunks. However, this approach still suffers from the problem that many localized video chunks often have very slight differences from their corresponding original video chunks. In such cases, a large portion of the original video data is still over-encoded when encoding the localized video chunks.

[0041] Encode the difference between the localized video and the original video To address the above issues, in some embodiments, compute instance 110(1) includes, but is not limited to, a localized video encoding application 120. As described in more detail below, in some embodiments, localized video encoding application 120 does not encode video data of the localized video, but encodes differences between the video data of the localized video and the corresponding video data of the decoded original video to generate an encoded localization layer. The generated encoded localization layer can then be combined with the encoded original video and decoded to generate the decoded localized video. In some embodiments, each encoded chunk included in the encoded localization layer can be combined with a corresponding encoded chunk included in the encoded original video and decoded to generate the corresponding video chunk included in the decoded localized video.

[0042] For ease of explanation, this specification may refer to a video chunk included in an original video as an "original video chunk," and a video chunk included in a localized video as a "localized video chunk." Furthermore, this specification may refer to a coded video chunk included in an encoded original video as an "encoded original video chunk," a coded video chunk included in an encoded localization layer as an "encoded localization layer chunk," and a coded video chunk included in an encoded localized video as an "encoded localized video chunk." Furthermore, this specification may refer to a video chunk included in a decoded original video as a "decoded original video chunk," and a video chunk included in a decoded localized video as a "decoded localized video chunk."

[0043] As described in more detail below, in some embodiments, compute instance 110(2) includes, but is not limited to, a playback application 170. In some embodiments, playback application 170 constructs encoded localized video chunks based on the encoded localization layer chunks and the encoded original video chunks. Playback application 170 then decodes the encoded localized video chunks to generate decoded localized video chunks. In some embodiments, playback application 170 sequentially displays the decoded localized video chunks on display device 102 to play the corresponding localized video.

[0044] This has the advantage that the amount of original video data redundantly coded by the localized video coding application 120 when encoding localized video can be significantly reduced compared to the amount of original video data redundantly coded by conventional encoders when encoding localized video. Therefore, the size of the encoded localization layers for localized video can be significantly reduced compared to the encoded versions of the localized video. Furthermore, when multiple localized videos are derived from the same original video, the corresponding multiple encoded localization layers can be combined and decoded with the encoded original video. Therefore, each time a localized video is added, the amount of additional memory used by the origin server device to store the encoded video data can be reduced from the size of the encoded version of the localized video to the size of the encoded localization layers corresponding to the localized video. This results in improved cache efficiency for the associated CDN and ultimately improved end-user QoE.

[0045] As shown, in some embodiments, localized video encoding application 120 resides in memory 116(1) of compute instance 110(1) and executes on processor 112(1) of compute instance 110(1). Also, in the same or other embodiments, playback application 170 resides in memory 116(2) of compute instance 110(2) and executes on processor 112(2) of compute instance 110(2). In other embodiments, any number of portions, including all of the functionality described herein for localized video encoding application 120 and playback application 170, may be distributed across any number of compute instances in any technically feasible manner. Also, in some embodiments, playback application 170 may be omitted from system 100.

[0046] For ease of explanation, the functionality of some embodiments of the localized video encoding application 120 will be described in terms of generating one encoded version of the original video 122 and generating different encoded localization layers for each of the localized videos 124(1), ..., 124(K), where K can be any positive integer. In the same or other embodiments, the localized video encoding application 120 encodes the original video 122 and the localized videos 124(1), ..., 124(K) using the same set of encoding parameter values.

[0047] However, those skilled in the art will recognize that the techniques described herein for the localized video encoding application 120 and playback application 170 are illustrative and not intended to be limiting, and that modifications may be made without departing from the spirit and scope of the present invention broadly construed. Many modifications and variations of the functionality of the localized video encoding application 120 and playback application 170 described herein will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described herein. For example, in some embodiments, the localized video encoding application 120 may generate J encoded versions of the original video 122 and J encoded localization layers for each of the localized videos 124(1), ..., 124(K) based on J sets of encoding parameter values, where J is an integer greater than 1.

[0048] As shown, in some embodiments, localized video encoding application 120 generates encoded original video 132 and encoded localization layers 150(1), ..., 150(K) based on original video 122 and localized videos 124(1), ..., 124(K). For ease of explanation, localized videos 124(1), ..., 124(K) may be individually referred to herein as "localized video 124" and collectively referred to herein as "localized videos 124" and "localized videos 124(1)-124(K)." Furthermore, in this specification, the encoded localization layers 150(1), ..., encoded localization layers 150(K) may be individually referred to as "encoded localization layers 150" or collectively referred to as "encoded localization layers 150" and "encoded localization layers 150(1) to 150(K)."

[0049] As shown, in some embodiments, localized video encoding application 120 includes, but is not limited to, video encoder 130, encoded original video 132, localization encoders 140(1), ..., localization encoder 140(K), and encoded localization layer 150. Localization encoders 140(1), ..., localization encoder 140(K) are different instances of a single localization encoder (which may be referred to herein as a "localization encoder 140"). For ease of explanation, localization encoders 140(1), ..., localization encoder 140(K) may be individually referred to herein as a "localization encoder 140" or collectively referred to herein as "localization encoders 140" and "localization encoders 140(1)-140(K)."

[0050] As shown, in some embodiments, video encoder 130 encodes original video 122 to generate encoded original video 132. Thus, encoded original video 132 is an encoded version of original video 122. For an integer k between 1 and K, localization encoder 140(k) generates encoded localization layer 150(k) based on localized video 124(k) and encoded original video 132. As described in more detail below with respect to FIG. 2, in some embodiments, localization encoder 140(k) can optionally obtain and use any amount and / or type of additional data to identify localized video data. For example, in some embodiments, localized video encoding application 120 inputs original video 122, metadata generated by the localization application in generating localized video 124(k), or both, to localization encoder 140(k).

[0051] Localization encoder 140 may implement any number and / or types of encoding techniques and any number and / or types of decoding techniques to generate encoded localization layer 150. In some embodiments, localization encoder 140 may implement scalable encoding techniques to generate encoded localization layer 150. Scalable encoding may encode modified versions of original video frames in the form of differences between the original and modified video frames. Examples of codecs that implement scalable encoding techniques for different encoded versions of a video include, but are not limited to, the AV1 (AOMedia Video 1) codec, some H.245 / HEVC (High Efficiency Video Coding) codecs, and some H.264 / AVC (Advanced Video Coding) codecs. In contrast, localization encoder 140 implements scalable encoding techniques for different but related encoded versions of video.

[0052] For ease of explanation, the following description of the functionality of localization encoder 140 in some embodiments will use localization encoder 140(1) as an example. As described in more detail below with respect to FIG. 2, in some embodiments, localization encoder 140(1) decodes encoded original video 132 to generate decoded original video (not shown in FIG. 1). Localization encoder 140(1) uses frames of the decoded original video as base data for generating frames of encoded localization layer 150(1) according to frames of localized video 124(1). In the same or other embodiments, each frame of encoded localization layer 150(1) specifies, without limitation, encoded predictive metadata (hereinafter "encoded predictive metadata") and, optionally, encoded residual data (hereinafter "encoded residual data"). In some embodiments, the encoded predictive metadata instructs the video decoder to copy one or more portions of corresponding frames of the decoded original video into the reconstructed predicted frame (optionally, to move the one or more portions relative to the reconstructed predicted frame). In the same or other embodiments, the encoded residual data specifies modifications the video decoder needs to make to the reconstructed predicted frame to generate the frame of decoded localized video. In some embodiments, the decoded localized video is a less-than-perfect replica of localized video 124(1).

[0053] Although not shown, in some embodiments, any software application may generate any portion of the encoded localized video based on a corresponding portion of the encoded original video and a corresponding portion of the encoded localized layer corresponding to that portion of the encoded localized video. For ease of explanation, this specification will use the process of generating a chunk of the encoded localized video 124(1) (“encoded localized video chunk”) as an example to describe the process of generating a portion of the encoded localized video. However, the same techniques may be applied to generate any portion of the encoded localized video based on a corresponding portion of the encoded original video and a corresponding portion of the encoded localization layer corresponding to that portion of the encoded localized video.

[0054] Although not shown, in some embodiments, any software application or any hardware module, such as localization encoder 140, localized video encoding application 120, or playback application 170, may execute a "layer interleaving algorithm" to generate any portion of the encoded localized video (including the entire video). In the same or other embodiments, the layer interleaving algorithm specifies a set of instructions for generating any portion of the encoded localized video based on, but not limited to, a corresponding portion of the encoded original video corresponding to that portion of the encoded localized video, a corresponding portion of the encoded localization layer corresponding to that portion of the encoded localized video, and optionally any amount and / or type of metadata.

[0055] In some embodiments, a software application or hardware module executes a layer interleaving algorithm to generate encoded localized video chunks based on encoded original video chunks associated with original video chunks and encoded localization layer chunks associated with localized video chunks. In the same or other embodiments, the techniques of this disclosure can be used to generate any number of portions of encoded localized video of a localized video based on portions of encoded original video corresponding to any number (including one) of the encoded localized video and portions of encoded localization layers corresponding to any number of portions of the encoded localized video.

[0056] In some embodiments, to generate the encoded localization video chunks, a software application or hardware module identifies frames of the localization video chunks that have differences relative to corresponding frames of the original video chunks. The software application or hardware module may determine whether a frame of the localization video chunk has differences relative to corresponding frames of the original video chunks in any technically feasible manner. For example, in some embodiments, the software application or hardware module may determine whether a frame of the localization video chunk has differences relative to corresponding frames of the original video chunks based on a frame of the encoded localization layer chunk that corresponds to a frame of the localization video chunk, any amount and / or type of metadata, or any combination thereof.

[0057] In some embodiments, a software application or hardware module generates a coded base layer (hereinafter "coded base layer") by, for each frame identified in the localization video chunk, marking the corresponding frame of the coded original video chunk as a reference-only frame. Thus, each frame included in the coded base layer is a selectively marked copy of a frame included in the coded original video chunk. As used herein, a "reference-only frame" refers to a frame that can be used as a reference for other frames but is not displayed or presented. The software application or hardware module then interleaves the frames of the coded base layer with frames of the coded localization layer chunk to generate the coded localization video chunk.

[0058] As shown, in some embodiments, localized video encoding application 120 transmits encoded original video 132 and each encoded localization layer 150 separately to any number of origin server devices (e.g., compute instance 110(3)). In the same or other embodiments, any number of CDN edge server devices (e.g., compute instance 110(4)) can stream encoded original video 132 and each encoded localization layer 150 separately to any number of end-user devices (e.g., compute instance 110(2)).

[0059] In some embodiments, when streaming one of the localized videos 124 to an end user device (e.g., compute instance 110(2)), the CDN edge server device delivers two separate streams to the end user device: an encoded original video stream and an encoded localization layer stream. In some embodiments, the CDN edge server device delivers chunks of the encoded original video 132 via the encoded original video stream and chunks of the encoded localization layer 150 corresponding to the localized video 124 via the encoded localization layer stream.

[0060] Although not shown, in some other embodiments, chunks of encoded original video 132 and chunks of encoded localization layer 150 corresponding to the localized video 124 may be delivered to the end user device in one stream or two separate streams by one or two edge server devices, one or two origin server devices, one or two other devices, or any combination thereof, in any technically feasible manner. In some embodiments, encoded original video 132 may be downloaded to the end user device at some point before encoded localization layer 150 is streamed to the end user device, and the techniques described herein may be modified accordingly.

[0061] For purposes of explanation, Figure 1 illustrates an example of high-level events performed by playback application 170 to play localized video 124(1). In some embodiments, playback application 170 opens one or more network connections to cloud-based video service 104, thereby allowing an end user of playback application 170 to select videos to stream to compute instance 110(2). In the same or other embodiments, playback application 170 can send or receive any amount and / or type of data (including none) to or from cloud-based video service 104 in any manner that is technically feasible.

[0062] In some embodiments, end user or playback application 170 selects localized video 124(1) to stream in any technically feasible manner. For example, in some embodiments, playback application 170 may select localized video 124(1) to stream based on any number and / or types of end user preferences, any number and / or types of end user favorites, the language selected by the end user when playing the previously selected localized video, or any combination thereof. In some embodiments, once end user or playback application 170 selects localized video 124(1) to stream, playback application 170 sends a manifest request to cloud-based video service 104 specifying localized video 124(1). In response, cloud-based video service 104 creates manifest file 172 based on localized video 124(1), optionally a CDN, and optionally compute instance 110(2). Cloud-based video service 104 then sends manifest file 172 to playback application 170.

[0063] In some embodiments, manifest file 172 specifies, without limitation, a bitrate and resolution combination, an average quality level, and location data associated with the encoded version of localized video 124(1). In the same or other embodiments, the location data specifies, without limitation, the location of encoded original video chunks for original video 122 and the location of encoded localization layer chunks for localized video 124(1) on each of one or more CDN edge server devices proximate to compute instance 110(2). As noted above, the encoded original video chunks for original video 122 are chunks of encoded original video 132. The encoded localization layer chunks for localized video 124(1) are chunks of encoded localization layer 150(1). The one or more CDN edge server devices proximate to compute instance 110(2) include, without limitation, compute instance 110(4).

[0064] In some embodiments, playback application 170 selects a sequence of encoded original video chunks and a corresponding sequence of encoded localization layer chunks to send from compute instance 110(4) to compute instance 110(2) based on manifest file 172. The order of the selected sequence of encoded original video chunks corresponds to the display order of the video chunks (and frames within the video chunks) of original video 122, and the order of the selected sequence of encoded localization layer chunks corresponds to the display order of the video chunks (and frames within the video chunks) of localized video 124(1).

[0065] For ease of explanation, Figure 1 illustrates an example period, but in some embodiments, the period illustrated in Figure 1 may begin when playback application 170 selects the location of encoded original video chunk 186 according to the selected sequence of encoded original video chunks and the location of encoded localization layer chunk 188 according to the selected sequence of encoded localization layer chunks. Encoded original video chunk 186 and encoded localization layer chunk 188 are chunks corresponding to the same localized video chunk.

[0066] As shown, playback application 170 issues encoded video chunk request 182 and encoded layer chunk request 184 to a video distribution application (not shown) running on compute instance 110(4). In some embodiments, encoded video chunk request 182 requests bytes corresponding to encoded original video chunk 186. In the same or other embodiments, encoded layer chunk request 184 requests bytes corresponding to encoded localization layer chunk 188.

[0067] In response to encoded video chunk request 182 and encoded layer chunk request 184, in some embodiments, the video distribution application obtains encoded original video chunk 186 and encoded localization layer chunk 188 and transmits them in separate streams. In other embodiments, the video distribution application provides encoded original video chunk 186 and encoded localization layer chunk 188 in a single stream (not shown) in any technically feasible manner. As playback application 170 receives encoded original video chunk 186 and encoded localization layer chunk 188 incrementally, it stores the encoded original video chunk 186 and encoded localization layer chunk 188 incrementally in playback buffer 174.

[0068] Then, in some embodiments, as playback application 170 receives and / or stores frames of encoded original video chunks 186 and encoded localization layer chunks 188 in playback buffer 174, playback application 170 executes the layer interleaving algorithm described above incrementally to build encoded localized video chunks 176 incrementally. Also, in the same or other embodiments, playback application 170 incrementally uses multi-layer video decoder 178 to decode encoded localized video chunks 176 incrementally, thereby generating decoded localized video chunks 190 incrementally.

[0069] Multi-layer video decoder 178 may include, but is not limited to, zero or more software applications, zero or more hardware modules, or any combination thereof, which cooperate to decode encoded localized video chunks 176 to generate decoded localized video chunks 190. In some embodiments, multi-layer video decoder 178 decodes associated multiple-layer encoded video data represented in any technically feasible manner by implementing scalable decoding techniques, any number and / or types of other decoding techniques, or both. For example, in some embodiments, multi-layer video decoder 178 supports the Scalable Video Coding (SVC) extension of H.264 / AVC.

[0070] In some embodiments, playback application 170 incrementally renders decoded localized video chunk 190 and zero or more other decoded localized video chunks (not shown) to display device 102 in a display order associated with localized video 124(1), thereby playing localized video 124(1) chunk by chunk. In the same or other embodiments, playback application 170 may incrementally render and play frames of a partially generated version of decoded localized video chunk 190 before receiving all frames of encoded original video chunk 186 and / or all frames of encoded localization layer chunk 188.

[0071] It should be noted that the techniques described herein are illustrative and not limiting, and that modifications may be made without departing from the spirit and scope of the present invention broadly construed. Many modifications and variations of the functionality of the localized video encoding application 120, localization encoder 140, and playback application 170 described herein will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described herein. Similarly, many modifications and variations of the encoding original video, encoding localization layer, and storing and delivering encoded localized video described herein will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described herein.

[0072] For example, in some embodiments, original video 122 may include, but is not limited to, any amount of basic video data that is not intended for direct presentation to an end user. Alternatively, original video 122 may be used to generate localized video 124, and decoded encoded original video 132 may be used to generate encoded localization layer 150. Note that encoded original video 132 and encoded localization layer 150 are not decoded and presented independently, but rather are used to generate at least a portion of localized encoded video (e.g., encoded localized video chunk 176), which is then decoded and presented.

[0073] In particular, in some embodiments, the original video 122 is configured to be a version of the original language video with the lip portion blurred, and one of the plurality of localized videos 124 is configured to be the original language video. In the same or other embodiments, each of the coded localization layers 150 (including the coded localization layer 150 corresponding to the original language video) is coded to add lip appearance data for the corresponding language to the original video 122, but does not need to be coded to remove lip appearance data for the original language from the original video 122. Thus, the number of differential pixels can be reduced compared to an embodiment in which the original video is the original language video.

[0074] In some embodiments, the original video 122 is configured to be a version of the original language video with, but not limited to, blanked text bubbles, and one of the localized videos 124 is configured to be the original language video. In the same or other embodiments, each of the coded localization layers 150 (including the coded localization layer 150 corresponding to the original language video) is coded to add text in the corresponding language to the text bubbles, but does not need to be coded to remove the original language from the text bubbles in the original video 122. Thus, the number of pixels of difference can be reduced compared to an embodiment in which the original video is the original language video.

[0075] It will be understood that the system 100 illustrated herein is illustrative and that variations and modifications are possible. For example, the functionality provided by the localized video encoding application 120, localization encoder 140, playback application 170, and cloud-based video service 104 described herein may be integrated or distributed across any number of software applications (including one) and any number of components of the system 100. Furthermore, the connection topology between the various units illustrated in FIG. 1 may be modified as desired.

[0076] Figure 2 illustrates in more detail one of the localization encoders 140 shown in Figure 1, according to various embodiments. More precisely, Figure 2 illustrates localization encoder 140(1), which, in some embodiments, generates encoded localization layer 150(1) based on localized video 124(1) and encoded original video 132. As noted above in the discussion of Figure 1, localized video 124(1) is a modified version of original video 122. Localized video 124(1) can be derived from original video 122 in any technically feasible manner.

[0077] Although not shown, in some embodiments, the original video 122 includes a frame sequence consisting of, but not limited to, N frames, where N can be any positive integer. For ease of explanation, the frames of the original video 122 are sometimes referred to individually as "original video frames" or collectively as "original video frames." Each original video frame is an image that includes, but is not limited to, any amount and / or type of video data. The video data included in the original video frame is sometimes referred to herein as "original video data." For ease of explanation, each original video frame is associated with a frame index. The frame index is a number from 1 to N that is assigned according to a display order, which is the order in which the frames are displayed sequentially during playback. More precisely, during normal playback of the original video 122, the original video frames corresponding to frame indexes "1" through "N" are displayed sequentially.

[0078] As shown, in some embodiments, localized video 124(1) includes, but is not limited to, a frame sequence of localized video frames 224(1), ..., 224(N), where N is the total number of original video frames. In the same or other embodiments, localized video frames 224(1), ..., 224(N) are modified versions of, and therefore correspond to, original video frames with frame index "1" ..., original video frames with frame index "N," respectively. For purposes of illustration, localized video frames 224(1), ..., 224(N) are associated with frame indices 1 through N, respectively. In some embodiments, the number of frames in localized video 124(1) may be different from the number of frames in the original video, and the techniques described herein may be modified accordingly.

[0079] For ease of explanation, localized video frames 224(1), ..., localized video frames 224(N) may be individually referred to herein as "localized video frames 224" and collectively referred to herein as "localized video frames 224" and "localized video frames 224(1)-224(N)." In some embodiments, each localized video frame 224 is a frame of localized video 124(1) that includes, but is not limited to, any amount of original video data that matches the original video data in the corresponding original video frame, any amount of localized video data that is modified from the original video data in the corresponding original video frame, or both.

[0080] Encoded original video 132 is an encoded version of original video 122. Encoded original video 132 may be generated in any technically feasible manner. As noted above in the description of FIG. 1 , in some embodiments, video encoder 130 encodes original video 122 to generate encoded original video 132. As shown, in some embodiments, encoded original video 132 includes, but is not limited to, encoded original video frames 232(1), ..., encoded original video frame 232(N).

[0081] The encoded original video frames 232(1), ..., 232(N) are encoded versions of, and therefore correspond to, the original video frames with frame index "1" ..., and frame index "N," respectively. For ease of explanation, the encoded original video frames 232(1), ..., 232(N) are associated with frame indices 1 through N, respectively. In this specification, the encoded original video frames 232(1), ..., 232(N) may be individually referred to as "encoded original video frames 232" or collectively referred to as "encoded original video frames 232" and "encoded original video frames 232(1) through 232(N)."

[0082] As shown, in some embodiments, localization encoder 140(1) includes, but is not limited to, a video decoder 202, a decoded original video 210, a frame comparison engine 240, a modified frame index list 250, comparison metadata 252, an incremental encoding engine 260, a localization frame encoder 270(1), ..., a localization frame encoder 270(M), and an encoded localization layer 150(1). In some embodiments, M can be any integer between 1 and N, inclusive.

[0083] In some embodiments, localization frame encoder 270(1), ..., localization frame encoder 270(M) are different instances of a localization frame encoder (which may be referred to herein as "localization frame encoder 270"). For ease of explanation, localization frame encoder 270(1), ..., localization frame encoder 270(M) may be individually referred to herein as "localization frame encoder 270" or collectively referred to herein as "localization frame encoders 270" and "localization frame encoders 270(1)-270(M)."

[0084] As shown, video decoder 202 decodes encoded original video 132 to generate decoded original video 210. Video decoder 202 can be any video decoder or any part of a codec capable of decoding encoded original video 132. In some embodiments, video decoder 202 decodes single-layer encoded video data by implementing any number and / or type of decoding techniques. In the same or other embodiments, video decoder 202 decodes multiple-layer encoded video data represented in any technically feasible manner by implementing any number and / or type of decoding techniques. In some embodiments, video decoder 202 is an instance of any multi-layer video encoder, such as multi-layer video decoder 178 shown in FIG. 1. In the same or other embodiments, video decoder 202 supports the SVC extension of H.264 / AVC.

[0085] As shown, the decoded original video 210 includes, but is not limited to, decoded original video frames 212(1), ..., 212(N). The decoded original video frames 212(1), ..., 212(N) are decoded versions of the encoded original video frames 232(1)-232(N), respectively, and thus correspond to the original video frame with frame index "1" ..., 212(N), respectively. For ease of explanation, the decoded original video frames 212(1), ..., 212(N) are associated with frame indices 1 through N, respectively. In this specification, the decoded original video frames 212(1), ..., decoded original video frames 212(N) may be individually referred to as "decoded original video frames 212" or collectively referred to as "decoded original video frames 212" or "decoded original video frames 212(1)-212(N)."

[0086] As those skilled in the art will recognize, video encoders typically implement lossy compression techniques, and the decoded video data may contain any amount and / or type of noise due to compression. When the decoded video data is displayed, such noise may result in visible distortions known as "encoding artifacts." Thus, in some embodiments, the decoded original video frames 212 are less-than-perfect replicas of the corresponding original video frames, and the decoded original video 210 is a less-than-perfect replica of the original video 122.

[0087] As shown, in some embodiments, frame comparison engine 240 generates modified frame index list 250 and comparison metadata 252 based on localized video 124(1) and decoded original video 210. In some embodiments, modified frame index list 250 specifies, but is not limited to, frame indices of a subset of localized video frames 224(1)-224(N) that have differences relative to corresponding frames in decoded original video frames 212(1)-212(N). In the same or other embodiments, modified frame index list 250 specifies, but is not limited to, frame indices of a subset of localized video frames 224(1)-224(N) that contain visible modifications resulting from the video localization process. For ease of explanation, in some embodiments, modified frame index list 250 includes, but is not limited to, M frame indices. These frame indices are represented herein as idx1-idxM in display order. Here, 1≦idx1≦idxM≦M≦N.

[0088] In some embodiments, comparison metadata 252 identifies, without limitation, any amount and / or type of original video data included in any portion (including entire frames) of localized video frames 224(1)-224(N). In the same or other embodiments, comparison metadata 252 identifies, without limitation, any number and / or type of differences between any portion (including entire frames) of localized video frames 224(1)-224(N) and the corresponding portion of decoded original video frames 212(1)-212(N). For example, without limitation, in some embodiments, comparison metadata 252 specifies frame indices corresponding to a subset of localized video frames 224 that do not include localized video data, and additionally or alternatively specifies the locations of one or more non-contiguous localized video data fragment portions for each of any number of localized video frames 224 included in the subset of localized video frames 224 that include localized video data.

[0089] Frame comparison engine 240 may identify any number and / or type of differences in localized video frames 224 and / or detect localized video data in any technically feasible manner. Frame comparison engine 240 may generate modified frame index list 250, comparison metadata 252, any amount and / or type of other comparison data, or any combination thereof, in any technically feasible manner based on any number and / or type of differences associated with localized video frames 224, localized video data, or both.

[0090] In some embodiments, the frame comparison engine 240 compares the localized video frame 224(x) with the decoded original video frame 212(x) for frame index x ranging from "1" to "N" to determine whether to add the frame index x to the modified frame index list 250. In the same or other embodiments, the frame comparison engine 240 also calculates pixel-by-pixel differences between the localized video frame 224(x) and the decoded original video frame 212(x) to identify a "modification" frame associated with the localized video frame 224(x). In some embodiments, the frame comparison engine 240 can implement any minimum difference criteria (e.g., minimum pixel distance value, minimum modification size, etc.) and can filter out or ignore any differences that do not meet the minimum difference criteria.

[0091] In some embodiments, frame comparison engine 240 may perform any number and / or types of operations based on any amount and / or type of data in any technically feasible manner to distinguish differences corresponding to localized video data from differences caused by noise or coding artifacts. Additionally, in the same or other embodiments, frame comparison engine 240 may ignore or filter out differences caused by noise or coding artifacts. In some embodiments, frame comparison engine 240 may generate modified frame index list 250, comparison metadata 252, any amount and / or type of other output data, or any combination thereof, based on the differences corresponding to localized video data (i.e., differences caused by the video localization process).

[0092] Although not shown, in some embodiments, frame comparison engine 240 can compare localized video 124(1) to original video 122 rather than decoded original video 210 to identify differences that correspond to localized video data. In the same or other embodiments, frame comparison engine 240 can use any amount and / or type of metadata to identify differences that correspond to localized video data. For example, in some embodiments, when a localization application modifies original video 122 to generate localized video 124(1), it can also generate metadata that specifies the modifications made to each original video frame. Then, in the same or other embodiments, frame comparison engine 240 can filter out or ignore any differences that do not correspond to modifications made to the original video frames according to any amount and / or type of metadata.

[0093] As shown, in some embodiments, incremental encoding engine 260 generates encoded localization layer 150(1) based on decoded original video 210, localized video 124(1), modified frame index list 250, and comparison metadata 252. In some embodiments, encoded localization layer 150(1) includes, but is not limited to, any amount (including zero) and / or any encoded metadata (hereinafter "encoded metadata") 256 and encoded localization layer frames 258(1), ..., encoded localization layer frames 258(N), where N is the total number of frames included in localized video 124(1).

[0094] In some embodiments, the encoding metadata 256 may indicate that any number of the decoded original video frames 212 are to be reused as is (without modification) as part of the decoded localized video (not shown). In the same or other embodiments, the encoded localization layer frames 258(1), ..., 258(N) are encoded versions of the localized video frames 224(1) through 224(N), respectively. For purposes of illustration, the encoded localization layer frames 258(1), ..., 258(N) are associated with frame indices 1 through N, respectively. Additionally, in this specification, the encoded localization layer frames 258(1), ..., encoded localization layer frames 258(N) may be individually referred to as "encoded localization layer frames 258" or collectively referred to as "encoded localization layer frames 258" and "encoded localization layer frames 258(1) to 258(N)."

[0095] While not shown, in some embodiments, each encoded localization layer frame 258 specifies, without limitation, any amount of encoding predictive metadata (including zero) and any amount of encoding residual data (including zero). In some embodiments, the encoding predictive metadata included in encoded localization layer frame 258(x) instructs a video decoder to copy one or more portions of decoded original video frame 212(x) into a reconstructed predicted frame (and, optionally, move the one or more portions relative to the reconstructed predicted frame), where x can be any integer from 1 to N. In the same or other embodiments, the encoded residual data also specifies modifications the video decoder needs to make to the reconstructed predicted frame to generate a decoded localization video frame at frame index "x" in a decoded localization video (not shown). In some embodiments, the decoded localization video frame at frame index "x" is a less-than-perfect replica of localization video frame 224(x).

[0096] In some other embodiments, one or more coded localization layer frames 258 may be omitted from coded localization layer 150(1), with the omission indicating that the frame of decoded original video 210 corresponding to each omitted coded localization layer frame 258 should be reused (without modification) as the corresponding decoded localized video frame in the decoded localized video. For example, if coded localization layer 150(1) does not include coded localization layer frame 258 with frame index "x," then coded localization layer 150(1) implicitly indicates to the video decoder that the decoded localized video frame with frame index "x" should be decoded original video frame 212(x). Note that the techniques described herein can be modified to account for the omission of any number of coded localization layer frames 258.

[0097] As detailed in the description of Figure 1 above, in some embodiments, any software application or any hardware module, such as localization encoder 140, localized video encoding application 120, or playback application 170, may perform a layer interleaving algorithm to generate any portion of the encoded localized video (including the entire video). In the same or other embodiments, the layer interleaving algorithm may also be used to generate any portion of the encoded localized video based on corresponding portions of encoded original video 132, corresponding portions of encoded localization layer 150(1), and optionally any amount and / or type of metadata. In some embodiments, any portion of the encoded localized video may be decoded to generate a corresponding portion of the decoded localized video.

[0098] In some embodiments, any portion of the encoded localization layer 150(1) can be used in combination with a corresponding portion of the decoded original video 210 to generate a corresponding portion of the decoded localized video. In the same or other embodiments, any portion of the encoded localization layer 150(1) can be combined with a corresponding portion of the encoded original video 132 and decoded to generate a corresponding portion of the decoded localized video.

[0099] In some embodiments, the incremental encoding engine 260 incrementally generates the encoded localization layer 150(1) bit by bit by sequentially encoding the localized video frames 224 according to an "encode / decode" order. In the same or other embodiments, each of the encoded localization layer frames 258 is a predicted frame (P-frame). In general, during encoding, if any portion of the decoded localized video frames or the decoded original video frames 212 that is later in display order can be used to "backward predict" a predicted frame, the code / decode order may differ from the display order. In the same or other embodiments, of the encoded localization layer frames 258, zero or more of the encoded localization layer frames 258 are intra-coded frames (I-frames), zero or more of the encoded localization layer frames 258 are P-frames, and zero or more of the encoded localization layer frames 258 are bidirectional predicted frames (B-frames).

[0100] For ease of explanation, the functionality of the incremental encoding engine 260 will be described herein in connection with some embodiments in which there are no backward predicted frames, and therefore the encoding / decoding order matches the display order of the localized video 124(1). However, in other embodiments, any number and / or types of portions of each predicted frame may be predicted using intra-prediction, forward-prediction, backward-prediction, or any combination thereof, resulting in a coding / decoding order that may differ from the display order, and the techniques described herein may be modified accordingly.

[0101] In some embodiments, incremental encoding engine 260 may initialize encoded localization layer 150(1) based on any amount and / or type of data and in any manner that is technically feasible. For example, in some embodiments, incremental encoding engine 260 may specify any amount and / or type of encoding metadata 256 that identifies any number and / or type of characteristics of localized video 124(1), any amount and / or type of encoding-related information, etc.

[0102] In the same or other embodiments, the incremental encoding engine 260 sequentially encodes localized video frames 224(1)-224(N) to sequentially generate encoded localization layer frames 258(1)-258(N). For ease of explanation, the functionality of the incremental encoding engine 260 in some embodiments will be described using the process of encoding localized video frame 224(x), where x can be any integer between 1 and N.

[0103] In some embodiments, if localized video frame 224(x) is not included in modified frame index list 250, incremental encoding engine 260 instructs encoding localization layer 150(1) to reuse decoded original video frame 212(x) as is (without modification) as the decoded localized video frame for frame index "x." Incremental encoding engine 260 can instruct encoding localization layer 150(1) to reuse decoded original video frame 212(x) as is (without modification) with any number of modifications (including zero).

[0104] In some embodiments, the incremental encoding engine 260 generates an encoded localization layer frame 258(x) that specifies that the decoded original video frame 212(x) be reused as is (without modification) as the decoded localization video frame for frame index "x" and appends the encoded localization layer frame 258(x) to the encoded localization layer 150(1). The incremental encoding engine 260 may specify the reuse of the decoded original video frame 212(x) via the encoded localization layer frame 258(x) in any technically feasible manner.

[0105] In some embodiments, to specify the reuse of the decoded original video frame 212(x) via the encoded localization layer frame 258(x), the incremental encoding engine 260 generates one or more "skip" instructions and / or any amount and / or type of skip metadata indicating that the encoded localization video frame at frame index "x" is the same as the encoded original video frame at frame index "x." The incremental encoding engine 260 then encodes the skip instructions and / or skip metadata to generate the encoding prediction metadata. The incremental encoding engine 260 then generates the encoded localization layer frame 258(x) including, but not limited to, the encoding prediction metadata 256.

[0106] In some embodiments, if modified frame index list 250 does not include frame index "x," incremental encoding engine 260 does not generate encoded localization layer frame 258 for frame index "x." Thus, in that case, encoded localization layer 150(1) does not include an encoded localization layer frame for frame index "x." In the same or other embodiments, omitting the encoded localization layer frame for frame index "x" from encoded localization layer 150(1) implicitly indicates that decoded original video frame 212(x) should be reused exactly (without modification) as the decoded localized video frame for frame index "x."

[0107] On the other hand, as shown, in some embodiments, if modified frame index list 250 includes frame index "x," incremental encoding engine 260 causes an instance of localization frame encoder 270 to perform one or more encoding operations on localized video frame 224(x) to generate encoded localization layer frame 258(x). Then, in the same or other embodiments, instance of incremental encoding engine 260 or localization frame encoder 270 appends encoded localization layer frame 258(x) to encoded localization layer 150(1).

[0108] As noted above, in some embodiments, modified frame index list 250 includes, but is not limited to, M frame indices. These frame indices are represented herein as idx1 through idxM in display order, where 1≦idx1≦idxM≦M≦N. In the same or other embodiments, incremental encoding engine 260 configures localization frame encoders 270(1) through 270(M) to encode localization video frames 224(idx1) through 224(idxM) to generate encoded localization layer frames 258(idx1) through 258(idxM), respectively. Note that in other embodiments, the number of instances of localization frame encoder 270 can vary, and the techniques described herein can be modified accordingly. For ease of explanation, the functionality of localization frame encoder 270 will be illustrated and described in detail using localization frame encoder 270(1) as an example.

[0109] As shown, in some embodiments, localization frame encoder 270(1) generates encoded localization layer frame 258(idx1) and decoded localization video frame 292(idx1) based on localization video frame 224(idx1), decoded frame buffer (hereinafter "decoded frame buffer") 262(idx1), and comparison metadata 252. Localization frame encoder 270(1) may implement any number and / or type of encoding ("coding") techniques to generate encoded localization layer frame 258(idx1). Also, in some embodiments, localization frame encoder 270(1) may implement any number and / or type of decoding techniques to generate decoded localization video frame 292(idx1). In other embodiments, localization frame encoder 270(1) does not generate decoded localization video frames.

[0110] Although not shown, the localization frame encoder 270(1) may receive or identify any number and / or types of encoding parameter values, and optionally, any number and / or types of decoding parameter values, in any technically feasible manner. Returning to FIG. 1 , in some embodiments, the localized video encoding application 120 generates the encoded original video 132 and the encoded localization layers 150(1)-150(K) based on a set of encoding parameter values. In the same or other embodiments, the localized video encoding application 120 generates the decoded original video 210 and the decoded localized video frame 292(idx1) based on a set of decoding parameter values.

[0111] In some embodiments, decode frame buffer 262(idx1) points to, but is not limited to, decoded original video frame 212(idx1), zero or more other decoded original video frames 212, and zero or more decoded localized video frames. The decoded frame(s) (hereinafter "decoded frames") pointed to by decode frame buffer 262(idx1) may be individually referred to as a "reference frame" or collectively referred to as "reference frames" herein.

[0112] In some embodiments, localization frame encoder 270(1) can identify and exploit zero or more spatial redundancies within localization video frame 224(idx1), zero or more spatial redundancies between localization video frame 224(idx1) and decoded original video frame 212(idx1), and zero or more temporal redundancies between localization video frame 224(idx1) and each of zero or more other reference frames. As used herein, "exploiting" redundancy refers to reducing the number of bits allocated to representing that redundancy in the encoded localization video frame of frame index "idx1."

[0113] For ease of explanation, the functionality of localization frame encoder 270(1) will be described herein using decoded frame buffer 262(idx1), which includes, but is not limited to, decoded frame buffer 262(idx1) and at most one other reference frame. More precisely, if idx1 is "1," i.e., localized video frame 224(idx1) is the first localized video frame 224 in display order, decoded frame buffer 262(idx1) points to, but is not limited to, decoded original video frame 212(idx1). On the other hand, if idx1 is not "1," decoded frame buffer 262(idx1) points to, but is not limited to, decoded original video frame 212(idx1) and the decoded localized video frame 292 (idx1-1: the frame index of idx1 minus 1) that immediately precedes localized video frame 224(idx1) in display order.

[0114] As shown, in some embodiments, localization frame encoder 270(1) includes, but is not limited to, a prediction engine 280, prediction metadata 282, a residual frame 284, quantized coefficients 286, an entropy coding engine 290, an encoded localization layer frame 258(idx1), and a partial decoder 278. In some embodiments, localization frame encoder 270(1) divides each of localization video frame 224(idx1), decoded original video frame 212(idx1), and any other reference frames into any number and / or type of non-overlapping portions, such as non-overlapping processing units, in any technically feasible manner. In the same or other embodiments, localization frame encoder 270(1) specifies the size and / or type of portions, such as processing units, according to a video compression format and / or image format.

[0115] For example, in some embodiments, localization frame encoder 270(1) divides each of localization video frame 224(idx1), decoded original video frame 212(idx1), and any other reference frames into processing units known as macroblocks. In the same or other embodiments, each macroblock may include, but is not limited to, one or more sample blocks for any number and / or type of color component. For example, in some embodiments, each macroblock may include, but is not limited to, four 8x8 sample blocks for the luma component and a different 8x8 sample block for each of the two chroma components.

[0116] In some embodiments, the prediction engine 280 independently processes any number of non-overlapping portions of the localized video frame 224 (idx1) to identify co-located portions of a predicted frame (not shown) and corresponding portions of the predictive metadata 282. In this manner, in some embodiments, the prediction engine 280 identifies non-overlapping portions of a predicted frame and their corresponding predictive metadata 282. Note that, as used herein, a "co-located" portion of a frame refers to a portion that is at the same location within the frame. In some embodiments, a portion of the predictive metadata 282 that corresponds to a portion of the localized video frame 224 (idx1) specifies one or more instructions for reconstructing the co-located portion of the predicted frame. In the same or other embodiments, the one or more instructions for reconstructing a portion of the predicted frame specify, but are not limited to, one or more predictors for that portion of the predicted frame. The predictors specify different portions of the localized video frame 224, the decoded original video frame 212 (idx1), or any other reference frame.

[0117] For ease of explanation, the following detailed description of the functionality of the prediction engine 280 in some embodiments will be provided using an example in which the non-overlapping portions of a frame are non-overlapping macroblocks contained in the frame. Thus, in some embodiments, the prediction engine 280 independently processes any number of non-overlapping macroblocks contained in the localized video frame 224 (idx1) to identify co-located macroblocks in a predicted frame (not shown) and corresponding portions of predictive metadata 282. In this manner, in some embodiments, the prediction engine 280 identifies non-overlapping macroblocks contained in the predicted frame and their predictive metadata 282.

[0118] In some embodiments, a portion of the prediction metadata 282 corresponding to a macroblock in the localized video frame 224(idx1) specifies, but is not limited to, one or more instructions for reconstructing a co-located macroblock in a predicted frame. Note that, as used herein, a "co-located" macroblock refers to a macroblock that has the same size and location in a different frame. For ease of explanation, a macroblock in a predicted frame may be referred to herein as a "prediction macroblock." In some embodiments, the one or more instructions for reconstructing a prediction macroblock specify, but is not limited to, one or more predictors for the prediction macroblock. The predictors specify different macroblocks for each predictor in the localized video frame 224, the decoded original video frame 212(idx1), or any other reference frame.

[0119] In some embodiments, the one or more instructions specifying a predictor for the predicted macroblock include, but are not limited to, a reference frame identifier and a motion vector. In the same or other embodiments, the reference frame identifier and motion vector identify a macroblock in localized video frame 224, decoded original video frame 212(idx1), or any other reference frame as a predictor to use to construct the predicted macroblock in any technically feasible manner. In some embodiments, the reference frame identifier specifies one of localized video frame 224, decoded original video frame 212(idx1), or any other reference frame in any technically feasible manner. Also, in the same or other embodiments, the motion vector specifies the distance and direction from the predicted macroblock in the predicted frame to the macroblock in the frame identified by the reference frame identifier.

[0120] For ease of explanation, the functionality of the prediction engine 280 in some embodiments will be described in terms of processing one macroblock in the localized video frame 224 (idx1). Note that the localized video frame 224 being processed by the prediction engine 280 may be referred to herein as the "target frame," and the macroblocks of the target frame may be referred to herein as "portions of the target frame" and "target macroblocks."

[0121] In some embodiments, to process a target macroblock, prediction engine 280 determines whether the target macroblock contains visible localized video data based on comparison metadata 252. In other embodiments, prediction engine 280 may identify or estimate whether the target macroblock contains visible localized video data and / or localized video data in any other technically feasible manner. For example, in some embodiments, prediction engine 280 compares the target macroblock with a co-located macroblock in decoded original video frame 212 (idx1) to estimate whether the target macroblock contains visible localized video data.

[0122] In some embodiments, if prediction engine 280 determines that the target macroblock does not contain visible localized video data, prediction engine 280 sets the predicted macroblock in the predicted frame to be the co-located macroblock in decoded original video frame 212(idx1). Also, in the same or other embodiments, prediction engine 280 adds one or more instructions to prediction metadata 282 that instruct the decoder to reuse the co-located macroblock in decoded original video frame 212(idx1) as the predicted macroblock in the predicted frame. For example, in some embodiments, the one or more instructions instruct the decoder to set the predicted macroblock to be the co-located macroblock in decoded original video frame 212(idx1).

[0123] In some embodiments, the one or more instructions instructing the decoder to set the predicted macroblock to be the co-located macroblock in decoded original video frame 212(idx1) include, but are not limited to, specifying a reference frame identifier corresponding to decoded original video frame 212(idx1) and a motion vector with a distance of zero. In the same or other embodiments, a motion vector with a distance of zero indicates that the predictor of the predicted macroblock in the predicted frame is the co-located macroblock in the frame corresponding to the reference frame identifier.

[0124] On the other hand, if the target macroblock contains visible localized video data, the prediction engine 280 selects zero or more macroblocks contained in the localized video frame 224 (idx1), the decoded original video frame 212 (idx1), and any number of other reference frames as candidate macroblocks. The prediction engine 280 may select candidate macroblocks in any technically feasible manner. For example, in some embodiments, the prediction engine 280 may select candidate macroblocks by implementing any number and / or types of search algorithms. The prediction engine 280 then evaluates the selected candidate macroblocks to identify the macroblock that best matches the target macroblock (hereinafter, the “best match macroblock”). The prediction engine 280 may evaluate candidate macroblocks in any technically feasible manner.

[0125] In some embodiments, for each candidate macroblock, prediction engine 280 calculates the mean squared error, mean absolute difference, peak signal-to-noise ratio, or any combination thereof, between the pixel values ​​of the target macroblock and the pixel values ​​of the corresponding candidate prediction. For example, if the candidate macroblock is a macroblock in decoded original video frame 212 (idx1), its corresponding candidate prediction is the candidate macroblock shifted to be co-located with the target macroblock. After evaluating the candidate macroblocks, prediction engine 280 sets the candidate macroblock with the least error, most similarity, or least dissimilarity to the target macroblock as the best match macroblock.

[0126] In some embodiments, prediction engine 280 sets a predicted macroblock in the predicted frame as the best match macroblock. Also, in the same or other embodiments, prediction engine 280 adds one or more instructions to prediction metadata 282 that instruct the decoder to set the predicted macroblock as the best match macroblock. In some embodiments, the instructions that instruct the decoder to set the predicted macroblock as the best match macroblock specify, without limitation, a reference frame identifier corresponding to the frame containing the best match macroblock and a motion vector from the predicted macroblock to the best match macroblock. More specifically, in some embodiments, the motion vector specifies the distance and direction from the location of the predicted macroblock in the predicted frame to the location of the best match macroblock in the frame corresponding to the reference frame identifier. In some embodiments, if the best match macroblock is co-located with the target macroblock, prediction engine 280 sets the motion vector distance to zero.

[0127] In some embodiments, the video decoder identifies a reference frame based on a portion of prediction metadata 282 specifying a reference frame identifier and a motion vector that identifies a predictor of a predicted macroblock in the predicted frame. The video decoder then maps the location of the predicted macroblock in the predicted frame to a best match macroblock in the reference frame according to the motion vector. In some embodiments, the decoder implements a virtually reconstructed predicted frame and sets a virtually reconstructed predicted macroblock in the virtually reconstructed predicted frame as the best match macroblock. In the same or other embodiments, the decoder copies the best match macroblock from the reference frame to the predicted macroblock in the reconstructed predicted frame.

[0128] In some embodiments, after generating the last predicted macroblock in a predicted frame, the prediction engine 280 subtracts the predicted frame from the target frame to generate a residual frame 284. Thus, in some embodiments, the residual frame 284 represents the prediction error between the predicted frame and the target frame in the spatial domain. The localization frame encoder 270(1) then generates an encoded localization layer frame 258(idx1), which encodes, without limitation, instructions for reconstructing a predicted frame and zero or more modifications to the reconstructed predicted frame to eliminate or reduce the prediction error of the reconstructed predicted frame. The localization frame encoder 270(1) can generate the encoded localization layer frame 258(1) in any technically feasible manner.

[0129] In some embodiments, the localization frame encoder 270(1) transforms the residual frame 284 from the spatial domain to the frequency domain by performing any number and / or types of transform operations on the residual frame 284. For example, in some embodiments, the localization frame encoder 270(1) divides the residual frame 284 into any number of non-overlapping blocks in any technically feasible manner. In the same or other embodiments, the localization frame encoder 270(1) transforms pixel values ​​in each block of the residual frame 284 from the spatial domain to the frequency domain to generate transform coefficients (not shown). In the same or other embodiments, the localization frame encoder 270(1) may perform any type of two-dimensional linear transform on the pixel values ​​in each block of the residual frame 284 to generate the transform coefficients. For example, in some embodiments, the localization frame encoder 270(1) converts pixel values ​​into transform coefficients using, but not limited to, a two-dimensional Discrete Cosine Transform (DCT), a Discrete Fourier Transform, a Discrete Sine Transform, or a Discrete Haar Transform.

[0130] In some embodiments, localization frame encoder 270(1) performs any number and / or type of quantization operations on the transform coefficients to generate quantized coefficients 286. Localization frame encoder 270(1) may quantize the transform coefficients in any technically feasible manner by performing any number and / or type of quantization operations on the transform coefficients. In some embodiments, localization frame encoder 270(1) then runs an instance of entropy coding engine 290 on quantized coefficients 286 to generate coded residual data (not shown). In the same or other embodiments, localization frame encoder 270(1) also runs an instance of entropy coding engine 290 on prediction metadata 282 to generate coded prediction metadata (not shown).

[0131] In some embodiments, entropy coding engine 290 may perform any number and / or types of lossless compression operations on any amount and / or types of data or symbols, and / or may additionally or alternatively apply any number and / or types of lossless compression techniques to generate small-sized encoded data (i.e., codes) for the data or symbols. For example, in some embodiments, entropy coding engine 290 may use run-length coding techniques to replace consecutive occurrences of a symbol with one and its repetitions. In the same or other embodiments, entropy coding engine 290 may use variable-length coding techniques to assign short codes to symbols with high probability of occurrence and long codes to symbols with low probability of occurrence.

[0132] More specifically, in some embodiments, the entropy coding engine 290 performs any number and / or type of run-length coding operations, any number and / or type of variable-length coding operations, any number and / or type of other lossless compression operations, or any combination thereof, on the quantized coefficients 286 to generate small-sized encoded residual data for the quantized coefficients 286. Also, in the same or other embodiments, the entropy coding engine 290 performs any number and / or type of run-length coding operations, any number and / or type of variable-length coding operations, any number and / or type of other lossless compression operations, or any combination thereof, on the prediction metadata 282 to generate small-sized encoded prediction metadata (not shown) for the prediction metadata 282. As shown, in some embodiments, the localization frame encoder 270(1) generates an encoded localization layer frame 258(idx1) that includes, but is not limited to, the encoded residual data and the encoded prediction metadata.

[0133] In some embodiments, localization frame encoder 270(1) generates encoded localization layer frame 258(idx1) in accordance with one or more video standards, encoding standards, and / or compression standards. For example, in some embodiments, encoded localization layer frame 258(idx1) is a P frame. Then, once localization frame encoder 270(1) generates encoded localization layer frame 258(idx1), localization frame encoder 270(1) or incremental encoding engine 260 adds encoded localization layer frame 258(idx1) to encoded localization layer 150(1).

[0134] In some embodiments, localization frame encoder 270(1) uses partial decoder 278 to generate decoded localized video frame 292(idx1) that matches the coded data included in coded localization layer frame 258(idx1) in any technically feasible manner. In the same or other embodiments, partial decoder 278 generates decoded localized video frame 292(idx1) based on quantized coefficients 286 and the predicted frame by implementing any number and / or types of decoding techniques.

[0135] More specifically, in some embodiments, the partial decoder 278 dequantizes the quantized coefficients 286 in any technically feasible manner to generate decoded transform coefficients (not shown). For example, in some embodiments, the partial decoder 278 may generate the decoded transform coefficients by performing any number and / or types of dequantization operations on the quantized coefficients 286. The partial decoder 278 then performs an inverse transform (e.g., an inverse 2D DCT) on each block of decoded transform coefficients to generate a decoded residual frame (not shown). In some embodiments, the decoded residual frame is a less-than-perfect replica of the residual frame 284. The partial decoder 278 then generates the decoded localized video frame 292(idx1) by adding the decoded residual frame to the predicted frame. Although not shown, in some embodiments, the incremental encoding engine 260 includes the decoded localized video frame 292(idx1) in the decoded frame buffer 262 associated with a frame index (idx1+1) that is idx1 plus one.

[0136] In some embodiments, the functionality of other instances of localization frame encoder 270 is similar to that described above. Specifically, as shown, localization frame encoder 270(M) generates encoded localization layer frame 258(idxM) and, optionally, decoded localized video frame 292(idxM) based on localized video frame 224(idxM), decoding frame buffer 262(idxM), and comparison metadata 252. Also, in the same or other embodiments, decoding frame buffer 262(idxM) points to, but is not limited to, decoded original video frame 212(idxM) and decoded localized video frame 292 (idxM-1: a frame index equal to idxM minus 1). Those skilled in the art will recognize that the techniques described herein can be modified to reflect any number and / or type of reference frames.

[0137] Although not shown, in some embodiments, localization encoder 140(1) can generate any amount and / or type of “localization” metadata related to streaming localized video 124(1), constructing corresponding encoded localized video, or both. For example, in some embodiments, localization metadata includes, but is not limited to, any amount and / or type of data that cloud-based video service 104 or any other software application can use to generate a manifest file. In the same or other embodiments, localization metadata includes, but is not limited to, any amount and / or type of data that playback application 170, or any other software application, or any hardware module that implements a layer interleaving algorithm, can use in combination with chunks of encoded original video 132 and corresponding chunks of encoded localization layer 150(1) to generate chunks of the corresponding encoded version of localized video 124(1). Additionally, in some embodiments, the localization encoder 140(1), the localized video encoding application 120, or both, can transmit any portion of the localization metadata to any number of software applications, hardware modules, or any combination thereof.

[0138] Figure 3 is a flow diagram illustrating method steps for encoding localized video, according to various embodiments. Note that the method steps are described with reference to the systems shown in Figures 1 and 2, but one skilled in the art will understand that the scope of various embodiments includes any system configured to perform the method steps in any order.

[0139] As shown, method 300 begins at step 302, in which localization encoder 140 obtains encoded original video 132 and localized video 124 associated with original video 122. Then, at step 304, localization encoder 140 decodes encoded original video 132 to generate decoded original video 210. Next, at step 306, localization encoder 140 identifies visible localized video data included in frames of localized video 124 by evaluating localized video 124 together with one or more of decoded original video 210, original video 122, and metadata. Then, at step 308, localization encoder 140 initializes encoded localization layer 150 and selects the first frame of localized video 124.

[0140] Next, in step 310, localization encoder 140 determines whether the selected frame contains perceptible localized video data. If, in step 310, localization encoder 140 determines that the selected frame does not contain visible localized video data, method 300 proceeds to step 312. Then, in step 312, localization encoder 140 instructs encoded localization layer 150 to reuse the corresponding frame of decoded original video 210 as is (without modification). Method 300 then proceeds directly to step 318.

[0141] On the other hand, if, at step 310, localization encoder 140 determines that the selected frame contains visible localization video data, method 300 proceeds directly to step 314. Then, at step 314, localization frame encoder 270 generates an encoded localization layer frame 258 corresponding to the selected frame by encoding the selected frame using zero or more decoded original video frames 212, zero or more decoded localization video frames, or both, as reference frames. Then, at step 316, localization encoder 140 adds the encoded localization layer frame 258 corresponding to the selected frame to encoded localization layer 150.

[0142] Then, in step 318, the localization encoder 140 determines whether the selected frame is the last frame of the localized video 124. If, in step 318, the localization encoder 140 determines that the selected frame is not the last frame of the localized video 124, the method 300 proceeds to step 320. Then, in step 320, the localization encoder 140 selects the next frame of the localized video 124, and the method 300 returns to step 310, where the localization encoder 140 determines whether the selected frame contains visible localized video data.

[0143] On the other hand, if the localization encoder 140 determines at step 318 that the selected frame is the last frame of the localized video 124, then the method 300 proceeds directly to step 322. Then, at step 322, the localized video encoding application 120 transmits the encoded original video 132 and the encoded localization layer 150 to one or more server devices for streaming the localized video 124 to the end user device via the one or more server devices. The method 300 then ends.

[0144] In other words, the techniques of this disclosure can be used to reduce the amount of original video data that is over-encoded when encoding localized video. In some embodiments, a localized video encoding application encodes the original video to generate encoded original video. The localized video encoding application then runs a localization encoder on the localized video and the encoded original video. The localization encoder decodes the encoded original video to generate decoded original video. The localization encoder compares frames of the localized video with frames of the decoded original video to generate a list of frames of the localized video that contain perceptible localized video data (and therefore, when displayed, are visibly different from corresponding portions of the decoded original video). In some embodiments, the localization encoder also generates comparison metadata that indicates zero or more portions of each frame of the localized video that contain perceptible localized video data.

[0145] The localization encoder initializes the encoding localization layer and then encodes each frame of the localized video in display order. For each frame of the localized video, the localization encoder performs the following process: If a frame of the localized video does not contain visible localized video data, the localization encoder instructs the encoding localization layer to reuse the corresponding frame of the decoded original video as is (without modification) as part of the decoded localized video.

[0146] On the other hand, if a frame of the localized video includes visible localized video data, the localization encoder adds the corresponding frame of the decoded original video to a decoding frame buffer of multiple decoded frames that can be reused during encoding. The localization encoder then encodes the frame of the localized video based on the decoding frame buffer to generate an encoded localization layer frame. In some embodiments, the encoded localization layer frame includes, but is not limited to, encoding prediction metadata and, optionally, encoded residual data. In some embodiments, the encoding prediction metadata encodes instructions for constructing a reconstructed predicted frame based on, but is not limited to, one or more portions of the corresponding frame of the decoded original video, one or more portions of one or more other decoded frames specified in the decoding frame buffer, or both. Note that the reconstructed predicted frame may include, but is not limited to, copies of any number of portions of the corresponding decoded original video frame, and optionally, the location within the reconstructed predicted frame for each copy of the portion of the corresponding decoded original video frame may be shifted from its location in the decoded original video frame. In some embodiments, the coded residual data specifies modifications to be made to the reconstructed predicted frames to generate frames of decoded localized video in order to reduce the residual error.

[0147] In some embodiments, the encoded original video and the encoded localization layer are stored independently and delivered to the end-user device via separate streams. A playback application running on the end-user device constructs each encoded localization video chunk based on its corresponding encoded original video chunk and its corresponding encoded localization layer chunk. More specifically, the playback application identifies frames of the localization video chunk that have differences from the corresponding frames of the original video chunk based on the encoded localization video chunk. For each identified frame in the localization video chunk, the playback application generates an encoded base layer chunk by marking the corresponding frame of the encoded original video chunk as a reference-only frame. The playback application then interleaves the frames of the encoded base layer chunk with the frames of the encoded localization layer chunk to generate the encoded localization video chunk.

[0148] One technical advantage of the technology of the present disclosure over the prior art is that the technology of the present disclosure at least reduces the amount of original video data that is over-encoded when encoding localized video. In this sense, each frame of the encoded localization layer can be specified to reuse any number of portions (including entire frames) of the corresponding frame of the reconstructed original video. This significantly reduces the amount of memory used to store the encoded localization layer chunks used to construct the encoded localization video chunks compared to the amount of memory required to store the encoded localization video chunks using prior art techniques. Another technical advantage of the technology of the present disclosure is that storing the encoded localization layer chunks rather than the encoded localization video chunks for streaming localized video over a CDN can improve the caching efficiency of the CDN and ultimately improve the QoE for end users. These technical advantages result in one or more technical improvements over prior art approaches.

[0149] 1. In some embodiments, a computer-implemented method for encoding localized video includes calculating a predicted frame based on a target frame of localized video and at least a portion of a reference frame of decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localized video; performing one or more encoding operations on the residual frame to generate a frame of an encoded localization layer; and transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding.

[0150] 2. The computer-implemented method of claim 1, wherein the step of calculating the predicted frame includes the steps of: determining that the similarity between the portion of the reference frame of the decoded original video and the portion of the target frame of the localized video is higher than the similarity between the portion of the frame of the decoded localized video and the portion of the target frame of the localized video; and generating prediction metadata indicating that the portion of the reference frame of the decoded original video is a predictor of the portion of the predicted frame corresponding to the portion of the target frame.

[0151] 3. The computer-implemented method of claim 1 or 2, further comprising performing one or more decoding operations on the encoded residual data included in the frame of the encoded localization layer to generate a frame of decoded localized video.

[0152] 4. A computer-implemented method according to any one of paragraphs 1 to 3, wherein the step of performing one or more encoding operations on the residual frame includes performing at least one of a transform operation, a quantization operation, and a lossless compression operation on the residual frame to generate encoded residual data.

[0153] 5. A computer-implemented method according to any one of paragraphs 1 to 4, wherein the frame of the coding localization layer includes at least one of coded residual data associated with the residual frame and coded predictive metadata associated with the predicted frame.

[0154] 6. A computer-implemented method according to any one of claims 1 to 5, further comprising the step of decoding the at least one frame of the encoded original video to generate the reference frame of the decoded original video.

[0155] 7. The computer-implemented method of any one of claims 1 to 6, wherein the step of calculating the residual frame includes a step of subtracting the predicted frame from the target frame of the localization video.

[0156] 8. A computer-implemented method described in any of paragraphs 1 to 7, further comprising the steps of: performing one or more comparison operations between a frame of the localized video and a frame of the decoded original video to determine that the frame of the localized video does not contain any visible modifications due to the video localization process; and instructing the encoding localization layer to reuse the frame of the decoded original video as part of the decoded localized video.

[0157] 9. The computer-implemented method of any one of clauses 1 to 8, further comprising the steps of: generating a coded base layer by marking the at least one frame of the coded original video as a reference-only frame; and interleaving a plurality of frames included in the coded base layer with a plurality of frames included in the coded localization layer to generate at least one chunk of coded localized video.

[0158] 10. The computer-implemented method of any one of claims 1 to 9, wherein the frames of the coding localization layer include P frames (predicted frames) or B frames (bidirectional predicted frames).

[0159] 11. In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, cause the one or more processors to encode localized video, the instructions causing the one or more processors to: calculate a predicted frame based on a target frame of localized video and at least a portion of a reference frame of decoded original video, calculate a residual frame based on the predicted frame and the target frame of the localized video, perform one or more encoding operations on the residual frame to generate a frame of an encoded localization layer, and transmit the frame of the encoded localization layer and at least one frame of encoded original video to another device for decoding.

[0160] 12. One or more non-transitory computer-readable media described in paragraph 11, wherein the step of calculating the predicted frame includes a step of determining that the similarity between the portion of the reference frame of the decoded original video and the portion of the target frame of the localized video is higher than the similarity between a second portion of the reference frame of the decoded original video and the portion of the target frame of the localized video.

[0161] 13. One or more non-transitory computer-readable media described in clause 11 or clause 12, further comprising a step of generating encoding prediction metadata to be included in the frame of the encoding localization layer by encoding a motion vector from a portion of the predicted frame to the portion of the reference frame of the decoded original video.

[0162] 14. One or more non-transitory computer-readable media described in any of clauses 11 to 13, wherein the step of performing the one or more encoding operations on the residual frame includes performing at least one of a transform operation, a quantization operation, and a lossless compression operation on the residual frame to generate encoded residual data.

[0163] 15. One or more non-transitory computer-readable media described in any of clauses 11 to 14, wherein the frame of the encoding localization layer includes at least one of encoded residual data associated with the residual frame and encoded predictive metadata associated with the predicted frame.

[0164] 16. One or more non-transitory computer-readable media described in any of clauses 11 to 15, wherein the instructions further cause the one or more processors to perform a step of decoding the encoded original video to generate the decoded original video.

[0165] 17. One or more non-transitory computer-readable media described in any of paragraphs 11 to 16, wherein the step of calculating the residual frame includes a step of subtracting the predicted frame from the target frame of the localized video.

[0166] 18. One or more non-transitory computer-readable media described in any of clauses 11 to 17, wherein the frames of the encoded localization layer are subsequently decoded in combination with the at least one frame of the encoded original video to generate a first chunk of decoded localized video.

[0167] 19. One or more non-transitory computer-readable media described in any of clauses 11 to 18, wherein the instructions further cause the one or more processors to perform the steps of generating a coded base layer by marking at least one frame of the coded original video as a reference-only frame, and interleaving a plurality of frames included in the coded base layer with a plurality of frames included in the coded localization layer to generate at least one chunk of coded localized video.

[0168] 20. In some embodiments, a system includes one or more memories storing instructions and one or more processors coupled to the one or more memories, wherein by executing the instructions, the one or more processors perform the following steps: calculating a predicted frame based on a target frame of a localized video and at least a portion of a reference frame of a decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localized video; performing one or more encoding operations on the residual frame to generate a frame of an encoded localization layer; and transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding.

[0169] Any and all combinations of any element recited in any claim and / or any element described herein, in any combination whatsoever, are intended to be within the contemplated scope of the invention and protection.

[0170] While various embodiments have been described, this has been done for illustrative purposes only, and the embodiments presented are not intended to be exhaustive or limiting of the embodiments of the present disclosure. Many modifications and variations will be apparent to those skilled in the art that do not depart from the scope and spirit of the embodiments described herein.

[0171] Aspects of the present disclosure may be embodied as a system, a method, or a computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be collectively referred to herein as a "module" or a "system." Furthermore, aspects of the present disclosure may take the form of a computer program product or may be embodied as computer-readable program code on one or more computer-readable medium(s).

[0172] Also, any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific (but not exhaustive) examples of computer-readable storage media include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium capable of storing (storing) a program for use by or in association with a system, apparatus, or device that executes instructions.

[0173] Aspects of the present disclosure have been described above with reference to flowchart diagrams and / or block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart diagrams and / or block diagrams, and combinations of blocks in the flowchart diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can then be provided to a processor of a programmable data processing apparatus, such as a general-purpose computer or a special-purpose computer, to produce a machine. The instructions, when executed by the processor of the programmable data processing apparatus, such as a computer, cause the implementation of the functions / acts specified in one or more blocks of the flowchart diagrams and / or block diagrams. Such a processor may include, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array.

[0174] The flowcharts and block diagrams depicted in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can be considered to represent a module, segment, or portion of code, including one or more executable instructions for implementing the logical function(s) identified therein. It should also be noted that in some alternative implementations, the functions noted in the blocks can be executed in an order different from that depicted in the figures. For example, two blocks shown as consecutive blocks can in fact be executed substantially simultaneously, or in some cases, the blocks can be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the functions or operations identified therein, or by a combination of dedicated hardware and computer instructions.

[0175] It should be noted that while the foregoing description is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is defined by the following claims. Preferred embodiments of the present invention will be described below in detail. Embodiment 1 1. A computer-implemented method for encoding localized video, comprising: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; A method comprising: Embodiment 2 the step of calculating a predicted frame comprises: determining that a similarity between the portion of the reference frame of the decoded original video and the portion of the target frame of the localized video is higher than a similarity between a portion of a frame of the decoded localized video and the portion of the target frame of the localized video; generating prediction metadata indicating that the portion of the reference frame of the decoded original video is a predictor of a portion of the predicted frame that corresponds to the portion of the target frame; 2. The computer-implemented method of claim 1, comprising: Embodiment 3 2. The computer-implemented method of claim 1, further comprising: performing one or more decoding operations on encoded residual data included in the frames of the encoded localization layer to generate frames of decoded localized video. Embodiment 4 2. The computer-implemented method of claim 1, wherein performing the one or more encoding operations on the residual frame includes performing at least one of a transform operation, a quantization operation, and a lossless compression operation on the residual frame to generate encoded residual data. Embodiment 5 2. The computer-implemented method of claim 1, wherein the frame of the encoding localization layer includes at least one of encoded residual data associated with the residual frame and encoded predictive metadata associated with the predicted frame. Embodiment 6 2. The computer-implemented method of embodiment 1, further comprising decoding the at least one frame of the encoded original video to generate the reference frame of the decoded original video. Embodiment 7 2. The computer-implemented method of claim 1, wherein calculating the residual frame comprises subtracting the predicted frame from the target frame of the localization video. Embodiment 8 performing one or more comparison operations between frames of the localized video and frames of the decoded original video to determine that the frames of the localized video do not contain visible modifications due to a video localization process; indicating at the coding localization layer that the frames of the decoded original video should be reused as part of a decoded localized video; 2. The computer-implemented method of claim 1, further comprising: Embodiment 9 generating a coded base layer by marking the at least one frame of the coded original video as a reference-only frame; interleaving a plurality of frames in the coded base layer with a plurality of frames in the coded localization layer to generate at least one chunk of coded localized video; 2. The computer-implemented method of claim 1, further comprising: Embodiment 10 2. The computer-implemented method of embodiment 1, wherein the frames of the coding localization layer include predicted frames (P frames) or bidirectional predicted frames (B frames). Embodiment 11 One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to encode localized video, the instructions causing the one or more processors to: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; One or more non-transitory computer-readable media that cause the execution of Embodiment 12 One or more non-transitory computer-readable media as described in embodiment 11, wherein the step of calculating the predicted frame includes a step of determining that a similarity between the portion of the reference frame of the decoded original video and a portion of the target frame of the localized video is higher than a similarity between a second portion of the reference frame of the decoded original video and the portion of the target frame of the localized video. Embodiment 13 One or more non-transitory computer-readable media as described in embodiment 11, further comprising generating encoding prediction metadata to be included in the frame of the encoding localization layer by encoding a motion vector from a portion of the predicted frame to the portion of the reference frame of the decoded original video. Embodiment 14 One or more non-transitory computer-readable media as described in embodiment 11, wherein the step of performing the one or more encoding operations on the residual frame includes performing at least one of a transform operation, a quantization operation, and a lossless compression operation on the residual frame to generate encoded residual data. Embodiment 15 One or more non-transitory computer-readable media as described in embodiment 11, wherein the frame of the encoding localization layer includes at least one of encoded residual data associated with the residual frame and encoded predictive metadata associated with the predicted frame. Embodiment 16 One or more non-transitory computer-readable media as described in embodiment 11, wherein the instructions further cause the one or more processors to perform a step of decoding the encoded original video to generate the decoded original video. Embodiment 17 12. The one or more non-transitory computer-readable media of embodiment 11, wherein the step of calculating the residual frame includes a step of subtracting the predicted frame from the target frame of the localized video. Embodiment 18 One or more non-transitory computer-readable media as described in embodiment 11, wherein the frames of the encoded localization layer are subsequently decoded in combination with the at least one frame of the encoded original video to generate a first chunk of decoded localized video. Embodiment 19 The instructions to the one or more processors: generating a coded base layer by marking the at least one frame of the coded original video as a reference-only frame; interleaving a plurality of frames in the coded base layer with a plurality of frames in the coded localization layer to generate at least one chunk of coded localized video; 12. One or more non-transitory computer-readable media as described in embodiment 11, further causing the computer to perform the following steps: Embodiment 20 one or more memories for storing instructions; one or more processors coupled to the one or more memories; A system comprising: Execution of the instructions causes the one or more processors to: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; To run the system. [Explanation of symbols]

[0176] 100 systems 102 Display Devices 104 Cloud-based video services 110 compute instances 112 processors 116 memory 120 Localized Video Coding Applications 122 Original Video 124 localized footage 130 Video Encoder 132 encoded original video 140 Localization Encoder 150 Coding Localization Layer 170 Playback Applications 172 Manifest File 174 Playback Buffer 176 Encoded Localized Video Chunks 178 Multilayer Video Decoder 182 Encoded Video Chunk Request 184 Encoding Layer Chunk Request 186 encoded original video chunk 188 Encoding Localization Layer Chunks 190 Decode Localized Video Chunks 202 Video Decoder 210 Decoded Original Video 212 decoded original video frames 224 localized video frames 232 encoded original video frame 240 Frame Comparison Engine 250 Modified Frame Index List 252 Comparison Metadata 256 encoding metadata, encoding prediction metadata 258 coded localization layer frames 260 Incremental Encoding Engine 262 Decoding Frame Buffer 270 Localization Frame Encoder 278 Partial Decoder 280 Prediction Engine 282 Prediction Metadata 284 residual frames 286 quantized coefficients 290 Entropy Coding Engine 292 decoded localized video frames

Claims

1. 1. A computer-implemented method for encoding localized video, comprising: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; A method comprising:

2. the step of calculating a predicted frame comprises: determining that a similarity between the portion of the reference frame of the decoded original video and the portion of the target frame of the localized video is higher than a similarity between a portion of a frame of the decoded localized video and the portion of the target frame of the localized video; generating prediction metadata indicating that the portion of the reference frame of the decoded original video is a predictor of a portion of the predicted frame that corresponds to the portion of the target frame; 10. The computer-implemented method of claim 1, comprising:

3. 2. The computer-implemented method of claim 1, further comprising: performing one or more decoding operations on encoded residual data included in the frames of the encoded localization layer to generate frames of decoded localized video.

4. 2. The computer-implemented method of claim 1, wherein performing the one or more encoding operations on the residual frame comprises performing at least one of a transform operation, a quantization operation, and a lossless compression operation on the residual frame to generate encoded residual data.

5. 2. The computer-implemented method of claim 1, wherein the frames of the coded localization layer include at least one of coded residual data associated with the residual frame and coded predictive metadata associated with the predicted frame.

6. The computer-implemented method of claim 1 , further comprising decoding the at least one frame of the encoded original video to generate the reference frame of the decoded original video.

7. The computer-implemented method of claim 1 , wherein calculating the residual frame comprises subtracting the predicted frame from the target frame of the localized video.

8. performing one or more comparison operations between frames of the localized video and frames of the decoded original video to determine that the frames of the localized video do not contain visible modifications due to a video localization process; indicating at the coding localization layer that the frames of the decoded original video should be reused as part of a decoded localized video; The computer-implemented method of claim 1 further comprising:

9. generating a coded base layer by marking the at least one frame of the coded original video as a reference-only frame; interleaving a plurality of frames in the coded base layer with a plurality of frames in the coded localization layer to generate at least one chunk of coded localized video; The computer-implemented method of claim 1 further comprising:

10. The computer-implemented method of claim 1 , wherein the frames of the coding localization layer comprise predicted frames (P frames) or bidirectional predicted frames (B frames).

11. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to encode localized video, the instructions causing the one or more processors to: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; One or more non-transitory computer-readable media that execute the method.

12. one or more memories for storing instructions; one or more processors coupled to the one or more memories; A system comprising: Execution of the instructions causes the one or more processors to: calculating a predicted frame based on a target frame of the localized video and at least a portion of a reference frame of the decoded original video; calculating a residual frame based on the predicted frame and the target frame of the localization video; performing one or more encoding operations on the residual frame to generate an encoded localization layer frame; transmitting the frame of the encoded localization layer and at least one frame of the encoded original video to another device for decoding; To run the system.

Images