Resilient decoding of image sequences.

Abstract

A method for error-resilient decoding of independently coded video frames is disclosed, which does not require a frame buffer to store the most reliable decoded value for each sample location. Instead, the described method only requires buffering the original coded code blocks in a reference buffer. The original coded blocks of a previous frame are accessed from the reference buffer. These blocks are decoded to retrieve the sample values ​​of the blocks from the previous frame. The decoded sample values ​​from the current block and the corresponding reference block are grouped into cells with one or more correspondences. Corresponding cells are compared to generate an initial classification label for each cell, which has one of at least three values ​​that identify the cell as ambiguous, consistent, or inconsistent. Cells initially labeled as ambiguous or inconsistent are subsequently resolved to consistent using the labels assigned to other cells.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to image and video decoding methods, and more particularly to decoding a sequence of related coded images (frames) of non-uniform quality, where the quality of any spatial region within the frames varies from frame to frame in the sequence. The disclosed methods are applicable to video content that is subject to multi-resolution spatial transforms, such as wavelet transforms, as well as block-based coding of transform subbands, such as those used in JPEG2000 and several other media encoding algorithms. [Background technology]

[0002] When decoding a sequence of related coded images (frames), uneven quality or quality variations may be encountered. Such quality variations may be caused by the presence of errors in the coded data stream. However, quality variations may also occur for other reasons, such as changes in the communication bandwidth available for communicating the coded image sequence or preferential coding of regions of interest that vary over the sequence. The decoding method must accommodate such quality variations.

[0003] Error concealment strategies for video coding are well studied. Video decoders typically incorporate one or more frame buffers that hold reference values ​​for each sample in a frame. In traditional predictive video coding algorithms, samples of the current frame are encoded, typically as quantized prediction residuals, with respect to reference values ​​held in the encoder's frame buffer, and predicted values ​​for each sample are derived from the frame buffer, either directly, through motion compensation, or through other transformations. In this very general context, since the frame buffers used by the encoder are also required during decoding, there is little or no additional implementation cost associated with making the values ​​stored in these frame buffers available to the error concealment algorithm that estimates samples of the current frame if the encoded data is somehow erroneous or missing. The same approach can be used even when video frames are encoded independently, by maintaining a frame buffer (usually only one) in the decoder that tracks the most recent reliable values ​​of each sample in the frame. However, in this case, the frame buffering is used only for error concealment, as opposed to the decoding procedure itself, which can significantly increase the implementation cost of the decoder. Definition

[0004] The term "codeblock" here is borrowed from JPEG2000, as defined in ISO / IEC 15444-1 and other members of the JPEG2000 family of standards, where a codeblock is an independently coded block of samples from one subband generated by applying a spatial discrete wavelet transform (DWT) to an image. However, the method of the present invention is also applicable when the codeblock is an independently coded collection of samples generated by other transform methods, or when the original frame sample data is not transformed, or when the codeblock extends across an entire video frame. In particular, the term codeblock simply refers to a unit in a coded representation that may be successfully decoded even if other codeblocks in the same frame are corrupted.

[0005] In this disclosure, the term JPEG2000 is taken to refer to all elements of the JPEG2000 family of standards, and in particular includes both the coding extensions defined in ISO / IEC 15444-2 and the high-throughput JPEG2000 coding technology defined in ISO / IEC 15444-15. Summary of the Invention

[0006] An embodiment of the present invention provides a method for resilient decoding of a sequence of video frames each transformed to produce a plurality of independently coded sample blocks, the coded sample blocks herein identified as code blocks, the method including a reference buffer capable of selectively storing previously received code blocks from the coded video sequence, wherein a code block in a given frame identified as the current code block of the current frame is processed using the following steps: a) decoding the current code block to generate a current block of sample values; b) obtaining a corresponding reference block of sample values ​​of the decoded reference block of a previous frame in the video sequence by accessing the reference buffer and performing the necessary decoding operations to obtain the reference samples; c) dividing each of the current block and the reference block of decoded sample values ​​into one or more cells, each cell of the current block having a one-to-one correspondence with a cell of the reference block; d) comparing corresponding cells of the current block and the reference block to generate an initial classification label for each cell pair, the label having at least three possible values ​​identifying the cell as inconsistent, ambiguous, or consistent; e) Deciding whether pairs of cells initially labeled as "ambiguous" are consistent or inconsistent by a label adjustment step using the labels assigned to other cells.

[0007] In some embodiments, the initial classification label for a pair of cells has four possible values ​​that identify the cells as one of: a) inconsistent, b) ambiguous, c) consistent, with a preference for using a reference cell in the representation of the current frame, and d) consistent, with a preference for using a current cell in the representation of the current frame.

[0008] In some embodiments, a) if a quantization interval associated with at least one decoded sample in the current cell is inconsistent with the quantization interval associated with a corresponding sample in a reference cell, the pair of cells is labeled as inconsistent, where inconsistency means that the quantization intervals do not intersect; b) if there is no pair of non-zero corresponding samples in both the current cell and the reference cell, labeling the pair of cells as ambiguous; c) otherwise, the pair of cells is assigned a consistency label; Thus, the comparison of corresponding cells is based on a comparison of the quantization intervals associated with the samples of the current cell and the reference cell, respectively.

[0009] In some embodiments, whether to prioritize use of the reference cell or the current cell in the representation of the current frame is determined based on information identifying whether the current cell or the reference cell is more finely quantized.

[0010] In some embodiments, a) a distortion reduction value that evaluates the maximum amount that the quantization distortion can be reduced within the two finer quantized cells on sample positions where the current quantization interval and the reference quantization interval are consistent and where both the current sample and the reference sample are non-zero; b) a distortion expansion value that evaluates the distortion between the current cell and the reference cell at sample locations where the current quantization interval and the reference quantization interval are inconsistent; and c) an energy value accumulating energy statistics corresponding to distortion between the decoded sample value and 0 on all samples of the current cell and the reference cell included in the calculation of the distortion reduction value and the distortion expansion value; A comparison of corresponding cells is made based on If the distortion expansion term exceeds a certain percentage of the energy value, the cell pair is labeled as inconsistent; if the energy value is 0, the cell pair is labeled as ambiguous; otherwise, the cell pair is labeled as consistent, and whether to prefer using the reference cell or the current cell in representing the current frame is determined based on a comparison of the distortion reduction value and the distortion expansion value, together with information identifying whether the current cell or the reference cell is more finely quantized.

[0011] In some embodiments, the codeblocks are independently coded blocks of samples from subbands generated by a discrete wavelet transform of the frame sample values.

[0012] In some embodiments, ambiguity in classification labels assigned to a pair of cells is resolved with the help of temporal inconsistency tracking information that identifies whether a temporal change has been detected over the support region of said cells in any frame following one associated with the reference code block of said cells, and if such a temporal change is detected, the ambiguous label is adjusted to an inconsistent label.

[0013] In some embodiments, cells identified as co-located cells and having similar areas of influence within a video frame are collected into groups, and a co-inconsistency step modifies the temporal inconsistency tracking information to identify a temporal change in one cell within the group based on the detection of a new temporal inconsistency in any cell within the same group, where a new temporal inconsistency in a cell means that a temporal change has been detected in that cell for the first time since the frame was associated with the cell's reference code block.

[0014] In some embodiments, temporal inconsistency tracking is performed by assigning binary state variables to cells within a frame, where the binary state variable of a cell is set to a default value that is in an active state after a temporal change is identified, and is reset to another value that is in an inactive state when future frames use a new reference code block for the cell when the reference buffer is updated.

[0015] In some embodiments, temporal inconsistency tracking information is performed jointly for all co-located cells in a group using a single state variable that serves to identify the most recent frame in which a new temporal inconsistency was detected in any cell of the group.

[0016] In some embodiments, if the temporal inconsistency tracking information for the cell does not identify a temporal change since the frame associated with the cell's reference code block, and another cell in the same position within the same group whose reference code block originates from the same frame or the previous frame is labeled as consistent, a co-consistency step adjusts the classification label of the cell from ambiguous to consistent.

[0017] In some embodiments, if a cell belonging to a low-resolution sub-band of a frame whose area of ​​influence in the video frame overlaps with the area of ​​influence of a cell from a high-resolution sub-band is identified as an ancestor of the high-resolution cell, and the temporal inconsistency tracking information of the high-resolution cell does not identify any temporal changes since the frame associated with the reference code block of the cell, and the reference code block of the ancestor cell originates from the same frame or the previous frame and is labeled as consistent, an inter-consistency step adjusts the classification label of the high-resolution cell from ambiguous to consistent.

[0018] In some embodiments, an inter-inconsistency step adjusts the temporal inconsistency tracking information of the high resolution cells and identifies temporal changes of the high resolution cells when new temporal inconsistencies are detected in the ancestor cells.

[0019] In some other embodiments, the inter-inconsistency step adjusts the temporal inconsistency tracking information of the high resolution cell to identify a temporal change of the high resolution cell when a new temporal inconsistency is detected in an ancestor cell or an ancestor of the ancestor cell, but does not adjust the temporal inconsistency tracking information when an immediate ancestor is ambiguous and a power inversion characteristic is observed meaning that a measurement of signal power in a reference sample in the high resolution cell is greater than a measurement of signal power in a reference sample of the immediate ancestor. In one embodiment, a group of co-located cells from a high resolution subband has an identified ancestor group consisting of a low resolution ancestor cell of each cell in the high resolution group, and the signal power measurement used to identify the power inversion is applied to the group of co-located cells so as to be calculated on all cells in the high resolution group and on all cells in the low resolution ancestor group.

[0020] In some embodiments, each ambiguous cell classification label that remains ambiguous is adjusted to an unambiguous label, the adjusted label being inconsistent if the associated cell's temporal inconsistency tracking information identifies a temporal change, and consistent if other power inversion characteristics associated with the cell are observed; otherwise, determined using a default policy that specifies either inconsistent or consistent as the default label for all cells that remain ambiguous.

[0021] In some embodiments, the reference buffer is updated based on context hints that enable a resilient decoder to determine whether code blocks of said current video frame should be used as reference code blocks for future frames, such context hints including protocol information and designated marker codes accompanying the encoded video content received by the decoder.

[0022] In some embodiments, the reference buffer is updated based on the statistics of quantization interval lengths and cell classification labels such that current code blocks with a sufficient proportion of cells labeled as inconsistent or consistent to favor using the current cells in the representation of the current frame are entered into the reference buffer such that the sufficient proportion itself becomes a reference cell for a future frame dependent on the statistics of the quantization interval lengths.

[0023] In some embodiments, a video frame is reconstructed from the combined cells of said frame; a) using the decoded sample values ​​of the current cell if the final classification label of the cell is inconsistent or if statistics of classification labels or quantization interval lengths identify the current cell as preferred for representing the current frame; b) using the decoded sample values ​​of said reference cells; derives a merged cell from the current cell and the reference cell of a cell pair.

[0024] In some embodiments, each code block from the selected high-resolution subband is assigned only one cell, and the final classification label of that cell is predetermined based on the classification labels assigned to the low-resolution cells; only the reference version or the current version of each high-resolution code block, rather than both versions, needs to be decoded, which depends on the predetermined classification label; the high-resolution cells are initially assigned ambiguous labels; and the method of claim 12 is used in combination with the method of any one of claims 13 to 16 to convert these ambiguous labels into inconsistent or consistent final classification labels. [Brief explanation of the drawings]

[0025] An embodiment incorporating all aspects of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1]FIG. 1 is a diagram providing an overview of the present invention and identifying some of the key concepts, notations, and aspects of the present invention. [Figure 2] FIG. 2 is an illustration of an example of temporal inconsistency tracking. [Figure 3] FIG. 3 is an explanatory diagram of reference buffer management. DETAILED DESCRIPTION OF THE INVENTION

[0026] [Detailed explanation] Embodiments of the present invention provide a method for decoding independently coded video frames in an error-resilient manner that can be updated from frame to frame with the most reliable decoded value for each sample location, without requiring a frame buffer in the traditional sense. Instead, the method described in this invention only requires buffering the original coded codeblocks in a reference buffer.

[0027] An embodiment of the resilient decoding method is applied to a sequence of video frames, where each video frame is transformed to generate a plurality of independently coded sample blocks, referred to herein as code blocks. The method includes a reference buffer that can selectively store previously received code blocks from the coded video sequence. A code block within a particular frame, identified herein as the current code block for the current frame, is processed as follows: a) Decode the current code block to generate a current block of sample values. b) obtaining a corresponding reference block of decoded sample values ​​of a previous frame in the video sequence by accessing a reference buffer and performing any decoding operations necessary to obtain said reference samples; c) Dividing each of the current block and the reference block of decoded sample values ​​into one or more cells, each cell of the current block having a one-to-one correspondence with a cell of the reference block. d) Comparing corresponding cells in the current block and the reference block to generate an initial classification label for each cell pair, which includes at least three positive values ​​that identify the cell as "inconsistent," "ambiguous," or "consistent." e) A label adjustment step using the labels assigned to the other cells determines whether pairs of cells initially labeled as "ambiguous" are "consistent" or "inconsistent."

[0028] The original coded blocks of the previous frame are accessed from a reference buffer. These blocks are decoded to obtain the sample values ​​of the blocks from the previous frame. The decoded sample values ​​from the current block and the corresponding reference block are divided into cells with one-to-one correspondence between the current frame and the reference frame. Corresponding cells are compared, and an initial classification label for each cell is generated. The classification label has one of at least three values ​​that identify the cell as "ambiguous," "consistent," or "inconsistent." Cells initially labeled as "ambiguous" or "inconsistent" are then determined to be "consistent" using the labels assigned to other cells.

[0029] The methods described in this disclosure are particularly useful when applied to video content in which frames are coded independently (also known as intra-coding), since errors or quality variations in one frame need not subsequently affect the decoding of other frames in the sequence. Furthermore, in this context, the methods described in this disclosure do not require a frame buffer to track decoded sample values ​​from previous frames, which is particularly useful for lightweight deployments on embedded hardware platforms. More specifically, the methods described in this disclosure are particularly useful when used with block-based encoding techniques that independently code blocks of samples, either in the image domain or in a transform domain such as the wavelet domain, since errors or quality variations in one block do not necessarily affect the decoding of other blocks in the same frame of the sequence. Finally, the methods described in this disclosure are particularly useful when applied to video content that is subject to multi-resolution spatial transforms, such as the wavelet transform, as well as block-based coding of transform subbands, such as those used in JPEG2000 and several other media coding algorithms.

[0030] [Summary and Background of the Invention] While error resilience is typically used only to combat unintentional corruption of coded media data streams, the same techniques can be used to improve the quality of non-uniformly coded video sequences, in which some frames or frame regions are significantly lower quality than others. In particular, low-quality code blocks in the current frame can be treated as if they were corrupted, as long as high-quality reference code blocks from the previous frame are available. The main difference between non-uniform quality decoding applications and error-resilient decoding is that the former introduces more ambiguity regarding the choice of reference code block to track in the reference buffer. This is because, as a video sequence evolves over time, distinguishing between corrupted and uncorrupted code blocks is often trivial compared to distinguishing between high-quality and low-quality versions of a code block. However, in the case of JPEG2000, error-resilient decoding applications can be equivalent to non-uniform quality decoding applications. This is because JPEG2000 code blocks are coded incrementally using an embedded block coding algorithm, so an error in the bitstream of a single code block only leads to a degradation of quality, as opposed to completely corrupting the decoded sample values.

[0031] In some cases, encoders or content delivery servers intentionally introduce significant quality variations from codeblock to codeblock as a way to reduce communication bandwidth. A good example is the "JSIV" video communication framework (A. Naman and D. Taubman, "JPEG2000-Based Scalable Interactive Video (JSIV)," IEEE Transactions on Image Processing, Vol. 20, no. 5, pp. 1435–1449, 2011). In this framework, the server dynamically optimizes the delivery of codeblock bitstreams over bandwidth-limited channels, assuming that a client receiving a completely empty codeblock (no coded bits at all) will use the most recent non-empty version of the same codeblock from the previous frame. In other words, JSIV assumes that the decoder has a reference buffer containing the most recent non-empty version of each codeblock in a frame, and addresses the uneven quality of each frame by simply replacing empty codeblocks in the current frame with the corresponding codeblock bitstream from the reference buffer. JSIV can also be used in combination with motion compensation (as described in A. Naman and D. Taubman, "Motion-Compensated JPEG2000-Based Scalable Interactive Video (JSIV)," IEEE Transactions on Image Processing, vol. 20, no. 9, pp. 2650-2663, 2011). In this case, the use of reference buffers becomes more complex, but the principle remains essentially the same. The method described in this invention can be used to further improve the quality of the decoded video associated with content delivered by a JSIV server.

[0032] In its simplest form, JSIV is a "conditional filling" scheme. In this scheme, non-empty codeblocks in the current frame are used to update (or "fill") their corresponding sample values, while empty codeblocks retain their previous values ​​(i.e., are not filled). In JSIV, conditional filling operates in the wavelet domain because JPEG2000 codeblocks partition wavelet subbands rather than the image data associated with each video frame. However, more generally, conditional filling has a very long history of application within video codecs. Since the first video coding standards, such as H.261, conditional filling has been an important mode for block-based motion-compensated video codecs, which can clearly identify (e.g., through mode flags) blocks within a given frame that will not be updated (filled) with new data. These video codecs primarily operate on image-domain blocks, such as "macroblocks," that are subject to block-based transform, quantization, and coding operations. Apart from operating in the image domain rather than the wavelet domain, another difference between JSIV and conditional replenishment-based video codecs is that JSIV is an open-loop scheme, meaning the decoder does not need to adopt a predetermined strategy for processing the received content. In contrast, most video codecs use a closed-loop approach, where the decoder incrementally updates at least one frame buffer replicated within the encoder. The JSIV server makes rate-distortion-optimizing decisions regarding the content to send to remote clients, based on the assumption that the decoder will adopt a sensible method for restoring the non-uniform quality of the received content, but the decoder is free to use the reference buffer in any way it sees fit.

[0033] As mentioned above, the method described in this invention can be used to address both error-resilient decoding and the decoding of content that may intentionally drop code blocks from a transmitted video stream to save bandwidth, which can be thought of as a form of conditional replenishment. More generally, however, the method of this invention can be used in less extreme cases where certain code blocks are significantly lower quality than other code blocks in a particular frame, without necessarily being completely empty or corrupted. Video encoders or distribution servers may intentionally create such content as a way to reduce communication bandwidth while maintaining a minimum level of video quality even for non-resilient decoders. Our goal is to provide a robust method for decoding such content, whether it is created by an encoder or server, such as JSIV, that carefully selects code blocks to be transmitted at high quality based on temporal changes in the scene, or by a simple encoder that employs a data-agnostic strategy for rotating code blocks transmitted at high quality.

[0034] Importantly, all of the methods of the present invention can be applied using a reference buffer consisting only of the original codeblocks received by the decoder from the encoded content, without the need to store decoded sample values ​​from previous frames. This property is extremely beneficial in embedded hardware systems where memory and memory bandwidth are severely constrained resources. One way to address the memory and memory bandwidth issues associated with maintaining a frame buffer is to employ lightweight data compression techniques on the frame buffer itself, as described, for example, in U.S. Patent Application Publication No. 2011 / 0310974, entitled "Frame buffer compression for video processing devices." However, the reference buffer employed in the present invention stores only the original codeblock bitstream received by the decoder; no encoding is performed within the decoder to maintain the reference buffer.

[0035] Figure 1 provides an overview of the present invention, highlighting some of the key concepts, notations, and inventive aspects of the present invention. In the figure, a frame 100 is independently compressed using a discrete wavelet transform (DWT) where subbands 105 are divided into independently coded codeblocks 110a-n, as is done in JPEG2000. The incoming data consists of compressed codeblocks 110a-n, and a reference buffer is further constructed for various frame indexes k. ref together with the previously received code block B ref 120a-n. The lightly shaded areas within the subbands of the current (arriving) frame represent areas of reduced quality (larger

number

[0036] The present invention comprises the following aspects, many of which are illustrated in FIG. 1, with notations described in more detail in the following sections.

[0037] Aspect 1: Code block B from the current frame cur and the corresponding code block B from the reference buffer ref A method for decoding and comparing B cur Decoded samples from one or more cells Ccur Divide into B ref The corresponding cell C derived from ref and assigning each such cell a classification label Λ based on a comparison with C. In a preferred embodiment, there are four labels Λ∈{-1, 0, 1, 2}, where Λ=-1 is C ref and C cur Λ>0 identifies inconsistency between C and Λ, Λ>0 identifies consistency, and Λ=0 means the comparison is ambiguous. Resolving such ambiguity is the subject of the second, third, fourth and fifth aspects of the present invention. The difference between the labels Λ=1 and Λ=2 is that Λ=2 is C cur identifies as preferred for use in the reconstruction of the current frame, while Λ=1 identifies C ref as preferred for use in reconstructing the current frame. In some embodiments of the present invention, it may not be necessary to distinguish between these two cases of temporal consistency Λ>1 on a cell-by-cell basis, but the distinction is useful for describing the method of the present invention generically.

[0038] Aspect 2: Tracking inconsistencies on a cell-by-cell basis across multiple frames of a video sequence and identifying cell reference code blocks B ref and the current frame. This aspect is useful for resolving ambiguous cell classification results for Λ=0.

[0039] Aspect 3: A method for using cell classification results and temporal inconsistency tracking information of cells belonging to one sub-band to enhance cell classification results and inconsistency tracking of co-located cells belonging to another sub-band. In some embodiments, this aspect of the invention also enables combining temporal inconsistency tracking information associated with co-located cells to reduce the memory footprint associated with tracking state variables.

[0040] Aspect 4: A method for improving classification results and inconsistency tracking information for cells belonging to high-resolution code blocks using classification results and temporal inconsistency tracking information from cells belonging to low-resolution code blocks, which is applicable when encoded frames are generated using multi-resolution transforms.

[0041] Aspect 5: A method for resolving ambiguous cell classification results Λ=0 using temporally inconsistent tracking information and other context information.

[0042] Aspect 6: A method for managing the contents of a reference buffer, comprising: ref the corresponding code block B from the current frame cur These methods include methods for determining when to replace a communication error with any available information, ref and B cur the length of the bitstream or the number of coding passes associated with B, the explicit information provided in the coded data, or ref and B cur , and (potentially) take into account the cell classification labels inferred by the method of any of aspects 1 to 5.

[0043] Aspect 7: A method for decoding a video sequence from coded video frames consisting of code blocks of non-uniform quality, comprising: a cell classification and reference buffer management method according to any one of aspects 1 to 6; and a reference buffer management method according to any one of aspects 1 to 6. ref and the current B cur a merging procedure that combines cells from each decoded pair of code blocks.

[0044] Aspect 8: In applications where the coded subbands are derived from multi-resolution transformations of each video frame, a method for reducing block decoding resources by using information obtained from the lower resolution code blocks to pre-determine the final classification label Λ for code blocks belonging to one or more highest resolutions, such that only the current version or the reference version of each high resolution code block needs to be decoded, and not both.

[0045] [Description of the embodiments of the present invention] 1.1 First Aspect: Decoding and Comparing Current and Reference Code Blocks Our method is described in terms of codeblocks, where the current video frame is (optionally) transformed to generate a collection of subbands, each of which is then divided into one or more codeblocks and (optionally) quantized and coded to generate a codeblock bitstream, with the coded representation of the frame consisting of the collection of codeblock bitstreams. An important example to note is JPEG2000, where the frame is transformed using a discrete wavelet transform (DWT) before optional quantization of the wavelet subbands, followed by independent embedded block coding. However, as described in Section 2, our method can also be applied to other coding strategies.

[0046] 1.1.1 Code Block B cur sample values, quantization intervals and related quantities Let k be the index (or time) for the current frame in the video sequence. cur and any code block B in that frame cur Considering the subband sample positions that the code block spans,

number

number

number

number

number

number

number

number

[0047] In JPEG2000, by following the midpoint reconstruction rule and recognizing that JPEG2000 uses a dead-zone quantizer, the quantization interval I[n] can be implicitly embedded within the reconstructed sample value y[n], so that y[n] can be expressed as:

number

[0048] where Δ is the subband width quantization step size,

number

number

number

number

number

[0049]

number

number

number

number

number

number

number

number

[0050] A similar strategy can be applied to quantization schemes other than the JPEG2000 described above, where both the reconstructed sample values ​​and the quantization interval for each sample can be described via one or more global quantization step sizes and a sign-magnitude integer q[n].

number

number

number

number

[0051]

number

[0052] For clarity, JPEG2000 codeblocks are recovered from J2K packets, and the packet header identifies the number of coding passes present in each block, along with the number of most significant absolute bitplanes P that are "missing" in the sense that the first coding pass encodes the next most significant absolute bitplane. Furthermore, the total number of coding passes that may exist within a codeblock is 3(M b -P)-2, and M b is inferred from the quantization parameters of the associated subband as the maximum number of absolute bitplanes that can be coded for any sample. cur The number of missing best coding paths from B cur The number of coding passes actually available and this maximum value 3 (M b -P)-2. The reason for dividing by 3 is that there are three JPEG2000 coding passes for each magnitude bitplane. Other coding techniques may provide their own means of estimating the logarithmic quantization interval within a codeblock based solely on parameters extracted from the codestream.

[0053] 1.1.2 Reference Code Block B ref Sample values ​​and related quantities

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0054] In some embodiments of the present invention, the reference buffer may consist of decoded sample data instead of a codeblock bitstream, in which case:

number

number

number

number

[0055] 1.1.3 Splitting code blocks into cells In the present invention, the code block area

number

number

number

number

number

[0056]

number

number

number

number

number

number

number

number

number

[0057] As already mentioned, the classification labels of interest are Λ = -1 (inconsistent), Λ = 0 (ambiguous), and Λ = 1 (consistent,

number

number

number

number

number

number

[0058]

number

[0059] 1.1.4 Cell Classification Label Generation for Screen Content Decoding Screen content is special in that large areas of the image often remain completely unchanged over many frames, so any detectable change between frames can be considered a sign of inconsistency. When a decoder is expected to process a video sequence of screen content, the cell

number

number

number

number

number

number

number

number

number

number

number

[0060] If both code blocks use the same quantization step size Δ,

number

number

number

number

[0061] In summary, the cell classification labels for decoding of screen content are set as follows:

number

[0062]

number

[0063] 1.1.5 Cell Classification Label Generation for Photographic Video Decoding For photographic content, the cells are sensitive to photon shot noise and other small environmental factors that can cause inconsistent intervals to appear even when there is no structural change in the scene content over time.

number

number

number

number

number

number

number

number

number

number

number

[0064] At least from an implementation point of view, we can define normalized correlation metrics as

number

number

number

number

[0065] where:

number

number

number

number

number

[0066] Also,

number

number

number

number

number

number

number

number

[0067] Cell classification labels for photo and video content

number

Number

[0068] Here, 0 < T < 1 is a threshold value such as 0.9.

[0069] As required in many embodiments, when both code blocks B cur and B ref include the same quantization step size Δ, these quantities are compared only with each other or with zero, so the common coefficient Δ 2 is

Number

Number

Number

[0070] It is clear to those skilled in the art that the methods described herein have many variations that can be expected to have similar characteristics. For example,

Number

Number

[0071] 1.2 Second Aspect: Tracking Temporal Inconsistencies The purpose of inconsistency tracking is to

number

number

number

number

number

number

number

number

number

number

number

[0072]

number

number

number

number

[0073] The first of these trigger signals is given by the following equation:

number

number

number

number

number

number

[0074] Figure 2 illustrates some of the concepts and notations for temporal inconsistency tracking used here and in later aspects of the invention. This figure illustrates the novel inconsistency triggers from the third and fourth aspects of the invention.

number

number

number

number

number

number

[0075] 2 is a diagram illustrating the temporal inconsistency tracking of six cells 210a-f forming two cell groups 220, 230 according to a third aspect of the present invention, showing the "co-inconsistency induction" within a cell group and the "inter-inconsistency induction" between cell groups of different resolutions (fourth aspect of the present invention). For simplicity, cell indexes c=1, 2, 3, 4, 5, 6 are used here.

number

number

number

[0076] In a resilient decoder

number

number

number

number

number

number

number

[0077]

number

[0078] In some embodiments of the present invention, ambiguous labels are generated by the method described in Section 3.1.

number

number

number

number

number

number

number

[0079] In some embodiments of the present invention,

number

number

[0080] 1.3 Third aspect: Combining information from co-located cells In many embodiments of the present invention, the compressed frame data is composed of multiple subbands that can be divided into equal-sized cells, such that cells in one subband are associated with essentially the same image region as corresponding cells in other subbands. In the case of JPEG2000 and other wavelet-based coding techniques, each resolution level of a component is typically composed of three subbands, referred to as the HL, LH, and HH subbands. This characteristic naturally leads to cells being organized into triplets consisting of one HL cell, one LH cell, and one HH cell, as illustrated in Figures 1 and 2. Color images also consist of multiple color components (or color planes), each with its own subband, and cells in one color component's subband can be associated with the same image region as corresponding cells in the corresponding subbands of each of the other color components.

[0081] Thus, in many applications, nine subbands consisting of the HL, LH, and HH subbands from each of the three color components at a given resolution can be grouped together, with each cell in one subband naturally associated with a corresponding cell in each of the other eight subbands in the group. If some components are subsampled, such associations may require the use of different cell sizes in corresponding subbands of different color components, or may result in a smaller set of grouped subbands. However, in general,

number

number

number

[0082] Since all cells in a group describe the same image region, they can be used to provide additional evidence of both temporal inconsistency and temporal consistency. This is done using two steps, called "co-consistency induction" and "co-consistency induction", applied in that order.

[0083] First, a "coin consistency induction" step is performed. In this step,

number

number

number

[0084] Following the method in Section 3.2, this second inconsistency trigger signal is the time-inconsistency variable

number

number

number

number

[0085] Grouping cells actually creates an opportunity to reduce the amount of persistent memory required to determine these time-inconsistent variables: for large groups, instead of allocating one persistent binary state variable for each cell, we allocate one for each group.

number

number

number

number

number

number

number

number

number

number

number

number

[0086] Finally, the group

number

number

number

[0087] This alternative method for tracking temporal inconsistencies involves using a reference frame gap for every cell in a frame.

number

number

number

[0088] After the "Co-in Consistency Induction" step is performed. In this step, the group

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0089] In some embodiments of the present invention, classification labels have three states given by {-1, 0, 1} instead of the usual four states {-1, 0, 1, 2}.

number

number

[0090] 1.4 Fourth Aspect: Inferring Cell Coherence from Lower Resolution Levels The fourth aspect of the present invention is applicable when the coding of each video frame is based on a multi-resolution transform structure, where each subband has an associated resolution level r, where r=0 is the lowest resolution and typically consists of a single base subband (or LL band), and the subbands associated with each high resolution r can be understood as detail bands. In the case of a general DWT with a classical Mallat structure, each resolution level r>0 consists of three detail subbands HL r ,LH r and H.H. r and its dimensions are at least 2 r As a result, assuming the same cell dimensions for all subbands, in most, if not all, cases, HL r The subband cells are HL r+1 The affected area in the restored image spans four cells from the subband, if any, and the same is true for LH r Sub-band and HH rThe same is true for the subbands. This is illustrated in Figure 2 for two resolution levels.

[0091] Reversing this relationship, for each resolution level r>1, HL r Each cell from a subband

number

number

number

number

number

number

number

number

number

number

[0092] Since a cell's domain of influence is covered by the domains of influence of its ancestors, ancestors can be used to provide additional evidence of both temporal inconsistency and temporal consistency. This is done using two steps called "inter-consistency induction" and "inter-inconsistency induction", applied in that order:

[0093] A group of cells at the same resolution and in the same location

number

number

number

number

number

number

number

number

[0094] In a multi-resolution embodiment of the invention, the method of this and all previous aspects of the invention involves computing the corresponding ancestor group at resolution level r-1.

number

number

[0095] In an embodiment of the present invention, the "interconsistency induction" step is performed immediately after the "coconsistency" step described in Section 3.3. In this step, the group

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0096] The cell label

number

number

number

number

number

[0097] As mentioned above, the group

number

number

number

number

[0098] In some embodiments of the present invention, the group

number

number

number

number

number

number

number

number

number

number

number

[0099] In these embodiments,

number

number

number

number

number

number

[0100] In another embodiment of the present invention,

number

number

[0101] where:

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0102] The above

number

[0103] The above

number

number

number

number

number

number

number

number

[0104] Those skilled in the art will recognize that there are many variations of the method described herein for estimating "power inversion" that are expected to have similar properties. For example:

number

number

[0105] In all embodiments of the present invention described herein,

number

number

number

number

number

number

[0106] 1.5 Fifth Aspect: Adjustment of Cell Classification Labels After applying the methods related to the first four aspects of the present invention, the ambiguous classification labels

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0107]

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0108] In a further embodiment of the present invention, prior evidence of temporal inconsistency is used to:

number

number

number

number

number

number

number

number

number

[0109] In a further embodiment of the present invention,

number

number

number

number

number

number

number

number

number

number

[0110] In a particularly conservative embodiment of the present invention,

number

number

number

number

number

number

number

[0111] 1.6 Sixth Aspect: Determining and Managing Reference Code Blocks As previously mentioned, the present invention relies on a reference buffer to store code blocks received from previous frames. While it is possible to store the decoded sample values ​​of these code blocks, preferred embodiments of the present invention store only the reference code block bitstreams received by the decoder in order to minimize the memory size and memory bandwidth associated with reference buffer transactions.

[0112] If the reference buffer stores the decoded sample values ​​and these sample values ​​can be updated cell by cell,

number

number

number

number

number

number

[0113] In the reference buffer management method described below, we assume that the reference buffer contains only codeblock bitstreams previously received by the decoder. We further assume that if the dimensions of the codeblocks change from frame to frame, reference codeblocks with incompatible dimensions are ignored or discarded. As a result, whenever a reference codeblock exists, B ref is B cur Not only does it have the same support area as

number

number

number

number

number

number

[0114] In some embodiments of the present invention, the reference buffer contains a reference for each codeblock B that may appear in the current video frame. cur One reference code block B for ref In other embodiments of the invention, a rotating buffer can be employed to store all received code blocks from the past k frames, where k is the separation of the current frame from all reference frames that are likely to be of interest.

number

number

number

[0115] Regardless of how the reference buffer is organized, an important consideration for this second aspect of the invention is the

number

number

number

[0116] In some cases, the nature of the application, or the operating context of an equally resilient decoder,

number

number

[0117] As another example, if the present invention is used for resilient decoding of content delivered by a JSIV server that actively selects codeblocks to be assigned non-empty bitstreams based on a temporal change detection algorithm, the natural choice is

number

[0118] In other applications, an encoder or video distribution server may intentionally degrade the quality associated with certain code blocks to conserve communication bandwidth, while including enough information in the communication data to identify (mark) those code blocks whose quality has not been degraded. This can be done as part of the communication protocol, such as by inserting markers into the code stream or otherwise. If a resilient decoder recognizes the protocol or marker codes, it can then use them to decode the

number

number

[0119] These are all examples of context hints, which allow a resilient decoder to make decisions without any actual decision process.

number

[0120] In the absence of context hints, embodiments of the present invention may, after taking into account any adjustments made by various aspects of the present invention, cur All cells associated with

number

number

number

number

number

number

[0121] Figure 3 illustrates reference buffer management. A code block 310 arriving as part of the compressed data for a frame 320 is used to manage the reference buffer using context hints, or adjusted cell labels and log quantization interval statistics, either implicit or explicitly included within the received compressed video data.

number

number

number

[0122] To understand the decision process, note that only those Λ=1 cells actually benefit from the reference code block currently in place, and this is likely to remain true in the future.

number

number

number

number

number

[0123] Some cells

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0124]

number

number

number

number

[0125] In fact,

number

number

number

number

[0126] Conversely, the frame index

number

number

number

number

number

number

number

number

number

[0127] In some embodiments of the present invention, the average

number

number

number

number

number

number

number

number

[0128] 1.7 Seventh aspect: Cell merging and reconstruction of decoded frames According to all the previous aspects of the present invention, after the generation and adjustment of the labels of the cells, no ambiguity remains, so that each cell

number

number

number

number

number

number

number

number

number

number

number

[0129] In embodiments that process code blocks generated by block-based transforms, as opposed to overlapping transforms such as the DWT, two candidates for the reconstructed sample values ​​of a cell are selected in a manner that is less likely to introduce artificial discontinuities at the boundaries between transform blocks.

number

number

[0130] 1.8 Eighth Aspect: Method for Reducing Block Decoding Resources The resilient decoding process described in this invention requires decoding up to two separate codeblocks for each actual codeblock in the compressed representation of a video frame. This doubling of decoding resources compared to a simple non-resilient decoder is the primary implementation cost of this approach. Of course, B cur and B ref If both are non-empty, then only two complete code blocks need to be decoded, but real-time hardware deployments usually need to provide enough resources to handle the worst case, even if one of the two is often empty.

[0131] As explained in Section 3.4, when subbands result from multi-resolution transformations of each video frame, one way to significantly reduce the decoding resources required for resilient decoding is to arrange for codeblocks from one or more of the highest-resolution subbands to be assigned classification labels that depend only on information that can be inferred from lower resolution levels, along with codeblock attributes that can be inferred without actual decoding. Specifically, for such high-resolution codeblocks B cur For each of the

number

number

number

number

number

[0132] In this method, B cur or its reference B ref without actually decrypting

number

number

[0133] It will be apparent to those skilled in the art that many modifications may be made without departing from the spirit and scope of the invention.

[0134] In the claims that follow and in the foregoing description of the invention, unless the context requires otherwise by express language or necessary implication, the word "comprises" or variations such as "comprises" or "comprises" are used in their inclusive sense, i.e., they are used to specify the presence of stated features but do not exclude the presence or addition of further features in various embodiments of the invention.

[0135] Where a prior art document is referred to herein, it should be understood that such reference is not an admission that the document forms part of the common general knowledge in the art in Australia or anywhere else. [Explanation of symbols]

[0136] 100...frame, 105...subband, 110...codeblock, 120..., 130...decoded codeblock, 135...cell, 140..., 150...reference codeblock, 155...cell, 180...recommended version of each cell (current or reference), 190...video frame, 210a-f...cell, 220, 230...cell group, 310...codeblock, 320...current frame, 330...reference buffer.

Claims

1. 1. A method for resilient decoding of a sequence of video frames each transformed to produce a plurality of independently coded sample blocks, the coded sample blocks herein identified as code blocks, the method including a reference buffer capable of selectively storing previously received code blocks from the coded video sequence, wherein a code block in a given frame identified as the current code block of the current frame is processed using the following steps: a) decoding the current code block to generate a current block of sample values; b) obtaining a corresponding reference block of decoded sample values ​​for a previous frame in the video sequence by accessing the reference buffer and performing the necessary decoding operations to obtain reference samples; c) dividing each of the current block and the reference block of decoded sample values ​​into one or more cells, each cell of the current block having a one-to-one correspondence with a cell of the reference block; d) comparing corresponding cells of the current block and the reference block to generate an initial classification label for each cell pair, the label having at least three possible values ​​identifying the cell as inconsistent, ambiguous, or consistent; e) Deciding whether pairs of cells initially labeled as "ambiguous" are consistent or inconsistent by a label adjustment step using the labels assigned to other cells; method.

2. The initial classification label for a pair of cells has four possible values ​​that identify the cell as one of: a) inconsistent; b) ambiguous; c) consistent, with a preference for using a reference cell in the representation of the current frame; and d) consistent, with a preference for using the current cell in the representation of the current frame. The method of claim 1.

3. a) if a quantization interval associated with at least one decoded sample in the current cell is inconsistent with the quantization interval associated with a corresponding sample in a reference cell, the pair of cells is labeled inconsistent, where inconsistency means that the quantization intervals do not intersect; b) if there is no pair of non-zero corresponding samples in both the current cell and the reference cell, then the pair of cells is labeled as ambiguous; c) otherwise, the pair of cells is assigned a consistency label; so that the comparison of corresponding cells is based on a comparison of the quantization intervals associated with the samples of the current cell and the reference cell, respectively. The method according to claim 1 or claim 2.

4. whether to prioritize use of the reference cell or the current cell in the representation of the current frame is determined based on information identifying which of the current cell and the reference cell is more finely quantized; The method according to claim 3, which is dependent on claim 2.

5. a) a distortion reduction value that evaluates the maximum amount that the quantization distortion can be reduced within two finer quantized cells on a sample position where the current and reference quantization intervals are consistent and both the current and reference samples are non-zero; b) a distortion expansion value that measures the distortion between the current cell and the reference cell over sample locations where the current quantization interval and the reference quantization interval are inconsistent; and c) an energy value accumulating energy statistics corresponding to distortion between the decoded sample value and 0 over all samples of the current cell and the reference cell included in the calculation of the distortion reduction value and the distortion expansion value; A comparison of corresponding cells is made based on If the distortion expansion term exceeds a certain percentage of the energy value, the pair of cells is labeled as inconsistent, if the energy value is 0 the pair of cells is labeled as ambiguous, otherwise the pair of cells is labeled as consistent, and whether to favor use of the reference cell or the current cell in the representation of the current frame is determined based on a comparison of the distortion reduction value and the distortion expansion value, together with information identifying whether the current cell or the reference cell is more finely quantized. The method of claim 2.

6. the codeblocks are independently coded blocks of samples from subbands generated by a discrete wavelet transform of frame sample values; 6. The method according to any one of claims 1 to 5.

7. Ambiguity in classification labels assigned to pairs of cells is resolved with the aid of temporal inconsistency tracking information that identifies whether a temporal change has been detected on the support region of the cell in any frame following a frame associated with the reference code block of the cell, and if such a temporal change is detected, the ambiguous label is adjusted to an inconsistent label.

7. The method according to any one of claims 1 to 6.

8. Cells identified as co-located cells and having similar areas of influence within a video frame are collected into groups, and a co-inconsistency step modifies the temporal inconsistency tracking information to identify a temporal change in one cell within the group based on the detection of a new temporal inconsistency in any cell within the same group, where a new temporal inconsistency in a cell means that a temporal change has been detected in that cell for the first time since the frame was associated with the cell's reference code block. The method of claim 7.

9. Temporal inconsistency tracking is performed by assigning binary state variables to cells within a frame, and the binary state variables of a cell are set to a default value, which is an active state, after a temporal change is identified, and when the reference buffer is updated, future frames use a new reference code block of the cell, and are reset to another value, which is an inactive state. The method according to claim 7 or claim 8.

10. the temporal inconsistency tracking information is performed jointly for all of the co-located cells in the group using a single state variable that serves to identify the most recent frame in which a new temporal inconsistency was detected in any cell of the group; The method of claim 8.

11. a co-consistency step adjusting the classification label of a cell from ambiguous to consistent if the cell's temporal inconsistency tracking information does not identify a temporal change since the frame associated with the cell's reference code block, and another cell in the same group at the same position whose reference code block originates from the same frame or a previous frame is labeled as consistent; 11. The method according to any one of claims 8 to 10.

12. a cell belonging to a low-resolution sub-band of a frame, whose area of ​​influence in the video frame overlaps with the area of ​​influence of a cell from a high-resolution sub-band, is identified as an ancestor of the high-resolution cell, and if the temporal inconsistency tracking information of the high-resolution cell does not identify a temporal change since the frame associated with the reference code block of the cell, and the reference code block of the ancestor cell originates from the same frame or the previous frame and is labeled as consistent, an inter-consistency step adjusts the classification label of the high-resolution cell from ambiguous to consistent; 12. The method according to any one of claims 7 to 11.

13. an inter-inconsistency step for adjusting the temporal inconsistency tracking information of the high-resolution cells and identifying a temporal change of the high-resolution cells when a new temporal inconsistency is detected in the ancestor cells; The method of claim 12.

14. The inter-inconsistency step adjusts the temporal inconsistency tracking information of the high-resolution cell and identifies a temporal change of the high-resolution cell when a new temporal inconsistency is detected in an ancestor cell or an ancestor of the ancestor cell, but does not adjust the temporal inconsistency tracking information when an immediate ancestor is ambiguous and a power inversion characteristic is observed, meaning that the measured signal power in the reference sample of the high-resolution cell is greater than the measured signal power in the reference sample of the immediate ancestor. The method of claim 12.

15. a group of co-located cells from the high-resolution subband having an identified ancestor group consisting of a low-resolution ancestor cell of each cell in the high-resolution group, and wherein the signal power measurements used to identify the power inversion are applied to the group of co-located cells such that they are calculated on all cells in the high-resolution group and on all cells in the low-resolution ancestor group; 15. The method of claim 14.

16. Each ambiguous cell classification label that remains ambiguous is adjusted to an unambiguous label, the adjusted label being inconsistent if the associated cell's temporal inconsistency tracking information identifies a temporal change, and consistent if other power inversion characteristics associated with the cell are observed, otherwise determined using a default policy that specifies either inconsistent or consistent as the default label for all cells that remain ambiguous.

16. The method of any one of claims 1 to 15.

17. the reference buffer is updated based on context hints that enable a resilient decoder to determine whether code blocks of the current video frame should be used as reference code blocks for future frames, such context hints including protocol information and designated marker codes that accompany the encoded video content received by the decoder; 17. The method of any one of claims 1 to 16.

18. the reference buffer is updated based on the statistics of quantization interval lengths and cell classification labels such that current codeblocks having a sufficient proportion of cells labeled as inconsistent or consistent to favor using the current cells in the representation of the current frame are entered into the reference buffer to become reference cells for future frames, the sufficient proportion itself being dependent on the statistics of quantization interval lengths; 18. The method of any one of claims 2 to 17.

19. a video frame is reconstructed from the combined cells of said frame; a) using the decoded sample values ​​of the current cell if the final classification label of the cell is inconsistent or if the statistics of classification labels or quantization interval lengths identify the current cell as preferred for representing the current frame; b) using the decoded sample values ​​of the reference cells; derives a merged cell from the current cell and the reference cell of a cell pair, 19. The method of any one of claims 1 to 18.

20. Each code block from the selected high-resolution subband is assigned only one cell, and the final classification label of that cell is predetermined based on the classification labels assigned to the low-resolution cells; only the reference version or the current version of each high-resolution code block, rather than both versions, needs to be decoded, which depends on the predetermined classification label; the high-resolution cells are initially assigned ambiguous labels; and the method of claim 12 is used in combination with the method of any one of claims 13 to 16 to convert these ambiguous labels into inconsistent or consistent final classification labels.

20. A method according to any one of claims 13 to 19 when dependent on claim 12.

Images