Image data encoding and decoding

Abstract

To provide image data encoding and decoding configured to meet a constraint defining an upper limit to the number of binarized symbols relative to a byte size of an output data unit.SOLUTION: In an image data encoding apparatus constituting an entropy encoder, a controller 2020 includes: an entropy encoder for generating encoded binarized symbols; an attribute detector configured to detect an encoding attribute applicable to an output data unit; and a selector for selecting, in response to the detected encoding attribute, a constraint from two or more candidate constraints. An output data stream is subject to a constraint defining an upper limit to the number of binarized symbols that is expressed by a byte size of any individual output data unit relative to the byte size of the output data unit. The entropy encoder is configured to provide padding data, for each output data unit which does not meet the constraint, so as to increase a size in a byte unit of the output data unit in order to meet the constraint.SELECTED DRAWING: Figure 21

Description

Technical Field

[0001] The present disclosure relates to encoding and decoding of image data.

Background Art

[0002] The description of "Background Art" in this specification is for generally explaining the background in this application. The techniques of the present inventors should not be regarded as prior art at the filing time of this application if they are not prior art within the scope described in this background art section, whether explicitly or implicitly, and are not recognized as prior art for this application.

[0003] There are some video data encoding systems and video data decoding systems that convert video data into a frequency domain representation, quantize the obtained frequency domain coefficients, and then apply a certain type of entropy encoding to the quantized coefficients. Thereby, video data can be compressed. By applying the corresponding decoding or decompression technique, the original video data is reconstructed and restored.

[0004] As a successor technology to H.264 / MPEG-4 AVC, HEVC (High Efficiency Video Coding), also known as H.265 or MPEG-H Part 2, has been proposed. This aims to improve the video quality of HEVC and double the data compression ratio of H.264, and to make H.264 scalable in the range of 128×96 to 7680×4320 pixel resolutions. This corresponds to a bit rate of approximately 128 kilobits / second to 800 megabits / second.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure aims to address or mitigate the problems associated with the above processing.

Means for Solving the Problems

[0006] The present disclosure relates to an entropy encoder configured to selectively encode data items representing image data so as to generate encoded binary symbols of successive output data portions, an attribute detector configured to detect encoding attributes applicable to a given output data portion, and a selector configured to select, from among two or more constraint candidates, a constraint for use with the given output data portion in response to the detected encoding attributes and includes, wherein the entropy encoder is configured to generate an output data stream that is subject to a constraint defining an upper limit on the number of binary symbols represented by any individual output data portion relative to the byte size of that output data portion, and the entropy encoder is configured to provide padding data for each output data portion that does not satisfy the constraint in order to increase the size of that output data portion in byte units in order to satisfy the constraint. An image data encoding apparatus is provided.

[0007] The present disclosure includes selectively encoding data items representing image data to generate encoded binary symbols of successive output data portions, generating an output data stream that is subject to a constraint defining an upper limit on the number of binary symbols represented by any individual output data portion relative to the byte size of that output data portion, providing padding data for each output data portion that does not satisfy the constraint in order to increase the size of that output data portion in byte units in order to satisfy the constraint, detecting encoding attributes applicable to a given output data portion, and selecting, in response to the detected encoding attributes, a constraint for use with the given output data portion from among two or more candidate constraints. and includes To further provide an image data encoding method.

[0008] The present disclosure further provides a suitable decoding device for decoding a data signal generated by the above-described method or apparatus.

[0009] Each further aspect and feature of the present disclosure is defined in the appended claims.

[0010] It should be understood that the above general description and the following detailed description are examples of the present technology and do not limit the present technology.

Brief Description of the Drawings

[0011] By referring to the following detailed description together with the accompanying drawings, a complete understanding of the present technology and many of its advantages can be easily understood.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0012] Next, referring to each drawing, FIGS. 1 to 4 schematically show a device or a system that uses a compression device and / or a decompression device according to each embodiment of the present technology described below.

[0013] All data compression devices and / or data decompression devices described below may be implemented in hardware, or may be implemented in software that operates on a general-purpose data processing device such as a general-purpose computer, as programmable hardware such as, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or a combination thereof. In the case of embodiments implemented in software and / or firmware, it will be understood that such software and / or firmware, and the non-transitory data recording medium in which such software and / or firmware is stored or provided, are considered embodiments of the present technology.

[0014] FIG. 1 is a schematic diagram showing an audio / video data transmission / reception system that compresses video data and decompresses video data.

[0015] The input audio / video signal 10 is supplied to a video data compression device 20 that compresses at least the video elements of the audio / video signal 10, and is transmitted along a transmission route 30 such as, for example, a cable, an optical fiber, or a wireless link. The compressed signal is processed by a decompression device 40, whereby an output audio / video signal 50 is provided. In the return path, a compression device 60 compresses the audio / video signal, and the audio / video signal is transmitted to a decompression device 70 along the transmission route 30.

[0016] Therefore, the compression device 20 and the decompression device 70 can constitute one node of the transmission link. Also, the decompression device 40 and the compression device 60 can constitute another node of the transmission link. Of course, if the transmission link is unidirectional, only one of these nodes will require a compression device and only the other node will require a decompression device.

[0017] FIG. 2 is a schematic diagram showing a video display system that performs video data decompression. Specifically, the compressed audio / video signal 100 is processed by the decompression device 110, thereby providing a decompressed signal that can be displayed on the display device 120. The decompression device 110 may be integrally formed with the display device 120, for example, by being provided in the same housing as the display device 120. Alternatively, the decompression device 110 may be provided as a so-called set-top box (STB). Note that the term "set-top" does not necessarily mean that the box needs to be placed in a specific direction or position with respect to the display device 120. This term is merely used in the art to indicate a device that can be connected to the display unit as a peripheral device.

[0018] FIG. 3 is a schematic diagram showing an audio / video storage system that performs video data compression and video data decompression. The input audio / video signal 130 is supplied to a compression device 140 that generates a compressed signal, and is stored in a storage device 150 such as a magnetic disk device, an optical disk device, a magnetic tape device, or a solid-state storage device such as a semiconductor memory or other storage devices. During playback, the compressed data is read from the storage device 150, sent to the decompression device 160, and decompressed. Thereby, an output audio / video signal 170 is provided.

[0019] It will be understood that storage media such as compressed or encoded signals and non-transitory device-readable storage media that store such signals are regarded as embodiments of the present technology.

[0020] FIG. 4 is a schematic diagram showing a video camera that compresses video data. In FIG. 4, an image capture device 180 such as a CCD (Charge Coupled Device) image sensor and its associated control and readout electronics generate a video signal to be sent to a compression device 190. One or more microphones 200 generate an audio signal to be sent to the compression device 190. The compression device 190 generates a compressed audio / video signal 210 to be stored and / or transmitted (collectively represented as stage 220).

[0021] The techniques described below are mainly related to the compression and decompression of video data. It will be understood that in order to compress audio data, existing techniques may be used together with the video data compression techniques described below to generate a compressed audio / video signal. Therefore, no separate description of the compression of audio data will be given. Also, it will be understood that in broadcast quality video data, the data rate associated with video data is generally much higher than the data rate associated with audio data (regardless of compression or non-compression). Therefore, it will be understood that uncompressed audio data can be added to compressed video data to form a compressed audio / video signal. Further, although the embodiments of the present invention (see FIGS. 1 - 4) relate to audio / video data, it will be understood that the techniques described below may also be used in a system that simply processes video data (i.e., compresses, decompresses, stores, displays, and / or transmits). That is, these embodiments do not necessarily need to be related to audio data processing and can be applied to the compression of video data.

[0022] Therefore, FIG. 4 provides an example of a video capture device that includes an image sensor and an encoder of the type described below. Therefore, FIG. 2 provides an example of a decoder of the type described below and a display device to which the decoded image is output.

[0023] The combination of FIGS. 2 and 4 can provide a video capture device including an image sensor 180, an encoding device 190, a decoding device 110, and a display device 120 to which the decoded image is output.

[0024] FIGS. 5 and 6 are schematic diagrams showing, for example, compressed data generated by devices 20, 60, compressed data input to device 110, or a storage medium that stores a storage medium, that is, storages 150, 220. FIG. 5 is a schematic diagram showing a disk-type storage medium such as a magnetic disk or an optical disk. FIG. 6 is a schematic diagram showing a solid-state storage medium such as a flash memory. Note that FIGS. 5 and 6 also show examples of non-transitory device-readable storage media that store computer software for causing a computer to execute one or more of the methods described below when executed by the computer.

[0025] Therefore, the above configuration provides examples of a video storage device, a capture device, and a transmission / reception device that embody any of the present technologies.

[0026] FIG. 7 is a schematic diagram of a video data compression and decompression device.

[0027] The control unit 343 controls the overall operation of the video data compression and decompression device. In particular, in terms of the compression mode, the control unit 343 controls the trial encoding process by operating as a selector that selects various operation modes such as the size and shape of blocks. In addition, the control unit 343 controls whether or not a loss occurs when the video data is encoded. This control unit is regarded as constituting a part of an image encoder or an image decoder (depending on the case). Successive images of the input video signal 300 are supplied to the addition unit 310 and the image prediction unit 320. The image prediction unit 320 will be described in detail later with reference to FIG. 8. The image encoder or the image decoder (depending on the case) may use the features from the device shown in FIG. 7 together with the intra-image prediction unit of FIG. 8. However, this image encoder or image decoder does not necessarily require all the features shown in FIG. 7.

[0028] The adder 310 receives the input video signal 300 on the "+" input and performs a virtual subtraction (negative addition) operation of receiving the output of the image prediction unit 320 on the "-" input. As a result, the predicted image is subtracted from the input image. As a result, a so-called residual image signal 330 representing the difference between the real image and the predicted image is generated.

[0029] One of the reasons for generating the residual image signal is as follows. The data encoding technology to be described, that is, the technology applied to the residual image signal, tends to act more efficiently when the "energy" in the image to be encoded is small. Here, the term "efficient" indicates that the amount of generated encoded data is small. At a specific image quality level, it is desirable (and considered "efficient") that the amount of data to be generated is as small as possible. The "energy" in the residual image is related to the amount of information contained in the residual image. If the predicted image and the real image are the same, the difference between these two images (that is, the residual image) contains zero information (zero energy) and can be very easily encoded into a small amount of encoded data. Generally, when the prediction process can be executed to a certain extent well so that the content of the predicted image is similar to the content of the image to be encoded, it is expected that the residual image data contains less information (less energy) than the input image and can be easily encoded into a small amount of encoded data.

[0030] The rest of the device operating as an encoder (encoding the residual or difference image) is described. The residual image data 330 is supplied to a conversion unit or circuit 340 that generates a discrete cosine transform (DCT) representation of a block or region of the residual image data. Since this DCT technology itself is widely known, it will not be described in detail here. Also, the use of DCT is only an exemplification of one configuration example. As other conversion methods, for example, the Discrete Sine Transform (DST) can be used. The conversion methods may combine individual conversion methods, for example, in a configuration where one conversion method is followed by another conversion method (directly or indirectly). The selection of the conversion method may be explicitly determined and / or may depend on side information used to configure the encoder and decoder.

[0031] The output of the conversion unit 340, that is, a series of DCT coefficients for each conversion block in the image data, is supplied to the quantization unit 350. A variety of quantization techniques are widely known in the field of video data compression, ranging from simple multiplication by a quantization scaling factor to the application of complex look-up tables under the control of quantization parameters. There are generally two purposes for this. The first is to reduce the values that the conversion data can take through quantization processing. The second is to increase the possibility that the value of the conversion data is zero through quantization processing. By these means, in the generation of a small amount of compressed video data, the entropy encoding process described later can be performed more efficiently.

[0032] The data scan process is applied by the scan unit 360. The purpose of the scan process is to rearrange the quantized conversion data in order to group non-zero quantized conversion coefficients together as much as possible, and of course, thereby group zero-valued coefficients together as much as possible. With these functions, so-called run-length encoding or similar techniques can be applied efficiently. Therefore, the scan process involves selecting coefficients from the quantized transform data, and in particular, from the block of coefficients corresponding to the blocks of transformed and quantized image data, according to a "scan order" such that (a) all coefficients are selected at least once as part of the scan, and (b) the desired rearrangement can be performed by the scan. An example of a scan order that yields valid results is the so-called Up-right Diagonal scan order.

[0033] The scanned coefficients are then sent to an entropy encoder (EE) 370. Again, various entropy codings may be performed. Two examples are variations of the so-called CABAC (Context Adaptive Binary Coding) system and variations of the so-called CAVLC (Context Adaptive Variable-Length Coding) system. Generally, CABAC is considered to be more efficient. Some studies have shown that the amount of encoded output data in CABAC is 10 - 20% less than that in CAVLC for the same image quality. However, the level of complexity shown (in terms of implementation) by CAVLC is considered to be much lower than that of CABAC. Note that the scan process and the entropy coding process are shown as separate processes, but in reality, they can be combined or treated together. That is, the data can be read out to the entropy encoder in the scan order. This is the same for each of the inverse processes described later.

[0034] The output of the entropy encoder 370 provides a compressed output video signal 380, together with additional data (described above and / or below) that defines, for example, how the prediction unit 320 generates a predicted image.

[0035] On the other hand, since the operation of the prediction unit 320 itself depends on the decompressed compressed output data, a return path is also provided.

[0036] The reason for this function is as follows. At an appropriate stage in the decompression process (described later), decompressed residual data is generated. This decompressed residual data needs to be added to the predicted image in order to generate the output image (because the original residual data was the difference between the input image and the predicted image). The predicted image generated by the prediction unit 320 should be the same during the compression process and the decompression process so that this process is equivalent on the compression side and the decompression side. Of course, the device cannot access the original input image during decompression. The only thing the device can access is the decompressed image. Therefore, during compression, the prediction unit 320 makes its prediction based on the decompressed compressed image (at least for intra-image coding).

[0037] The entropy encoding process executed by the entropy encoder 370 is considered to be "lossless" (at least in some examples). That is, it can be replaced with exactly the same data as the data initially supplied to the entropy encoder 370. Therefore, in such an example, the return path can be implemented before the entropy encoding stage. In fact, the scan process executed by the scan unit 360 is also considered to be lossless, but in this embodiment, the return path 390 is from the output of the quantization unit 350 to the input of the supplementary inverse quantization unit 420. If a stage causes or may cause a loss, that stage may be included in the feedback loop formed by the return path. For example, the entropy encoding stage can be considered to cause at least in principle a loss by a technique such as encoding bits in parity information. In such an example, the entropy encoding and decoding need to form part of the feedback loop.

[0038] Generally, the entropy decoder 410, inverse scan unit 400, inverse quantization unit 420, and inverse transform unit i.e., circuit 430 respectively provide inverse functions of the entropy encoder 370, scan unit 360, quantization unit 350, and transform unit 340. Here, the compression process will be continued to be described, and the process for decompressing the input compressed video signal will be described separately later.

[0039] In the compression process, the scanned coefficients are sent from the quantization unit 350 to the inverse quantization unit 420 that performs the inverse operation of the scan unit 360 via the return path 390. The inverse quantization process and the inverse transform process are executed by the inverse quantization unit 420 and the inverse transform unit 430, and the compressed-decompressed residual image signal 440 is generated.

[0040] The image signal 440 is added to the output of the prediction unit 320 by the addition unit 450, and the reconstructed output image 460 is generated. Thus, as will be described later, one input to the image prediction unit 320 is configured.

[0041] The process applied to decompress the received compressed video signal 470 will be described. The compressed video signal 470 is first supplied to the entropy decoder 410, and then supplied in order to the inverse scan unit 400, inverse quantization unit 420, and inverse transform unit 430. Then, it is added to the output of the image prediction unit 320 by the addition unit 450. Therefore, on the decoder side, the decoder reconstructs the residual image and applies this (in block units) (by the addition unit 450) to the predicted image to decode each block. Briefly speaking, the output 460 of the addition unit 450 forms the output decompressed video signal 480. Actually, before outputting the signal, further filtering (for example, using the filter 560) may be optionally performed. This filter 560 is shown in FIG. 8. In FIG. 7 showing the overall configuration compared to FIG. 8, the filter 560 is omitted for clarity.

[0042] The devices shown in FIGS. 7 and 8 can operate as a compression (encoding) device or a decompression (decoding) device. The functions of the two types of devices substantially overlap. The scan unit 360 and the entropy encoder 370 are not used in the decompression mode. The operations of the prediction unit 320 (to be described in detail later) and other units follow the mode and parameter information included in the received compressed bitstream and do not generate this information themselves.

[0043] FIG. 8 is a schematic diagram showing the generation of a predicted image, and particularly shows the operation of the image prediction unit 320.

[0044] Two basic prediction modes are executed by the image prediction unit 320. The two basic prediction modes are so-called intra-image prediction and so-called inter-image prediction or motion-compensated (MC) prediction. On the encoder side, these predictions each include detecting a prediction direction for the current block to be predicted and generating a predicted block of samples according to other samples (in the same (intra) or another (inter) image). The adder 310 or 450 encodes or decodes each block by encoding or decoding the difference between the predicted block and the actual block.

[0045] (On the decoder side or the inverse decoding side of the encoder, the detection of this prediction direction may depend on the data associated with the encoded data by the encoder, which indicates which direction was used by the encoder. Alternatively, the detection of the prediction direction may depend on the same elements determined by the encoder.)

[0046] Intra-image prediction is based on predicting the content of image blocks or regions in the data obtained from within the same image. This corresponds to what is known as I-frame coding in other video compression techniques. However, in contrast to I-frame coding which encodes the entire image by intra-coding, in this embodiment, the selection between intra-coding and inter-coding can be done on a block-by-block basis. In other embodiments, the selection is still done on an image-by-image basis.

[0047] Motion-compensated prediction is an example of inter-image prediction, and motion information is used to define the source of the image details to be encoded in the current image in other adjacent or neighboring images. Thus, in an ideal example, the content of a block of image data in the predicted image can be very easily encoded as a reference (motion vector) indicating a corresponding block that exists at the same or slightly different position in the adjacent image.

[0048] The technique known as "block copy" prediction uses a vector indicating a block of samples at a position displaced from the current prediction block within the same image that should be copied to generate the current prediction block, and in some respects, can be said to be a hybrid of the above two predictions.

[0049] Returning to FIG. 8, FIG. 8 shows two image prediction configurations (corresponding to intra-image prediction and inter-image prediction), and the prediction results are selected by a multiplier 500 under the control of a mode signal 510 (e.g., of a control unit 343) to provide blocks of the predicted image for supply to adder units 310 and 450. The selection is made based on which selection results in the least "energy" (which may be considered the amount of information that needs to be encoded as described above), and the selection is notified to the decoder in the encoded output data stream. In this regard, for example, from an input image, the regions of two versions of a predicted image are trial-subtracted, the pixel value of each pixel in the difference image is squared, the multiplied values are summed, and between the two versions, by identifying which version has a lower average multiplication value of the difference image related to the image region, the image energy can be detected. In other examples, trial encoding can be performed for each selection or for each possible selection. Then, the selection is made according to the cost for each possible selection regarding one or both of the number of bits required for encoding and the distortion with respect to the image.

[0050] In the intra prediction system, the actual prediction is performed based on an image block received as part of signal 460. That is, the prediction is performed based on the encoded-decoded image block so that the same prediction can be made in the decompressor. However, data can also be derived from the input video signal 300 to control the operation of the intra image prediction unit 530 by the intra mode selection unit 520.

[0051] In inter image prediction, the motion compensation (MC) prediction unit 540 uses motion information such as motion vectors derived from the input video signal 300 by the motion estimation unit 550, for example. The motion compensation prediction unit 540 applies these motion vectors to the reconstructed image 460 to generate blocks for inter image prediction.

[0052] Therefore, the intra image prediction unit 530 and the motion compensation prediction unit 540 (operating together with the estimation unit 550) each operate as a detection unit that detects the prediction direction for the current block to be predicted, and also as a generation unit that generates a predicted block of samples (which forms part of the prediction result sent to the addition units 310 and 450) according to other samples defined by the prediction direction.

[0053] Here, the processing applied to signal 460 will be described. First, signal 460 is optionally filtered by filter section 560. The filter section 560 will be described in more detail below. In this processing, a "deblocking" filter is applied to eliminate or at least mitigate the influence on the block-based processing and subsequent operations executed by conversion section 340. It is also possible to use a Sample Adaptive Offsetting (SAO) filter. Also, an adaptive loop filter is optionally applied using the coefficients obtained by processing the reconstructed signal 460 and the input video signal 300. This adaptive loop filter is a type of filter that applies adaptive filter coefficients to the data to be filtered using known techniques. That is, the filter coefficients can vary based on various factors. The data defining which filter coefficients to use is inserted into a part of the encoded output data stream.

[0054] When the device is operating as a decompression device, the filtered output from filter section 560 actually forms the output video signal 480. This signal is stored in one or more image or frame storage sections 570. The storage of consecutive images is required in motion compensation prediction processing, particularly in the generation of motion vectors. To ensure the necessary memory, the images stored in image storage section 570 are held in a compressed format and may then be decompressed for use in the generation of motion vectors. For this specific purpose, any known compression / decompression system may be used. The stored images are sent to interpolation filter 580 that generates higher-resolution stored images. In this example, intermediate samples (sub-samples) are generated such that the resolution of the interpolated image output by interpolation filter 580 is four times (in each dimension) the image stored in image storage section 570 when the color channels are 4:2:0 and eight times (in each dimension) the image stored in image storage section 570 when the luminance channel is 4:2:0. The interpolated image is sent as an input to motion estimation section 550 and motion compensation prediction section 540.

[0055] Here, a method for dividing an image for compression processing will be described. At a basic level, an image to be compressed can be considered as an array of blocks or regions consisting of samples. The division of such an image into blocks or regions can be performed by a decision tree as described in "High Efficiency Video Coding (HEVC) text specification draft 6" by Bross et al., JCTVC-H1003_d0 (November 2011), the content of which is incorporated herein by reference. In some examples, the resulting blocks or regions have various sizes and, in some cases, have a shape along the array of image features within the image as a whole by the decision tree. This alone can improve the coding efficiency. This is because samples representing or along similar image features tend to be grouped by such a configuration. In some examples, square blocks or regions of different sizes (e.g., 4×4 samples to, for example, 64×64, or larger blocks) are available for selection. In other configuration examples, blocks or regions of different shapes such as rectangular blocks (e.g., oriented vertically or horizontally) can be used. Other non-square and non-rectangular blocks are also included. As a result of the division of such an image into blocks or regions, (at least in this example) each sample of the image is assigned to one, and only one, such block or region sample.

[0056] Next, the intra prediction process will be described. Generally, intra prediction includes generating a prediction result of the current block of samples from samples that have been previously encoded and decoded in the same image.

[0057] FIG. 9 is a schematic diagram showing a partially encoded image 800. Here, the image is encoded in block units from the upper left to the lower right. An example of a block that is in the process of being encoded in the processing of the entire image is shown as block 810. From the upper shaded area 820 to the left side of block 810, it has already been encoded. For intra-image prediction of the content of block 810, any of the shaded areas 820 can be used, but the unshaded area below it cannot be used.

[0058] In some examples, the image is encoded in block units such that larger blocks (referred to as CUs (Coding Units)) are encoded in an order such as the order described with reference to FIG. 9. For each CU, (depending on the block splitting process performed), the CU may be processed as a set of two or more smaller blocks or TUs (Transform Units). This gives an encoding hierarchical order such that the image is encoded in CU units. Each CU may potentially be encoded in TU units. For an individual TU (the largest node in the block splitting tree structure) within the current CTU (Coding Tree Unit), the above-described (for each CU and then for each TU) encoding hierarchical order means that there are samples that have been encoded earlier in the current CU and that may be available for encoding of the TU. These samples may be present, for example, to the upper right or lower left of the TU.

[0059] Block 810 represents a CU. As described above, for intra-image prediction processing, this may be subdivided into a set of smaller units. An example of the current TU 830 is shown within CU 810. More generally, the image is divided into regions or groups of samples so that signaling information and the transformed data can be efficiently encoded. Signaling of the information may actually require different tree structures consisting of those subdivided with respect to the structure of the transformation, or rather with respect to the structure of the prediction information or the prediction itself. For these reasons, the CU may have different tree structures for the conversion block or region, the prediction block or region, and the structure of the prediction information. In some examples such as HEVC, the structure can be a so-called quadtree of the CU where the leaf nodes include one or more prediction units and one or more TUs. The TU can include a plurality of conversion blocks corresponding to the luma and chroma representations of the image, and it can be considered that the prediction method is applicable at the conversion block level. In some examples, it can be considered that the parameters applied to a specific sample group are mainly defined at the block level. This block level may not have the same granularity as the conversion structure.

[0060] Intra-image prediction considers the samples encoded before considering the current TU. The samples are the samples above and / or to the left of the current TU. The samples serving as the prediction sources for the required samples may exist at different positions or in different directions with respect to the current TU. To determine which direction is suitable for the current prediction unit, the exemplary encoder mode selection unit 520 can try all the combinations of available TU structures for each candidate direction and select the prediction direction and TU structure that result in the highest compression efficiency.

[0061] The image may be encoded slice by slice. In one example, a slice is a group of adjacent CUs in the horizontal direction. However, more generally, a slice can be composed of the entire residual image, or a slice can be a single CU or a row of CUs, etc. Since a slice is encoded as an independent unit, a certain degree of resilience to errors can be obtained. The states of the encoder and decoder are completely reset at the slice boundary. For example, intra prediction is not performed across the slice boundary. Therefore, the slice boundary is treated as an image boundary.

[0062] FIG. 10 is a schematic diagram showing a set of possible (candidate) prediction directions. All direction candidates are available for a prediction unit. The direction is determined by horizontal and vertical movements relative to the current block position, but is encoded as a prediction "mode". The set of directions of the said set is shown in FIG. 11. Note that the so-called DC mode represents the simple arithmetic mean of the surrounding upper and left samples. Also, the set of directions shown in FIG. 10 is just an example. In other examples, as schematically shown in FIG. 12, a set (a total of 67 modes) combining DC and planar in, for example, a 65-degree angular mode is used as one set. Other numbers of modes can also be adopted.

[0063] Generally, after detecting the prediction direction, the system is operable to generate a predicted block of samples according to other samples determined by the prediction direction. In some examples, an image encoder is configured to encode data identifying a selected prediction direction for each sample or region of an image (and an image decoder is configured to detect such data).

[0064] FIG. 13 is a schematic diagram showing an intra prediction process. In this intra prediction process, samples 900 of a block or region 910 composed of samples are derived from other reference samples 920 of the same image according to a direction 930 determined by an intra prediction mode related to the said samples. The reference samples 920 in this example are based on the blocks above and to the left of the target block 910, and the predicted value of the samples 900 is obtained by tracking along the direction 930 with respect to the reference samples 920. The direction 930 may indicate a single individual reference sample, but more generally, an interpolated value of surrounding reference samples is used as the predicted value. Note that the block 910 may be square as shown in FIG. 13, or may have other shapes such as a rectangle.

[0065] FIG. 14 is a schematic diagram showing the operation of a CABAC entropy encoder.

[0066] The CABAC encoder operates on binary data, i.e., data represented by only two symbols, 0 and 1. The encoder performs a so-called context modeling process of selecting a "context", i.e., a probability model, for the next data based on the already encoded data. The selection of the context is performed in a deterministic manner such that the same decision is made in the decoder without requiring additional data (identifying the context) to be added to the encoded data stream passed to the decoder based on the already decoded data.

[0067] Referring to FIG. 14, the input data to be encoded may be passed to a binary converter 1400 if it is not already in binary form. If the data is already in binary form, the converter 1400 is bypassed (by switch 1410 in the figure). In this embodiment, the conversion to binary form is actually performed by representing the quantized DCT coefficient data as a series of binary "maps". Binary maps will be described later.

[0068] The binary data may then be processed by one of two processing paths, a "normal" path and a "bypass" path (although shown schematically as separate paths, in some embodiments of the present invention described later, they can actually be executed at the same processing stage using slightly different parameters). In the bypass path, a so-called bypass coder 1420 is used that does not necessarily utilize the same form of context modeling as the normal path. In some examples of CABAC encoding, when it is necessary to process a series of data particularly rapidly, the bypass path can be selected. However, in this embodiment, two characteristics of the so-called "bypass" data are mentioned. The first characteristic is that the bypass data is processed by the CABAC encoders (950, 1460) using only a fixed context model representing a 50% probability. The second feature is that the bypass data relates to a certain category of data. A specific example of such data is coefficient sign data. If the bypass path is not selected, the normal path is selected by switches 1430, 1440 shown in the figure that operate under the control of control circuit 1435. This includes data that is processed by context modeler 1450 and subsequently by encoding engine 1460.

[0069] The entropy encoder shown in FIG. 14 encodes a block of data (i.e., data corresponding to, for example, a block of coefficients for a block of a residual image) as a single value when the entire block consists of data having a value of 0. For each block not included in this category, i.e., a block containing at least some non-zero data, a "significance map" is created. The significance map indicates, for each position in the block of data to be encoded, whether the corresponding coefficient in the block is non-zero. The significance map data itself, which is in binary form, is CABAC encoded. Utilizing the significance map is useful for compression because it is not necessary to encode data for coefficients whose magnitudes are indicated as zero by the significance map. Also, the significance map can include a special code indicating the last non-zero coefficient in the block. This allows all of the last high-frequency / trailing zero coefficients to be omitted from encoding. In the encoded bitstream, following the significance map is data that defines the values of the non-zero coefficients specified by the significance map.

[0070] Also, another level of map data is created and CABAC encoded. As an example, there is a map that defines, as a binary value (1 = yes, 0 = no), whether the coefficient data at the map positions indicated as "non-zero" by the significance map actually has a value of "1". Other maps define whether the coefficient data at the map positions indicated as "non-zero" by the significance map actually has a value of "2". Still another map indicates whether the data has a value of "3 or more" for these map positions where the coefficient data is indicated as "non-zero" by the significance map. Also, still another map indicates the sign of the data value (using a predetermined binary notation such as 1 for +, 0 for -, or vice versa, etc.) for the data specified as "non-zero".

[0071] In an embodiment of the present invention, the significance map and other maps are assigned in a predetermined manner to either a CABAC encoder or a bypass encoder, and each represents a different attribute or value range of the same initial data item. In one example, at least the significance map is CABAC encoded and at least a part of the remaining maps (such as sign data) is bypass encoded. Thus, each data item is divided into respective subsets, and each subset is encoded by a first (e.g., CABAC) and a second (e.g., bypass) encoding system. The nature of the data, CABAC, and bypass encoding is such that for a predetermined amount of CABAC encoded data, a variable amount of bypass data greater than or equal to zero is generated for the same initial data item. Thus, for example, if the quantized and reordered DCT data contains substantially all zero values, no bypass data may be generated, or a very small amount of bypass data may be generated. This is because the bypass data relates only to the map positions indicating that the value of the significance map is not zero. In another example, in quantized re-ordered DCT data having many high value coefficients, a significant amount of bypass data may be generated.

[0072] In an embodiment of the present invention, the significance map and other maps are generated from quantized DCT coefficients, for example, by the scanning unit 360, and are subjected to zigzag scanning processing (or scanning processing selected from zigzag, horizontal raster, and vertical raster scanning) before being subjected to CABAC encoding.

[0073] Generally, CABAC encoding involves predicting the context of the next bit to be encoded, i.e., the probability model, based on other previously encoded data. If the next bit is the same as the bit identified as "most likely" by the probability model, the encoding of the information that "the next bit matches the probability model" can be encoded with high efficiency. Encoding that "the next bit does not match the probability model" is inefficient, so the derivation of context data is important for the good operation of the encoder. The term "adaptive" means that the context or probability model is adapted or changed during encoding in an attempt to provide a good match with the next data (which has not yet been encoded).

[0074] Using a simple analogy, in written English, the letter "U" is relatively rare. However, at the character position immediately following the letter "Q", it is very common. Therefore, in the probability model, the probability of "U" can be set to a very low value, but if the current character is "Q", the probability model of "U" as the next character can be set to a very high probability value.

[0075] In the current configuration, CABAC encoding is used for at least the significance map and a map indicating whether a non-zero value is 1 or 2. Bypass processing - In these embodiments, it is the same as CABAC encoding, but for at least the coded data and a map indicating whether the value is > 2, in view of the fact that the probability model is fixed with an equal (0.5:0.5) probability distribution of 1s and 0s. For those data positions identified as > 2, the actual value of the data can be encoded using a separate so-called escape data encoding. This may include Golomb-Rice encoding techniques.

[0076] For CABAC context modeling and encoding processing, it is further described in detail in WD4: Working Draft 4 of High Efficiency Video Coding, JCTVC-F803_d5, Draft ISO I / EC 23008-HEVC; 201x(E) 2011-10-28.

[0077] Here, referring to FIGS. 15 and 16, the entropy encoder forming part of the video coding apparatus includes a first coding system (e.g., an arithmetic coding system such as CABAC encoder 1500) and a second coding system (such as bypass encoder 1510), and is arranged such that a particular data word or value is encoded into the final output data stream by either the CABAC encoder or the bypass encoder, but not both.

[0078] In embodiments of the present invention, the data values passed to the CABAC encoder and the bypass encoder are each subset of the ordered data values divided or derived from the initial input data (reordered quantized DCT data in this example), and represent different ones of the set of "maps" generated from the input data.

[0079] The schematic diagram treats the CABAC encoder and the bypass encoder as separate arrangements. Although this is often true in practice, in another possibility schematically shown in FIG. 16, a single CABAC encoder 1620 is used as both the CABAC encoder 1500 and the bypass encoder 1510 of FIG. 15. Encoder 1620 operates under the control of encoder 1630 such that it operates in an adaptive context model when in the mode of CABAC encoder 1500 (as described above), and in a fixed 50% probability context model when in the mode of bypass encoder 1510.

[0080] A third possibility is to combine these two in that two substantially identical CABAC encoders can be operated in parallel (similar to the parallel arrangement of FIG. 15). The difference is that the CABAC encoder operating as the bypass encoder 1510 has its context model fixed as a 50% probability context model.

[0081] The outputs of the CABAC encoding process and the bypass encoding process can be stored (at least temporarily) in their respective buffers 1540, 1550. In the case of FIG. 16, switch or demultiplexer 1660 functions under the control of mode signal 1630, routing the CABAC encoded data to buffer 1550 and bypassing the encoded data to buffer 1540.

[0082] FIGS. 17 and 18 schematically show an example of an entropy decoder forming part of a video decoding apparatus. Referring to FIG. 17, each buffer 1710, 1700 passes data to CABAC decoder 1730 and bypass decoder 1720, arranged such that a particular encoded data word or value is decoded by either the CABAC decoder or the bypass decoder, but not both. The decoded data is reordered by logic 1740 into the appropriate order for subsequent decoding stages.

[0083] The schematic diagram of FIG. 17 treats the CABAC decoder and the bypass decoder as separate arrangements. This is often the case in reality, but in another possibility schematically shown in FIG. 18, a single CABAC decoder 1850 is used as both the CABAC decoder 1730 and the bypass decoder 1720 of FIG. 17. Decoder 1850 operates in an adaptive context model when in the mode of CABAC decoder 1730 (as described above), and operates in a fixed 50% probability context model when in the mode of bypass encoder 1720, under the control of decoder 1860.

[0084] As described above, the third possibility is that two substantially identical CABAC decoders can operate in parallel (similar to the parallel arrangement in FIG. 17), and the difference is that the CABAC decoder operating as the bypass decoder 1720 has a context model fixed with a 50% probability context model.

[0085] In the case of FIG. 18, switch or multiplexer 1870 acts under the control of mode signal 1860 to route the CABAC encoded data from buffer 1700 or buffer 1710 to decoder 1850 as appropriate.

[0086] FIG. 19 schematically shows picture 1900 and is used to show various picture segmentation schemes relevant to the following discussion.

[0087] An example of picture segmentation is in slices or "regular slices". Each regular slice is encapsulated in its own network abstraction layer (NAL) unit. Prediction within the picture (e.g., in-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are not permitted. That is, a normal slice can be reconstructed independently of other regular slices within the same picture. So-called tiles define horizontal and vertical boundaries for splitting a picture into rows and columns of tiles. In a way corresponding to normal slices, picture prediction dependencies are not permitted across tile boundaries, nor are there entropy decoding dependencies. However, tiles are not restricted to be contained within individual NAL units.

[0088] Generally speaking, there can be multiple tiles within one slice, multiple slices within one tile, or one or more slices within one picture.

[0089] The schematic example in Figure 19 shows four slices 1910, 1920, 1930, 1940, and slice 1940 contains two tiles 1950, 1960. However, as mentioned, this is merely an arbitrary schematic example.

[0090] In some configuration examples, there is a threshold for the number of bins (either EP or CABAC) that can be encoded in a slice or picture according to the following formula.

[0091] BinCountsinNalUnits <= (4 / 3) * NumBitsInVclNalUnits + (RawMinCuBits*PicSizelnMinCbsY) / 32 (Equation 1)

[0092] The right side of the equation depends on the sum of two parts. These are a constant value (RawMinCuBits*PicSizelnMinCbsY) for a specific image region, related to the size of the slice or picture, and a dynamic value (NumBitsInVclNalUnits) which is the number of bits encoded in the output stream of the slice or picture. Note that the value 4 / 3 represents the number of bins per bit.

[0093] RawMinCuBits is the number of bits of a raw CU of the minimum size (usually 4*4), and PicSizelnMinCbsY is the number of minimum size CUs of the slice or picture.

[0094] When this threshold is exceeded, CABAC zero words (3 bytes having the value 00 00 03) are added to the stream until the threshold is reached. Each such zero word increases the dynamic value by three.

[0095] This constraint may be expressed as follows. N <= K1 * B + (K2 * CU) Here, N = the number of binarized symbols in the output data section, K1 is a constant, B = the number of encoded bytes in the output data section, K2 is a variable that depends on the characteristics of the minimum size coding unit employed by the image data encoding apparatus, CU = the size of the picture, slice, or tile represented by the output data section expressed in the number of minimum size coding units.

[0096] In previously proposed examples, this threshold check is performed at the picture and slice levels.

[0097] As another method, the following formula of the same threshold may be used. Here, the difference lies in the reference to bytes instead of bits. BinCountsinNalUnits <= (32 / 3) * NumByteslnVclNalUnits + (RawMinCuBits * PicSizelnMinCbsY) / 32 (Equation 2)

[0098] The threshold of Equation 1 (or the threshold equivalently represented by Equation 2) may be uniformly applied, that is, regardless of other encoding or decoding parameters, attributes, etc.

[0099] An example of the technical justification for using such a threshold is that the threshold itself indirectly defines the maximum level of processing performance required by the real-time decoder. Real-time decoding depends on decoding each frame within a time period to achieve a specific output frame rate (e.g., number of frames per second). The constraint on the processing performance of the real-time decoder depends not on the amount of resulting decoded data, but on the speed at which binary values need to be decoded. To enable relatively simple rate control on the encoder side, the technique used to control this situation is to limit the number of binary values per slice or picture (e.g., as an upper limit or threshold or constraint). Then, the maximum processing performance required by the CABAC decoder can be considered as the product or function of this constraint and the speed at which the slice or picture has to be decoded.

[0100] Therefore, it is important to set the threshold (or upper limit or constraint) appropriately. If the threshold is too high, there may be challenges in implementing or the normal operation of the real-time decoder. Similarly, the use of an incorrect or inappropriate threshold may lead to excessive use of padding data (e.g., dummy or other data simply aimed at occupying a portion of the encoded data stream to increase the size of the encoded output data portion), reducing the efficiency of encoding.

[0101] As noted with reference to FIG. 19, a picture or slice can be divided into a plurality of tiles. An example of the reason for this to be done is to enable the use of multiple simultaneous (parallel) decoders.

[0102] Under the previously proposed arrangement, each tile does not necessarily meet the threshold calculation discussed previously. For example, when a tile is used or decoded alone like a picture, or different tiles (such as different quantization parameters or different sources) are combined together, there may be no guarantee that the combined slice or picture complies with the above specifications.

[0103] To address this issue, in an exemplary embodiment, the CABAC threshold is applied at the end of each tile rather than at the end of each slice or the image alone. Therefore, the application of the threshold is performed at the end of the encoding of any of the tile, slice, or picture. Stated otherwise, if each tile within an image complies with the threshold, it can be assumed that the entire picture must also comply, so in the case of a picture divided into slices or tiles, there is no need to apply the threshold again at the end of the encoding of the picture.

[0104] The terms "tile" and "slice" mean independently decodable units and represent the names used at the priority date of this application. In the case of subsequent or other changes to the names, the arrangement is applicable to other such independently decodable units. Therefore, in an exemplary configuration, the output data portion may be an independently decodable data unit. For example, the image portion (represented by the output data portion) may be one of an image, a slice, or a tile.

[0105] To apply the above formula, the dynamic value represents the number of bytes encoded in the output stream of the tile, and the fixed value depends on the number of minimum-size coding units (CUs) within the tile.

[0106] FIG. 20 schematically shows an apparatus configured to perform this test. Referring to FIG. 20, at input 2000, a CABAC / EP encoded stream is received from the encoder. Padding data detector 2010 detects, at a predetermined stage, with reference to the completion of a slice or tile such as the end of encoding of a slice or tile, whether the above-described threshold calculation is compliant. Controller 2020 controls padding data generator 2030 in response to the detection by detector 2010, generates padding data 2040 such as the above-described CABAC zero word, and adds this to the stream by combiner 2050 to form output stream 2060. The generation of the zero word can also send a signal back to detector 2010 to continue monitoring whether padding data detector 2010 is compliant with the threshold when the zero word is added, and after being compliant with the threshold, cause controller 2020 to stop generating the zero word.

[0107] In other examples, controller 2020 may also function as a predictor configured to generate a prediction as to whether constraints are satisfied by its output data portion in relation to the embodiments described herein during the generation of the output data portion.

[0108] The predetermined stage may be, for example, for every n (where n is an integer greater than or equal to 1) encoded binary values, but in the embodiment, the predetermined stage is the end of encoding the current output data portion.

[0109] Accordingly, FIG. 20 discloses an example of padding data detector 2010 configured to detect whether constraints are satisfied by the current output data portion at a predetermined stage for encoding the current output data portion, and padding data generator 2030 configured to generate and insert sufficient padding data into the current output data portion so that the output data portion including the inserted padding data satisfies the constraints.

[0110] Referring to FIG. 21, in some examples, the controller 2020 includes an attribute detector 2070 configured to detect encoding attributes applicable to a given output data portion, and a selector 2080 configured to select a constraint from two or more candidate constraints 2082 for use with the given output data portion according to the detected encoding attributes.

[0111] Further, the controller 2020 may include a comparator 2090 configured to compare a threshold derived from the currently selected constraint with the detection by the padding data detector 2010 in order to derive a control signal for controlling the operation of the padding data generator 2030.

[0112] The attributes detected by the detector 2070 may be, for example, encoding attributes (e.g., encoding mode or profile, e.g., enabling of dependent quantization to be discussed below), which means that the selection of quantization parameters used for quantization of current data values depends at least in part on the characteristics of previously encoded data values and is included in the subsequently encoded data stream or represented by flag data (schematically shown as 2072) related to the encoded data stream, and is an operation mode that can be detected by a decoder later. For example, the flag data may be included in header data such as the header data of the output data portion (e.g., slice header data). The detector 2070 itself does not need to generate or insert flag data. This aspect of the process is shown in FIG. 21 for merely schematic purposes for the benefit of this description. The attribute can be considered applicable to a given output data portion even if (as in some examples) the attribute is also applicable to other (e.g., preceding or subsequent) output data portions.

[0113] Therefore, the apparatuses of FIGS. 7 and 14 operating according to the techniques described with respect to FIGS. 19 and 20 (and those described below) provide an example of an image data encoding apparatus. This apparatus It includes a first data encoder 1450, 1460 and a second data encoder 1420 configured to generate output data bits representing binarized symbols from consecutive symbols representing image data. The first data encoder is configured to generate output data bits representing the encoded symbols at a variable data bit ratio with respect to the number of encoded symbols. The second data encoder is configured to generate a fixed number of output data bits representing each encoded symbol. The entropy encoder is configured to generate an output data stream (e.g., using controller 2020 and / or controller 1435). This output data stream is subject to a constraint that defines an upper limit on the number of binarized symbols represented by the byte size of any individual output data portion with respect to the byte size of its output data portion, and the entropy encoder is configured to provide padding data for each output data portion that does not meet the constraint in order to increase the byte unit size of its output data portion to meet the constraint. The apparatus includes an attribute detector 2070 configured to detect encoding attributes applicable to a given output data portion, and a selector 2080 configured to select a constraint for use with a given output data portion from two or more candidate constraints according to the detected encoding attributes.

[0114] Using the technique shown in FIG. 14, the first data encoder / decoder may be a context adaptive binary arithmetic coding (CABAC) encoder / decoder. The second data encoder / decoder may be a bypass encoder / decoder. The second data encoder / decoder may be a binary arithmetic coder / decoder using a fixed 50% probability context model.

[0115] Examples of the application of thresholds or constraints are as follows.

[0116] Generally, the test is "Does the amount of generated output data meet the threshold test?" This can be performed, for example, last when encoding the output data section.

[0117] The apparatus of FIG. 7 operating according to the techniques discussed herein provides an example of an image data encoding apparatus. The image data encoding apparatus comprises an entropy encoder configured to selectively encode data items representing image data so as to generate encoded binary symbols for successive output data sections. The entropy encoder is configured to generate an output data stream, which is subject to a constraint that defines an upper limit on the number of binary symbols represented by the byte size of any individual output data section relative to the byte size of that output data section. To meet this constraint, the entropy encoder is configured to provide padding data for each output data section that does not meet this constraint in order to increase the size of that output data section in byte units. The image data encoding apparatus comprises an attribute detector configured to detect encoding attributes applicable to a given output data section, and a selector configured to select a constraint for use with a given output data section from among two or more constraint candidates according to the detected encoding attributes.

[0118] Also, the present disclosure provides a suitable decoding apparatus for decoding a data signal generated by the method or apparatus described herein.

[0119] Here, a further example of a constraint will be described.

[0120] In the above example, a single threshold derivation or equation is consistently used. In the alternative examples described below, a choice is implemented between two or more candidate thresholds or constraints. The alternative or choice may be in response to one or more encoded attributes (e.g., a parameter, an attribute or a mode, e.g., a parameter or mode signaled from the encoder side to the decoder side by a flag or the like, or a parameter, an attribute or a mode derivable in a corresponding or matching manner on the encoder side and the decoder side).

[0121] An example of a further candidate equation for a threshold or constraint is as follows. BinCountsinNalUnits <= 10 * NumByteslnVclNalUnits + (RawMinCuBits * PicSizelnMinCbsY) / 16 (Equation 3)

[0122] Thus, this Equation 3 can be used uniformly as described above, but in an embodiment, the choice is made between two or more candidate expressions, e.g., Equation 1 / 2 and Equation 3. In some examples, the choice may be made depending on whether a given encoded attribute signaled in the encoded data stream or signaled together with the encoded data stream is in a first or second state having respective states of a given attribute corresponding to the choice on the encoder side and the decoder side (e.g., Equation 1 / 2 or Equation 3).

[0123] An example of such an attribute is the so-called "dep_quant_enabled_flag". This indicates in the slice header whether a technique called dependent quantization is enabled for the slice to which the slice header applies.

[0124] Instead of using the dep_quant_enabled_flag, it actually depends on the availability of the tool rather than whether this (dep_quant) tool is actually enabled. Thus, for example, in the case of a profile, the profile itself may define that "the dep_quant_enabled_flag should be off (not enabled)". Also, there may be no such constraint and the dep_quant tool can be turned on (enabled) or off (disabled). Thus, this selection can be made based on profile constraints rather than whether the dep_quant tool is currently enabled or disabled.

[0125] Dependent quantization is defined in "Versatile Video Coding (Draft 5), Bross et al, JVET-N1001-v10, July 2019" (incorporated herein by reference). For example, refer to Section 8.7.3. This relates to techniques where the decoding process selects between multiple possible quantization parameters or a sequence of quantization parameters in response to properties of previously encoded and decoded sample values (e.g., parity properties). Thus, when dep_quant_enabled_flag = 1 (enabled), such an ongoing dependent quantization selection is made. When dep_quant_enabled_flag = 0 (disabled), such an ongoing dependent quantization selection is not made. As described above, the flag dep_quant_enabled_flag is provided, for example, in the slice header (as an example of a coding attribute), and enabling or disabling dependent quantization is applied to the entire slice.

[0126] When dependent quantization is used, or at least when it is possible and applicable, different constraints may be relevant due to the empirical observation that the expected relationship between the encoded and decoded binary values can be changed. Different constraints, such as those of Equation (3), may be more suitable for use with dependent quantization.

[0127] If multiple candidate constraints are applicable, the decoder is expected to be subject to design constraints in order to provide sufficient processing power, speed, or capacity to handle the encoded data generated under the more difficult (or most difficult) of the different available constraints.

[0128] However, more generally, such any attribute (e.g., a flag or parameter) can be used, either explicitly within the data stream or whether or not sent with the data stream as a signal. For example, different candidate expressions can be selected for each different instance of what is called a "profile", where the profile in this context defines a set or basket type of parameters such as bit depth, chrominance sampling (e.g., 4:0:0 (monochrome), 4:2:0, 4:2:2, 4:4:4, etc.), restrictions on the encoding type (intra-frame encoding only, etc.).

[0129] In an exemplary embodiment, since the relevant threshold derivation can be applied to its output data portion, attributes that define some aspect of the output data portion can be conveniently used. Examples of the output data portion in this context can include an output data portion representing each image portion such as a slice, tile, or picture. Other examples of suitable attributes include, in the case of a slice, whether the slice type is intra-only or unrestricted (can include intra). As an example of a tile, consider the case of a composition of images from multiple sources (one tile per source). Each tile can have its own threshold derivation depending on the original encoding method. Or the attribute can also be determined using the value of the attribute of the previous tile / output data portion at the same position within the picture.

[0130] In some examples, as described above, the constraint or threshold is defined by an expression such as the following. N <= K1 * B + (K2 * CU) Here N = The number of binarized symbols in the output data part, K1 is a constant, B = The number of encoded bytes in the output data part, K2 is a variable that depends on the characteristics of the minimum size encoding unit adopted by the image data encoding device. CU = The size of the picture, slice, or tile represented by the output data part expressed in the number of minimum size encoding units.

[0131] It should be noted that this is actually a generalization of the above-mentioned Equations 1, 2, and 3. An example of a list of constraint candidates referring to K1 and K2 is as follows. [Table 1]

[0132] Regarding the example of Equation 5, the variable vcIByteScaleFactor can be expressed as follows. vcIByteScaleFactor = (32 + 4 * general_tier_flag) / 3 Here, general_tier_flag is an indicator of the encoding layer and varies as a flag value of 0 or 1 (in at least some examples). 0 indicates the so-called main layer, and 1 indicates the so-called high layer. For a specific encoding level (representing the maximum dimension of the image to be encoded), the high layer generally corresponds to a higher bitrate representation than the main layer. Therefore, in this example, the image data encoding device operates with an encoding layer selected from at least two candidate encoding layers and is configured to generate layer parameters (e.g., encoded image data or related to or within the bitstream) that define the currently selected encoding layer, where at least the constant K1 depends on the layer parameters. For example, the higher the layer (tier) parameter, the higher quality output encoded for a given image size, and parameter K1 may increase according to the layer parameter.

[0133] Thus, the processing, circuitry, code, or logic used to generate the threshold is conveniently the same or substantially the same in each case, and parameters K1 and K2 are simply varied for each of the exemplary expressions (1 / 2) - (7). However, it is understood that one or more different expressions or representations (or potentially different fixed thresholds) can be used, such as between the different candidate thresholds or constraints described above.

[0134] Thus, candidate constraint 2082 can be stored or represented as a pair of (K1, K2) to be selected by selector 2080. The pair of (K1, K2) can be stored in advance, and the encoder and decoder, or the pair (or an indication of a larger subset of a predetermined pair) can be sent from the encoder to the decoder as part of, for example, profile or parameter setting data. Then, the expression N <= K1 * B + (K2 * CU) can be tested in comparator 2090 using the generally selected (K1, K2).

[0135] Figures 22A and 22B provide two schematic examples of the output data portion, extended across a page, and include binarized data 2200 and pad data 2210 provided and inserted by the techniques described above. The amount of padding data can be determined by the particular image data being encoded (in terms of how efficiently it can be encoded) and the constraints in use.

[0136] Figure 23 is a schematic flowchart showing an image data encoding method, the method comprising: (at step 2300) selectively encoding data items representing the image data to generate encoded binarized symbols for successive output data portions; Generating an output data stream that receives a constraint defining an upper limit on the number of binarized symbols represented by any individual output data portion with respect to the size of the output data portion in bytes (at step 2310); Providing padding data for each output data portion that does not satisfy the constraint so as to increase the size of the output data portion in bytes in order to satisfy this constraint (at step 2320); Detecting encoding attributes applicable to a given output data portion (at step 2330); Selecting a constraint for use with a given output data portion from two or more candidate constraints in response to the detected encoding attributes (at step 2340) including.

[0137] Also, an exemplary embodiment provides an image decoder including a circuit configured to interpret an encoded signal generated by controlling an image data encoding apparatus of any one or more of the embodiments described herein and output a decoded video image.

[0138] Also, an exemplary embodiment provides an image decoder including a circuit configured to interpret an encoded signal generated by controlling a first and a second data encoder and any one or more controllers of the embodiments described herein and output a decoded video image.

[0139] The embodiments described herein are implemented in any suitable form including hardware, software, firmware, or any combination thereof. The embodiments described herein may optionally be implemented at least in part as computer software executed on one or more data processors and / or digital signal processors. The components and elements in any embodiment are physically, functionally, and logically implemented in any suitable way. In fact, the functions can be implemented in a single unit, in multiple units, or as part of other functional units. Therefore, the embodiments of the present disclosure may be implemented in a single unit or may be physically and functionally distributed among different units, circuits, and / or processors.

[0140] In light of the above teachings, it will be apparent that numerous modifications and variations of the present disclosure are possible. Therefore, within the scope of the appended claims, it should be understood that the technology can be practiced in ways other than those specifically described herein.

[0141] Each aspect and feature is defined by the following numbered clauses. (1) An entropy encoder configured to selectively encode data items representing image data so as to generate encoded binary symbols of consecutive output data portions, An attribute detector configured to detect encoding attributes applicable to a given output data portion, A selector configured to select, from two or more constraint candidates, a constraint for use with the given output data portion according to the detected encoding attributes comprising The entropy encoder is configured to generate an output data stream, and this output data stream is subject to a constraint that defines an upper limit on the number of binary symbols represented by the byte size of any individual output data portion with respect to the byte size of its output data portion, To satisfy this constraint, the entropy encoder is configured to provide padding data for each output data portion that does not satisfy the constraint in order to increase the size of that output data portion in byte units. An image data encoding apparatus. (2) The entropy encoder is configured to selectively encode data items representing image data to be encoded by a first, context-adaptive binary arithmetic coding (CABAC), coding system, or a second, bypass, coding system in order to generate encoded binary symbols. The image data encoding device according to (1). (3) The image data represents one or more pictures, and each picture (i) one or more slices within each network abstraction layer (NAL) unit, and (ii) zero or more tiles that define horizontal and vertical boundaries of each picture area and are not restricted to being encapsulated within each NAL unit, and includes data representing Each slice of a picture is decodable independently of other slices of the same picture, and the tiles are decodable independently of other tiles of the same picture. The output data unit includes one or more pictures, slices, and tiles. The image data encoding device according to (1). (4) The second coding system is a binary arithmetic coding system that uses a fixed 50% probability context model. The image data encoding device according to (1) or (2). (5) The constraint is N <= K1 * B + (K2 * CU) (Constraint formula 1) where N = the number of binary symbols in the output data unit, K1 is a constant, B = the number of encoded bytes in the output data unit, K2 is a variable that depends on the characteristics of the minimum size coding unit employed by the image data encoding device, CU = the size of the picture, slice, or tile represented by the output data unit, represented by the number of minimum size coding units. is defined as The image data encoding device according to any one of (1) to (4). (6) At least two candidate constraints are defined by the constraint formula 1, each of a series of (K1, K2) is associated with each of the at least two candidate constraints, the selector is configured to select a series of (K1, K2) for the given output data part The image data encoding device according to (5). (7) The controller is configured to encode the representation of the encoding attribute applicable to the predetermined output data part in relation to the output data stream representing the predetermined output data part The image data encoding device according to any one of (1) to (6). (8) The image data encoding device includes a quantization unit configured to selectively operate in a dependent quantization mode, the encoding attribute indicates whether the dependent quantization mode is valid or invalid for a given output data part The image data encoding device according to (7). (9) The entropy encoder includes a detector configured to detect whether the constraint is satisfied by the current output data part at a predetermined stage for encoding the current output data part, and a padding data generator configured to generate and insert sufficient padding data into the current output data part so that the output data part including the inserted padding data satisfies the constraint and The image data encoding device according to any one of (1) to (8). (10) The predetermined stage is the end of encoding the current output data part The image data encoding device according to (6). (11) A video storage device, a capture device, and a transmission / reception device including the image data encoding device according to any one of (1) to (10). Selectively encoding data items representing image data to generate encoded binarized symbols in successive output data portions; Generating an output data stream that is subject to a constraint that defines an upper limit on the number of binarized symbols represented by any individual output data portion relative to the size of the output data portion in bytes; Providing padding data for each output data portion that does not satisfy the constraint so as to increase the size of the output data portion in bytes to satisfy the constraint; Detecting encoding attributes applicable to a given output data portion; Selecting, in response to the detected encoding attributes, a constraint for use with the given output data portion from among two or more candidate constraints; Including An image data encoding method. (13) Computer software that, when executed by a computer, causes the computer to execute the method according to (12). (14) A machine-readable non-transitory storage medium storing the computer software according to (13). (15) A data signal including encoded data generated according to the method according to (12). (16) An image data decoder configured to decode the data signal according to (15).

Claims

Storing in a buffer an input data stream representing image data that conforms to a constraint that defines an upper limit on the number of binarized symbols that can be represented by each output data unit according to the byte size of the output data unit, and providing padding data for input data units that do not satisfy the constraint, thereby increasing the byte size of the input data unit so as to satisfy the constraint, where the constraint is selected based on a mode or profile of the input data stream that satisfies the design constraints of an entropy decoder. Decoding data items of the input data stream representing the image data by a first context-adaptive binary arithmetic coding (CABAC) decoding system or a second bypass decoding system, and using attributes within the input data stream to determine the ability to handle the selected constraint, thereby generating decoded binarized symbols. Including An image data encoding method.

2. The constraint is N <= K1 * B + (K2 * CU) (Constraint Equation 1) where N = the number of binarized symbols in the output data portion, K1 is a constant, B = the number of encoded bytes in the output data portion, K2 is a variable that depends on the characteristics of the minimum-size coding unit employed by the image data encoding apparatus, CU = the size of the picture, slice, or tile represented by the output data portion, expressed in the number of minimum-size coding units defined as The image data encoding method according to Claim 1.

3. At least two candidate constraints are defined by the Constraint Equation 1, each of a series of (K1, K2) is associated with each of the at least two candidate constraints, and the selector selects a series of (K1, K2) for the given output data portion. The image data encoding method according to Claim 2.

4. The mode or profile is notified by a flag within the data stream. The image data encoding method according to Claim 1.

5. An attribute within the data stream is a flag derived by a decoding circuit. The image data encoding method according to Claim 1.

6. A flag determining the selection of the constraint is used to determine the ability of a decoder circuit to handle the selected constraint. The image data encoding method according to Claim 4.

7. The flag for notifying the profile defines a series of restrictions applied to the input stream The method for encoding image data according to claim 4 **Claim 8**: A computer program which, when executed by a computer, causes the computer to execute the method according to claim 1 **Claim 9**: A computer-readable storage medium storing the computer program according to claim 8 **Claim 10**: A buffer configured to receive and store an input data stream representing image data that conforms to a constraint that defines an upper limit on the number of binarized symbols that can be represented by each output data unit with respect to the byte size of the output data unit, wherein padding data is provided for each input data unit that does not meet the constraint, and the byte size of the input data unit is increased to meet a constraint selected based on the mode or profile of the input data stream that meets the design constraints of the entropy decoder A decoding circuit that decodes data items of the input data stream representing the image data by a first context-adaptive binary arithmetic coding (CABAC) decoding system or a second bypass decoding system, and generates decoded binarized symbols by determining the ability to handle the selected constraints using attributes within the input data stream An image data encoding apparatus having the same **Claim 11** The constraint is N <= K1 * B + (K2 * CU) (Constraint formula 1) where N = the number of binarized symbols in the output data part K1 is a constant B = the number of encoded bytes in the output data part K2 is a variable that depends on the characteristics of the minimum-size coding unit adopted by the image data encoding apparatus and CU = the size of the picture, slice or tile represented by the output data part represented by the number of minimum-size coding units is defined as The image data encoding apparatus according to claim 10 **Claim 12** At least two candidate constraints are defined by the constraint formula 1 Each of a series of (K1, K2) is associated with each of the at least two candidate constraints The selector is configured to select a series of (K1, K2) for the given output data part The image data encoding apparatus according to claim 11 **Claim 13** The mode or profile is signaled by a flag within the data stream The image data encoding apparatus according to claim 10 **Claim 14** An attribute within the data stream is a flag derived by a decoding circuit The image data encoding apparatus according to claim 10 **Claim 15** A flag that determines a selection of constraints is used to determine the capabilities of a decoder circuit to handle the selected constraints The image data encoding apparatus according to claim 14 **Claim 16** A flag that signals a profile defines a set of restrictions applied to an input stream The image data encoding apparatus according to claim 14 **Claim 17** A video storage device, a capture device, and a transmission / reception device including the image data encoding apparatus according to claim 10

Images