Method for encoding image data, encoder for performing same, and non-transitory computer readable medium - Patents.com

Abstract

To provide a coding method capable of improving display stream technology.SOLUTION: A method for coding an image data block indicating an image by using an encoder includes a step of defining first and second regions of an image by an encoder, and a step of assigning a first number of bits, containing basic bits, to coding of the first region, and assigning a second number of bits, containing basic bits and enhancement bits, to coding of the second region, by means of the encoder. Sum of the number of first pixels in the first region and the number of second pixels in the second region is same as the total number of pixels of the image, sum of the number of first bits and the number of second bits is same as the total number of bits for encoding all pixels, and the second region is encoded with the number of bits per pixel larger than that of the first region.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to display compression, and more particularly to a method for encoding image data, an encoder and a non-transitory computer readable medium for performing the same.

[0002] This application claims priority to U.S. Patent Application No. 62 / 713,464, filed in the United States Patent and Trademark Office on August 1, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0003] As display technology advances, the resolution of the data that generates images on display devices has improved significantly. As pixel bandwidth associated with display technology continues to increase, especially for mobile and virtual reality (VR) display devices, the Video Electronics Standard Association (VESA) has announced a Call for Technologies (CfT) for display stream codec, which supports higher compression rates at the expense of more complex calculations than Display Stream Compression (DSC).

[0004] Display compression allows higher resolution data to be transmitted over the existing display link bandwidth (e.g. DSC and VDC-M standardized by VESA). VESA has developed a new display interface compression standard for mobile / smartphone display devices called VDC-M (VESA Display Codec for Mobile). VDC-M, as a new display compression standard, aims to achieve a higher compression ratio than the DSC (Display Stream Compression) standard while maintaining the same visual lossless quality as the DSC standard, which is more difficult to achieve.

[0005] Compared to image / video standards (e.g. JPEG2000 and HEVC standards), display compression coders / encoders (e.g. DSC, VDC-M) are generally lightweight and offer visually lossless performance. DSC and VDC-M are fixed rate coders that can compress down to 8 and 6 bits per pixel, respectively, for a 24-bit color source. For example, VDC-M is a fixed rate display stream codec that has a fixed bit rate and can compress down to 6 bits per pixel for an RGB 4:4:4 source. VDC-M is a block-based codec with 8x2 pixel blocks.

[0006] The field of view (FOV) of a VR headset is, for example, about 150 degrees horizontally and 135 degrees vertically, respectively. However, due to the inherent characteristics of the human eye, humans cannot uniformly perceive the entire 150 x 135 degree field of view.

[0007] For example, the small depression in the retina of the human eye is called the macula (fovea), and it is the most visually sensitive area. The retinal cones are particularly concentrated in this area, so the center of the human eye's field of vision is focused on this area. Therefore, the area of ​​best human perception (or highest resolution) is the area covered by the macula. The field of vision corresponding to the macula is approximately + / - 3 degrees.

[0008] The contents described in this Background of the Invention are merely intended to aid in understanding of the embodiments of the present invention, and may include technical information obtained in the process of implementing the present invention. Therefore, some of the contents described above may not be related art. Summary of the Invention [Problem to be solved by the invention]

[0009] The problem that the present invention seeks to solve is to improve display device technology in general, and to improve virtual reality technology in particular, by improving display stream compression. [Means for solving the problem]

[0010] An encoding method according to one embodiment of the present invention is a method of encoding an image data block representing an image using an encoder, comprising the steps of: defining a first region of the image and a second region of the image by the encoder; and allocating, by the encoder, a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, wherein a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels of the image, a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.

[0011] The second region may include a high resolution region corresponding to a gaze fixation point of a user viewing the image on a display device, and the first region may include a low resolution region outside the gaze fixation point.

[0012] The method may further include defining, by the encoder, a third region of the image, the third region being encoded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region.

[0013] The method may further include the steps of determining, by the encoder, a number of pixels per block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter for the current block; encoding, by the encoder, the current block with the quantization parameter; updating, by the encoder, a number of remaining pixels of the image to be encoded; and updating, by the encoder, a number of remaining bits for encoding the remaining pixels.

[0014] The method may further include determining, by the encoder, a basic number of allocated bits per block based on the remaining number of pixels of the image to be encoded, the remaining number of bits available for encoding the remaining pixels of the image, and the remaining number of pixels of the second region to be encoded.

[0015] If the current block belongs to the second region, the method may further include calculating, by the encoder, a number of remaining enhancement bits available for encoding the current block based on the number of pixels of the second region and a number of enhancement bits per block, where the number of enhancement bits per block may indicate a difference between a target bit per block of the second region and a target bit per block of the first region.

[0016] The method may further include a step of calculating, by the encoder, a difference between the remaining number of bits available for encoding the remaining pixels of the image and the remaining number of enhancement bits, and a step of calculating, by the encoder, an allocated number of basic bits per block of the first region and the second region based on the remaining number of pixels of the image and the difference.

[0017] The encoder may further include calculating, if the current block of the image is within the second region, a total number of allocated bits per block for encoding the current block based on a sum of the number of allocated basic bits per block of the first region and the number of enhanced bits per block.

[0018] The quantization parameter of the current block may be determined based on a quantization parameter of a block encoded immediately before the current block, the number of bits used to encode the block immediately before the current block, and the number of bits allocated to the current block, and if the current block of the image is within the first region, the number of bits allocated to the current block may be the basic bits allocated per block of the first region, and if the current block of the image is within the second region, the number of bits allocated to the current block may be the total number of bits allocated per block of the second region.

[0019] According to another embodiment of the present invention, an image data encoder includes a memory for buffering image data, and a processor for controlling the memory, wherein the processor defines a first region of the image and a second region of the image, and the encoder allocates a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels of the image, a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.

[0020] The second region may include a high resolution region corresponding to a gaze fixation point of a user viewing the image on a display device, and the first region may include a low resolution region outside the gaze fixation point.

[0021] The processor may define a third region of the image, the third region being encoded with a greater number of bits per pixel than the first region and a lesser number of bits per pixel than the second region.

[0022] The processor may determine a number of pixels per block, receive a current block of the image, determine a quantization parameter for the current block, encode the current block with the quantization parameter, update a number of remaining pixels of the image to be encoded, and update a number of remaining bits for encoding the remaining pixels.

[0023] The processor may determine an allocated base number of bits per block based on the remaining number of pixels of the image to be encoded, the remaining number of bits available for encoding the remaining pixels of the image, and the remaining number of pixels of the second region to be encoded.

[0024] When the current block belongs to the second region, the processor may calculate a remaining number of enhancement bits available for encoding the current block based on the second region pixel count and a number of enhancement bits per block, and the number of enhancement bits per block may indicate a difference between a target bit per block of the second region and a target bit per block of the first region.

[0025] The processor may calculate a difference between the remaining number of bits available for encoding the remaining pixels of the image and the remaining enhancement number of bits, and calculate an allocated basic number of bits per block of the first region and the second region based on the remaining number of pixels of the image and the difference.

[0026] The processor may calculate, if the current block of the image is within the second region, a total number of allocated bits per block for encoding the current block based on the sum of the allocated basic bits per block and the enhanced bits per block of the first and second regions.

[0027] The quantization parameter of the current block may be determined based on a quantization parameter of a block encoded immediately before the current block, the number of bits used to encode the block immediately before the current block, and the number of bits allocated to the current block, and if the current block of the image is within the first region, the number of bits allocated to the current block may be the basic bits allocated per block of the first region, and if the current block of the image is within the second region, the number of bits allocated to the current block may be the total number of bits allocated per block of the second region.

[0028] According to another embodiment of the present invention, a non-transitory computer readable medium is provided for an image data encoder including a memory for encoding image data and buffering the image data and a processor for controlling the memory, the non-transitory computer readable medium storing instructions for performing an image data encoding method when executed by the processor, the image data encoding method including the steps of: defining a first region of the image and a second region of the image; and allocating a first number of bits including base bits to encoding the first region and a second number of bits including base bits and enhancement bits to encoding the second region, wherein a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels of the image, a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.

[0029] The method of encoding image data may further include defining a third region of the image, the third region being encoded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region. Effect of the Invention

[0030] Therefore, a codec or encoder according to an embodiment of the present invention can improve the method of compressing image data by defining different resolution regions, allocating bits differently to image blocks depending on the corresponding region, and updating the available remainder after block encoding. [Brief description of the drawings]

[0031] [Figure 1] 1 illustrates an example of an image including different resolution regions having different resolutions according to one embodiment of the present invention. [Diagram 2] 4 is a flowchart illustrating a method of allocating various numbers of bits to blocks during compression of a display stream according to an embodiment of the present invention. [Diagram 3] 1 illustrates an image having different resolution regions and a conceptual diagram of multi-resolution chiaroscuro according to an embodiment of the present invention; [Figure 4] 4 is a flowchart illustrating a method of allocating different numbers of bits to blocks in different resolution regions during compression of a display stream according to an embodiment of the present invention. [Diagram 5] 2 shows an image of a display device having four resolution regions with different resolutions according to one embodiment of the present invention. [Figure 6] 1 is a flowchart illustrating a method of allocating various numbers of bits to a plurality of blocks (i.e., three or more regions) of various resolution regions during a display stream compression process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] As mentioned above, the perception of the human eye is not uniform from the center to the outer edge of the visual field. In order to reduce the bandwidth, it is useful to compress image data according to how the displayed image data is perceived by the user. That is, in order to reduce the bandwidth, it is more useful to compress image data corresponding to the outer edge of the image than image data corresponding to the center of the image.

[0033] Also, while the display resolution of display screens continues to increase, the bandwidth of the internal links that carry data from the application processor to the display device has not increased at the same rate.

[0034] It would therefore be useful to enhance the system by providing areas with different levels of resolution on the display screen of a device such as a virtual reality headset.

[0035] FIG. 1 illustrates an example of an image including different resolution regions having different resolutions according to one embodiment of the present invention.

[0036] Referring to Figure 1, an image 100 is shown on a display device, such as a display device of a virtual reality headset. Image 100 may include N x M pixels, where N and M are integers. Although the term "image" is used herein, it should be understood that this term can equally be used for a "slice of an image" (or a portion of an image). That is, although discussion is given of compressing an image, it should be understood that the discussion can equally be used for compressing a slice of an image.

[0037] In an embodiment of the present invention, multi-resolution shading is used for the displayed image 100 to reflect human perception. That is, the image 100 implements multi-resolution shading to reflect the differences in human perception corresponding to the human field of vision. Thus, since the human visual acuity is not uniform across the entire field of vision, the visual quality is also not uniform across the entire image 100.

[0038] For example, a higher resolution may be used for high resolution regions 110 of image 100 in areas corresponding to where human perception is highly sensitive (e.g., areas of the display device corresponding to the macula of the human eye). In one example, high resolution regions 110 may be visually lossless (e.g., high resolution regions 110 may display uncompressed display image data or display image data compressed with a lossless compression ratio). Additionally, a lower resolution may be used for low resolution regions 130 of display device image 100 corresponding to less sensitive areas (e.g., areas of the display device corresponding to regions toward the outer edges of the human visual field).

[0039] Between the high resolution region 110 and the low resolution region 130 of the display device image 100, one or more different intermediate resolution regions 120 having intermediate resolution are provided to allow a smooth transition of perception from the center to the edge of the image 100 (see, for example, FIG. 5). That is, the display device of this embodiment has one intermediate resolution region 120 having a resolution at an intermediate level between the resolutions of the high resolution region 110 and the low resolution region 130, but is not limited to this in other embodiments. According to other embodiments, a plurality of layers or regions having different resolution levels (e.g., various intermediate resolution levels) are provided to allow a virtual reality headset user's perception to transition naturally from the low resolution region 130 to the high resolution region 110.

[0040] The various high resolution regions 110, medium resolution regions 120, and low resolution regions 130 shown in Figure 1 are merely examples. In other embodiments, the number, location, and pattern of the regions can be determined through analysis and testing (e.g., after compensation with virtual reality headset lenses). For example, in other embodiments, the image 100 in Figure 1 has high resolution, medium resolution, and low resolution regions 110, 120, 130 that are rectangular, but the various high resolution, medium resolution, and low resolution regions may be circular, elliptical, or discontinuous.

[0041] Thus, blocks of an image may be coded differently depending on the resolution level of the region in which they are located, and the display device may use multi-resolution virtual reality compression based on a multi-resolution shading scheme. For example, different high resolution regions 110, medium resolution regions 120, and low resolution regions 130 may be assigned bits (e.g., regions having higher resolution may be assigned more bits per pixel or more bits per block) depending on the expected human perceptual sensitivity level of the user of the virtual reality headset that includes the display device. However, the encoder of the display codec is generally unable to analyze the entire image before image compression begins due to memory limitations. That is, the encoder generally has no prior knowledge of the characteristics of different image blocks that it has not yet received, but analyzes those blocks after they have been received.

[0042] In the case of single resolution ratio control in VDC-M, if there is an image 100 having a total of P pixels and a total of B bits (P and B are integers) allocated to the pixels, in an embodiment of the present invention, how to allocate or allocate the number of bits per block (e.g., 8×2 pixel block) to improve or maximize quality can be determined as follows: That is, since a certain pixel block is assigned a larger number of bits than other blocks, and since it is generally not known how bits will be allocated to various blocks before starting compression of the image 100 (e.g., pixel blocks are compressed in series when image data is input to a compression device such as a VDC-M encoder), in this embodiment, a method of determining how to allocate bits as image compression progresses using a multi-resolution contrast method will be presented below.

[0043] For each pixel block (e.g., each pixel block contains 16 pixels and is arranged in an 8×2 pixel array), the remaining or available bits of the image (e.g., image 100 in FIG. 1) are calculated as “B r " and the remaining pixels of image 100 "P rBased on this, we can determine or calculate "bitsAllocated" (e.g., the number of bits allocated for the block, or the average number of bits per block). In the example presented below, for the first block of the display stream, r =B and P r If the block corresponds to 16 pixels, the average number of bits allocated to each block can be calculated as in Equation 1.

number

[0044] Therefore, according to Equation 1, the remaining bits can be divided equally among the remaining pixels (e.g., each block can be assigned the same number of bits), which may be an acceptable strategy, or even the best strategy, when the complexity of the next block is unknown (e.g., when the compression ratio of the block being compressed is unknown).

[0045] However, because image 100 may have varying levels of complexity or texture in different regions, it may be useful to vary the number of bits allocated to different blocks as the compression process proceeds, i.e., it may be useful to adjust bitsAllocated based on the average compression ratio as the compression process proceeds.

[0046] Therefore, the complexity of the current pixel block relative to other neighboring pixel blocks (e.g., previous blocks located in the area of ​​the current block) can be estimated based on the bitsAllocated determined by Equation 1. The complexity measure is calculated by dividing the quantization parameter (QP) value of the current block ("curBlock QP") by the quantization parameter (QP) value of the previous block ("prevBlock QP") using Equation 2. QP ") can be used to determine or calculate

number

[0047] In Equation 2, δ QP is a positive number if the complexity of the current block is greater than that of the adjacent previous block, is a negative number if the complexity of the current block is less than that of the adjacent previous block, and may be 0 if the complexity of the current block and the previous block are the same or similar.

[0048] By constantly computing the complexity for each block, and then δ QP Based on curBlock QP By adjusting the number of bits, the number of bits that is not used will not exceed the total number of bits ("B") allocated to the image 100, and the number of bits that are left when the processing of the image 100 is completed ("B r "). That is, in conjunction with the remainder of the processing according to this embodiment, Equation 2 can ensure that too many or too few bits are not used in processing image 100.

[0049] Therefore, "curBlock QP The number of bits allocated to the current block by the determined QP, which can be used to calculate ", may be adjusted to be more or less than the average number of bits allocated to the remaining blocks (e.g., "bitsAllocated"). As the QP increases, the step size corresponding to the image also increases along with the image distortion, but the corresponding bit rate decreases.

[0050] (For example, curBlock QP Once the number of bits allocated to the current block has been determined (based on B r ") can then be updated. The first and second operations described above can then be repeated for the next block.

[0051] The flowchart of FIG. 2, which summarizes the above description, illustrates a method of allocating various numbers of bits to blocks during compression of a display stream according to an embodiment of the present invention.

[0052] Referring to FIG. 2, in step S210, the encoder according to this embodiment can determine (e.g., according to Equation 1) the number of bits allocated per block (e.g., the average number of bits per block for the image) based on the number of remaining bits of the image and the number of remaining pixels in the image.

[0053] However, it should be noted that determining "bitsAllocated" may involve floating point calculations between two large integers. Also, the number of remaining pixels (P r ) changes as pixel blocks are processed and encoded, it may be impractical to pre-calculate "bitsAllocated". To avoid such floating-point calculations, a fixed-point divider can be used to calculate "bitsAllocated". Also, after determining a fixed-point target rate approximation, a quadratic LUT can be used to adjust the target rate approximation. In addition, bitshifting can be used to avoid a division operation to calculate "bitsAllocated".

[0054] For example, for each pixel block, a target bit rate per block (TR) is calculated based on the number of bits (B) and the number of pixels (P) remaining in the image or video slice. ideal ) is updated. Therefore, if there are 16 pixels per block, TIFF0007621031000003.tif15168. To avoid floating point calculations, Use a fixed-point approximation such as TIFF0007621031000004.tif12168.

[0055] Therefore, the denominator (P) in the formula is It can be converted to TIFF0007621031000005.tif14168. Then, TIFF0007621031000006.tif21168, LUT function for TIFF0007621031000007.tif14168 TIFF0007621031000008.tif18168 can be saved into a LUT.

[0056] Also, since the quality of the first line of slices (FLS) is useful (e.g., by prediction), TR for FLS is 0 You can add an offset parameter to the TIFF0007621031000009.tif13168, where δ FLS allocates extra bits to blocks in the first row of an image or video slice, TIFF0007621031000010.tif13168=16·2=32.

[0057] In step S220, the encoder may determine a quantization parameter for a current block received by the encoder for compression based on a quantization parameter of at least one neighboring block. According to one embodiment, the at least one neighboring block is a block that was encoded immediately before the current block. The quantization parameter for the current block is Find TIFF0007621031000011.tif17168.

[0058] In step S230, the encoder may determine the number of bits to encode the current block based on the determined quantization parameter of the current block (e.g., the encoder may determine the number of bits to encode the current block based on the determined quantization parameter of the current block). QP(The current block may be encoded based on the determined QP.) Then, in step S240, the encoder may encode the current block with the determined QP and transmit the block to the decoder.

[0059] In step S250, the encoder updates the number of remaining bits to be used in compressing the image based on the number of remaining pixels of the image to be compressed (e.g., the encoder updates the number of remaining bits to be used in compressing the image based on the number of remaining pixels of the image to be compressed). r and P r can be updated).

[0060] In step S260, the encoder may receive the next pixel block to be compressed, and may return to step S210 until it has coded the last block of image 100. If no more blocks remain, the encoder may end image compression in step S270.

[0061] In the conventional case, for single resolution compression, the VDC-M rate control allocates bits based on the quantization parameters of blocks in a given portion of the image. Thus, more bits can be allocated to blocks in complex portions (e.g., image regions that compress less, such as regions corresponding to more textured natural images) and fewer bits can be allocated to blocks in uniform or flat portions (e.g., computer-generated or natural regions that can be compressed at higher rates without significantly sacrificing the quality of the image 100).

[0062] Alternatively, in accordance with the present embodiment, in the case of multi-resolution compression, the encoder may allocate more bits to blocks in the high resolution region 110 than to blocks in the low resolution region 130. For example, fewer bits may be allocated to the medium resolution region 120 and low resolution region 130, more bits may be allocated to the high resolution region 110 to maintain high quality, and more bits may be allocated to the medium resolution region 120 than to the low resolution region 130.

[0063] Given an image 100 having a total of P pixels, and an encoder having a total of B bits available for encoding the image 100, embodiments of the present invention may appropriately allocate the available bits for a current pixel block differently based on quality differences for the different high resolution regions 110, medium resolution regions 120, and low resolution regions 130, or may do so based on complexity differences within one or more of the high resolution regions 110, medium resolution regions 120, and low resolution regions 130.

[0064] FIG. 3 illustrates an image having different resolution regions and the concept of multi-resolution chiaroscuro according to one embodiment of the present invention.

[0065] Referring to FIG. 3, the multi-resolution chiaroscuro method according to the present embodiment can reflect the difference in human perception because the difference corresponds to different regions of the visual field (e.g., the region corresponding to the center of the visual field corresponds to higher visual acuity and can correspond to a gaze fixation region or a high resolution region), and can selectively allocate the bit rate of the encoder based on the difference. For ease of explanation, the display image 300 of the present embodiment includes only two different resolution regions [e.g., a high resolution region 310 and a low resolution region 330]. However, the present embodiment can be extended to additional resolution regions [e.g., the medium resolution region 120 of the image 100 shown in FIG. 1 or the medium resolution regions 520, 525 of the image 500 shown in FIG. 5 described below]. In the embodiment, the high resolution and medium resolution regions can also be referred to as enhanced regions.

[0066] In this embodiment, and for ease of explanation of the equations described below, the low resolution region 330 is referred to as "region 1" and the high resolution region 310 is referred to as "region 0." During processing, these regions 310, 330 of the image 300 may be defined by a bitmask, where one region may be defined by one value in the bitmask (e.g., a value of 1) and the other region may be defined by another value in the bitmask (e.g., a value of 0).

[0067] The bits per block allocated to a given region [for example, “region i” (i is an integer corresponding to the region, i=1 or 0 in this embodiment)] is “b i According to the present embodiment, the number of bits per block (b 0 ) is the number of bits per block (b 1 ) is greater than.

number

[0068] The number of pixels in region i is P i The total number of pixels in the image (P) is the number of pixels in each region (P i In other words, as shown in Equation 4, in this embodiment, the total number of pixels (P) in the image 300 is the sum of the number of pixels (P) in the high resolution region 310. 0 ) and the number of pixels in the low-resolution region 330 (P 1 ) is the sum of

number

[0069] Similarly, the number of bits allocated for coding region i is B i It is. B i is the number of bits per block for region i (b i ) to the total number of pixels in that area (P i ) divided by the number of pixels per block (16 per block for an 8x2 pixel block in this example). This can be expressed as Equation 5:

number

[0070] Thus, in this example, the total number of bits (B) allocated to coding image 300 is less than the total number of bits for low resolution region 330 ("B 1") and the total number of bits for the high resolution region 310 ("B 0 In a manner similar to Equation 4, the total number of bits (B) for the entire image 300 can be expressed by Equation 6.

number

[0071] This embodiment presents a new solution by dividing the bits per block for the high resolution region 310 into "base bits" per block and "enhancement bits" per block. Since no enhancement bits are allocated to blocks in the low resolution region 330, the base bits for each block in the high resolution region 310 will be the same as the bits per block available in the low resolution region 330 (e.g., b 1 That is, the total number of bits (B) will be partitioned into base bits for the entire image (e.g., all of regions 310, 330) and enhancement bits for only the enhancement regions (e.g., high resolution region 310 of image 300).

[0072] Therefore, the total number of bits allocated to the high resolution region 310 (B 0 ) can be expressed by Equation 7.

number

[0073] where TIFF0007621031000017.tif14168 The term is basic bits, which corresponds to the bits per block for both regions 310,330. The TIFF0007621031000018.tif15168 term is the enhancement bit and is allocated to the high resolution region 310. The number of available enhancement bits can be used to calculate the "enhancement bits per block" for the blocks of the high resolution region 310. Because the two regions 310, 330 use the same basic number of bits per block, the number of enhancement bits per block is the difference between the target number of bits per block for the high resolution region 310 and the target number of bits per block for the low resolution region 330 [i.e., (b 0 -b 1 According to one embodiment, the enhancement bits per block is a fixed number.

[0074] For example, if the size of the image 300 is 160×160 pixels (i.e., P=160 2 ), and the size of the high-resolution region 310 is 32×32 pixels (i.e., P 0 =32 2 ), then according to the above formula, the bits per block in the high resolution region 310 is 85 [i.e., b 0 = 85, bits per pixel ("bpp 0 b = 5.3125, which corresponds to a compression ratio of 1.5:1 assuming that each pixel in image 300 is represented using 8 bits. Also, in this example, the bits per block for low resolution region 330 are 16 [i.e., b 1 = 16, and bpp 1 =1.0, which corresponds to a compression ratio of 8:1, assuming that each pixel in image 300 is represented using 8 bits.

[0075] Therefore, by using the above Equation 7, the total number of bits (B 0) is 5,440 bits, which is the sum of 1,024 basic bits and 4,416 enhancement bits in the high resolution region 310. Also, in this example, according to Equation 6 above, the total number of bits available for the entire image 300 is 30,016 bits, of which 25,600 basic bits are evenly distributed to the entire image 300 including the high resolution region 310 and the low resolution region 330, and 4,416 enhancement bits are used only for the high resolution region 310.

[0076] Since enhancement bits can be scarce, the present embodiment tracks the number of available enhancement bits remaining during the encoding process. Thus, the bitsAllocated per block for the high resolution region 310 and the low resolution region 330 can be determined as follows:

[0077] The basic number of allocated bits per block can be determined for the entire image 300, including the high resolution region 310 and the low resolution region 330. The basic number of allocated bits per block can be determined by multiplying the number of allocated bits per block for the low resolution region (e.g., (bitsAllocated) described below). 1 ]. The number of enhancement bits from the total number of bits allocated to the image 300 may be the same as The value subtracted from TIFF0007621031000019.tif16168 is allocated to the entire image 300 based on the complexity of the entire image 300 (eg, in the same manner as in the case of VDC-M single resolution).

[0078] For the high resolution region 310, the base bits per block plus the enhancement bits per block may determine the total allocated bits per block for the high resolution region 310 (i.e., the base bits per block plus the enhancement bits per block). Thus, the additional enhancement bits (e.g., the bits remaining after the base bits are allocated) may be allocated to the high resolution region 310, which may generally improve image quality.

[0079] The multi-resolution chiaroscuro method according to the present embodiment is It can be distinguished from VDC-M single resolution in that it continues to track the remaining pixel blocks of the high resolution region 310 remaining in the image 300 being coded by its dependency on TIFF0007621031000020.tif18168. Once all blocks of the high resolution region 310 have been coded, The file becomes TIFF0007621031000021.tif15168.

[0080] According to this embodiment, the encoder may encode the bits as follows:

[0081] The bitsAllocated of each block is the remaining bits of the image 300 (i.e., B r ), the remaining pixels of the image 300 (i.e., P r ) and based on the high resolution region 310 (P 0 r ) may be continually updated based on the remaining pixels in the low-resolution region 330. Thus, if each pixel block is 8×2 (i.e., 16 pixels per block), then the number of bitsAllocated per block for the low-resolution region 330 is “(bitsAllocated) 1 " and can be expressed by Equation 8.

number

[0082] For blocks in the high resolution region 310, the bits allocated per block ["(bitsAllocated) 0 "] is determined using Equation 9.

number

[0083] Here, (b 0 -b 1 ) may indicate the enhancement bits per block that can be allocated to the high resolution region 310 relative to the allocated bits per block of the low resolution region 330 (e.g., since the low resolution region 330 is coded with only basic bits, the allocated basic bits per block for the entire image 300 is (bitsAllocated) 1 (It could be).

[0084] As in the VDC-M rate control, the QP value of the current block to be coded received by the encoder is determined (bitsAllocated). 1 and may be determined based on the complexity of the current block relative to the neighboring blocks.

[0085] FIG. 4, which summarizes the above description, is a flow chart illustrating a method of allocating various numbers of bits to blocks in various resolution regions during compression of a display stream according to an embodiment of the present invention.

[0086] Referring to FIG. 4, in step S400, the encoder according to the present embodiment can determine the number of basic bits of the entire image. TIFF0007621031000024.tif16168 can be determined.

[0087] In step S405, the encoder can determine the number of enhancement bits only for the high resolution region. TIFF0007621031000025.tif17168 can be determined.

[0088] In step S410, the encoder can determine the number of basic bits allocated per block for both the low-resolution region and the high-resolution region. TIFF0007621031000026.tif23164], which may be identical to the base number of bits allocated for all remaining blocks.

[0089] In step S415, the encoder may determine a total number of allocated bits per block for the high resolution region by adding the allocated number of base bits per block to the allocated number of enhancement bits per block for the high resolution region. TIFF0007621031000027.tif11164 can be determined).

[0090] In step S420, the encoder may determine a quantization parameter for the current block based on the quantization parameters of one or more neighboring blocks that have been previously encoded. TIFF0007621031000028.tif13164 can be determined.

[0091] In step S430, the encoder may determine the number of bits for the current block based on the quantization parameter of the current block, and then in step S440, the encoder may remove bits from a pool of remaining bits that are not used by encoding the current block with the QP and transmit the current block to the decoder.

[0092] At step S450, the encoder can update the number of remaining bits, the number of remaining pixels of the image, and the number of remaining pixels of the high resolution region.

[0093] In step S460, the encoder may receive the next pixel block to be compressed, or may return to step S410 until it has coded the last block of image 300. If no more blocks remain, the encoder may end image compression in step S470.

[0094] FIG. 5 shows an image of a display device having four resolution regions with different resolutions according to one embodiment of the present invention.

[0095] 5, an embodiment of the present invention, as previously described, may be modified to be applicable to a display device having R regions (R is an integer, in this example 4). The image 500 in this embodiment has a high resolution region 510, a low resolution region 530 and two intermediate resolution regions 520, 525.

[0096] According to this embodiment, Equation 3 described above can be changed to express bits per block "" for a given region "region i" as in Equation 10.

number

[0097] Equation 4 described above can be changed to express the total number of pixels (P) in image 500 as in Equation 11 below.

number

[0098] Similarly, Equation 6 described above can be changed to express the total number of bits (B) allocated to the image 500 as in Equation 12 below.

number

[0099] Therefore, the above-described Equation 7 can be expressed as the total bits “B i " can be transformed into something that expresses "

number

[0100] Here, the basic bits assigned to all the regions 510, 520, 525, and 530 of the image 500 are The enhancement bits allocated to the remaining regions 510, 520, and 525 except for the low resolution region 530 are It may also be represented as TIFF0007621031000034.tif16168.

[0101] Using such a formula, the above embodiment can be extended to more regions by separating each region i (i ≠ R-1) into base bits and enhancement bits.

[0102] For each block, the remaining bits (B r ), the remaining pixels in image 500 (P r ), and the remaining pixels located in regions 0, 1, …, R-2, respectively. TIFF0007621031000035.tif15168 We can update bitsAllocated (size 8×2) based on

[0103] Therefore, the above-described Equation 8 can be changed to the following Equation 14, which expresses the number of basic bits allocated per block.

number

[0104] For blocks in region R-1 [i.e., low-resolution region 530], the allocated bits per block are (bitsAllocated) R-1 That is, the basic bits per block allocated for all regions may be the same as the bits per block allocated for all low resolution regions 530.

[0105] The enhancement bits per block for the regions 510, 520, 525 other than the low resolution region 530 [eg, region i (i ≠ R−1)] can be determined by changing Equation 9 to the following Equation 15.

number

[0106] Here, the enhancement bits per block corresponding to the blocks in each enhancement region i (i ≠ R-1) are (b i -b R-1 ) may also be expressed as

[0107] The QP value for the current block that the encoder is to encode has been determined (bitsAllocated) (e.g., as done in VDC-M rate control). 1 and the complexity of the current block relative to the neighboring blocks. Much like the example described above with reference to Figure 4, the encoder can then assign a QP to the current block based on the determined complexity, encode and transmit the current block, and update the number of remaining base bits and the number of remaining enhancement bits. The encoder then moves on to the next block.

[0108] FIG. 6, which summarizes the above-described contents, is a flowchart showing a method of allocating various numbers of bits to multiple blocks (i.e., three or more regions) of various resolution regions during the compression of a display stream according to one embodiment of the present invention.

[0109] Referring to FIG. 6, in step S600, the encoder according to the present embodiment can determine the number of basic bits of the entire image. TIFF0007621031000038.tif17170 can be determined.

[0110] In step S605, the encoder may determine the number of enhancement bits for the regions other than the low resolution region. TIFF0007621031000039.tif18170 can be determined.

[0111] In step S610, the encoder can determine the number of basic bits to be allocated per block for both the low-resolution region and the high-resolution region. TIFF0007621031000040.tif18170 can be determined.

[0112] In step S615, the encoder may determine a total number of allocated bits per block for each region other than the low resolution region, which may be equal to the average number of allocated enhancement bits per block for the enhancement region. TIFF0007621031000041.tif14170 can be determined.

[0113] In step S620, the encoder may determine a quantization parameter for the current block based on previously determined quantization parameters of one or more blocks adjacent to the current block that were previously encoded. TIFF0007621031000042.tif13170 can be determined.

[0114] In step S630, the encoder may determine the number of bits for encoding the current block based on the determined quantization parameter and based on the resolution domain the current block is located in. Then, in step S640, the encoder may encode the current block with the determined QP and transmit the encoded block to the decoder.

[0115] At step (S650), the encoder may update the number of remaining bits, the number of remaining pixels of the image, and the number of remaining pixels of each region other than the low resolution region.

[0116] In step S660, the encoder may receive the next pixel block to be compressed, and may return to step S610 until it has coded the last block of image 500. If no more blocks remain, the encoder may end image compression in step S670.

[0117] According to the above example, the fixation point (e.g., the point of the image corresponding to the user's focus) can be considered to be at the center of the image. However, due to eye movement, the fixation point may not correspond to the center of the image. Therefore, the embodiment of the present invention can be equally applied to the case where the fixation point is not at the center of the image. In the latter case, the high resolution, medium resolution and low resolution regions may be defined differently depending on the selected fixation point, and the presented technique may be equally applied.

[0118] Also, embodiments of the present invention are applicable to all fixed rate codecs that use a block-based solution of size NxM. For example, embodiments of the present invention are applicable to codecs with or without a rate buffer. If the codec includes a rate buffer, the determined bitsAllocated per block are 1 may be further adjusted depending on how full or empty the encoder buffer is (e.g., depending on the buffer status or depending on the buffer fullness). That is, before determining the QP value described above, the bits per block "bitsAllocated" may be adjusted depending on the "buffer fullness" value as shown in the following Equation 16.

number

[0119] Here, buffer fullness (BF) is an abstract value indicating the number of bits in the encoder buffer, and δ(BF) is an offset value that is a function (e.g., a monotonically decreasing function) of BF. According to one embodiment, δ(BF) may be zero for all regions except the high resolution region or the enhancement region. According to another embodiment, δ(BF) may be zero only for the low resolution region.

[0120] Thus, based on the limitations of a codec designed according to the embodiments described herein, the encoder can subsequently determine whether the buffer is relatively full. If the buffer is relatively full (e.g., close to a state that would cause an overflow error), the encoder can increase the QP and reduce the bitrate. Similarly, if the buffer is relatively empty, the encoder can decrease the QP and increase the bitrate.

[0121] Thus, the encoder can use the available bits to encode the image while intelligently allocating different numbers of bits per block to different regions to provide different regions with different levels of resolution, and can encode the image without using up all the available bits before encoding the last block of the image. Thus, by using the embodiments described above, display stream technology can be improved.

[0122] The concept of the present invention and the gist of the method for achieving the same can be easily understood through the detailed description of the invention and the accompanying drawings. The present invention can be realized in various different forms and is not limited to the embodiments described herein. By providing such embodiments, the detailed description of the invention is sufficient and rich, and various aspects and features of the invention can be fully shown to those skilled in the art. Therefore, the description of processes, devices, techniques, etc. that are not necessary for those skilled in the art to fully understand the various aspects and features of the present invention can be omitted. Unless otherwise specified, the same reference numerals in the drawings and the specification indicate the same components, and the description thereof will be omitted. In addition, in order to clarify the description, parts that are not related to the description of the embodiments may not be shown. In the drawings, parts, layers, regions, etc. may be exaggerated for clear understanding.

[0123] Various embodiments will be described herein with reference to cross-sectional views showing schematic structures and / or intermediate structures of the embodiments. The illustrated patterns may be modified or changed in various ways, for example, depending on manufacturing techniques and / or tolerances. Furthermore, the description of specific structures or functions described herein is merely exemplary to explain the embodiments according to the concepts of the present invention. Thus, the embodiments described herein are not limited to the specific patterns of the illustrated regions, but may include variations in the patterns, for example, depending on the manufacturing method. Thus, the patterns of the regions illustrated in the drawings are schematic in nature and are not intended to show or limit the actual patterns of the regions in the device. In addition, it will be appreciated that those skilled in the art may modify or change the above-described embodiments in various different ways without departing from the spirit or scope of the present invention.

[0124] In the detailed description, various conditions are specified to provide a thorough description of the various embodiments. However, it will be apparent that the embodiments may be practiced without these conditions or their equivalent. Instead, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the various embodiments.

[0125] The terms used in the specification are used only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the embodiments, unless a number is specifically stated, the singular and plural cases are all included. The expression "comprising" a feature, step, operation, part, component, etc. means that other features, steps, operations, parts, components, etc. may be included in addition to the part in question. The expression "and / or" includes all combinations of one or more of the items in the list.

[0126] If a particular embodiment is implemented differently, the order of certain processes may vary from that described. For example, two processes described as being performed sequentially may be performed simultaneously or in the reverse order from that described.

[0127] The electronic, electrical devices and / or other related devices or portions described in accordance with the embodiments of the present invention may be implemented using appropriate hardware, firmware (e.g., application-customized integrated circuits), software, or a combination thereof. For example, the various components of the devices may be implemented on a single integrated circuit chip, or on different integrated circuit chips. The various components of the devices may also be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board, or the like, or may be formed on a single substrate. The various components of the devices may also be processes or threads that may be executed by one or more processors in one or more computing devices that execute computer program instructions and interact with other system elements to perform the various functions described herein. The computer program instructions may be stored in memory implemented in the computing device using standard memory devices such as random access memory (RAM). Moreover, those skilled in the art may combine or consolidate the functionality of the various computing devices into a single computing device, or distribute the functionality of a particular computing device to one or more other computing devices, without departing from the spirit and scope of the embodiments of the present invention.

[0128] Unless otherwise specified, all terms (including technical and scientific terms) used herein have the same meaning as commonly known to those skilled in the art to which the present invention belongs. Terms, such as terms defined in commonly used dictionaries, are to be interpreted as having a meaning consistent with the meaning in the relevant technical field and / or in this specification, and should not be interpreted in an ideal or overly strict sense unless expressly stated herein.

[0129] Although specific terms have been used to describe the embodiments, such terms should be interpreted in a general sense for the purpose of description and not for the purpose of limitation. For example, unless otherwise specified, it is clear to a person skilled in the art at the time of filing that the features, characteristics and / or components described in connection with a particular embodiment can be used alone or in combination with features, characteristics and / or components described in connection with other embodiments. Therefore, it is clear to a person skilled in the art that various changes in form and details can be made without departing from the spirit and scope of the present invention as presented in the claims and the functional equivalents inherent therein. [Explanation of symbols]

[0130] 100, 300, 500:Image 110, 310, 510: High resolution area 120, 520, 525: Medium resolution area 130, 330, 530: High resolution area

Claims

1. 1. A method of encoding a block of image data representing an image using an encoder, comprising the steps of: defining, by the encoder, a first region of the image and a second region of the image; allocating, by the encoder, a first number of bits, including only base bits, to the coding of the first region, and a second number of bits, including only base bits and enhancement bits, to the coding of the second region; determining, by said encoder, a number of pixels per block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter for the current block; encoding the current block with the quantization parameter by the encoder; updating, by the encoder, a number of pixels remaining of the image to be encoded; updating, by the encoder, a number of remaining bits for encoding pixels corresponding to the number of remaining pixels; Including, a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels in the image; a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels; the second region is encoded with a larger number of bits per pixel than the first region; Encoding method.

2. the second region includes a high resolution region corresponding to a gaze fixation point indicating a point on the image corresponding to a focal point of a user viewing the image on a display device; the first region includes a low-resolution region outside the gaze fixation point; The encoding method according to claim 1 .

3. 2. The method of claim 1, further comprising determining, by the encoder, an allocated basic number of bits per block based on the remaining number of pixels of the image to be encoded, the remaining number of bits available for encoding pixels corresponding to the remaining number of pixels of the image, and the remaining number of pixels of the second region to be encoded.

4. If the current block belongs to the second region, the encoder may further include calculating, by the encoder, a number of remaining enhancement bits available for encoding the current block based on a number of remaining pixels of the second region and a number of enhancement bits per block; the enhanced number of bits per block indicates a difference between a target number of bits per block of the second region and a target number of bits per block of the first region; The encoding method according to claim 3.

5. calculating, by the encoder, a difference between the number of remaining bits available for encoding the remaining pixels of the image and the number of remaining enhancement bits; calculating, by the encoder, an allocated number of basic bits per block of the first region and the second region based on the remaining number of pixels of the image and the difference; The encoding method of claim 4 further comprising:

6. 6. The method of claim 5, further comprising the step of: calculating, by the encoder, if the current block of the image is within the second region, a total number of allocated bits per block for encoding the current block based on a sum of the number of allocated basic bits per block of the first region and the number of enhanced bits per block.

7. the quantization parameter of the current block is determined based on a quantization parameter of a block encoded immediately before the current block, a number of bits used to encode the block immediately before the current block, and a number of bits allocated to the current block; if the current block of the image is within the first region, the number of bits allocated to the current block is the basic bits allocated per block of the first region; if the current block of the image is within the second region, the number of bits allocated to the current block is the total number of bits allocated per block of the second region; The encoding method according to claim 6.

8. a memory for buffering image data representing an image; A processor for controlling the memory; Including, The processor, Defining a first region of the image and a second region of the image; allocating, by an encoder, a first number of bits for encoding the first region, the first number of bits including only base bits, and a second number of bits for encoding the second region, the second number of bits including only base bits and enhancement bits; Calculate the number of pixels per block. receiving a current block of the image; determining a quantization parameter for the current block; encoding the current block with the quantization parameter; updating the number of pixels remaining of the image to be encoded; updating a number of remaining bits for encoding pixels corresponding to the number of remaining pixels; a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels in the image; a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels; the second region is encoded with a greater number of bits per pixel than the first region; Image data encoder.

9. the second region includes a high resolution region corresponding to a gaze fixation point indicating a point on the image corresponding to a focal point of a user viewing the image on a display device; the first region includes a low-resolution region outside the gaze fixation point; 9. The image data encoder of claim 8.

10. 9. The image data encoder of claim 8, wherein the processor determines an allocated basic number of bits per block based on the remaining number of pixels of the image to be encoded, the remaining number of bits available for encoding pixels corresponding to the remaining number of pixels of the image, and the remaining number of pixels of the second region to be encoded.

11. the processor calculates a number of remaining enhancement bits available for encoding the current block based on a number of remaining pixels of the second region and a number of enhancement bits per block when the current block belongs to the second region; the enhanced number of bits per block indicates a difference between a target number of bits per block of the second region and a target number of bits per block of the first region; 11. The image data encoder of claim 10.

12. The processor, calculating a difference between the remaining number of bits available for encoding the remaining pixels of the image and the remaining enhancement number of bits; calculating an allocated basic number of bits per block of the first region and the second region based on the remaining number of pixels of the image and the difference; 12. The image data encoder of claim 11.

13. the processor calculates a total number of allocated bits per block for encoding the current block of the image based on a sum of the number of allocated basic bits per block and the number of enhanced bits per block of the first and second regions if the current block of the image is within the second region; 13. The image data encoder of claim 12.

14. the quantization parameter of the current block is determined based on a quantization parameter of a block encoded immediately before the current block, a number of bits used to encode the block immediately before the current block, and a number of bits allocated to the current block; if the current block of the image is within the first region, the number of bits allocated to the current block is the basic bits allocated per block of the first region; if the current block of the image is within the second region, the number of bits allocated to the current block is the total number of bits allocated per block of the second region; 14. The image data encoder of claim 13.

15. 1. A non-transitory computer-readable medium for an image data encoder that encodes image data representing an image, the non-transitory computer-readable medium including a memory for buffering the image data and a processor for controlling the memory, the non-transitory computer-readable medium comprising: A program for causing a computer to execute a method for encoding image data when executed by the processor, comprising: The image data encoding method includes: defining a first region of the image and a second region of the image; allocating a first number of bits to the coding of the first region, the first number including only base bits, and allocating a second number of bits to the coding of the second region, the second number including only base bits and enhancement bits; determining a number of pixels per block; receiving a current block of the image; determining a quantization parameter for the current block; encoding the current block with the quantization parameter; updating, by the encoder, a number of pixels remaining of the image to be encoded; updating, by the encoder, a number of remaining bits for encoding pixels corresponding to the number of remaining pixels; Including, a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels in the image; a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels; a program recorded on the computer to cause the second region to be encoded with a greater number of bits per pixel than the first region; A computer-readable recording medium.

Images