Method for encoding image data blocks, image data encoder and non-transitory computer-readable medium

Abstract

To provide an encoder that can improve a display stream technology.SOLUTION: An encoding method according to an embodiment of the present invention is a method for encoding an image data block representing an image using an encoder, includes the steps of defining a first region of the image and a second region of the image by the encoder, and allocating a first number of bits including base bits to encoding of the first region and a second number of bits including base bits and enhancement bits to encoding of the second region by the encoder, where a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels of the image, a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to display compression, and more particularly to a method for encoding image data, an encoder for performing the same, and a non-transitory computer-readable medium.

[0002] This application claims priority to U.S. Patent Application No. 62 / 713,464, filed in the United States Patent and Trademark Office on August 1, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0003] As display technology advances, the resolution of the data that generates images on display devices has improved significantly. As pixel bandwidth associated with display technology continues to increase, especially for mobile display devices and virtual reality (VR) display devices, the Video Electronics Standards Association (VESA) announced a Call for Technologies (CfT) for display stream codecs, which support higher compression rates despite being more computationally complex than Display Stream Compression (DSC).

[0004] Display compression allows higher resolution data to be transmitted over existing display link bandwidth (e.g., DSC and VDC-M standards developed by VESA). VESA has developed a new display interface compression standard for mobile / smartphone display devices, called VDC-M (VESA Display Codec for Mobile). As a new display compression standard, VDC-M aims to achieve a higher compression ratio than the DSC (Display Stream Compression) standard while maintaining the same lossless visual quality as the DSC standard, which makes it more difficult to achieve.

[0005] Compared to image / video standards (e.g., JPEG2000 and HEVC), display compression coders / encoders (e.g., DSC, VDC-M) are generally lightweight and offer visually lossless performance. DSC and VDC-M are fixed-rate coders that can achieve compression rates down to 8 and 6 bits per pixel, respectively, for a 24-bit color source. For example, VDC-M is a fixed-rate display stream codec that has a constant bitrate and can achieve compression rates down to 6 bits per pixel for an RGB 4:4:4 source. VDC-M is a block-based codec with 8x2 pixel blocks.

[0006] The field of view (FOV) of a VR headset is, for example, approximately 150 degrees horizontally and 135 degrees vertically, but due to the inherent characteristics of the human eye, humans cannot uniformly perceive the entire 150 x 135 degree field of view.

[0007] For example, the macula is a small depression in the retina of the human eye, and it is the most visually sensitive area. The retinal cones are particularly concentrated in this area, so the center of the human eye's visual field is focused on this area. Therefore, the area with the best human perception (or the highest resolution) is the area covered by the macula. The visual field corresponding to the macula is approximately + / - 3 degrees.

[0008] The content described in this Background Art section is intended to aid in understanding the embodiments of the present invention and may include technical information obtained in the process of implementing the present invention. Therefore, some of the content described above may not be prior art. Summary of the Invention [Problem to be solved by the invention]

[0009] The problem that the present invention seeks to solve is to improve display technology in general, and in particular to improve virtual reality technology, for example, by improving display stream compression. [Means for solving the problem]

[0010] An encoding method according to one embodiment of the present invention is a method for encoding an image data block representing an image using an encoder, comprising the steps of: defining a first region of the image and a second region of the image by the encoder; and allocating a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region by the encoder; wherein a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels of the image; a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all pixels; and wherein the second region is encoded with a larger number of bits per pixel than the first region.

[0011] The second region may include a high-resolution region corresponding to a gaze fixation point of a user viewing the image on a display device, and the first region may include a low-resolution region outside the gaze fixation point.

[0012] The method may further include defining a third region of the image using the encoder, the third region being encoded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region.

[0013] The method may further include determining, by the encoder, the number of pixels per block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter for the current block; encoding, by the encoder, the current block with the quantization parameter; updating, by the encoder, the number of remaining pixels of the image to be encoded; and updating, by the encoder, the number of remaining bits for encoding the remaining pixels.

[0014] The method may further include determining, by the encoder, a number of basic bits allocated per block based on the number of remaining pixels of the image to be encoded, the number of remaining bits available for encoding the remaining pixels of the image, and the number of remaining pixels of the second region to be encoded.

[0015] If the current block belongs to the second region, the encoder may further calculate a number of remaining enhancement bits available for encoding the current block based on the number of pixels in the second region and a number of enhancement bits per block, where the number of enhancement bits per block may indicate a difference between a target bit per block for the second region and a target bit per block for the first region.

[0016] The method may further include a step of calculating, by the encoder, a difference between the number of remaining bits available for encoding the remaining pixels of the image and the number of remaining enhancement bits, and a step of calculating, by the encoder, a number of basic bits allocated per block of the first region and the second region based on the number of remaining pixels of the image and the difference.

[0017] The encoder may further include calculating, if the current block of the image is within the second region, a total number of allocated bits per block for encoding the current block based on the sum of the number of basic bits allocated per block of the first region and the number of enhancement bits per block.

[0018] The quantization parameter of the current block is determined based on the quantization parameter of the block coded immediately before the current block, the number of bits used to code the block immediately before the current block, and the number of bits allocated to the current block. If the current block of the image is in the first region, the number of bits allocated to the current block may be the basic number of bits allocated per block of the first region, and if the current block of the image is in the second region, the number of bits allocated to the current block may be the total number of bits allocated per block of the second region.

[0019] According to another embodiment of the present invention, an image data encoder includes a memory for buffering image data and a processor for controlling the memory, wherein the processor defines a first region of the image and a second region of the image, and the encoder allocates a first number of bits including basic bits for encoding the first region and a second number of bits including basic bits and enhancement bits for encoding the second region, wherein the sum of the first number of pixels in the first region and the second number of pixels in the second region is equal to the total number of pixels of the image, the sum of the first number of bits and the second number of bits is equal to the total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.

[0020] The second region may include a high-resolution region corresponding to a gaze fixation point of a user viewing the image on a display device, and the first region may include a low-resolution region outside the gaze fixation point.

[0021] The processor may define a third region of the image, the third region being coded with a greater number of bits per pixel than the first region and a lesser number of bits per pixel than the second region.

[0022] The processor may determine the number of pixels per block, receive a current block of the image, determine a quantization parameter for the current block, encode the current block with the quantization parameter, update the number of remaining pixels of the image to be encoded, and update the number of remaining bits for encoding the remaining pixels.

[0023] The processor may determine the basic number of allocated bits per block based on the remaining number of pixels of the image to be encoded, the remaining number of bits available for encoding the remaining pixels of the image, and the remaining number of pixels of the second region to be encoded.

[0024] If the current block belongs to the second region, the processor calculates the number of remaining enhancement bits available for encoding the current block based on the number of pixels in the second region and the number of enhancement bits per block, and the number of enhancement bits per block may indicate the difference between the target bits per block of the second region and the target bits per block of the first region.

[0025] The processor can calculate the difference between the number of remaining bits available for encoding the remaining pixels of the image and the number of remaining enhancement bits, and calculate the allocated basic number of bits per block of the first region and the second region based on the number of remaining pixels of the image and the difference.

[0026] If the current block of the image is within the second region, the processor may calculate a total number of allocated bits per block for encoding the current block based on the sum of the basic number of allocated bits per block and the enhancement number of bits per block of the first and second regions.

[0027] The quantization parameter of the current block is determined based on the quantization parameter of the block coded immediately before the current block, the number of bits used to code the block immediately before the current block, and the number of bits allocated to the current block. If the current block of the image is in the first region, the number of bits allocated to the current block may be the basic number of bits allocated per block of the first region. If the current block of the image is in the second region, the number of bits allocated to the current block may be the total number of bits allocated per block of the second region.

[0028] According to another embodiment of the present invention, a non-transitory computer-readable medium is provided for an image data encoder that encodes image data and includes a memory for buffering the image data and a processor for controlling the memory, and stores instructions for performing an image data encoding method when executed by the processor, the image data encoding method including the steps of defining a first region of the image and a second region of the image, and allocating a first number of bits including base bits to encoding the first region and a second number of bits including base bits and enhancement bits to encoding the second region, wherein the sum of the first number of pixels in the first region and the second number of pixels in the second region is equal to the total number of pixels of the image, the sum of the first number of bits and the second number of bits is equal to the total number of bits for encoding all pixels, and the second region is encoded with a larger number of bits per pixel than the first region.

[0029] The image data encoding method may further include defining a third region of the image, the third region being encoded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region. [Effects of the Invention]

[0030] Therefore, a codec or encoder according to an embodiment of the present invention can improve the image data compression method by defining different resolution regions, allocating bits to image blocks differently depending on the corresponding region, and updating the available remainder after block encoding. [Brief explanation of the drawings]

[0031] [Figure 1] 1 illustrates an example of an image including different resolution regions having different resolutions according to one embodiment of the present invention. [Figure 2] 1 is a flowchart illustrating a method of allocating various numbers of bits to blocks during a display stream compression process according to an embodiment of the present invention. [Figure 3] 1 illustrates an image having different resolution regions and a conceptual diagram of multi-resolution chiaroscuro according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating a method of allocating different numbers of bits to blocks in different resolution regions during a display stream compression process according to an embodiment of the present invention. [Figure 5] 1 shows an image of a display device having four resolution regions with different resolutions according to one embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating a method of allocating various numbers of bits to a plurality of blocks (i.e., three or more regions) in various resolution regions during a display stream compression process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] As mentioned above, the perception of the human eye is not uniform from the center to the outer edge of the field of view. To reduce bandwidth, it is useful to compress image data depending on how the displayed image data is perceived by the user. That is, to reduce bandwidth, it is more useful to compress image data corresponding to the outer edge of the image than image data corresponding to the center of the image.

[0033] Additionally, while display resolutions on display screens have continually increased, the bandwidth of the internal links that carry data from the application processor to the display device has not increased at the same rate.

[0034] Therefore, it is useful to enhance the system by providing areas with different levels of resolution on the display screen of a device such as a virtual reality headset.

[0035] FIG. 1 illustrates an example of an image including different resolution regions having different resolutions according to one embodiment of the present invention.

[0036] Referring to Figure 1, an image 100 is shown on a display device, such as a display device in a virtual reality headset. Image 100 can include N x M pixels (N and M are integers). While the term "image" is used herein, it should be understood that this term can equally be used for a "slice of an image" (or a portion of an image). That is, while discussion of compressing an image is provided herein, it should be understood that the discussion can also be used for compressing a slice of an image.

[0037] In an embodiment of the present invention, multi-resolution shading is used for the displayed image 100 to reflect human perception. That is, the image 100 implements multi-resolution shading to reflect the differences in human perception corresponding to the human field of vision. Therefore, because human visual acuity is not uniform across the entire field of vision, the visual quality is also not uniform across the entire image 100.

[0038] For example, a higher resolution may be used in high-resolution regions 110 of image 100 in areas corresponding to where human perception is most sensitive (e.g., areas of the display device corresponding to the macula of the human eye). By way of example, high-resolution regions 110 may be visually lossless (e.g., high-resolution regions 110 may display uncompressed display image data or display image data compressed at a lossless compression ratio). Additionally, a lower resolution may be used in low-resolution regions 130 of display device image 100 corresponding to less sensitive areas (e.g., areas of the display device corresponding to regions toward the periphery of the human visual field).

[0039] One or more different intermediate resolution regions 120 having intermediate resolutions are provided between the high resolution region 110 and the low resolution region 130 of the display device image 100, allowing for a smooth transition of perception from the center to the edges of the image 100 (see, for example, FIG. 5). That is, the display device of this embodiment has one intermediate resolution region 120 having a resolution intermediate between the resolutions of the high resolution region 110 and the low resolution region 130, but is not limited to this in other embodiments. According to other embodiments, multiple layers or regions having different resolution levels (e.g., various intermediate resolution levels) are provided, allowing for a natural transition of perception of the virtual reality headset user from the low resolution region 130 to the high resolution region 110.

[0040] The various high-resolution regions 110, medium-resolution regions 120, and low-resolution regions 130 shown in Figure 1 are exemplary only. In other embodiments, the number, location, and pattern of the regions can be determined through analysis and testing (e.g., after compensation with virtual reality headset lenses). For example, in other embodiments, the high-resolution, medium-resolution, and low-resolution regions 110, 120, and 130 in image 100 of Figure 1 are rectangular, but the various high-resolution, medium-resolution, and low-resolution regions can be circular, elliptical, or non-contiguous.

[0041] Thus, image blocks may be coded differently depending on the resolution level of the region in which they are located, and a display device may use multi-resolution virtual reality compression based on multi-resolution shading. For example, bits may be allocated to different high-resolution, medium-resolution, and low-resolution regions 110, 120, and 130 depending on the expected level of human perceptual sensitivity of a user of a virtual reality headset including the display device (e.g., a higher number of bits per pixel or per block may be allocated to regions with higher resolution). However, due to memory limitations, the encoder of a display codec typically cannot analyze the entire image before starting image compression. That is, the encoder typically does not have prior knowledge of the characteristics of different image blocks that have not yet been received, but analyzes those blocks after they are received.

[0042] In the case of single resolution ratio control in VDC-M, if there is an image 100 having a total of P pixels and a total of B bits (P and B are integers) allocated to those pixels, in an embodiment of the present invention, how to allocate or allocate the number of bits per block (e.g., an 8x2 pixel block) to improve or maximize quality can be determined as follows: That is, because a specific pixel block is assigned a larger number of bits than other blocks, and because it is generally not known how bits will be allocated to various blocks before starting compression of the image 100 (e.g., because pixel blocks are compressed serially when image data is input to a compression device such as a VDC-M encoder), in this embodiment, a method for determining how to allocate bits as image compression progresses using a multi-resolution contrast method will be presented below.

[0043] For each pixel block (e.g., containing 16 pixels each and arranged in an 8x2 pixel array), the remaining or available bits of the image (e.g., image 100 in Figure 1) are calculated as "B r ” and the remaining pixels of image 100 “P r" can be used to determine or calculate "bitsAllocated" (e.g., the number of bits allocated to the block, or the average number of bits per block). In the example presented below, for the first block of the display stream, B r =B and P r If the block corresponds to 16 pixels, the average number of bits allocated to each block can be calculated as in Equation 1.

number

[0044] Therefore, according to Equation 1, the remaining bits can be divided evenly among the remaining pixels (e.g., each block can be assigned the same number of bits), which may be an acceptable strategy, or even the best strategy, when the complexity of the next block is unknown (e.g., when the compression ratio of the block being compressed is unknown).

[0045] However, because image 100 can have varying levels of complexity or texture in different regions, it is useful to vary the number of bits allocated to different blocks as the compression process progresses. In other words, it is useful to adjust bitsAllocated based on the average compression rate as the compression process progresses.

[0046] Therefore, the complexity of the current pixel block relative to other neighboring pixel blocks (e.g., previous blocks located in the area of ​​the current block) can be estimated based on the bitsAllocated determined by Equation 1. The complexity measure is calculated by using Equation 2 to calculate the QP value of the current block ("curBlock QP") based on the quantization parameter (QP) value of the previous block ("prevBlock QP"). QP ").

number

[0047] In Equation 2, δ QP is a positive number if the complexity of the current block is greater than that of the adjacent previous block, a negative number if the complexity of the current block is less than that of the adjacent previous block, and 0 if the complexity of the current block and the previous block are the same or similar.

[0048] By constantly calculating the complexity for each block, and δ QP Based on curBlock QP By adjusting the number of bits, the number of bits that will be used will not exceed the total number of bits ("B") allocated to the image 100, and the number of bits that will be left when the processing of the image 100 is complete ("B") r "). That is, in conjunction with the remainder of the processing according to this embodiment, Equation 2 can ensure that too many or too few bits are not used in processing image 100.

[0049] Therefore, "curBlock QP The number of bits allocated to the current block by the determined QP, which can be used to calculate ", may be adjusted to be more or less than the average number of bits allocated to the remaining blocks (e.g., "bitsAllocated"). As the QP increases, the step size corresponding to the image also increases along with the image distortion, but the corresponding bit rate decreases.

[0050] (e.g. curBlock QP Once the number of bits allocated to the current block is determined (based on B r ") can be updated. Then, the first and second operations described above can be repeated for the next block.

[0051] The flowchart of FIG. 2, which summarizes the above-described contents, illustrates a method of allocating various numbers of bits to blocks during the compression of a display stream according to an embodiment of the present invention.

[0052] Referring to FIG. 2, in step S210, the encoder according to this embodiment can determine the number of bits allocated per block (e.g., the average number of bits per block for the image) based on the number of remaining bits of the image and the number of remaining pixels in the image (e.g., using Equation 1).

[0053] However, it should be noted that determining "bitsAllocated" may involve floating point calculations between two large integers. r Because 'bitsAllocated' changes as pixel blocks are processed and encoded, it may be impractical to pre-calculate 'bitsAllocated'. To avoid such floating-point calculations, 'bitsAllocated' can be calculated using a fixed-point divider. Also, after determining the fixed-point target rate approximation, a quadratic LUT can be used to adjust the target rate approximation. In addition, bitshifting can be used to avoid the division operation used to calculate 'bitsAllocated'.

[0054] For example, for each pixel block, a target bit rate per block (TR) is determined based on the number of bits (B) and number of pixels (P) remaining in the image or video slice. ideal ) is updated. Therefore, if there are 16 pixels per block, JPEG0007793017000003.jpg1230. To avoid floating point calculations, Use a fixed-point approximation such as JPEG0007793017000004.jpg827.

[0055] Therefore, the denominator (P) in the formula is It can be converted to JPEG0007793017000005.jpg854. Then, JPEG0007793017000006.jpg1378, LUT function for JPEG0007793017000007.jpg935 JPEG0007793017000008.jpg1327 can be saved in the LUT.

[0056] Also, since the quality of the first line of slice (FLS) is useful (e.g., by prediction), an offset parameter can be added to TR0 relative to the FLS. Thus, JPEG0007793017000009.jpg1151, where δ FLS allocates extra bits to blocks in the first row of the image or video slice, JPEG0007793017000010.jpg978=16·2=32.

[0057] In step S220, the encoder may determine a quantization parameter of the current block received for compression based on the quantization parameter of at least one neighboring block. According to one embodiment, the at least one neighboring block is a block encoded immediately before the current block. The quantization parameter of the current block is Calculate using JPEG0007793017000011.jpg11102.

[0058] In step S230, the encoder may determine the number of bits to encode the current block based on the determined quantization parameter of the current block (e.g., the encoder may determine the number of bits to encode the current block based on the determined quantization parameter of the current block). QP(The current block can be encoded based on the QP.) Then, in step S240, the encoder can encode the current block with the determined QP and transmit the block to the decoder.

[0059] In step S250, the encoder updates the number of remaining bits to be used in compressing the image based on the number of remaining pixels of the image to be compressed (e.g., the encoder updates B r and P r can be updated).

[0060] In step S260, the encoder can receive the next pixel block to be compressed and can return to step S210 until it has coded the last block of image 100. If there are no more blocks remaining, the encoder can end image compression in step S270.

[0061] In the conventional case, for single-resolution compression, the VDC-M rate control allocates bits based on the quantization parameters of blocks in a given portion of the image. Thus, more bits can be allocated to blocks in complex portions (e.g., image regions that compress less, such as regions corresponding to more textured natural images) and fewer bits can be allocated to blocks in uniform or flat portions (e.g., computer-generated or natural regions that can be compressed at a higher rate without significantly sacrificing the quality of the image 100).

[0062] Alternatively, in the case of multi-resolution compression, the encoder may allocate more bits to blocks in the high resolution region 110 than to blocks in the low resolution region 130. For example, fewer bits may be allocated to the medium resolution region 120 and the low resolution region 130, and more bits may be allocated to the high resolution region 110 to maintain high quality, and more bits may be allocated to the medium resolution region 120 than to the low resolution region 130.

[0063] Given an image 100 having a total of P pixels, and an encoder capable of using a total of B bits to encode the image 100, embodiments of the present invention may allocate the available bits to the current pixel block in different ways based on the quality differences between the different high resolution regions 110, medium resolution regions 120, and low resolution regions 130, or may allocate the bits in different ways based on the complexity differences within one or more of the high resolution regions 110, medium resolution regions 120, and low resolution regions 130.

[0064] FIG. 3 illustrates an image having different resolution regions and the concept of multi-resolution chiaroscuro according to one embodiment of the present invention.

[0065] Referring to FIG. 3, the multi-resolution chiaroscuro method according to this embodiment can reflect differences in human perception because they correspond to different regions of the visual field (e.g., the region corresponding to the center of the visual field corresponds to higher visual acuity and may correspond to a gaze fixation region or a high-resolution region), and can selectively allocate the bit rate of the encoder based on these differences. For simplicity, the display image 300 of this embodiment includes only two different resolution regions [e.g., high-resolution region 310 and low-resolution region 330]. However, this embodiment can be extended to additional resolution regions [e.g., the medium-resolution region 120 of image 100 shown in FIG. 1 or the medium-resolution regions 520 and 525 of image 500 shown in FIG. 5, described below]. In this embodiment, the high-resolution and medium-resolution regions may also be referred to as enhanced regions.

[0066] In this embodiment, and for ease of explanation of the equations described below, the low-resolution region 330 is referred to as "region 1," and the high-resolution region 310 is referred to as "region 0." During processing, these regions 310, 330 of the image 300 may be defined by a bitmask, with one region being defined by one value (e.g., a value of 1) in the bitmask and the other region being defined by another value (e.g., a value of 0) in the bitmask.

[0067] The bits per block allocated to a given region (for example, "region i" (i is an integer corresponding to the region, i=1 or 0 in this embodiment)) is "b i According to this embodiment, as shown in Equation 3, the number of bits per block (b0) for the high resolution region 310 (i.e., region 0) is greater than the number of bits per block (b1) for the low resolution region 330 (i.e., region 1). b0>b1 (Equation 3)

[0068] The number of pixels in region i is "P i The total number of pixels in the image (P) is the number of pixels in each region (P i ) In other words, as shown in Equation 4, in this embodiment, the total number of pixels (P) of the image 300 is the sum of the number of pixels (P0) in the high-resolution region 310 and the number of pixels (P1) in the low-resolution region 330. P = P1 + P0 (Equation 4)

[0069] Similarly, the number of bits allocated for coding region i is B i B i is the number of bits per block (b i ) to the total number of pixels in that area (P i ) divided by the number of pixels per block (16 per block for an 8x2 pixel block in this example). This can be expressed as Equation 5: B i =b i *(P i / 16) (Equation 5)

[0070] Therefore, in this example, the total number of bits (B) allocated to encoding image 300 is the sum of the total number of bits (B1) for low-resolution region 330 and the total number of bits (B0) for high-resolution region 310. In a manner similar to Equation 4, the total number of bits (B) for the entire image 300 can be expressed as Equation 6: B = B1 + B0 (Equation 6)

[0071] This embodiment presents a novel solution by partitioning the bits per block for the high resolution region 310 into "base bits" per block and "enhancement bits" per block. Because no enhancement bits are allocated to blocks in the low resolution region 330, the base bits for each block in the high resolution region 310 will be the same as the bits per block available to the low resolution region 330 (e.g., equal to b1). That is, the total number of bits (B) will be partitioned into base bits for the entire image (e.g., all of regions 310, 330) and enhancement bits for only the enhancement region (e.g., high resolution region 310 of image 300).

[0072] Therefore, the total number of bits (B0) allocated to the high resolution region 310 can be expressed by Equation 7.

number

[0073] where JPEG0007793017000013.jpg815 item is the basic bit, which corresponds to the bits per block for both areas 310 and 330. Item JPEG0007793017000014.jpg623 is the enhancement bit and is allocated to the high resolution region 310. The number of available enhancement bits can be used to calculate the "enhancement bits per block" for the blocks in the high resolution region 310. Because the two regions 310, 330 use the same base number of bits per block, the number of enhancement bits per block corresponds to the difference between the target number of bits per block for the high resolution region 310 and the target number of bits per block for the low resolution region 330 [i.e., (b0-b1)]. According to one embodiment, the enhancement bits per block is a fixed number.

[0074] For example, if the size of the image 300 is 160×160 pixels (i.e., P=160 2), and the size of the high-resolution area 310 is 32×32 pixels (i.e., P0=32 2 ), then according to the above formula, the bits per block for the high resolution region 310 is 85 [i.e., b0 = 85, bits per pixel ("bpp0") = 5.3125, corresponding to a 1.5:1 compression ratio, assuming each pixel in the image 300 is represented using 8 bits]. Also, in this example, the bits per block for the low resolution region 330 is 16 [i.e., b1 = 16, and bpp1 = 1.0, corresponding to an 8:1 compression ratio, assuming each pixel in the image 300 is represented using 8 bits].

[0075] Therefore, by using Equation 7 above, the total number of bits (B0) for the high-resolution region 310 is 5,440 bits, which is the sum of the 1,024 basic bits and 4,416 enhancement bits for the high-resolution region 310. Also, in this example, according to Equation 6 above, the total number of bits available for the entire image 300 is 30,016 bits, of which 25,600 basic bits are evenly distributed to the entire image 300 including the high-resolution region 310 and the low-resolution region 330, and the 4,416 enhancement bits are used only for the high-resolution region 310.

[0076] Since enhancement bits can become scarce, the present embodiment tracks the number of available enhancement bits remaining during the encoding process. Therefore, bitsAllocated per block for the high resolution region 310 and the low resolution region 330 can be determined as follows:

[0077] The basic number of allocated bits per block can be determined for the entire image 300 including the high resolution region 310 and the low resolution region 330. The basic number of allocated bits per block can be the same as the number of allocated bits per block for the low resolution region (e.g., (bitsAllocated)1 described below). The value obtained by subtracting JPEG0007793017000015.jpg1350 is distributed to the entire image 300 based on the complexity of the entire image 300 (for example, in the same manner as in the case of VDC-M single resolution).

[0078] For the high resolution region 310, the basic bits per block plus the enhancement bits per block can be used to determine the total number of allocated bits per block for the high resolution region 310 (i.e., the basic bits per block plus the enhancement bits per block). Thus, additional enhancement bits (e.g., the bits remaining after the basic bits have been allocated) may be allocated to the high resolution region 310, which can generally improve image quality.

[0079] The multi-resolution chiaroscuro method according to the present embodiment is [ JPEG0007793017000016.jpg1133] This can be distinguished from VDC-M single resolution in that it continues to track the remaining pixel blocks of the high resolution region 310 that remain in the image 300 being coded by its dependency on JPEG0007793017000017.jpg826=0 becomes.

[0080] According to this embodiment, the encoder can encode the bits as follows:

[0081] The bitsAllocated for each block is the remaining bits of the image 300 (i.e., B r ), the remaining pixels of the image 300 (i.e., JPEG0007793017000018.jpg56) and a high resolution region 310 within the image 300 It may be continually updated based on the remaining pixels of JPEG0007793017000019.jpg67. Thus, if each pixel block is 8x2 (i.e., 16 pixels per block), the number of bitsAllocated per block for the low-resolution region 330 can be written as "(bitsAllocated)1" and expressed as Equation 8.

number

[0082] For blocks in the high resolution region 310, the allocated bits per block ["(bitsAllocated)0"] is determined using Equation 9.

number

[0083] Here, (b0-b1) may indicate the enhancement bits per block that can be allocated to the high resolution region 310 relative to the allocated bits per block of the low resolution region 330 (e.g., since the low resolution region 330 is coded with only basic bits, the allocated basic bits per block for the entire image 300 may be (bitsAllocated) 1).

[0084] As in VDC-M rate control, the QP value of the current block to be coded received by the encoder is determined. JPEG0007793017000024.jpg31(bitsAllocated)1 and can be determined based on the complexity of the current block relative to neighboring blocks.

[0085] FIG. 4, which summarizes the above-described contents, is a flowchart illustrating a method of allocating various numbers of bits to blocks in various resolution regions during the process of compressing a display stream according to an embodiment of the present invention.

[0086] Referring to FIG. 4, in step S400, the encoder according to the present embodiment can determine the number of basic bits of the entire image. JPEG0007793017000025.jpg1118 can be determined].

[0087] In step S405, the encoder can determine the number of enhancement bits only for the high resolution region. JPEG0007793017000026.jpg1337 can be determined].

[0088] In step S410, the encoder can determine the number of basic bits allocated per block for both the low-resolution region and the high-resolution region. JPEG0007793017000027.jpg1685 can be determined, which may be the same as the base number of bits allocated for all remaining blocks.

[0089] In step S415, the encoder may determine the total number of allocated bits per block for the high resolution region by adding the allocated number of base bits per block to the allocated number of enhancement bits per block for the high resolution region. JPEG0007793017000028.jpg9117 can be determined).

[0090] In step S420, the encoder may determine the quantization parameters of the current block based on the quantization parameters of one or more previously encoded neighboring blocks. JPEG0007793017000029.jpg1090 can be determined.

[0091] In step S430, the encoder may determine the number of bits for the current block based on the quantization parameter of the current block. Then, in step S440, the encoder may remove bits from a pool of remaining bits that are not used by encoding the current block with the QP, and transmit the current block to the decoder.

[0092] In step S450, the encoder can update the number of remaining bits, the number of remaining pixels of the image, and the number of remaining pixels of the high resolution region.

[0093] In step S460, the encoder can receive the next pixel block to be compressed and can return to step S410 until it has coded the last block of image 300. If there are no more blocks remaining, the encoder can end image compression in step S470.

[0094] FIG. 5 shows an image of a display device having four resolution regions with different resolutions according to one embodiment of the present invention.

[0095] 5, an embodiment of the present invention may be modified to be applicable to a display device having R regions (R is an integer, in this example 4), as previously described. In this embodiment, image 500 has a high resolution region 510, a low resolution region 530, and two intermediate resolution regions 520, 525.

[0096] According to this embodiment, the above-described Equation 3 is expressed as the following Equation 10, which is the number of bits per block for a given area "area i". JPEG0007793017000030.jpg66".

number

[0097] Equation 4 described above can be changed to express the total number of pixels (P) in the image 500 as in Equation 11 below.

number

[0098] Similarly, Equation 6 described above can be changed to express the total number of bits (B) allocated to the image 500 as in Equation 12 below.

number

[0099] Therefore, the above-described Equation 7 can be expressed as the total bits of region i, "B i " can be transformed into something that expresses "

number

[0100] Here, the basic bits allocated to all the regions 510, 520, 525, and 530 of the image 500 are JPEG0007793017000035.jpg1125, and enhancement bits allocated to the remaining areas 510, 520, and 525 excluding the low-resolution area 530 are It may also be expressed as JPEG0007793017000036.jpg1247.

[0101] Using such a formula, the above embodiment can be extended to more regions by separating each region i (i≠R-1) into base bits and enhancement bits.

[0102] For each block, the remaining bits ( JPEG0007793017000037.jpg45), the remaining pixels in image 500 ( JPEG0007793017000038.jpg45), and the remaining pixels located in regions 0, 1, ...R-2, respectively. We can update bitsAllocated (size 8x2) based on JPEG0007793017000039.jpg1040.

[0103] Therefore, the above-described Equation 8 can be changed to the following Equation 14, which expresses the number of basic bits allocated per block.

number

[0104] For blocks in region R-1 (i.e., low-resolution region 530), the allocated bits per block are (bitsAllocated) R-1 That is, the basic bits per block allocated for all regions may be the same as the bits per block allocated for all low resolution regions 530.

[0105] The enhancement bits per block for regions 510, 520, 525 other than the low resolution region 530 [eg, region i (i≠R-1)] can be determined by changing Equation 9 to the following Equation 15:

number

[0106] Here, the enhancement bits per block corresponding to the blocks in each enhancement region i (i ≠ R-1) are (b i -b R-1 ) may also be expressed as

[0107] The QP value for the current block that the encoder encodes may be determined based on the determined bitsAllocated and the complexity of the current block relative to neighboring blocks (e.g., as done in VDC-M rate control). Much like the example described above with reference to FIG. 4, the encoder can then assign a QP to the current block based on the determined complexity, encode and transmit the current block, and update the number of remaining base bits and the number of remaining enhancement bits. The encoder then moves on to the next block.

[0108] Figure 6, which summarizes the above-described content, is a flowchart showing a method of allocating various numbers of bits to multiple blocks (i.e., three or more regions) of various resolution regions during the display stream compression process according to one embodiment of the present invention.

[0109] Referring to FIG. 6, in step S600, the encoder according to this embodiment can determine the number of basic bits of the entire image. JPEG0007793017000042.jpg1229 can be determined].

[0110] In step S605, the encoder can determine the number of enhancement bits for the region other than the low-resolution region. JPEG0007793017000043.jpg1364 can be determined].

[0111] In step S610, the encoder can determine the number of basic bits allocated per block for both the low-resolution region and the high-resolution region. JPEG0007793017000044.jpg1395 can be determined].

[0112] In step S615, the encoder may determine a total number of allocated bits per block for each region other than the low-resolution region, which may be equal to the average number of allocated enhancement bits per block for the enhancement region. JPEG0007793017000045.jpg12128 can be determined].

[0113] In step S620, the encoder may determine the quantization parameters of the current block based on previously determined quantization parameters of one or more blocks adjacent to the current block that have been previously encoded. JPEG0007793017000046.jpg1186 can be determined.

[0114] In step S630, the encoder may determine the number of bits for encoding the current block based on the determined quantization parameter and based on the resolution domain the current block is located in. Then, in step S640, the encoder may encode the current block with the determined QP and transmit the encoded block to the decoder.

[0115] In step S650, the encoder can update the number of remaining bits, the number of remaining pixels of the image, and the number of remaining pixels of each region other than the low-resolution region.

[0116] In step S660, the encoder can receive the next pixel block to be compressed and can return to step S610 until it has coded the last block of image 500. If there are no more blocks remaining, the encoder can end image compression in step S670.

[0117] According to the above example, the fixation point (e.g., the point on the image corresponding to the user's focus) can be considered to be at the center of the image. However, due to eye movement, the fixation point may not correspond to the center of the image. Therefore, embodiments of the present invention can be equally applied even when the fixation point is not at the center of the image. In the latter case, the high-resolution, medium-resolution, and low-resolution regions may be defined differently depending on the selected fixation point, and the presented technology can be equally applied.

[0118] Furthermore, embodiments of the present invention are applicable to all fixed rate codecs that use a block-based solution of size NxM. For example, embodiments of the present invention are applicable to codecs with or without a rate buffer. If the codec includes a rate buffer, the determined bitsAllocated per block may be further adjusted depending on how full or empty the encoder buffer is (e.g., depending on the buffer status or buffer fullness). That is, before determining the QP value described above, the bits per block, 'bitsAllocated', may be adjusted depending on the 'buffer fullness' value, as shown in Equation 16 below.

number

[0119] Here, buffer fullness (BF) is an abstract value indicating the number of bits in the encoder buffer, and δ(BF) is an offset value that is a function (e.g., a monotonically decreasing function) of BF. According to one embodiment, δ(BF) may be 0 for all regions except for the high-resolution region or the enhancement region. According to another embodiment, δ(BF) may be 0 only for the low-resolution region.

[0120] Therefore, based on the limitations of a codec designed according to the embodiments described herein, the encoder can subsequently determine whether the buffer is relatively full. If the buffer is relatively full (e.g., close to overflow error), the encoder can increase the QP and decrease the bitrate. Similarly, if the buffer is relatively empty, the encoder can decrease the QP and increase the bitrate.

[0121] Thus, the encoder can use the available bits to encode the image by judiciously allocating different numbers of bits per block to different regions to achieve different regions with different levels of resolution, and can encode the image without using up all the available bits before encoding the last block of the image. Thus, by using the embodiments described above, display stream technology can be improved.

[0122] The concept of the present invention and the gist of the method for achieving the same can be easily understood through the detailed description of the invention and the accompanying drawings. The present invention can be realized in various different forms and is not limited to the embodiments described herein. By providing these embodiments, the detailed description of the invention will be thorough and complete, and various aspects and features of the invention will be fully explained to those skilled in the art. Therefore, descriptions of processes, devices, techniques, etc. that are not necessary for those skilled in the art to fully understand various aspects and features of the present invention may be omitted. Unless otherwise specified, the same reference numerals denote the same components throughout the drawings and specification, and descriptions thereof will be omitted. Furthermore, for clarity of explanation, parts unrelated to the description of the embodiments may not be shown. In the drawings, parts, layers, regions, etc. may be exaggerated for clarity.

[0123] Various embodiments will be described herein with reference to cross-sectional views showing schematic structures and / or intermediate structures of the embodiments. The illustrated patterns may be variously modified or changed due to, for example, manufacturing techniques and / or tolerances. Furthermore, the description of specific structures or functions described herein is merely an example for explaining embodiments according to the concepts of the present invention. Therefore, the embodiments described herein are not limited to the specific patterns of the illustrated regions, but may also include variations in patterns due to, for example, manufacturing methods. Therefore, the patterns of regions shown in the drawings are schematic in nature and are not intended to represent or limit the actual patterns of regions on a device. Furthermore, it will be appreciated that those skilled in the art will be able to modify or change the above-described embodiments in various different ways without departing from the spirit or scope of the present invention.

[0124] In the detailed description, various conditions are specified to provide a thorough description of the various embodiments. However, it will be apparent that the embodiments may be practiced without these specific conditions or their equivalents. Instead, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the various embodiments.

[0125] The terms used in the specification are used only to describe particular embodiments and are not intended to limit the present invention. Furthermore, in the present embodiments, unless a number is specifically stated, the singular and plural cases are all included. The expression "comprising" a feature, step, operation, part, component, etc. means that other features, steps, operations, parts, components, etc. may be included in addition to the part in question. The expression "and / or" includes all combinations of one or more of the listed items.

[0126] When implementing a particular embodiment differently, the order of the particular processes may be varied from that described. For example, two processes described as being performed sequentially may be performed simultaneously or in the reverse order from that described.

[0127] The electronic, electrical, and / or other related devices or components described in accordance with embodiments of the present invention may be implemented using appropriate hardware, firmware (e.g., application-specific integrated circuits), software, or a combination thereof. For example, various components of these devices may be implemented on a single integrated circuit chip, or on separate integrated circuit chips. Additionally, various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board, or the like, or may be formed on a single substrate. Additionally, various components of these devices may be processes or threads that may be executed by one or more processors within one or more computing devices that execute computer program instructions and interact with other system elements to perform the various functions described herein. The computer program instructions may be stored in memory implemented in the computing device using standard memory devices such as random access memory (RAM). Furthermore, those skilled in the art may combine or integrate the functionality of various computing devices into a single computing device, or distribute the functionality of a particular computing device across one or more other computing devices, without departing from the spirit and scope of embodiments of the present invention.

[0128] Unless otherwise specified, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with the meaning in the relevant art and / or this specification, and should not be interpreted in an ideal or overly strict sense unless expressly stated herein.

[0129] While specific terms have been used to describe the embodiments, these terms should be construed in a general sense for descriptive purposes and not for limiting purposes. For example, unless otherwise specified, it would be clear to one skilled in the art at the time of filing that features, characteristics, and / or components described in connection with a particular embodiment can be used alone or in combination with features, characteristics, and / or components described in connection with other embodiments. Therefore, it will be apparent to one skilled in the art that various changes in form and detail can be made without departing from the spirit and scope of the present invention, as defined by the claims and their inherent functional equivalents. [Explanation of symbols]

[0130] 100, 300, 500: Images 110, 310, 510: High resolution area 120, 520, 525: Medium resolution range 130, 330, 530: High resolution area

Claims

1. 1. A method of encoding image data using an encoder, comprising: defining, by the encoder, a first region of an image, a second region of the image, and a third region of the image; allocating, by the encoder, a first number of bits consisting of base bits to encode the first region, a second number of bits consisting of base bits and enhancement bits to encode the second region, and a third number of bits consisting of base bits and enhancement bits to encode the third region; obtaining, by said encoder, a number of pixels per block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter for the current block based on a previous quantization parameter of a previous block in the same image frame in the region of the current block; encoding the current block with the quantization parameter by the encoder; updating, by the encoder, the number of pixels remaining in the image to be encoded; updating, by the encoder, a number of remaining bits for encoding pixels corresponding to the number of remaining pixels; determining, by the encoder, a basic number of allocated bits per block for a remaining number of blocks based on a remaining number of pixels of the image to be encoded, a remaining number of bits available for encoding pixels corresponding to the remaining number of pixels of the image, and a remaining number of pixels in the second region to be encoded; the number of allocated basic bits per block is the same for all remaining blocks, the second region is encoded with a larger number of bits per pixel than the first region; the third region is coded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region; the third region surrounds the second region and is surrounded by the first region; Encoding method.

2. the second region includes a high-resolution region corresponding to a gaze fixation point indicating a point on the image corresponding to a focus of a user viewing the image on the display device; the first region includes a low-resolution region outside the gaze fixation point; The encoding method of claim 1 .

3. determining that the current block belongs to the second region; calculating, by the encoder, a number of remaining enhancement bits available for encoding the current block based on the number of remaining pixels in the second region and the number of enhancement bits per block; the enhanced number of bits per block represents a difference between a target number of bits per block in the second region and a target number of bits per block in the first region; The encoding method of claim 1 .

4. calculating, by the encoder, a difference between the number of remaining bits available for encoding the remaining pixels of the image and the number of remaining enhancement bits; the number of basic bits allocated per block in the first region and the number of basic bits allocated per block in the second region correspond to the number of basic bits allocated per block based on the difference between the numbers of basic bits allocated per block in the first region and the number of basic bits allocated per block in the second region. The encoding method according to claim 3.

5. and calculating, by the encoder, when the current block of the image is in the second region, a total number of allocated bits per block for encoding the current block based on a sum of the number of basic bits allocated per block of the first region and the number of enhancement bits per block of the second region. The encoding method according to claim 4.

6. the quantization parameter of the current block is determined based on a quantization parameter of a block encoded immediately before the current block, a number of bits used to encode the block immediately before the current block, and a number of bits allocated to the current block; if the current block of the image is within the first region, the number of bits allocated to the current block is the basic bits allocated per block of the first region; If the current block of the image is within the second region, the number of bits allocated to the current block is the total number of bits allocated per block of the second region. The encoding method according to claim 5.

7. An image data encoder for encoding image data, comprising: a memory for buffering image data; a processor configured to control the encoder; The encoder comprises: defining a first region of an image, a second region of the image, and a third region of the image; allocating a first number of bits consisting of base bits to encode the first region, a second number of bits consisting of base bits and enhancement bits to encode the second region, and a third number of bits consisting of base bits and enhancement bits to encode the third region; Get the number of pixels per block, receiving a current block of the image; determining a quantization parameter for the current block based on a previous quantization parameter for a previous block in the region of the current block within the same image frame; encoding the current block with the quantization parameter; updating the number of pixels remaining in the image to be encoded; updating the number of remaining bits for encoding pixels corresponding to the number of remaining pixels; determining an allocated basic number of bits per block for a remaining number of blocks based on a remaining number of pixels of the image to be coded, a remaining number of bits available for coding pixels corresponding to the remaining number of pixels of the image, and a remaining number of pixels in the second region to be coded; the number of allocated basic bits per block is the same for all remaining blocks, the second region is encoded with a greater number of bits per pixel than the first region; the third region is encoded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region; the third region surrounds the second region and is surrounded by the first region; Image data encoder.

8. the second region includes a high-resolution region corresponding to a gaze fixation point indicating a point on the image corresponding to a focus of a user viewing the image on the display device; the first region includes a low-resolution region outside the gaze fixation point; 8. The image data encoder of claim 7.

9. The image data encoder further comprises: determining that the current block belongs to the second region; calculating a number of remaining enhancement bits available for encoding the current block based on the number of remaining pixels in the second region and the number of enhancement bits per block; the number of enhanced bits per block represents a difference between a target bit per block of the second region and a target bit per block of the first region; 8. The image data encoder of claim 7.

10. the image data encoder is further configured to calculate a difference between the remaining number of bits available for encoding remaining pixels of the image and the remaining number of enhancement bits, and the allocated basic number of bits per block of the first region and the allocated basic number of bits per block of the second region correspond to an allocated basic number of bits per block based on the difference.

10. The image data encoder of claim 9.

11. the image data encoder is further configured to, when the current block of the image is in the second region, calculate a total number of allocated bits per block for encoding the current block based on a sum of a number of allocated basic bits per block and a number of enhanced bits per block of the first region and the second region.

11. The image data encoder of claim 10.

12. the quantization parameter of the current block is determined based on a quantization parameter of a block coded immediately before the current block, a number of bits used to code the block immediately before the current block, and a number of bits allocated to the current block; if the current block of the image is within the first region, the number of bits allocated to the current block is the basic bits allocated per block of the first region; If the current block of the image is within the second region, the number of bits allocated to the current block is the total number of bits allocated per block of the second region.

12. The image data encoder of claim 11.

13. 1. A non-transitory computer-readable medium implemented in an image data encoder for encoding image data, comprising: the image data encoder comprises a memory for buffering the image data and a processor for controlling the image data encoder; When executed, defining a first region of an image, a second region of the image, and a third region of the image; allocating a first number of bits consisting of base bits for encoding the first region, a second number of bits consisting of base bits and enhancement bits for encoding the second region, and a third number of bits consisting of base bits and enhancement bits for encoding the third region; Get the number of pixels per block, receiving a current block of the image; determining a quantization parameter for the current block based on a previous quantization parameter for a previous block in the region of the current block within the same image frame; encoding the current block with the quantization parameter; updating the number of pixels remaining in the image to be encoded; updating the number of remaining bits for encoding the remaining pixels; instructions are stored that cause the image data encoder to perform operations to determine an allocated base number of bits per block for a remaining number of blocks based on a remaining number of pixels of the image to be encoded, a remaining number of bits available for encoding the remaining pixels of the image, and a remaining number of pixels in the second region to be encoded; the number of allocated basic bits per block is the same for all remaining blocks, the second region is encoded with a greater number of bits per pixel than the first region; the third region is coded with a larger number of bits per pixel than the first region and a smaller number of bits per pixel than the second region; the third region surrounds the second region and is surrounded by the first region; Computer-readable medium.

14. and further comprising adjusting the quantization parameter or the bit rate corresponding to the image based on a level of a buffer for buffering the image after the encoding of the image by the encoder has started. The encoding method of claim 1 .

Images