Method and Encoder for Encoding Image Data and Computer-Readable Medium

Abstract

The present invention discloses a method and an encoder for encoding image data and a computer-readable medium. There is provided a method of encoding video data blocks representing an image using an encoder, the method comprising the encoder identifying a first region of the image and a second region of the image, the sum of a first number of pixels in the first region and a second number of pixels in the second region being equal to the total number of pixels in the image; and the encoder allocating a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, the sum of the first number of bits and the second number of bits being equal to the total number of bits for encoding all of the pixels, wherein the second region is encoded using a greater number of bits per pixel than the first region.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims the priority and benefit of U.S. Provisional Patent Application No. 62 / 713,464, filed on August 1, 2018, entitled "Rate Control for Fixed Rate Foveated Display Compression", U.S. Provisional Patent Application No. 62 / 784,346, filed on December 21, 2018, entitled "Rate Control for Fixed Rate Foveated Display Compression", and U.S. Provisional Patent Application No. 16 / 290,598, filed on March 1, 2019. Technical Field

[0003] One or more aspects of embodiments of the present disclosure generally relate to display compression, and may also relate to multi - resolution virtual reality (VR) compression and VDC - M rate control. Background Art

[0004] As display technologies improve, the resolution of the data used to generate images on a display has increased significantly. Due to the continuously increasing pixel bandwidth associated with display technologies (especially for mobile displays and virtual reality (VR) displays), the Video Electronics Standards Association (VESA) has issued a Call for Technologies (CfT) for display stream codecs that support higher compression ratios, which has increased the computational complexity regarding display stream compression (DSC).

[0005] Display compression enables the transmission of data with a larger resolution over an existing display link bandwidth (e.g., DSC and VDC - M standardized by VESA). VESA has developed a new display interface compression standard for mobile / smartphone displays, called the VESA Mobile Device Display Codec (VDC - M). As a new display compression standard, VDC - M aims for a higher compression ratio than the display stream compression (DSC) standard while still maintaining visually lossless quality, but at the cost of increased complexity.

[0006] Compared with image / video compression standards (e.g., JPEG 2000 and HEVC standards), display compression encoders (e.g., DSC, VDC-M) are generally lightweight and can provide visually lossless performance. DSC and VDC-M are fixed bitrate encoders and can support compression ratios as low as 8 and 6 bits / pixel respectively for 24-bit color sources. For example, VDC-M is a fixed bitrate display stream codec and thus has a fixed bitrate, and can support a compression ratio as low as 6 bits / pixel for an RGB 4:4:4 source. VDC-M is a block-based codec with a block size of 8×2 pixels.

[0007] The field of view (FOV) of a VR headset can be, for example, approximately 150 degrees and 135 degrees in the horizontal and vertical directions respectively. However, due to the inherent characteristics of the human eye, human perception is not uniform across the entire 150×135-degree FOV.

[0008] For example, the fovea in the human retina with the highest visual acuity is called the foveal pit. The center of the human eye's FOV is focused in this area because this is where the retinal cones are particularly concentrated. Therefore, the area where human perception is best (e.g., with the highest resolution) corresponds to the area covered by the foveal pit. The FOV corresponding to the foveal pit can be about + / - 3 degrees.

[0009] It should be noted that the information disclosed in this background art section is only for enhancing the understanding of the embodiments of the present disclosure and may include technical information obtained during the process of implementing the inventive concept. Therefore, it may contain information that does not form part of the prior art. Summary of the Invention

[0010] The embodiments described herein generally provide improvements to display technology, including improvements to display stream compression, which in turn can improve virtual reality (VR) technology.

[0011] According to an embodiment of the present disclosure, a method of encoding video data blocks representing an image using an encoder is provided. The method includes the encoder identifying a first region of the image and a second region of the image, the sum of a first number of pixels in the first region and a second number of pixels in the second region being equal to the total number of pixels in the image; and the encoder allocating a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, the sum of the first number of bits and the second number of bits being equal to the total number of bits for encoding all of the pixels, wherein the second region is encoded using a greater number of bits per pixel compared to the first region.

[0012] The second region may include a high-resolution region corresponding to the fixation point of a user viewing an image on the display, wherein the first region includes a low-resolution region located outside the fixation point.

[0013] The method may further include identifying, by an encoder, a third region of the image, wherein the third region is encoded using a greater number of bits per pixel as compared to the first region and using a lesser number of bits per pixel as compared to the second region.

[0014] The method may further include obtaining, by the encoder, the number of pixels of each block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter of the current block; encoding, by the encoder, the current block using the quantization parameter; updating, by the encoder, the number of remaining pixels to be encoded in the image; and updating, by the encoder, the number of remaining bits for encoding the remaining pixels.

[0015] The method may further include determining, by the encoder, a base number of bits allocated to each block based on the number of remaining pixels to be encoded in the image, the number of remaining bits available for encoding the remaining pixels in the image, and the number of remaining pixels to be encoded in the second region.

[0016] If the current block belongs to the second region, the method may further include calculating, by the encoder, a remaining number of enhanced bits available for encoding the current block based on the number of remaining pixels in the second region and the number of enhanced bits per block, the number of enhanced bits per block representing the difference between the target number of bits per block in the second region and the target number of bits per block in the first region.

[0017] The method may further include calculating, by the encoder, the difference between the number of remaining bits available for encoding the remaining pixels of the image and the remaining number of enhanced bits; and calculating, by the encoder, the base number of bits allocated to each block in the first region and the second region based on the number of remaining pixels of the image and the difference.

[0018] The method may further include calculating, by the encoder, a total number of bits allocated to each block for encoding the current block based on the sum of the base number of bits allocated to each block in the first region and the second region and the number of enhanced bits per block when the current block of the image is located in the second region.

[0019] The quantization parameter of the current block can be determined based on the quantization parameter of the block encoded immediately before the current block, the number of bits used to encode the block immediately before the current block, and the number of bits allocated for the current block. When the current block of the image is in the first region, the number of bits allocated for the current block is the base number of bits allocated for each block in the first region. When the current block of the image is in the second region, the number of bits allocated for the current block is the total number of bits allocated for each block in the second region.

[0020] According to another embodiment of the present disclosure, there is provided a video data encoder for encoding video data. The encoder includes a memory for buffering video data and a processor configured to control the memory. The processor is configured to: identify a first region of an image and a second region of the image, where the sum of the number of first pixels in the first region and the number of second pixels in the second region is equal to the total number of pixels in the image; and allocate a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, where the sum of the first number of bits and the second number of bits is equal to the total number of bits for encoding all of the pixels, and where the second region is encoded using a greater number of bits per pixel than the first region.

[0021] The second region may include a high-resolution region corresponding to the gaze point of a user viewing the image on a display, and the first region includes a low-resolution region located outside the gaze point.

[0022] The processor may further be configured to identify a third region of the image, where the third region is encoded using a greater number of bits per pixel than the first region and a lesser number of bits per pixel than the second region.

[0023] The processor may further be configured to: obtain the number of pixels in each block; receive the current block of the image; determine the quantization parameter of the current block; encode the current block using the quantization parameter; update the number of remaining pixels to be encoded in the image; and update the number of remaining bits for encoding the remaining pixels.

[0024] The processor may further be configured to determine the base number of bits allocated for each block based on the number of remaining pixels to be encoded in the image, the number of remaining bits available for encoding the remaining pixels in the image, and the number of remaining pixels to be encoded in the second region.

[0025] If the current block belongs to the second region, the processor may be further configured to calculate the remaining enhancement bits available for encoding the current block based on the number of remaining pixels in the second region and the enhancement bits per block, where the enhancement bits per block represents the difference between the target bits per block in the second region and the target bits per block in the first region.

[0026] The processor may be further configured to calculate the difference between the remaining bits available for encoding the remaining pixels of the image and the remaining enhancement bits; and calculate the base bits assigned to each block in the first region and the second region based on the number of remaining pixels of the image and the difference.

[0027] The processor may be further configured to, when the current block of the image is in the second region, calculate the total bits per block assigned for encoding the current block based on the sum of the base bits per block assigned to each block in the first region and the second region and the enhancement bits per block.

[0028] The quantization parameter of the current block may be determined based on the quantization parameter of the block encoded immediately before the current block, the number of bits used for encoding the block immediately before the current block, and the number of bits assigned to the current block, where when the current block of the image is in the first region, the number of bits assigned to the current block is the base bits per block assigned to each block in the first region, and when the current block of the image is in the second region, the number of bits assigned to the current block is the total bits per block assigned to each block in the second region.

[0029] According to another embodiment of the present disclosure, there is provided a non - transitory computer - readable medium implemented on a video data encoder for encoding video data, the encoder including a memory for caching video data and a processor for controlling the memory, where instructions are stored on the medium, and when executed, the instructions cause the processor to: identify a first region of an image and a second region of the image, the sum of the number of first pixels in the first region and the number of second pixels in the second region being equal to the total number of pixels in the image; and allocate a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, the sum of the first number of bits and the second number of bits being equal to the total number of bits for encoding all of the pixels, where the second region is encoded using a greater number of bits per pixel than the first region.

[0030] When executed by the processor, the instructions may further cause the processor to identify a third region of the image, where the third region is encoded using a greater number of bits per pixel than the first region and a lesser number of bits per pixel than the second region.

[0031] Accordingly, the codec or encoder of embodiments of the present disclosure can improve the method of image data compression by dividing different resolution regions, by allocating bits to a block according to the region corresponding to the block of the image, and by updating the count of remaining available bits after encoding the block. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and / or other aspects will become apparent and be more readily understood from the following description of embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 An image including different resolution regions having different respective resolutions according to an embodiment of the present disclosure is shown;

[0034] Figure 2 is a flowchart showing a method for allocating a varying number of bits to each block during compression of a display stream according to an embodiment of the present disclosure;

[0035] Figure 3 An image including different resolution regions and the concept of multi-resolution shading according to an embodiment of the present disclosure is shown;

[0036] Figure 4 is a flowchart showing a method for allocating a varying number of bits to each block in regions having different resolutions during compression of a display stream according to an embodiment of the present disclosure;

[0037] Figure 5 An image of a display having four different resolution regions having different respective resolutions according to an embodiment of the present disclosure is shown; and

[0038] Figure 6 is a flowchart showing a method for allocating a varying number of bits to each block in multiple regions having different resolutions (e.g., 3 or more regions) during compression of a display stream according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] As described above, from the center of the field of view (FOV) to the periphery of the FOV, the perception of the human eye is inconsistent. To reduce bandwidth, it may be beneficial to compress the image data according to how the displayed image data is expected to be perceived by the user. That is, to reduce bandwidth, it may be beneficial to compress the image data corresponding to the periphery of the image more than the image data corresponding to the center of the image.

[0040] In addition, although the display resolution of the display screen continues to increase, the bandwidth of the underlying link that presents data from the application processor to the display has not increased at the same rate.

[0041] Therefore, an improved system for implementing different resolution levels or different levels of resolution in corresponding resolution regions on a display screen (such as the display screen of a virtual reality (VR) headset) may be beneficial.

[0042] Figure 1 An image is shown that includes different resolution regions having different corresponding resolutions, according to an embodiment of the present disclosure.

[0043] Reference Figure 1 , an image 100 of a display (e.g., the display of a VR headset) is shown. The image 100 may include N×M pixels, where N and M are integers. It should be noted that although the term "image" is used herein, this term may equally apply to a "slice of a macroblock of an image". That is, although an image is described as being compressed herein, it should be understood that the disclosed embodiments may also be used to compress a slice of a macroblock of an image.

[0044] This embodiment uses multi - resolution shading of the image 100 to be displayed to explain human perception. That is, the image 100 implements multi - resolution shading to explain the differences in human perception corresponding to the FOV of the human eye. Therefore, since visual acuity is not uniform in the FOV of the human eye, the visual quality is not uniform across the image 100.

[0045] For example, a higher resolution may be used in a high - resolution region 110 in the image 100 that corresponds to a region that is highly sensitive to human perception (e.g., the region in the display corresponding to the fovea of the human eye). As an example, the high - resolution region 110 may be a visually lossless region (e.g., uncompressed image data may be displayed, or image data compressed using a lossless compression ratio may be displayed). Additionally, a lower resolution may be used in a low - resolution region 130 in the image 100 of the display that corresponds to a less sensitive region (e.g., the region in the display corresponding to the region towards the periphery of the FOV of the human eye).

[0046] One or more different medium - resolution regions 120 having a medium resolution in the image 100 may be present between the high - resolution region 110 and the low - resolution region 130 of the image 100 of the display to enable a smoother transition of perception from the center to the edge of the image 100 (e.g., reference Figure 5)。That is, although the display of this embodiment indicates a medium-resolution region 120 with an intermediate / medium-level resolution located between the high-resolution region 110 and the low-resolution region 130, other embodiments are not limited thereto. In other embodiments, multiple layers or regions with different resolution levels (e.g., varying medium-resolution levels) may enable the user of the VR headset to perceive a smoother transition from the low-resolution region 130 to the high-resolution region 110.

[0047] It should be noted that Figure 1 the various high, medium, and low-resolution regions 110, 120, and 130 shown in are merely examples. In other embodiments, analysis and testing may be utilized to determine the number, location, and shape of the regions (e.g., after dewarping through the lens of the VR headset). For example, although the high, medium, and low-resolution regions 110, 120, and 130 are shown as rectangles in the Figure 1 image 100, according to other embodiments, the various high, medium, and low-resolution regions may be circular or oval, or may be discontinuous.

[0048] Thus, each block of the image may be encoded differently according to the resolution level of the region in which it is located, and the display may utilize multi-resolution VR compression based on multi-resolution shading. For example, according to the sensitivity level of the expected human perception of the user of the VR headset including the display, corresponding bits may be allocated to the different high, medium, and low-resolution regions 110, 120, and 130 (e.g., a larger number of bits per pixel or a larger number of bits per block are allocated to the region with a higher resolution). However, before the encoder of the display stream codec begins to compress the image, it generally cannot analyze the entire image due to memory limitations. That is, the encoder typically analyzes the blocks of the image as it receives them without foreknowledge of the characteristics of the different blocks of the image that it has yet to receive.

[0049] For single-resolution bitrate control in VDC-M, assuming that the image 100 has a total of P pixels and a total of B bits that can be allocated to the pixels (P and B are integers), embodiments of the present disclosure may enable determination of how to distribute or allocate the corresponding bits for each block (e.g., a block with a block size of 8×2 pixels) to improve or maximize quality, as described below. That is, because certain blocks of pixels will be provided with a greater number of bits compared to other blocks, and because it is generally not known how to allocate the bits to the individual blocks before the compression of the image 100 begins (e.g., because pixel blocks may be serially compressed as the image data enters a compression device such as a VDC-M encoder), so this embodiment provides a method for determining how to improve bit allocation as the image compression proceeds and according to multi-resolution shading.

[0050] For each pixel block (e.g., each block has 8×2 pixels, for a total of 16 pixels per block), "bitsAllocated" (e.g., the number of bits allocated to the block, or the average number of bits per block) can be based on the remaining or available number of bits "B" of the image (e.g., Figure 1 image 100) and on the remaining number of pixels "P" in image 100 r ". In the example provided below, for the first block in the display stream, B r = B and P r = P. When the block corresponds to 16 pixels, the average number of bits to be allocated per block can be calculated by Equation 1. r

[0051] Equation 1

[0052]

[0053] Thus, Equation 1 equalizes the remaining bits among the remaining pixels (e.g., such that each block is allocated the same number of bits). This may be an acceptable strategy, or even the best strategy, when the complexity of future blocks is unknown (e.g., when the compression ratio of the blocks yet to be compressed is unknown).

[0054] However, because image 100 typically has varying levels of complexity or texture in different regions of image 100, it may be useful to vary the number of bits allocated to different blocks as compression continues. That is, as compression continues, it may be useful to adjust bitsAllocated based on the ongoing average compression ratio.

[0055] Accordingly, based on the determination of bitsAllocated in Equation 1, the complexity of the current pixel block relative to other pixel blocks in its neighborhood (e.g., previous blocks in the region of the current block) can be estimated. The QP value "curBlock" of the current block can be determined or calculated using Equation 2 based on the quantization parameter ("QP") value "prevBlock" of the previous block QP " of the previous block. QP

[0056] Equation 2

[0057] curBlock QP = prevBlock QP + δ QP

[0058] In Equation 2, if the complexity of the current block is greater than that of the previous block in its neighborhood, then δ QPis a positive number, and if the current block is less complex than the previous block in its neighborhood, then δ QP is a negative number, and if the complexity between the current block and the previous block is the same or similar, then δ QP can be zero.

[0059] By continuously measuring the complexity of each block and based on δ QP to adjust curBlock QP , it is possible to process Image 100 without using more bits than the total number of bits "B" allocated to Image 100, and there is no large remaining number of bits "B r " when the processing of Image 100 is completed. That is, Equation 2 together with the rest of the processing of this embodiment helps to ensure that neither too many nor too few bits are used in the processing of Image 100.

[0060] Therefore, based on the determined QP that can be used to calculate "curBlock QP ", the number of bits allocated to the current block can be adjusted up or down from the average number of bits allocated to the remaining blocks (e.g., "bitsAllocated"). As the QP increases, the step size corresponding to the image also increases with the distortion of the image, while the corresponding bit rate decreases.

[0061] Once the number of bits allocated to the current block is determined (e.g., based on curBlock QP ), then the current block is encoded, and the actual number of bits used to encode the current block is subtracted from the remaining number of bits in Image 100, so that the remaining number of bits "B r " can be updated. Thereafter, the above first operation and second operation can be repeated for the next block.

[0062] Summarizing the above, Figure 2 is a flowchart showing a method for allocating a varying number of bits to each block during the compression of a display stream according to an embodiment of the present disclosure.

[0063] Referring to Figure 2 , in S210, the encoder of this embodiment can determine the number of bits allocated to each block (e.g., the average number of bits for each block of the image) based on the remaining number of bits for the image and the remaining number of pixels in the image (e.g., the encoder can determine ).

[0064] However, it should be noted that the determination of bitsAllocated involves floating-point calculations between two large integers. Additionally, because the remaining number of pixels P rAs the pixel blocks are processed and encoded, the pre-computation of bitsAllocated may be impractical. To avoid such floating-point calculations, the calculation of bitsAllocated can be implemented using a fixed-point divider. Additionally, after determining the fixed-point target bitrate approximation, an auxiliary lookup table (LUT) can be used to adjust the target bitrate approximation. Further, bit shifting can be used to avoid using division operations to calculate bitsAllocated.

[0065] For example, for each pixel block, the target bitrate (TR ideal ) of each block is updated based on the number of bits (B) and the number of remaining pixels (P) in the macroblock strip or image. Thus, when each block has 16 pixels, To avoid floating-point calculations, fixed-point estimation is used such that TR0≈TR ideal .

[0066] Therefore, the denominator (P) of the equation can be converted to P = 2 N ·p, where p ∈ [0.5, 1). Then, And the function for can be stored in the LUT.

[0067] Additionally, since the quality of the first line (FLS) of the macroblock strip may be useful (e.g., due to prediction), there may be an offset parameter added to TR0 for the FLS. Thus, TR = TR0 + δ FLS , where δ FLS allocates additional bits to the blocks within the first line of the macroblock strip or image, and where δ FLS = 16·FirstLineBpgOffset = 16·2 = 32.

[0068] In S220, the encoder can determine the quantization parameter of the current block received by the encoder for compression based on the quantization parameters of at least one adjacent block. In one embodiment, at least one adjacent block is the block encoded immediately before the current block. And the quantization parameter of the current block is given by curBlock QP = prevBlock QP + δ QP .

[0069] In S230, the encoder can determine the number of bits for encoding the current block based on the determined quantization parameter of the current block (e.g., the encoder can encode the current block based on curBlock QP ). Then, in S240, the encoder can encode the current block with the determined QP and can send the block to the decoder.

[0070] At S250, the encoder may update the remaining number of bits available for compressing the image and the remaining number of pixels to be compressed in the image (e.g., the encoder may update B r and P r ).

[0071] At S260, the encoder may receive the next pixel block for compression and may return to S210 until the last block of image 100 has been encoded. Once there are no additional blocks remaining, the encoder may end the compression of the image at S270.

[0072] Traditionally, for single-resolution compression, VDC-M rate control allocates bits based on the quantization parameters of the blocks in a given portion of the image. Thus, more bits may be allocated to the blocks in complex portions (e.g., regions or areas in the image with a lower compression ratio, such as regions corresponding to natural images with more texture), and fewer bits may be allocated to flat or smooth portions (e.g., computer-generated regions or natural regions that can be compressed at a higher rate without sacrificing the high quality of image 100).

[0073] In contrast, according to this embodiment, for multi-resolution compression, the encoder may allocate more bits to the blocks in the high-resolution region 110 than to the blocks in the low-resolution region 130. For example, more bits may be allocated to the high-resolution region 110 to maintain a higher quality, while fewer bits may be allocated to the mid-resolution region 120 and the low-resolution region 130, and more bits may be allocated to the mid-resolution region 120 than to the low-resolution region 130.

[0074] Assuming that image 100 has a total of P pixels and a total of B bits available for the encoder to encode image 100, embodiments of the present disclosure may appropriately allocate different amounts of the available bits to the current pixel block based on the quality variation across different regions 110, 120, and 130, and may also do so based on the complexity variation within one or more of the regions 110, 120, and 130.

[0075] Figure 3 Shows the concept of an image and multi-resolution shading including different resolution regions according to an embodiment of the present disclosure.

[0076] Reference Figure 3, According to this embodiment, multi - resolution shading can explain differences in human perception because they correspond to different regions corresponding to the FOV of the eye (for example, the region corresponding to the center of the FOV corresponds to higher visual acuity and can also correspond to a fixation region or a high - resolution region), and multi - resolution shading can selectively budget the bitrate of the encoder based on these differences. For simplicity, the display image 300 of this example only includes two different resolution regions (for example, a high - resolution region 310 and a low - resolution region 330). However, this example can be extended to other embodiments that include additional resolution regions (for example, the medium - resolution region 120 in the image 100 of Figure 1 or the medium - resolution regions 520 and 525 in the image 500 of Figure 5 as described below). The high - resolution regions and medium - resolution regions of the described embodiments can also be referred to as enhanced regions. Figure 1 in the medium - resolution region 120 of the image 100 or the Figure 5 medium - resolution regions 520 and 525 in the image 500 as described below).

[0077] In this example, for ease of describing the equations discussed below, the low - resolution region 330 is referred to as "region 1", and the high - resolution region 310 is referred to as "region 0". During processing, these regions 310, 330 of the image 300 can be defined by a bit - mask, where one region is defined by one value of the bit - mask (for example, value 1), and the other region is defined by another value of the bit - mask (for example, value 0).

[0078] The number of bits for each block assigned to a given region (for example, "region i", where i is an integer corresponding to that region, such that in this example i = 1 or 0) is "b i " bits for each block. In this example, the number of bits per block b0 in the high - resolution region 310 (i.e., region 0) is greater than the number of bits per block b1 in the low - resolution region 330 (i.e., region 1), as shown in Equation 3.

[0079] Equation 3

[0080] b0 > b1

[0081] The number of pixels in region i is "P i ". The total number of pixels P in the image is equal to the sum of the number of pixels P i in each region. That is, the P of the image 300 in this example is equal to the sum of the number of pixels P0 in the high - resolution region 310 and the number of pixels P1 in the low - resolution region 330, as shown in Equation 4.

[0082] Equation 4

[0083] P = P1 + P0

[0084] Similarly, the number of bits assigned to region i for encoding is B i . B iis determined by multiplying the number of bits b of each block in region i i by the total number of pixels P in that region i and dividing by the number of pixels per block (for an 8×2 pixel block in this example, 16 pixels per block). This can be represented by Equation 5.

[0085] Equation 5

[0086] B i = b i * (P i / 16)

[0087] Thus, in this example, the total number of bits B allocated for encoding image 300 is equal to the sum of the total number of bits "B1" allocated to the low-resolution region 330 and the total number of bits "B0" allocated to the high-resolution region 310. In a manner similar to Equation 4, the total number of bits B for the entire image 300 can be represented by Equation 6.

[0088] Equation 6

[0089] B = B1 + B0

[0090] This embodiment provides a novel solution by dividing the bits of each block allocated to the high-resolution region 310 into "base bits" per block and "enhancement bits" per block. The base bits of each block in the high-resolution region 310 will be the same bits available for each block in the low-resolution region 330 (e.g., will be the same as b1), since the blocks in the low-resolution region 330 will not be allocated any enhancement bits. That is, the total number of bits B can be divided into base bits for the entire image (e.g., the two regions 310, 330) and enhancement bits only for the enhancement region (e.g., the high-resolution region 310 of image 300).

[0091] Therefore, the total number of bits B0 allocated to the high-resolution region 310 can be represented by Equation 7.

[0092] Equation 7

[0093]

[0094] where the term can be referred to as the number of base bits, which corresponds to the number of bits per block for the two regions 310, 330, while It can be referred to as the number of enhanced bits allocated for the high-resolution region 310. For each block in the high-resolution region 310, the "enhanced bits per block" can be calculated using the number of available enhanced bits. The number of enhanced bits per block can correspond to the difference between the target bits per block in the high-resolution region 310 and the target bits per block in the low-resolution region 330 (i.e., (b0 - b1)), because each block in the two regions 310 and 330 can use the same number of base bits. In one embodiment, the enhanced bits per block are a fixed number.

[0095] As an example, if the size of the image 300 is 160×160 pixels (i.e., P = 160 2 ), and if the size of the high-resolution region 310 is 32×32 pixels (i.e., P0 = 32 2 ), then according to the above equation, the bits allocated to each block in the high-resolution region 310 are 85 bits (i.e., b0 = 85, and the bits per pixel "bpp0" = 5.3125, which corresponds to a compression ratio of 1.5:1, assuming each pixel in the image 300 uses 8 bits to represent). In addition, in this example, the bits allocated to each block in the low-resolution region 330 are 16 bits (i.e., b1 = 16, and bpp1 = 1.0, which corresponds to a compression ratio of 8:1, assuming each pixel in the image 300 uses 8 bits to represent).

[0096] Therefore, by using Equation 7 above, the total number of bits B0 allocated to the high-resolution region 310 is 5440 bits, which is equal to the sum of 1024 base bits and 4416 enhanced bits in the high-resolution region 310. In addition, in this example, and according to Equation 6 above, the total number of bits available for the entire image 300 is 30016 bits, where 25600 base bits are evenly distributed over the entire image 300 including both the high-resolution region 310 and the low-resolution region 330, while 4416 enhanced bits are only used in the high-resolution region 310.

[0097] Since the enhanced bits can be very precious, this embodiment can track the number of available enhanced bits remaining during the encoding process. Therefore, the bitsAllocated for each block in the high-resolution region 310 and the low-resolution region 330 can be determined as follows.

[0098] The base number of bits allocated per block can be determined for the entire image 300 including both the high-resolution region 310 and the low-resolution region 330. The base number of bits allocated per block can be equal to the number of bits allocated per block of the low-resolution region 330 (e.g., (bitsAllocated)1, which will be further described below). The total number of bits B allocated for the image 300 minus the enhancement bits can be distributed over the entire image 300 based on the overall complexity of the image 300 (e.g., in a manner similar to the VDC-M single-resolution case).

[0099] For the high-resolution region 310, the enhancement bits per block can be added to the base bits per block to determine the total number of bits allocated per block of the high-resolution region 310 (i.e., both the base bits per block and the enhancement bits per block). Thus, additional enhancement bits (e.g., the bits remaining after the base bits are allocated) can be allocated to the high-resolution region 310, thereby generally providing a higher-quality image therein.

[0100] By keeping track of the remaining pixel blocks in the high-resolution region 310 that are retained in the image 300 to be encoded due to the dependence of the enhancement bits the multi-resolution shading of the present embodiment can be distinguished from the VDC-M single resolution. When encoding all the blocks of the high-resolution region 310, the term

[0101] According to the present embodiment, the encoder can encode the image as follows.

[0102] Based on the remaining number of bits in the image 300 (i.e., B in Equation 1 r ), based on the remaining number of pixels in the image 300 (i.e., P in Equation 1 r ), and based on the remaining number of pixels in the high-resolution region 310 of the image 300 the bitsAllocated per block can be continuously updated. Thus, if each pixel block is 8×2 (i.e., 16 pixels per block), the number of bits allocated per block of the low-resolution region 330 is represented by "(bitsAllocated)1", which can be represented by Equation 8.

[0103] Equation 8

[0104]

[0105] A term similar to but not to be confused with the total number of enhancement bits of the high-resolution region 310 Refers to the remaining enhancement bits available for the high - resolution region 310. When b0 = b1, this term will be equal to zero (e.g., this can be reduced to a single - resolution case).

[0106] For a block in the high - resolution region 310, Equation 9 can be used to determine the number of bits “(bitsAllocated)0” allocated to each block.

[0107] Equation 9

[0108] (bitsAllocated)0=(bitsAllocated)1+(b0 - b1)

[0109] Where (b0 - b1) represents the number of enhancement bits per block that can be allocated in the high - resolution region 310 compared to the number of bits allocated to each block in the low - resolution region 330 (e.g., the base number of bits allocated to each block of the entire image 300 can be equal to (bitsAllocated)1, since the low - resolution region 330 is encoded only with base bits).

[0110] The QP value of the current block to be encoded received by the encoder can be determined based on the determined bitsAllocated and the complexity of the current block relative to adjacent blocks, as can be done in VDC - M rate control.

[0111] Summarizing the above, Figure 4 is a flowchart showing a method for allocating a varying number of bits to blocks in regions of different resolutions during compressed display streaming according to an embodiment of the present disclosure.

[0112] Referring to Figure 4 , at S400, the encoder of this embodiment can determine the base number of bits for the entire image (e.g., the encoder can determine ).

[0113] At S405, the encoder can determine the number of enhancement bits only for the high - resolution region (e.g., the encoder can determine ).

[0114] At S410, the encoder can determine the base number of bits allocated to each block for both the low - resolution region and the high - resolution region (e.g., the encoder can determine ), which can be equal to the base number of bits allocated to each block of all remaining blocks.

[0115] At S415, the encoder may determine the total number of bits allocated to each block in the high-resolution region by adding the number of base bits allocated to each block and the number of enhancement bits allocated to each block in the high-resolution region (e.g., the encoder may determine (bitsAllocated)0 = (bitsAllocated)1 + (b0 - b1)).

[0116] At S420, the encoder may determine the quantization parameter of the current block based on the quantization parameters of one or more previously encoded adjacent blocks (e.g., the encoder may determine curBlock QP = prevBlock QP + δ QP ).

[0117] At S430, the encoder may determine the number of bits for the current block based on the quantization parameter of the current block. Then, at S440, the encoder may encode the current block with the determined QP, thereby removing bits from the remaining unused bit pool and may send the current block to the decoder.

[0118] At S450, the encoder may update the remaining number of bits, the remaining number of pixels in the image, and the remaining number of pixels in the high-resolution region.

[0119] At S460, the encoder may receive the next pixel block for compression and may return to S410 until the last block of image 300 has been encoded. Once there are no additional blocks remaining, the encoder may end the compression of the image at S470.

[0120] Figure 5 Shows an image of a display with four different resolution regions having different respective resolutions according to an embodiment of the present disclosure.

[0121] Reference Figure 5 , as described above, embodiments of the present disclosure may be modified to apply to a display having regions ( is an integer, where in this example is 4). The image 500 of this embodiment has a high-resolution region 510, a low-resolution region 530, and two medium-resolution regions 520 and 525.

[0122] According to this embodiment, Equation 3 above may be modified to yield Equation 10 below for representing the number of bits “b i ” allocated to each block of a given region “region i”.

[0123] Equation 10

[0124]

[0125] The above Equation 4 can be modified to produce the following Equation 11 for representing the total number of pixels P in Image 500.

[0126] Equation 11

[0127]

[0128] Similarly, the above Equation 6 can be modified to produce the following Equation 12 for representing the total number of bits B allocated to Image 500.

[0129] Equation 12

[0130]

[0131] Therefore, the above Equation 7 can be modified to produce the following Equation 13 for representing the total number of bits “B i ” allocated to Region i.

[0132] Equation 13

[0133]

[0134] where the base number of bits for all regions in Regions 510, 520, 525, 530 of Image 500 that are transmitted can be represented by and where the enhancement number of bits for each of Regions 510, 520, 525 that are transmitted to regions other than the low-resolution Region 530 can be represented by represented.

[0135] Using these equations, the above embodiments can be extended to more regions by dividing each region i (where ) into base bits and enhancement bits.

[0136] For each block (size 8×2 pixels), bitsAllocated can be updated based on the remaining number of bits B r for encoding Image 500, the remaining number of pixels P in Image 500 r and the remaining number of pixels respectively located in regions in .

[0137] Therefore, the above Equation 8 can be modified to produce the following Equation 14 for representing the base number of bits allocated to each block.

[0138] Equation 14

[0139]

[0140] For region (i.e., the low - resolution region 530), for each block, the number of bits allocated is set to be equal to That is, the base number of bits for each block allocated for all regions can be equal to the number of bits for each block allocated for the low - resolution region 530.

[0141] By modifying Equation 9 to Equation 15 below, the enhanced number of bits for each block in regions 510, 520, 525 other than the low - resolution region 530 can be determined (e.g., for each block in region i, where ).

[0142] Equation 15

[0143]

[0144] where the enhanced number of bits for each block corresponding to each block in the enhanced region i (where ) can be represented by the term shown.

[0145] The QP value of the current block to be encoded by the encoder can be determined based on the determined bitsAllocated and the complexity of the current block relative to adjacent blocks (e.g., as done in the VDC - M rate control). Similar to the example described above regarding Figure 4 the encoder can then assign a QP to the current block based on the determined complexity, encode the current block and send the current block, and can update the remaining base number of bits and the remaining enhanced number of bits. Then the encoder can proceed to the next block.

[0146] Summarizing the above, Figure 6 is a flowchart showing a method for allocating a varying number of bits to blocks in multiple regions (e.g., 3 or more regions) of different resolutions during the compression of a display stream according to an embodiment of the present disclosure.

[0147] Referring to Figure 6 , at S600, the encoder of this embodiment can determine the base number of bits for the entire image (e.g., the encoder can determine

[0148] At S605, the encoder can determine the enhanced number of bits for regions other than the low - resolution region (e.g., the encoder can determine

[0149] At S610, the encoder can determine the base number of bits allocated for each block in both the low - resolution region and the high - resolution region (e.g., the encoder can determine

[0150] At S615, the encoder can determine the total number of bits allocated to each block in each resolution region except for the low-resolution region, which can be equal to the average number of bits that can be allocated to each block in the enhancement region (e.g., the encoder can determine

[0151] At S620, the encoder can determine the quantization parameter of the current block based on the quantization parameters of one or more blocks previously encoded by the encoder and adjacent to the current block (e.g., the encoder can determine curBlock QP = prevBlock QP + δ QP ).

[0152] At S630, the encoder can determine the number of bits used to encode the current block based on the determined quantization parameter and based on the resolution region in which the current block is located. Then, at S640, the encoder can encode the current block with the determined QP and can send the encoded block to the decoder.

[0153] At S650, the encoder can update the remaining number of bits, the remaining number of pixels in the image, and the remaining number of pixels in each region except for the low-resolution region.

[0154] At S660, the encoder can receive the next pixel block for compression and can return to S610 until the last block of image 500 has been encoded. Once there are no additional blocks remaining, the encoder can end the compression of the image at S670.

[0155] According to the above example, it can be assumed that the fixation point (e.g., the point in the image corresponding to the user's focus) is located at the center of the image. However, due to eye movement, the fixation point may not always correspond to the center of the image. Therefore, even when the fixation point is not at the center of the image, this embodiment can be equally applied. In the latter case, the high, medium, and low-resolution regions can be differently identified according to the selected fixation point, and the proposed technique can be equally applied.

[0156] In addition, this embodiment can be applied to any fixed-rate codec using a block-based solution of size NxM. For example, this embodiment can be applied to a codec with or without a rate buffer. If the codec does include a rate buffer, the determined bitsAllocated for each block can be further adjusted according to how full or empty the encoder's buffer can be (e.g., based on the state of the buffer, or based on the buffer fullness). That is, before determining the QP value discussed above, the number of bits "bitsAllocated" for each block can be adjusted according to the "buffer fullness" value, as shown in Equation 16 below.

[0157] Equation 16

[0158] bitsAllocated = bitsAllocated + δ(BF)

[0159] where the buffer fullness (BF) is an abstract value representing the number of bits in the encoder's buffer, and where δ(BF) is an offset value that is a function of BF (e.g., monotonically decreasing). In one example, for all regions except the high-resolution region or except the enhancement region, δ(BF) will be zero. In another example, δ(BF) is zero only for the low-resolution region.

[0160] Thus, based on the limitations of the codec designed according to the embodiments disclosed herein, the encoder can continuously determine how full its buffer is. If the buffer is relatively full (e.g., close to causing an overflow error), the encoder can increase the QP to reduce the bit rate. Similarly, if the buffer is relatively empty, the encoder can reduce the QP to increase the bit rate.

[0161] Thus, the encoder is able to intelligently allocate a different number of bits to each block for different regions to obtain different regions with different corresponding resolution levels, while ensuring that the image is encoded without under-utilizing the bits available for encoding the image and without using all of the available bits before encoding the last block of the image. Thus, the embodiments described herein provide an improvement to display stream technology.

[0162] The features and implementation methods of the inventive concept can be more easily understood by referring to the detailed description of the embodiments and the drawings. Hereinafter, the embodiments will be described in more detail with reference to the drawings. However, the described embodiments may be embodied in various different forms and should not be construed as limited to the embodiments shown herein. Instead, these embodiments are provided as examples so that the present disclosure will be thorough and complete, and will fully convey the aspects and features of the inventive concept to those skilled in the art. Thus, processes, elements, and techniques that are unnecessary for those of ordinary skill in the art to fully understand the aspects and features of the inventive concept may not be described. Unless otherwise noted, the same reference numerals denote the same elements throughout the drawings and the written description, and thus, their description is not repeated. In addition, parts irrelevant to the description of the embodiments may not be shown to make the description clear. In the drawings, the relative sizes of elements, layers, and regions may be enlarged for clarity.

[0163] In this document, various embodiments are described with reference to cross-sectional views, which are schematic views of the embodiments and / or intermediate structures. As such, variations from the illustrated shapes due to, for example, manufacturing techniques and / or tolerances are to be expected. In addition, the specific structural or functional descriptions disclosed herein are merely illustrative for the purpose of describing embodiments in accordance with the concepts of the present disclosure. Accordingly, the embodiments disclosed herein should not be construed as limited to the particular regional shapes shown, but include shape deviations caused by, for example, manufacturing. For example, an implantation region shown as rectangular will generally have rounded or arcuate features and / or a gradient in the implantation concentration at its edges, rather than a binary change from the implantation region to the non-implantation region. Similarly, a buried region formed by implantation can result in some implantation in the region between the buried region and the surface through which the implantation occurs. Accordingly, the regions shown in the figures are actually schematic, their shapes are not intended to show the actual shape of the regions of the device, and are not intended to be limiting. In addition, as will be recognized by those skilled in the art, the described embodiments can be modified in various different ways, all of which do not depart from the spirit or scope of the present disclosure.

[0164] In the description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, it will be apparent that the various embodiments can be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form in order not to unnecessarily obscure the various embodiments.

[0165] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprises", "comprising", and "has", "having" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0166] When a particular embodiment can be implemented in different ways, the specific order of processing can be performed differently from the order described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to the described order.

[0167] An electrical or electronic device and / or any other related device or component according to an embodiment of the present disclosure described herein may be implemented using any suitable hardware, firmware (e.g., an application specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, various components of these devices may be formed on one integrated circuit (IC) chip or on separate IC chips. Additionally, various components of these devices may be implemented on a flexible printed circuit film, tape carrier package (TCP), printed circuit board (PCB), or formed on a substrate. Further, various components of these devices may be processes or threads running on one or more processors, where the processors are located in one or more computing devices for executing computer program instructions and interacting with other system components to perform the various functions described herein. The computer program instructions are stored in a memory that may be implemented in a computing device using a standard storage device, such as random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer-readable media, such as a CD-ROM, flash drive, etc. Additionally, those skilled in the art should recognize that the functions of individual computing devices may be combined or integrated into a single computing device, or the functions of a particular computing device may be distributed among one or more other computing devices without departing from the spirit and scope of the embodiments of the present disclosure.

[0168] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the technical field to which the inventive concept belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their context in the relevant art and / or the present specification, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0169] Embodiments have been disclosed herein, and although specific terms have been used, they have been used and interpreted in a general and descriptive sense only and not for purposes of limitation. In some cases, as will be apparent to those of ordinary skill in the art to which this application pertains, features, characteristics, and / or elements described in connection with a particular embodiment may be used alone or may also be combined with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise indicated, for example. Accordingly, various changes in form and detail may be made without departing from the spirit and scope of the present disclosure as set forth in the appended claims (the functional equivalents of which are included herein).

Claims

1. A method for encoding a video data block representing an image using an encoder, the method comprising: identifying, by the encoder, a first region of the image and a second region of the image, wherein a sum of a first number of pixels in the first region and a second number of pixels in the second region is equal to a total number of pixels in the image; and allocating, by the encoder, a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, wherein a sum of the first number of bits and the second number of bits is equal to a total number of bits for encoding all of the pixels, wherein the second region is encoded using a greater number of bits per pixel as compared to the first region, and wherein the method further comprises, for each block in sequence, obtaining, by the encoder, a number of pixels of each block; receiving, by the encoder, a current block of the image; determining, by the encoder, a quantization parameter of the current block; encoding, by the encoder, the current block using the quantization parameter; updating, by the encoder, a number of remaining pixels to be encoded in the image; and updating, by the encoder, a number of remaining bits for encoding the remaining pixels, wherein the method further comprises: determining, by the encoder, a number of base bits allocated to each block based on the number of remaining pixels to be encoded in the image, the number of remaining bits available for encoding the remaining pixels in the image, and the number of remaining pixels to be encoded in the second region, and wherein for pixels in the first region, the number of bits allocated to each block is determined by: subtracting a remaining number of enhancement bits from the remaining bits; multiplying by the number of pixels of each block; and dividing by the number of remaining pixels in the image, and for pixels in the second region, the number of bits allocated is determined by adding the remaining number of enhancement bits of each block to the number of bits allocated to each block determined for the pixels in the first region.

2. The method according to claim 1, wherein The second region includes a high-resolution region corresponding to a fixation point of a user viewing the image on a display, and wherein the first region includes a low-resolution region located outside the fixation point.

3. The method according to claim 1, further comprising identifying, by the encoder, a third region of the image, Among them, wherein the third region is encoded using a greater number of bits per pixel as compared to the first region and a lesser number of bits per pixel as compared to the second region.

4. The method according to claim 3, wherein, If the current block belongs to the second region, the method further comprises: calculating, by the encoder, a remaining number of enhancement bits available for encoding the current block based on the number of remaining pixels in the second region and the number of enhancement bits of each block, the number of enhancement bits of each block representing a difference between a target number of bits of each block in the second region and a target number of bits of each block in the first region.

5. The method according to claim 4, further comprising: calculating, by the encoder, a difference between the remaining number of bits available for encoding the remaining pixels in the image and the remaining enhancement bits; and calculating, by the encoder, the number of base bits allocated to each block in the first region and the second region based on the number of the remaining pixels in the image and the difference.

6. The method according to claim 5, further comprising: when the current block of the image is in the second region, calculating, by the encoder, the total number of bits allocated to each block for encoding the current block based on the sum of the number of base bits allocated to each block in the first region and the second region and the number of enhancement bits of each block.

7. The method according to claim 6, wherein The quantization parameter of the current block is determined based on the quantization parameter of the block encoded immediately before the current block, the number of bits used for encoding the block immediately before the current block, and the number of bits allocated to the current block, wherein when the current block of the image is in the first region, the number of bits allocated to the current block is the number of base bits allocated to each block in the first region, and when the current block of the image is in the second region, the number of bits allocated to the current block is the total number of bits allocated to each block in the second region.

8. A video data encoder for encoding video data, the video data encoder comprising: a memory for caching the video data; and a processor configured to control the memory, wherein the processor is configured to: identify a first region of an image and a second region of the image, the sum of a first number of pixels in the first region and a second number of pixels in the second region being equal to the total number of pixels in the image; and allocate a first number of bits including base bits for encoding the first region and a second number of bits including base bits and enhancement bits for encoding the second region, the sum of the first number of bits and the second number of bits being equal to the total number of bits for encoding all of the pixels, wherein the second region is encoded using a greater number of bits per pixel than the first region, and wherein the processor is further configured to, for each block in sequence, obtain the number of pixels of each block; receive the current block of the image; determine the quantization parameter of the current block; encode the current block using the quantization parameter; update the number of remaining pixels to be encoded in the image; and update the remaining number of bits available for encoding the remaining pixels, wherein the processor is further configured to: determine, by the encoder, the number of base bits allocated to each block based on the number of remaining pixels to be encoded in the image, the remaining number of bits available for encoding the remaining pixels in the image, and the number of remaining pixels to be encoded in the second region, and For the pixels in the first region, the number of bits allocated to each block is determined as follows: subtracting the remaining enhancement bits from the remaining bits; multiplying by the number of pixels in each block; and dividing by the number of remaining pixels in the image. And for the pixels in the second region, the number of bits allocated is determined by adding the remaining enhancement bits of each block to the number of bits allocated to each block determined for the pixels in the first region.

9. The video data encoder according to claim 8, wherein, The second region includes a high-resolution region that corresponds to the fixation point of a user viewing the image on the display, and wherein the first region includes a low-resolution region located outside the fixation point.

10. The video data encoder according to claim 8, wherein, The processor is further configured to identify a third region of the image, wherein, compared to the first region, the third region is encoded using a greater number of bits per pixel, and compared to the second region, the third region is encoded using a smaller number of bits per pixel.

11. The video data encoder according to claim 10, wherein, If the current block belongs to the second region, the processor is further configured to: Based on the number of remaining pixels in the second region and the enhancement bits of each block, calculate the remaining enhancement bits available for encoding the current block, where the enhancement bits of each block represent the difference between the target number of bits of each block in the second region and the target number of bits of each block in the first region.

12. The video data encoder according to claim 11, wherein, The processor is further configured to: Calculate the difference between the remaining bits available for encoding the remaining pixels in the image and the remaining enhancement bits; And Based on the number of remaining pixels in the image and the difference, calculate the base number of bits allocated to each block in the first region and the second region.

13. The video data encoder according to claim 12, wherein, The processor is further configured to: when the current block of the image is located in the second region, calculate the total number of bits allocated to each block for encoding the current block based on the sum of the base number of bits allocated to each block in the first region and the second region and the enhancement bits of each block.

14. The video data encoder according to claim 13, wherein, The quantization parameter of the current block is determined based on the quantization parameter of the block encoded immediately before the current block, the number of bits used to encode the block immediately before the current block, and the number of bits allocated to the current block, where when the current block of the image is located in the first region, the number of bits allocated to the current block is the base number of bits allocated to each block in the first region, and when the current block of the image is located in the second region, the number of bits allocated to the current block is the total number of bits allocated to each block in the second region.

15. A non-transitory computer-readable medium, implemented on a video data encoder for encoding video data, the video data encoder including a memory for caching the video data and a processor for controlling the memory, wherein instructions are stored on the non-transitory computer-readable medium, and when executed, the instructions cause the processor to implement the method according to any one of claims 1 to 7.

Images