Method and device for intracoding image frames - Patents.com

The image pipeline adjusts pixel values in intra-coded frames using intra-prediction modes to minimize residual differences, particularly in non-interest regions, effectively reducing frame size and resource requirements.

JP7578566B2Active Publication Date: 2024-11-06AXIS
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021139551
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-09-08
Filing Date
2021-08-30
Publication Date
2024-11-06
Estimated Expiration
2041-08-30

AI Technical Summary

Technical Problem

Intra-coded image frames, or I-frames, are larger in size compared to inter-coded frames due to exploiting only spatial redundancy, leading to increased storage and transmission requirements, and existing methods to reduce their size often result in loss of image information or inefficient compression.

Method used

An image pipeline that adjusts pixel values within blocks of an image frame by a predefined amount to reduce residual differences, using multiple intra-prediction modes, particularly in regions of non-interest, to minimize frame size while maintaining image quality.

Benefits of technology

The method reduces the size of intra-coded frames by optimizing pixel values, thereby decreasing storage and transmission demands without compromising image quality, especially in regions of non-interest.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578566000001
    Figure 0007578566000001
  • Figure 0007578566000002
    Figure 0007578566000002
  • Figure 0007578566000003
    Figure 0007578566000003
Patent Text Reader

Abstract

To provide a method for processing an image frame before intra-coding the image frame and an image pipeline.SOLUTION: In an imaging system 100, an image pipeline comprises a processing part, and an encoding part supporting a plurality of intra-prediction modes used for intra-coding image frames. A method comprises predicting pixel values of at least one block of pixel values using an intra-prediction mode supported by the encoding part for the at least one block of the pixel value in the image frame, updating the image frame by adjusting at least one pixel value within the at least one block of pixel values so as to reduce a residual between the at least one pixel value and a corresponding predicted pixel value, and intra-coding the updated image frame using at least one out of the plurality of intra-prediction modes.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of intracoding image frames, in particular to the processing of image frames prior to intracoding of said image frames. [Background technology]

[0002] In digital video systems, such as network camera monitoring systems, video sequences are usually compressed before being stored or before being transmitted over a communication network. The reason for compressing video sequences before being stored is to reduce the storage resources required compared to the storage resources required to store uncompressed video sequences. Correspondingly, video sequences are compressed before being transmitted to reduce the network resources used, e.g., network bandwidth, compared to the network resources used to transmit uncompressed video sequences. If the available network resources are limited, it may even be impossible to transmit the video sequences without compression.

[0003] Video sequences can be compressed, i.e., encoded by an encoder, using a variety of video encoding methods. In many digital video encoding systems, two main modes are used to compress image frames of a video sequence: intra-mode and inter-mode. In this disclosure, intra-mode compression is also referred to as intra-coding, and inter-mode compression is correspondingly referred to as inter-coding.

[0004] In intra-coding, luminance and chrominance channels are encoded by exploiting spatial redundancy of pixels within a given channel of a single frame by prediction, transformation, and entropy coding. The encoded image frame is called an intra-frame, which can also be called an I-frame or keyframe. Within an intra-frame, a block of pixels or a macroblock containing multiple blocks is encoded in intra-mode, meaning that they are encoded as intra-blocks, i.e. individually, without reference to other blocks.

[0005] Inter-coding instead exploits temporal redundancy between separate image frames and relies on motion-compensation prediction techniques that predict a given portion of an image frame from one or more previous image frames by encoding the motion of pixels from one image frame to another for selected blocks of pixels. In particular, motion vectors and the difference in motion between the pixels of the image frame and pixels of a reference frame are encoded. The encoded frames are called inter-frames and can be called P-frames (forward-predicted frames) that can refer to the previous frame in decoding order, or B-frames (bidirectionally predicted frames) that can refer to two or more previously decoded frames, and can have any arbitrary display-order relationship of the frames used for prediction. Within an inter-frame, blocks of pixels can be encoded in inter-mode, meaning that they are encoded with reference to similar blocks in a previously decoded image frame, or in intra-mode, meaning that they are encoded as intra-blocks, i.e., individually, without reference to other blocks.

[0006] When performing intra-coding of a block of pixels, the encoder will use intra-prediction, in which the encoder will predict pixel values ​​within the block of pixels being intra-coded using the values ​​of the pixels above and to the left of the block.

[0007] After predicting pixel values ​​using one of the intra-prediction modes, the predicted pixel values ​​are subtracted from the actual pixel values, and the difference (also known as the residual) is quantized and entropy encoded before being submitted in the bitstream. If the prediction were perfect, there would be no residual. However, in practice, there would be a residual that contributes to the size of the intra-coded frame.

[0008] Many encoders have different modes for performing intra prediction, and can select the intra prediction mode that gives the smallest residual for a given block of pixels. For example, the compression standard Advanced Video Coding (AVC), sometimes referred to as H.264, supports nine different intra prediction modes, the compression standard High Efficiency Video Coding (HEVC) supports 35 different intra prediction modes, and the compression standard AO Media Video 1 (AV1) supports 54 different intra prediction modes.

[0009] A disadvantage of intra-coding image frames compared to inter-coding image frames is that the resulting I-frames have a larger size and cause spikes in the bitstream compared to the inter-coded P-frames. The reason for the larger size of I-frames is that only spatial redundancy is exploited when performing intra-coding compared to inter-coding P-frames where both spatial and temporal redundancy are exploited.

[0010] One approach to reduce the size of intra-coded image frames, i.e., I-frames, is described in US Patent Application Publication No. 2013 / 0251279, in which an adaptive pre-processing scheme is used in image encoding. In the adaptive pre-processing scheme, a block of an original image, which has a determined size, is compared with a block of a reference image (an image for which pre-processing has already been performed), and the block of the reference image is duplicated to the block of the original image when the block has a similarity within a certain threshold. After duplication, the original image frame including the duplicated reference block is encoded. The encoder then evaluates whether the decoded image meets the desired quality requirements. If the quality of the decoded image is too low, the threshold must be controlled upwards so that pre-processing of the upcoming image does not result in a degraded image quality. A drawback with the prior art is that the entire block of the original image is replaced with the entire reference block, resulting in a loss of image information for the part of the image including the replaced block. Eventually, the resulting image may be useless for some purposes, such as surveillance purposes, due to the loss of image information when many blocks of the original image are replaced with reference blocks. The prior art has the drawback of requiring a previously preprocessed image that contains the blocks used for comparison and duplication. Furthermore, the need to adjust the threshold is determined when at least one preprocessed and encoded image of reduced quality is obtained. This results in a delay in adjusting the threshold. Furthermore, if the reduced quality image is not transmitted to the receiver, the original image must be preprocessed again using a new threshold, resulting in a decrease in compression efficiency.

[0011] Thus, there is room for improvement.

[0012] Another approach to reducing the size of intra-coded image frames is disclosed in US Patent Application Publication No. 2020 / 021817, which describes a coding technique using Position Dependent Prediction Combination (PDPC) with wide-angle intra prediction. In particular, a method is described for determining a predictive block for a current block of a current picture using wide-angle intra prediction, and then using PDPC to modify a predictive sample of the predictive block. Furthermore, a method is described for determining a residual value for a residual block based on the modified predictive sample, and a method is described for signaling information indicating the residual value.

[0013] US Patent Application Publication No. 2011 / 002385 discloses an image coding apparatus, which divides an input picture into multiple macroblocks, and for pixel values ​​contained in the macroblock, determines whether the macroblock is a flat macroblock based on a calculated flatness. If the macroblock is determined to be a flat macroblock and belongs to a flat area, the apparatus comprises a unit for selecting one of an inter prediction mode and an intra prediction mode as a prediction mode for encoding the macroblock based on the determination result and based on cost values ​​of the inter prediction mode and the intra prediction mode. The apparatus further comprises a unit for performing predictive coding according to the selected prediction mode.

[0014] The paper "AHG6: A threshold for SAO edge offset" (Sugio T et al. no. JCTVC-J0213 10 July 2012 (2012-07-10), XP03020234791) proposes a threshold for SAO EO to control the classification of strong and weak edges to improve flexibility in terms of subjective and objective quality. Summary of the Invention

[0015] In view of the above, it is therefore an object of the present invention to provide techniques for mitigating the above problems and reducing the size of intra-coded image frames, or I-frames.

[0016] According to a first aspect of the invention, the above object is achieved by a method implemented by an image pipeline for intra-coding an image frame, the image pipeline comprising a processing unit followed by an encoding unit supporting multiple intra-prediction modes used for intra-coding the image frame, the method comprising: receiving an image frame, predicting, by the processing unit, for at least one block of pixel values ​​in the image frame, pixel values ​​of the at least one block of pixel values ​​using an intra-prediction mode supported by the encoding unit, reducing a residual between at least one pixel value in the at least one block of pixel values ​​and a corresponding predicted pixel value by adjusting at least one pixel value in the at least one block of pixel values ​​upwards by a predefined amount within a pixel value range when the corresponding predicted pixel value is greater than the at least one pixel value, or downwards by a predefined amount within a pixel value range when the corresponding predicted pixel value is less than the at least one pixel value, and updating the image frame with the adjusted at least one pixel value in the at least one block of pixel values. The method further includes intra-coding, by an encoding unit, the updated image frame using at least one intra-prediction mode from among a plurality of intra-prediction modes, where the intra-coding includes, for each block of pixel values ​​in the updated image frame, predicting pixel values ​​and calculating a residual using one or more of the supported intra-prediction modes, and for intra-coding of each respective block of pixel values, selecting the intra-prediction mode that provides the smallest residual for the respective block of pixel values.

[0017] Therefore, a processing unit preceding the encoding unit in the image pipeline updates the image frame by adjusting at least one pixel value upwards or downwards by a predefined amount within a pixel value range to reduce a residual between the at least one pixel value and a corresponding predicted pixel value. The predefined amount may be one or two steps upwards or downwards within the pixel value range. For example, in a video stream having 8 bits per pixel (bpp), at least one pixel value is an integer in the range of 0 to 255, and in a video stream having 10 bpp, the pixel value range is 0 to 1023. Therefore, it should be understood that the pixel value range depends on the number of bits per pixel of the video stream and may be another range other than the range shown above as an example. The processing unit predicts corresponding predicted pixel values ​​using one of the intra-prediction modes supported by the encoding unit, and then adjusts at least one pixel value to reduce residual error, so that when the encoding unit performs prediction of pixel values ​​in the intra-coding of the image frame to be updated, it will use the same intra-prediction mode of the intra-prediction modes, and therefore the residual error determined by the encoding unit is likely to be smaller than it would have been in the absence of the adjustment made by the processing unit. Thanks to the smaller residual error, the size of the intra-coded image frame is reduced compared to the scenario in the absence of the pixel value adjustment made by the processing unit. Therefore, this method is useful for reducing the size of I-frame.

[0018] As used herein, the term intra-coding refers to encoding of an image frame using one or more intra-prediction modes supported by an encoding unit. The one or more intra-prediction modes may be one or more from among the nine different intra-prediction modes supported by AVC, one or more from among the 35 different intra-prediction modes supported by compression standard HEVC, and / or one or more from among the 54 different intra-prediction modes supported by compression standard AV1, to name a few examples. However. other compression standards may additionally or alternatively support other intra-prediction modes supported by the encoding unit. Furthermore, the supported intra-prediction modes may include one or more luma (luma) intra-prediction modes and / or one or more chrominance (chroma) intra-prediction modes.

[0019] By the term block of pixel values, a unit of pixel values, e.g. a processing unit of pixel values, used in compression is meant as a whole. For example, the processing unit can be used in image and video compression formats based on a linear block transform, typically the Discrete Cosine Transform (DCT). A block can sometimes be called a prediction block or a transformation block, depending on whether it is used in prediction or compression. A block can be part of a so-called macroblock. In the AVC compression standard, a block of pixel values ​​typically contains 4x4 pixel values ​​or 8x8 pixel values. However, it should be understood that the block size can be something else, such as 16x16 or 32x32, or the block size can be a macroblock base size depending on the compression standard. Therefore, it should be understood that the number of pixel values ​​in a block of pixel values ​​can vary, for example, depending on the compression technique used.

[0020] As mentioned above, by the term residual as used in this disclosure, we mean the difference between the actual pixel value and the predicted pixel value. Sometimes, the residual is expressed as the absolute value of the difference between the actual pixel value and the predicted pixel value.

[0021] In some variations of the method, the method further includes identifying at least one region of non-interest within the image frame, wherein at least one block of pixel values ​​falls within the at least one region of non-interest.

[0022] By the term non-interest region as used in this disclosure, it is meant a region different from the region of interest ROI that includes one or more objects or details of interest. In other words, a non-interest region is a region that does not include objects or details of interest, such as a background region. For example, a non-interest region can be a region that includes a portion of the sky, ocean, forest, grass, or other vegetation, to name a few examples.

[0023] Therefore, the adjustment of at least one pixel value contained in a block in a region of non-interest to reduce a residual between the pixel value and a corresponding predicted pixel value as described above would have the advantage of providing a better match with the intra-prediction mode that the encoder would use in the subsequent encoding of the image frame, without affecting the image quality of the encoded image frame, because the adjustment of the at least one pixel value does not affect the object or detail of interest.

[0024] Identification of the at least one region of non-interest can be achieved in a number of ways, some of which are described below with reference to the first and second groups of embodiments.

[0025] In a first group of embodiments, identifying at least one non-interest region in the image frame includes dividing the image frame into image segments, classifying the image segments, and identifying at least one of the image segments as being at least one non-interest region when the at least one of the image segments belongs to a group of predetermined classes.

[0026] One or more of the above-mentioned dividing, classifying and identifying steps can be performed by an image segmentation procedure, in which a label is assigned to all pixels in an image frame, such that pixels with the same label are considered to share certain characteristics and belong to the same image segment. The label can be associated with or indicate one of a predefined class. Typically, the result of the image segmentation is a set of image segments that collectively cover the entire image frame. An example of an image segmentation procedure is semantic segmentation. Some other examples of image segmentation procedures are motion-based segmentation, and threshold, histogram or clustering method-based segmentation, or other procedures for determining certain areas of an image area as more important than other areas due to the presence of objects, people or movements in those areas, and classifying those areas as ROI and other areas as non-interest areas.

[0027] The set of predefined classes may include classes defining homogeneous image segments, i.e., image segments that are approximately uniform in composition, e.g., color and / or texture, or image segments that have spatial variability below a threshold. For example, the set of predefined classes may include classes such as sky and vegetation, and / or subclasses such as daytime sky, nighttime sky, clouds, and trees, shrubs, and grass, respectively.

[0028] In a first group of embodiments, the processing unit can select an intra-prediction mode based on a group of predetermined classes. Thanks to the segmentation into multiple classes, it is easier to select an intra-prediction mode that is more suitable for a certain class.

[0029] In some variations of the method, the processing unit calculates a predicted pixel value for each pixel value in the at least one block of pixel values ​​using at least two intra prediction modes of the multiple intra prediction modes, and selects an intra prediction mode of the at least two intra prediction modes of the multiple intra prediction modes that provides a minimum sum of absolute values ​​of residuals between the pixel value and the corresponding predicted pixel value. By selecting the intra prediction mode with the minimum sum of absolute values ​​of residuals, the encoding unit will select the same intra prediction mode when performing intra prediction of the image frame, increasing the likelihood that resulting in a lower encoded frame size compared to a scenario in which the encoding unit selects the other of the two intra prediction modes.

[0030] In a second group of embodiments, identifying at least one non-interest region in the image frame includes determining a spatial variability of pixel values ​​contained in at least one region in the image frame, and identifying the at least one non-interest region as a region containing pixel values ​​having a spatial variability less than a region threshold. A spatial variability less than the region threshold indicates that the region is homogeneous, i.e., does not contain an object or detail of interest. It is therefore advantageous to identify the region as a non-interest region.

[0031] At least one block of pixel values ​​included in the non-interest region may have a spatial variability of pixel values ​​greater than a certain block threshold, so when selecting a block in which at least one pixel value is adjusted, it is advantageous to select a block with a spatial variability greater than the block threshold, because in such a block the pixel values ​​can be adjusted more to obtain a reduced encoded frame size without compromising image quality.

[0032] In some variations of the method, adjusting at least one pixel value in the at least one block of pixel values ​​includes adjusting at least one pixel value in the at least one block of pixel values ​​to reduce a spatial variability of the pixel values ​​below a block threshold, such that a residual error for the at least one pixel value is also reduced.

[0033] In some variations of the method, the intra-prediction mode used by the processing unit is selected from a subset of multiple intra-prediction modes supported by the encoding unit.

[0034] The method may further include: calculating, by the processing unit, a residual between the pixel value and the corresponding predicted pixel value; and selecting at least one pixel value to be adjusted to be a pixel value for which the absolute value of the calculated residual is greater than a pixel threshold. Adjusting at least one pixel value in the at least one block of pixel values ​​may include adjusting at least one pixel value in the at least one block of pixel values ​​such that the absolute value of the residual is less than or equal to the pixel threshold. By comparing the pixel value to the pixel threshold, pixel values ​​that, if not adjusted, may result in an increase in the encoded frame size and a spike in the encoded bitstream may be selected for adjustment, thereby resulting in a reduced encoded frame size.

[0035] The predefined amount can be one or two steps up or down in the pixel value range. For example, in a video stream with 8 bits per pixel (bpp), at least one pixel value is an integer in the range of 0 to 255, and in a video stream with 10 bpp, the pixel value range is 0 to 1023. It should be understood that the pixel value range is therefore dependent on the number of bits per pixel of the video stream and may be another range other than the ranges given above as examples.

[0036] In some variations of the method, adjusting at least one pixel value includes adjusting at least one pixel value in at least one block of pixel values ​​to be equal to a neighboring pixel value, such that subsequent encoding of the image frame is simplified since neighboring pixels having the same pixel value can be encoded in the same manner. For example, this may be advantageous for a vertical intra-prediction mode (e.g., AVC mode 0) or a horizontal intra-prediction mode (e.g., AVC mode 1).

[0037] In some variations of the method, adjusting the at least one pixel value includes adjusting one or more but not all of the pixel values ​​in the at least one block of pixel values. Typically, the number of adjusted pixel values ​​is kept as small as possible to obtain a sought-after reduction in encoded frame size, while simultaneously avoiding degrading the image quality of the encoded image frame.

[0038] According to a second aspect of the present invention, the above object is achieved by an image pipeline for intra-coding an image frame, the image pipeline comprising a processing unit followed by an encoding unit configured to support multiple intra-prediction modes used for intra-coding the image frame, the processing unit being configured to receive the image frame and configured to predict, for at least one block of pixel values ​​in the image frame, pixel values ​​of the at least one block of pixel values ​​using an intra-prediction mode supported by the encoding unit, reduce a residual between at least one pixel value in the at least one block of pixel values ​​and a corresponding predicted pixel value by adjusting at least one pixel value in the at least one block of pixel values ​​upwards by a predefined amount within a pixel value range when the corresponding predicted pixel value is greater than the at least one pixel value, or downwards by a predefined amount within a pixel value range when the corresponding predicted pixel value is less than the at least one pixel value, and update the image frame with the adjusted at least one pixel value in the at least one block of pixel values. The encoding unit is configured to intra-code the updated image frame using at least one intra-prediction mode from among the multiple intra-prediction modes by being configured to, for each block of pixel values ​​in the updated image frame, predict pixel values ​​using one or more of the supported intra-prediction modes, calculate a residual, and for intra-coding of each respective block of pixel values, select the intra-prediction mode that gives the smallest residual for the respective block of pixel values.

[0039] According to a third aspect of the present invention, the above object is achieved by a non-transitory computer readable medium having stored thereon computer code instructions adapted to perform the method of the first aspect when executed by a device having processing capability.

[0040] The second and third aspects may generally have the same features and advantages as the first aspect. It is further noted that the present invention relates to all possible combinations of features disclosed herein unless expressly stated otherwise.

[0041] The above, as well as further objects, features, and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of embodiments thereof, with reference to the accompanying drawings, in which like reference numerals will be used for similar elements, in which: [Brief description of the drawings]

[0042] [Figure 1] FIG. 1 illustrates a schematic diagram of an imaging system with an image pipeline according to an embodiment. [Diagram 2] 1 is a flowchart of a method for intracoding an image frame according to an embodiment. [Diagram 3] FIG. 2 illustrates a schematic diagram of labelling of predicted pixel values ​​within a block containing 4×4 pixels; [Figure 4] FIG. 1 is a diagram illustrating the three intra-prediction modes, namely mode 0, mode 1 and mode 2, of the compression standard AVC. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown.

[0044] 1 illustrates one embodiment of an imaging system 100. The imaging system 100 includes an image source 110 and an image pipeline 120 for intra-coding one or more image frames 112.

[0045] The image source 110 may be or may comprise an image sensor (not shown) configured to capture one or more image frames of a scene. The image source 110 may form part of a video camera, such as a surveillance camera. Alternatively or additionally, the image source 110 may be an image rendering engine. The image rendering engine is configured to render the image frames. The image rendering engine may be configured by a computer program to render photorealistic or non-photorealistic image frames from a 2D or 3D model. Alternatively or additionally, the image source 110 may be a storage configured to store one or more image frames received from an image sensor (not shown) located outside the image source 110 and in communication with the image source 110. Furthermore, the image source 110 may be configured to forward the image frames 112 to the image pipeline 120. The image pipeline 120 may form part of a video camera, such as a surveillance camera, as discussed above. Thus, in some embodiments. The video camera comprises the imaging system 100.

[0046] 1, the image pipeline 120 comprises a processing unit 122 and an encoding unit 124. The processing unit 122 is disposed prior to the encoding unit 124 in the image pipeline 120, such that the encoding unit 124 follows the processing unit 122. The encoding unit 124 is configured to support multiple intra-prediction modes used to intra-code the image frames 112. The image pipeline 120 may also comprise a receiving and transmitting unit (not shown) configured to receive image frames from the image source 110 and transmit the image frames to a client device.

[0047] As such, the image pipeline 120 is configured to process one or more image frames 112 received from the image source 110 and transmit the processed and encoded image frames 112″ in one or more bitstreams, e.g., over a wireless or wired communication network (not shown), to one or more clients. In particular, the image pipeline 120 is configured to process the image frames to reduce the encoded frame size of intra-coded image frames, i.e., the encoded frame size of I-frames transmitted from the encoding unit 124 in one or more bitstreams. As also shown in FIG. 1, each client may have a decoding unit 130 configured to decode the received bitstream of encoded image frames 112″ and present the decoded image frames on a display device 140, such as on a video monitor.

[0048] A method for intra-coding an image frame 112 will now be described with reference to the flow chart of FIG. 2. The method may be performed by the image pipeline 120 of the imaging system 100. The method or one or more steps of the method may be performed independently in several parts of the image pipeline 120. For illustrative purposes, in the following it is assumed that the steps of the method are performed by the processing unit 122 or the encoding unit 124 of the image pipeline 120. However, examples of performing some steps of the method in another part, such as in a pre-processing unit (not shown), will be given as well. Furthermore, it should be noted that some of the steps may be optional and that the steps can be performed in another suitable order. Furthermore, in the present disclosure, one or more steps are described as being performed on at least one block of pixel values. This should be understood, since one or more steps may be performed on multiple blocks of pixel values, e.g., all blocks of pixel values, of an image frame or of a portion of an image frame.

[0049] In step S202, the processing unit 122 receives the image frame 112. As mentioned above, the processing unit 122 may obtain the image frame 112 from the image source 110. In the present disclosure, the image frame 112 is an intra-coded frame, i.e., the frame is encoded using intra-mode compression.

[0050] In optional step S204, at least one non-interest region in the image frame 112 is identified. The at least one non-interest region includes at least one block of pixel values. As described in more detail below with reference to steps S206 and S208, the image frame 112 is updated by adjusting at least one pixel value of the at least one block of pixel values ​​to reduce a residual between the at least one pixel value and a corresponding predicted pixel value. By adjusting the at least one pixel value in the non-interest region, a possible drawback of a reduction in image quality is less significant since it will occur in a non-interest region of the image frame 112 and will not negatively affect the viewing experience. Thus, in some embodiments, at least one block of pixel values ​​is selected to be included in the non-interest region.

[0051] The at least one region of non-interest may be identified in a number of ways by image pipeline 120. For example, the at least one region of non-interest may be identified by processing unit 122, but may alternatively be identified by another portion of image pipeline 120, such as by a pre-processing unit (not shown) preceding processing unit 122.

[0052] In a first group of embodiments, the image pipeline 120 may identify at least one non-interest region in the image frame 112 by dividing the image frame 112 into image segments, classifying the image segments, and identifying at least one of the image segments as at least one non-interest region when the at least one of the image segments belongs to a group of predefined classes. For example, the group of predefined classes may include classes such as sky and vegetation, and / or subclasses such as daytime sky, nighttime sky, clouds, and trees, shrubs, and grass, respectively.

[0053] In a second group of embodiments, the image pipeline 120 may identify at least one region of non-interest in the image frame 112 by determining the spatial variability of pixel values ​​contained in at least one region of the image frame 112 and identifying the at least one region of non-interest as a region containing pixel values ​​having a spatial variability less than a region threshold. Thus, the region threshold may be a value below which the spatial variability is sufficiently low to consider the region as non-interesting. For example, the region threshold may be a value corresponding to a 10th percentile of the spatial variability of the pixels in the frame, such that statistically, 10 percent of all pixel values ​​will be less than the region threshold and therefore may be considered to belong to a region of non-interest. The spatial variability for a region may be the variance or standard deviation of the pixel values ​​in the region.

[0054] Alternatively or additionally, at least one block of pixel values ​​included in the non-interest region may have a spatial variability of pixel values ​​greater than the block threshold. If several blocks in the non-interest region have a spatial variability of pixel values ​​greater than the block threshold, a block with the maximum spatial variability among the several blocks may be selected. The processing unit 122 may then adjust at least one pixel value in the at least one block of pixel values ​​to reduce the spatial variability of pixel values ​​below the block threshold. For example, the block threshold may be set to a value close to the spatial variability of the block, such as within ±1 to 10 percent of the spatial variability of the block, such as within ±1 to 5 percent of the spatial variability of the block. The spatial variability for a block may be the variance or standard deviation of pixel values ​​in the block. This may be used in intra prediction modes that use average values ​​of top and left hand samples, such as in AVC intra prediction mode 2 (DC).

[0055] In step S206, the processing unit 122 predicts pixel values ​​for at least one block of pixel values ​​in the image frame 112 by using an intra prediction mode supported by the encoding unit 124. As mentioned above, the at least one block of pixel values ​​may be included in a region of the image frame, such as a region of non-interest. Furthermore, the processing unit 122 may predict pixel values ​​for all blocks of pixel values ​​included in a region of the image frame, such as a region of non-interest.

[0056] As mentioned above, the encoding unit 124 supports multiple intra prediction modes. Therefore, in the case of prediction of pixel values, i.e., in the case of intra prediction, the processing unit 122 should use one intra prediction mode from among the multiple intra prediction modes supported by the encoding unit 124. For example, the encoding unit 124 can support one or more of the 9 different intra prediction modes supported by AVC, one or more of the 35 different intra prediction modes supported by the compression standard HEVC, and / or one or more of the 54 different intra prediction modes supported by the compression standard AV1, to name a few examples.

[0057] In a first set of embodiments, the processing unit 122 may select an intra-prediction mode from among a plurality of intra-prediction modes supported by the encoding unit 124 based on a set of predefined classes. To that end, the processing unit 122 may be configured to select an intra-prediction mode based on a classification of an image segment. If the classification indicates that the region is homogeneous, then an intra-prediction mode suitable for homogeneous regions should be selected and used.

[0058] Alternatively, in a second group of embodiments, the processing unit 122 may select an intra-prediction mode from among multiple intra-prediction modes supported by the encoding unit 124 based on the spatial variability. For example, if the spatial variability is smaller than a region threshold, it may indicate that the region is homogeneous and that an intra-prediction mode suitable for homogeneous regions should be selected and used. However, it should be understood that different intra-prediction modes suitable for homogeneous regions may be used for different homogeneous regions. Therefore, the same intra-prediction mode does not need to be used for all homogeneous regions. For example, one or more further region thresholds successively smaller than the region threshold may be available, and when the spatial variability is smaller than one region threshold of the one or more further region thresholds, it may indicate that another intra-prediction mode suitable for homogeneous regions should be used. For example, it may indicate that intra-prediction mode 2 (DC) should be used instead of intra-prediction modes 0 and 1.

[0059] 3 shows an example in which a predicted pixel value in a block containing 4×4 pixels is labeled ap, and pixel values ​​above and to the left of the block are labeled M, AH, and IL, respectively. Depending on the intra prediction mode used, pixel values ​​above and / or to the left of the block are used to predict the pixel value ap.

[0060] 4 illustrates diagrammatically three first intra-prediction modes, namely mode 0, mode 1, and mode 2 (also called DC mode), out of nine intra-prediction modes for the compression standard AVC. In each of the intra-prediction modes, namely mode 0, mode 1, and mode 2, it is shown how pixel values ​​above and / or to the left of the block are used to predict the pixel value ap.

[0061] In AVC intra-prediction mode 0, pixel values ​​above each column of pixel values ​​in a block will be used for prediction, giving all pixel values ​​in the same column the same predicted pixel value. Thus, pixel values ​​a, e, i, and m will be predicted to have pixel value A; pixel values ​​b, f, j, and n will be predicted to have pixel value B; pixel values ​​c, g, k, and o will be predicted to have pixel value C; and pixel values ​​d, h, l, and p will be predicted to have pixel value D.

[0062] In AVC intra prediction mode 1, the left pixel value of each row of pixel values ​​in a block will be used for prediction, giving all pixel values ​​in the same row the same predicted pixel value.

[0063] In AVC intra-prediction mode 2, the average of the pixel values ​​to the left and above a pixel value in a block will be used for prediction, giving all pixel values ​​in the block the same pixel value. In the example shown in Figure 4, all pixel values ​​in the block will be predicted to have the average value of pixel values ​​AD and IL.

[0064] For homogeneous regions or regions with low spatial variability, i.e., containing pixel values ​​below a region threshold, one of the three first modes of intra prediction modes supported by AVC, namely mode 0, mode 1, or mode 2 (DC), may be suitable to select for prediction of pixel values ​​within at least one block of pixel values.

[0065] In another alternative, the processing unit 122 may use at least two of the multiple intra prediction modes to calculate a predicted pixel value for each pixel value in at least one block of pixel values, and select an intra prediction mode of at least two of the multiple intra prediction modes that provides a minimum sum of absolute values ​​of residuals between the pixel value and the corresponding predicted pixel value. Thus, the processing unit 122 may use all of the multiple supported intra prediction modes to calculate a predicted pixel value for each pixel value in at least one block of pixel values. In such a scenario, the processing unit 122 may select an intra prediction mode of the multiple intra prediction modes that provides a minimum sum of absolute values ​​of residuals between the pixel value and the corresponding predicted pixel value.

[0066] In some embodiments, processing unit 122 may use multiple intra prediction modes supported by encoding unit 124. In such embodiments, processing unit 122 may select an intra prediction mode from multiple supported intra prediction modes. For example, processing unit 122 may select an intra prediction mode from a subset of multiple intra prediction modes supported by encoding unit 124. The subset of multiple intra prediction modes may be a proper subset and thus may not include all of the intra prediction modes supported by encoding unit 124.

[0067] 2, in step S208, processing unit 122 updates image frame 112 by adjusting at least one pixel value in at least one block of pixel values ​​in image frame 112 to reduce a residual error between the at least one pixel value and a corresponding predicted pixel value. This can also be expressed as processing unit 122 reducing a residual error between at least one pixel value in at least one block of pixel values ​​and a corresponding predicted pixel value in step S208 by adjusting at least one pixel value in the at least one block and updating the image frame with the adjusted at least one pixel value in the at least one block of pixel values.

[0068] The processing unit 122 can adjust at least one pixel value in at least one block of pixel values ​​upward or downward by a predefined amount within the pixel value range to reduce the residual. For example, when the corresponding predicted pixel value is greater than the at least one pixel value, the processing unit 122 adjusts at least one pixel value in at least one block of pixel values ​​upward by a predefined amount within the pixel value range to reduce the residual. Correspondingly, and to reduce the residual when the corresponding predicted pixel value is less than the at least one pixel value, the processing unit 122 adjusts at least one pixel value in at least one block of pixel values ​​downward by a predefined amount within the pixel value range.

[0069] It should be noted that reducing the residual completely, i.e., to zero, is not the goal, which is typically not the case, since a complete reduction of the residual would result in an excessive loss of image information. Instead, it is noted that a further slight reduction of the residual increases the likelihood that the encoding unit 124 will subsequently encode an updated image frame with at least one adjusted pixel value according to the intra-prediction mode used by the processing unit 122 for prediction. Encoding the updated image frame 112' according to the same intra-prediction mode will result in a reduced encoded frame size of the I-frame compared to the I-frame resulting from encoding the original, unupdated image frame 112.

[0070] The processing unit 122 may also select at least one pixel value to be adjusted. For example, this may be done by the processing unit 122 calculating a residual between a pixel value and a corresponding predicted pixel value, and selecting at least one pixel value to be adjusted such that the pixel value for which the absolute value of the calculated residual is greater than a pixel threshold. Thus, if the absolute value of the difference is greater than the pixel threshold for a particular pixel value, that pixel value may be selected for adjustment. If several pixel values ​​have an absolute difference greater than the pixel threshold, at least one or all of those pixel values ​​may be adjusted. For example, the pixel threshold may be set to a value close to the spatial variability of the block, such as within ±1 to 10 percent of the spatial variability of the block, e.g., within ±1 to 5 percent. This may be used in intra-prediction modes that use average values ​​of upper and left hand samples, such as in AVC intra-prediction mode 2 (DC).

[0071] Once the pixel values ​​are selected, they should be adjusted, which can be performed in different ways: for example, the processing unit 122 adjusts at least one pixel value in at least one block of pixel values ​​such that the absolute value of the residual is less than or equal to a pixel threshold value.

[0072] Alternatively or additionally, the processing unit 122 adjusts the at least one pixel value by a predefined amount. The defined amount by which the at least one pixel value may be adjusted may be one or two steps up or down within the pixel value range. As mentioned above, the pixel value range is 0 to 255 for an 8 bpp video stream and 0 to 1023 for a 10 bpp video stream, to give two examples.

[0073] Alternatively or additionally, the processing unit 122 may adjust at least one pixel value in at least one block of pixel values ​​to be equal to a neighboring pixel value, whereby subsequent encoding of the image frame is simplified since neighboring pixels having the same pixel value can be encoded in the same way. As mentioned above, this may be advantageous for vertical intra-prediction modes (e.g., AVC mode 0) or horizontal intra-prediction modes (e.g., AVC mode 1). Furthermore, this may be advantageous when the adjustment results in a reduction in residuals, since this may simplify encoding for the encoding unit 126 and may require less encoding resources compared to the case where the neighboring pixel values ​​are different.

[0074] However, it should be understood that the processing unit 122 may adjust at least one pixel value in at least one block of pixel values ​​to be equal to the pixel value that is most favorable for the intra-prediction mode used, i.e., the pixel value that gives the smallest residual for the intra-prediction mode used.

[0075] Furthermore, the processing unit 122 may adjust one or more, but not all, of the pixel values ​​in the at least one block of pixel values. In other words, only a proper subset of pixel values ​​from the total number of pixel values ​​in the at least one block that are adjusted. It should be noted that the proper subset does not include all pixel values ​​of the block. Thus, the intention is not to replace a block of pixel values ​​with another block of pixel values, since that would excessively reduce image quality. Instead, the intention is to adjust only one or more pixel values, or as many pixel values ​​as necessary, as necessary, to increase the probability that the encoding unit 124 will select the same intra-prediction mode that the processing unit 122 used to predict the pixel values. As mentioned above, if the encoding unit 124 selects the same intra-prediction mode as the intra-prediction mode used by the processing unit 122, the desired reduction in the encoded frame size is obtained.

[0076] In some alternative embodiments, the processing unit 122 may adjust all of the pixel values ​​in at least one block of pixel values ​​so long as the residual for at least one pixel is not reduced to zero, i.e., so long as the residual for at least one pixel is greater than zero and less than a pixel threshold.

[0077] In step S210, the encoding unit 124 performs intra-coding on the updated image frame 112' using at least one intra-prediction mode from among a plurality of intra-prediction modes. The intra-coding performed by the encoding unit 124 includes predicting pixel values ​​for blocks of pixel values ​​in the updated image frame and calculating residuals using one or more of the supported intra-prediction modes, according to standard techniques, and selecting an intra-prediction mode that gives the smallest residual for each block of pixel values ​​for intra-coding of each respective block of pixel values. As known to those skilled in the art, the encoding unit 124 will encode different blocks of the updated image frame 112' differently depending on the region in which the block is located. For example, if the block is located in a uniform region, AVC mode 2 (DC) may give the smallest residual and therefore may be selected. As another example, if the block is located in a region containing a vertical pattern, AVC mode 0 (vertical) may give the smallest residual and therefore may be selected. Therefore, the encoding unit 125 can use different intra-prediction modes to encode different blocks of the updated image frame 112'. For example, if a block of the updated image frame 112' is included in an ROI, the encoding unit 124 will perform intra-coding using an intra-prediction mode suitable for encoding objects or details, and if the block is included in a non-interest region, the encoding unit 124 will perform intra-coding using an intra-prediction mode suitable for encoding a non-interest region, for example, one mode from among mode 0, mode 1, and mode 2 of AVC.Since the encoding unit 124 will calculate the residual using one or more of the supported intra prediction modes and select the intra prediction mode that gives the smallest residual for intra coding, the likelihood that the encoding unit 124 will select the same intra prediction mode as the intra prediction mode used by the processing unit 122 will increase after adjusting at least one pixel value, because the result of the adjustment is a reduction in the residual for the selected intra prediction mode. Therefore, if the encoding unit 124 selects the same intra prediction mode as the intra prediction mode used by the processing unit 122, the size of the encoded image frame 122'' will be reduced. Furthermore, thanks to the encoding unit 124 operating according to standard techniques, a standard decoding unit 130, e.g., a legacy decoding unit, can decode the received bitstream of the encoded image frame 122'' without requiring any modification of the decoding unit 130. Therefore, the encoding unit 124 is compatible with standard decoding units.

[0078] As described above, the image pipeline 120 may be configured to implement a method for intracoding image frames 112. To this end, the image pipeline 120 may include circuitry configured to implement the various method steps described above.

[0079] In a hardware implementation, the circuitry may be dedicated and specifically designed to implement one or more of the method steps. The circuitry may be in the form of one or more integrated circuits, such as one or more application specific integrated circuits or one or more field programmable gate arrays. By way of example, the image pipeline 120 may thus comprise a circuitry that, when in use, predicts pixel values ​​of at least one block of pixel values ​​and adjusts at least one of the pixel values ​​using an intra-prediction mode supported by the encoding unit 124.

[0080] In a software implementation, the circuitry may instead be in the form of a processor, such as a microprocessor, which, in conjunction with computer code instructions stored on a (non-transitory) computer-readable medium, such as a non-volatile memory, causes the image pipeline 120 to perform any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, optical disks, and the like. In the case of software, each of the method steps described above may thus correspond to a portion of computer code instructions stored on a computer-readable medium, which, when executed by the processor, causes the image pipeline 120 to perform any of the methods disclosed herein.

[0081] It will be appreciated that it is also possible to have a combination of hardware and software implementations, meaning that some method steps are implemented in hardware and other method steps are implemented in software.

[0082] It will be appreciated that a person skilled in the art can modify the above described embodiment in many ways and still use the advantages of the present invention shown in the above embodiment. For example, the image pipeline 120 does not have to be a single unit with the processing unit 122 and the encoding unit 124 in one location, but can be a virtual unit in which the processing unit 122 and the encoding unit 124 work together but are provided in different locations. Alternatively, the processing unit 122 can be an integral part of the encoding unit 124. Furthermore, possible pre-processing units can be located outside the image pipeline 120, for example between the image source 110 and the image pipeline 120. Therefore, the present invention should not be limited to the illustrated embodiment, but should only be defined by the appended claims. Furthermore, the illustrated embodiments can be combined as a person skilled in the art will understand.

Claims

1. A method implemented by an image pipeline (120) for intra-coding an image frame (112), the image pipeline (120) comprising a processing unit (122) and an encoding unit (124) following the processing unit (122) that supports multiple intra-prediction modes used to intra-code the image frame, the method comprising: by said processing unit (122), receiving (202) an image frame; for at least one block of pixel values ​​in said image frame (112), - predicting (206) pixel values ​​of said at least one block of pixel values ​​using intra-prediction modes supported by said encoding unit (124); reducing a residual between at least one pixel value in said at least one block of pixel values ​​and a corresponding predicted pixel value by adjusting said at least one pixel value in said at least one block of pixel values ​​upwards by a predefined amount within a pixel value range when said corresponding predicted pixel value is greater than said at least one pixel value, or downwards by said predefined amount within said pixel value range when said corresponding predicted pixel value is less than said at least one pixel value; updating (208) said image frame (112) with the adjusted value of said at least one pixel in said at least one block of pixel values; To carry out the following: by said encoding unit (124), - intra-coding (210) said updated image frame (112') using at least one intra-prediction mode from said plurality of intra-prediction modes. wherein the intra-coding (210) includes performing - for each block of pixel values ​​in said updated image frame (112'), predicting pixel values ​​using one or more of said supported intra prediction modes and calculating residuals; and - for intra-coding of each respective block of pixel values, selecting the intra-prediction mode that gives the smallest residual for each said block of pixel values; A method comprising:

2. 2. The method of claim 1, further comprising: identifying (204) at least one region of non-interest within the image frame (112), wherein the at least one block of pixel values ​​is included in the at least one region of non-interest.

3. The identifying (204) of the at least one region of non-interest within the image frame may include: - dividing said image frames into image segments; - classifying said image segments; and - identifying at least one of said image segments as being a region of non-interest when said at least one of said image segments belongs to a group of predefined classes; The method of claim 2 , comprising:

4. The method of claim 3 , wherein the intra-prediction mode used by the processing unit (122) is selected based on the group of predetermined classes.

5. - calculating, by said processing unit (122), a predicted pixel value for each pixel value in said at least one block of pixel values ​​using at least two intra prediction modes of said plurality of intra prediction modes, and selecting, from said at least two intra prediction modes of said plurality of intra prediction modes, an intra prediction mode that gives a minimum sum of absolute values ​​of said residuals between said pixel values ​​and said corresponding predicted pixel values. The method of claim 2 , further comprising:

6. The identifying (204) of the at least one region of non-interest within the image frame (112) may include: determining the spatial variability of pixel values ​​contained in at least one region within said image frame (112); and - identifying said at least one region of non-interest as a region containing pixel values ​​having a spatial variability below a certain region threshold; The method of claim 2 , comprising:

7. The method of claim 2 , wherein the at least one block of pixel values ​​included in the region of non-interest has a spatial variability of pixel values ​​that is greater than a block threshold.

8. The adjusting of the at least one pixel value in the at least one block of pixel values ​​includes: adjusting said at least one pixel value within said at least one block of pixel values ​​to reduce said spatial variability of said pixel values ​​below said block threshold. The method of claim 7, comprising:

9. The method of claim 1 , wherein the intra-prediction mode used by the processing unit (122) is selected from a subset of the plurality of intra-prediction modes supported by the encoding unit (124).

10. - calculating, by said processing unit (122), said residual between said pixel value and said corresponding predicted pixel value and selecting said at least one pixel value to be adjusted such that the absolute value of said calculated residual is a pixel value greater than a certain pixel threshold value. The method of claim 1 , further comprising:

11. The adjusting of the at least one pixel value in the at least one block of pixel values ​​includes: - adjusting said at least one pixel value within said at least one block of pixel values ​​such that the absolute value of said residual is less than or equal to said pixel threshold value. The method of claim 10, comprising:

12. The adjusting of the at least one pixel value comprises: - adjusting said at least one pixel value within said at least one block of pixel values ​​to be equal to a neighboring pixel value; The method of any one of claims 1 to 11, comprising:

13. The adjusting of the at least one pixel value comprises: adjusting one or more, but not all, of said pixel values ​​in said at least one block of pixel values; 13. The method of any one of claims 1 to 12, comprising:

14. An image pipeline (120) for intra-coding an image frame (112), comprising: a processing unit (122); and an encoding unit (124) following the processing unit (122), the encoding unit (124) being configured to support multiple intra-prediction modes used to intra-code the image frame; The processing unit (122) - configured to receive image frames (112), for at least one block of pixel values ​​in said image frame (112), predicting pixel values ​​of said at least one block of pixel values ​​using intra prediction modes supported by said encoding unit (124); - configured to reduce a residual between at least one pixel value in said at least one block of pixel values ​​and a corresponding predicted pixel value by adjusting said at least one pixel value in said at least one block of pixel values ​​upwards by a predefined amount within a pixel value range when said corresponding predicted pixel value is greater than said at least one pixel value, or downwards by said predefined amount within said pixel value range when said corresponding predicted pixel value is less than said at least one pixel value, and to update said image frame (112) with said adjusted at least one pixel value in said at least one block of pixel values, The encoding unit (124) - for each block of pixel values ​​in said updated image frame (112'), predicting pixel values ​​using one or more of said supported intra prediction modes and calculating residuals; for intra-coding of each respective block of pixel values, selecting the intra-prediction mode that gives the smallest residual for each said block of pixel values; By being configured as such, an image pipeline (120) configured to intra-code said updated image frame (112') using at least one intra-prediction mode from among said plurality of intra-prediction modes;

15. A non-transitory computer readable medium having stored thereon computer code instructions adapted to perform the method of any one of claims 1 to 13 when executed by a device having processing capability.

Citation Information

Patent Citations

  • Device and method for image processing and storage medium

    JP2001218062A

  • Coding device, coding method, and program

    JP2006340134A

  • Image processing apparatus, image processing method, and program

    JP2009089356A

  • Position dependent intra prediction combination with wide angle intra prediction

    US20200021817A1