Method, device, and recording medium for generating luma or chroma based on a linear model

Abstract

The linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure may include the steps of: obtaining a background region map by distinguishing a region of interest and a background region within an image; obtaining a background region patch and a region of interest patch based on the background region map; obtaining a linear model applied to the background region patch; and restoring a Luma component of the background region using a Chroma component of the background region based on the linear model.

Description

Technology Field

[0001] The present disclosure may relate to the technical field utilizing a linear model-based luma or chroma generation method, apparatus, and recording medium for application in Video Coding for Machines (VCM). Background Technology

[0002] Existing video encoding technologies have achieved improvements in compression efficiency and image quality to be optimized for the human visual system. However, future video encoding technologies are expected to be utilized not only for human vision but also extensively in the field of machine vision, including surveillance functions, intelligent transportation, smart cities, and intelligent industries. Therefore, it is necessary to develop video encoding technologies that can achieve high-efficiency compression and recognition accuracy by considering both human and machine vision simultaneously. Accordingly, ISO / IEC MPEG formed the VCM Ad-hoc Group in July 2019 and is currently developing standards for machine video encoding technologies that are efficient for environments performing machine inference tasks such as object detection, object segmentation, and object tracking. The problem to be solved

[0003] The present disclosure may relate to a method, apparatus, and recording medium for improving image encoding and decoding performance for a machine through linear model-based luma or chroma generation. means of solving the problem

[0004] The linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure may include the steps of: obtaining a background region map by distinguishing a region of interest and a background region within an image; obtaining a background region patch and a region of interest patch based on the background region map; obtaining a linear model applied to the background region patch; and restoring a Luma component of the background region using a Chroma component of the background region based on the linear model.

[0005] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the background region and the region of interest patch may be block units comprising at least one pixel.

[0006] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the block size of the block unit may be 16×16.

[0007] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the linear model may be determined by a set of parameters including weights and bias values.

[0008] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the weights may include weights applied to a Chroma-Blue component and weights applied to a Chroma-Red component.

[0009] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the parameter set may be determined by an index signaled from a bitstream specifying the parameter set among a plurality of parameter sets.

[0010] In the linear model-based Luma or Chroma generation method, apparatus, and recording medium of the present disclosure, the method for acquiring the plurality of parameter sets may include: acquiring a first plurality of parameter sets using some combinations among all combinations of the weights and the bias values ​​that minimize the difference between the original Luma image and the generated Luma image values; clustering the first plurality of parameter sets to acquire a second plurality of parameter sets; and acquiring the plurality of parameter sets using some parameter sets with a high selection frequency among the second plurality of parameter sets. Effects of the invention

[0011] Through the present disclosure, encoding efficiency can be improved by encoding only the luminance or chroma in all or part of an image and generating the luminance or chroma through a linear model in a decoder. Brief explanation of the drawing

[0012] FIG. 1 is a block diagram of an image encoder according to one embodiment of the present disclosure. FIG. 2 is a block diagram of an image decoder according to one embodiment of the present disclosure. FIG. 3 is a flowchart of an image encoding / decoding method based on a region of interest according to one embodiment of the present disclosure. Figure 4 illustrates an example of an image encoding and decoding structure for a machine. Figure 5 illustrates an example of generating luminance in a background area using chroma values ​​and a linear model. Figure 6 is an enlarged view of the linear model of Figure 5. Figure 7 illustrates an example of an image sub / decoder structure for a machine to which a linear model-based luma generation module is applied. Figure 8 illustrates an example of an information map of the region of interest and a background region. Figure 9 illustrates examples of a luminance image of a region of interest within the image and luminance and chroma (Cb, Cr) images of the entire image. Figure 10 illustrates an example of the linear model derivation process in the VCM encoder and the background region luma generation process in the VCM decoder. FIG. 11 illustrates an example of linear model parameters. Specific details for implementing the invention

[0013] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0014] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0015] Where it is stated that any component of the present disclosure is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, where it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0016] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0017] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0018] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0020] FIG. 1 is a block diagram of an image encoder according to one embodiment of the present disclosure.

[0021] Referring to FIG. 1, the image encoder may include a preprocessing unit (110) and an image encoding unit (120).

[0022] The preprocessing unit (110) performs a preprocessing process to convert the input original images into images suitable for image encoding. At this time, the images input to the preprocessing unit (110) may be color or black-and-white images following the YUV or YCbCr format.

[0023] The preprocessing unit (110) may include at least one of a temporal resampling unit (112), a spatial resampling unit (114), or a region of interest-based processing unit (116).

[0024] The temporal resampling unit (112) resamples the images temporally. Only the resampled images can be selected as targets for image encoding. That is, through temporal resampling, encoding for some of the images input to the preprocessing unit (110) can be omitted. For example, odd-numbered images of a 60fps (frames per second) image can be omitted to convert the 60fps image into a 30fps image. Alternatively, images of a specific output order can be omitted by considering the temporal overlap between the images.

[0025] The spatial resampling unit (114) spatially resamples the image. Through spatial resampling, the size and / or spatial resolution of the image may be reduced. For example, the spatial resampling unit may convert the image so that an image with a resolution of 1920x1080 can be converted and encoded into an image such as 960x540 or 480x270 by the image encoding unit (120), and may transmit resampling information to the image encoding unit (120).

[0026] The region of interest-based processing unit (116) sets a region of interest within the image so that image encoding / decoding is performed primarily on information important for machine inference tasks. The region of interest-based processing unit (116) can remove background areas excluding the set region of interest, or adjust the size and / or position of the region of interest within the image so that the region of interest is encoded / decoded with high quality.

[0027] The video encoding unit (120) encodes the video output from the preprocessing unit (110). Meanwhile, the video encoding unit (120) can encode the video by utilizing a conventional codec technology or a codec technology modified for VCM (Video Coding for Machine) based on a conventional codec technology. For example, the video encoding unit (120) can encode the video based on HEVC, VVC, or AV1. A bitstream is generated as a result of the video encoding, and the generated bitstream can be transmitted to a video decoder.

[0028] FIG. 2 is a block diagram of an image decoder according to one embodiment of the present disclosure.

[0029] Referring to FIG. 2, the image decoder may include an image decoder (210) and a post-processing unit (220).

[0030] The image decoding unit (210) decodes the bitstream received from the image encoder (110) to generate a decoded (or reconstructed) image. The image decoding unit (210) can decode the bitstream based on the codec technology used in the image encoding unit (120).

[0031] The post-processing unit (220) performs post-processing on the decoded image. Through post-processing, the size and frame rate of the image can be restored to match the original image.

[0032] The post-processing unit (220) may include at least one of a post-filtering unit (222), a region of interest-based restoration unit (224), a spatial restoration unit (226), or a temporal restoration unit (218).

[0033] The post-filtering unit (222) applies filtering to reduce the restoration error of the decoded image. For example, the post-filtering unit (222) may apply an in-loop filter to the decoded image. The in-loop filter may include at least one of a deblocking filter, a sample adaptive offset filter, an LMCS (Luma mapping chroma scaling), an adaptive loop filter, or a neural network-based filter.

[0034] In the region of interest-based restoration unit (224), an image of the same size as the original image is obtained based on the region of interest information. For example, if a cropped image containing the region of interest is encoded, the decoded image will have a different size from the original image. Accordingly, the region of interest-based restoration unit can adjust the retargeted image to the original size. Here, the retargeted image may represent the decoded image or an image that has been upscaled through the spatial restoration unit (226). Alternatively, if the size or location of the region of interest within the image to be encoded is adjusted, the region of interest-based restoration unit (224) can adjust the location and size of the region of interest within the retargeted image to match the original image.

[0035] In the spatial restoration unit (226), upscaling is performed on the decoded image. Through upscaling, the decoded image can be restored to the same size and / or spatial resolution as the original image.

[0036] In the temporal restoration unit (228), an image of a temporal location where encoding / decoding was omitted is restored by temporal resampling. Specifically, the temporal restoration unit (228) can generate an image of a temporal location where encoding / decoding was omitted through interpolation between decoded images.

[0037] Meanwhile, additional information may be encoded and signaled to perform reverse processing on the image processing performed in the preprocessing unit (110). In the postprocessing unit (220), postprocessing of the decoded image may be performed based on the additional information to generate an image for machine inference to be performed on a machine. Meanwhile, the additional information may be referred to as 'metadata'.

[0038] Metadata may include at least one of temporal resampling information, spatial resampling information, or region of interest processing information.

[0039] Temporal resampling information may include at least one of a flag indicating whether temporal resampling has been performed or information indicating a temporal resampling rate.

[0040] For example, the above flag being 1 indicates that temporal resampling has been performed. In this case, information indicating the temporal resampling rate may be additionally encoded / decoded. When temporal resampling is performed, fewer images than the number of original images may be encoded / decoded. In the image decoder, images that have been omitted from encoding / decoding can be restored through temporal restoration.

[0041] On the other hand, the above flag being 0 indicates that temporal resampling was not performed.

[0042] The temporal resampling rate can be expressed as a power of 2. For example, a temporal resampling rate of 2^N indicates that one of 2^N images is selected as the image to be encoded / decoded. For instance, only images with a POC (Picture Order Count) that is a multiple of 2^N can be the targets for encoding / decoding. Information representing the temporal resampling rate can represent the power of the temporal resampling rate (i.e., N). For example, the above information can represent the value of the power of the temporal resampling rate or the value obtained by subtracting 1 from the power of the rate.

[0043] Spatial resampling information may include at least one of a flag indicating whether spatial resampling has been performed, or information indicating a scaling parameter for spatial resampling.

[0044] For example, the above flag being 1 indicates that spatial resampling has been performed. In this case, information representing scaling parameters may be additionally encoded. Specifically, information representing horizontal scaling parameters and information representing vertical scaling parameters may each be encoded and signaled. When spatial resampling is performed, the size and / or spatial resolution of the image may be reduced. In the image decoder, the size of the decoded image can be restored to the size of the original image or a preset size through spatial reconstruction. Meanwhile, information for specifying the preset size may be additionally encoded / decoded.

[0045] The above flag being 0 indicates that spatial resampling was not performed.

[0046] The region of interest processing information may include at least one of image size information or region of interest information.

[0047] The image size information may include information indicating whether retargeting has been performed. A retargeting flag of 1 indicates that the retargeted image is encoded / decoded instead of the original image. On the other hand, a retargeting flag of 0 indicates that the original image is encoded / decoded as is.

[0048] A retargeting image represents an image generated by performing at least one of resolution adjustment and position adjustment on at least one region of interest within an original image. Accordingly, the resolution or position of the region of interest within the retargeted image may differ from that of the original image. Additionally, the size of the retargeting image may be the same as or smaller than that of the original image.

[0049] If retargeting is allowed (i.e., when the retargeting flag is 1), size information of the retargeted image can be encoded / decoded. The size information of the retargeted image may include width information of the image and height information of the image.

[0050] Meanwhile, additional encoding / decoding information indicating the size difference between the original image and the retargeted image may be performed. For example, information indicating whether the difference in size between the retargeted image and the original image is encoded / decoded may be encoded / decoded.

[0051] For example, information indicating whether the magnitude difference is encoded / decoded is 0, which indicates that the magnitude difference between the retargeted image and the original image is not encoded / decoded.

[0052] On the other hand, if the information indicating whether the size difference is encoded / decoded is 1, it indicates that the size difference between the retargeted image and the original image is encoded / decoded. In this case, information indicating the difference in size between the retargeted image and the original image may be additionally encoded / decoded.

[0053] Information indicating the size difference represents the size difference between the original image and the retargeted image. Meanwhile, information indicating the size difference in the horizontal direction and information indicating the size difference in the horizontal direction can each be encoded and signaled.

[0054] The region of interest information may include at least one of a flag indicating whether a region of interest exists, number information of the region of interest, scaling parameters of the region of interest, or location information of the region of interest.

[0055] For example, the above flag being 1 indicates that information about the region of interest is encoded / decoded. In this case, at least one of the number of regions of interest, scaling parameter information of the region of interest, location information of the region of interest, or size information of the region of interest may be additionally encoded / decoded.

[0056] On the other hand, if the above flag is 0, it indicates that there is no region of interest.

[0057] Information on the number of regions of interest indicates the number of regions of interest. Meanwhile, the number of regions of interest can be calculated in units of image groups containing at least one image.

[0058] The scaling parameter of the region of interest represents the scaling parameter for the region of interest. Depending on the scaling parameter of the region of interest, the size of the region of interest can be adjusted.

[0059] The scaling parameter information of the region of interest may include information indicating whether the scaling parameter of the region of interest has been updated. If the information indicating whether to update indicates that the scaling parameter of the region of interest has not been updated, the scaling parameter of the region of interest may be set to a default value or the same value as in the previous frame. On the other hand, if the information indicating whether to update indicates that the scaling parameter of the region of interest should be updated, the information indicating the scaling parameter of the region of interest may be additionally encoded / decoded.

[0060] Meanwhile, scaling parameter information for the region of interest can be encoded / decoded individually for each region of interest.

[0061] The location information of the region of interest indicates the location of the region of interest in the original image. At this time, the horizontal position information (i.e., x-axis coordinate) and the vertical position information (i.e., y-axis coordinate) of the region of interest can be encoded / decoded, respectively.

[0062] The size information of the region of interest represents the size of the region of interest in the original image. In this case, the horizontal size (i.e., width) and vertical size (i.e., height) information of the region of interest can be encoded and decoded, respectively.

[0063] As described above, according to the present disclosure, through an image preprocessing process, the encoding / decoding efficiency of the image can be improved while maintaining machine mission performance.

[0064] FIG. 3 is a flowchart of an image encoding / decoding method based on a region of interest according to one embodiment of the present disclosure.

[0065] In the image encoder, a region of interest within the image can be determined (S310). Specifically, in the region of interest-based processing unit within the image encoder, the image can be divided into a region of interest and a background region.

[0066] The determination of the region of interest may be performed at the image level or at the image group level. Here, an image group may consist of multiple images. For example, images belonging to an intra period or a predefined number of images may be set as an image group.

[0067] In the image encoder, a transformed image based on the region of interest can be generated / derived (S320).

[0068] Here, the image transformed based on the region of interest may be one in which the location of the region of interest within the image has been moved to a predefined location. The predefined location may be the top-left, top-right, bottom-left, bottom-right, or center of the image.

[0069] The image transformed based on the region of interest may be encoded through at least one of temporal resampling or spatial resampling (S330, S340). Alternatively, the image transformed based on the region of interest may be generated / derived after at least one of temporal resampling or spatial resampling has been performed.

[0070] In the image decoder, the converted image can be decoded based on the region of interest (S350). Then, using metadata, the decoded image can be converted into an image of the same size as the original image, or the region of interest within the decoded image can be moved to its original position to obtain a restored image (S370). Meanwhile, at least one of temporal restoration or spatial restoration can be performed on the decoded image (S360), and the restored image can be obtained through region of interest restoration on the image on which temporal restoration or spatial restoration has been performed (S370). Alternatively, after region of interest restoration is performed, at least one of temporal restoration or spatial restoration can be performed to obtain a restored image.

[0071] Figure 4 illustrates an example of an image encoding and decoding structure for a machine.

[0072] The role of each module in the VCM (Video Coding for Machines) Encoder can be as follows.

[0073] Temporal resampling is a module that performs resampling of an image or the entire video in a video, and can change the temporal resolution of the input video. For example, an odd-numbered frame of a 60fps (frames per second) video can be omitted to generate a 30fps video. However, the resampling method is not limited to the above, and resampling can be performed in arbitrary units by considering the temporal overlap between images. Temporal resampling can correspond to the temporal resampling unit (112) of FIG. 1.

[0074] Spatial resampling can change the spatial resolution of a video at the video unit level or the entire video unit level. For example, encoding performance can be improved by changing a video with a resolution of 1920x1080 to a smaller resolution such as 960x540 or 480x270. Spatial resampling can correspond to the spatial resampling unit (114) of FIG. 1. In one embodiment, spatial resampling can downsample an input frame when the object occupancy exceeds a predefined threshold.

[0075] ROI-based processing is an image processing part that considers the Region of Interest (ROI) within the video. For example, only important parts (such as people) within the video are selected, and the remaining background parts are removed, thereby increasing encoding performance while maintaining machine task performance. Alternatively, the background parts may be resampled to a low resolution, and the resolution of the important parts may be maintained while recombining the background and important parts to produce an output image. ROI-based processing can correspond to the region of interest-based processing unit (116) of FIG. 1.

[0076] Bit depth truncation is a part that adjusts the luminance and chroma values ​​of an image, for example, converting the luminance value to a value shifted right by 1 bit and outputting it. Bit depth truncation may be included in the preprocessing unit (110) and, although not disclosed in FIG. 1, may be represented as a bit depth truncation unit.

[0077] The Inner Encoder is a module that receives video input and encodes the video using VTM (VVC Test Model), etc., and can correspond to the video encoding unit (120) of FIG. 1.

[0078] The role of each module in the VCM (Video Coding for Machines) Decoder, corresponding to the VCM (Video Coding for Machines) Encoder, is as follows.

[0079] The inner decoder is a part that receives an encoded bitstream and restores an image, and can restore the image using a decoder corresponding to the encoder used during encoding. The inner decoder can correspond to the image decoder (210) of FIG. 2.

[0080] ROI-based restoration is a processing part for an image that is output from an Inner Decoder after being processed by considering the Region of Interest (ROI) area within the video. For example, it can restore an image in which the important area and background area within the image have been recombined to their original positions and resolutions. ROI-based restoration can correspond to the region of interest-based restoration part (224) of FIG. 2.

[0081] Spatial restoration can restore a spatially altered resolution to the resolution of the original video. Spatial restoration can correspond to the spatial restoration unit (226) of FIG. 2. In one embodiment, spatial restoration can use a fixed interpolation filter.

[0082] Temporal restoration can restore the resolution that has been changed over time to the resolution of the original video. Temporal restoration can correspond to the temporal restoration unit (228).

[0083] Bit depth restoration is a module that restores the value adjusted through bit truncation via shift left. Bit depth restoration may be included in the post-processing unit (220) and, although not disclosed in FIG. 2, may be represented as a bit depth restoration unit.

[0084] The present disclosure describes a method for improving encoding efficiency by generating luminance or chroma based on a linear model. In an encoder, only one of the chroma or luminance is encoded into an inner encoder, and the component not encoded into the inner encoder is generated through a linear model with the decoded component, and the linear model information can be encoded. For example, if only the luminance component is encoded into the inner encoder in an encoder, the chroma component, which is the component not encoded into the inner encoder, is generated through a linear model with the decoded luminance component, and the linear model information can be encoded. For example, if only the chroma component is encoded into the inner encoder in an encoder, the luminance component, which is the component not encoded into the inner encoder, is generated through a linear model with the decoded chroma component, and the linear model information can be encoded.

[0085] In the decoder, missing components can be generated from the image decoded through the inner decoder using linear model information.

[0086] The area to be encoded for chroma or luma only can be limited to a portion of the image (e.g., a region of interest or a background area).

[0087] In the following description, a case is described in which the encoder encodes both luminance and chroma in the region of interest within the image, encodes chroma excluding luminance in the background region, and encodes linear model information to be used for luminance generation together, so that the decoder generates luminance information using the chroma information and linear model information decoded for the background region. However, the present disclosure may not be limited thereto.

[0088] Figure 5 illustrates an example of generating luminance in a background area using chroma values ​​and a linear model.

[0089] Figure 6 is an enlarged view of the linear model of Figure 5.

[0090] In the image, the person and the ball can be separated into a region of interest, and the remaining area into a background region. Chroma Cb and chroma Cr are encoded in both the background region and the region of interest, while luminance can be encoded only in the region of interest. Luminance information for the background region can be encoded by applying a linear model to the chroma values ​​decoded at the same location to generate it. Figure 6 is an enlarged view of the linear model used in Figure 5, and 15 linear models are used. In the decoder, luminance can be generated through Equation 1 according to the linear model being signaled.

[0091] Luma(x,y) = w × Cb(x,y) + w × Cr(x,y) + b (Equation 1)

[0092] Here, x and y are pixel coordinate values ​​within the image, and w and b can be weighting factor and bias values ​​applied at the pixel level, block level (such as 8×8, 16×16, 32×32), or the entire image level. The weighting factors applied to Cb and Cr in Equation 1 may be the same or different. The linear model also does not need to be limited to 15 and can be reduced or increased depending on the situation.

[0093] In one embodiment, the w applied to Cb and the w applied to Cr may be the same. In this case, the parameter set may be defined as {w, b}.

[0094] In one embodiment, the w applied to Cb and the w applied to Cr may be different. In this case, for convenience of explanation, the weights may be expressed separately. As an example, Equation 1 is Luma(x,y) = w Cb × Cb(x,y) + w Cr It can also be expressed as × Cr(x,y) + b. In addition, the parameter set is {w Cb , w Cr It can be defined as { , b}.

[0095] In one embodiment, the luminance component for the background area may be initialized to a non-zero, pre-defined value. In Formula 1, the term for the initialized luminance component may be omitted by considering the luminance component for the background area as zero.

[0096] If the luminance component is initialized to a non-zero, pre-defined value, the generated luminance component can be expressed as in Equation 2.

[0097] Luma(x,y) = w L × Luma ini (x,y) + w Cb × Cb(x,y) + w Cr × Cr(x,y) + b (Equation 2)

[0098] In this case, the parameter set is {w L , w Cb , w Cr It can be defined as { , b}.

[0099] In one embodiment, Luma ini (x,y) can be a predefined integer.

[0100] In one embodiment, Luma ini (x,y) can be pre-defined as a value of 1, a midpoint of bit depth, a value representing gray, etc.

[0101] In one embodiment, Luma ini (x,y) may differ at the pixel level. Or, Luma ini (x,y) can have all pixels with the same value in block units such as 8×8, 16×16, 32×32, or in the entire image unit.

[0102] In one embodiment, Luma ini If is a fixed integer regardless of pixels, w L × Luma ini The part can be expressed by being included in b of Equation 1. Or, w L × Luma iniThe part can be expressed as separate, distinct terms as in Equation 2.

[0103] In one embodiment, when the parameters of a sharpness model (weights, bias values, etc.) are defined as a parameter set, the parameters of the sharpness model may be determined by one of the determined parameter sets.

[0104] Figure 7 illustrates an example of an image sub / decoder structure for a machine with a linear model-based luma generation module applied.

[0105] Figure 8 illustrates an example of an information map of the region of interest and a background region.

[0106] Figure 9 illustrates examples of a luminance image of a region of interest within the image and luminance and chroma (Cb, Cr) images of the entire image.

[0107] In the ROI-based Processing module of Fig. 7, the region of interest and the background region can be identified to generate information about the region of interest and a background region map as shown in Fig. 8. Additionally, based on this, region-specific luminance / chroma images can be generated, such as a luminance image of the region of interest within the image, and chroma (Cb, Cr) and luminance images of the entire image (region of interest and background region), as shown in Fig. 9. Among these, the luminance image of the region of interest and the chroma image of the entire image undergo bit depth truncation and are encoded through an inner encoder. The luminance image of the entire image is sent to an LM Index Derivation module, where it is compared with the luminance component generated using a linear model from the chroma of the background region within the image restored by the Inner Encoder to determine the linear model to be used for the corresponding block (16×16 pixels). The determined linear model can be encoded using methods such as arithmetic coding.

[0108] The region of interest information on the left side of Fig. 8 may represent multiple rectangular regions formed by pairs of top-left pixel coordinates and bottom-right pixel coordinates. In the background area map on the right side of Fig. 8, 0 represents the background area and 1 represents the region of interest.

[0109] Figure 10 illustrates examples of the linear model derivation process in the VCM encoder and the background region luma generation process in the VCM decoder.

[0110] Referring to Fig. 10, the LM Index Derivation module can output an LM index stream through the following processes.

[0111] Information regarding the region of interest and background region (ROI position, size, etc.) can be verified by passing through the encoder's ROI-based Processing module, and a background map can be generated in 16×16 pixel block units through the Background Classifier. In Patch Extraction, the chroma image restored after encoding in the inner encoder and the original luminance image are received as input, and processing can be performed in patch units (16×16 pixel blocks) at the same location.

[0112] For example, if the patch is in the Foreground (FG), since both the luminance and chroma are encoded in the inner encoder, there is no need to generate the luminance, so it can be bypassed. For example, if the patch is in the Background (BG), since the luminance component is not encoded in the inner encoder, a linear model to be used to generate the luminance component can be selected using the chroma patch restored after encoding in the inner encoder, and the information can be arithmetic encoded.

[0113] After the inner decoder in the VCM decoder, the Luma BG generation module can generate the Luma of the background area.

[0114] Similar to the encoder side, the same background map can be generated through a background classifier, and subsequently, patch extraction can perform processing at the same location patch unit (16×16 pixel block) on the restored chroma and lumina images. By identifying the region of interest and the background region, a linear model to be used only for the background region can be decoded and applied to the chroma component to generate the lumina component. Afterward, the lumina and chroma components of the region of interest and the background region can be combined to output the restored image.

[0115] Processing of candidate linear models may be required for linear model selection. To use a linear model such as Equation 1, values ​​for w and b may be necessary. To this end, encoding efficiency can be improved by defining and indexing a limited number of (w, b) combinations. These values ​​can be derived from pre-derived images or from the images to be encoded. When using pre-derived values, the w and b values ​​that minimize the difference between the original luminance image and the luminance image values ​​generated by Equation 1 within the unit (e.g., 16x16 pixel block) where the linear model selected via the VCM encoder of Fig. 7 is applied can be obtained using methods such as ridge regression. Subsequently, the (w, b) values ​​obtained by encoding the entire image are grouped into K (w, b) pairs using K-means clustering, and the (w, b) pair with the highest frequency in each cluster can be indexed and used. Although expressed only as (w, b) for the sake of convenience of explanation, (w, b) can be {w, b}, {w L , w Cb , w Cr , b}, {w Cb , w CrIt may include , b}.

[0116] Information regarding the linear model to be used and the indexed parameter set can be predefined or encoded at the unit of the image to be used and transmitted to the decoder. When deriving values ​​from the image to be encoded, the linear model can be defined in the manner described above by encoding the first frame or a portion of the image. Additionally, encoding efficiency can be improved by reducing the number of indices by excluding clusters with low frequency based on the frequency of occurrence of (w, b) pairs within a cluster. For example, 16 (w, b) pairs can be selected and linear model information can be encoded using 4 bits per 16x16 pixel block to generate a luma most similar to the original luma.

[0117] Meanwhile, to further reduce the amount of bits required to encode the selected linear model, the selected linear model can be referenced from blocks surrounding the pixel being encoded. For example, the index of the selected linear model can be referenced from the top-left block, top block, top-right block, left block that are adjacent to the block currently being encoded and have already been encoded, or from blocks encoded at the same location in the previous frame.

[0118] Specifically, encoding efficiency can be improved by performing arithmetic encoding while considering the probability of the index to be encoded in the current block based on the indices encoded in the top and left blocks.

[0119] FIG. 11 illustrates an example of linear model parameters.

[0120] A linear model can use the decoded luminance component to generate the final luminance component. That is, unlike the above embodiments, the luminance component of the background region is encoded in the encoder and transmitted to the decoder, and the final luminance component of the background region can be generated using the decoded luminance component and the linear model. Equation 3 may be an example of a linear model used to obtain the final luminance component.

[0121] Luma final (x,y) = w Y × Luma decoded (x,y) + w U × Cb(x,y) + w V × Cr(x,y) + b (Equation 3)

[0122] Here, w Y is the weight for the restored luma component, w U is the weight for the Cb component, w V represents the weight for the Cr component, and b can represent the bias value. Also, Luma decoded (x,y) is the Luma component recovered from the bitstream, Luma final (x,y) can represent the final luminance component. In this case, the parameter set is {w Y , w U , w V It can be defined as { , b}.

[0123] The parameter set can be determined in pixel units, block units such as 8×8, 16×16, 32×32, or entire image units. The units may be optionally determined or fixed. For example, the parameter set can be determined in fixed 16×16 block (patch) units.

[0124] The method for generating the final luminance component can be applied equally to the region of interest as well as the background region. For example, the final luminance component of the region of interest can be obtained by using the reconstructed luminance component of the region of interest and the chroma component of the region of interest.

[0125] In one embodiment, each patch is an input matrix It can be expressed as (bold R denotes the set of real numbers). Here, n can represent the number of pixels within Patch 2. For example, n can be 256. R i The heat is the decoded luma component , chroma-blue component , chroma-red ingredient , and bias vector It can be composed of. The chroma components may be upsampled to have the same resolution as the lumina component to generate the lumina component. In this case, the parameter set θ = [w Y , w U , w V , b] T ∈ It can be expressed as.

[0126] In a first embodiment, optimal parameter sets can be obtained by minimizing the L2-regularized cost (L2 regularized cost function) defined as Equation 4.

[0127] (Equation 4)

[0128] The first term of the above formula represents performing MSE (Mean Squared Error), and the second term is an L2 penalty that limits the size of the weight vector. It may be the case where a weight α is applied to it. In this case, the closed-form solution can be obtained through the regression method (Ridge Regression) as shown in Equation 5.

[0129] (Equation 5)

[0130] In a second embodiment, the optimal parameter sets are characterized by having similar patterns, and K-means clustering is performed on the parameter sets obtained in the first embodiment. It can be applied to. Here, the clustering operation is K c When expressed as (·), the centroid set is the result of clustering. can be obtained as in Equation 6. Here, C can represent the number of centroids.

[0131] (Equation 6)

[0132] In summary, the optimal parameter sets of the second embodiment can be obtained by applying K-means clustering to the parameter sets obtained in the first embodiment.

[0133] In a third embodiment, among the parameter sets of the centroid set obtained in the second embodiment, M parameter sets that are frequently used and have good compression efficiency relative to accuracy can be obtained to obtain optimal parameter sets (hereinafter, representative parameter sets). The representative parameter sets can be obtained by the following formulas (formulas 7 to 10).

[0134] First, the representative parameter sets Γ can be determined such that the average cost function of Equation 7 is minimized.

[0135] (Equation 7)

[0136] Here, can represent a cost function for selecting representative parameter sets T. The cost function can be calculated as shown in Equation 8 by combining the minimum distortion (D) and the transmission rate (R).

[0137]

[0138] (Equation 8)

[0139] Here, D is is a distortion term corresponding to the Mean Squared Error (MSE), and is the transmission rate term for the index, R' is the evaluated LM index bit, and β may be a weighting parameter that adjusts the balance between distortion and transmission rate. Also, can be the probability that a specific representative model will be selected.

[0140] Additionally, for each parameter set of the centroid set, the selection frequency f(θ) according to the best-fit assignment criteria in the training data is calculated, and after sorting the parameter sets according to the calculated selection frequency, the top M parameters are selected to obtain a temporary parameter set as in Equation 10.

[0141] (Equation 10)

[0142] In the temporary parameter set obtained from the above formula By adding, the final representative parameter set is You can obtain.

[0143] In one embodiment, the optimal parameter set may be performed in the same way in the encoder / decoder, or performed only in the encoder and index information specifying the optimal parameter set may be transmitted to the decoder.

[0144] The names of the syntax elements introduced in the embodiments described above are merely temporary for the purpose of describing the embodiments according to the present disclosure. Syntax elements may be named with names different from those proposed in the present disclosure.

[0145] The components described in the exemplary embodiments of the present disclosure may be implemented by hardware elements. For example, the hardware elements may include at least one of a digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element such as an FPGA, a GPU, other electronic devices, or a combination thereof. At least some of the functions or processes described in the exemplary embodiments of the present disclosure may be implemented in software, and the software may be recorded on a recording medium. The components, functions, and processes described in the exemplary embodiments may be implemented by a combination of hardware and software.

[0146] A method according to one embodiment of the present disclosure may be implemented as a program that can be executed by a computer, and said computer program may be recorded on various recording media such as magnetic storage media, optical reading media, digital storage media, etc.

[0147] The various technologies described in this disclosure may be implemented as digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The technologies may be implemented as computer program products, namely, computer programs tangibly implemented on information media or computer programs (e.g., machine-readable storage devices (e.g., computer-readable media) or data processing devices), or as computer programs implemented as signals processed by or propagated to perform operations of data processing devices (e.g., programmable processors, computers, or a plurality of computers).

[0148] Computer program(s) may be written in any form of programming language, including compiled or interpreted languages, and may be distributed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may be executed on a single computer, or by multiple computers distributed across one site or multiple sites and interconnected by a communication network.

[0149] Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and one or more processors of a digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. Components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Additionally, the computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be connected to said mass storage devices to receive and / or transmit data. Examples of information media suitable for implementing computer program instructions and data include semiconductor memory devices (magnetic media such as hard disks, floppy disks, and magnetic tapes), optical media such as compact disc read-only memory (CD-ROM) and digital video discs (DVD), magneto-optical media such as floptical disks, and Read Only Memory (ROM), Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), and other known computer-readable media. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0150] A processor may execute an operating system (OS) and one or more software applications running on the OS. A processor unit may also access, store, manipulate, process, and generate data in response to software execution. For simplification, a processor unit is described in the singular; however, those skilled in the art will understand that the processor unit may include multiple processing elements and / or various types of processing elements. For example, a processor unit may include multiple processors or a processor and a controller. It may also constitute different processing structures, such as parallel processors. Furthermore, a computer-readable medium means any medium accessible to a computer and may include both computer storage media and transmission media.

[0151] The present disclosure includes detailed descriptions of various detailed embodiments, but such details are not intended to limit the invention or claims proposed in the present disclosure and should be understood as describing the features of specific exemplary embodiments.

[0152] Features individually described in exemplary embodiments in this disclosure may be implemented by a single exemplary embodiment. Conversely, various features described with respect to a single exemplary embodiment in this disclosure may be implemented by a combination of multiple exemplary embodiments or a suitable sub-combination. Furthermore, in this disclosure, said features may operate by a specific combination and may be described as said combination first claimed, but in some cases, one or more features may be excluded from the claimed combination, or the claimed combination may be changed into a sub-combination or a modified form of a sub-combination.

[0153] Likewise, even if operations are described in a specific order in the drawings, it should not be understood that it is necessary to execute the operations in a specific sequence or order, or that all operations must be performed, in order to obtain the desired result. In certain cases, multitasking and parallel processing may be useful. Furthermore, it should not be understood that the various device components in the exemplary embodiments of all embodiments must be separated, and the aforementioned program components and devices may be packaged into a single software product or multiple software products.

[0154] The exemplary embodiments disclosed in this specification are merely illustrative and are not intended to limit the scope of this disclosure. Those skilled in the art will recognize that various modifications to the exemplary embodiments may be made without departing from the spirit and scope of the claims and their equivalents.

[0155] Accordingly, the present disclosure shall be deemed to include all other substitutions, modifications, and changes falling within the scope of the following claims.

Claims

Claim 1 A method for decoding an image based on a region of interest, characterized by comprising: a step of obtaining a background region map by distinguishing a region of interest and a background region within an image; a step of obtaining a background region patch and a region of interest patch based on the background region map; a step of obtaining a linear model applied to the background region patch; and a step of restoring the luminance component of the background region using the chroma component of the background region based on the linear model. Claim 2 A method for decoding images based on a region of interest, wherein, in claim 1, the background region and the region of interest patch are block units comprising at least one pixel. Claim 3 A region of interest-based image decoding method according to claim 2, characterized in that the block size of the block unit is 16×16. Claim 4 A region of interest-based image decoding method according to claim 1, wherein the linear model is determined by a set of parameters including weights and bias values. Claim 5 A region of interest-based image decoding method according to claim 4, wherein the weights include weights applied to the chroma-blue component and weights applied to the chroma-red component. Claim 6 A region of interest-based image decoding method according to claim 4, wherein the parameter set is determined by an index signaled from a bitstream specifying the parameter set among a plurality of parameter sets. Claim 7 In claim 6, the method for obtaining the plurality of parameter sets comprises: obtaining a first plurality of parameter sets using some combinations among all combinations of the weights and bias values ​​that minimize the difference between the original luminous image and the generated luminous image values; clustering the first plurality of parameter sets to obtain a second plurality of parameter sets; and obtaining the plurality of parameter sets using some parameter sets with a high selection frequency among the second plurality of parameter sets, characterized in that the method is an interest region-based image decoding method. Claim 8 A method for region-based image encoding comprising: a step of obtaining a background region map by distinguishing a region of interest and a background region within an image; a step of obtaining a background region patch and a region of interest patch based on the background region map; and a step of obtaining a linear model applied to the background region patch, wherein the linear model is a model that restores the luminance component of the background region using the chroma component of the background region. Claim 9 An interest region-based image encoding method according to claim 8, wherein the background region and the interest region patch are block units comprising at least one pixel. Claim 10 A region of interest-based image encoding method according to claim 9, characterized in that the block size of the block unit is 16×16. Claim 11 A region of interest-based image encoding method according to claim 8, wherein the linear model is determined by a set of parameters including weights and bias values. Claim 12 A region-of-interest-based image encoding method according to claim 11, wherein the weights include weights applied to the chroma-blue component and weights applied to the chroma-red component. Claim 13 An interest region-based image encoding method according to claim 11, characterized in that an index specifying a parameter set among a plurality of parameter sets is encoded in a bitstream. Claim 14 In claim 13, the method for obtaining the plurality of parameter sets comprises: obtaining a first plurality of parameter sets using some combinations among all combinations of the weights and bias values ​​that minimize the difference between the original luminance image and the generated luminance image values; clustering the first plurality of parameter sets to obtain a second plurality of parameter sets; and obtaining the plurality of parameter sets using some parameter sets with a high selection frequency among the second plurality of parameter sets, characterized in that the method is an interest region-based image encoding method. Claim 15 A non-transient computer-readable recording medium for storing a bitstream generated by a region of interest-based image encoding method, wherein the image encoding method comprises the steps of: distinguishing a region of interest and a background region within an image to obtain a background region map; obtaining a background region patch and a region of interest patch based on the background region map; and obtaining a linear model applied to the background region patch, wherein the linear model is a model that restores the luminance component of the background region using the chroma component of the background region.

Images