Iterative Graph-Based Image Enhancement with Object Segmentation

Abstract

A system and method for enhancing an image using graph-based inter-object and intra-object separation. One method includes receiving an object in an image frame, the object comprising a plurality of pixels, performing an inter-object point cloud separation operation on the image, and expanding the plurality of pixels of the object. The method includes performing a spatial enhancement operation on the plurality of pixels of the object, and generating an output image based on the inter-object point cloud separation operation, the expanding of the plurality of pixels, and the spatial enhancement operation.

Description

[Technical field]

[0001] 1. CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Patent Application No. 63 / 285,570 and European Patent Application No. 21212271.7, both filed on December 03, 2021, each of which is incorporated by reference in its entirety.

[0002] 2. Field of disclosure This application relates generally to a system and method for enhancing images using graph-based inter-object and intra-object separation. [Background technology]

[0003] 3. Background Non-Patent Document 1 discloses an object-based multilevel contrast stretching method for enhancing image structure. The purpose of image enhancement is to improve the perceptibility of information contained in an image. Since the human visual system has a tendency to extract image structures, enhancing structural features can improve the perceived image quality. The method first uses morphological watersheds and region merging to segment an image into its constituent objects, which are treated as image structural components, and then stretches image contrast at inter-object and intra-object levels separately in different ways. At the inter-object level, a stretching technique between adjacent extremes is used to fully expand the local dynamic range of gray levels between objects. At the intra-object level, a uniform linear stretch is used to enhance the texture features of objects while maintaining their uniformity. Since the method operates directly on objects, it can avoid introducing ringing, blocking or other false contour artifacts in the structural appearance, and furthermore, it can effectively suppress overemphasis of noise and roughly preserve the overall brightness of the image. Experimental results show that this method can generate enhanced images with more natural appearance compared to some classical methods. [Non-Patent Document 1] Beilei Xu et al., "Object-based multilevel contrast stretching method for image enhancement", IEEE Transactions on Consumer Electronics, IEEE Service Center, New York, USA, vol. 56, no. 3, 1 August 2010, p.1746-1754, XP011320092

[0004] US 2007 / 0133093 A1 discloses an apparatus for performing color enhancement of an image having a segmenter that generates image segments, which may be relatively small. An analyzer identifies neighboring segments for a first segment, and a color enhancer applies a color enhancement algorithm to the first segment. An adjuster is configured to adjust characteristics of the color enhancement algorithm for the first segment depending on relative geometric characteristics of a resulting group of color points in color space and a neighboring group of color points in color space. The resulting group of color points includes color points for at least some color-enhanced pixels of the first segment. The neighboring group of color points includes color points of at least some pixels of the at least one neighboring segment. Segmentation-based color enhancement taking into account color characteristics between segments can provide improved image quality. [Patent Document 1] International Publication No. 2011 / 141853A1

[0005] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to "scene-referred" intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular width. In this sense, DR relates to "display-referred" intensity. Unless a particular meaning is explicitly specified as having special importance at any point in the description of this specification, it should be presumed that the term may be used in either sense, e.g., interchangeably.

[0006] As used herein, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR that humans can simultaneously perceive a wide range of intensity ranges may be somewhat truncated for HDR. As used herein, the terms extended dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably refer to the DR perceivable in a scene or image by the human visual system (HVS), including eye movements, taking into account some light adaptation changes across the scene or image.

[0007] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), with each color component represented with n bits of precision per pixel (e.g., n=8). Using linear luminance encoding, images with n<8 may be considered standard dynamic range images, and images with n>8 (e.g., color 24-bit JPEG images) may be considered extended dynamic range images. EDR and HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0008] As used herein, the term "metadata" refers to any auxiliary information transmitted as part of an encoded bitstream that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0009] Most consumer desktop displays today have a resolution of 200-300cd / m 2 or nits of luminance. Most consumer HDTVs are in the 300-500 nits range, with newer models reaching 1000 nits (cd / m 2). Thus, such conventional displays are representative of low dynamic range (LDR), also called standard dynamic range (SDR), in relation to HDR or EDR. As the availability of HDR content increases due to advances in both capture devices (e.g., cameras) and HDR displays (e.g., the PRM-4200 Professional Reference Monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support a higher dynamic range (e.g., 1000 nits to 5000 nits or more).

[0010] Early methods of digital image enhancement enhanced the entire image using global contrast adjustment using histogram equalization, color correction using color balancing techniques, or a combination of both. Advanced image enhancement techniques use information about a pixel's surroundings to locally enhance the image using local contrast enhancement, local tone mapping, image sharpening, and bilateral filtering. In addition, image segmentation methods have provided precise identification of objects within an image. Summary of the Invention [Means for solving the problem]

[0011] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention. The embodiments provided herein utilize segmentation information to improve the visual appeal of an image. For example, segmentation information can be used to (i) enhance an object independently and (ii) enhance an object relative to other objects in its vicinity. This object-specific enhancement increases the visual quality of the image by improving the contrast within and between objects.

[0012] Although the proposed method can improve any kind of image, further benefits may be found in enhancing the subjective quality of HDR images displayed on mobile screens. HDR images have a higher luminance range than traditional standard dynamic range (SDR) images. This increased luminance range allows HDR images to effectively represent details in dark and bright areas without clipping the dark areas or oversaturating the bright areas. Furthermore, HDR images have a wider color representation compared to SDR images. Due to the size of mobile screens, these advantages of HDR images are often limited. The embodiments described herein utilize knowledge of objects in the image to visually enhance HDR images for mobile screens.

[0013] Various aspects of the present disclosure relate to devices, systems, and methods for enhancing images using graph-based inter-object and intra-object separation. Although certain embodiments are directed to HDR video data, the video data may also include standard dynamic range (SDR) video data, and other user generated content (UGC), such as gaming content.

[0014] In one exemplary aspect of the present disclosure, a video delivery system for iterative graph-based image enhancement of image frames is provided. The video delivery system includes a processor configured to perform processing of the image frames. The processor is configured to receive an object in the image frames, the object including a plurality of pixels. The processor is configured to perform an inter-object point cloud separation operation on the image, dilate the plurality of pixels of the object, and perform a spatial enhancement operation on the plurality of pixels of the object. The processor is configured to generate an output image based on the inter-object point cloud separation operation, the dilate the plurality of pixels, and the spatial enhancement operation.

[0015] In another exemplary aspect of the present disclosure, an iterative method for image enhancement of an image frame is provided. The method includes receiving an object in the image frame, the object including a plurality of pixels. The method includes performing an inter-object point cloud separation operation on the image, expanding the plurality of pixels of the object, and performing a spatial enhancement operation on the plurality of pixels of the object. The method includes generating an output image based on the inter-object point cloud separation operation, the expanding the plurality of pixels, and the spatial enhancement operation.

[0016] In another example aspect of the present disclosure, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by a processor of the video delivery system, cause the video delivery system to perform operations including receiving an object comprising a plurality of pixels in an image frame, performing an inter-object point cloud separation operation on the image, enlarging a plurality of pixels of the object, performing a spatial enhancement operation on a plurality of pixels of the object, and generating an output image based on the inter-object point cloud separation operation, the enlarging of the plurality of pixels, and the spatial enhancement operation.

[0017] In this manner, various aspects of the present disclosure provide provisions for the display of images having high dynamic range and either high or standard resolution, leading to improvements in at least the fields of image projection, holography, signal processing, and the like. [Brief description of the drawings]

[0018] These and other more detailed and specific features of the various embodiments are more fully disclosed in the following description, which refers to the accompanying drawings.

[0019] [Figure 1] 1 illustrates an example process for a video delivery pipeline.

[0020] [Figure 2A] 1 shows an exemplary HDR image.

[0021] [Figure 2B] Figure 2A shows an example HDR image after the image segmentation process.

[0022] [Diagram 3] An example pixel level in a BFS search is shown.

[0023] [Figure 4] AB show exemplary image-to-graph structure relationships.

[0024] [Figure 5A] 4B illustrates an exemplary method for using a priority queue to process the graph structure of FIG. [Figure 5B] 4B illustrates an exemplary method for using a priority queue to process the graph structure of FIG. [Figure 5C] 4B illustrates an exemplary method for using a priority queue to process the graph structure of FIG. [Figure 5D] 4B illustrates an exemplary method for using a priority queue to process the graph structure of FIG. [Figure 5E] 4B illustrates an exemplary method for using a priority queue to process the graph structure of FIG.

[0025] [Figure 6] 1 illustrates an exemplary method for enhancing an image.

[0026] [Figure 7] 1 illustrates an example graph illustrating an object enhancement process.

[0027] [Figure 8] 1 shows an example graph illustrating multiple sigmoid curves.

[0028] [Figure 9A] 1 illustrates an exemplary inter-object point cloud separation process. [Figure 9B]1 illustrates an exemplary inter-object point cloud separation process. [Figure 9C] 1 illustrates an exemplary inter-object point cloud separation process.

[0029] [Figure 10] 9A-9C show example luminance-saturation graphs of objects before and after the inter-object point cloud separation process.

[0030] [Figure 11] 1 illustrates an exemplary method for inter-object point cloud separation process.

[0031] [Figure 12] 4 shows an example luminance-saturation graph of an object before and after an intra-object point cloud growth process.

[0032] [Figure 13] 4 shows exemplary average luminance and luminance-saturation graphs of objects in an image undergoing inter-object point cloud separation and intra-object point cloud growth processes.

[0033] [Figure 14] 1 illustrates an exemplary method for an inside-object point cloud growth process.

[0034] [Figure 15A] 1 shows an example segmented luminance square of an object undergoing an intra-object spatial enhancement process. [Figure 15B] 1 shows an example segmented luminance square of an object undergoing an intra-object spatial enhancement process. [Figure 15C] 1 shows an example segmented luminance square of an object undergoing an intra-object spatial enhancement process. [Figure 15D]1 shows an example segmented luminance square of an object undergoing an intra-object spatial enhancement process. [Figure 15E] 1 shows an example segmented luminance square of an object undergoing an intra-object spatial enhancement process.

[0035] [Figure 16] 1 illustrates an exemplary method for an intra-object spatial enhancement process.

[0036] [Figure 17] 1 illustrates an exemplary method for image quality assessment.

[0037] [Figure 18] 1 illustrates an exemplary method for an iterative image enhancement process.

[0038] [Figure 19] 19A-B show example graphs identifying stopping criteria for the iterative image enhancement process of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is in no way intended to limit the scope of the present disclosure.

[0040] In the following description, numerous details are set forth, such as optical device configurations, timing, operation, etc., to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely examples and are not intended to limit the scope of the present application.

[0041] Additionally, while this disclosure will primarily focus on examples in which the various circuits are used in digital projection systems, it will be understood that these are merely examples. The disclosed systems and methods may be implemented in display devices having OLED displays, LCD displays, quantum dot displays, etc. It will be further understood that the disclosed systems and methods may be used in any device that needs to project light, such as cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, etc.

[0042] Video encoding of HDR signals FIG. 1 illustrates an exemplary process of a video delivery pipeline (100) showing various stages from video capture to video content display. A sequence of video frames (102) are captured or generated using an image generation block (105). The video frames (102) may be captured digitally (e.g., by a digital camera) or computer generated (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide the video data (107). In a production phase (110), the video data (107) is edited to provide a video production stream (112).

[0043] The video data of the production stream (112) is then provided to a processor (or one or more processors, such as a central processing unit (CPU)) in block (115) for post-production editing. The post-production editing in block (115) may include adjusting or modifying the color or brightness in certain areas of the image to improve image quality or achieve a particular look for the image according to the video creator's creative intent. This is sometimes called "color timing" or "color grading". Other editing (e.g., scene selection and ordering, image cropping, adding computer-generated visual special effects, etc.) may be performed in block (115) to provide a final version (117) of the production for distribution. During post-production editing (115), the video images are viewed on a reference display (125).

[0044] Following post-production (115), the video data of the final production (117) may be delivered to an encoding block (120) for delivery to downstream decoding and playback devices such as television sets, set-top boxes, movie theaters, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate an encoded bitstream (122). The methods described herein may be performed by a processor in the block (120). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents the same or a close approximation of the signal (117). The receiver may be attached to a target display (140), which may have completely different characteristics than the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display-mapped signal (137). Additional methods described herein may be performed by the decode unit (130) or the display management block (135). Both the decode unit (130) and the display management block (135) may include their own processors or may be integrated into a single processing unit.

[0045] Image segmentation maps and preprocessing operations To process individual objects within an image, the locations of the pixels that make up that particular object are known. This information is stored in a segmentation map. Using the image segmentation map, individual objects are extracted from the image and used to generate a graph that characterizes the objects in the image. This graph provides the iterative process with structural information about which object to visit first and how to process each object within the image.

[0046] At the local level, individual pixels in an image are used to characterize objects. At the global level, an image is characterized by the objects that compose it. The ability to process images at the object level allows for balanced global and local image enhancement. This balance is achieved by processing objects to stand out from their neighbors (global) for improved visual quality, while also enhancing the interiors of these objects (local).

[0047] To perform processing at the object level, knowledge of which pixels belong to each object in the image is provided as a segmentation map. H×W For each image, the segmentation map associated with the image L∈N H×W where each integer pixel value represents a label corresponding to the object to which it belongs. Sample image-segmentation map pairs are provided in FIG. 2A and B. Image (200) is the original image and segmentation map (250) is its corresponding segmentation map. In some embodiments, image (200) is the original HDR image. Segmentation map (250) can include labels overlaid on image (200). Segmentation map (250) can be created by "cutting out" object boundaries using rectangles, polylines, polygons, and pixel regions of interest (ROIs). For example, let L represent the number of distinct categories of objects in the image. Then, all pixels in the output segmentation map are

number

[0048] In some embodiments, the generated segmentation map fails to assign pixels along boundaries between objects or along image boundaries. Such problems may be corrected by reassigning the value of each zero pixel (e.g., unlabeled pixel) in the image segmentation map with a value equal to its closest non-zero pixel value. In some implementations, a breadth first search (BFS) is employed to find the closest non-zero pixel value to a particular zero-valued pixel. BFS is a tree data structure search algorithm in which each level of the tree is fully explored before moving to the next level of the tree. Each tree level represents the pixels that directly bound the previous layer of the tree constructed up to that point. The "root" of the tree is the first level explored and is given by the unlabeled pixel to which the algorithm has assigned a label. The second level of the tree consists of the eight pixels surrounding the unlabeled pixel. The third level of the tree consists of the 16 pixels surrounding the second level, and so on. The tree generation and exploration is performed until the first labeled pixel is encountered. Any unlabeled pixel to which the algorithm is attempting to assign a label is given this label. In Figure 3, the root of the tree is provided as an unlabeled pixel (300). The unlabeled pixel (300) is surrounded by a second level (302). The second level (302) is surrounded by a third level (304).

[0049] The levels of the tree are built and explored using a queue. Each time an unlabeled pixel is encountered during the search, the eight surrounding pixels are checked in clockwise order. In some implementations, the top left pixel is checked first. The neighboring pixels surrounding this pixel that have not been explored are added to the queue. In FIG. 3, the unlabeled pixel (300) has eight neighboring pixels, not all of which have been checked in the queue. Thus, all pixels of the second level (302) are added to the queue, starting with the top left pixel (303) in the top left corner. When the top left pixel (303) is pulled from the queue to be processed, it is first checked to see if it has a label. If the top left pixel (303) has a label, the BFS process ends and the unlabeled pixel (300) is assigned this label. If not, all of the neighboring pixels of the top left pixel (303) that have not been added to the queue are added. In this case, the top left pixel (303) has eight neighbors, but three have been checked into the queue, so only the five pixels in the third level (304) that border the top left pixel (303) are added to the queue. These pixels are added to the end of the queue, so they will not be explored until the rest of the second level (302) has been explored.

[0050] As one specific example, the following pseudocode provides an example for labeling unlabeled pixels. In this pseudocode, GetUnvisitedNeighbors returns a list of unvisited neighboring pixels for the current pixel. QueueFrontElement, QueuePop, and QueueInsert are standard queuing operations. This process is performed for every unlabeled pixel in the original segmentation map. When finished, all pixels in the new segmentation map are assigned labels from 1,…,L. By putting this code in a loop, all pixels inside the segmentation map are found. [Table 1]

[0051] Hierarchical Graph Generation As mentioned above, an image may be composed of several objects. Enhancing certain objects results in a greater improvement in image quality than enhancing other less important objects. Objects that have a greater impact on the overall image quality are more important than objects that have a lesser impact on the overall image quality. Thus, in some implementations, objects are ordered and processed based on their importance. The methods described herein capture this hierarchical structure of object importance using a graph structure described below with respect to Figs. 4A-B. Larger, more important objects are located closer to the "root node" of the graph structure and are processed earlier, allowing more freedom in decision making. Smaller objects are located further away from the root node and are processed later. These constraints enhance objects while maintaining the relative luminance and saturation levels between objects. For example, a dark object surrounded by bright objects should not become brighter than its neighbors during enhancement.

[0052] FIG. 4A provides an image (400) consisting of multiple objects, such as object A, object B, object C, object D, object E, object F, and object G. A graph structure (420) includes multiple nodes, such as node A, node B, node C, node D, node E, node F, and node G. Each node represents a respective object. For example, node A represents object A, node B represents object B, etc. Each node is connected by an edge. If two nodes share an edge in the graph structure (420), then the respective objects in the image (400) share a boundary (or border) with each other. The graph structure (420) may be directed. As shown in the graph structure (440) of FIG. 4B, the direction of the edges in the graph conveys the importance of the nodes. Specifically, consider a graph G defined on the pair (V,E). Here, V is the set of all nodes, and E is the set of all directed edges between vertices, E⊆{(u,v)|u,v∈V}, where the edge direction of the edge is from u to v. Then, the importance of the object associated with node u is greater than v. The importance of the object is determined by a metric that may be user-defined. In the methods described herein, the size of the object is used to determine the importance. However, other characteristics of the object may be used to determine the importance. For example, the category of the object may be used to determine the importance. In such an example, a human may have greater importance in an image than a wall or a floor, regardless of the size of the background wall. In other embodiments, the connectivity of a node or object with neighboring nodes or objects may also be used to determine the importance. For example, a node with a larger number of connected edges may have a higher importance than a node or object with a smaller number of connected edges. In some embodiments, the importance is based on a combination of the described object characteristics.

[0053] In the example of Figure 4A-B, object A is the largest object in the image (400). The larger object has an arrow pointing out towards the smaller object with which it shares a border. Thus, the arrow points from object A to both object B and object C.

[0054] As one particular example, the following pseudocode provides an example for generating a graph data structure, such as the graph structure (440): An object boundary map is generated from the segmentation map. The boundary pixels are on the boundary between two objects. When each of these boundary pixel locations is visited, their neighbors are checked in the object segmentation map to determine connected object pairs. The function CheckIfDifferentFromNeighborPixels in the pseudocode finds connected object pairs from the object boundary map and the segmentation map. Two edges are stored between these two objects (representing both directions) in the adjacency matrix. This generates an undirected adjacency matrix or an undirected graph. The adjacency matrix is ​​then made directed by going through each edge and checking which node connected to that edge is larger based on a size metric. The CheckLargerMetricValueBetweenNodes function in the pseudocode below finds the smaller or less important nodes. Edges directed from smaller nodes to larger nodes in the adjacency matrix are then removed. The result is a directed adjacency matrix or graph. In some implementations, if objects share boundary pixels, their nodes are connected by edges. In other implementations, some boundary pixels between objects are used instead to avoid weakly connected objects. [Table 2]

[0055] In some implementations, the decision-making process of the order in which objects are processed is implemented using a priority queue. In the examples described herein, such a priority queue is based on both the size of the object and the relationship between the object and neighboring objects. The largest object in the image is placed at the top of the queue for priority. Once the largest object is processed, the associated node is removed from the graph. The nodes that share a boundary with the node associated with the largest object are then examined to determine whether they have any remaining ancestors. If any nodes do not have any remaining ancestors, the corresponding object is larger than the remaining unprocessed nodes that they border. Thus, these nodes are now positioned according to size and placed in a priority queue. In this way, larger objects are processed first. All objects processed in later iterations internalize the improvements of their ancestors.

[0056] As one particular example, the following pseudocode provides an example for using a priority queue: In the pseudocode, the GetPredecessorsOfCurrentNode function gets the ancestors or parents of the current node, while the GetSucessorsOfNode function gets the children or descendants of the current node. The AddNodeToQueue function adds a node to the priority queue based on its importance. The RemoveFirstNodeInQueue function gets the first node in the priority queue. [Table 3]

[0057] In some implementations, attributes used to process an object are stored within its corresponding node. This provides for precomputing many quantities used during processing. Such information includes the size of the object, a list of the object's ancestors, the average of the pixel intensities that form the object, an object label, an object luminance histogram, and an object saturation histogram. The list of the object's ancestors includes a list of all nodes that share edges with the current node and the direction of the edge towards the current node. The object label includes the segmentation map label that corresponds to the current node. The object luminance histogram describes the luminance of the object. The object saturation histogram describes the saturation of the object.

[0058] 5A-5E show an exemplary process for traversing the priority queue. Nodes A-G shown in FIGS. 5A-5E correspond to nodes A-G shown in FIGS. 4A-4B. Nodes A-G in FIGS. 5A-5E form a graph structure (500). The graph structure (500) may correspond to graph structure (440) in FIG. 4B. The priority queue (520) indicates the priority of nodes A-G for processing. The priority queue (520) also includes parentheses next to the node label indicating the number of parents for that node. The priority queue (520) is ordered according to the number of parents. When an object is processed, the corresponding node is removed from the priority queue (520). The parent numbers for all its children are decremented by one. The priority queue is then sorted by the updated parent numbers.

[0059] In Figure 5A, node A has no parent. Node A is processed and removed from the priority queue (520).

[0060] In Figure 5B, the priority queue (520) is updated because node A has been removed. Node C has no remaining parents and is processed and removed from the priority queue (520).

[0061] In Figure 5C, since node C has been removed, the priority queue (520) is updated and reordered. Both nodes B and G have no remaining parents and can be processed. Since object B is larger than object G, node B is processed first and is removed from the priority queue (520).

[0062] In Figure 5D, since node B has been removed, the priority queue (520) is updated and reordered. Both nodes G and D have no remaining parents and can be processed. Since object G is larger than object D, node G is processed first and is removed from the priority queue (520).

[0063] In Figure 5C, node G has been removed, so the priority queue is updated and reordered. Nodes D, E, and F all have no remaining parents and may be processed. Object F is processed first because it is the largest remaining object. Object E is larger than object D, so node E is processed next. Object D is processed last.

[0064] Following construction of the graph structure (500), each node is visited and each object is processed using a three-stage enhancement process, which is described in more detail below. After each node in the graph structure (500) has been processed once, the image quality is checked and if a stopping criterion has not been met, a second round of enhancement is initiated. This process continues until convergence.

[0065] Iterative improvement process The conversion of an SDR image to an HDR image, or the incremental enhancement of an existing HDR image, can be achieved at the object level using a graph-based iterative process. The iterative approach makes incremental changes, avoiding clipping in highlights and loss of detail in low-luminance areas that can be experienced with known upconversion methods.

[0066] FIG. 6 illustrates a method (600) for an iterative image enhancement process. The method (600) may be performed by a central processing unit in block (115) for post-production editing. In some embodiments, the method (600) is performed by an encoder in the encoding block (120). In other embodiments, the method (600) is performed by a decoding unit (130), a display management block (135), or a combination thereof. In block (602), the method (600) includes receiving an input image, such as an SDR image or an HDR image. The input image may be received in the RGB color domain or the YCbCr color domain. In some implementations, when the input image is in the RGB color domain, the method (600) includes converting the input image to the YCbCr color domain.

[0067] At block (604), the method (600) includes constructing a graph structure based on the input image, such as the graph structure (440), as described above. In some implementations, the graph structure is constructed using a segmentation map associated with the input image. At block (606), the method (600) includes assessing the quality of the input image using a global image quality metric (GIQM).

[0068] At block (608), the method (600) includes selecting a next node in the graph hierarchy. For example, the next node in the priority queue (520) is selected. At process (650), the method (600) includes performing an iterative enhancement process. This process is performed for each object in the graph structure. The processing is performed in three main modules: (1) point cloud separation at block (610), (2) point cloud expansion at block (612), and (3) spatial enhancement at block (614). Point cloud separation is for inter-object enhancement, and point cloud expansion and spatial enhancement are for intra-object enhancement. At block (616), the method includes using a lightweight local image quality metric (LIQM) to monitor the quality of the enhanced objects as the iterative process progresses.

[0069] When the last object in the image has been processed, in block (618), the method (600) includes evaluating the GIQM to evaluate the overall objective quality of the image after the changes made in the current iteration. If the image quality has improved, a new iteration is started to process the image output from the current iteration (block 620). If a stopping criterion is met in the current iteration in block (620), the method (600) ends the sequential iteration process, and in block (622), the highest quality enhanced image up to that iteration is output as the HDR image. In some implementations, the method (600) includes converting the HDR image to the RGB domain.

[0070] Returning to the iterative enhancement process described by process (650), FIG. 7 illustrates an exemplary point cloud separation process (700), an exemplary point cloud expansion process (720), and an exemplary spatial enhancement process (740). In each process, the y-axis represents luminance and the x-axis represents saturation. Point cloud separation is a form of inter-object enhancement where processing occurs between objects, rather than within each object. Point cloud separation separates the luminance-saturation point clouds for objects from each other, thereby making the objects more distinct from each other in the final image. Point cloud expansion and spatial enhancement are forms of intra-object enhancement and are performed to enhance pixels within a particular object. Point cloud expansion increases the spread of pixels within a particular object, thereby making the intra-object pixel values ​​stand out and cover a larger luminance-saturation range. Spatial enhancement increases the detail of each object, thereby making high frequency information such as edges and textures appear more "crisp" and easily discernible by viewers of the image.

[0071] The three-stage enhancement process of process (650) relies on the characteristics of an object and its relationships with its neighboring objects (obtained from the graph structure) to decide on the enhancement of the current object. The order in which the neighboring objects are processed affects the enhancement performed on the current object. These complex interactions between objects are difficult to capture in a one-shot optimization framework. To resolve these complexities, the iterative enhancement process makes incremental changes to the objects in the image at each iteration. As all objects are incrementally advanced at each iteration, the algorithm obtains the latest information from the neighboring objects and makes an informed decision on enhancing the current object by balancing between inter-object and intra-object enhancement techniques.

[0072] The notation used in the iterative process is explained as follows: RGB ∈RH×W×3 Let I be an RGB image of size H×W. YCbCr ∈R H×W×3 Define Y, C to be the same image after transformation to the YCbCr domain. b , and C r I YCbCr where Y is the luminance component of the original image, and S is the chrominance component, S=√(C b 2 +C r 2 ) Suppose there are N objects in an image, object1, ..., objectN, and without loss of generality, we assume that they are processed in ascending order during the iterative process. Y k and S k Let be the array of luminance and chrominance components of the kth object. The lth pixel value within the luminance component of object k is p k,l Furthermore, m k =mean(Y k Let m denote the average luminance of the kth object. In a graph structure, objects have descendants / children (nodes they point to) and ancestors / parents (nodes that point towards them). Ak and m Dk Define r and r to be the luminance averages of the ancestors and descendants of object k, respectively. The current iteration step for a particular value is denoted using a superscript. For example, r (i) is the learning rate used in the ith iteration.

[0073] Certain values ​​change between modules in the iteration process. These are Y k , S k , p k,l , m k , m Ak , and m DkTo distinguish these values ​​between modules, dots are placed on them. A single dot on a value indicates that point cloud separation has been applied; a double dot on a value indicates that point cloud expansion has been applied; a triple dot on a value indicates that spatial enhancement has been applied; no dot on a value indicates the completion of the entire iteration process. For example,

number

[0074] For ease of reference, an expanded list of this notation is given in Table 1. [Table 4] Table 1: Notation used in the iterative process

[0075] Inter-object point cloud separation Phone displays are much smaller than television screens and computer monitors. Performing detail enhancement on an image can lead to improved visual quality on larger displays, but these benefits are not always fully realized on phone screens, as the small size inhibits enhancement. The use of inter-object contrast results in perceptible visual quality differences on smaller screens, and is achieved via point cloud separation in the YCbCr color domain (where luminance and saturation are easily adjusted).

[0076] Using the notation above, the luminance of the kth object, after point cloud separation in iteration i, can be calculated as:

number

number

[0077] In Equation 1, r1 represents a learning rate, which may be adjusted at each iteration to ensure that the value of the image quality metric (GIQM) is improving at each iteration. In the embodiments provided herein, the image is (0) = 0.2. Symbol Δ k (i) is the total luminance displacement for the kth object before applying the learning rate. Δ k (i) is expressed by the following formula:

number

[0078] In Equation 2, ψ b is a modulated sigmoid curve used to shift the input mean of the current object to a new mean.

number

number

[0079] Equation 3 defines a sigmoid curve centered along a 45° line that passes through the origin. The left tail of the curve passes through the point (0,0) and the right tail passes through the point (2 β-1 ,2 β-1) where β is a constant that specifies the bit depth of the image, which is 16 for HDR images. The parameter d is used to center the sigmoid curve along a 45° line. The parameter x is a value between 0 and 2 that determines the bit depth at which the sigmoid curve will generate the output. β-1 is used to specify the input pixel value of . The parameter b is a hyperparameter that modifies the slope of the sigmoid curve as it crosses the 45° line. In the example provided herein, b is set to -2. The slope of this curve is higher in the mid-tones and tapers gradually towards the light or dark intensity regions. Figure 8 shows several of these sigmoid curves centered at various points on the 45° line.

[0080] In formula 2,

number

number

number

number

[0081] When an object is associated with a root node in a graph structure, as mentioned before, the object has no ancestors. Nevertheless, it is important that the shift applied to it increases its separation from other objects around it, making it stand out in the image. Thus, to perform the operation on object 1 (or object A in Figure 4A), which is the root node:

number

[0082] A sigmoid curve is incorporated into the point cloud separation operation because it shifts objects that are closer in luminance further apart. The sigmoid curve function ramps down as you approach a sufficiently large distance from the center. Thus, objects that already have sufficient contrast with their neighbors are shifted less. Thus, nearby objects with less contrast are pushed further apart from each other, and objects that are far enough apart are not pushed apart as much.

[0083] Treating saturation independently of luminance can result in unnatural, overly saturated images. When the luminance of an object is increased to become brighter, the object may begin to appear "washed out." Conversely, when a bright object becomes darker, the brightness blinding effect is lost and the object appears more colorful. Based on this relationship between luminance and saturation, Equation 7 is used to update the saturation component of object k in iteration step i once the luminance component has been updated:

number

[0084] 9A-C illustrate the point cloud separation operation in the ith iteration. A first object (900) and a second object (910) share a border with a third object (920) in the image, and thus share an edge in their respective graph structures. Furthermore, the first object (900) and the second object (910) are the larger ancestors of the third object (920). As a result, the third object (920) is shifted according to the luminance histogram of its ancestor. Center of the sigmoid curve

number

number

[0085] In FIG. 9A, the point cloud separation operation is

number

number

number

[0086] FIG. 11 illustrates a method (1100) for an inter-object point cloud separation operation for object k. Object k may be any object identified in the input image. In block (1102), the method (1100) includes receiving a current luminance array for object k. In block (1104), the method (1100) includes calculating an average luminance of object k based on the luminance array of object k. In block (1106), the method (1100) includes receiving an updated luminance array of an ancestor of object k. In block (1108), the method (1100) includes calculating an average luminance of an ancestor of object k based on the luminance array of the ancestor of object k. In block (1110), the method (1100) includes applying a sigmoid curve to obtain a luminance shift of object k. The sigmoid curve may, for example, be based on the average luminance of object k and the average luminance of object k's ancestors. In block (1112), the luminance array of object k is updated based on the obtained luminance shift. Then, in block (1102), the updated luminance array may be provided as the current luminance array for an additional iteration step of inter-object point cloud separation.

[0087] At block (1114), the method (1100) includes receiving a current saturation array for object k. At block (1116), the method (1100) includes updating the saturation of object k based on the current saturation array and the obtained luminance shift for object k. At block (1118), the method (1100) includes updating the saturation array for object k based on the updated saturation values.

[0088] Expanding point clouds within an object The inter-object point cloud separation operation moves objects based on their luminance and saturation, making them stand out from each other in the output image. This leads to an improvement in the overall visual quality in the output image, but further local or pixel-by-pixel enhancements are required to realize the full potential of the iterative module. The inter-object point cloud separation may make objects brighter and more saturated, or darker and less saturated. This makes the pixels in the image appear to be "washed out" as the original contrast is visibly lost. To counter this possibility, an intra-object point cloud expansion operation is incorporated. The intra-object point cloud expansion operation increases the contrast within the object by increasing the spread of the luminance and chrominance values ​​of the pixels in the image.

[0089]

number

number

[0090] In equation (8), r2 (i) represents the learning rate, which can be adjusted at each iteration to ensure that the value of the image quality metric (GIQM) is improving. f is a function that can take a variety of forms. For example, in some implementations

number

number

number

[0091] Since luminance and saturation share a relationship, their dependency can be taken into account during the intra-object point cloud growth operation. Thus, the saturation component of object k can be updated according to

number

[0092] In equation (9), the luma pixel values ​​of object k are spread out from the average of its luma component. As the chroma pixel values ​​of object k are updated according to the improvements made to the luma component, they are spread out as well. As a result, the pixels in the luma-chroma point cloud are spread out around the centroid of the point cloud, allowing for improved contrast within the object.

[0093] 12 illustrates a number of pixels forming an object k and undergoing an intra-object point cloud grow operation. A first graph (1200) illustrates the luminance and saturation values ​​of the number of pixels prior to the intra-object point cloud grow operation. A second graph (1220) illustrates the luminance and saturation values ​​of the number of pixels after the intra-object point cloud grow operation.

[0094] FIG. 13 shows multiple objects forming an image and undergoing both an inter-object point cloud separation operation and an intra-object point cloud expansion operation. The objects can be, for example, the first object (900), the second object (910), and the third object (920) from FIGS. 9A-9C. A first luminance histogram graph (1300) shows luminance histograms of the first object (900), the second object (910), and the third object (920) before processing begins. A first luminance-saturation graph (1310) shows the respective luminance and saturation values ​​of pixels in the first object (900), the second object (910), and the third object (920).

[0095] The second luminance histogram graph (1320) shows the luminance histograms of the first object (900), the second object (910), and the third object (920) after undergoing the inter-object point cloud separation operation. The second luminance-saturation graph (1330) shows the respective luminance and saturation values ​​of the pixels in the first object (900), the second object (910), and the third object (920) after the inter-object point cloud separation operation. As shown between the first luminance histogram graph (1300) and the second luminance histogram graph (1320), the average luminance of each object may change due to the inter-object point cloud separation operation.

[0096] A third luminance histogram graph (1340) shows the luminance histograms of the first object (900), the second object (910), and the third object (920) after undergoing the intra-object point cloud grow operation. A third luminance-saturation graph (1350) shows the respective luminance and saturation values ​​of the pixels in the first object (900), the second object (910), and the third object (920) after the intra-object point cloud grow operation. As shown between the second luminance histogram graph (1320) and the third luminance histogram graph (1340), the average luminance of each object may remain the same after the intra-object point cloud grow operation. Additionally, the luminance and saturation ranges for each object are increased due to the intra-object point cloud grow operation.

[0097] FIG. 14 illustrates a method (1400) for an intra-object point cloud expansion operation for object k. Object k may be any object identified in the input image. At block (1402), the method (1400) includes receiving a current luminance array for object k. At block (1404), the method (1400) includes calculating an average luminance for object k based on the luminance array for object k. At block (1406), the method (1400) includes updating the luminance for object k according to equation (7). At block (1408), the method (1400) includes updating the luminance array for object k based on the result of equation (7). The updated luminance array at block (1402) may then be provided as the current luminance array for an additional iteration of inter-object point cloud separation.

[0098] At block (1410), the method (1400) includes receiving a current saturation array for object k. At block (1412), the method (1400) includes updating the saturation of object k based on the current saturation array and the updated luminance of object k. At block (1414), the method (1400) includes updating the saturation array for object k based on the updated saturation values.

[0099] Intra-object spatial enhancement Inter-object point cloud separation and intra-object point cloud expansion provide the benefit of improving inter-object and intra-object contrast. While this leads to improved visual quality in the enhanced image, further improvements can be obtained by using intra-object spatial information during processing to bring out detailed contrast within the object. In the example described herein, a shape adaptive discrete cosine transform (SA-DCT) is used for intra-object spatial enhancement. At each iteration of the process (650), the SA-DCT is applied to the particular object being enhanced. The DCT coefficients are then modulated according to a weighting function before an inverse shape adaptive discrete cosine transform (ISA-DCT) is performed. It should be noted that although this SA-DCT algorithm is employed for intra-object spatial enhancement, it is also possible to process the frequency coefficients of the object in other domains, for example using wavelet or Fourier transforms. Furthermore, spatially based image sharpening tools such as unsharp masking may also be utilized instead of the SA-DCT.

[0100] 15A-15E show an implementation of the SA-DCT of the luminance component of an object. I Y,k Let be the segmentation of any kth object from the smallest square inside the image that the object fits into. An example of the segmentation of the kth object is shown in Figure 15A. IY,k Let N′ be the result of shifting all pixels in this object vertically towards the top of the square shown in FIG. 15B. The dimensions of this square are N k ×N k The size N k We define the DCT transformation matrix of as follows:

number

[0101] This matrix is ​​N k DCT Nk Contains a set of basis vectors. Nk vertical DCT Nk The coefficient vector is given by:

number

[0102] Where: I Y,k,i 'teeth I Y,k ' is the i-th column of . This complete transformation is D k As a result, the vertical dimension of the pixels in this segmented object is transformed as shown in FIG. 15C. D k Let ' be the result of shifting all pixels in this object horizontally towards the left of the square shown in Figure 15D. Nk The conversion is then D k ' and the fully transformed SA-DCT output shown in Fig. 15E. D k '' is obtained.

[0103] The inverse operations can be performed to transform objects in the SA-DCT domain to the spatial domain. In the following, the transformation from the spatial domain to the SA-DCT domain and vice versa is represented by the following operations: SADCT() and ISADCT(), respectively. In the above examples, for simplicity of notation, we use D k " was used, but the luminance component of the segmented kth object I Y,k To represent the complete SA-DCT transformation of D k is used. Therefore,

number

[0104] FIG. 16 shows a method (1600) for an intra-object spatial enhancement operation using an SA-DCT for an object k. At block (1602), the method (1600) includes receiving a segmented luminance square of the object k as described with respect to FIGs. 15A-15E. At block (1604), the method (1600) includes computing an SA-DCT transform. At block (1606), the method (1600) includes applying a weighting function to the SA-DCT coefficients computed at block (1604). At block (1608), the method (1600) includes computing an ISA-DCT transform. At block (1610), the method (1600) includes updating the segmented luminance square of the object k.

[0105] The method (1600) provides for computing the DCT of arbitrarily shaped objects in an image, in this manner taking into account the DCT coefficients of each object that may be differently modulated according to the iterative process (650).

[0106] Similar to the DCT coefficients, the SA-DCT coefficients in the upper left corner of the transformation matrix correspond to lower frequency information. To increase spatial contrast inside objects, the higher valued transform coefficients are multiplied using the following weights:

number

number

number

[0107] In formula (13), r3 (i) represents the learning rate, which may be adjusted at each iteration to ensure that the value of the image quality metric (GIQM) is improving.

number

number

number

number

[0108] Local Quality Metrics The improvement applied to each object during each iteration of the process (650) may be monitored. To ensure that image quality is improved each time an object is processed, a modified form of a lightweight image quality metric (LIQM) may be used to measure the local improvement within the image as each object is processed within an iteration. A formula for an exemplary LIQM function is given as Equation (14).

number

[0109] In equation (14), δ and γ are hyperparameters empirically set to 1 and 1 / 4, respectively. c and A.C. c AIE measures the average amount of entropy information and contrast in an image, respectively. c and A.C. c Here is the image I RGB It is defined according to the R, G, and B components of the AIE c is given by equation (15).

number

[0110] In equation (15), the IE term is defined according to the average entropy of the R, G or B color component with which it is associated.

number

[0111] AC c is defined according to equation (17).

number

[0112] By applying LIQM to the entire image after each object is processed, the amount of contrast that the improved object adds to the overall image is captured, since the current object is the only object that has been improved since LIQM was last applied to the image. If the LIQM value reported after processing the object is greater than the LIQM value reported before the object was processed, then the changes applied to the current object are accepted; otherwise, the improvement is not accepted (i.e., the object is not changed at each iteration).

[0113] Global Quality Metrics Returning to Figure 6, before the iterative iterations of process (650) begin, a global image quality metric (GIQM) is applied to determine the image quality of the original input image in block (606). Once the iterative process begins, the GIQM is recalculated at the end of each iteration in block (618) to ensure that the improvements made to the image in each iteration improve the objective image quality.

[0114] FIG. 17 shows an example architecture for a GIQM network (1700). In block (1702), the GIQM network (1700) receives an input image, such as the input image in block (602). In block (1704), the GIQM network (1700) analyzes the input image. In implementations where the improvements made to the current image should be similar to the improvements of a reference image to which the model is given access, a full reference image quality assessment (FR-IQA) model may be used for the analysis. However, in many cases, the model may not have access to a reference image. In such cases, a no-reference image quality assessment (NR-IQA) may be used. In some implementations, NR-IQA is modeled under the assumption that certain forms of distortion are present in the scene and need to be removed. In other implementations, NR-IQA may instead use natural scene statistics (NSS) techniques that aim to find a subspace in a larger image space that contains undistorted images from the natural world. In some implementations, the GIQM network (1700) is a suitable neural network (NN) model, such as a convolutional neural network (CNN) like the Neural Image Assessment (NIMA) model or the Explicit Implicit Image Quality Assessment (EI-IQA) model. In block (1706), the GIQM network (1700) outputs a quality score that indicates the image quality of the input image.

[0115] Iterative process flowchart with learning rate FIG. 18 provides a method (1800) illustrating the sequential iteration process (650) in more detail. In block (1802), the method (1800) includes processing a first object. In block (1804), the method (1800) includes processing a second object. Each object is processed according to the graph structure described above until the final object is processed by block (1806). The GIQM is calculated in block (1808) at the end of each iteration on the output image. If the GIQM score has increased, the next iteration is started with the same learning rate. If the GIQM score has decreased but the stopping criteria are not met, the next iteration may also be started with the same learning rate. The GIQM may have decreased but the LIQM may have increased, and therefore it may not be desirable to change the learning rate. If the GIQM score has not changed from the previous iteration, no changes have been made to any objects in the current iteration and the learning rate is decreased in block (1810).

[0116] The method (1800) is repeated until the learning rate becomes so small that improvements no longer lead to any significant improvement in the visual quality of the image. (i-1) Let denote one of these learning rates at the end of the i-1th iteration. If the GIQM score has not changed since the i-1th iteration, the new learning rate is r (i) =r (i-1) / τ. In some implementations, τ is a positive constant set to 2. If the learning rate falls below a threshold, the iterative process terminates.

[0117] Stopping criteria If not checked, the iterative process may run for many iterations without any visible improvement to the image. Furthermore, if the image starts to deteriorate due to excessive improvement, the algorithm may be stopped. The GIQM score is used to determine when to stop the iterative process. FIG. 19A shows a sample of the trend of the GIQM score through all the iterations. To avoid observing only local minima, the average of the last Q GIQM scores is used to see the smoothed trend shown in FIG. 19B. This trend increases consistently until it converges. If the smoothed score in FIG. 19B stagnates or decreases, the algorithm is terminated and the best image up to that iteration is output.

[0118] The above video delivery systems and methods may provide for enhancing images using graph-based inter-object and intra-object separation. The systems, methods, and devices according to the present disclosure may have any one or more of the following configurations:

[0119] (1) A video delivery system for iterative graph-based image enhancement of image frames, comprising a processor for performing processing of the image frames, the processor being configured to receive an object in the image frame, the object comprising a plurality of pixels, perform an inter-object point cloud separation operation on the image, enlarge the plurality of pixels of the object, perform a spatial enhancement operation on the plurality of pixels of the object, and generate an output image based on the inter-object point cloud separation operation, the enlargement of the plurality of pixels, and the spatial enhancement operation.

[0120] (2) The video delivery system of (1), wherein the processor is further configured to replace unlabeled pixels in the image frame using a breadth-first search operation.

[0121] (3) The video delivery system of any one of (1) to (2), wherein the object is a first object of a plurality of objects, and the processor is further configured to perform a segmentation operation to obtain the plurality of objects, determine a priority queue for the plurality of objects, and determine one or more attributes of each object of the plurality of objects, the one or more attributes including at least one selected from the group consisting of a size of the object, a list of neighboring objects, an average pixel intensity of the object, a segmentation map label of the object, a saturation histogram of the object, and a luminance histogram of the object.

[0122] (4) The video delivery system of any one of (1) to (3), wherein when performing the inter-object point cloud separation operation on the image, the processor is configured to calculate an average luminance of the object, calculate an average luminance of the object's ancestors from one or more previous iteration steps, and apply a sigmoid curve to the object based on the average luminance of the object and the average luminance of the object's ancestors to obtain a luminance shift of the object.

[0123] (5) The video delivery system of (4), wherein when performing the inter-object point cloud separation operation on the image, the processor is further configured to receive a saturation array of the object and update the saturation array of the object based on the luminance shift of the object.

[0124] (6) A video delivery system described in any one of (1) to (5), wherein when the plurality of pixels of the object are enlarged, the processor is configured to determine an average luminance of the object, determine a current saturation of the object, update a luminance array of the object, and update a saturation array of the object based on the updated luminance array.

[0125] (7) A video delivery system as described in any one of (1) to (6), wherein when performing the spatial enhancement operation on the plurality of pixels of the object, the processor is configured to segment luminance squares of the object, calculate shape adaptive discrete cosine transform (SA-DCT) coefficients, and apply a weighting function to the SA-DCT coefficients.

[0126] (8) A video delivery system as described in any one of (1) to (7), wherein when performing the spatial enhancement operation on the plurality of pixels of the object, the processor is further configured to calculate inverse SA-DCT coefficients and update the segmented luminance squares of the object based on the SA-DCT.

[0127] (9) A video delivery system described in any one of (1) to (8), wherein the processor is further configured to apply a global image quality metric (GIQM) to the image frame and determine a quality of the image frame based on the output of the GIQM.

[0128] (10) The video delivery system of (9), wherein the processor is further configured to sequentially repeat the steps of performing the inter-object point cloud separation operation on the image, enlarging the plurality of pixels of the object, and performing the spatial enhancement operation on the plurality of pixels of the object until the quality of the image frame meets a quality threshold.

[0129] (11) An iterative method for image enhancement of an image frame, the method comprising: receiving an object in the image frame, the object comprising a plurality of pixels; performing an inter-object point cloud separation operation on the image; enlarging the plurality of pixels of the object; performing a spatial enhancement operation on the plurality of pixels of the object; and generating an output image based on the inter-object point cloud separation operation, the enlarging of the plurality of pixels, and the spatial enhancement operation.

[0130] (12) The method of (11), wherein the object is a first object of a plurality of objects, the method further comprising: performing a segmentation operation to obtain the plurality of objects; determining a priority queue for the plurality of objects; and determining one or more attributes of each object of the plurality of objects, the one or more attributes comprising at least one selected from the group consisting of a size of the object, a list of neighboring objects, an average pixel intensity of the object, a segmentation map label of the object, a saturation histogram of the object, and a luminance histogram of the object.

[0131] (13) A method according to any one of (11) to (12), wherein the step of performing the inter-object point cloud separation operation on the image frame includes the steps of calculating an average luminance of the object, calculating an average luminance of ancestors of the object from one or more previous iterations, and applying a sigmoid curve to the object based on the average luminance of the object and the average luminance of the ancestors of the object to obtain a luminance shift of the object.

[0132] (14) The method of (13), wherein the step of performing the inter-object point cloud separation operation on the image frame further includes the step of receiving a saturation array of the object and updating the saturation array of the object based on the luminance shift of the object.

[0133] (15) A method according to any one of (11) to (14), wherein enlarging the plurality of pixels of the object includes determining an average luminance of the object, determining a current saturation of the object, updating a luminance array of the object, and updating a saturation array of the object based on the updated luminance array.

[0134] (16) A method according to any one of (11) to (15), wherein performing the spatial enhancement operation on the plurality of pixels of the object includes segmenting luminance squares of the object, computing shape adaptive discrete cosine transform (SA-DCT) coefficients, and applying a weighting function to the SA-DCT coefficients.

[0135] (17) The method of (16), wherein performing the spatial enhancement operation on the plurality of pixels of the object further comprises calculating inverse SA-DCT coefficients and updating the segmented luminance squares of the object based on the SA-DCT.

[0136] (18) A method according to any one of (11) to (17), further comprising the step of applying a global image quality metric (GIQM) to the image frame and determining the quality of the image frame based on the output of the GIQM.

[0137] (19) The method of (18), further comprising sequentially repeating the steps of performing the inter-object point cloud separation operation on the image, expanding the plurality of pixels of the object, and performing the spatial enhancement operation on the plurality of pixels of the object until a quality of the frame meets a quality threshold.

[0138] (20) A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method according to any one of (11) to (19).

[0139] With respect to the processes, systems, methods, heuristics, etc. described herein, steps of such processes, etc. are described as occurring according to a certain ordered sequence, but it should be understood that such processes may be practiced with the described steps performed in orders other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the scope of the claims in any way.

[0140] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology described herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0141] All terms used in the claims are intended to be given their broadest reasonable interpretation and ordinary meaning as understood by one skilled in the art described herein, unless expressly indicated to the contrary herein. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless the claim describes an explicit limitation to the contrary.

[0142] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of improving the flow of the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. 1. A video delivery system for iterative graph-based image enhancement of image frames, the video delivery system comprising: a processor for performing processing of the image frames, the processor comprising: receiving, for a plurality of objects within the image frame, locations of pixels that constitute each object; storing the received locations corresponding to the plurality of objects in a segmentation map; extracting individual objects from the image frames by using the segmentation map to generate a graph characterizing the plurality of objects, the graph providing structural information regarding which object should be visited first and how each object within the image frames should be processed; performing an inter-object point cloud separation operation on the image, the inter-object point cloud separation separating luma-saturation point clouds for objects from each other such that the objects stand out more from each other in a final image; Expanding the pixels of a particular object by increasing the spread of pixels within the object, so that intra-object pixel values ​​stand out and cover a larger luma-chroma range; performing a spatial enhancement operation on the plurality of pixels of the object; generating an output image based on the inter-object point cloud separation operation, the dilation of the plurality of pixels, and the spatial enhancement operation, wherein the steps of performing the inter-object point cloud separation operation on the image, dilation of the plurality of pixels of the object, and performing the spatial enhancement operation on the plurality of pixels of the object are iteratively repeated until a quality of the frame satisfies a quality threshold; is configured to run Video delivery system.

2. When performing the object-to-object point cloud separation operation on the image, the processor: Calculate an average luminance of the object; Calculate an average luminance of ancestors of the object from one or more previous iterations; Applying a sigmoid curve to the object based on the average luminance of the object and the average luminance of the object's ancestors to obtain a luminance shift of the object. The video delivery system of claim 1 , configured to:

3. When expanding the plurality of pixels of the object, the processor: determining an average luminance of the object; determining a current saturation of the object; updating a luminance array of the object; and updating a saturation array of the object based on the updated luminance array.

3. A video delivery system according to claim 1 or 2.

4. When performing the spatial enhancement operation on the plurality of pixels of the object, the processor: Segmenting a luminance square of the object; Calculate the shape adaptive discrete cosine transform (SA-DCT) coefficients; configured to apply a weighting function to the SA-DCT coefficients. A video delivery system according to any one of claims 1 to 3.

5. When further performing the spatial enhancement operation on the plurality of pixels of the object, the processor: Calculate the inverse SA-DCT coefficients; and updating the segmented luminance squares of the object based on the SA-DCT. The video delivery system of claim 4.

6. The processor: applying a global image quality metric (GIQM) to the image frames; and determining a quality of the frame based on an output of the GIQM. A video delivery system according to any one of claims 1 to 5.

7. 1. An iterative method for image enhancement of an image frame, the method comprising: receiving, for a plurality of objects within the image frame, locations of pixels that constitute each object; storing the received locations corresponding to the plurality of objects in a segmentation map; extracting individual objects from the image frames by using the segmentation map to generate a graph characterizing the plurality of objects, the graph providing structural information regarding which object should be visited first and how each object within the image frames should be processed; performing an inter-object point cloud separation operation on the image, the inter-object point cloud separation separating luma-saturation point clouds for objects from each other such that the objects stand out more from each other in a final image; Expanding the pixels of a particular object by increasing the spread of pixels within the object, so that intra-object pixel values ​​stand out and cover a larger luma-chroma range; performing a spatial enhancement operation on the plurality of pixels of the object; generating an output image based on the inter-object point cloud separation operation, the dilation of the plurality of pixels, and the spatial enhancement operation, wherein the steps of performing the inter-object point cloud separation operation on the image, dilation of the plurality of pixels of the object, and performing the spatial enhancement operation on the plurality of pixels of the object are iteratively repeated until a quality of the frame satisfies a quality threshold. method.

8. Performing the object-to-object point cloud separation operation on the image frames includes: Calculate an average luminance of the object; Calculate an average luminance of ancestors of the object from one or more previous iterations; applying a sigmoid curve to the object based on an average luminance of the object and an average luminance of ancestors of the object to obtain a luminance shift of the object. The method of claim 7.

9. Performing the object-to-object point cloud separation operation on the image frames further comprises: receiving a saturation array for the object; updating the saturation array of the object based on the luminance shift of the object. The method according to claim 8.

10. Magnifying the plurality of pixels of the object comprises: determining an average luminance of the object; determining a current saturation of the object; updating a luminance array of the object; updating a saturation array of the object based on the updated luminance array.

10. The method according to any one of claims 7 to 9.

11. Performing the spatial enhancement operation on the plurality of pixels of the object comprises: Segmenting a luminance square of the object; Calculate the shape adaptive discrete cosine transform (SA-DCT) coefficients; applying a weighting function to the SA-DCT coefficients.

11. The method according to any one of claims 7 to 10.

12. applying a global image quality metric (GIQM) to the image frames; determining a quality of the frame based on an output of the GIQM; 12. The method according to any one of claims 7 to 11.

13. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method of any one of claims 7 to 12.

Images