Dynamic spatial metadata for image and video processing

Abstract

Methods and systems for generating and using dynamic spatial metadata in image and video processing are disclosed. At the encoder, in addition to global metadata, local spatial metadata for two or more image regions or image objects is generated, smoothed, and embedded as spatial metadata values. At the decoder, the decoder can reconstruct the spatial metadata and use interpolation techniques to generate metadata for specific regions of interest. An example of spatial metadata generation is provided for minimum, median, and maximum luminance values ​​in an image.

Description

[Technical field]

[0001] The present invention relates generally to images. More specifically, embodiments of the present invention relate to generating and applying dynamic spatial metadata in image and video processing. [Background technology]

[0002] The creation of content, such as movies, television programs, or cartoons, and the capture of content, such as the recording of live sporting or news events, is characterized by a number of factors, including pixel resolution, aspect ratio (16:9, 9:16, etc.), and dynamic range (e.g., standard dynamic range, high dynamic range, HDR10, etc.). Once the content is created or captured, it is then distributed for playback on devices that may have a wide variety of aspect ratios, resolutions, and dynamic ranges. All such playback devices will then be required to adapt the display of the content by performing various operations, including upsampling, downsampling, cropping or padding the content to fit the resolution and aspect ratio of the target display, and / or display mapping that maps the incoming dynamic range of the content to the dynamic range of the target display. Users can also adapt the incoming video or image to their preferences, for example by screen rotation, cropping, zooming, etc.

[0003] As used herein, the terms "display management" or "display mapping" refer to processes performed at a receiver to render a picture on a target display. For example, without limitation, such processes may include tone mapping, gamut mapping, color management, frame rate conversion, etc.

[0004] In most cases, the transmitted content is also accompanied by metadata.

[0005] As used herein, the term "metadata" refers to any auxiliary information transmitted as part of a coded bitstream or still image that aids a decoder in rendering a decoded image. Such metadata can be "static", i.e., metadata that remains unchanged within the video bitstream or within the captured set of images. Typically, static metadata characterizes the main monitor and / or environment used to create the video (e.g., the dynamic range and color gamut of the reference display, the ambient light during content creation, the color gamut of the bitstream, the encoding of the bitstream (e.g., the use of gamma, HLG, or PQ coding for the video (Reference [5])), etc.). Some metadata can be "dynamic", i.e., it can change on a picture, frame, or scene basis within the video stream or the captured set of still images. Image metadata plays an important role in the correct processing and display of the incoming content. SUMMARY OF THE DISCLOSURE As recognized by the inventors herein, in order to improve upon existing image processing and display schemes, improved techniques for dynamic image metadata generation are developed.

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously considered or pursued. Thus, unless otherwise indicated, none of the approaches described in this section are qualified as prior art by virtue of their inclusion in this section alone. Similarly, it should not be assumed that problems identified with one or more of the approaches have been recognized in the prior art based on this section, unless otherwise indicated.

[0007] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numbers refer to similar elements and in which: [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates an example of a video delivery pipeline process. [Figure 2A] 4 illustrates an example process for generating spatial metadata according to an embodiment of the present invention. [Figure 2B] 1 illustrates an example process for extracting and applying spatial metadata according to an embodiment of the present invention. [Figure 2C] 4 illustrates an example process for generating and applying spatial metadata according to an embodiment of the present invention. [Diagram 3] 2 illustrates an example of an image, a region of interest, and associated spatial metadata regions within the image according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A method for generating and using dynamic spatial metadata in image and video processing is described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present invention.

[0010] [overview] Exemplary embodiments described herein relate to methods for generating and using dynamic spatial metadata in image and video processing. In an embodiment, a processor receives an input image (202) at an input spatial resolution and an input aspect ratio. The processor: Calculating (210) first metadata parameters for the input image at a first spatial resolution equal to or less than the input spatial resolution; determining (205) a second spatial resolution of the spatial metadata parameter that is smaller than the first spatial resolution; calculating global metadata parameters of the input image based on the first metadata parameters; generating smoothed metadata parameters by smoothing the first metadata parameters (220); downsampling (225) the smoothed metadata parameters to generate spatial metadata parameters at a second spatial resolution; A metadata bitstream is generated based on the global metadata parameters and the spatial metadata parameters.

[0011] In a second embodiment, the processor receives an input image (202) at an input spatial resolution and image metadata (232), the image metadata including global metadata parameters for the entire input image and encoded spatial metadata parameters for two or more regions of the input image; generating spatial metadata parameters for two or more regions based on the encoded spatial metadata parameters and the global metadata parameters; receiving region of interest (ROI) parameters defining an ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; Generate an ROI image based on the input image and the ROI parameters; An image processing operation is performed on the ROI image based at least on pixel values ​​in the ROI image and the ROI metadata.

[0012] In a third embodiment, a processor receives an input image (202) at an input spatial resolution and an input aspect ratio. The processor: Calculating (210) first metadata parameters for the input image at a first spatial resolution equal to or less than the input spatial resolution; determining (205) a second spatial resolution of the spatial metadata parameter that is smaller than the first spatial resolution; Calculating global metadata parameters of the input image based on the first metadata parameters; generating smoothed metadata parameters by smoothing the first metadata parameters (220); downsampling (225) the smoothed metadata parameters to generate spatial metadata parameters at a second spatial resolution; receiving region of interest (ROI) parameters defining an ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; Generate an ROI image based on the input image and the ROI parameters; An image processing operation is performed on the ROI image based at least on pixel values ​​in the ROI image and the ROI metadata.

[0013] [Dynamic spatial metadata] Video Coding Pipeline Figure 1 illustrates an example of a conventional video distribution pipeline (100) process showing various stages from video capture to video content display. A sequence of video frames (102) is captured or generated using an image generation block (105). The video frames (102) may be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In a production phase (110), the video data (107) is edited to provide a video production stream (112).

[0014] The video data of the production stream (112) is then provided to a processor in a block (115) for post-production editing, which may include adjusting or modifying the color or brightness in certain areas of the image to enhance image quality or achieve a certain look of the image according to the creative intent of the video creator. This is sometimes called "color timing" or "color grading". Other editing (e.g., scene selection and ordering, image cropping, addition of computer-generated visual special effects, etc.) may be performed in block (115) to obtain a final version of the product (117) for distribution. During post-production editing (115), the video image is viewed on a reference display (125). Following post-production (115), the video data of the final product (117) may be provided to an encoding block (120) for downstream delivery to decoding and playback devices such as televisions, set-top boxes, movie theaters, etc. In some embodiments, the coding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blue-Ray, and other distribution formats, to generate a coded bitstream (122). At the receiver, the coded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents the same or a close approximation of the signal (117). The receiver may be attached to a target display (140), which may have characteristics quite different from the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). Without limitation, examples of display management processes are described in References [1] and [2].

[0015] Example process 100 may also be applied to processing of still images, which may include, for example, capture (105) on a capture device, user enhancement or editing (110, 115), coding (120), decoding (130), and display management on the capture device or other playback device.

[0016] Display Mapping and Metadata In traditional display mapping (DM), the mapping algorithm applies a sigmoid function (see, for example, references [3] and [4]) to map the input dynamic range to the dynamic range of the target display. Such mapping functions can be expressed as piecewise linear or nonlinear polynomials characterized by anchor points, pivots, and other polynomial parameters that are generated using the characteristics of the input source and the target display. For example, in references [3] and [4], the mapping functions use anchor points based on the luminance characteristics of the input image and the display (e.g., maximum, mean (average), and maximum luminance). Note that other mapping functions may use different statistics, such as luminance variance or luminance standard deviation values ​​at the block level or for the entire image. For standard dynamic range (SDR) images, the process may be aided by additional metadata that is either transmitted as part of the transmitted video or calculated by the decoder or display. For example, if a content provider has both SDR and high dynamic range (HDR) versions of source content, the source may use both versions to generate metadata (e.g., piecewise linear approximations of forward or inverse reshaping functions) to help the decoder convert the incoming SDR image into an HDR image.

[0017] As used herein, the term "Level 1 (L1) metadata" refers to the minimum, mean, and maximum luminance values ​​for the entire input frame or image. L1 metadata is considered dynamic metadata and may be calculated by converting the RGB data to luma-chroma format (e.g., YCbCr) and then calculating the minimum, mean, and maximum values ​​in the intensity / luma plane, or they may be calculated directly in RGB space. For example, in an embodiment, L1min represents the minimum of the PQ-encoded (reference [5]) min(RGB) values ​​of an image while taking into account active regions (e.g., by excluding gray or black bars, letterbox bars, etc.). min(RGB) represents the minimum of the color component values ​​{R,G,B} of a pixel. The values ​​of L1mid and L1max may be calculated similarly. For example, in an embodiment, L1mid represents the average of the PQ-encoded max(RGB) values ​​of an image, and L1max represents the maximum of the PQ-encoded max(RGB) values ​​of an image. In some embodiments, the L1 metadata may be normalized to be in [0,1].

[0018] Considering the maximum (peak) and minimum (black) luminance of the target display, denoted as Tmax and Tmin, as well as L1Min, L1Mid, and L1Max of the original metadata, an intensity tone mapping curve can be generated that maps the intensity of the input image to the dynamic range of the target display, as described in references [3] and [4]. Such a mapping works very well when mapping the entire image to the display, but if the input image is cropped or zoomed before being displayed, unexpected results may occur, since the luminance information for the selected part of the image may be quite different from the luminance information of the entire image. As recognized by the inventors, image processing such as tone mapping of cropped or zoomed regions, also called regions of interest (ROI), may be improved by using metadata based on local areas, or spatial metadata.

[0019] As used herein, the term "spatial metadata" refers to image metadata that represents spatial pixel information within a sub-region of an image. For example, given an input image subdivided into M×N non-overlapping rectangular blocks, the L1 spatial metadata may include the minimum, median, and maximum luminance values ​​in each one of these M×N blocks. In other embodiments, the image may be segmented into objects, and each such object is associated with its own spatial metadata. As recognized by the inventors, in order to avoid image artifacts or degraded user experience, the spatial metadata may be characterized by the following properties: Represents an "envelope" of global metadata (i.e. metadata that encompasses minimum and maximum values ​​within surrounding local regions). Smooth to allow smooth transitions when the user manipulates adjacent parts of the image region under observation. Generated automatically. ·Do not impose excessive computational load during content creation. Without loss of generality, an example of the process of generating L1 spatial metadata is given below.

[0020] Generating spatial metadata Consider an input image with spatial resolution of width x height. Either the entire image is used to generate the global and spatial metadata, or, due to computational resource limitations, a decision is made to generate the global and spatial metadata based on an input subset. In an embodiment, and without limitation, the aspect ratio of the resolution of the spatial metadata may match the aspect ratio of the image data used to generate the global metadata. Note: In the remaining discussion, without loss of generality, specific computational examples may be given in pseudocode or in MATLAB code.

[0021] Step 1: Determine the metadata size from the image dimensions. In this example, it can be determined that the longer image dimension (length or width) of the spatial metadata is fixed to have M (e.g., M=16) samples. The other dimension (N) can then be derived using the matching aspect ratio. In this example, the full image is used. For example, if M=16:

number

[0022] At the end of step 1, vector MDsz specifies the size of an M×N array of spatial metadata. If these metadata represent L1 metadata (e.g., minimum, middle, and maximum luminance values), then, as an example, steps 2-3 below calculate the corresponding minimum and maximum values ​​in the RGB domain. Similar steps can be applied to calculate L1 metadata in the luma-chroma domain, or to calculate other dynamic spatial metadata values.

[0023] Step 2: Calculate min(RGB) and max(RGB).

number

[0024] Step 3: Compute global (full-frame) metadata.

number

[0025] Step 4: Dilate and erode the minimum and maximum values. This may also be called the windowed maximum and windowed minimum. The radius (R) of the dilation filter is calculated from the ratio of the image size to the spatial metadata size multiplied by some factor (e.g., 2 in this case). In the following example, a rectangular filter is used, but other shapes such as a circle are possible. This is a preprocessing step to improve the metadata smoothing operation that will follow in the next step.

number

[0026] Step 5: Smooth the minimum and maximum values. This step ensures that the cropped area changes smoothly and so does the metadata, preventing sudden, jarring changes across boundaries. In an embodiment, the radius of the spatial filter is half the radius used in the previous step to ensure that the minima / maxima do not change. In this example, a 2D Gaussian filter is used.

number

[0027] Step 6: Downsample the filtered image to the desired metadata resolution.

number

[0028] Given the spatial metadata of step 6, there are many alternatives for packing them and transmitting them as part of the image data. In an embodiment, the L1 spatial metadata may be transmitted as an offset value from the global L1 metadata. For example, in an embodiment, the global L1 metadata and the spatial L1 metadata may be transmitted as follows:

number

[0029] Assuming a differential coding scheme, the offset value may be calculated as shown in step 7. Step 7: Calculate the offset from the global metadata.

number

[0030] FIG. 2A illustrates an example process of steps 1 through 7 for generating spatial metadata.

[0031] Given an input image 202, as represented in FIG. 2A, step 205 determines the resolution (e.g., M×N) of the spatial metadata. In step 210, metadata parameters may be calculated at a first spatial resolution that represents the highest possible resolution of available values. For example, for L1 metadata, parameters for minimum, medium, and maximum luminance values ​​may be calculated at the full resolution pixel level. In step 215, global metadata for the entire image may be calculated using the results of step 210. In step 220, the metadata generated at the first spatial resolution may need to be pre-processed and smoothed to generate smooth metadata. As indicated above, smoothing may include various operations such as dilation, erosion, and spatial filtering. In step 225, the smoothed metadata is downsampled to generate spatial metadata at the desired resolution of the spatial metadata. Finally, in step 230, both the global metadata and the corresponding spatial metadata are packed together in a suitable file format (232) to be transmitted downstream along with the image data. For example, the spatial metadata may be transmitted as an offset of the global metadata.

[0032] In an embodiment, spatial metadata may also be computed at the "object" level. For example, an image may be segmented into two or more arbitrarily shaped objects (overlapping or non-overlapping). Then, after smoothing the image-level metadata parameters (step 220), the encoder may generate spatial metadata parameters for each object by a corresponding segmentation of the image-level metadata parameters. To pack the metadata in step 230, the metadata format may include the number of available objects, a label for each object, and the corresponding metadata parameters for each object.

[0033] Reconstruction and application of spatial metadata Given an image (202) with both global metadata and coded spatial metadata (232), Figure 2B illustrates an example process for reconstructing the metadata and applying them to an image processing application, such as display mapping. For example, assuming the encoder transmitted the spatial metadata as offset values ​​to the global metadata, in step 240 the actual spatial L1 metadata values ​​can be reconstructed as follows:

number

number

[0034] Then, in step 245, the M×N metadata may be upscaled to a desired resolution, for example the same resolution as the input image. For example: Phase 2: Upscale metadata to image resolution

number

[0035] Next, given a region of interest in the input image (202), specified by ROI image parameters (247), step 250 selects appropriate spatial metadata for that region. For example, if the application performs display mapping of the region of interest (e.g., a region generated by cropping, zooming, etc.), in an embodiment, the spatial metadata generated for the entire image in step 245 may be cropped, zoomed, etc. to fit the region of interest. Note that the ROI may be rectangular, circular, elliptical, or any shape based on user control or other means of the application. In that case, ROI image parameters 247 may define appropriate parameters for a circle, rectangle, polygon, etc., while defining the region of interest and its resolution. For example, for a rectangular ROI: Step 3: Select metadata to fit the selected region of interest in the displayed image

number

[0036] In some embodiments, step 245 can be skipped by modifying step 250, thus reducing the computational burden. For example, in step 250, a) calculating an interpolated value for every edge of a region of interest (ROI) in the low resolution spatial metadata based on the distance from the edge of the ROI to adjacent spatial metadata values; b) Calculate the ROI metadata based on the interpolated metadata values ​​(eg, max, min, average) of the edge values ​​and all metadata values ​​that are entirely within the region of interest.

[0037] As an example, consider Figure 3, which depicts an image 305 that has been subdivided into 4 x 3 spatial metadata regions. Consider an ROI image 310 specified within spatial metadata regions 315, 320, 325, 330, 335, and 340. ROI metadata may be calculated based on the interpolated metadata across all edge points of the ROI (310) and the spatial metadata of metadata region 330. For example, to calculate the metadata across the top edge of ROI 310, one interpolates between local metadata values ​​in regions 315 and 325 and regions 320 and 330.

[0038] Returning to Figure 2B, given a region of interest and its corresponding spatial metadata, the image processor may then act on the selected region using information from its corresponding spatial metadata, in step 255. For example, a display mapping process (see References [1]-[4]) may tone map the ROI to a target display using the target display characteristics and the ROI metadata.

[0039] In some embodiments, the spatial metadata values ​​may be further scaled or generally transformed before being applied to imaging operations (e.g., tone mapping, contrast enhancement, etc.) within the region of interest. For example, consider an image captured by a device with a 1,000 nit display (e.g., a mobile phone with Dolby Vision). The captured image may contain a very bright light source (e.g., the sun, or a window with a bright sky) and other less luminous items (e.g., clouds, curtains, people, etc.). In an encoder, the image capture device may map the brightest light source to 1,000 nits and adjust the next brightest items (e.g., clouds) to an appropriate brightness, e.g., 200 nits, to accommodate the typically very high brightness ratio between the two objects. When displaying the entire image on a similar device (e.g., a similar phone) at 1,000 nits, the reference display and the target display have the same dynamic range, so under the normal display mapping process, there is no dynamic range conversion and all image components will be displayed as they were captured. However, if the user of the second device decides to crop the brightest light source, it may be beneficial to use the spatial metadata to remap all other items in the image to take full advantage of the HDR display. For example, even if the L1 spatial metadata for a cloud area specifies that the local maximum luminance is 200 nits, the decoder may decide to multiply it by a scale factor such as 2 or 4 to map it to a 400 nit or 800 nit display, thereby improving the user experience. These scaling factors may be determined by a tone mapping algorithm, via a user interface on the display device, or they may be part of the image metadata as additional metadata, for example a lookup table that associates each object in the image or each M×N region with a weight or scaler.

[0040] Typically, when cropping or zooming on an image, a user may use a fairly crude interface (e.g., their finger), and the region of interest, e.g., a person's face, may include nearby areas, and local spatial metadata may distort the parameters required for proper display mapping. Furthermore, as mentioned before, removing the cropped region may provide more dynamic range room for expanding the dynamic range of the cropped or zoomed region. In such a scenario, the decoder may derive a "difference metric" by comparing the difference between the extracted local metadata and the global metadata, and use that metric to assign a higher weight to the spatial metadata values ​​of a particular region than the values ​​generated by steps 245 and 250 above. These weights or scaling values ​​may also be part of the image metadata as a look-up table, so that each captured image (or scene in a video) has its own local optimization. Alternatively, these scaling factors or weights may be calculated by the decoder on the fly based on the global metadata values ​​and the ROI metadata values. Given the practicality of such scaling, for example, the reconstruction stage 1 above may be rewritten as follows:

number

[0041] In an embodiment, a decoder (or playback) device can be both a producer and consumer of spatial metadata. For example, a user may capture an image with his phone or camera and then want to display it with a crop or zoom operation. Alternatively, a user may receive an image without spatial metadata, but want to display only a portion of the image. In such a scenario, if a full-size image is loaded, the device may generate both global metadata and full-size spatial metadata. Then, when the image is zoomed, cropped, resized, etc., given a ROI image, the display mapping function a) selects the appropriate ROI metadata from the full-size spatial metadata, and b) applies tone mapping or other image processing functions (e.g., contrast enhancement, etc.) using the selected ROI metadata. This has the advantage that the full image metadata is calculated only once and the ROI metadata can be updated immediately without further analysis of the ROI image of interest. An example of such a process combines the processing steps discussed in Figures 2A and 2B and is depicted in Figure 2C. As depicted in FIG. 2C, processing steps 205 through 225 are calculated only once, but steps 250 and 255 may be calculated each time the ROI parameters (247) change.

[0042] A number of embodiments of the present disclosure have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Thus, the present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE) which describe the structure, features, and functions of some portions of the present invention.

[0043] EEE1. 1. A method of processing spatial metadata by a processor, comprising: accessing an input image (202) at an input spatial resolution and an input aspect ratio; Calculating (210) a first metadata parameter of the input image at a first spatial resolution equal to or smaller than the input spatial resolution; determining (205) a second spatial resolution of the spatial metadata parameter that is smaller than the first spatial resolution; calculating global metadata parameters of the input image based on the first metadata parameters; smoothing the first metadata parameter to generate a smoothed metadata parameter (220); downsampling (225) the smoothed metadata parameters to generate the spatial metadata parameters at the second spatial resolution; The method according to claim 1,

[0044] EEE2. receiving region of interest (ROI) parameters defining a ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; generating a ROI image based on the input image and the ROI parameters; performing an image processing operation on the ROI image based at least on pixel values ​​within the ROI image and the ROI metadata; The method according to EEE1, further comprising:

[0045] EEE3. generating a metadata bitstream based on the global metadata parameters and the spatial metadata parameters. The method according to EEE1 or EEE2.

[0046] EEE4. the second spatial resolution and the first spatial resolution have the same aspect ratio. A method according to any one of EEE1 to EEE3.

[0047] EEE5. Smoothing the first metadata parameter comprises: applying a dilation filter to the first metadata parameters to generate dilated metadata parameters; applying an erosion filter to the dilation metadata parameters to generate erosion metadata parameters; applying a 2D spatial filter to the eroded metadata parameters to generate the smoothed metadata parameters; having A method according to any one of EEE1 to EEE4.

[0048] EEE6. the 2D spatial filter comprises a 2D Gaussian filter; The method described in EEE5.

[0049] EEE7. generating the metadata bitstream calculating a metadata offset between the spatial metadata parameters and the global metadata parameters; generating the metadata bitstream based on the global metadata parameters and the metadata offset; having A method according to any one of claims 8 to 10.

[0050] EEE8. 1. A method for processing an image containing spatial metadata, comprising the steps of: receiving an input image (202) at an input spatial resolution and image metadata (232), the image metadata including global metadata parameters for the entire input image and encoded spatial metadata parameters for two or more regions of the input image; generating spatial metadata parameters for the two or more regions based on the encoded spatial metadata parameters and the global metadata parameters; receiving region of interest (ROI) parameters defining a ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; generating a ROI image based on the input image and the ROI parameters; performing an image processing operation on the ROI image based at least on pixel values ​​within the ROI image and the ROI metadata; The method according to claim 1,

[0051] EEE9. the encoded spatial metadata parameters comprise difference values ​​between the global metadata parameters and the spatial metadata parameters. Method as described in EEE8.

[0052] EEE10. generating the spatial metadata parameters includes subtracting the difference values ​​from the global metadata parameters; The method described in EEE9.

[0053] EEE11. Generating the ROI metadata based on the spatial metadata parameters and the ROI parameters includes: upscaling the spatial metadata parameters to a spatial metadata resolution, a third spatial resolution, to generate upscaled metadata; extracting the ROI metadata from the upscale metadata based on the ROI parameters; having A method according to any one of EEE2 to EEE10.

[0054] EEE12. Generating the ROI metadata based on the spatial metadata parameters and the ROI parameters includes: generating edge spatial metadata parameters by interpolating between spatial metadata parameters in regions surrounding edges of the ROI image; generating the ROI metadata based on spatial metadata parameters associated with a region within the ROI image and the edge spatial metadata parameters; having A method according to any one of EEE2 to EEE10.

[0055] EEE13. The image processing operation includes a tone mapping operation of the ROI image to a target display. A method according to any one of EEE2 to EEE12.

[0056] EEE14. scaling ROI metadata parameters in the ROI metadata by a scaling factor before applying the ROI metadata to the image processing operation. A method according to any one of EEE2 to EEE13.

[0057] EEE15. the scaling factor is based on a difference between the ROI metadata parameters and the global metadata parameters; Method according to EEE14.

[0058] EEE16. the global metadata parameters include one of a maximum luminance value, an average luminance value, or a minimum luminance value of the input image; A method according to any one of claims 1 to 15.

[0059] EEE17. the spatial metadata parameters are generated for M×N non-overlapping regions of the input image, where M and M are integers and M×N represents the second spatial resolution; A method according to any one of claims 1 to 15.

[0060] EEE18. An apparatus having a processor and configured to perform any one of the methods described in EEE1 to EEE17.

[0061] EEE19. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing, by one or more processors, a method according to any one of claims 1 to 17.

[0062] References Each one of the references cited herein is hereby incorporated by reference in its entirety. [1] U.S. Patent No. 9,961,237, “Display management for high dynamic range video”, R. Atkins [2] International Patent Application No. PCT / US2020 / 028552, filed April 16, 2020, International Publication No. WO2020 / 219341, “Display management for high dynamic range images”, R. Atkins et al. [3] U.S. Patent No. 8,593,480, “Method and apparatus for image data transformation”, A. Ballestad and A. Kostin [4] U.S. Patent No. 10,600,166, “Tone curve mapping for high dynamic range images”, J. A. Pytlarz and R. Atkins [5]Rec. ITU-R BT-2100-2, “Image parameter values ​​for high dynamic range television for use in production and international program exchange”, ITU, July 2018

[0063] [Example of computer system] Embodiments of the invention may be implemented by computer systems, systems configured in electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete time or digital signal processors (DSPs), application specific ICs (ASICs), and / or apparatuses including one or more of such systems, devices, or components. The computers and / or ICs may execute, control, or perform instructions related to image manipulation as described herein. The computers and / or ICs may calculate any of a variety of parameters or values ​​related to the dynamic spatial metadata of the images as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0064] A particular implementation of the invention comprises a computer processor executing software instructions that cause the processor to perform the inventive method. For example, one or more processors of a display, encoder, set-top box, transcoder, etc., may perform the above-described method relating to dynamic spatial metadata of images by executing software instructions in a program memory accessible to the processor. The invention may be provided in the form of a program product. The program product may comprise any tangible, non-transitory medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the inventive method. A program product according to the invention may be in any of a wide variety of tangible forms. The program product may comprise a physical medium, such as, for example, a magnetic data storage medium, including a floppy diskette or a hard disk drive, an optical data storage medium, including a CD ROM or a DVD, an electronic data storage medium, including a ROM or a flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0065] When a component (e.g., a software module, a processor, an assembly, a device, a circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including the reference to a "means") should be interpreted as including any component that performs the function of the described component (e.g., is functionally equivalent) as an equivalent of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in the represented embodiments of the invention.

[0066] [Equivalents, Extensions, Substitutes, and Others] Exemplary embodiments relating to dynamic spatial metadata of images have thus been described. In the above specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what is, and what is intended by the applicant to be, the invention is the set of claims issuing from this application, and the specific form in which such claims issue, including any subsequent amendments. The definitions expressly set forth herein for terms contained in such claims shall govern the meaning of the terms used in the claims. Thus, no limitations, elements, properties, features, advantages, or attributes not expressly recited in the claims shall limit the scope of such claims in any manner. Thus, the specification and drawings are to be regarded in an illustrative sense, and not in a restrictive sense.

[0067] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 62 / 250407 (Reference No. D21111USP1), filed September 30, 2021, and European Patent Application No. 21208718.3 (Reference No. D21111EP), filed November 17, 2021, the contents of each of which are incorporated herein by reference in their entirety.

Claims

1. 1. A method of processing spatial metadata by a processor, comprising: accessing an input image at an input spatial resolution and an input aspect ratio; calculating a first metadata parameter of the input image at a first spatial resolution equal to or smaller than the input spatial resolution; determining a second spatial resolution of the spatial metadata parameter, the second spatial resolution being smaller than the first spatial resolution; calculating global metadata parameters of the input image based on the first metadata parameters; smoothing the first metadata parameter to generate a smoothed metadata parameter; downsampling the smoothed metadata parameters to generate the spatial metadata parameters at the second spatial resolution; and The method according to claim 1,

2. receiving region of interest (ROI) parameters defining a ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; generating an ROI image based on the input image and the ROI parameters; performing an image processing operation on the ROI image based at least on pixel values ​​within the ROI image and the ROI metadata; The method of claim 1 further comprising:

3. generating a metadata bitstream based on the global metadata parameters and the spatial metadata parameters. The method of claim 1.

4. the second spatial resolution and the first spatial resolution have the same aspect ratio. The method of claim 1.

5. Smoothing the first metadata parameter comprises: applying a dilation filter to the first metadata parameters to generate dilated metadata parameters; applying an erosion filter to the dilation metadata parameters to generate erosion metadata parameters; applying a 2D spatial filter to the eroded metadata parameters to generate the smoothed metadata parameters; having The method of claim 1.

6. the 2D spatial filter comprises a 2D Gaussian filter; The method according to claim 5.

7. generating the metadata bitstream, calculating a metadata offset between the spatial metadata parameters and the global metadata parameters; generating the metadata bitstream based on the global metadata parameters and the metadata offset; having The method according to claim 3.

8. 1. A method for processing an image containing spatial metadata, comprising the steps of: receiving an input image at an input spatial resolution and image metadata, the image metadata including global metadata parameters for the entire input image and encoded spatial metadata parameters for two or more regions of the input image; generating spatial metadata parameters for the two or more regions based on the encoded spatial metadata parameters and the global metadata parameters; receiving region of interest (ROI) parameters defining a ROI image based on the input image; generating ROI metadata based on the spatial metadata parameters and the ROI parameters; generating an ROI image based on the input image and the ROI parameters; performing an image processing operation on the ROI image based at least on pixel values ​​within the ROI image and the ROI metadata; The method according to claim 1,

9. the encoded spatial metadata parameters comprise difference values ​​between the global metadata parameters and the spatial metadata parameters. The method according to claim 8.

10. generating the spatial metadata parameters includes subtracting the difference values ​​from the global metadata parameters; 10. The method of claim 9.

11. Generating the ROI metadata based on the spatial metadata parameters and the ROI parameters includes: upscaling the spatial metadata parameters to a spatial metadata resolution, a third spatial resolution, to generate upscaled metadata; extracting the ROI metadata from the upscale metadata based on the ROI parameters; having The method according to claim 2 or 8.

12. Generating the ROI metadata based on the spatial metadata parameters and the ROI parameters includes: generating edge spatial metadata parameters by interpolating between spatial metadata parameters in regions surrounding an edge of the ROI image; generating the ROI metadata based on spatial metadata parameters associated with a region within the ROI image and the edge spatial metadata parameters; having The method according to claim 2 or 8.

13. The image processing operation includes a tone mapping operation of the ROI image to a target display. The method according to claim 2 or 8.

14. and scaling ROI metadata parameters in the ROI metadata by a scaling factor prior to applying the ROI metadata to the image processing operation. The method according to claim 2 or 8.

15. the scaling factor is based on a difference between the ROI metadata parameters and the global metadata parameters; The method of claim 14.

16. the spatial metadata parameters represent pixel values ​​within a sub-region of the input image; The method according to claim 1 or 8.

17. The spatial metadata parameters represent minimum, average and maximum luminance values ​​within a sub-region of the input image.

17. The method of claim 16.

18. the spatial metadata parameters represent minimum, average and maximum color component values ​​within a sub-region of the input image; 17. The method of claim 16.

19. Apparatus comprising a processor and configured to perform the method of any one of claims 1 to 10.

20. A non-transitory computer readable storage medium having stored thereon computer executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 10.

Images