Pruning stage metadata prediction in video sequence using neural network

Abstract

Methods and systems for generating pruning phase metadata for high dynamic range (HDR) video are described. The pruning stage prediction pipeline includes a feature extraction network followed by a fully connected network that maps the extracted features to pruning stage values. In the first architecture, the feature extraction network is based on four cascaded convolutional networks. In the second architecture, the feature extraction network is based on a modified MobileNetV3 neural network. In both architectures, the fully connected network is formed from a set of three linear networks, each set being customized to optimally match its corresponding feature extraction network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 342,306 filed on May 16, 2022 and European Patent Application No. 22 182 506.0 filed on July 1, 2022, each of which is incorporated by reference in its entirety. Technical Field

[0003] The present invention relates generally to images. More particularly, embodiments of the present invention relate to techniques for predicting trim pass metadata in video sequences using a neural network. Background Art

[0004] As used herein, the term 'dynamic range' (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness, luminance signals) in an image (e.g., from the darkest gray (black) to the brightest white (highlight)). In this sense, DR relates to the intensity of a 'reference scene'. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular width. In this sense, DR relates to the intensity of a 'reference display'. Unless at any point in the description herein a particular meaning is expressly specified to have particular significance, it should be inferred that the term may be used in either sense (e.g., interchangeably).

[0005] As used herein, the term high dynamic range (HDR) refers to a DR width that spans about 14-15 orders of magnitude across the human visual system (HVS). In practice, the DR in the intensity range in which humans can simultaneously perceive a wide width may be slightly truncated relative to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may refer, individually or interchangeably, to the DR perceptible in a scene or image by the human visual system (HVS) including eye movement to allow for some light adaptation changes on the scene or image.

[0006] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits per pixel (e.g., n=8) of precision. For example, using gamma luma encoding, images where n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images where n≥10 can be considered enhanced dynamic range images. EDR and HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats such as the OpenEXR file format developed by Industrial Light and Magic.

[0007] Most consumer desktop monitors currently support 200 to 300 cd / m 2 Most consumer HDTVs range from 300 to 500 nits, with newer models reaching 1,000 nits (cd / m 2 ). Thus, such conventional displays represent a lower dynamic range (LDR) relative to HDR or EDR, also referred to as standard dynamic range (SDR). As the availability of HDR content grows due to advances in both capture equipment (e.g., cameras) and HDR displays (e.g., the PRM-4200 professional reference monitor from Dolby Laboratories), HDR content can be color graded and displayed on HDR displays that support a higher dynamic range (e.g., from 1000 nits to 5000 nits or more). In general, without limitation, the methods of the present disclosure relate to any dynamic range above SDR.

[0008] As used herein, the term "display management" refers to the processes performed on a receiver to render a picture to a target display. For example, and without limitation, such processes may include tone mapping, gamut mapping, color management, frame rate conversion, etc.

[0009] As used herein, the term "trim stage" refers to a video post-production process in which the colorist or creative personnel responsible for the content go through the master grading of the content shot by shot and adjust the lift-gamma-gain primaries and / or other color parameters to create a desired color or effect. Parameters associated with this process (e.g., lift, gain, and gamma values) may also be embedded within the video content as trim stage metadata or "trim" for later use as part of the display management process.

[0010] The creation and playback of high dynamic range (HDR) content is now becoming commonplace, as HDR technology provides more realistic and lifelike images than earlier formats. However, when converting HDR content to SDR content for legacy displays, the broadcast infrastructure may not support the generation and transmission of custom trims. To improve existing encoding schemes, as the inventors herein have realized, improved techniques for automatically generating trim stage metadata have been developed.

[0011] US2021 / 076042A1 discloses a method for generating metadata for use by a video decoder to display video content encoded by a video encoder, the method comprising accessing a target tone mapping curve; accessing a decoder tone curve corresponding to the tone curve used by the video decoder to tone map the video content; generating multiple parameters of a pruning stage function used by the video decoder to be applied after the decoder tone curve is applied to the video content, wherein the parameters of the pruning stage function are generated to approximate the target tone curve using a combination of the pruning stage function and the decoder tone curve, and generating metadata for use by the video decoder, including the multiple parameters of the pruning stage function.

[0012] US2021 / 350512A1 discloses a method for automatic display management generation for games or SDR+ content. Different candidate image data feature types are evaluated to identify one or more specific image data feature types to be used in training a predictive model for optimizing one or more image metadata parameters. Multiple image data features of one or more selected image data feature types are extracted from one or more images. The multiple image data features of one or more selected image data feature types are reduced to multiple significant image data features. The total number of image data features in the multiple significant image data features is not greater than the total number of image data features in the multiple image data features of the one or more selected image data feature types. Multiple significant image data features are applied to train a predictive model for optimizing one or more image metadata parameters.

[0013] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously thought of or pursued. Therefore, unless otherwise indicated, none of the approaches described in this section should be considered to qualify as prior art simply by virtue of their inclusion in this section. Similarly, unless otherwise indicated, problems identified with respect to one or more approaches should not be considered to have been recognized in any prior art based on this section. Summary of the invention

[0014] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0016] Figure 1A depicts an example process for pruning stage metadata for HDR video prediction according to an example embodiment of the present invention;

[0017] Figure 1BDescribes the training Figure 1A An example process of a neural network depicted in ;

[0018] Figure 2 depicts a first example of a neural network architecture for generating pruning stage metadata according to an example embodiment of the present invention; and

[0019] Figure 3 A second example of a neural network architecture for generating pruning stage metadata according to another example embodiment of the present invention is depicted. DETAILED DESCRIPTION

[0020] Methods for predicting pruning stage metadata for videos are described herein. In the following description, for the purpose of explanation, many specific details are set forth to provide a thorough understanding of the present invention. However, it will be clear that the present invention can be practiced without these specific details. In other cases, well-known structures and equipment are not described in detail to avoid unnecessarily obscuring, blurring or confusing the present invention.

[0021] Overview

[0022] The example embodiments described herein relate to a method for generating pruning-stage metadata for a video sequence. In an embodiment, a processor receives a picture in a video sequence. A feature extraction neural network extracts image features from the picture and then passes them to a fully connected neural network that maps the image features to output pruning-stage metadata values ​​for the input picture.

[0023] Pruning Phase Metadata Prediction Pipeline

[0024] In conventional display mapping (DM), a mapping algorithm applies a sigmoid-like function (e.g., see references [1-2]) to map the input dynamic range to the dynamic range of the target display. Such mapping functions can be expressed as piecewise linear or nonlinear polynomials characterized by anchor points, pivots, and other polynomial parameters generated using the characteristics of the input source and the target display. For example, in references [1-2], the mapping function uses anchor points based on the brightness characteristics of the input image and the display (e.g., minimum, median (average), and maximum brightness). However, other mapping functions may use different statistics, such as brightness variance or brightness standard deviation values ​​at the block level or for the entire image. For SDR images, the process can also be assisted by additional metadata, which are transmitted as part of the transmitted video, or they are calculated by the decoder or display. For example, when a content provider has both an SDR version and an HDR version of the source content, the source can use both versions to generate metadata (such as a piecewise linear approximation of a forward or backward shaping function) to assist the decoder in converting the incoming SDR image into an HDR image.

[0025] As used herein, the term "L1 metadata" refers to the minimum, middle, and maximum brightness values ​​associated with an input frame or image. The L1 metadata can be calculated by converting the RGB data into a brightness-chrominance format (e.g., YCbCr) and then calculating the minimum, middle (average), and maximum values ​​in the Y plane, or they can be calculated directly in the RGB space. For example, in an embodiment, L1Min represents the minimum value of the min(RGB) value of the PQ encoding of the image, while taking into account the valid area (e.g., by excluding gray or black bars, envelope bars, etc.). Min(RGB) represents the minimum value of the color component value {R, G, B} of the pixel. The values ​​of L1Mid and L1Max can also be calculated in the same manner by replacing the min() function with the average() and max() functions. For example, L1Mid represents the average value of the max(RGB) value of the PQ encoding of the image, and L1Max represents the maximum value of the max(RGB) value of the PQ encoding of the image. In some embodiments, the L1 metadata can be normalized to be in [0, 1].

[0026] Considering the L1Min, L1Mid and L1Max values ​​of the original HDR metadata, and the maximum (peak) and minimum (black) luminance of the target display denoted as Tmax and Tmin. Then, as described in references [1-2], an intensity tone mapping curve can be generated that maps the intensity of the input image to the dynamic range of the target display. This can be viewed as an ideal single-level tone mapping curve to be matched using the reconstructed metadata.

[0027] As mentioned previously, the term "clipping" refers to tone curve adjustments performed by colorists to improve tone mapping operations. Clipping is typically applied to the SDR range (e.g., 100 nits maximum brightness, 0.005 nits minimum brightness). These values ​​are then linearly interpolated to the target brightness range, which depends only on the maximum brightness. These values ​​modify the default tone curve and exist for each clip.

[0028] Information about clipping can be part of the HDR metadata and can be used to adjust the original tone mapping curve (see references [1-3]). For example, in Dolby Vision, clipping can be delivered as level 2 (L2) or level 8 (L8) metadata, which includes Slope, Offset, and Power variables (collectively referred to as SOP parameters) representing Gain and Gamma to adjust pixel values. For example, if Slope, Offset, and Power are in [-0.5, 0.5], then given Lift, Gain, and Gamma:

[0029]

[0030] In some content creation scenarios, it may not be possible to employ a full range of color grading tools to derive SDR content from HDR content. The embodiments described herein propose the use of a neural network-based architecture to automatically generate such trimming stage metadata. The trimming stage metadata is configured to perform adjustments to the tone mapping curve applied to the input picture when displayed on a target display. Although the examples presented herein primarily describe how to predict Slope, Offset, and Power values, a similar architecture can be applied to directly predict Lift, Gain, or Gamma, or other trimming as described in reference [3].

[0031] Figure 1A An example pruning-prediction pipeline (100) for an HDR image (102) is depicted in accordance with an embodiment. Figure 1A As depicted in FIG. 1 , the pipeline 100 includes the following modules or components:

[0032] HDR input (102)

[0033] Neural Networks for Feature Extraction (105)

[0034] A fully connected neural network (110) that maps the extracted features to metadata from the pruning phase

[0035] Pruning stage metadata output (112) (e.g., predicted SOP data)

[0036] The network (100) takes a given frame (102) as input and passes it to a convolutional neural network (105) for feature extraction, which identifies high-level features of the image, which are then passed to a fully connected network (110) that provides a mapping between features and pruning metadata (112). Each of the high-level features corresponds to an image feature type from a set of image feature types. Each image feature type is represented by a plurality of associated image features extracted from a corresponding set of training images and used to train the convolutional neural network (105) for feature extraction. This network can be used as a standalone component, or it can be integrated in a video processing pipeline. In order to maintain temporal consistency of the metadata at the pruning stage, the network can also take multiple frames as input.

[0037] In an embodiment, the pipeline is formed by neural network (NN) blocks that are trained to process HDR images encoded using perceptual quantization (PQ) in the ICtCp color space as defined in Rec. BT.2390 "High dynamic range television for production and international programme exchange". The ICtCp space is preferred because it is a perceptually uniform color space, thereby improving spatiotemporal consistency.

[0038] Figure 1B An example pipeline (130) for training network 100 is depicted. Compared to system (100), this system also includes:

[0039] Training input for real pruning phase metadata (132)

[0040] An error / loss calculation module (120) that calculates an error function (e.g., mean square error (MSE) or mean absolute error (MAE)) between the training data (132) and the predicted value (112)

[0041] • The back-propagation path (122) of training networks 105 and 110.

[0042] Pruning phase prediction architecture

[0043] Two different pruning stage prediction architectures (100) have been designed. They provide a balance between accuracy and speed. Figure 2 A first architecture according to an embodiment is depicted.

[0044] In an embodiment, a neural network may be defined as a set of 4-dimensional convolutions, each of which is followed by adding a constant bias value to all results. In some layers, the convolutions are followed by clamping negative values ​​to 0. Convolutions are defined by their pixel size (M×N), the number of image channels they operate on (C), and the number of such kernels in the filter bank (K). In this sense, each convolution may be described by the size of the filter bank M×N×C×K. As an example, a filter bank of size 3×3×1×2 consists of 2 convolution kernels, each of which operates on one channel and has a size of 3 pixels×3 pixels. Input and output sizes are expressed as height×width×channels.

[0045] Some filter banks can also have a stride, which means that some of the results of the convolution are thrown away. A stride of 1 means that every input pixel produces an output pixel. A stride of 2 means that only every other pixel in each dimension produces output, etc. Therefore, a filter bank with a stride of 2 will produce an output of (M / 2)×(N / 2) pixels, where M×N is the input image size. If padding=1, then all inputs except the input to the fully connected kernel are padded, so that setting the stride to 1 will produce an output with the same number of pixels as the input. The output of each convolutional group is fed as input to the next convolutional layer.

[0046] like Figure 2 As depicted in FIG. 1 , in a first architecture, the feature extraction network (105) includes four convolutional networks configured as follows:

[0047] Convolution 1: Input 540×960×1, 3×3×1×4, stride=2, output: 270×480×4,

[0048] Bias, Rectified Linear Unit (ReLU) activation

[0049] Convolution 2: 3×3×4×8, stride=2, bias, ReLU activation, output: 135×240×8

[0050] Convolution 3: 7×7×8×16, stride=5, bias, ReLU activation, output: 27×48×16

[0051] Convolution 4: 27×48×16×3 (fully connected), stride=1, bias, output: 1×1×3 After the feature extraction network, the fully connected network (110) includes three linear networks, which are configured as follows:

[0052] Linear 1: Input: 1×3, Output 1×6, Batch Norm, ReLU, DropOut

[0053] Linear 2: Input: 1×6, Output 1×6, Batch Norm, ReLU, DropOut

[0054] Linear 3: Input: 1×6, Output 1×3

[0055] DropOut refers to using Dropout regularization, and BatchNorm refers to applying batch normalization (reference [5]) to increase training speed.

[0056] Figure 3 An example of the second architecture is depicted. Figure 3 As depicted in , in the second architecture, the feature extraction network (105) includes a modified version of the MobileNet_V3 network, which was originally described in reference [4] for images with a 1:1 aspect ratio, but is now extended to operate on images of arbitrary aspect ratios. Table 1 (based on Table 2 in reference [4]) provides additional details of the operations in the modified MobileNetV3 architecture for an example 540×960×3 input. Compared to reference [4], after the second conv2d, 1×1 stage, the original pooling, 7×7 stage is replaced by an average pooling, 1×1 stage to generate the final output (e.g., 1×576), and the two subsequent conv2d, 1×1 non-batch normalization (NBN) stages are removed.

[0057] In Table 1, "bneck" indicates bottleneck and inverted residual blocks, Exp size indicates the expansion block size, SE indicates whether squeeze and excitation are enabled, NL indicates a nonlinear activation function, HS indicates a hard-swish activation function, RE indicates a ReLU activation function, and s indicates a step size. The output of a neural network stage matches the input of the subsequent neural network stage.

[0058] Table 1. Specifications of MobileNetV3 (Small)

[0059]

[0060] After network 105, the fully connected network (110) includes three linear networks, which are configured as follows:

[0061] Linear 1: Input: 1×576, Output: 1×128, Batch Norm, ReLU,

[0062] DropOut

[0063] Linear 2: Input: 1×128, Output: 1×64, Batch Norm, ReLU, DropOut

[0064] Linear 3: Input: 1×64, Output: 1×3

[0065] As described in [4], MobileNetV3 is adapted to mobile phone CPUs for object detection and semantic segmentation (or dense pixel prediction). It uses a variety of innovative tools, including:

[0066] Deep convolutional filters

[0067] Bottleneck and inverted residual blocks (bneck)

[0068] Squeeze and excite (SE) block (reference [6]); and

[0069] Hard-Swish (HS) activation function, which is defined as

[0070]

[0071] Wherein ReLU6 represents an activation function known in the art.

[0072] Fully connected network

[0073] The output of the feature extraction module (105) is fed to a fully connected network to get the predicted pruning values. Based on the architecture of the feature extraction module, the fully connected network can have different input sizes and therefore may require different output sizes. This module learns the mapping between the high-level features extracted from the image and its associated Slope, Offset and Power. As mentioned above, the same network design can be used to directly calculate Lift, Gain and Gamma, or any other parameters related to the pruning stage metadata.

[0074] In terms of complexity, the first architecture is simpler and less computationally intensive than the MobileNet-based architecture; however, the MobileNet-based architecture is more accurate.

[0075] The second architecture can directly take three input channels (eg, RGB or ICtCp); however, it is straightforward to modify the first architecture to also consider chrominance by allowing three input channels instead of just one.

[0076] Network training

[0077] like Figure 1BAs depicted in , during training, during the error calculation (120), an L1 loss (e.g., MAE) or an L2 loss (e.g., MSE) may be applied. This loss is calculated between the predicted pruning stage metadata values ​​(112) and the true pruning stage metadata values ​​(132). Alternatively, a tone curve may be used as a loss function. Essentially, the pruning stage metadata values ​​may be used together with other metadata values ​​(such as L1) and a tone curve corresponding to the image may be defined. Then, the L1 loss or L2 loss between the tone curves obtained using the predicted pruning stage metadata and the true pruning stage metadata may be minimized.

[0078] For example, let f true (i) represents the tone curve generated by using L1 metadata as input (102) and training the pruning stage metadata (132), and let f pred (i) represents the tone curve generated using the same L1 metadata and predicted pruning stage metadata (112). Then, during training, one may wish to minimize the mean squared error, which is defined as:

[0079]

[0080] Where |i| represents the cardinality of the i value for which the MSE is calculated. Alternatively, the L1 metric can be applied as follows:

[0081]

[0082] References

[0083] Each of these references is incorporated by reference in its entirety.

[0084] 1. A. Ballestad and A. Kostin, U.S. Patent 8,593,480, “Method and apparatus for image data transformation”

[0085] 2. J.A. Pytlarz and R. Atkins, U.S. Patent 10,600,166, “Tone curve mapping for high dynamic range image.”

[0086] 3. “Module 2.8–The Dolby Vision metadata Trim Pass,” https: / / learning.dolby.com / hc / en-us / articles / 360056574431-Module-2-8-The-Dolby- Vision-Metadata-Trim-Pass- , downloaded on May 3, 2022.

[0087] 4. A. Howard et al., “Searching for MobileNetV3,” Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019, see also arXiv:1905.02244v5, 20 November 2019.

[0088] 5. S. Ioffe and C. Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." International conference on machine learning. PMLR, 2015. See also arXiv:1502.03167v3, 2 March 2015.

[0089] 6. J. Hu et al., “Squeeze-and-excitation networks,” arXiv:1709.01507v4, May 16, 2019.

[0090] Example Computer System Implementation

[0091] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC)), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to image transformations (such as those described herein). The computer and / or IC may calculate any of the various parameters or values ​​involved in the pruning stage metadata prediction process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0092] Certain implementations of the invention include a computer processor that executes software instructions that cause the processor to perform the method of the invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement methods related to the pruning phase metadata prediction process as described above by executing software instructions in a program memory accessible to the processor. The invention can also be provided in the form of a program product. The program product may include any tangible and non-transitory medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of the invention. The program product according to the invention may be any of a variety of tangible forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAMs, etc. The computer-readable signals on the program product may be optionally compressed or encrypted.

[0093] When referring to a component (e.g., a software module, a processor, an assembly, a device, a circuit, etc.) above, unless otherwise specified, reference to that component (including reference to a "part") should be interpreted to include equivalents of that component, any component that performs the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0094] Equivalence, extensions, substitutions, and more

[0095] Thus described are example embodiments involving a pruning phase metadata prediction process. In the foregoing specification, embodiments of the present invention have been described with reference to many specific details, which may vary between implementations. Therefore, the sole and exclusive indication of what the invention is and what the applicant intends to invent is a set of claims, which are issued from this application in the specific form in which such claims are issued (including any subsequent corrections). Any definitions expressly set forth herein for terms contained in such claims will affect the meaning of such terms when used in the claims. Therefore, any limitation, element, property, feature, advantage or attribute not expressly recited in a claim should not limit the scope of such claim in any way. The specification and drawings are accordingly to be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for generating cropping-stage metadata for a picture in a video sequence, wherein the cropping-stage metadata is configured to perform an adjustment of a tone mapping curve applied to an input picture when displayed on a target display, the method comprising: Receiving the input picture; Providing a feature extraction network, the feature extraction network comprising a convolutional neural network for feature extraction, the convolutional neural network being trained to recognize high-level image features of the input picture; Applying the feature extraction network to the input image to generate the image features; providing a fully connected network, the fully connected network comprising a plurality of cascaded linear neural networks, the plurality of cascaded linear neural networks being trained to map the image features to output pruning-stage metadata values ​​for the input picture; as well as The fully connected network is applied to the image features to map the image features to the output pruning-stage metadata values ​​for the input picture.

2. The method of claim 1, wherein the input picture is a high dynamic range (HDR) picture encoded using PQ coding in an ICtCp color space.

3. The method of claim 1 or claim 2, wherein the feature extraction network comprises four cascaded convolutional networks.

4. The method of claim 3, wherein the fully connected network comprises three cascaded linear networks.

5. The method of any one of claims 1-4, wherein the feature extraction network comprises a modified MobileNetV3 neural network that accepts inputs with non-square aspect ratios. The method of claim 5 , wherein the fully connected network comprises three cascaded linear networks.

7. The method according to any one of claims 1 to 6, further comprising: Receiving input training pruning phase parameters corresponding to the input picture; applying an error loss unit to generate an error metric based on the input training pruning phase parameters and the output pruning phase metadata; as well as The feature extraction network and the fully connected network are trained by minimizing the error metric.

8. The method of claim 7, wherein calculating the error metric comprises calculating a minimum absolute error or a mean square error between the input training pruning phase parameters and the output pruning phase metadata.

9. The method of claim 7, wherein calculating the error metric comprises: generating a first tone mapping function based at least on the input training pruning phase parameters; generating a second tone mapping function based at least on the output pruning stage metadata; as well as The minimum absolute error or mean square error between the value of the first tone mapping function and the value of the second tone mapping function is calculated.

10. An apparatus comprising a processor and configured to execute any one of the methods recited in claims 1-9.

11. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for executing the method according to any one of claims 1-9 using one or more processors.

Images