Enforcing Minimum and Average Distance Constraints in Video Streaming

Abstract

An input image is received as an input to the multi-node system. The input image is divided into segments that are assigned to respective nodes of the multi-node system. Primary and secondary scenes are identified within the segments to ensure compliance with minimum and average distance constraints. A scene-level forward reconstruction mapping is generated for the scene by each node of the assigned segment. Forward reconstructed images within the segments are generated by the nodes using the forward reconstruction mapping and encoded into an output video signal, which enables a receiving device to generate a reconstructed image and render a display image derived from the reconstructed image on an image display.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to European Patent Application No. 21191429.6, filed August 16, 2021, and U.S. Provisional Patent Application No. 63 / 233,458, filed August 16, 2021, each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to image processing operations. More particularly, one embodiment of the present disclosure relates to video codecs. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image (e.g., from darkest black (shadows) to brightest white (highlights)). In this sense, DR relates to scene-referred intensities. DR may also relate to the ability of a display device to reasonably or approximately render an intensity range having a certain width. In this sense, DR relates to display-referred intensities. Anywhere in this specification, the term may be used in either sense (e.g., interchangeably) unless a particular meaning is specifically and explicitly specified.

[0004] In this specification, the term high dynamic range (HDR) refers to a DR width that spans 14-15 orders of magnitude or more in the human visual system (HVS). In practice, the DR that humans can simultaneously perceive a wide range of intensities may be somewhat truncated for HDR. In this specification, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR), individually or indistinguishably, refer to the DR that the human visual system (HVS) (including eye movements and allowing some degree of photopic change across a scene or image) can perceive in a scene or image. In this specification, EDR may refer to a DR that spans 5-6 orders of magnitude. Although somewhat narrower for true scene-referred HDR, EDR represents a wide DR width and may also be referred to as HDR.

[0005] In practice, an image has one or more color components / channels of a color space (e.g. luma Y and chroma Cb and Cr), each color component / channel being represented with n bits of precision per pixel (e.g. n=8). When using non-linear luminance coding (e.g. gamma coding), images with n≦8 (e.g. color 24-bit JPEG images) are said to be standard dynamic range images, while images with n>8 are said to be enhanced dynamic range images.

[0006] The reference electro-optical transfer function (EOTF) for a display characterizes the relationship from the color values ​​of the input video signal (e.g., luminance represented in the codewords representing the image) to the output screen brightness (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT. 1886, “Reference electro-optical transfer function for flat panel displays used in HDTV studio production” (March 2011), which is incorporated herein by reference in its entirety, defines the reference EOTF for flat panel displays. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. In this specification, the term “metadata” refers to any auxiliary information that is transmitted as part of the encoded bitstream and helps a decoder to render the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

[0007] In this specification, the term "PQ" refers to the quantization of perceptual luminance amplitude. The human visual system (HVS) responds to increasing light levels in a highly nonlinear manner. A human's ability to see a stimulus is affected by the luminance of the stimulus, its size, the spatial frequencies that compose the stimulus, and the luminance level to which the eye has adapted up to the moment the stimulus is viewed. In some embodiments, the perceptual quantizer function maps linear input gray levels to output gray levels that are more closely matched to the contrast sensitivity thresholds in the human visual system. An example of a PQ mapping function is described in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter "SMPTE"), the disclosure of which is incorporated herein in its entirety. Here, for a fixed stimulus size, for each luminance level (e.g., stimulus level), the minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0008] 302cd / m 2 ~1000cd / m 2Displays supporting 100 nits of luminance are representative of low dynamic range (LDR), also referred to as standard dynamic range (SDR) as opposed to EDR (or HDR). EDR content may be displayed on EDR displays supporting a higher dynamic range (e.g., 1000 nits to 5000 nits or more). Such displays may be defined with an alternative EOTF supporting higher luminance capabilities (e.g., 0 nits to 10000 nits or more). Exemplary (e.g., HDR, Hybrid Log Gamma, or HLG) EOTFs are defined in SMPTE 2044 and Rec. ITU-R BT.2060, “Image parameter values ​​for high dynamic range television for use in production and international programme exchange” (06 / 2017). See also ITU Rec. ITU-R BT.3040-2, "Parameter values ​​for ultra-high definition television systems for production and international programme exchange," October 2015, which is incorporated herein by reference in its entirety and relates to the Rec. 3040 or BT.3040 color space. As appreciated by the inventors herein, it is desirable to improve techniques for coding high quality video content data that is rendered using a wide variety of display devices.

[0009] EP 3510772 discloses a segment-based method for generating backward-compatible reconstructed SDR video that preserves the expressive intent or "look" of a given EDR input. For each frame in the segment, a reconstruction function is generated based on a supporting frame set determined based on a sliding window of frames that is adjusted based on scene cuts in the segment and may include frames from both the current segment and adjacent segments. For luma reconstruction, a mapping that preserves the cumulative density function of luminance histogram values ​​in the EDR and SDR inputs is combined with a minimum codeword allocation derived based on the EDR signal and the supporting frame set. For chroma reconstruction, a method of segment-based forward and backward reconstruction using a multivariate multiple regression analysis model is also presented.

[0010] The approaches described in this section could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise specified, it should not be assumed that any of the approaches described in this section qualify as prior art by virtue of their inclusion in this section. Similarly, it should not be assumed, based on this section, that problems identified with one or more approaches have been recognized in any prior art, unless otherwise specified. Summary of the Invention

[0011] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments of the invention.

[0012] An embodiment of the present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference symbols refer to similar elements and in which: [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 illustrates an example video / image processing pipeline.

[0014] [Figure 2A] FIG. 1 illustrates an example process flow for performing tasks in two passes. [Figure 2B] FIG. 1 illustrates an exemplary process flow for performing tasks in a second of two passes. [Figure 2C] FIG. 2 illustrates an exemplary process flow by which a node traverses an input image / frame. [Figure 2D] FIG. 1 illustrates an example process flow for preemptive action to avoid average distance constraint violations. [Figure 2E] FIG. 1 illustrates an exemplary process flow for a variable length scene splitting operation. [Figure 2F] FIG. 1 illustrates an exemplary process flow for iterative overlap. [Figure 2G] FIG. 1 illustrates an example process flow for forward look-up table (FLUT) interpolation of secondary scenes. [Figure 2H] FIG. 2 illustrates an example process flow for scene-based encoding and related operations. [Figure 2I] FIG. 2 illustrates an example process flow for scene-based encoding and related operations. [Figure 2J] FIG. 2 illustrates an example process flow for scene-based encoding and related operations. [Figure 2K] FIG. 2 illustrates an example process flow for scene-based encoding and related operations.

[0015] [Figure 3A] A diagram showing example fixed-length segments assigned to each node in multiple computing nodes (or a cloud-based computer) in a multi-node video encoding system. [Figure 3B] FIG. 2 illustrates exemplary scene cuts identified by a video professional. [Figure 3C]FIG. 2 illustrates an exemplary manual automatic scene cut identified by automatic scene detection. [Figure 3D] FIG. 2 illustrates an example merged scene cut. [Figure 3E] FIG. 2 illustrates a first list of exemplary scenes and mandatory scene cuts. [Figure 3F] FIG. 13 illustrates an example scene cut after removing the first encountered short scene cut with a single traversal direction approach. [Figure 3G] A diagram showing example operations and scene cuts after removing short scene cuts in a multi-traversal direction approach. [Figure 3H] A diagram showing example operations and scene cuts after removing short scene cuts in a multi-traversal direction approach. [Figure 3I] FIG. 2 illustrates an exemplary fixed-length scene partitioning. [Figure 3J] FIG. 2 illustrates an exemplary distribution of key scenes. [Figure 3K] FIG. 1 illustrates an example parent scene distributed across two nodes. [Figure 3L] FIG. 13 illustrates an exemplary statistics window including a bumper image / frame. [Figure 3M] A diagram illustrating an example operation of splitting portions of a primary / parent scene into secondary scenes across two nodes. [Figure 3N] FIG. 2 illustrates an exemplary anchor sub-scene. [Figure 3O] FIG. 1 illustrates an example segment in a fixed length split operation. [Figure 3P] FIG. 1 illustrates an example segment in a fixed length split operation. [Figure 3Q] FIG. 1 illustrates an example segment in a fixed length split operation. [Figure 3R] FIG. 1 illustrates an example primary / parent scene whole that is entirely within a segment assigned to a node. [Figure 3S]FIG. 2 illustrates an exemplary sub-scene split from a primary / parent scene. [Figure 3T] FIG. 2 illustrates an exemplary sub-scene split from a primary / parent scene. [Figure 3U] FIG. 2 illustrates an exemplary sub-scene split from a primary / parent scene. [Figure 3V] FIG. 2 illustrates an exemplary anchor sub-scene in variable length scene partitioning. [Figure 3W] FIG. 2 illustrates an exemplary anchor sub-scene in variable length scene partitioning. [Figure 3Z] FIG. 1 illustrates an exemplary temporal violation in a frame direct current (DC) value. [Figure 3AA] FIG. 2 illustrates an example partition of a primary or parent scene. [Figure 3AB] A diagram showing example DC changes across a partition of a primary or parent scene after adjustment and / or offset. [Figure 3AC] A diagram showing example DC changes across a partition of a primary or parent scene after adjustment and / or offset. [Figure 3AD] A diagram showing example DC changes across a partition of a primary or parent scene after adjustment and / or offset. [Figure 3AF] A diagram showing example DC changes across a partition of a primary or parent scene after adjustment and / or offset. [Figure 3A-G] A diagram showing an example initial scene cut after short scene merging and an example final scene cut after scene splitting. [Figure 3AH] FIG. 1 illustrates an example forward reconstruction statistics window in two segments assigned to two nodes. [Figure 3AI] FIG. 13 illustrates example display management (DM) and cumulative density function (CDF) curves before and after trim correction.

[0016] [Figure 4A] FIG. 1 illustrates an exemplary process flow. [Figure 4B] FIG. 1 illustrates an exemplary process flow.

[0017] [Diagram 5] FIG. 1 is a simplified block diagram illustrating an example hardware platform on which a computer or computing device described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following description, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the present disclosure, although it will be apparent that the present disclosure may be practiced without these details. In other instances, detailed details of well-known structures and devices are not described in order to unnecessarily clutter, obscure, or obfuscate the description of the present disclosure.

[0019] overview An operation mode referred to as Minimum and Average Distance Constraint (MADC) mode may be implemented or operated by a video codec. Exemplary video codecs described herein may include, but are not necessarily limited to, single layer backward compatible (SLBC) codecs. MADC may be used to ensure a minimum image metadata (e.g., reference processing unit data or RPU data, etc.) refresh time interval and maintain an average number of image metadata refreshes in a given time interval while encoding fixed-length video segments in an upstream video encoding system, such as a cloud-based multi-node video encoding system.

[0020] In some operating scenarios, the MADC mode may be implemented or operated by an SLBC video encoder designed to generate a composite video bitstream, each of which includes SDR video content / data optimized for viewing on an SDR image display / screen and image metadata that enables reconstructing, from the SDR video content / data, HDR video content / data optimized for viewing on an HDR (or compatible) display / screen. The composite video bitstream may include a base layer video stream carrying the SDR video content / data as well as other streams / containers to carry some or all image metadata used to map lower dynamic range base layer video content / data, such as SDR video content / data, to higher dynamic range reconstructed video content / data, such as reconstructed HDR video content / data for an HDR display.

[0021] To achieve the best visual quality on an HDR display, dynamic image metadata that may change from image / frame to frame may be encoded in the video signal. Some receiving devices, such as relatively low-end battery-powered mobile devices, may have difficulty keeping up with the rapidly changing image metadata that fluctuates from frame to frame in the video signal. As a result, some video images / frames and / or their corresponding image metadata may be dropped by these receiving devices.

[0022] These receiving devices can function more reliably the less frequently the image metadata changes. It can be difficult to maintain reasonable visual quality while allowing image metadata to change only after a specified number (sometimes referred to as a minimum distance constraint of (consecutive) images / frames). The minimum distance constraint (or minimum distance) can be specifically selected to provide additional decoding time to avoid frame drops that may otherwise occur due to incorrect display timing caused by the extra decoding time at the decoder side. Decoding, for example, a "refresh" composer metadata portion may require additional computation / time to build lookup table(s) (e.g., three-dimensional lookup table(s) or 3D-LUT(s), etc.) used in the decoding operation at the decoder side. If the distance between two temporally adjacent "refresh" composer metadata portions is too small, there may not be enough time to allow for computing or building the lookup table(s) or 3D-LUT(s), thereby resulting in playback jitter and even frame drops. By way of example and not limitation, the minimum distance constraint, or minimum distance, may be selected to provide additional decoding time to avoid frame drops that may otherwise occur due to incorrect display timing caused by the extra decoding time at the decoder side. min The minimum distance, denoted as , can be 1, 4, 8, or 10 frames or more.

[0023] The bitrate of dynamic image metadata (e.g. composer metadata, display management or DM metadata, etc.) can increase significantly with increasing frame rate. For a relatively low resolution base layer, the image metadata overhead per frame can be relatively large. MADC mode not only helps to keep the image metadata constant for a certain number of images / frames, but also reduces the image metadata bitrate at the same time.

[0024] In some operating scenarios, for a (current) image / frame, the composer metadata that points to a portion of the image metadata used to reconstruct a higher dynamic range image from a corresponding lower dynamic range image carried in the base layer can point to previous composer metadata for one of the previous frames preceding that image / frame. This can be done by simply setting the composer metadata ID for one of the previous frames in the current image / frame (e.g., in its non-payload header, etc.) without any image metadata payload, thereby achieving a relatively high composer metadata compression.

[0025] In addition to keeping or enforcing image metadata to remain unchanged or fixed for a number of consecutive image / frame chunks, the MADC mode can be implemented or operated to deliver a further reduced bitrate for conveying image metadata bitrate, for example, by controlling the average number of image metadata changes in a given time interval or duration, which may be referred to as an average distance constraint. Reducing the metadata update frequency saves valuable battery life in a receiving device, such as a relatively low-end mobile phone. The average distance constraint (or average distance) may be specifically selected to comply with a desired or allocated power consumption and / or to comply with a desired or allocated bitstream overhead. For example, building a 3D-LUT consumes computational power. The higher the average distance for refreshing composer metadata, the lower the power consumed by the decoding device, and the less composer metadata present in the coded bitstream, the lower the bitstream overhead. By way of example and not limitation, the average distance constraint or D avg The average distance, denoted as , may be 12 frames or more or less.

[0026] For images / frames belonging to the same scene, the image metadata may change little from one frame to the next. The same image metadata or the same set of metadata parameters may be used for all images / frames in a scene. The first image / frame in a scene may be given the (fresh) image metadata in the video signal, while the remaining images / frames in the scene may be signaled to use or reuse the image metadata already provided in the video signal for the first image / frame. In an operating scenario where all scenes have a total number of images / frames greater than the minimum distance and greater than the average frame number, both the minimum distance constraint and the average distance constraint may be satisfied simultaneously and still achieve consistent visual quality for all images / frames of the scene.

[0027] In many, but not all, operating scenarios, D avg ≧D min For example, the average distance constraint may be 12 frames and the minimum distance constraint may be 10 frames. Thus, assuming that the entire sequence of images / frames is encoded consecutively by a single processing node (or a single video coding node), every scene should have 10 or more frames to satisfy the minimum distance constraint and 12 or more frames on average to satisfy the average distance constraint.

[0028] An upstream encoding system may include multiple processing nodes (or multiple video coding nodes). A sequence of images / frames that make up a scene may be partitioned into multiple fixed-length segments, assigned to multiple nodes, and coded in parallel using the multiple nodes.

[0029] By partitioning the same scene into multiple fixed-length segments, one part of a scene can be one node and a different part of the same scene can be a different node. As a result, satisfying both the minimum and average distance constraints becomes a relatively complex task, since extra (e.g., fragmented, partial) scenes (or fragments for short) may be created.

[0030] Furthermore, each node assigned to process fragments of the same (original) scene may not be fully aware of those parts or fragments in the same (original) scene assigned to other nodes, and it may become impossible to send a single set of the same image metadata or metadata parameters for all images / frames of the original scene.

[0031] Although a node may send the same image metadata for all images / frames in a fragment or partial scene to which it has access, image metadata for different fragments or different partial scenes encoded by different nodes relative to the entire original scene may have significantly, if not significantly, different metadata parameters (e.g., reconstruction parameters, etc.) because these metadata parameters are calculated on different partial scene video data by different nodes.

[0032] More specifically, a first node may encode a first image / frame belonging to an original scene, as well as a first metadata parameter of the first image / frame. A second, different node may encode a second image / frame (e.g., a next scene of the first image / frame, etc.) belonging to the same original scene, as well as a second metadata parameter of the second image / frame. Since two different nodes may accumulate different image-related statistics in their respective encoding operations, the first metadata parameter encoded by the first node for the first image / frame may be significantly different from the second metadata parameter encoded by the second node for the second image / frame. As a result, when a receiving side uses these metadata parameters to render the first and second images, the first image / frame may have a relatively large visual difference compared to the second image / frame. These visual differences may result in temporal inconsistencies in the visual quality of the images / frames, rapid or sudden brightness changes, flashing, flickering, color changes, visual artifacts, etc.

[0033] Exemplary embodiments described herein relate to encoding video images. A sequence of input images and one or more input lists of scenes in an input video signal are received as inputs to a multi-node video encoding system having a plurality of video encoding nodes. The one or more input lists of scenes collectively identify a plurality of primary scenes in the sequence of input images. The sequence of input images is divided into a plurality of non-overlapping segments. Each segment in the plurality of non-overlapping segments is assigned to a respective node in the plurality of video encoding nodes. Each node receives an input image in the segment as its intra-segment image of the segment and receives an input image in one or two bumper sections of the segment as bumper images of the segment. Compliance with the minimum distance constraint and the average distance constraint is ensured by generating a second list of scenes using the plurality of primary scenes identified in the one or more input lists of scenes and segment boundaries defining adjacent segments in the plurality of non-overlapping segments. The second list of scenes identifies one or more intra-segment scenes in the segment. The one or more intra-segment scenes include zero or more primary scenes that are not on any segment boundary of the segment, and one or more secondary scenes split from the one or more primary scenes on one or both segment boundaries of the segment. One or more scene-level forward reconstruction mappings are generated for the one or more secondary scenes. Each scene-level forward reconstruction mapping of the one or more scene-level forward reconstruction mappings is generated at least in part from reconstruction statistics collected from input images within an overlap window determined for each secondary scene of the one or more secondary scenes in the segment. The forward reconstructed images in the segment are encoded by the respective nodes into an output video signal collectively encoded by the multiple video encoding nodes. The output video signal includes image metadata portions generated by the respective nodes and enables a receiving device to generate a reconstructed image from the forward reconstructed images in the segment and render a display image derived from the reconstructed image on an image display.

[0034] Exemplary embodiments described herein relate to decoding video images. A sequence of forward reconstructed images is decoded from a video signal. The video signal has been generated by a plurality of video encoding nodes in a multi-node video encoding system. The video signal includes image metadata generated by the plurality of video encoding nodes. The sequence of forward reconstructed images includes forward reconstructed images encoded into the video signal by a node of the plurality of video encoding nodes for a segment of a plurality of segments used to partition a sequence of source images. The sequence of forward reconstructed images includes segments of forward reconstructed images generated by a node of the multi-node video encoding system using a forward reconstruction function of one or more scenes identified in a segment of a pre-reconstructed source image. The one or more scenes identified in the segment of the pre-reconstructed source image have been guaranteed by the node to comply with a minimum distance constraint and an average distance constraint. An image metadata portion in the image metadata is used to generate a reconstructed image from the forward reconstructed images in the segment. The image metadata portion has been generated by the node. A display image derived from the reconstructed images is rendered on an image display.

[0035] An Example Image Processing Pipeline FIG. 1 illustrates an exemplary process of a video / image delivery pipeline 100 showing various stages from video capture to video content display. A series of video frames 102 are captured or generated using an image generation block 105. The video frames 102 can be captured digitally (e.g., by a digital camera, etc.) or generated by a computer (e.g., by computer animation, etc.) to provide video data 107. Additionally, optionally, or alternatively, the video frames 102 can be captured on film by a film camera. The film can be converted to a digital format to provide the video data 107. The film can be converted to a digital format to provide the video data 107. Additionally, optionally, or alternatively, some or all of the video frames 102 can be partially or fully generated by a computer using image rendering / generation and / or computer graphics techniques. In a production phase 110, the video data 107 is edited to provide a video production stream 112.

[0036] The video data of the production stream 112 is then provided to a processor for post-production editing 115. Post-production editing 115 may include adjusting or modifying (e.g., automatically, manually, partially automatically, partially manually, etc.) the color or brightness of certain regions of an image to enhance the image or achieve a particular look according to the video creator's creative intent. This may be referred to as "color timing" or "color grading." Other editing (e.g., scene selection and alignment, manual and / or automatic scene cut information generation, image cropping, addition of computer-generated visual special effects, etc.) may be performed in post-production editing 115 to bring the original input video signal 117 (e.g., HDR image, etc.) through content mapping and / or color grading to a coding block 120 collectively implemented by multiple video encoding nodes in a multi-node video encoding system. Additionally, optionally or alternatively, the post-production editing 115 can provide a reference video signal 117-1 (such as an SDR image 117-1 depicting the same visual semantic content as the HDR image in the original input video signal) to the coding block 120 from the video data of the production stream 112 through content mapping and / or color grading. The reference color grade 117-1 may be derived from the original input video signal through content mapping and / or color grading performed manually, automatically, or with a combination of manual and automatic image processing operations.

[0037] The original input video signal and / or the reference video signal 117-1 may be used by the coding block 120 or a multi-node coding system to perform segment-based video encoding and generate an output video signal 122. The output video signal 122 may include reconstructed images that are the same as or closely approximate (e.g., through minimization of prediction error, through closed-form solutions to unconstrained or constrained optimization problems, etc.) images in the reference video signal 117-1. An exemplary reconstruction operation is described in U.S. Patent No. 10,080,026, entitled "Signal reshaping approximation" by GM. Su et al., the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0038] For illustrative purposes only, the original input video signal 117 represents an input color grade of an HDR image. The reference video signal 117-1 represents a reference color grade of a reference image that is approximated by a reconstructed color grade represented in the output video signal 122. In some operating scenarios, the reference image may be content mapped (e.g., using an appropriate profile of a video coding tool, such as a Dolby Vision coding tool, commercially available from Dolby Laboratories, San Francisco, Calif., etc.) from an HDR image in the original input video signal 117. In some operating scenarios, during post-production editing 115, the HDR image in the input video signal 117 is viewed or reviewed on an HDR reference display supporting high dynamic range by a colorist performing post-production editing operations on the HDR image. Additionally, optionally, or alternatively, in some operating scenarios, during post-production editing 115, the SDR image in the reference video signal 117-1 is viewed or reviewed on an SDR reference display supporting standard dynamic range (e.g., 125 in FIG. 1 ).

[0039] Coding block 120 may implement some or all of the multi-node segment-based video coding operations as described herein to generate reconstruction (e.g., forward, reverse, etc.) operating parameters for mapping input images in the input video signal to reconstructed images and / or for mapping the reconstructed images to reconstructed images that approximate the input images. The reconstructed images (e.g., forward, etc.) may be compressed / encoded by coding block 120 into an output video signal 122 or a coded bitstream representing the output video signal 122. Some or all of the operating parameters of the reconstruction function (e.g., reverse reconstruction function, inverse reconstruction function, etc.) may be included in the same coded bitstream or may be coded as part of the image metadata.

[0040] The coding block 120 may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate a coded bitstream 122.

[0041] In some operating scenarios, the output video signal 122 may represent a video signal (e.g., an 8-bit SDR video signal, a 10-bit SDR video signal, etc.) that is backward compatible with a wide variety of SDR display devices (e.g., SDR displays, etc.). In a non-limiting example, the video signal encoded with the reconstructed SDR image may be a single-layer backward-compatible video signal. Here, a "single-layer backward-compatible video signal" may refer to a video signal that carries an SDR image specifically optimized or color-graded for an SDR display in a single signal layer. An exemplary single-layer video coding operation is described in U.S. Patent Application Publication No. 2019 / 0110054, entitled "Encoding and decoding reversible production-quality single-layer video signals," by GM. Su et al., the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0042] The output video signal 122 is then distributed downstream to a receiver, such as a mobile device, a handset, a tablet computer, a decoding and playback device, a media source device, a media streaming client device, a television set (e.g., a smart TV, etc.), a set-top box, a movie theater, etc. At the receiver (or downstream device), the output video signal 122 is decoded by a decoding block 130 to generate a decoded image 182, which may be the same as the image (e.g., a forward reconstructed SDR image, etc.) encoded into the bitstream 122 by the coding block 120, but subject to quantization errors that occur in the compression performed by the coding block 120 and the decompression performed by the decoding block 130.

[0043] In some operating scenarios, the decoded image represents an SDR image that has been forward reconstructed by an upstream video encoder (e.g., using coding block 120, etc.) from an HDR image that has been color graded to approximate a reference SDR color grade.

[0044] The operating parameters of the reconstruction function can be decoded by a receiving device of the video signal 122 or coded bitstream and used in a prediction operation to generate a reconstructed image of one color grade from a decoded image of another color grade.

[0045] Additionally, optionally or alternatively, the output video signal 122 or coded bitstream is encoded with additional image metadata, including, but not limited to, display management (DM) metadata that can be used by a downstream decoder to perform display management operations on the decoded images or reverse reconstructed images to generate display images optimized for rendering on a target display, which may or may not have the same display capabilities as the reference display to which the input color grade and reference color grade were graded.

[0046] In an operating scenario in which the receiver operates with (or is attached or operably linked to) a target display 140 that supports rendering decoded images 182, the decoding block 130 can directly or indirectly use the decoded images 182 (e.g., a forward reconstructed SDR image, etc.) to decode the images 182 from the coded bitstream 122 (e.g., a single layer therein, etc.) and render them on the target display 140.

[0047] In some operating scenarios, the target display 140 has similar characteristics as the SDR reference display 125 and the decoded image 182 is a forward reconstructed SDR image that can be viewed directly on the target display 140.

[0048] In some embodiments, the receiver operates with (or is attached to or operatively linked to) a target display that has display capabilities that differ from those of the reference display for which the decoded image 182 was optimized. Some or all of the operational parameters for the reconstruction function in the image metadata (or composer metadata) may be used to compose or reconstruct an image from the decoded image 182 optimized for the target display.

[0049] For example, the receiver may operate with an HDR target display 140-1 that supports a resolution (e.g., 100 nits, 302 nits, 300 nits, 500 nits, 1000 nits, 4000 nits, 10000 nits or more, etc.) higher than the high dynamic range of the decoded image 182. The receiver may extract image metadata from the coded bitstream (e.g., metadata container(s) therein) representing the output video signal 122 and use the operating parameters of the reconstruction function in the image metadata (or composer metadata) to compose or reconstruct image 132-1 from the decoded image 182, such as a forward reconstructed SDR image.

[0050] In some operating scenarios, reconstructed image 132-1 represents a reconstructed image optimized for viewing on a display that is the same as or comparable to the target display with which the receiver operates (e.g., HDR, EDR, an image optimized for a 1000 nits display device, an image optimized for a 4000 nits display device, etc.) The receiver can directly use reconstructed image 132-1 for rendering on the target display.

[0051] In some operating scenarios, the reconstructed image 132-1 represents a reconstructed image optimized for viewing on a display that is not the same (e.g., a reference, etc.) as the target display 140-1 operating with the receiver. A display management block (e.g., 135-1, etc.), which may be in the receiver, the target display 140-1, or a separate device, further adjusts the reconstructed image 132-1 to the characteristics of the target display 140-1 by generating a display mapped signal 137-1 adapted to the characteristics of the target display 140-1. The displayed image or the adjusted reconstructed image can be rendered on the target display 140-1.

[0052] SLBC MADC operation 2A illustrates an example process flow of SLBC MADC operations. These SLBC MADC operations may be performed by a parallel video encoding system to reduce the bit rate used to transmit image metadata in an SLBC video signal generated by multiple nodes of the video coding system, and to ensure or adhere to minimal separation between image metadata updates, while maintaining or preserving temporal consistency between (e.g., all, 99%, etc.) images / frames within a scene, even when the scene is partitioned for processing by more than one node of the video coding system.

[0053] As shown in Figure 2A, an input video signal (denoted as "Mezzanine") may be received as an input. The input video signal may comprise or be decoded into a sequence of images / frames (e.g., continuous, sequential, etc.) (e.g., 302 in Figure 3A, etc.) and input scene data (denoted as "XML+Auto-SCD").

[0054] The multi-node video encoding operation of Figure 2A may include a pre-processing step that distributes a sequence of images / frames (302 in Figure 3A) received in an input video signal ("mezzanine") into a sequence of fixed-length segments and associated bumper frames. Each fixed-length segment may be assigned to a different computing node (or video encoding node) among multiple computing nodes (or video encoding nodes) of the multi-node video encoding system.

[0055] For illustrative purposes only, and referring to Figure 3A, a sequence of fixed length segments may include three segments 304-1, 304-2, 304-3, etc. These segments (e.g., 304-1, 304-2, 304-3, etc.) are mutually exclusive or have no frames in common. Each segment includes a subsequence of images / frames (e.g., consecutive, sequential, etc.) in the sequence 302. The three segments (e.g., 304-1, 304-2, 304-3, etc.) may be assigned to three different computing nodes, denoted as "node-1", "node-2", and "node-3", respectively. The input images / frames in the three segments (e.g., 304-1, 304-2, 304-3, etc.) can be provided to computing nodes ("node-1," "node-2," and "node-3"), respectively, along with the input images / frames in respective sections 306-1 through 306-4, etc. of the bumper frame.

[0056] As shown in FIG. 3A, segment 304-1 is assigned to the first of the three nodes, i.e., node-1. The input images / frames in segment 304-1 are provided to node-1 along with the input images / frames in section 306-2 of the bumper frame. Similarly, segment 304-2 is assigned to the second of the three nodes, i.e., node-2. The input images / frames in segment 304-2 are provided to node-2 along with the input images / frames in sections 306-1 and 306-4 of the bumper frame. Segment 304-3 is assigned to the third of the three nodes, i.e., node-3. The input images / frames in segment 304-3 are provided to node-3 along with the input images / frames in section 306-3 of the bumper frame.

[0057] Each section of the bumper frame (one of 306-1 through 306-4) includes a certain (e.g., pre-configured, dynamically adjustable, 4, 8, 16, 32, etc.) number of images in the sequence 302 that precede the first image / frame or follow the last image / frame of the fixed length segment (one of 304-1, 304-2, 304-3, etc.). In some operating scenarios, the bumper section includes more input images / frames than the minimum and average distances, respectively.

[0058] The immediately preceding / following bumper frames of a fixed length segment provide the current node (e.g., node-2, etc.) to which the segment is assigned additional image data that may be used to maintain temporal consistency with adjacent (e.g., previous, next, etc.) fixed length segment(s) encoded by adjacent (e.g., previous, next, etc.) node(s) to the current node. The current node is assigned to encode the fixed length segment and does not encode the preceding or following bumper frames of the fixed length segment into the video signal.

[0059] Thus, each node is given a share of images / frames in the form of segments and bumper images / frames. In some operational scenarios, except for the first node and the last node (e.g., of three or more nodes, etc.), any other node, e.g., an intermediate node assigned an intermediate segment in a sequence of fixed-length segments, is provided with the same number of input images / frames in relation to the fixed-length segment to which such other node is assigned. In one example, the segment length (or total number of input images / frames) in the fixed-length segment may cover 30 seconds of normal play or playback time, and each bumper section may cover 2 seconds of normal play or playback time. Thus, for a 24 fps video, the fixed-length segment contains 720 input images / frames, and each bumper section contains 48 input images / frames.

[0060] Referring back to FIG. 2A, after the pre-processing step is completed, each node receives or gains access to its assigned fixed-length segment(s) and its share of input images / frames within its respective bumper section. In some operational scenarios, the multi-node encoding operation of FIG. 2A performed by each node after the pre-processing step may be implemented or performed using a two-pass technique. For simplicity, the two passes may be denoted as "pass 1" and "pass 2," respectively.

[0061] Pass 1 or the tasks therein primarily focus on generating a list of scenes in fixed-length segments to which nodes are assigned. Scene cuts from input scene data provided with an XML file and an automatic scene cut detector (Auto-SCD) can be combined to generate a first list of scenes. Scenes can be identified, for example, in the list of scenes by scene cuts that represent a combination of a start image / frame index and an end image / frame index in a sequence of consecutive images / frames.

[0062] One or more fixed or variable length scene segmentation operations / algorithms / methods / processes may be implemented or executed with the goal of maintaining temporal consistency in scenes distributed across multiple nodes, using or analyzing input images / frames in assigned segments with their respective bumper (frame) section(s) to avoid minimum and average distance constraint violations. Exemplary scene segmentation operations, including variable length scene segmentation, are described in U.S. Provisional Patent Application No. 63 / 080,255, entitled “RECURSIVE SEGMENT TO SCENE SEGMENTATION FOR CLOUD-BASED CODING OF HDR VIDEO,” filed September 18, 2020, by HARSHAD KADU et al., the entire contents of which are incorporated herein by reference as if fully set forth herein. These scene segmentation operations / algorithms / methods / processes generate additional scenes that are added to the first list of scenes to generate a second list of scenes, while still respecting or complying with the minimum and average distance constraints. The second list of scenes is provided to pass 2 or a task therein. Meanwhile, auxiliary data (denoted as “Aux Data”) is made accessible to pass 2, such as by being cached or stored, for the purpose of reducing computational overhead in pass 2.

[0063] Path 2 or a task therein performs forward and backward reconstruction for each scene identified in the second list for the fixed length segment using the second list of scenes and Aux data received from Path 1. Trim correction can be performed in the forward (processing) path that implements or performs forward reconstruction. The forward reconstruction generates a reconstructed SDR video image / frame from the input images / frames in the fixed length segment, while the backward reconstruction, implemented in the backward (processing) path, for example, generates a single set of composer metadata parameters for a given scene identified in the second list for the fixed length segment. Each image / frame in the scene can use the same set of composer metadata parameters to reconstruct an HDR image from the reconstructed SDR video image / frame decodable from the video signal generated from the multi-node video encoding operation.

[0064] Let L be the total number of input images / frames in the segment, and B be the total number of input images / frames in each bumper section. Let f be the input image / frame at the ith frame index in the sequence of input images / frames (the "mezzanine"). i This is expressed as:

[0065] A first node of the plurality of nodes selects images / frames f0 to f1 within a first fixed-length segment. L-1 The first fixed length segment does not have a left (i.e. leading) bumper section. The right (i.e. trailing) bumper section of the first fixed length segment is encoded within the image / frame range f L ~f L+B-1 This ranges from...

[0066] The Nth node or node-N of the multiple nodes is the image / frame f in the Nth fixed-length segment. (N-1)L ~f NL-1 The Nth fixed-length segment is encoded as the input image / frame f (N-1)L-B ~f( N-1)L-1and an input image / frame f NL ~f NL+B-1 and a right (or trailing) bumper section having

[0067] The last node of the plurality of nodes encodes the images / frames that are in the last fixed-length segment. The last fixed-length segment has only a left (i.e., leading) bumper section and no right (i.e., trailing) bumper section. In addition, the last fixed-length segment may have fewer than L input images / frames.

[0068] The nodes may be ordered, for example, in sequential order of node index / identifiers. For node N, node N-1 is the left or preceding (i.e., earlier) neighbor. Node N+1 is the right or succeeding (i.e., next) neighbor. The left node of N may include all nodes from 0 to N-1. The right node of N may include all nodes from N+1 to the last node.

[0069] As shown in FIG. 2A, the tasks of pass 1 performed by a node with respect to the assigned fixed-length segment may include preemptively avoiding violating minimum and average distance constraints, splitting scenes only at segment boundaries, generating auxiliary data ("Aux data"), etc. FIG. 2B shows an example process flow for performing these tasks in pass 1. The purpose of these tasks in FIG. 2B is to generate or obtain a (second) list of scenes in the assigned segment that satisfy the minimum and average distance constraints / requirements. For scene cuts within the segment and bumper section(s), only the scene cuts inside the segment are ultimately used by scene-based coding (pass 2, not shown in FIG. 2B). Scene cuts within the bumper section(s) are still useful for maintaining temporal consistency (pass 2, not shown in FIG. 2B) with adjacent nodes or adjacent segments assigned to these nodes.

[0070] Referring to FIG. 2B, block 202 includes receiving an input image / frame within an assigned fixed length segment and a section or sections of a bumper frame preceding and / or following the assigned fixed length segment.

[0071] Colorist-specified scene cuts can be read from an XML file. Additionally, optionally, or alternatively, an automatic scene cut detector identifies scene cuts or image / frame positions / indexes that define scene cuts (e.g., potential, candidate, etc.).

[0072] Block 204 involves merging these scene cuts from the colorist and the automatic detector to generate or obtain a first list of scenes for the segment.

[0073] Let D be the optimal (e.g., selected, pre-configured, preferred, etc.) distance between two successive sets of fresh image metadata. optIn some operating scenarios, the optimal distance may be given as:

number

[0074] Scenes that are smaller than the optimal distance or have fewer images / frames than the optimal distance can be merged with adjacent scenes until the combined scene is larger than the optimal distance or has more images / frames than the optimal distance. This generates an updated first list of scenes, sometimes referred to as primary scenes. At this point, all primary scenes in the segment assigned to the node have input images / frames that are larger than the optimal distance or more than the optimal distance. Thus, the minimum distance constraint and the average distance constraint are already satisfied by the primary scenes identified in the updated first list of scenes.

[0075] Block 206 includes splitting the scene by applying or executing a fixed or variable length scene splitting algorithm / method / process. In this block, the primary scene on the segment boundary (separating the segment from the bumper section(s)) is split into secondary scenes. The scene splitting algorithm / method / process may be implemented or executed in a way that complies with or inherently handles the minimum distance constraint, but may violate the average distance requirement in the secondary scene after the primary scene split. MADC operations as described herein may be implemented or executed to prevent this violation already before it occurs. These MADC operations may be used or implemented to estimate or establish a worst case number, represented by the maximum number of scenes that may be created in the assigned segment after the scene split, but with the predicted worst case number meeting the average distance constraint. In an operating scenario where the worst case number does not meet or comply with the average distance constraint, preemptive actions may be taken to avoid or prevent such violation.

[0076] Block 208 includes splitting the primary scenes on segment boundaries into secondary scenes (including, but not limited to, input images / frames in bumper (frame) section(s)) using fixed-length or variable-length scene splitting techniques, as further conditioned by possible preemptive actions (if applicable). The secondary scenes can be added to the updated first list of scenes to generate or obtain a second list of scenes. This second list can then be provided to the tasks in pass 2 for scene-based encoding. Due to worst-case handling (preemptive actions can be taken to remedy possible or potential violations of the average distance constraints), the minimum and average distance constraints are already satisfied at this point.

[0077] Blocks 210 and 212 each include performing iterative heavy overlap operations and forward lookup table (FLUT) interpolation on the input images / frames in the secondary scenes, for example, with the aim of achieving temporal consistency in the scene division between different nodes. These operations generate auxiliary data ("Aux data") in addition to the second list of scenes, which can be used to forward reconstruct the input images / frames in the secondary scenes.

[0078] Scene Cut Identification As described above, the scene cuts from the XML file and the automatic scene cut detector (Auto-SCD) can be merged to generate a first list of scenes that contain scene cuts. These cuts constitute the first list of scenes.

[0079] For example, colorists and professional color graders can manually review a sequence of images / frames ("mezzanine") that contain visual depictions of multiple scenes (e.g., input, consecutive, etc.) and detect candidate scene cuts (shown in FIG. 3B as "XML Scene Cuts" or "XML Scene Cut") within the sequence.

[0080] As shown in FIG. 3B, XML scene cuts can be represented and stored in an XML file. The dashed vertical line indicates the first frame of a candidate scene represented in the XML file. Each node (e.g., Node-N, etc.) can read (e.g., only or including) a subset of the XML scene cuts from the XML file. The subset of XML scene cuts can include XML scene cuts that are located within the node's assigned (e.g., video, fixed-length, etc.) segment or within the bumper section(s) immediately adjacent to the assigned segment.

[0081] Even if XML scene cuts are detected by a colorist, they may or may not be completely accurate. For grading purposes, a colorist may introduce scene cuts in the middle of a dissolving scene or at the beginning of a fade-in or fade-out portion of a scene. These scene cuts may cause flashing (or false positives) and can be avoided or removed, for example, by combining the XML scene cuts with automatic scene cut information generated by an automatic scene cut detector (Auto-SCD) that identifies candidate scene cuts for the assigned segment (shown in FIG. 3C as “Automatic Scene Cuts” or “Automatic Scene Cut”).

[0082] An automatic scene cut detector or Auto-SCD can detect candidate scene changes (or cuts) using changes in luminance or brightness levels within a sequence of images / frames or different subsequences therein. This automatic detector can be unaware or agnostic of dissolve, fade-in, or fade-out portions of the video, thus avoiding the false positive problems associated with colorists. As shown in FIG. 3C, the automatic scene cuts generated by Auto-SCD can also be within the assigned segment or its bumper section(s).

[0083] However, the auto-detector may have its own false positive problems. Sometimes there are luminance or brightness changes in a scene due to camera panning, motion, occlusion, etc. These luminance or brightness changes may be falsely detected as automatic scene cuts by Auto-SCD. To discard or remove these false positives, the candidate scene cuts from the XML file and Auto-SCD can be merged, for example, in block 206 of FIG. 2B.

[0084] XML and Auto-Discovery Merging JPEG0007602694000002.jpg58 shall denote a collection of frame indexes each representing a scene start frame of a respective XML scene cut in multiple XML scene cuts (as shown in Figures 3B and 3D) within the assigned segment and its bumper section(s) provided to node N (or node-N), where the subscripts indicate that these scene start frames are read from or represented in an XML file.

[0085] Similarly, Let JPEG0007602694000003.jpg515 denote a collection of frame indices each representing a scene start frame of a respective automatic scene cut in multiple automatic scene cuts (as shown in Figures 3C and 3D) within the assigned segment and its bumper section(s) provided to Node N (or Node-N), where the subscripts indicate that these scene start frames are detected by Auto-SCD.

[0086] Merging the XML and automatic scene cuts from these two collections is equivalent to taking the intersection of these two collections:

number

[0087] Merging Short Scenes The first list of scenes may contain scene cuts that are closer than the optimal distance. These short scenes may be merged to ensure that the scene has more images / frames than the optimal distance. For example, In JPEG0007602694000006.jpg55, two adjacent scene cuts occur at frame f i and f j Located in |ji| <D opt If ,then one of the scene cuts can be removed to avoid the constraint violation.,Removing a scene cut is equivalent to merging a scene with its,neighbors (or adjacent scenes).

[0088] To facilitate segmented processing across multiple nodes, a scene cut removal strategy may be designed, implemented, or executed such that consistent results are obtained or achieved across different adjacent nodes that receive overlapping input images / frames.

[0089] Figure 3E shows the scene in a sequence of input images / frames ("mezzanine"). 1 illustrates an exemplary first list of JPEG0007602694000007.jpg55, as well as mandatory scene cuts at two adjacent nodes, node N and node N+1. For each of the two nodes, node N and node N+1, the scene cuts derived from the first list of scenes are indicated by dashed vertical lines, and mandatory scene cuts occurring at segment boundaries are indicated by solid vertical lines. As illustrated, the common area (f NL-B ~f NL+B-1 ) are perfectly aligned.

[0090] A simple technique for removing short scene cuts is to traverse the segment and its bumper section(s) from left to right (or right to left) and remove any scene cut that violates the optimal distance (e.g., first found, etc.). However, this simple strategy may not be able to maintain perfect alignment of the resulting scene cuts in the common area of ​​two adjacent nodes N and N+1.

[0091] FIG. 3F shows an example scene cut after removing the first encountered short scene cut in the single traversal direction approach. As shown, the resulting scene cuts in the common area of ​​node N and node N+1 are misaligned. Node N removes both scene cuts in the common area on both sides of the solid vertical line, while node N+1 removes only one of these two scene cuts on only one side of the solid vertical line. As a result of using different scene cuts in the common area, temporal inconsistencies may be generated in the base layer video data of the video signal encoded by multiple nodes including these two adjacent nodes N and N+1.

[0092] A better approach can be used or implemented to have a node (e.g., node N) traverse an image / frame in multiple directions, as shown in FIG. 3G. As shown, under this approach, traversal of an image / frame by a node (e.g., either two adjacent nodes N or N+1) starts from the same location in a common area and moves in the same direction in both nodes. In other words, the same traversal (process) is replicated or implemented in two adjacent nodes with respect to the common area. More specifically, the input image / frame in the segment provided to a node (node ​​N in this example) and its bumper section(s) can be partitioned into five parts P1-P5 with respective traversal directions, as shown in FIG. 3G.

[0093] P1 represents the preceding bumper section of the segment, if any; P2 represents the first sub-segment (of the assigned segment) that overlaps with the last bumper section of the previous segment assigned to the previous node (node ​​N-1 in this example, not shown in FIG. 3G); P3 represents the second sub-segment (of the assigned segment) that does not overlap with any input image / frame provided to either the previous node (node ​​N-1) or the subsequent node (node ​​N+1); P4 represents the third sub-segment (of the assigned segment) that overlaps with the leading (or previous) bumper section of the subsequent segment assigned to the subsequent node (node ​​N+1 in this example); P5 represents the last (or subsequent) bumper section of the segment, if any.

[0094] FIG. 2C illustrates an exemplary process flow in which a node (eg, Node-N, etc.) traverses an input image / frame of segments assigned to the node and bumper section(s).

[0095] In P4, the traversal is performed by detecting the mandatory scene cut at the end of the segment (f NL ) and move from right to left to find the first frame (f NL-B ), where short scenes are merged along the way (whenever they are first encountered). The same traversal (or path) is followed or duplicated by node N+1.

[0096] In P5, traversal is performed on the essential scene cuts (f NL ), from left to right, the last frame (f NL+B-1 ) The same traversal (or path) is followed or mimicked / replicated at node N+1.

[0097] A similar strategy can be followed by node N in traversing P1 and P2, such that these traversals by node N in P1 and P2 mirror or follow the same traversals by node N-1 in the same overlapping input images / frames of P1 and P2. More specifically, the traversal in P1 is f (N-1)L From f (N-1)L-B , and the traversal in P2 is f (N-1)L From f (N-1)L+B-1 Proceed to.

[0098] Traversal P3 may occur after the scene cuts at P2 and P4 have been fully resolved by the traversals at P2 and P4, and may utilize information about the scene cuts at P2 and P4. Because the input images / frames at P3 are not common to the input images / frames assigned to either the previous or next node, the traversal at P3 may proceed in any direction. For example, the traversal at P3 may (e.g., additionally, alternatively, optionally, etc.) be the same left-to-right as shown in FIG. 3G, with the final complete scene cut at P2 (f (N-1)L ≦f X ≦f (N-1)L+B-1 f such that X ) and the first complete scene cut (f NL ≧f Y ≧f NL-B f such that Y In other words, the traversal through node N in P3 starts at the last complete scene cut in P2, moves towards the right, and ends at the first complete scene cut in P4.

[0099] Short scenes encountered during traversal are merged along the way. The merging process is as shown in Figure 3H: Generate an updated list of scene cuts, denoted by JPEG0007602694000008.jpg55. The total number of images / frames in the ith primary scene that belong to this updated list is Here is the updated list, indicated by JPEG0007602694000009.jpg56 In JPEG0007602694000010.jpg55 The image is JPEG0007602694000011.jpg520.

[0100] For the example distribution of scene cuts shown in Fig. 3E, by applying the multi-way traversal strategy or process flow as shown in Fig. 2C, the primary scenes in the updated list of scene cuts after merging of short scenes are shown in Fig. 3H. The same scene cuts are removed in the common area of ​​node N and node N+1. Thus, the alignment of scene cuts is preserved across different adjacent nodes, providing temporal stability of scenes overlapping common sections between different adjacent nodes.

[0101] The same scene merging procedure may be performed by each node of the multiple nodes in the multi-node video coding system in a common region, so that the updated list of scenes generated by short scene merging from the first list of scenes maintains an optimal distance between any two adjacent scene cuts that separate the scenes.

[0102] Preemptive measures to meet the average distance constraint At this point, the updated list All primary scenes identified in JPEG0007602694000012.jpg55 are at least D opt =max(D min ,D avg ) images / frames. In other words, the minimum and average distance constraints are Filled with JPEG0007602694000013.jpg55.

[0103] For temporal consistency, primary scenes on (or overlapping) segment boundaries are split. This splitting can result in the creation of additional scenes, which can lead to constraint violations with respect to minimum and average distance constraints. To prevent such constraint violations, preemptive measures or actions can be taken, as described herein.

[0104] In some operational scenarios, preemptive measures or actions can be taken by applying fixed-length (primary) scene splitting operations to the primary scene on segment boundaries to generate secondary scenes (or sub-scenes).

[0105] The default length of the jth secondary scene, sometimes called a subscene, is JPEG0007602694000014.jpg85, where: JPEG0007602694000015.jpg820, which can be written alternatively as F by omitting the superscript j for simplicity. SS ≧D min Therefore, the (fixed-length) scene splitting operation satisfies or complies with the minimum distance constraint by design. Note that any other primary scene in the segment that is not on one of the segment boundaries already satisfies both the minimum and average distance constraints, because such a primary scene is D opt This is because the total number of images / frames is equal to or greater than images / frames.

[0106] However, a scene splitting operation increases the total number of scenes in a segment, which may lead to a violation of the average distance constraint. Figure 3I shows an example fixed-length scene split of three primary scenes covering a segment and its bumper section. To comply with the average distance constraint due to video coding operations of the segment by the nodes in a multi-node coding system, we assume that the maximum allowed number of scenes in a segment, excluding its bumper section, is six.

[0107] As shown in Figure 3I(b), F SS Given a default subscene length of 1 frame, after applying a fixed-length scene splitting operation to primary scenes 1 and 3 on the segment boundaries of the segment, in the worst case scenario, eight scenes may be generated within the segment.

[0108] To comply with the allowed limit of six scenes, the default secondary subscene length is set to 100, with the goal of reducing the total number of subscenes generated from applying fixed-length scene splitting operations to split primary scenes on segment boundaries. JPEG0007602694000016.jpg55 can be increased.

[0109] As shown in FIG. 3I(c), having a larger subscene is equivalent to having fewer subscenes. As will be explained in more detail later, the subscenes include anchor subscenes (e.g., subscenes 1 and 8 in FIG. 3I(b), subscenes 1 and 6 in FIG. 3I(c), etc.), which include D min etc., a certain length (or number) of images / frames may be provided.

[0110] As part of the fixed-length scene segmentation operation, the worst-case total number of scenes after scene segmentation can be estimated and used as a reliable indicator of possible average distance constraint violations.

[0111] More specifically, if the worst-case total number of scenes is still less than or equal to the maximum allowed number of scenes in a segment derived from the average distance constraint, then do nothing. For example, the updated default minimum size for secondary scenes (or subscenes) is JPEG0007602694000017.jpg55 can remain the same as the initial default size, The result is JPEG0007602694000018.jpg515.

[0112] On the other hand, if the worst-case total number of scenes is greater than the maximum allowed number of scenes in a segment derived from the average distance constraint, the default size of the secondary scenes (or subscenes) is increased to reduce the total number of new secondary scenes generated from primary scenes on segment boundaries after scene splitting. As mentioned above, both the minimum and average distance constraints are already met or complied with by primary scenes that are not on any segment boundaries (non-boundary). Increasing the default size of the subscenes generated after splitting to an appropriate size / length (or appropriate total number of images / frames) ensures that the minimum and average distance constraints are met everywhere in the segment.

[0113] FIG. 2D illustrates an example process flow for a preemptive measure (or process) to avoid violation of the average distance constraint. The preemptive process described herein sets the default minimum length of a subscene to F SS to some new length JPEG0007602694000019.jpg55. More specifically, the maximum allowed number of scenes in a segment JPEG0007602694000020.jpg55 is calculated. The worst case total number of scenes in the segment JPEG0007602694000021.jpg55 also has the default minimum length of the subscene F SS Then, the worst case total JPEG0007602694000022.jpg55 is the maximum allowed It is judged whether it is less than or equal to JPEG0007602694000023.jpg55. Worst Case Total JPEG0007602694000024.jpg55 is the maximum allowed If it is determined that the subscene is less than or equal to JPEG0007602694000025.jpg55, the default minimum length of the subscene is F SS The initial value of is kept or used to split the primary scene on the segment boundary of the segment. Meanwhile, the worst case total JPEG0007602694000026.jpg55 is the maximum allowed If it is determined to be larger than JPEG0007602694000027.jpg55, the default minimum length of the subscene F SS From the initial value, the default minimum length F SS Instead of the initial value of JPEG0007602694000028.jpg55, split the primary scenes on the segment boundaries of the segment, and the sum of the resulting total number of secondary scenes and the total number of primary scenes that are not on any segment boundaries in the segment is equal to or greater than the maximum allowed number. Make sure the size is JPEG0007602694000029.jpg55 or smaller.

[0114] The process flow implements a minimum length adjustment algorithm to take or receive one or more input parameters and determine the new minimum length of the subscene. A single output can be generated as JPEG0007602694000030.jpg55.

[0115] More specifically, for the segments assigned to node N, Z identified in JPEG0007602694000031.jpg55 n Suppose there are first-order scenes, where the first-order scenes are denoted by {0,1…,Z n-1}, respectively. It is shown as JPEG0007602694000032.jpg523.

[0116] For illustrative purposes only, the primary scene on the left segment boundary of the segment has index l, and the primary scene on the right segment boundary of the segment has index r. Thus, the segment of node N has Z s =r-l+1 primary scenes.

[0117] As mentioned above, the worst-case total number of scenes in a segment after scene division JPEG0007602694000033.jpg55 can be estimated, and the maximum allowed number of scenes in a segment can be calculated. It can be calculated as JPEG0007602694000034.jpg1018, where: JPEG0007602694000035.jpg54 is a floor function (or operator) that selects an integer value at or just below the values ​​bracketed by the function, and L is the length of the segment.

[0118] If it is determined that the second scene is JPEG0007602694000036.jpg514, then the default length (or initial value of the minimum length) of the second scene F is set to ensure that the average distance constraint is met or complied with after scene segmentation. SS For a new larger value The following changes to JPEG0007602694000037.jpg55 can be made. In the following case ( JPEG0007602694000038.jpg514) can be processed in a minimum length adjustment algorithm.

[0119] Case 1: (a) the segment is the first segment in a sequence of images / frames processed by multiple nodes of a multi-node video coding system, and (b) the total number of primary scenes identified by the updated list of scenes in the (first) segment is greater than 1, i.e., Z s >1.

[0120] In this case, the first primary scene P on the left segment boundary l Since the first segment contains the entire scene, there is no need to split the first primary scene. SS (or D min ) on the right boundary of the (first) segment in the worst-case scenario. r Total number of scenes after splitting only JPEG0007602694000039.jpg55 is given as follows.

number

[0121] The maximum number of secondary scenes allowed in the (last) primary scene, if the average distance constraint is satisfied by these secondary scenes. Shown as JPEG0007602694000042.jpg1018.

[0122] Total worst case count of scenes JPEG0007602694000043.jpg55 as the maximum allowed number of scenes JPEG0007602694000044.jpg55 or less, the default minimum size of the secondary scene is as follows: JPEG0007602694000045.jpg56 can be increased.

number

[0123] As mentioned above, anchor subscenes can be used to facilitate or ensure temporal consistency across different segments processed by different nodes. All primary scenes located on segment boundaries are assigned a subscene of length D min Therefore, in the above formula (4), D min is the total number of frames in the (last) primary scene JPEG0007602694000047.jpg55 is subtracted from it, and then, to take into account the anchor sub-scene in the (last) primary scene, JPEG0007602694000048.jpg is divided by 511.

[0124] In a working scenario where JPEG0007602694000049.jpg511, a new value for the minimum length of a subscene may be given as:

number

[0125] Assuming short scene merging (e.g., in block 206 of FIG. 2B), the primary scene, including the last primary scene here, should have more images / frames than both the minimum distance and the average distance. Thus, JPEG0007602694000051.jpg55 is at least 1. Note that the minimum length / size change is applied to the segment. Whether other segments require changes is determined for those other segments individually.

[0126] Case 2: (a) the segment is the last segment in a sequence of images / frames processed by multiple nodes of a multi-node video coding system, and (b) the total number of primary scenes identified by the updated list of scenes in the (last) segment is greater than 1, i.e., Z s >1.

[0127] In this case, there is no need to split the last primary scene on the right segment boundary, since this last primary scene is entirely contained in the (last) segment. Only the first primary scene on the left segment boundary (or left edge) of the (last) segment is split. SS (or D min ) on the left boundary of the (last) segment in the worst-case scenario with l Total number of scenes after splitting only JPEG0007602694000052.jpg55 is given as follows.

number

[0128] The maximum number of secondary scenes allowed in a (first) primary scene, if the average distance constraint is satisfied by these secondary scenes, Shown as JPEG0007602694000055.jpg818.

[0129] Total worst case count of scenes JPEG0007602694000056.jpg55 as the maximum allowed number of scenes JPEG0007602694000057.jpg55 or less, the default minimum size of the secondary scene is as follows: JPEG0007602694000058.jpg56 can be increased.

number

[0130] As mentioned above, anchor subscenes can be used to facilitate or ensure temporal consistency across different segments processed by different nodes. All primary scenes located on segment boundaries are assigned a subscene of length D min Therefore, in the above formula (7), D min is the total number of frames in the (last) primary scene JPEG0007602694000060.jpg55 is subtracted from it, and then, to take into account the anchor sub-scene in the (first) primary scene, JPEG0007602694000061.jpg is divided by 511.

[0131] In a working scenario where the image is JPEG0007602694000062.jpg511, the new value for the minimum length of a subscene may be given by equation (5) above.

[0132] Assuming short scene merging (e.g., in block 206 of FIG. 2B), the primary scene, including the first primary scene here, should have more images / frames than both the minimum distance and the average distance. Thus, JPEG0007602694000063.jpg55 is at least 1.

[0133] Case 3: (a) the segment is an intra-segment within a sequence of images / frames processed by multiple nodes of a multi-node video coding system, and (b) the total number of primary scenes identified by the updated list of scenes in the (intra-)segment is 1, i.e., Z s =1.

[0134] If there is only one primary scene in the current segment, then the worst-case total number of scenes in the segment is:

number

[0135] JPEG0007602694000065.jpg55 To make the secondary scene's default size (or minimum length) smaller than JPEG0007602694000066.jpg55, JPEG0007602694000067.jpg56 can be increased.

number

[0136] This primary scene can be assigned or provided with two anchor sub-scenes, and therefore adjustments are made in equation (9) above.

[0137] For JPEG0007602694000069.jpg511, the result would be:

number

[0138] Case 4: (a) the segment is an intra-segment within a sequence of images / frames processed by multiple nodes of a multi-node video coding system, and (b) the total number of primary scenes identified by the updated list of scenes in the (intra-)segment is greater than 1, i.e., Z s >1.

[0139] There are two primary scenes on each of the two segment boundaries of a segment. The maximum number of subscenes in the first primary scene is JPEG0007602694000071.jpg818, and the maximum number of subscenes in the last primary scene is JPEG0007602694000072.jpg818. The worst case count for the scene can be derived as follows:

number

[0140] The default size (or minimum length) of the secondary scene is: JPEG0007602694000074.jpg56 can be increased.

number

[0141] JPEG0007602694000076.jpg511 and / or JPEG0007602694000077.jpg511, the corresponding term in the above equation (12) is D opt More specifically, If the file is JPEG0007602694000078.jpg511, JPEG0007602694000079.jpg1015 is D opt is replaced by. If the file is JPEG0007602694000080.jpg511, JPEG0007602694000081.jpg1015 is D opt is replaced by.

[0142] The preemptive measures described herein result in a new default (or minimum) length for secondary scenes. JPEG0007602694000082.jpg55 is generated. Every secondary scene, except the anchor subscene, must contain at least JPEG0007602694000083.jpg has 56 images / frames. The new default (or minimum) length ensures compliance with the average distance constraint, even in the worst case scenario.

[0143] Secondary scene cut FIG. 3J illustrates an exemplary distribution of primary scenes associated with segments assigned to a node (e.g., node N, etc.). These primary scenes may be defined with merged scene cuts after a merge operation (e.g., short scenes, etc.). As shown, primary scene 2 is entirely within the segment and therefore may be processed entirely by node N (e.g., using the pass 2 operation of FIG. 2A, etc.). Meanwhile, primary scenes 1 and 3 are on the segment boundary of the segment. These primary scenes on the segment boundary, sometimes referred to as parent scenes, are distributed across multiple nodes and are processed independently by those nodes. Special processing may be implemented for primary scenes 1 and 3 to ensure a consistent appearance in the boundary images / frames encoded by different nodes.

[0144] 3K illustrates an exemplary (simple) scenario in which a parent scene P is distributed across two (adjacent) nodes, node N and node N+1, both of which may have access to only their respective portions of the parent scene (or respective subsets or windows of the total number of images / frames) for generating forward reconstruction statistics. These respective portions of the parent scene may be referred to as forward reconstruction statistics windows, as illustrated in FIG. 3K.

[0145] In an operational scenario where these two nodes process and encode their respective parts of the parent scene (without including the bumper image in each part), the reconstruction parameters are calculated for different subsets or windows of the images / frames of the parent scene. As a result, the reconstruction parameters are calculated for at least the last image / frame (f (N+1)L-1 ) and the first image / frame in the (subsequent) segment of node N+1 (f (N+1)LFor a given SDR image, the reconstructed SDR image encoded within the video signal and the reconstructed EDR generated by applying backward reconstruction with composer metadata to the reconstructed SDR image may appear visually different. Such visual differences may appear in the form of flickering, flashing, and sudden brightness changes, and are referred to as temporal inconsistency across different nodes. The reason for such temporal inconsistency is, at least in part, due to the lack of common images / frames in the portions of the forward reconstruction statistics window or images / frames used by two adjacent nodes to compute forward reconstruction statistics and reconstruction parameters.

[0146] FIG. 3L illustrates an example of providing / inclusion of bumper images / frames in a forward reconstruction statistics window used by a node to calculate forward reconstruction statistics and reconstruction parameters. These bumper images / frames can be used to calculate, generate, or supplement the reconstruction statistics used to generate the operational parameters and composer metadata of the reconstruction operation. These operational parameters enable the reconstruction operations that generate the reconstructed SDR and reconstructed HDR images to be implemented or performed with relatively smooth transitions across the nodes.

[0147] Each of the bumper sections may contain a small total number of images / frames compared to the total number of images / frames in the parent scene, which may not be sufficient to guarantee temporal consistency without additional processing.

[0148] To solve the problems of temporal inconsistency and small number of images / frames in the bumper section, the received / provided portions of the parent scene on node N and node N+1 may be divided into secondary scenes (or sub-scenes) as shown in FIG. 3M. Even if the reconstruction statistics in a scene (e.g., parent scene, primary scene, etc.) may change significantly from the beginning of the scene to the end of the scene, the reconstruction statistics may not change much (e.g., not enough to cause perceptible visual artifacts, etc.) from one frame to the next. The secondary scenes described herein represent a relatively small neighborhood (or a relatively small portion of consecutive images / frames in the primary scene) for which forward reconstruction statistics can be generated for the purpose of evaluating or generating reconstruction parameters for forward and backward reconstruction. The reconstruction parameters calculated using the forward reconstruction statistics window in the sub-scenes do not change much (e.g., not enough to cause perceptible visual artifacts, etc.) from one sub-scene to the next, which helps the scene division achieve temporal consistency. Note that the adjacent sub-scenes described herein may also be on previous / next nodes.

[0149] Scene splits create additional scenes and increase the bitrate used to transmit composer metadata. The goal is to achieve temporal consistency using a minimum number of scene splits to help keep the bitrate of composer metadata relatively low. As mentioned above, the minimum and average distance constraints must be respected / compliant after a scene split (e.g., finalized or adopted). Bumper images / frames and anchor subscenes can play a relatively important role in ensuring a smooth visual transition from an image / frame coded by a given node to a frame coded by the next node of a given node.

[0150] FIG. 3N illustrates an exemplary primary scene in which anchor subscenes are incorporated to provide or support temporal consistency across different adjacent nodes. As illustrated, a parent scene P (in the sequence of input images / frames of the input video signal "mezzanine") spans two or more nodes. Anchor subscenes X and Y are assigned or allocated before scene splitting operations begin in the primary scene on node N and node N+1. Each of the anchor subscenes is assigned or allocated to D min images / frames. Therefore, anchor subscene X has the first frame index range JPEG0007602694000084.jpg523, anchor subscene Y is the second frame index range JPEG0007602694000085.jpg523. Both anchor subscenes X and Y compute forward reconstruction statistics and composer metadata using all images / frames within the joint region of anchor subscenes X and Y. The joint region is the combined frame index range JPEG0007602694000086.jpg531. As a result, the exact same forward reconstruction parameters for the forward reconstruction and backward reconstruction parameters (or composer metadata) for the backward reconstruction are calculated by each of node N and node N+1. Thus, the reconstructed base layer has smooth continuity at the segment boundaries even if the images / frames in the first frame index range and the images / frames in the second frame index range are respectively coded (e.g., independently, without synchronization, etc.) by two different nodes N and N+1. If the primary scene boundary (e.g., the vertical dashed line in FIG. 3N) is too close to one or both of the anchor sub-scenes (e.g., D min In a motion scenario with less than 100 images / frames, one or both of the anchor sub-scenes may be min It is possible to have more than one image / frame.

[0151] The merging of short scenes described in this specification is performed when the anchor subscene is D min In other words, the primary scene cut (vertical dotted line in FIG. 3N) helps to ensure that the anchor subscene has at least one frame index range. JPEG0007602694000087.jpg531) Mandatory scene cuts occur at the start of a segment.

[0152] As shown in Figure 3N, anchor subscenes exist on either side of the right segment boundary (or mandatory scene cut) of the segment assigned to node N. Similarly, there may be one or more anchor subscenes (one or two, depending on whether the segment represents the very first segment of all segments) on either or both sides of the left segment boundary (or another mandatory scene cut) of the segment assigned to node N.

[0153] A scene split operation may begin to execute after an anchor subscene is assigned or allocated. A scene split may include a fixed-length scene split or a variable-length scene split.

[0154] Fixed-length scene partitioning generates secondary scenes or subscenes that have a fixed, predefined length (or a fixed, predefined total number of images / frames). Each subscene, except for the anchor subscene, JPEG0007602694000088.jpg Contains 56 or more images / frames. The anchor subscene contains min It has one or more frames.

[0155] Variable-length scene splitting allows the creation of sub-scenes with different lengths (or different total number of images / frames). Each sub-scene is JPEG0007602694000089.jpg Contains 56 or more images / frames, each with its own anchor subscene min It has one or more frames.

[0156] Fixed-length scene division As a general rule, fixed-length partitions are partitions of equal length, e.g., as derived in the process flow of FIG. 2D. JPEG0007602694000090.jpg56. There may be (e.g. two) exceptions to this general rule: In one example, a subscene close to a primary scene cut or anchor scene JPEG0007602694000091.jpg may have more than 56 frames. min It is possible to have more than one frame.

[0157] 3O, 3P, and 3Q illustrate an exemplary fixed-length split operation. As shown, each anchor region includes two adjacent anchor subscenes anchored or located on two sides of a segment boundary. Each anchor subscene anchored or located on the side of a segment boundary is a D min has a size equal to

[0158] Traversal of a primary scene in a segment means starting from an anchor subscene in the primary scene and moving away from the anchor subscene toward a scene cut that defines the primary scene, if the anchor subscene is the only anchor subscene in the primary scene. If there are two anchor subscenes in the primary scene, traversal can be from left to right or from right to left. Thus, the direction of traversal is determined depending on (e.g., inherently, exclusively, independently, etc.) the position(s) of the anchor subscene(s) in the primary scene. To determine the direction of traversal, there is no need to communicate information from adjacent nodes to the node to which the segment containing the primary scene is assigned. Thus, different nodes are enabled to make these decisions by themselves (e.g., inherently, exclusively, independently, etc.) in isolation. This greatly facilitates the parallel video encoding operations performed by these nodes.

[0159] In a first example, for a primary scene on the right segment boundary of a segment, the fixed-length split operation can begin moving left from the right anchor subscene (or anchor subscene anchored or located to the left of the right segment boundary, shown by the bottom arrow in FIG. 3O) toward the scene cut that defines the primary scene within the segment, as shown by primary scene P3 at node X in FIG. 3O.

[0160] In a second example, for a primary scene on the left segment boundary of a segment, the fixed-length split operation may begin moving right from the left anchor subscene (or an anchor subscene anchored or located to the right of the left segment boundary) toward the scene cut that defines the primary scene within the segment, as shown by primary scene P1 at node X in FIG. 3O.

[0161] In a third example, for a single primary scene on both the left and right segment boundaries of a segment, the fixed-length split operation may begin moving from the left anchor subscene (or an anchor subscene anchored or positioned to the right of the left segment boundary) to the right, as shown in primary scene P at node Y in FIG. 3O.

[0162] In some operating scenarios, the parent scene from which the primary scene in a segment is derived by segmentation / partitioning may span or at least partially cover more than two nodes. For example, as shown in FIG. 3P, the parent scene may span or at least partially cover nodes N-1, N, and N+1. The first primary scene derived from the parent scene may be the first part of the parent scene in the (N-1)th segment assigned to node N-1. The second primary scene derived from the parent scene may be the second part of the parent scene in the Nth segment assigned to node N. The third primary scene derived from the parent scene may be the third part of the parent scene in the (N+1)th segment assigned to node N+1.

[0163] As shown in Figure 3P, the traversal of the first primary scene in the (N-1)th segment assigned to node N-1 is the right anchor subscene. It may begin before JPEG0007602694000092.jpg522 (anchor subscene anchored or located to the left of the right segment boundary of the (N-1)th segment) and move leftward toward the left primary scene cut (indicated by the longer vertical dashed line) that defines the first primary scene cut. The secondary scene cut (indicated by the shorter vertical dashed line) may begin before the insertion JPEG0007602694000093.jpg Right to left, unless you create a subscene with fewer than 514 images / frames. JPEG0007602694000094.jpg is inserted after every 514 images / frames. As a result, the secondary scene bounded by the left primary scene cut that defines the first primary scene is JPEG0007602694000095.jpg can have more than 514 images / frames.

[0164] Similarly, as shown in Figure 3P, the traversal of the third primary scene in the (N+1)th segment assigned to node N+1 traverses the left anchor subscene It may begin before JPEG0007602694000096.jpg518 (anchor subscene anchored or located to the right of the left segment boundary of the (N+1)th segment) and move rightward toward a right primary scene cut (indicated by a longer vertical dashed line) that defines a third primary scene cut. A secondary scene cut (indicated by a shorter vertical dashed line) may be inserted before the first primary scene cut (indicated by a shorter vertical dashed line). JPEG0007602694000097.jpg From left to right, unless you create a subscene with fewer than 514 images / frames JPEG0007602694000098.jpg is inserted after every 514 images / frames. As a result, a secondary scene bounded by a right primary scene cut that defines a third primary scene is JPEG0007602694000099.jpg can have more than 514 images / frames.

[0165] Traversal of the second primary scene in the Nth segment assigned to node N can be left-to-right or right-to-left. For illustrative purposes only, a left-to-right traversal direction is shown in FIG. 3P. If the traversal Starting from the left anchor subscene (of the second primary scene), which corresponds to JPEG0007602694000100.jpg513, the secondary scene cut will cut images / frames until it reaches the right anchor subscene (of the second primary scene). The subscene immediately before the right anchor subscene is inserted after JPEG0007602694000101.jpg511. JPEG0007602694000102.jpg can have more than 511 frames.

[0166] Figure 3Q shows an example of the lengths of subscenes in different primary scenes in different segments assigned to different nodes. JPEG0007602694000103.jpg541 may be different on different nodes. As shown in Figure 3Q, An arrow points to or points to a subscene with a minimum length, such as one of JPEG0007602694000104.jpg541. Subscenes that do not have an arrow pointing to or pointing to within a node may be larger than the minimum size of a subscene for that node.

[0167] In some operating scenarios, the fixed-length scene splitting technique may be implemented or performed as the default method of scene splitting and may generate more sub-scenes than the variable-length scene splitting technique.

[0168] Variable length scene division For illustrative purposes only, Figure 2E illustrates an exemplary process flow of a variable length scene splitting operation performed on a parent scene (denoted as Scene P) that is entirely within one node. Thus, in this example, the parent scene is equivalent to the primary scene on that node. The variable length scene splitting operation illustrated with such a parent scene may be extended or modified for purposes of performing variable length scene splitting on parent scenes, each of which may be distributed across two or more nodes.

[0169] As shown in FIG. 3R, scene P is located at the Qth frame index (f Q The first image / frame includes M images / frames, with the first image / frame being denoted as

[0170] A variable length scene split operation such as that shown in Figure 2E may be used or executed to split a parent scene into sub-scenes for temporal consistency. Note that in some operational scenarios, a parent scene that is entirely within a segment assigned to one node may not need to be split, and thus this example is for illustrative purposes only.

[0171] Step A of FIG. 2E includes accessing SDR and HDR (or EDR) images / frames in the primary scene (P) and initializing FLUT, histogram (or HIST), and thresholds associated with the SDR and HDR images / frames in the primary scene (P), e.g., during an initialization phase.

[0172] The SDR and HDR images / frames can include multiple HDR and SDR image / frame pairs, each of which includes an HDR image and an SDR image depicting the same visual semantic content depicted in the HDR image.

[0173] For example, for each (HDR and SDR) image / frame (pair) in scene P, we calculate the frame FLUT ( JPEG0007602694000105.jpg819) and HDR histogram ( JPEG0007602694000106.jpg818) is calculated.

number

[0174] Scene FLUT JPEG0007602694000108.jpg54 can be constructed for a scene P by averaging all frame FLUTs in the scene as follows:

number

[0175] In some operating scenarios, a frame FLUT of a (HDR and SDR) frame / image (pair) represents a mapping of HDR codewords (e.g., input, pre-reconstruction, etc.) in the HDR image of the frame / image (pair) to reconstructed SDR codewords (e.g., forward, etc.) in a reconstructed SDR image that approximates the SDR image of the frame / image (pair). Elements of a FLUT described herein may be, for example: JPEG0007602694000110.jpg527. An exemplary reconstruction operation for a FLUT is described in PCT Application No. PCT / US2019 / 031620, filed May 9, 2019, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0176] The histogram or bins therein may be set with the same dimension or same codeword partition as the scene FLUT (or frame FLUT). Using the scene FLUT and the (frame) histogram, the DC values ​​(denoted as χ) of all images / frames (pairs) in the scene P can be predicted or generated as follows:

number

[0177] The DC value difference between a frame and its previous frame, or between a frame and its previous frame, is used as a threshold ( The image may be used as the input image (represented as JPEG0007602694000112.jpg56).

number

[0178] The threshold for making the scene splitting decision(s) may be calculated once for all frame indices j in equation (16) above during the initialization phase in step A of FIG. 2E and used in subsequent steps, e.g. during scene splitting.

[0179] Calculate the element-wise maximum absolute difference between the FLUT of an image / frame and the FLUT of its previous image / frame and set a threshold ( The resulting image (denoted as JPEG0007602694000114.jpg59) can be generated as follows:

number

[0180] The threshold for detecting the smoothness violation(s) may be calculated once for all frame indices j in equation (17) above during the initialization phase in step A of FIG. 2E and stored in an accessible storage or cache for other operations to access.

[0181] C g The secondary scene cut, denoted as,Ω, P The frames Q+M can be collected into a sorted set denoted as Q+M, where g is an index in the sorted set. The frame index Q+M serves as an end of list marker in the sorted set and is not used as a (secondary) scene cut. The secondary scene cuts in the sorted set can be initialized in step A of FIG. 2E as follows:

number

[0182] A violation set Υ can be used to store sub-scenes that violate the smoothness criterion / threshold and can be initialized as Υ={P} in step A of FIG. 2E. Only scenes or sub-scenes listed in the violation set are split in step B of FIG. 2E. The initialization phase implemented in step A includes the threshold of secondary scene cuts. JPEG0007602694000117.jpg826, the violation set Υ, and the sorted set Ω P Generate.

[0183] Step B of FIG. 2E involves splitting the subscene(s) in the violation set Υ. A new round of subscene splitting is then performed in this step on the violation set Υ and the sorted set Ω of secondary scene cuts. P We start with input,U,. All subscenes in the violation set,U,are iterated over and partitioned.

[0184] Subscenes in the violation set Υ are P g Subscene P g is the image / frame range [C g ,C g+1 -1]. Subscene P g but JPEG0007602694000118.jpgIf it is determined that the subscene has less than 512 images / frames, then the subscene is JPEG0007602694000119.jpg is not split into two to avoid creating a (new) subscene with less than 56 frames. Otherwise, subscene P g but JPEG0007602694000120.jpg512 or more images / frames are included in the subscene P g Subscenes of (or scenes from the initial or first round of scene division in step B) FLUT JPEG0007602694000121.jpg85 is subscene P g Each frame in FLUT Compared with JPEG0007602694000122.jpg54, subscene FLUT A particular frame index position C where the similarity between JPEG0007602694000123.jpg85 and a particular FLUT of a particular image / frame is highest s A particular image / frame in is picked up or selected as follows:

number

[0185] As shown in Figure 3S, subscene P g After being split into two sub-scenes or secondary scenes, at a specific frame index position C s The partition index, denoted by, is the sorted set Ω as follows: P is inserted in the correct position.

number

[0186] In step B of FIG. 2E, a stopping criterion (no new splits) can be used to determine whether the final secondary scene cut has been obtained. Under this stopping criterion, after iterating through all subscenes in the violation set Υ, the sorted set Ω P If we determine that no new scene segmentations are added to , the stopping criterion is met. Therefore, the sorted set Ω P is output containing the final secondary scene cut.

[0187] On the other hand, there is a sorted set Ω that contains at least one new scene partition after iterating through all the subscenes in the violation set Υ.P If we determine that all new partitions from all subscenes in the violation set Υ are added to the sorted set Ω in the corresponding sorted order, then the stopping criterion is not met. P The violation set Υ is reinitialized to the empty set as follows:

number

[0188] The sorted set Ω updated in step B P can be passed to the next step, i.e., step C of FIG. 2E.

[0189] Step C is the sorted set Ω P The step B involves collecting (reconstruction-relevant, image-content-dependent) statistics for all subscenes identified in . The subscene FLUT is then updated in step B by sorting the set Ω P For example purposes only, a sorted set Ω may be calculated or established for each secondary scene in P is as follows: C0 to C G Includes G+1 secondary scene cuts up to.

number

[0190] There are G subscenes in this round, and a sorted set Ω P The frame indices in are in corresponding order, such as ascending order, as follows:

number

[0191] Figure 3T shows the frame range [C g ,C g+1 An example subscene P among one or more subscenes derived from a parent scene P ranging from gFor all subscenes P with g∈[0,G-1], g Subscene for FLUT JPEG0007602694000130.jpg85 can be calculated or generated for the (current) round as follows:

number

[0192] Let λ be the DC value in the current round. g The predicted DC values ​​for all frames in can be calculated or generated for the (current) round as follows:

number

[0193] Sorted set Ω P After iterating through all subscenes in, the DC values ​​of all frames in the parent scene P are collected. These DC values ​​can be passed to step D of FIG. 2E and used to find or identify threshold violations at edges / boundaries that respectively separate two adjacent subscenes of a subscene.

[0194] Step D involves detecting temporal capability violations at subscene edges or boundaries between adjacent / neighboring subscenes.

[0195] Figure 3U shows subscene P (g-1) and P g An example sub-scene edge (or boundary) between C g It checks for multiple types of violations and if any one of those types of violations is on a subscene edge (or boundary) C g If the violation check (for any of the violation types) fails, then the subscene P (g-1) and P gThe same check is performed on each subscene boundary C in parent scene P except C0. g It can be applied in.

[0196] (e.g., in step C of Figure 2E) P Given the DC values ​​of all frames in the parent scene P after iterating through all subscenes in P All secondary scene cuts except C0 in g DC differences at sub-scene edges (or boundaries) can be calculated. JPEG0007602694000133.jpg56 is frame index C g The DC value of the image / frame with index C g It can be calculated as the difference between the DC value of the previous frame with -1 as follows:

number

[0197] Violation check 1 is absolute value of DC difference JPEG0007602694000135.jpg88 is DC thresholded as follows: JPEG0007602694000136.jpg88 (for example, frame index C based on the above formula (16) g ω) plus a non-negative bias ω.

number

[0198] Absolute DC Value Difference JPEG0007602694000138.jpg88 is the DC threshold If it is determined that the pixel value of JPEG0007602694000139.jpg88 is greater than the non-negative bias ω, then the violation check is deemed to have failed. g-1 and P g The transition point between g Any change in brightness in these sub-scenes P represents or is considered to be a threshold violation. g-1 and P g is added to the violation set Υ, which is further split in the next round. Otherwise, this violation check is considered to pass. This criterion or violation check may be a relatively important violation check because of its impact on the outcome of the scene splitting operation.

[0199] Violation check 2 is performed to determine whether the sign of the DC value difference is the same as the sign of the DC threshold as follows.

number

[0200] Positive DC Difference JPEG0007602694000141.jpg56 indicates that there is an increase in DC value from the previous frame to the current (or next) frame that follows the previous frame. Conversely, a negative DC difference JPEG0007602694000142.jpg56 indicates that there is a decrease in DC value from the previous frame to the current (or next) frame. Threshold The trend of DC value increase / decrease in parent scene P from the previous frame to the next frame as shown or represented by the code in JPEG0007602694000143.jpg86 is the DC value difference Temporal stability is deemed to be maintained under this violation check when the temporal stability is maintained the same as in the subscene as shown or represented by the code JPEG0007602694000144.jpg56. Otherwise, this violation check is deemed to have failed and the subscene P g-1 and P g Both are added or appended to the violation set Υ for further partitioning in the next round. This criterion or violation check helps to maintain temporal stability in fade-ins / outs, dissolving scenes, or any scenes with gradual brightness changes.

[0201] Violation check 3 is as follows: FLUT JPEG0007602694000145.jpg85 and The maximum absolute element-by-element difference between JPEG0007602694000146.jpg88 is C g Threshold at This is performed to determine whether the image is greater than JPEG0007602694000147.jpg89 (derived, for example, based on equation (17) above).

number

[0202] FLUT JPEG0007602694000149.jpg85 and The maximum absolute element-by-element difference between JPEG0007602694000150.jpg88 is the threshold If it is determined that the smoothness constraint is greater than JPEG0007602694000151.jpg89, then this violation check is considered to have failed (or the smoothness constraint has been violated), and these subscenes P g-1 and P g is included or added to the violation set Υ, which is further partitioned in the next round. Otherwise, this violation check is considered to pass (or the smoothness constraint is not violated).

[0203] By way of example and not limitation, all violation checks may be at subscene boundaries. If there is a violation, both subscenes on either side of the subscene boundary are added to a violation set Υ for further partitioning in the next round. Execution of these violation checks terminates the scene partitioning operation in the current round. If the violation set Υ (which may be updated with these violation checks) is not empty, the process flow of FIG. 2E continues by adding a sorted set Ω to the subscene boundary for the next round of scene partitioning. P Then, return to

[0182] using the violation set Y.

[0204] In step D of FIG. 2E, a second stopping criterion (no violation) can be used to determine whether the final secondary scene cut has been obtained. Under this stopping criterion in step D, the stopping criterion is met if it is determined that none of the sub-scenes in the parent scene P violate these criteria or the violation check. Thus, the sorted set Ω P is output containing the final secondary scene cut.

[0205] It should be noted that while the process flow in FIG. 2E is shown with a parent scene that is entirely within a segment, in various embodiments some or all of these scene splitting techniques may be extended or adapted to apply to primary scenes on either side of a segment boundary, such as a parent scene that is shared by (or resides within) two or more segments assigned to two or more nodes.

[0206] FIG. 3V shows an example of a parent scene present on two nodes. Each portion of the parent scene within a node (or corresponding segment) represents the primary scene of that node (or corresponding segment). Three anchor sub-scene cuts (indicated by solid vertical lines in FIG. 3V) in these two primary scenes define two anchor sub-scenes, which together form an overlapping window. For node N, the anchor scene cuts (or splits) are frame JPEG0007602694000152.jpg513, and for node N+1, the second anchor split is introduced in frame These initial sub-scene cuts (or splits) are already in place before any further variable length scene splitting operations are performed on these primary scenes.

[0207] In this case, if a parent scene exists in more than one node, the sorted set Ω P By including additional anchor scene cuts in a segment, relatively small changes or adjustments can be made in the initialization performed in step A of the process flow of FIG. 2E to scene split a primary scene in a segment by considering anchor sub-scenes in the primary scene. With these initial anchor scene cuts (splits) in place in adjacent segments, a visually consistent appearance at the segment transition boundaries can be maintained.

[0208] Then, the process flow of FIG. 2E can jump directly to step C or proceed to step C after initialization with relatively small changes in adjustment. The previous purpose of step B in the previous case where the primary scene only existed in a single segment is to split the primary scene in this previous case. In the present case where the parent scene exists in more than one node, the process flow can move directly to step C in the very first round because the primary scene in each node has already been split into two subscenes by the additional anchor scene cut (or split) deployed before the process flow of FIG. 2E. Subsequent rounds of the process flow in the present case where the parent scene exists in more than one node can operate or proceed in the same way as the process flow in the previous case where the primary scene only exists in a single segment, except that the already placed anchor subscenes are not further split in the subsequent scene splitting operation, even if there is a violation in the anchor subscene boundary frame (e.g., corresponding to an additional subscene cut inside the segment, etc.). FIG. 3W shows an example result of variable length scene splitting for the parent scene P of FIG. 3V.

[0209] Variable-length scene partitioning may be relatively complex and computationally intensive compared to fixed-length scene partitioning. However, variable-length scene partitioning may provide advantages / benefits over fixed-length scene partitioning in that a relatively smaller total number of sub-scenes may be created. In some operating scenarios, fixed-length scene partitioning may be used as a default in scene partitioning.

[0210] In some operational scenarios, only the primary scenes on either side of the segment boundary are split into secondary scenes. Scene cuts (or splits) are collected and appended to an updated first list of scenes, and the updated first list of scenes is added to a second list of scenes. The first list is used to obtain or generate JPEG0007602694000154.jpg55. The second list is passed as input to pass 2 for scene-based encoding. Temporal violations may occur at scene cut frames within scenes identified in the second list of JPEG0007602694000155.jpg55 (e.g., unsplit primary scenes, anchor and / or non-anchor sub-scenes, etc.) In some operating scenarios, temporal violations and inconsistencies are smoothed using iterative overlap and FLUT interpolation before moving to pass 2.

[0211] Repeated overlap of secondary scenes If the scene segmentation could be allowed down to a single image / frame such that JPEG0007602694000156.jpg512, all temporal violations could be removed. However, compliance with the minimum and average distance constraints could lead to temporal violations in the images / frames encoded in the video signal.

[0212] FIG. 2F illustrates an example process flow for iterative overlap to mitigate temporal violations associated with secondary scenes. The process flow is as follows: P and a violation set Υ as input and accesses all subscenes in the primary scene P as identified in the violation set Υ.

[0213] Potential temporal inconsistencies at subscene boundaries can be assessed using some or all of the criteria or validation checks in step D of Fig. 2E. Following a fixed or variable length partitioning, a sorted set Ω of primary scenes P in segments assigned to nodes N is P Frame index in {C0,C1…C g...}, the process flow of Figure 2F may end if it is determined that no violations are found or detected according to violation checks 1, 2, and 3 of step D of Figure 2E. Otherwise, having determined that at least one violation has been found or detected in the sub-scene cut, node N may start the very first iteration of the iterative overlap process as the current iteration.

[0214] In the current iteration of the iterative overlap process, node N accesses every subscene boundary violation identified in violation set Υ. For example, as shown in FIG. 3U, g-1 and P g Frame index C between g A violation may be found or detected in . The overlap length of each subscene adjacent to an identified boundary or subscene cut in the violation set . . . is initialized to the length of each such subscene.

[0215] The current iteration of the iterative overlap process consists of: (1) for each boundary or subscene cut identified in the violation set Υ, g-1 to the right (in terms of a boundary or subscene cut) of the left subscene P g-1 (2) determining or checking whether adjacent images / frames are available to increase the overlap length of the right sub-scene P for that boundary or sub-scene cut; g The right subscene P to the left (in terms of a boundary or subscene cut) of g and determining or checking whether adjacent images / frames are available to increase the overlap length of the sub-scenes in the segment assigned to node N. The overlap length of the sub-scenes in the segment assigned to node N cannot exceed the bumper frames available to node N.

[0216] In this example, the left subscene P g-1 and right side subscene P getc., if it determines that there are no adjacent images / frames available to increase the overlap length of the subscene, then it does nothing, i.e., the overlap length of the subscene is not incremented or increased. Note that in some operating scenarios, no overlap length increment operation is performed on the anchor subscene.

[0217] Otherwise, the adjacent image / frame, in this example the left subscene P g-1 and right side subscene P g By determining that the overlap length of the subscenes P1, P2, P3, P4, P5, P6, P7, P8, P9, P10, P11, P12, P13, P14, P15, P16, P17, P18, P19, P20, P21, P22, P23, P24, P25, P26, P27, P28, P29, P30, P31, P32, P41, P42, P53, P64, P70, P82, P93, P104, P116, P126, P138, P148, P156, P164, P176, P184, P296, P296, P296, P304, P31, P42, P53, P65, P70, P71, P82, P93, g The overlap length of the left subscene P is incremented by 1 to the left. g-1 The overlap length of is incremented by 1 to the right. The process flow proceeds to the next violation in the violation set Υ and performs the same overlap length increment operation as described herein.

[0218] If the subscene overlap length has not been updated or changed during this (current) iteration, the process flow of FIG. 2F ends ("2. No change in overlap" in FIG. 2F).

[0219] As shown in step C of FIG. 2E, images / frames with updated overlap lengths can be used to collect (reconstruction-relevant, image-content-dependent) statistics for all sub-scenes.

[0220] The current iteration of the iterative overlap process further includes determining or checking for violations ("3. Check for violations in the primary scene" in FIG. 2F), as shown in step D of FIG. 2E.

[0221] If it is determined that no violations are detected, the process flow of Figure 2F ends ("4. No Violation" in Figure 2F). Otherwise, the process flow of Figure 2F returns to step 1 of the iterative overlap process ("1. Increase Subscene Overlap by 1" in Figure 2F) ("5. Any Violation" in Figure 2F), increments the iteration count, and begins a new iteration as the current iteration.

[0222] For each of these subscenes, a FLUT can be calculated using the respective overlap length (or window). The FLUTs thus calculated can be stored or cached in a data structure used for FLUT interpolation. The left and / or right overlap lengths of each subscene can be passed to the next stage or pass 2 as part of the auxiliary data.

[0223] In a non-limiting illustrative example, a sample parent (or primary) scene of close to 180 frames is split across three nodes. The parent scene is split into fixed-length sub-scenes of 10 frames or more. These sub-scenes are: For anchor and non-anchor subscenes near a primary scene cut that may have lengths that exceed the lengths specified for the non-anchor subscenes, anchor subscenes, and anchor subscenes specified in JPEG0007602694000157.jpg55 JPEG0007602694000158.jpg55 may include non-anchor sub-scenes near the primary scene cut that may have a length greater than JPEG0007602694000158.jpg55.

[0224] When these subscenes are separated and processed separately on three different nodes with iterative overlap, a first absolute DC value difference between adjacent images / frames can be calculated. When these images / frames are processed on a single node without splitting the primary / parent scene into subscenes, and therefore without iterative overlap, a second absolute DC value difference between adjacent images / frames can also be calculated to demonstrate the effectiveness of iterative overlap.

[0225] The iterative overlap aims to make the first absolute DC value difference as close as possible to the second absolute DC value difference. Peaks in the first absolute DC value difference and the second absolute DC value difference correspond to large DC value differences between adjacent images / frames. Such peaks are an indication of possible temporal inconsistencies. It can be seen that after each iteration of the process flow of FIG. 2F of the iterative overlap, the peaks in the first absolute DC value difference are reduced and become closer and closer to the second absolute DC value difference (than without the iterative overlap), thereby achieving or improving temporal smoothness in the video signal encoded by the multi-node video encoding system. It can also be seen that most of the temporal smoothness violations as indicated by the remaining peaks appear to occur near the anchor sub-scene because the overlap length is not changed or expanded for the anchor sub-scene. The total number of iterations in the iterative overlap can be configurable or pre-configured. A hard limit on the total number can be implemented, regardless of whether the temporal smoothness violations are completely eliminated in the iterations. This hard limit can be set to a fixed number such as 15, 20, 25, etc., or can be set based on multiple factors such as the segment length (e.g., how many images / frames are in the segment), minimum distance constraints, and average distance constraints.

[0226] Although repeated overlap smooths sub-scene transitions, temporal smoothness violations (or temporal violations) may still persist. Figure 3Z shows exemplary temporal violations measured or expressed in frame (or per-frame) DC values.

[0227] As shown in the top graph plot of FIG. 3Z, frame DC values ​​(e.g., reconstructed base layer images, etc.) throughout a primary scene that is entirely processed, reconstructed, or encoded by a single node using a single set of forward reconstruction parameters for the entire scene exhibit continuity. This DC value trend / pattern is considered normal (e.g., as being the norm, etc.). Any significant or relatively large deviation from this trend / pattern is an indication of temporal variation and non-smoothness.

[0228] As shown in the bottom graph plot of Figure 3Z, frame DC values ​​(e.g., reconstructed base layer images, etc.) within a primary / parent scene that has been split into subscenes that have been separately processed, reconstructed, or coded using different sets of forward reconstruction parameters for the subscenes generated from portions of the primary / parent scene exhibit abrupt DC value changes or many discontinuities. These abrupt DC value changes are primarily found at the boundaries of the subscenes.

[0229] For example, temporal smoothness violations, as represented by abrupt DC value changes or discontinuities, can be exacerbated when sub-scenes within a parent / primary scene are in different segments that are assigned to different nodes and are processed, reconstructed, and encoded separately by the different nodes.

[0230] FLUT interpolation may be implemented to improve or avoid sudden DC value changes in operational scenarios where sub-scenes are processed separately by different nodes, with the aim of providing a better visual experience that is relatively free of visual artifacts caused by multi-node video coding. If it is determined that the temporal smoothness violation still persists, FLUT interpolation may be applied after iterative overlap. Otherwise, FLUT interpolation may be bypassed.

[0231] FIG. 2G illustrates an exemplary process flow for secondary scene (or sub-scene) FLUT interpolation that improves the temporal smoothness violations that remain after repeated overlap.

[0232] As shown in Figure 3AA, a primary or parent scene identified in the list of scenes may be partitioned, segmented, or otherwise divided among multiple nodes, such as nodes N-1, N, and N+1. Each node is assigned a portion of the scene. These portions of the parent scene may also be referred to as the primary scene in their respective segments.

[0233] A primary scene within a segment having a right segment boundary and a left segment boundary defining the segment may belong to or be classified as one of the following three intra-segment primary scene types: First, the primary scene within a segment assigned to, for example, node N-1 of FIG. 3AA includes only the right segment boundary; Second, the primary scene within a segment assigned to, for example, node N+1 of FIG. 3AA includes only the left segment boundary; Third, the primary scene within a segment assigned to, for example, node N of FIG. 3AA includes both the right and left segment boundaries.

[0234] As shown in FIG. 3AA, secondary scene cuts can be formed after scene splitting using anchor sub-scenes around segment boundaries.

[0235] As shown in FIG. 2G, the three intra-segment primary scene types can be treated differently in FLUT interpolation as cases 1, 2 and 3, respectively. In some operating scenarios, FLUT interpolation does not affect the frame FLUT in the anchor subscene. The frame FLUT in the anchor subscene serves as a reference for the frame FLUT in the rest of the subscene (or non-anchor subscene) to align to.

[0236] Let χ be the frame DC value when the primary scene P is processed as a single unit by a single node. JPEG0007602694000159.jpg54 and FLUT JPEG0007602694000160.jpg54 have the same dimension or total number of indices or entries, where j indicates the frame index for the jth image / frame. JPEG0007602694000161.jpg54 can be formed by averaging the FLUTs of all frame FLUTs for all images / frames in scene P. The frame DC value of the jth image / frame can be calculated as follows:

number

[0237] The DC difference between each image / frame and its previous image / frame, which is used as a reference in FLUT interpolation ( JPEG0007602694000163.jpg56) can be calculated as follows:

number

[0238] In case 1 of Fig. 2G, the primary scene includes only a right segment boundary, for example, in the segment assigned to node N-1 in Fig. 3AA. As shown in Fig. 2G, the FLUT in the (right) anchor subscene can be calculated first, and the FLUTs of the remaining subscenes other than the (right) anchor subscene in the primary scene can be adjusted to the FLUT in the (right) anchor subscene.

[0239] Right anchor subscene A r The subscene immediately to the left of the right anchor subscene is represented as A. r-1 Then, the next left subscene is represented as A r-2 and so on.

[0240] Anchor Subscene JPEG0007602694000165.jpg55's flut is the frame flut of the image / frame in the anchor subscene JPEG0007602694000166.jpg54 is obtained by averaging the other (non-anchor) sub-scenes. JPEG0007602694000167.jpg58, FLUTs such as JPEG0007602694000168.jpg58 are obtained by averaging frame FLUTs within an overlap length or window (which may be extended and thus include more images / frames than these subscenes). The FLUTs of these non-anchor subscenes may be initially calculated or reused / obtained (to the extent available) from the results of an iterative overlap operation performed before the FLUT interpolation operation. In some operational scenarios, the input codewords and mapped codewords in these FLUTs are calculated in the normalized codeword domain, e.g. Represented as JPEG0007602694000169.jpg523.

[0241] As shown in Figure 2G for Case 1, the left (A r-iThe FLUTs for some or all (non-anchor) subscenes (denoted as

number

[0242] Weight JPEG0007602694000172.jpg58 can be derived as follows:

number

[0243] The updated FLUTs for the (non-anchor) subscenes in equation (32) above and the HDR histograms for each frame available for each frame in these subscenes Using JPEG0007602694000174.jpg54, such a (non-anchor) subscene A r-i The DC value (denoted by a symbol different from λ) for each image / frame in (eg, the jth image / frame) may be calculated as follows:

number

[0244] The FLUT adjustment in equation (32) above ensures that the FLUT transition is sufficiently smooth from one subscene to the next. However, even after the FLUT adjustment, there may still be abrupt DC changes, as shown in the plot of the top graph in FIG. 3AB. To remove these abrupt DC value changes, an offset can be further applied to the adjusted FLUT.

[0245] Right Anchor Subscene A r Since the FLUT of (the rightmost part of the plot in Figure 3AB) is fixed, the adjusted FLUT of the remaining subscenes is the right anchor subscene A. r can be shifted by an offset (positive or negative) along the vertical axis of FIG. 3AB.

[0246] The adjusted FLUT translation for the subscene is Subscene A. r-1 For illustrative purposes only, Anchor Subscene A can be used from right to left, starting from r The frame index of the first image / frame in is q. Therefore, the adjacent non-anchor sub-scene A r-1 The last image / frame in is q-1. In some motion scenarios, overlapping images / frames added to the length of a subscene are not considered part of the subscene for the purposes of calculating temporal violations. r-1 and A r At the boundary between (or separating) q = λ q -λ q-1 The reference DC value difference for temporal consistency is given by The DC value difference between these two can be used to find the subscene A. r-1 FLUT The offset to move JPEG0007602694000177.jpg58 JPEG0007602694000178.jpg520 can be derived or generated. This offset can then be used to JPEG0007602694000179.jpg58, the (actual) DC value of image / frame q-1 is increased by ρ, which is the new (actual) DC value difference at q and q-1. This results in JPEG0007602694000180.jpg54, which is the reference DC value difference It is exactly equal to JPEG0007602694000181.jpg56. This can be inferred or proven as follows:

number

[0247] Therefore, by adding the offset, the updated DC value delta JPEG0007602694000183.jpg54 is the reference DC value difference The value is the same as JPEG0007602694000184.jpg56. Subscene A r-1 This movement of the FLUT of the adjacent non-anchor sub-scene A r-1 FLUT and Anchor Subscene A r Next, the alignment with the FLUT of the adjacent non-anchor sub-scene A is completed. r-1 The FLUT is fixed, and the subscene A r-1 Subscene A adjacent to r-2 FLUT is sub-scene A r-1 are shifted in a similar manner as described herein for the FLUTs of . This shifting process with offset can be repeated for all non-anchor subscenes in the scene at node N-1 until the last subscene on the left side is reached, as shown in the bottom graph plot of FIG. 3AB, thereby completely removing the temporal mismatch since the DC value differences are restored to their (original) values ​​without any scene partitioning / segmentation.

[0248] In case 2 of Fig. 2G, the primary scene includes only a left segment boundary, for example, in the segment assigned to node N+1 in Fig. 3AA. As shown in Fig. 2G, the FLUT in the (left) anchor subscene can be calculated first, and the FLUTs of the remaining subscenes other than the (left) anchor subscene in the primary scene can be adjusted to the FLUT in the (left) anchor subscene.

[0249] Left anchor subscene A l The subscene immediately to the right of the left anchor subscene is represented as A l+1 Then, the next right sub-scene is represented as A l+2 and so on.

[0250] Anchor Subscene JPEG0007602694000185.jpg55's flut is the frame flut of the image / frame in the anchor subscene JPEG0007602694000186.jpg54 is obtained by averaging the other (non-anchor) sub-scenes. JPEG0007602694000187.jpg57, FLUTs such as JPEG0007602694000188.jpg57 are obtained by averaging frame FLUTs within an overlap length or window (which may be extended and thus include more images / frames than these subscenes). The FLUTs of these non-anchor subscenes may be initially calculated or reused / obtained (to the extent available) from the results of an iterative overlap operation performed before the FLUT interpolation operation. In some operational scenarios, the input codewords and mapped codewords in these FLUTs are calculated in the normalized codeword domain, e.g. Represented as JPEG0007602694000189.jpg523.

[0251] For case 2, as shown in Figure 2G, the right (A l+iThe FLUTs for some or all (non-anchor) subscenes (denoted as

number

[0252] Weight JPEG0007602694000192.jpg58 can be derived as follows:

number

[0253] The updated FLUTs for the (non-anchor) subscenes in equation (36) above and the HDR histograms for each frame available for each frame in these subscenes Using JPEG0007602694000194.jpg54, such a (non-anchor) subscene A l+i The DC value (denoted by a symbol different from λ) for each image / frame in (eg, the jth image / frame) may be calculated as follows:

number

[0254] The FLUT adjustment in equation (36) above ensures that the FLUT transition is sufficiently smooth from one subscene to the next. However, even after the FLUT adjustment, there may still be abrupt DC changes, as shown in the plot of the top graph in FIG. 3AC. To remove these abrupt DC value changes, an offset can be further applied to the adjusted FLUT.

[0255] Left Anchor Subscene A l Since the FLUT of (the leftmost part of the plot in Figure 3AC) is fixed, the adjusted FLUT of the remaining subscenes is the left anchor subscene A. l can be shifted by an offset (positive or negative) along the vertical axis of FIG. 3AC.

[0256] The adjusted FLUT translation for the subscene is Subscene A. l+1 For illustrative purposes only, Anchor Subscene A can be used from left to right, starting with l The frame index of the last image / frame in is q-1. Therefore, the frame index of the adjacent non-anchor subscene A l+1 The first image / frame in is q. In some operating scenarios, overlapping images / frames added to the length of a subscene are not considered part of the subscene for the purposes of calculating temporal violations. l+1 and A l At the boundary between (or separating) q = λ q -λ q-1 The reference DC value difference for temporal consistency is given by The DC value difference between these two can be used to find the subscene A. l+1 FLUT The offset to move JPEG0007602694000197.jpg57 We can derive or generate JPEG0007602694000198.jpg520. We use this offset as the FLUT JPEG0007602694000199.jpg57, the (actual) DC value of image / frame q is increased by ρ, which is the new (actual) DC value difference at q and q-1. This results in JPEG0007602694000200.jpg54, which is the reference DC value difference It is exactly equal to JPEG0007602694000201.jpg56. This can be inferred or proven as follows:

number

[0257] Therefore, by adding the offset, the updated DC value delta JPEG0007602694000203.jpg54 is the reference DC value difference The value is the same as JPEG0007602694000204.jpg56. Subscene A l+1 This movement of the FLUT of the adjacent non-anchor sub-scene A l+1 FLUT and Anchor Subscene A l Next, the alignment with the FLUT of the adjacent non-anchor sub-scene A is completed. l+1 The FLUT is fixed, and the subscene A l+1 Subscene A adjacent to l+2 FLUT is sub-scene A l+1 are shifted in a similar manner as described herein for the FLUTs of . This shifting process with offset can be repeated for all non-anchor subscenes in the primary scene at node N+1 until the last subscene on the left side is reached, as shown in the bottom graph plot of FIG. 3AC, thereby completely removing the temporal mismatch since the DC value differences are restored to their (original) values ​​without scene partitioning / segmentation.

[0258] In case 2 of FIG. 2G, the primary scene contains both right and left segment boundaries, for example within the segment assigned to node N in FIG. 3AA. The interior subscene in this case needs to be aligned with both the right and left segment boundaries. l , right anchor subscene A r Let the FLUTs of these anchor subscenes be Let's say JPEG0007602694000205.jpg820. The FLUT of the internal subscene can be adjusted as follows:

number

number

[0259] To reduce abrupt DC changes, we can further add an offset to the adjusted FLUT (in equation (40) above) of the inner subscene. Because there are two anchor subscenes and the FLUTs of both anchor subscenes are fixed, adding an offset to the FLUT of the inner subscene to reduce temporal violations is a more constrained problem than adding an offset to the FLUT of a scene with only one anchor subscene.

[0260] Without loss of generality, adding an offset to the FLUT of an inner subscene is equivalent to adding an offset to the left anchor subscene A. l Start with the inner subscene adjacent to the right anchor subscene A r In another embodiment, adding an offset to the FLUT may be performed by traversing the right anchor subscene A toward the right anchor subscene B. r Starting from the inner subscene adjacent to Left Anchor Subscene A l Note that we can traverse along the reverse direction towards

[0261] The DC value of the first frame in the current inner subscene can have continuity with the DC value of the last frame in the left (or previous) subscene because an offset is added to the FLUT. r A adjacent to r-1 Right anchor subscene A r The last internal subscene A r-1 Right anchor to align with Subscene A r You cannot add an offset to the FLUT of the anchor subscene. The FLUT of the anchor subscene is not adjusted or changed.

[0262] A r-1 and A r The boundary between or A r-1 and A r The alignment error (denoted as ξ) at the boundary separating JPEG0007602694000212.jpg520, and q is calculated as the right anchor sub-scene A, as shown in the lower graph of Figure 3AD. r is the frame index of the first image / frame in the

[0263] Node N can determine whether the alignment error ξ is small enough (e.g., measured below a maximum alignment error threshold, etc.) If it determines that the alignment error ξ is small enough, the FLUT can be used for the sub-scene without further error diffusion techniques.

[0264] On the other hand, if it is determined that the alignment error ξ is sufficiently small, A r-1 and A r The alignment error ξ at the boundary between or separating the left anchor subscene A l Towards Subscene A r-1 to avoid a relatively large DC value change occurring in a single place or location.

[0265] Error diffusion can be achieved by adding offsets to the other subscene FLUTs while meeting the criteria associated with a particular error diffusion method. As noted above, the FLUTs of the left and right anchor subscenes remain unchanged.

[0266] Prior to or at the start of an error diffusion operation, the following error diffusion related data values ​​may be calculated or derived: Positive DC Difference JPEG0007602694000213.jpg56>The number of 0s is Represented by JPEG0007602694000214.jpg55 Negative DC Difference JPEG0007602694000215.jpg56<0's number is Represented by JPEG0007602694000216.jpg55 The sum of all positive DC differences is JPEG0007602694000217.jpg518, JPEG0007602694000218.jpg513 The sum of all negative DC differences is JPEG0007602694000219.jpg520, JPEG0007602694000220.jpg513 where q represents the boundary image / frame between or separating any two adjacent subscenes in scene P. These parameters JPEG0007602694000221.jpg536 can all be calculated as positive values ​​for computational efficiency.

[0267] DC value shift We denote the residuals as JPEG0007602694000222.jpg53, which can be used to diffuse or distribute the alignment errors to other sub-scenes, if desired. Let us denote the error remaining after all options for safely diffusing the alignment error to other sub-scenes without introducing any (e.g., additional, new, etc.) violations have been exhausted. Because the FLUT interpolation and error diffusion methods are deterministic, these parameters may be pre-computed even before the FLUT interpolation. In an operating scenario where ξ>0, these parameters may be calculated (e.g., using an ordered set of equations, etc.) as follows:

number

[0268] In an operating scenario where ξ<0, these parameters may be calculated (eg, using an ordered set of equations, etc.) as follows:

number

[0269] In equations (42) and (43) for ξ<0 and ξ>0, respectively, JPEG0007602694000226.jpg819 Adding to JPEG0007602694000227.jpg53 is done when the residual error is greater than zero (0), i.e. This is done when JPEG0007602694000228.jpg59. To some extent, this may reintroduce temporal violations at some or all of the subscene boundaries. JPEG0007602694000229.jpg59) occurs when there are not enough subscenes to diffuse the alignment error. In some operating scenarios, the average and minimum distance constraints may be relaxed for the current segment in order to mitigate, reduce, or avoid remaining errors in the error diffusion method.

[0270] Without loss of generality, the error diffusion method is l+1 Starting from A r-1 We can traverse it from left to right so that it ends with A. i and A i+1 The diffusion error offset ρ used to diffuse to the boundary between JPEG0007602694000230.jpg520, where q is the A i+1 is the frame index of the first frame in, which is further updated as follows:

number

[0271] The updated offset ρ is JPEG0007602694000232.jpg57 to diffuse the alignment error to the appropriate subscene boundary. Figure 3AF shows an example DC value plot (lower graph) after error diffusion. Compared to the DC value plots (top graph in Figure 3AF and bottom graph in Figure 3AD), the DC value plot after error diffusion appears more continuous.

[0272] After FLUT interpolation and / or error diffusion, pass 1 of Figure 2A ends. The FLUTs of the anchor and non-anchor subscenes, as well as primary scenes that are not divided into subscenes, may be stored as auxiliary data and passed as part of the input to pass 2 of Figure 2A.

[0273] Second list of scenes As used herein, a "scene...on a (segment) boundary" of a segment may mean that the scene is a parent scene identified prior to partitioning or segmenting a sequence of images / frames (e.g., continuous, sequential, etc.) into multiple segments to be respectively assigned to multiple nodes in a multi-node video coding system, where the parent scene has images / frames on both sides of the segment boundary. Additionally, optionally, or alternatively, a scene may refer to a primary segment within a segment derived from such a parent scene.

[0274] In the segment assigned to node N, JPEG0007602694000233.jpg55 represents the updated first list of scenes, which may be referred to as the second list of scenes obtained after merging of short scenes (e.g., in block 206 of FIG. 2B). These scenes may be referred to as primary scenes. A scene cut of a scene may be used to indicate the first frame index (corresponding to the first image / frame) of the scene. Scenes located on segment boundaries are split into secondary scenes or sub-scenes, such as primary scenes 1 and 3 shown in FIG. 3AG.

[0275] Secondary scenes or subscenes create additional scene cuts that are added to the first list of scenes to create an updated first list of scenes. A node is the first image / frame of a segment f NL Since we can only start encoding from the first image / frame of a segment, f NL The last image / frame of a segment, f(N+1)L-1 is at the end (not at the end of a scene (or the last scene) in the segment), but the last frame f (N+1)L-1 The image / frame immediately following (e.g., outside the segment, in a bumper section, etc.) may be included in the updated list of scenes as an end marker of the scene cut. Figure 3AG shows an example initial scene cut after short scene merging in the top graph ("Node-N (First List of Scenes)") and an example final scene cut after scene splitting in the bottom graph ("Node-N (Second List of Scenes)").

[0276] After scene splitting, primary scenes 1 and 3 in the first list are no longer present in the second list and have been replaced by secondary scenes in the second list. As shown in Figure 3AG, the second list of scenes for node N has one primary scene (P) and six secondary scenes (A, B, C, D, E, and F).

[0277] Let JPEG0007602694000234.jpg521 represent the left and right sets of scene cuts near the left and right segment boundaries of node N, respectively. JPEG0007602694000235.jpg55 can be represented mathematically by the following set operations:

number

[0278] In operating scenarios where the scene is longer than the segment length of the segment, there may not be separate left or right collections, but there will be a single collection of secondary scene cuts within the segment.

[0279] List for Segment Let S be the starting frame index for the kth scene among the K scenes identified in JPEG0007602694000237.jpg55. kwhere k ranges from 0 to K-1. The elements of the second list can be expressed as:

number

[0280] Second list of scenes JPEG0007602694000239.jpg55 may be passed or provided as input to pass 2 along with ancillary data.

[0281] Subscene auxiliary data Second list of scenes JPEG0007602694000240.jpg55 has detailed information about the primary and secondary scenes in the segment. In some operating scenarios, no auxiliary data is provided for the primary scene from pass 1 to pass 2, but auxiliary data is provided for the secondary scenes (or sub-scenes) from pass 1 to pass 2. The auxiliary data includes the total number(s) of overlapping images / frames on either or both the left and right sides of each secondary scene, identifying a specific statistics collection window for the secondary scene, and the FLUT for each secondary scene.

[0282] As mentioned above, a scene in the second list of scenes in the MADC architecture can be either a primary scene or a secondary scene. Pass 2 processes the scene to generate a set of composer metadata parameters for each frame in the scene. The composer metadata, or operational parameters therein, can be included in a video signal (denoted as "VES" or video encoded signal) received by a receiving device and specify a reverse reconstruction mapping (denoted as "RPU" in FIG. 2A) used by the receiving device to generate a reverse reconstruction or reconstructed (e.g., HDR, etc.) image from a forward reconstructed image (denoted as "Reconstructed SDR" in FIG. 2A) encoded in a base layer (denoted as "BL" in FIG. 2A) of the received video signal. Exemplary forward and reverse reconstructions using forward reconstruction metadata parameters (or simply reconstruction metadata or reconstruction parameters) and reverse reconstruction metadata parameters (or composer metadata), respectively, are described in U.S. Provisional Patent Application No. 63 / 013,063, entitled “Reshaping functions for HDR imaging with continuity and reversibility constraints,” by GM. Su, filed April 21, 2020, and U.S. Provisional Patent Application No. 63 / 013,807, entitled “Iterative optimization of reshaping functions in single-layer HDR image codec,” by GM. Su and H. Kadu, filed April 22, 2020, the contents of which are hereby incorporated by reference in their entirety as if fully set forth herein.

[0283] In the pass 2 forward reconstruction operation, the reconstruction parameters can be calculated from all images / frames within the statistics collection window of the scene. For a primary scene, the statistics collection window includes all images / frames in the primary scene, except for any images / frames outside the primary scene. For a secondary scene, the statistics collection window includes all images / frames in the secondary scene, plus some extra images / frames from either or both the previous and next subscenes. These extra images / frames are sometimes referred to as overlap images / frames.

[0284] Thus, for a statistics collection window, a primary scene does not overlap with any adjacent scenes, but a secondary scene is allowed to overlap (e.g., partially) with adjacent secondary scenes, and overlapping images / frames of a subscene are not from any adjacent primary scenes.

[0285] FIG. 3AH illustrates exemplary forward reconstruction statistics windows (or statistics collection windows) in two segments assigned to nodes N and N+1. As illustrated, there are two portions in the two segments, each representing a primary scene in a respective segment of parent scene P. The forward reconstruction statistics window of subscene A (not shown) may have no extra images / frames on the left side, but zero or more extra images / frames on the right side, in addition to the images / frames in subscene A. The forward reconstruction statistics window of subscene H (not shown) may have no extra images / frames on the right side, but may have extra images / frames on the left side. The forward reconstruction statistics windows (not shown) for each of subscenes B, E, F, G may have extra frames on both the left and right.

[0286] As shown in FIG. 3AH, the forward reconstruction statistics window of anchor subscene C has no overlapping images / frames on the left from subscene B, but receives all images / frames from right subscene D via the rear bumper section. The forward reconstruction statistics window of anchor subscene D has no overlapping images / frames on the right from subscene E, but receives all images / frames from left subscene C via the front bumper section. Thus, the forward reconstruction statistics windows of adjacent anchor subscenes C and D are the same. As a result, the same forward and / or backward reconstruction parameters can be calculated by node N and node N+1 for anchor subscenes C and D, even if these anchor scenes are coded separately (e.g., independently, etc.) by two different nodes. This helps to achieve a consistent appearance of adjacent subscenes across nodes.

[0287] In some operating scenarios, trim detection and correction is performed during construction of the FLUT of the subscene. Trim path correction information generated from the trim detection and correction may be included as part of the auxiliary data along with the FLUT of the subscene.

[0288] Scene-based coding in segments In pass 2 of the MADC operation as shown in Figure 2A, each node in the multiple nodes in the multi-node video coding system is allocated or assigned for scene-based video coding using fixed-length segments (except the last node in the multiple nodes). The images / frames in the segments are respectively encoded and combined (e.g., sequentially, etc.) by the multiple nodes into a video signal, such as a collection of multiple video segments or an entire coded bitstream.

[0289] As described above, pass 1 operation of the MADC operation as shown in FIG. 2A generates a list of scenes (e.g., a second list, an updated first list, etc.) that can help reduce the bitrate used to transmit or distribute image metadata within a video signal while maintaining temporal consistency across different portions of the video signal that are encoded by different (e.g., adjacent, etc.) nodes.

[0290] Nodes in pass 2 use this (second) list of scenes to perform scene-based encoding operations for the scenes or sub-scenes identified by the list in the segment. Although the nodes are provided with bumper images / frames in the leading and / or trailing bumper sections, the nodes do not need to encode these bumper images / frames into a video signal. These bumper images / frames, including any anchor sub-scenes therein, may be used to collect forward reconstruction statistics in a temporally stable forward reconstruction related operation.

[0291] Figure 2H illustrates an exemplary process flow for scene-based encoding, e.g., in pass 2 of Figure 2A. As shown, the process flow is provided with images / frames in fixed length segments and bumper images / frames in the leading and / or trailing bumper section(s). Also provided from pass 1 is a (second) list of scenes including primary scenes not subject to scene splits, sub-scenes replacing primary scenes that are subject to scene splits, and auxiliary data for the sub-scenes.

[0292] The (second) list of scenes can include scene cut information that identifies the scenes in the segment. The node can access the SDR image (denoted as "Scene SDR") and the HDR image (denoted as "Scene HDR") in the scene.

[0293] If the scene represents a secondary scene (or sub-scene), the auxiliary data may comprise a portion of the secondary scene. A node may use the auxiliary data portion of the scene to identify and access additional images / frames within the statistics collection window of the sub-scene.

[0294] Primary and secondary scenes can be processed in a similar manner, with two differences. First, the pass 2 operations performed by a node on a primary scene include constructing or generating a FLUT, and the pass 2 operations performed by a node on a secondary scene include copying a FLUT (e.g., frame, subscene, both, etc.) over the auxiliary data from pass 1. In some operating scenarios, the codewords represented within the normalized range in the copied FLUT are not necessarily represented within the codeword range / space available in (the base layer of) a particular bit depth video signal (e.g., 2x100 for an 8-bit bit depth video signal). 8 , 2 for a 10-bit bit-depth video signal 10 Second, the primary scene does not use any overlapping or extra frames for forward reconstruction, but the secondary subscenes can use overlapping or extra frames in adjacent subscenes.

[0295] The backward reconstruction process is exactly the same for the primary and secondary scenes. There is no overlap in the backward reconstruction phase. A combined bitstream consisting of RPU metadata and the compressed base layer is generated as output. The details of each block are described in the following subsections.

[0296] As shown in FIG. 2H, in some operating scenarios, forward reconstruction related operations in scene-based encoding operations include frame-level forward reconstruction statistics collection operations as further shown in FIG. 2I, and frame-level forward reconstruction operations as further shown in FIG. 2J.

[0297] The frame-level forward reconstruction statistics collection operation collects image content dependent statistics for each image / frame within a statistics collection window of a scene.

[0298] As mentioned above, a statistics collection window for a subscene may include overlapping images / frames in adjacent subscene(s), while a statistics collection window for a primary scene that includes only images / frames within the primary scene may not include overlapping images / frames. As mentioned above, information regarding overlapping images / frames for each subscene is included in and can be obtained from the auxiliary data.

[0299] The image content-dependent statistics collected by the frame-level forward reconstruction statistics collection operation of FIG. 2H may include luma (reconstruction) statistics and chroma (reconstruction) statistics, as shown in FIG. 2I. The statistics collected for luma (reconstruction) may be used to generate a scene-level FLUT that reconstructs the luma codeword, as shown in FIG. 2J. The statistics collected for chroma (reconstruction) may be used to generate a scene-level MMR mapping that reconstructs the chroma codeword, as shown in FIG. 2J. An exemplary MMR-based operation is described in U.S. Patent No. 8,811,490, which is incorporated herein by reference in its entirety as if fully set forth herein.

[0300] Luma statistics collection and luma forward reconstruction As shown in Figure 2I, for luma, the histogram ( JPEG0007602694000241.jpg59) can be used to cache or store (e.g., in a frame buffer, etc.) image-dependent statistics from the SDR luma codewords in each (or j-th) SDR image / frame in the acquisition window, while the histogram ( The jth SDR image and the jth HDR image depict the same visual content and form an HDR-SDR pair.

[0301] The (frame-level) forward reconstruction can use the scene-level FLUT to forward reconstruct HDR codewords in each HDR image in the multiple HDR images of the scene into reconstructed SDR codewords in a corresponding reconstructed SDR image in the multiple reconstructed SDR images of the scene.

[0302] As shown at the top of Figure 2J, statistics for each image / frame (or each image / frame pair) in the (current) scene, as collected for luma (reconstruction), can be used to calculate optimized forward reconstruction coefficients for a scene-level FLUT that is specifically optimized for the scene.

[0303] Take the kth scene from a list of K scenes. An image / frame in this (current) scene is denoted by frame index S k ~S k+1 As shown in FIG. 2J, all frame-level SDR histograms for all SDR images / frames in the kth scene may be collectively used or aggregated to generate a scene-level SDR histogram for the scene as follows:

number

[0304] Similarly, all frame-level HDR histograms for all HDR images / frames in a scene can be collectively used or aggregated to generate a scene-level HDR histogram for the scene, as follows:

number

[0305] As shown in the top of FIG. 2J, CDF matching with histograms of both scene levels may be used to generate or create a luma forward reconstruction mapping function from HDR codewords to SDR codewords. Exemplary cumulative density function (CDF) matching operations are described in PCT Application No. PCT / US2017 / 50980, filed September 11, 2017, and U.S. Provisional Patent Application No. 62 / 404,307, filed October 5, 2016 (also published April 5, 2018 as U.S. Patent Application Publication No. 2018 / 0098094), the entire contents of which are incorporated herein by reference as if fully set forth herein. For illustrative purposes only, the luma forward reconstruction mapping function (e.g., scene-level FLUT) obtained through CDF matching may be JPEG0007602694000245.jpg54 is represented as follows:

number

[0306] The FLUT (e.g., scene-level, frame-level, etc.) generated from the CDF matching can be modified to remove or prevent clipping artifacts (or visual artifacts generated from clipping), which may be caused by trim operations added by a colorist (e.g., manually, etc.).

[0307] The operations used to remove or prevent these artifacts are sometimes referred to as trim correction (e.g., luma trim pass correction as shown in FIG. 2J, etc.) and may be applied to remove or prevent visual artifacts caused by clipping from tone mapping, image editing, etc. For a secondary scene (or subscene), the trim-corrected frame-level FLUT of the images / frames within the statistics collection window of the subscene The trim correction operations for the primary and secondary scenes are similar, except that JPEG0007602694000247.jpg54 are averaged to generate the (scene-level or subscene-level) FLUTs for the subscenes, whereas for the primary scene, the trim-corrected scene-level FLUT is used directly for the primary scene. Thus, trim correction is performed at the frame level for the secondary scene, and trim correction is performed at the scene level for the primary scene.

[0308] The clipping-related statistics may be measured or collected in images / frames of the primary scene or sub-scenes. An exemplary trim-path correction operation, including clipping-related statistics, is described in U.S. Provisional Patent Application No. 63 / 089154, entitled “TRIM-PASS CORRECTION FOR CLOUD-BASED CODING OF HDR VIDEO,” filed on October 8, 2020, by HARSHAD KADU et al., the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0309] In some operating scenarios, if it is determined that no trim-related clipping is found, no trim correction is performed, and therefore JPEG0007602694000248.jpg513, where: JPEG0007602694000249.jpg54 shows the FLUT before trimming correction, and T F denotes the final FLUT used in the forward reconstruction. Otherwise, having determined that trim-related clipping has been found, trim correction is performed as follows:

[0310] First, the range-limited DM curve ( JPEG0007602694000250.jpg57) is compared to a reference (e.g., original, unconstrained) DM curve ( The reference DM curve may be constructed or constructed from a set of luminance or luma values ​​from a primary scene or sub-scene image / frame (shown as JPEG0007602694000251.jpg57). Initially, the reference DM curve may be specified or defined based on a group of luminance or luma values ​​from a primary scene or sub-scene image / frame. For example, initially, the reference DM curve JPEG0007602694000252.jpg57 takes as input parameters the minimum, average and maximum HDR luma values ​​in the scene or image / frame ( It can be configured using standard-based or proprietary Display Management (DM) tone mapping functions including JPEG0007602694000253.jpg523.

number

[0311] Differential CDF FLUT JPEG0007602694000255.jpg54 and DM FLUT JPEG0007602694000256.jpg56 are, respectively, their original counterparts as follows: Derived from JPEG0007602694000257.jpg522.

number

number

[0312] The bounded differential DM curve given by the above equation (52) is sometimes called a range-limited differential DM curve. The cumulative sum of the elements represented by the range-limited differential DM curve is expressed as follows: Provides JPEG0007602694000261.jpg57.

number

[0313] The trim correction may be applied at a brightness below the midpoint or mid-range of the overall luminance range, in which case the trim correction may be referred to as low-intensity trim correction. Additionally, optionally or alternatively, the trim correction may be applied at a brightness above the midpoint or mid-range of the overall luminance range, in which case the trim correction may be referred to as high-intensity trim correction.

[0314] For low intensity trim correction, the CDF and DM curves are The CDF and DM curves are merged or combined from JPEG0007602694000263.jpg57 to the merge point determined for the low intensity trim correction. Meanwhile, for the high intensity trim correction, the CDF and DM curves are merged or combined in the forward direction from the merge point of the high intensity trim correction. The merge point is the FLUT entry / index value that marks either the end of the CDF and DM merging in the low intensity trim correction or the start of the CDF and DM merging in the high intensity trim correction.

[0315] An initial estimate of the merging point may be made for the entire SDR codeword range. Exemplary operations for estimating the merging point(s) are described in the aforementioned U.S. Provisional Patent Application No. 63 / 089154. The operations described herein for a secondary scene (or subscene) of a segment may be similarly performed herein for the primary and secondary scenes of a segment.

[0316] The first estimates of the merge points are calculated for the low intensity and high intensity regions as follows: JPEG0007602694000264.jpg510 Shown as JPEG0007602694000265.jpg531.

[0317] CDF curve Using JPEG0007602694000266.jpg54, we inversely map the SDR codewords that represent the first estimates of the merge points in the SDR codeword range to HDR codewords to find the equivalent merge points ( JPEG0007602694000267.jpg524) can be determined or estimated.

number

[0318] These first estimates of the merge point are then refined using the range-restricted DM curve to obtain a second, more accurate estimate of the merge point within the SDR range, as follows: JPEG0007602694000269.jpg524 can be generated.

number

[0319] The second estimate of the merge point in the SDR domain is then multiplied by a second, more accurate estimate of the merge point in the HDR range as follows: It can be inversely mapped to an equivalent HDR merge point value that represents JPEG0007602694000271.jpg524.

number

[0320] The low-intensity and high-intensity portions of the differential CDF curve may be replaced by corresponding low-intensity and high-intensity portions of a range-limited differential DM curve to avoid or prevent clipping.

[0321] A trim correction can be implemented for the difference FLUT curve at an entry / index value located in the low intensity portion below the midpoint or mid-range as follows:

number

[0322] A trim correction can be implemented for the difference FLUT curve at an entry / index value located in the high intensity portion above the midpoint or mid-range as follows:

number

[0323] For entries / index values ​​that are not located in the low and / or high intensity portions below the midpoint or mid-range, no trim correction needs to be implemented for the differential FLUT curves, as follows:

number

[0324] Updated or trimmed difference CDF curve The cumulative sum of JPEG0007602694000276.jpg57 is the (e.g., final, trim-corrected, etc.) luma FLUTT F This results in:

number

[0325] 3AI shows example DM and CDF curves before trim correction (left), example range-limited DM and range-limited CDF curve ranges (center), and a trim-corrected CDF curve that represents a hybrid curve after merging the CDF and DM curves. As shown, the SDR range of the hybrid curve is the same as the SDR range of the CDF curve before trim correction.

[0326] As shown in the top part of Figure 2J, the SDR and HDR histograms are calculated, followed by CDF matching and trim correction in the primary scene. For the secondary scene, the FLUT can be copied from the auxiliary data provided from pass 1.

[0327] Chroma statistics collection and chroma forward reconstruction For chroma, the chroma forward reconstruction mapping used to generate the forward reconstructed chroma codeword may be represented by an MMR-based forward reconstruction mapping, which may be defined or specified in terms of (optimized) operational parameters such as MMR coefficients.

[0328] A three-dimensional mapping table (3DMT), or more precisely, a dynamic 3DMT (or d3DMT) (where "dynamic" means that the 3DMT changes from one HDR-SDR image pair to the next HDR-SDR image pair), can be constructed from the SDR and HDR codewords in the j-th SDR and HDR image / frame in the HDR-SDR image pair in the collection window.

[0329] Exemplary 3DMT or d3DMT construction operations are described in the aforementioned PCT Application No. PCT / US2019 / 031620. After constructing the d3DMT, a chromatic trim pass correction can be performed to update or modify the d3DMT at the frame level, for example as described in [1]. Additionally, optionally or alternatively, a letterbox color consistency process can also be performed to update or modify the d3DMT at the frame level, as described in the aforementioned PCT Application No. PCT / US2019 / 031620.

[0330] A data structure such as the a / B matrix can now be generated from the d3DMT table as follows:

number

[0331] In some operating scenarios, for chroma, chromatic trim correction and letterboxing are performed at the frame level, and the results of these operations are already built into or included in the a / B matrices. Chromatic trim correction can be applied in the same way to the primary or secondary scenes, except that the secondary scene may have extra images / frames as shown in its overlap window (or statistics collection window). Some or all of the information or statistics (e.g., variance, peaks, etc.) obtained from applying luma trim correction can be reused to apply chromatic trim correction.

[0332] A scene-level a / B matrix can be generated by averaging all the a / B matrices of all HDR-SDR image pairs in the scene, as follows:

number

[0333] The (optimized) MMR coefficients of the MMR-based chroma forward reconstruction mapping used to generate forward reconstructed (e.g., SDR) chroma codewords in a reconstructed SDR image from HDR codewords in the HDR image (which correspond to or depict the same visual semantic content as the reconstructed SDR image) can be solved or generated via Gaussian elimination as follows:

number

[0334] Base Layer Generation The optimized forward reconstruction functions / mappings generated at the scene level for the primary and secondary scenes identified in the (second) list of scenes may be applied by multiple nodes of the video encoding system to HDR images in a series of (e.g., input, successive, HDR, etc.) images / frames received with the input video signal to generate corresponding forward reconstructed SDR images in a series of (e.g., output, successive, SDR, etc.) forward reconstructed SDR images. These reconstructed SDR images (not the original SDR images in the HDR-SDR image pairs used to generate the forward reconstruction statistics) may be encoded into the output video signal (e.g., its base layer, etc.). In some operating scenarios, the forward reconstruction parameters generated from the (input, original, etc.) HDR-SDR image pairs may be saved or stored in a binary file in a data store. In some operating scenarios, before the reconstruction mappings / functions are generated, film grain noise may be added or inserted into the (e.g., input, original, etc.) HDR images or luma / luminance codewords therein for the purpose of reducing or avoiding banding, false contours, or visual artifacts caused by compression.

[0335] Scene-based backward reconstruction An output video signal encoded with a forward reconstructed image may include or carry reverse reconstruction metadata (or composer metadata) to enable a receiving device of the output video signal to generate a reconstructed (e.g., HDR, etc.) image that approximates the forward reconstructed input image to the forward reconstructed image. The reverse reconstruction metadata may provide scene-level reverse reconstruction mapping / functions as well as flags / indicators to signal reuse of previously transmitted scene-level reverse reconstruction mapping / functions.

[0336] The reverse reconstruction mapping / function can be specified in the composer metadata with (e.g., optimized, etc.) operational parameters such as polynomial coefficients, MMR coefficients, etc. The optimization used to generate the operational parameters (e.g., of the reverse reconstruction, in the reverse path, etc.) can be performed at the frame level and / or scene level in the same manner for the primary or secondary scenes.

[0337] A backward reconstruction statistics collection operation can be performed to collect backward reconstruction statistics of the (backward) luma reconstruction in a scene. In some operation scenarios, the (scene-level) forward luma reconstruction function can be represented by a single-channel (e.g., a mapping in a single channel of three color channels / components of a color space, a mapping in a luma / luminance channel, etc.) predictor that predicts an output codeword in a single channel from an input codeword in a corresponding single channel.

[0338] In these operating scenarios, a (scene-level) forward luma reconstruction function, such as a first polynomial, may be undoped to obtain a corresponding (scene-level) luma backward reconstruction function, such as a second polynomial, undoped from the first polynomial. The backward luma reconstruction function may be implemented using a backward lookup table or BLUT(T B (denoted as

[0339] The histogram(s) or histogram bins therein used to generate the BLUT can be used to store backward reconstructed statistics collected from the input luma (e.g., HDR, input, original, etc.) codewords and forward reconstructed luma codewords generated based on the scene-based FLUTs made available from the forward path (implementing forward reconstruction). As shown at the top of FIG. 2K, the BLUT can be constructed from the histogram(s) using a histogram-weighted BLUT construction method / algorithm. Exemplary histogram-weighted BLUT construction operations are described in the aforementioned PCT Application No. PCT / US2019 / 031620.

[0340] As shown at the top of Figure 2K, the BLUT can be approximated by a multi-piece polynomial, such as an 8-piece quadratic polynomial. The polynomial coefficients specifying the multi-piece polynomial can be used (in place of the BLUT) as a scene-based backward luma reconstruction function / mapping (denoted as "scene-based luma RPU" in Figure 2K) as part of the composer metadata carried in the output video signal.

[0341] A receiving device for the output video signal may perform a backward reconstruction operation to apply the BLUT to the forward reconstructed luma (e.g., SDR) codeword in each encoded (e.g., SDR, etc.) image of the scene to generate a corresponding backward reconstructed or reconstructed (e.g., HDR, etc.) codeword in a corresponding backward reconstructed or reconstructed (e.g., HDR, etc.) image of the scene.

[0342] The backward reconstruction statistics collection operation may also collect backward reconstruction statistics of the (backward) chroma reconstruction in a scene. In some operating scenarios, the (scene-level) backward luma reconstruction function may be expressed by an MMR-based backward reconstruction mapping with operating parameters such as MMR coefficients as follows: These operating parameters may be optimized using the collected backward reconstruction statistics of the (backward) chroma reconstruction:

[0343] For example, a d3DMT may be constructed based on a forward reconstructed SDR image and the original HDR image from which the forward reconstructed SDR image is derived, in a manner similar to how a d3DMT is constructed for chroma forward reconstruction. In some operating scenarios, chromatic trim path correction may already be performed in the forward path to ensure reversibility between the forward and reverse paths. In some operating scenarios, color correction may be performed for letterboxing.

[0344] The data structure of the a / B matrix etc. at the frame level can be calculated as follows.

number

[0345] The scene-level a / B matrix can be obtained by averaging the a / B matrices of all forward reconstructed images and all input images in the scene as follows:

number

[0346] The (optimized) MMR coefficients can be solved or generated via Gaussian elimination as follows:

number

[0347] Then, at the decoder, a reconstructed (eg, backward reconstructed, HDR, etc.) image may be generated as follows:

number

[0348] Exemplary Process Flow 4A illustrates an exemplary process flow according to one embodiment. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. At block 402, a multi-node video encoding system having multiple video encoding nodes receives as input a sequence of input images in an input video signal and one or more input lists of scenes. The one or more input lists of scenes collectively identify multiple primary scenes in the sequence of input images.

[0349] At block 404, the multi-node system divides the sequence of input images into a plurality of non-overlapping segments, each of which is assigned to a respective node of the plurality of video encoding nodes, each of which receives the input images within a segment as intra-segment images for that segment and receives the input images within one or two bumper sections of the segment as bumper images for the segment.

[0350] In block 406, the multi-node system generates a second list of scenes using the primary scenes identified in the one or more input lists of scenes and segment boundaries defining adjacent segments within the plurality of non-overlapping segments to ensure compliance with the minimum distance constraint and the average distance constraint. The second list of scenes identifies one or more intra-segment scenes within the segment. The one or more intra-segment scenes include zero or more primary scenes that are not on any segment boundary of the segment and one or more secondary scenes split from the one or more primary scenes on one or both segment boundaries of the segment.

[0351] At block 408, the multi-node system generates one or more scene-level forward reconstruction mappings for the one or more secondary scenes, each scene-level forward reconstruction mapping of the one or more scene-level forward reconstruction mappings being generated at least in part from reconstruction statistics collected from input images within an overlap window determined for a respective secondary scene of the one or more secondary scenes in the segment.

[0352] At block 410, the multi-node system, or each node therein, encodes the forward reconstructed images in the segment into an output video signal collectively encoded by the multiple video encoding nodes. The output video signal includes the image metadata portions generated by the respective nodes and enables a receiving device to generate a reconstructed image from the forward reconstructed images in the segment and render a display image derived from the reconstructed image on an image display.

[0353] In one embodiment, the video signal represents a single-layer backwards compatible signal.

[0354] In one embodiment, multiple primary scenes identified by one or more input lists of scenes are generated by (a) merging manually identified scene cuts in an external list (e.g., an XML file, a non-XML file or data, etc.) with scene cuts automatically detected by an automatic scene detector into combined scene cuts, and (b) merging short scenes represented in the combined scene cuts into longer primary scenes.

[0355] In one embodiment, the multi-node system implements fixed length scene partitioning that includes using an average distance constraint to determine a maximum allowable number of primary and secondary scenes for a segment, using an initial secondary scene length to determine a worst-case number of primary and secondary scenes for the segment, and if it determines that the worst-case number is greater than the maximum allowable number, determining a modified secondary scene length that generates one or more secondary scenes in the segment.

[0356] In one embodiment, the multi-node system implements variable length scene partitioning including: (a) generating a sorted set of candidate secondary scene cuts in a segment, a violation set, and a violation set that identifies candidate secondary scenes having smoothness violations within the sorted set of candidate scene cuts; (b) splitting the candidate secondary scenes in the violation set into additional candidate secondary scenes that are updated to the sorted list of candidate secondary scene cuts; (c) updating the violation set to include the additional candidate secondary scenes having smoothness violations in the sorted list of candidate secondary scene cuts; and (d) if it is determined that the one or more stopping criteria are not satisfied, repeating (a)-(d) using the sorted list of candidate secondary scene cuts and the updated violation set until at least one of the one or more stopping criteria is satisfied.

[0357] In one embodiment, one or more scene-level forward reconstruction mappings are adjusted by one or more of forward reconstruction mapping interpolation, offsets, and the like between adjacent secondary scenes.

[0358] In an embodiment, the zero or more primary scenes and the one or more secondary scenes include a specific scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the multi-node system further performs: generating a plurality of SDR histograms from luma SDR codewords in the plurality of SDR images covered by the overlap window of the specific scene; generating a plurality of HDR histograms from luma HDR codewords in the plurality of HDR images covered by the overlap window of the specific scene; generating a scene-level SDR histogram by averaging the plurality of SDR histograms and generating a scene-level HDR histogram by averaging the plurality of HDR histograms; applying CDF matching to the scene-level SDR histogram and the scene-level HDR histogram to generate a CDF matched luma forward reconstruction function; and applying luma trim correction to the CDF matched luma forward reconstruction function to generate a scene-level trim-corrected luma forward reconstruction function of the specific scene.

[0359] In one embodiment, the zero or more primary scenes and the one or more secondary scenes include a specific scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the multi-node system further performs: generating a plurality of frame-level three-dimensional mapping tables (3dMTs) from SDR codewords in the plurality of SDR images covered by the overlap window of the specific scene and from HDR codewords in the plurality of HDR images covered by the overlap window of the specific scene; applying trim correction and letterbox processing to the plurality of frame-level 3dMTs to generate a plurality of trim-corrected 3dMTs; deriving frame-level a / B matrices for the plurality of image pairs using the plurality of trim-corrected 3dMTs; generating a scene-level a / B matrix from the frame-level a / B matrix; and generating optimized multiple color channel multiple regression (MMR) coefficients by solving an optimization problem formulated using the scene-level a / B matrix.

[0360] In one embodiment, each segment boundary of a segment serves as a scene cut for a scene in the second list of scenes.

[0361] In one embodiment, a segment includes at least one of a primary scene entirely within the segment, a subscene derived from a portion of a parent scene representing a primary scene identified in the list of primary scenes, an anchor subscene adjacent to a segment boundary of the segment, a non-anchor subscene not adjacent to any segment boundary of the segment, etc.

[0362] In one embodiment, the segment is adjacent to a second segment in the plurality of segments, and the second segment is assigned to a second different node in the plurality of video encoding nodes for video encoding, and the segment and the second segment share a common anchor sub-scene overlap window formed by an anchor segment in the segment and a second anchor segment in the second segment.

[0363] In one embodiment, the sequence of input images represents a sequence of high dynamic range (HDR) images and the forward reconstructed images encoded into the video signal represent standard dynamic range (SDR) images.

[0364] FIG. 4B illustrates an exemplary process flow according to one embodiment. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) may perform this process flow. At block 452, the image processing system decodes a sequence of forward reconstructed images from a video signal. The video signal has been generated by a plurality of video encoding nodes in a multi-node video encoding system. The video signal includes image metadata generated by the plurality of video encoding nodes. The sequence of forward reconstructed images includes forward reconstructed images encoded into the video signal by one of the plurality of video encoding nodes for one of the plurality of segments used to partition the sequence of source images.

[0365] The sequence of forward reconstructed images includes segments of forward reconstructed images generated by nodes of the multi-node video encoding system using forward reconstruction functions of one or more scenes identified in the segments of the pre-reconstructed source images that are guaranteed by the nodes to comply with minimum and average distance constraints.

[0366] In block 454, the image processing system generates a reconstructed image from the forward reconstructed images in the segment using the image metadata portion in the image metadata, the image metadata portion having been generated by the node.

[0367] In block 456, the image processing system renders a display image derived from the reconstructed image on an image display.

[0368] In one embodiment, a computing device, such as a display device, a mobile device, a set-top box, a multimedia device, etc., is configured to perform any of the aforementioned methods. In one embodiment, an apparatus comprises a processor and is configured to perform any of the aforementioned methods. In one embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the aforementioned methods to be performed.

[0369] In one embodiment, a computing device comprises one or more processors and one or more storage media that store a set of instructions that, when executed by the one or more processors, cause the device to perform any of the methods described above.

[0370] Although separate embodiments are discussed herein, it should be noted that any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0371] Exemplary Computer System Implementation Embodiments of the invention may be implemented using a computer system, a system of electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific integrated circuit (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may implement, control, or execute instructions related to adaptive perceptual quantization of images having extended dynamic range as described herein. The computer and / or IC may calculate any of the various parameters or values ​​associated with the adaptive perceptual quantization process described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0372] Certain embodiments of the present invention comprise a computer processor that executes software instructions that cause the processor to perform the methods of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc., can perform the methods related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product can include any non-transitory medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the embodiments of the present invention. A program product according to an embodiment of the present invention can be in any of a wide variety of forms. The program product can include physical media, such as, for example, magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAMs, etc. The computer-readable signals on the program product can optionally be compressed or encrypted.

[0373] Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referenced above, unless otherwise indicated, the reference to that component (including the reference to "means") is intended to include any components that perform the function of the described component (i.e., functionally equivalent components) as equivalents of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in the illustrated exemplary embodiments of the invention.

[0374] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors that are programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to achieve the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that incorporate hardwired and / or program logic to implement the techniques.

[0375] For example, Figure 5 is a block diagram illustrating a computer system 500 in which one embodiment of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0376] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions executed by processor 504. Main memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0377] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0378] The computer system 500 can be coupled via a bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, which communicates directional information and command selections to the processor 504 and controls cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to specify a position in a plane.

[0379] The computer system 500 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that in combination with the computer system makes the computer system 500 a special-purpose machine or programs it to be a special-purpose machine. According to one embodiment, the techniques described herein are performed by the computer system 500 in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0380] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0381] Storage media is distinct from, but may be used in conjunction with, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0382] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on a storage device 510 either before or after execution by the processor 504.

[0383] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a local area network (LAN) card providing a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0384] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an internet service provider (ISP) 526. The ISP 526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 528. Both local network 522 and Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are exemplary forms of transmission media.

[0385] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0386] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0387] Equivalents, extensions, alternatives and others In the above specification, the embodiments of the present invention have been described with respect to numerous specific details that may vary from implementation to implementation. Therefore, the sole and exclusive indication of what are and are intended by the applicant to be the embodiments of the present invention is the claims, which are a set of claims in their particular form obtained from this application. These claims include any and all subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall determine the meaning of the terms used in the claims. Thus, any limitations, elements, properties, features, advantages or attributes not expressly recited in the claims should in no way limit the scope of such claims. The specification and drawings should therefore be regarded in an illustrative sense and not in a restrictive sense.

[0388] Enumerated exemplary embodiments The present invention may be embodied in any of the forms described herein, including, but not limited to, the following enumerated example embodiments (EEE), which illustrate the structure, features, and functionality of some portions of the embodiments of the present invention.

[0389] EEE1. A method, comprising: receiving as input to a multi-node video encoding system having a plurality of video encoding nodes a sequence of input images in an input video signal and one or more input lists of scenes, the one or more input lists of scenes collectively identifying a plurality of primary scenes in the sequence of input images; Dividing a sequence of input images into a plurality of non-overlapping segments, each segment within the plurality of non-overlapping segments being assigned to a respective node within the plurality of video coding nodes, each node receiving input images within the segment as intra-segment images for the segment and receiving input images within one or two bumper sections of the segment as bumper images for the segment; generating a second list of scenes that ensures compliance with the minimum distance constraint and the average distance constraint using the one or more input lists of scenes and a plurality of primary scenes identified at segment boundaries defining adjacent segments within the plurality of non-overlapping segments, the second list of scenes identifying one or more intra-segment scenes within the segment, the one or more intra-segment scenes including zero or more primary scenes that are not on any segment boundary of the segment and one or more secondary scenes split from the one or more primary scenes on one or both segment boundaries of the segment; generating one or more scene level forward reconstruction mappings for the one or more secondary scenes, each scene level forward reconstruction mapping of the one or more scene level forward reconstruction mappings being generated at least in part from reconstruction statistics collected from input images within an overlap window determined for a respective secondary scene of the one or more secondary scenes in the segment; encoding, by each node, the forward reconstructed images in the segment into an output video signal collectively encoded by the plurality of video encoding nodes, the output video signal including an image metadata portion generated by each node to enable a receiving device to generate a reconstructed image from the forward reconstructed images in the segment and render a display image derived from the reconstructed image on an image display; A method comprising:

[0390] EEE2. The method of EEE1, wherein the image metadata portion includes reverse reconstruction parameters generated for one or more secondary scenes, and the reverse reconstruction parameters are used by the receiving device to generate a reconstructed image by reverse reconstructing, at least in part, the forward reconstructed image using the reverse reconstruction parameters.

[0391] EEE3. The method according to EEE1 or 2, wherein the video signal represents a single-layer backwards compatible signal.

[0392] EEE4. The method of any one of EEE1-3, wherein a plurality of primary scenes identified by one or more input lists of scenes are generated by (a) merging manually identified scene cuts in an external list with scene cuts automatically detected by an automatic scene detector into combined scene cuts, and (b) merging short scenes represented in the combined scene cuts into longer primary scenes.

[0393] EEE5. The method implements fixed-length scene partitioning, the fixed-length scene partitioning comprising: determining a maximum allowable number of primary and secondary scenes for a segment using an average distance constraint; determining a worst case number of primary and secondary scenes for the segment using the initial secondary scene length; determining a modified secondary scene length that generates one or more secondary scenes in the segment if the worst case number is determined to be greater than the maximum allowed number; The method according to any one of EEE1 to 4, comprising:

[0394] EEE6. A method for implementing variable length scene segmentation, the variable length scene segmentation comprising: (a) generating a sorted set of candidate secondary scene cuts in a segment, a violation set, and a violation set identifying candidate secondary scenes having smoothness violations within the sorted set of candidate scene cuts; (b) splitting the candidate secondary scenes in the violation set into additional candidate secondary scenes that are updated into a sorted list of candidate secondary scene cuts; (c) updating the violation set to include the additional candidate secondary scenes in the updated sorted list of candidate secondary scene cuts having smoothness violations; and (d) if it is determined that the one or more stopping criteria are not satisfied, repeating (a)-(d) with the updated sorted list of candidate secondary scene cuts and the updated violation set until at least one of the one or more stopping criteria is satisfied; The method according to any one of EEE1 to 5, comprising:

[0395] EEE7. The method of any one of EEE1-6, wherein one or more scene-level forward reconstruction mappings are adjusted using one or more of forward reconstruction mapping interpolation between adjacent secondary scenes, or offsets.

[0396] EEE8. The zero or more primary scenes and the one or more secondary scenes include a particular scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the method further comprising: generating a plurality of SDR histograms from luma SDR codewords in a plurality of SDR images covered by overlapping windows of a particular scene; generating a plurality of HDR histograms from luma HDR codewords in a plurality of HDR images covered by overlapping windows of a particular scene; generating a scene-level SDR histogram by averaging a plurality of SDR histograms and generating a scene-level HDR histogram by averaging a plurality of HDR histograms; applying CDF matching to the scene-level SDR histogram and the scene-level HDR histogram to generate a CDF-matched luma forward reconstruction function; applying a luma trim correction to the CDF matched luma forward reconstruction function to generate a scene-level trim-corrected luma forward reconstruction function for the particular scene; The method of any one of EEE1-7, further comprising:

[0397] EEE9. The zero or more primary scenes and the one or more secondary scenes include a particular scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the method further comprising: generating a plurality of sets of SDR statistics from luma SDR codewords in a plurality of SDR images covered by overlapping windows of a particular scene; generating a plurality of sets of HDR statistics from luma HDR codewords in a plurality of HDR images covered by overlapping windows of a particular scene; generating a set of scene-level SDR statistics by averaging the multiple sets of SDR statistics and generating a set of scene-level HDR statistics by averaging the multiple sets of HDR statistics; generating a luma forward reconstruction function using the set of scene-level SDR statistics and the set of scene-level HDR statistics; applying a luma trim correction to the luma forward reconstruction function to generate a scene-level trim-corrected luma forward reconstruction function for a particular scene; The method of any one of EEE1-8, further comprising:

[0398] EEE10. Initializing the overlap window of a particular subscene to the length of the particular subscene; (a) using the sorted set of candidate secondary scene cuts and a violation set of subscenes that violate one or more smoothness criteria, determining whether a particular subscene having an overlapping window still violates at least one of the smoothness criteria; (b) if determining that a particular subscene having an overlap window still violates at least one of the smoothness criteria, determining whether adjacent images from adjacent subscenes are available for incrementing the overlap window; (c) incrementing the overlap window using adjacent images from the adjacent subscenes if the adjacent images from the adjacent subscenes are available for incrementing the overlap window; (d) repeating (a)-(d) until a particular subscene with an overlap window no longer violates any of the smoothness criteria or until no adjacent images from adjacent subscenes are available to increment the overlap window; The method of any one of claims 8 to 9, further comprising:

[0399] EEE11. Determining whether a segment includes one or both of a left anchor subscene and a right anchor subscene; If it determines that the segment contains only a right anchor subscene, Compute a right anchor forward reconstruction mapping for the right anchor subscene; Compute a left-adjacent forward reconstruction mapping for the left-adjacent subscene relative to the right anchor subscene; Using the right anchor forward reconstruction mapping for the right anchor subscene, modify the left adjacent forward reconstruction mapping for the left adjacent subscene to a modified left adjacent forward reconstruction mapping for the left adjacent subscene; evaluating a first differential current (DC) difference at one or more first subscene boundaries of the left-adjacent subscene using the modified left-adjacent forward reconstruction mapping for the left-adjacent subscene; performing adding a first offset to the modified left-adjacent forward reconstruction mapping to reduce a first DC difference at one or more first subscene boundaries of the left-adjacent subscenes; If it is determined that the segment contains only a left anchor subscene, Compute a left anchor forward reconstruction mapping for the left anchor subscene; Compute a right-adjacent forward reconstruction mapping for the right-adjacent subscene relative to the left anchor subscene; Using the left anchor forward reconstruction mapping for the left anchor subscene, modify the right adjacent forward reconstruction mapping for the right adjacent subscene to a modified right adjacent forward reconstruction mapping for the right adjacent subscene; evaluating a second DC difference at one or more second subscene boundaries of the right-adjacent subscene using the modified right-adjacent forward reconstruction mapping for the right-adjacent subscene; performing adding a second offset to the modified right-adjacent forward reconstruction mapping to reduce a second DC difference at one or more second subscene boundaries of the right-adjacent subscene; If a segment is determined to contain both left and right anchor subscenes, Calculate a particular anchor forward reconstruction mapping for a particular anchor subscene among the left anchor subscene and the right anchor subscene; Compute adjacent forward reconstruction mappings for adjacent subscenes for a particular anchor subscene; Using the particular anchor forward reconstruction mapping for the particular anchor subscene, modify the adjacent forward reconstruction mapping for the adjacent subscene to a modified adjacent forward reconstruction mapping for the adjacent subscene; evaluating a third DC difference at one or more third subscene boundaries of the adjacent subscenes using the modified adjacent forward reconstruction mapping for the adjacent subscenes; adding a third offset to the modified adjacent forward reconstruction mapping to reduce a third DC difference at one or more third subscene boundaries of the right-adjacent subscene; performing error diffusion calculated as a difference between the third DC difference and a reference frame-by-frame DC difference calculated for each image in the adjacent subscene; The method of any one of EEE1-10, further comprising:

[0400] EEE12. The zero or more primary scenes and the one or more secondary scenes include a particular scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the method further comprising: generating a plurality of frame-level three-dimensional mapping tables (3dMTs) from SDR codewords in a plurality of SDR images covered by overlapping windows of a particular scene and HDR codewords in a plurality of HDR images covered by overlapping windows of a particular scene; applying trim correction and letterboxing to the plurality of frame-level 3dMTs to generate a plurality of trim-corrected 3dMTs; deriving frame-level a / B matrices for multiple image pairs using multiple trim-corrected 3dMT; generating scene-level a / B matrices from the frame-level a / B matrices; generating optimized multi-color channel multiple regression (MMR) coefficients by solving an optimization problem formulated with the scene-level a / B matrices; The method of any one of EEE1-11, further comprising:

[0401] EEE13. The zero or more primary scenes and the one or more secondary scenes include a particular scene having an overlap window, the overlap window covering a plurality of image pairs formed by a plurality of standard dynamic range (SDR) images and a plurality of high dynamic range (HDR) images, and the method further comprising: Generate a plurality of frame level mapping tables from SDR codewords in a plurality of SDR images covered by overlapping windows of a particular scene and HDR codewords in a plurality of HDR images covered by overlapping windows of a particular scene; applying trim correction and letterbox processing to the plurality of frame level mapping tables to generate a plurality of trim-corrected frame level mapping tables; deriving frame-level a / B matrices for a plurality of image pairs using a plurality of trim-corrected frame-level mapping tables; generating scene-level a / B matrices from the frame-level a / B matrices; generating optimized multi-color channel multiple regression (MMR) coefficients by solving an optimization problem formulated with the scene-level a / B matrices; The method of any one of EEE1-12, further comprising:

[0402] EEE14. The method of any one of EEE1-13, wherein each segment boundary of the segment serves as a scene cut for a scene in the second list of scenes.

[0403] EEE15. A method according to any one of EEE1 to 14, wherein the segment includes at least one of a primary scene entirely within the segment, a subscene derived from a portion of a parent scene representing a primary scene identified in the list of primary scenes, an anchor subscene adjacent to a segment boundary of the segment, or a non-anchor subscene not adjacent to any segment boundary of the segment.

[0404] EEE16. The method of any one of EEE1-15, wherein the segment is adjacent to a second segment within the plurality of segments, the second segment being assigned to a second different node within the plurality of video encoding nodes for video encoding, and the segment and the second segment share a common anchor sub-scene overlap window formed by an anchor segment within the segment and a second anchor segment within the second segment.

[0405] EEE17. The method according to any one of EEE1 to 16, wherein the sequence of input images represents a sequence of high dynamic range (HDR) images and the forward reconstructed images encoded in the video signal represent standard dynamic range (SDR) images.

[0406] EEE18. Dividing a segment into one or more subsegments; determining whether each subsegment in the one or more subsegments represents one of the following: (a) preceding a bumper section of the segment; (b) overlapping the rearmost bumper section of the previous segment; (c) not overlapping either the previous segment or a subsequent segment; (d) overlapping the leading bumper section of a subsequent segment; or (e) the rearmost bumper section of the segment; if the sub-segment is determined to represent a bumper section of the segment, merging short scenes within the sub-segment from right to left; if it is determined that the sub-segment overlaps with the rear bumper section of the previous segment, merging the short scenes in the sub-segment from left to right; if it is determined that the sub-segment does not overlap with either the previous segment or the subsequent segment, merging short scenes in the sub-segment from one of the left or right to the other of the left or right; merging short scenes in the subsegment from right to left if it is determined that the subsegment does not overlap with the leading bumper section of the following segment; if it is determined that the sub-segment represents a tail bumper section of the segment, merging short scenes within the sub-segment from left to right; The method of any one of EEE1-17, further comprising:

[0407] EEE19. A method, comprising: Decoding a sequence of forward reconstructed images from a video signal, the video signal being generated by a plurality of video encoding nodes in a multi-node video encoding system, the video signal including image metadata generated by the plurality of video encoding nodes, the sequence of forward reconstructed images including forward reconstructed images encoded into the video signal by nodes in the plurality of video encoding nodes for segments in a plurality of segments used to partition a sequence of source images; the sequence of forward reconstructed images includes segments of forward reconstructed images generated by nodes of the multi-node video encoding system using a forward reconstruction function for one or more scenes identified in the segments of the pre-reconstructed source images, the one or more scenes identified in the segments of the pre-reconstructed source images being ensured by the nodes to comply with a minimum distance constraint and an average distance constraint; generating a reconstructed image from the forward reconstructed image in the segment using an image metadata portion in the image metadata, the image metadata portion being generated by the node; rendering a display image derived from the reconstructed image on an image display; A method comprising:

[0408] EEE20. An apparatus comprising a processor and configured to perform the method according to any one of claims 1 to 19.

[0409] EEE21. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing a method using one or more processors according to the method recited in any one of claims 1-19.

Claims

1. receiving as input to a multi-node video encoding system having a plurality of video encoding nodes a sequence of input images in an input video signal and one or more input lists of scenes, the one or more input lists of scenes collectively identifying a plurality of primary scenes in the sequence of input images, each of the plurality of primary scenes identified by the one or more input lists of scenes being generated by (a) merging manually identified scene cuts in an external list with scene cuts automatically detected by an automatic scene detector into combined scene cuts of adjacent scenes, or (b) merging adjacent scenes represented in the combined scene cuts into a longer primary scene, the adjacent scenes being merged until a length of the combined scene exceeds a threshold representing a predetermined average distance to a next adjacent scene, such that each combined scene satisfies an average distance constraint; dividing the sequence of input images into a plurality of non-overlapping segments, each segment within the plurality of non-overlapping segments being assigned to a corresponding node within the plurality of video coding nodes, the corresponding node receiving as intra-segment images of the segment an input image within the respective segment and an input image within one or two bumper sections of the respective segment, each bumper section including a plurality of images preceding a first image of the respective segment or following a last image of the respective segment; generating a second list of scenes from the plurality of primary scenes in the sequence of input images, the second list of scenes identifying one or more of the plurality of primary scenes as intra-segment scenes within the respective segments, the one or more intra-segment scenes including zero or more primary scenes not on any segment boundary of the respective segments and one or more secondary scenes split from one or more primary scenes on one or both segment boundaries of the respective segments, the secondary scenes including the images of the bumper sections on the respective segment boundaries, each secondary scene having a length, excluding its bumper section, that is greater than or equal to a predetermined minimum distance to a next adjacent scene, whereby each secondary scene satisfies a minimum distance constraint, the predetermined minimum distance being less than the predetermined average distance, and an average length of the scenes in the second list of scenes that is greater than or equal to the predetermined average distance to a next adjacent scene, excluding their bumper sections, whereby the scenes in the second list of scenes satisfy the average distance constraint; generating a scene-level forward reconstruction mapping for each scene of the second list of scenes in the respective segment, the scene-level forward reconstruction mapping capable of mapping an input image to a corresponding forward reconstructed image having a standard dynamic range (SDR) that is smaller than a high dynamic range (HDR) of the input image, each scene-level forward reconstruction mapping being generated at least in part from reconstruction statistics collected from input images within an overlap window determined for each scene of the second list of scenes in the respective segment that includes its bumper section, the reconstruction statistics including (i) luma codewords within an SDR version of each input image within the overlap window aggregated into a scene-level SDR histogram, and (ii) statistics from luma codewords within each input image within the overlap window aggregated into a scene-level HDR histogram; applying, by each node, the generated scene-level forward reconstruction mapping to the input images of the scenes of the second list of scenes for the respective segment excluding its bumper section to generate a forward reconstructed image of the respective segment; encoding, by said respective nodes, the forward reconstructed images in said respective segments into an output video signal collectively encoded by said plurality of video encoding nodes, said output video signal including image metadata portions generated by said respective nodes to enable a receiving device to generate reconstructed images from said forward reconstructed images in said respective segments and to render on an image display a display image derived from said reconstructed images, wherein the same image metadata portion is used for all images of the encoded scene such that said image metadata portion satisfies said minimum distance constraint and said average distance constraint; A method comprising:

2. 2. The method of claim 1 , wherein the image metadata portion includes reverse reconstruction parameters generated for the one or more secondary scenes, the reverse reconstruction parameters being used by the receiving device to generate the reconstructed image by at least in part reverse reconstructing the forward reconstructed image using the reverse reconstruction parameters.

3. 3. The method of claim 1 or 2, wherein the output video signal represents a single-layer backwards-compatible video signal carrying an SDR image optimized or color-graded for an SDR display in a single signal layer.

4. The method implements a fixed length scene partitioning of each of the scenes, the fixed length scene partitioning comprising: determining a maximum allowable number of primary and secondary scenes for the segment using an average distance constraint; determining a worst case number of primary and secondary scenes for the segment using an initial secondary scene length; determining a modified secondary scene length that generates the one or more secondary scenes within the segment if the worst case number is determined to be greater than the maximum allowed number; The method of claim 1 or 2, comprising:

5. The method implements a variable length scene partitioning of each of the scenes, the variable length scene partitioning comprising: (a) generating a sorted set of candidate secondary scene cuts in each of the segments and a violation set identifying candidate secondary scenes having smoothness violations within the sorted set of candidate scene cuts, the candidate secondary scene cuts being sorted in ascending order according to their image positions, and smoothness violations representing DC value differences between two adjacent images that exceed a threshold for making a scene segmentation decision; (b) dividing the candidate secondary scenes in the violation set into additional candidate secondary scenes that are updated to the sorted set of candidate secondary scene cuts; (c) updating the violation set to include additional candidate secondary scenes having the smoothness violations in the updated sorted set of candidate secondary scene cuts; (d) if determining that one or more stopping criteria are not satisfied, repeating (a)-(d) with the updated sorted set of candidate secondary scene cuts and the updated violation set until at least one of the one or more stopping criteria is satisfied, wherein the one or more stopping criteria include at least one of no new splits and no violation sets; The method of claim 1 or 2, comprising:

6. The method of claim 1 or 2, wherein the one or more scene-level forward reconstruction mappings are adjusted using one or more of: forward reconstruction mapping interpolation between adjacent secondary scenes; or offsets.

7. 3. The method of claim 1, wherein each of the segments includes at least one of a primary scene entirely within the respective segment, a subscene derived from a portion of a parent scene representing a primary scene identified in the list of primary scenes, an anchor subscene adjacent to a segment boundary of the respective segment, or a non-anchor subscene that is not adjacent to any segment boundary of the respective segment, an anchor subscene being a secondary scene split from a primary scene on a segment boundary that satisfies the minimum distance constraint, and the anchor subscene ensures temporal consistency across different segments processed by different nodes.

8. 3. The method of claim 1, wherein the respective segments are adjacent to a respective second segment in the plurality of segments, the respective second segment being assigned to a second different node in the plurality of video encoding nodes for video encoding, and the respective segment and the respective second segment share a common anchor sub-scene overlap window formed by an anchor segment in the respective segment and a second anchor segment in the respective second segment. receiving the output video signal at a decoder; decoding the forward reconstructed image from the output video signal in the decoder; generating the reconstructed image from the forward reconstructed image using the image metadata portion in the output video signal; rendering a display image derived from the reconstructed image on an image display; The method of claim 1 further comprising:

10. An apparatus comprising a processor and configured to carry out the method according to claim 1 or 2.

11. A computer program comprising computer executable instructions for carrying out the method according to claim 1 or 2 using one or more processors.

Images