Stereoscopic high dynamic range video

Abstract

Methods and systems for stereoscopic 3D video are described. The input HDR stereoscopic views in the first codeword representation are merged together by a first merging function to generate an input merged view to optimize a remodeling operation that generates a remodeled merged view in the second codeword representation and associated combiner metadata. The remodeled merge view may be segmented and remerged by a second frame packing function to optimize the video coding efficiency of the output encoded bitstream based on the remodeled merge view. In the decoder, after extracting the remodeled merge view from the encoded bitstream, the combiner function applies combiner metadata to the decoded remodeled merge view to generate an output merge view in the first codeword representation. It then generates an output HDR stereoscopic view based on the output merged view.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 338,781, filed on May 5, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to images and more particularly to techniques for stereoscopic transmission of high dynamic range video. Background Art

[0004] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, such as from the darkest gray (black) to the brightest white (highlight). In this sense, DR relates to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR relates to "display-referred" intensities. Unless a particular sense is explicitly specified at any point in the description herein to have a particular meaning, it should be inferred that the terms may be used in either sense, such as interchangeably.

[0005] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans approximately 14 to 15 orders of magnitude across the human visual system (HVS). In practice, the DR, over a wide range of intensity ranges that humans can simultaneously perceive, may be slightly truncated relative to HDR. As used herein, the terms "enhanced dynamic range (EDR) or visual dynamic range (VDR)" may be individually or interchangeably related to the DR that can be perceived within a scene or image by the human visual system (HVS) including eye movement, allowing for some light adaptation changes across the scene or image.

[0006] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). For example, using gamma light luma encoding, images where n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, while images where n≥10 can be considered enhanced dynamic range images. EDR and HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0007] Most consumer desktop monitors currently support 200 to 300 cd / m 2 Most consumer HDTVs have a brightness range of 300 to 500 nits, with newer models reaching 1,000 nits (cd / m 2 ). Thus, such conventional displays represent a lower dynamic range (LDR) relative to HDR or EDR, also referred to as standard dynamic range (SDR). As the availability of HDR content increases due to the development of both capture devices (e.g., cameras) and HDR displays (e.g., Dolby Laboratories' PRM-4200 Professional Reference Monitor), HDR content can be color graded and displayed on HDR displays that support a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more). In general, and without limitation, the methods of the present disclosure relate to any dynamic range above SDR.

[0008] As used herein, the term "display management" refers to the process performed on a receiver to render images for a target display. For example, without limitation, such processes may include tone mapping, gamut mapping, color management, frame rate conversion, etc.

[0009] The creation and playback of high dynamic range (HDR) content is now becoming commonplace, as HDR technology provides more realistic and lifelike images than earlier formats. It is expected that stereoscopic HDR combined with volumetric video will provide a more immersive experience. To improve upon existing encoding schemes, as appreciated by the inventors herein, improved techniques for stereoscopic transmission and display of stereoscopic HDR video have been developed.

[0010] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present disclosure are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0012] Figure 1A Depicting an encoding process for stereoscopic HDR video according to a first example embodiment of the present disclosure;

[0013] Figure 1B Depicting a decoding process for stereoscopic HDR video according to a first example embodiment of the present disclosure;

[0014] Figure 2A and Figure 2B Depicting an encoding process and a decoding process for stereoscopic HDR video according to a second example embodiment of the present disclosure;

[0015] Figure 2C depicts a decoding process for stereoscopic HDR video according to a third example embodiment of the present disclosure; and

[0016] Figure 3 Depicted is merging the left and right stereo views before reshaping according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Methods for stereoscopic HDR video encoding and decoding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the present disclosure may be practiced without these specific details. In other cases, well-known structures and devices are not described in detail in order to avoid unnecessarily obscuring, obscuring, or confusing the present disclosure. Summary of the invention

[0018] Example embodiments described herein relate to a method for a stereoscopic HDR video pipeline. In an embodiment, a processor receives a first view and a second view of a scene in a first codeword representation;

[0019] applying a merging function to merge the first view and the second view into an input merged view in the first codeword representation;

[0020] applying a reshaping process to the input merged view to generate a reshaped merged view in a second codeword representation and combiner metadata, wherein the combiner metadata allows a combiner function operating on the reshaped merged view to generate an output that approximates the input merged view in the first codeword representation;

[0021] applying a partitioning function to the reshaped merged view to generate a first reshaped view and a second reshaped view in the second codeword representation;

[0022] encoding the first reshaped view and the second reshaped view to generate an encoded bitstream;

[0023] and combining the encoded bitstream with the combiner metadata to generate an encoded output.

[0024] In a second embodiment, a processor receives a bitstream comprising reshaped coded data in a second codeword representation and combiner metadata;

[0025] demultiplexing the bitstream to extract the reshaped coded data and the combiner metadata;

[0026] decoding the reshaped encoded data to generate a reshaped merged view in the second codeword representation;

[0027] applying a combiner function to the reshaped merged view to generate an output merged view in a first codeword representation based on the combiner metadata; and

[0028] An output first view and an output second view in the first codeword representation are generated based on the output merged view.

[0029] Stereoscopic HDR Video Architecture

[0030] FIG1 depicts an example embodiment of an encoding pipeline for stereoscopic video. As used herein, the term "metadata" refers to any auxiliary information transmitted as part of an encoded bitstream or sequence and assisting a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or color gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0031] As used herein, the term "forward reshaping" refers to the process of sample-to-sample mapping or codeword-to-codeword mapping of a digital image from its original bit depth and original codeword distribution or representation (e.g., γ, PQ, HLG, etc.) to an image of the same or different bit depth and different codeword distribution or representation. Reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example, without limitation, reshaping can be applied to 10-bit or 12-bit PQ encoded HDR video to improve coding efficiency in a 10-bit video coding architecture. In a receiver, after decompressing the received signal (which may or may not have undergone reshaping), the receiver can apply an "inverse (or backward) reshaping function" to restore the signal to its original codeword distribution and / or achieve a higher dynamic range.

[0032] As depicted in FIG. 1 , the source content includes a left view, a right view, and metadata (102) to assist in proper display management (DM). For example, such metadata may include "L1 metadata" that represents the minimum, median, and maximum luminance values ​​associated with an input frame or image. The L1 metadata may be calculated by converting the RGB data to a luminance-chrominance format (e.g., YCbCr) and then calculating the minimum, median (average), and maximum values ​​in the Y plane, or the metadata may be calculated directly in the RGB space. For example, in an embodiment, L1Min represents the minimum value of the PQ-encoded min(RGB) values ​​of the image, while taking into account active areas (e.g., by excluding gray or black bars, video black edges, etc.). min(RGB) represents the minimum value of the color component values ​​{R, G, B} of a pixel. The values ​​of L1Mid and L1Max may also be calculated in the same manner, replacing the min() function with average() and max() functions. For example, L1Mid represents the average value of the PQ-encoded max(RGB) values ​​of the image, and L1Max represents the maximum value of the PQ-encoded max(RGB) values ​​of the image. In some embodiments, the L1 metadata may be normalized to [0, 1].

[0033] Given two views, the merger unit (105) merges the left view and the right view. The merging can be done horizontally or vertically, and the details will be explained later (depending on the detection of letterboxes and pillar boxes). The merger packs the left view video (whose size is H×W) and the right view video (whose size is H×W) together into a single frame. This merging is performed to optimize the efficiency of the reshaper (110). The packed image is passed to the reshaper (110) to generate the reshaped video and the corresponding combiner metadata (112). The reshaped bitstream is passed to the video codec (125) (e.g., HEVC, VVC, etc.) for compression. The multiplexer (130) multiplexes the encoded video bitstream (127) with the combiner metadata (112) and the DM metadata (102) to generate an output bitstream (132).

[0034] The output of the reshaper (110) is a merged reshape signal. In some embodiments, it may be split back into left and right views in the spatial splitter 115 and go through another frame packing process (120), this time to optimize video encoding (125). For example, depending on the desired encoding format, frame packing options include:

[0035] • Full resolution of each view in a side-by-side packed format, where the coded frame resolution is H×2W.

[0036] Downsample each view horizontally to size H×(W / 2) and pack them side-by-side (SbS) at the final image resolution H×W.

[0037] Downsample each view vertically to size (H / 2)×W and pack them top-bottom (TaB) at the final image resolution H×W.

[0038] In an embodiment, if the output format of the spatial merger unit 105 matches the frame packing format to be used for video encoding, the segmenter unit (115) and the frame packing unit (120) may be removed from the pipeline.

[0039] Note that if half resolution is selected (horizontally downsampled or vertically downsampled), resolution reformatting can also be performed before the left / right view merger to reduce the amount of reconstruction computation by half. Note that the merger (105) and the frame packing (120) can have different formats. For example, the left / right view merger can use TaB to merge the views to facilitate the reconstruction, but the frame packing (120) can split the left and right views and re-merge them for SbS packing to improve coding efficiency or solve other problems.

[0040] The combiner coefficients (112) will be generated from a single merged frame to ensure that the same reshaping function is applied to both views.

[0041] Figure 1B An example embodiment of the decoding process is depicted. The decoder side performs the reverse operation. After demultiplexing (135), a suitable video decoder (140) (matched with encoder 125) will decode the bitstream to obtain a reshaped signal. Then, a combiner (145) will apply the combiner metadata (112) to the decoder signal to reconstruct the HDR signal. Finally, a segmenter 150 will recreate the two views. The display metadata 102 can be passed directly to the display management process (not shown).

[0042] Figure 2A An alternative embodiment of the encoding process is depicted by utilizing a scalable video encoder (125B) (e.g., using HEVC temporal scalability). At the encoder end, for a given left / right pair, it will be spatially merged into a single frame and reshaped. The combiner coefficients will be generated from the single merged frame. Then, in block 115, the merged frame is split into a left reshaped image and a right reshaped image. The left view will be encoded as a first temporal sub-layer, and the right view will be encoded as a second temporal sub-layer.

[0043] At the decoder side, Figure 2BAs depicted in FIG, after demultiplexing (135), the scalable video decoder (140B) will generate two temporal sub-layers, which will be passed to the combiner (145) to generate a reconstructed HDR image. Then, in unit 150B, the left view and the right view will be temporally segmented for subsequent post-processing and display.

[0044] Other decoder implementations are possible, such as having the output of the scalable video decoder (140B) be output to two buffers, and each output processed by its own combiner. Thus, there are two combiners (145L, 145R) operating at half speed. This alternative is Figure 2C Depicted in.

[0045] In an alternative embodiment, a multi-view encoder (such as a multi-view HEVC encoder) may be used. Figure 2B In , the scalable video encoder (125B) is not used, but can be replaced by a multi-view video encoder. Similarly, for the decoder, in Figure 2C In the embodiment of the present invention, the scalable video decoder (140B) can be replaced by a multi-view video decoder.

[0046] Stereo Reshaper

[0047] As described previously, the input to the reshaper is a merged representation of the left and right views. A straightforward way to perform the reshaping and obtain the combiner coefficients is to run the reshaper independently on each view. However, since the reshaping operation is content-dependent, this approach will generate different combiner coefficients for each view. To ensure that both views share the same forward and backward reshaping functions, it is better if the reshaper input is the merged left and right views as a single image. By doing so, the statistics in both views are taken into account together and the reshaping function will be the same for both views. First, various compatibility scenarios are checked.

[0048] Non-Backward Compatibility (NBC)

[0049] In the context of this disclosure, backward compatibility refers to whether a decoder needs to be able to view the SDR version of the incoming content - that is, the incoming input is viewable even if the decoder cannot apply reverse or backward reshaping. In non-backward compatible systems, the decoder is required to apply the reshaping metadata (112) to generate a viewable HDR output.

[0050] In an embodiment, the NBC codec core takes a single image and calculates block-based statistics and techniques (e.g., see references [1-2]) to determine the number of codewords required in each luminance range. Using this information, a forward lookup table (LUT) can be built to reshape the input HDR signal to a lower bit depth in the reshaped domain. The reshaper will also output combiner coefficients that allow the decoder to build a reverse reshape LUT to reconstruct the HDR signal. In order to reuse this single view functionality for stereo signals, the left and right views can be merged together and sent to the single view NBC codec to output the reshaped stereo video and corresponding metadata. Although both side-by-side merging and top-bottom merging are possible, it is noted that a typical system may also have a letterbox detector and a pillarbox detector to exclude those non-textured dark areas from the reshape operation. In order to make existing detectors work without modification, if the video has letterboxes (black borders above and below), it is preferred to use the side-by-side merge format. If the video has pillarbox (black borders on the left and right) detection, the top-bottom merge format is preferred. Thus, at the output of the merger 105, the output images will be two views positioned side by side or two views positioned above and below.

[0051] After reshaping, in unit 115, depending on the selection of the next-stage video encoder (125), the reshaping merged video may be split again into a reshaping left view and a reshaping right view.

[0052] Single Layer Backward Compatibility (SLBC)

[0053] When encoding HDR video in the SLBC format, the design priority is to preserve the fidelity of the HDR content. Therefore, when creating the SDR version, the reshaping is optimized to maximize the preservation of the original HDR view. There are two types of SLBC codecs, depending on the availability of the SDR video version during the reshaping process. These will be discussed separately.

[0054] Reference SDR available.

[0055] When both a reference SDR image and an HDR image of the same view are available, the single-view SLBC reconstruction algorithm can take one HDR frame and one SDR frame as input and generate a reshaped SDR and corresponding combiner coefficients. If the device does not have HDR playback, it can still play the SDR base layer. Alternatively, the HDR decoder can apply the combiner metadata to reconstruct the HDR signal.

[0056] The single-view SLBC algorithm performs CDF matching by constructing a mapping curve to match the histogram between the HDR version and the SDR version of the input image (see, for example, references [3-4]). The algorithm also constructs a dynamic 3D mapping table (d3DMT) by scanning the chrominance color components to solve for the correct combiner-related metadata (reference [5]). Figure 3 As depicted, the left view HDR and the right view HDR can be merged into a single HDR image. In addition, the left view SDR and the right view SDR can also be merged into a single SDR image. Then, the forward reshaping can be reused to output the reshaped SDR signal. The corresponding combiner coefficients are also output.

[0057] Similar to the NBC case, the SLBC pipeline may have a letterbox / pillarbox detector. Thus, depending on the box type, a control signal (302) may be used to control whether the merge is performed in SbS or TaB format. As before, the reshaped merged SDR (307) signal may be split into left and right views after the reshaper, depending on the choice of the next level video codec.

[0058] Reference SDR is not available.

[0059] When the reference SDR is not available, the reshaper operates only on the HDR input to generate a reshape SDR output and corresponding combiner coefficients. Similar to the NBC case, the left view HDR and the right view HDR can be merged as input; and the merged HDR picture is passed to the non-reference SLBC codec (e.g., see references [6-7]) to output a merged reshape SDR and combiner coefficients. As before, depending on the choice of the next-level video codec, the reshape merged SDR image can be split into left and right views after the reshaper.

[0060] Single-layer inverse display mapping (SLiDM)

[0061] In some HDR profiles (such as Dolby Vision Profile 8.4), the original content is in SDR format. Although the decoder can reconstruct the HDR version, the reconstruction is optimized to preserve the original SDR content. With this format, the encoder can apply inverse mapping techniques (for example, inverse tone mapping or inverse display management) to generate the HDR signal (reference [4]). If both SDR and HDR versions are available (reference [4]), the merging of the left and right views is the same as the process described for the SLBC codec (see Figure 3). If no HDR signal is available (reference [8]) and a static mapping from SDR to HDR is chosen, the available SDR signal is transmitted as is, so there is no need to include units 105 and 115. The reshaping coefficients are simply multiplexed with the coded stereo signal.

[0062] For example, in Dolby Vision Profile 8.4, which converts HLG to PQ, the mode operates as an EOTF conversion, so access to the video content is not required. The combiner coefficients are static for the entire video sequence. In other profiles, an HDR version can be created from SDR using dynamic combiner coefficients. These coefficients are created based on the content and characteristics. In this case, the left and right views will need to be merged and content analysis performed to generate the correct metadata 112.

[0063] Stereo coding details

[0064] As previously discussed, the video encoder 125 can be one of a single-layer encoder, a scalable encoder, or a multi-view encoder. These options will be discussed in more detail in this section. Without loss of generality, HEVC (reference [9]) will be used as an example to describe additional details; however, the same concepts can be applied to other codecs (such as AVC, AV1, VVC, etc.). For this discussion, it is assumed that the left view is the base view and the right view is the enhancement view. HEVC specifies three methods to support stereoscopic 3D coding.

[0065] The first method is to use the frame packing arrangement SEI message for single-layer HEVC. For the syntax definition of the syntax parameter frame_packing_arrangement_type, it supports side-by-side (SbS), top-bottom (TaB) and time interleaved (TI), as indicated in Table D.8 of reference [9].

[0066] For SbS, the spatial resolution of the original video is reduced by half horizontally. For TaB, the spatial resolution of the original video is reduced by half vertically. At the decoder, an up-conversion process is performed to restore the decoded video to the original resolution.

[0067] When temporal interleaving is used, the frames retain the original resolution; however, by interleaving the two views, the video frame rate is doubled to achieve stereoscopic 3D. At the decoder, the left view is extracted from the even-numbered frames and the right view is extracted from the odd-numbered frames, or vice versa. In practice, for 8k×2k×96Hz content, this approach would require the use of HEVC Main10, Level 6.2.

[0068] Time Scalability

[0069] The second option is to use the temporal sublayer to support stereoscopic video coding in single-layer HEVC. For example, in this scenario, the left view is encoded with even frames and the right view is encoded with odd frames. This solution also supports backward compatibility in the single-view case, i.e., the decoder can decode only the left view.

[0070] In an embodiment, assuming that the maximum value of TemporalId is set equal to maxTemporalId (maxTemporalId>0), the right view is encoded with TemporalId equal to maxTemporalId, and the left view is encoded with TemporalIds less than maxTemporalId.

[0071] Specifically, in a sequence parameter set (SPS), the syntax sps_max_sub_layers_minus1 is signaled. The signal sps_max_sub_layers_minus1 plus 1 specifies the maximum number of temporal sub-layers that may be present in each coded video sequence (CVS) referencing the SPS. In an embodiment, maxTemporalId may be set equal to the value of sps_max_sub_layers_minus1.

[0072] In the Network Abstraction Layer (NAL) unit header, TemporalId is signaled by the syntax nuh_temporal_id_plus1, which is defined as follows:

[0073] nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit.

[0074] The value of nuh_temporal_id_plus1 is set equal to TemporalId+1.

[0075] Therefore, for the right view, TemporalId is set equal to maxTemporalId. For the left view, TemporalId is set less than maxTemporalId but greater than or equal to 0. With this setting, backward compatibility is supported by extracting a bitstream with TemporalId less than maxTemporalId.

[0076] To encode time-interleaved video more efficiently, the following encoder settings are proposed:

[0077] 1) For inter-view prediction, if a left view decoded picture is used to predict a right view picture, the left view reference picture should be marked as a long-term reference picture. In particular, to be consistent with the MV-HEVC design, only left view decoded pictures with the same time instance should be used as long-term reference pictures for the right view.

[0078] 2) QP adaptability: For conventional temporal scalability, a higher QP is usually used for the highest temporal sub-layer. In this application, since the highest temporal sub-layer is used for coding the right view, the QP rule should be adjusted to achieve the best stereo quality.

[0079] The main reason behind 1) is because of the picture order count (POC) setting in the proposed embodiment.

[0080] 1) POC: By definition, at the same time instance, the left view and the right view should have the same POC. In the proposed solution, POC needs to be assigned based on time interleaving, and the meaning of POC for the left view and the right view no longer has a real time meaning.

[0081] 2) Impact of POC on coding: In HEVC, in merge or advance motion vector prediction (AVMP) mode, a candidate list is created from the motion information of spatial or temporal neighboring prediction blocks. In this process, the motion vector (MV) from the neighboring block can be scaled in time using POC. Since POC is now false, in order not to harm coding efficiency, the inter-view reference picture needs to be marked as a long-term reference picture, thereby disabling the scaling of the MV associated with the long-term reference picture.

[0082] As an example, in HDR encoding, to associate metadata with left and right views, it is necessary to check that nuh_temporal_id_plus1 is different for left and right views, and associate left and right views with the same time instance using POC order and POC difference.

[0083] For example, if POC1 is assigned to the left view, then the associated right view needs to be found. The right view POC2 should have the following characteristics:

[0084] 1) POC2 is larger than POC1 and has the smallest POC gap with POC1;

[0085] 2)nuh_temporal_id_plus1 is equal to maxTemporalId+1.

[0086] In case of supporting temporal scalability, the proposed solution needs to be adjusted to allow additional temporal sub-layer indication within the left and right views. In this case, it may be much simpler to arrange SEI messages and MV-HEVC solutions using frame packing to not confuse intent.

[0087] Such a temporal scalability situation can be as follows: for example, most of a movie is authored at 24fps, but a few scenes are authored at 96fps. Not all decoders can support 96fps, so for compatible decoders, you might want a compatible 24 and an additional 96 decoder. You can set a base frame rate of 24fps, and use temporal scalability only for the 96fps scenes.

[0088] In fact, for 4k×2k×192fps, this implementation requires HEVC Main10 Level 6.1.

[0089] MV-HEVC

[0090] HEVC has two extensions for supporting 3D video: MV-HEVC and 3D-HEVC. The multi-view extension MV-HEVC allows efficient encoding of multiple camera views and associated auxiliary pictures by reusing a single-layer decoder without changing the block-level processing module. It allows inter-view prediction only through high-level syntax (HLS) changes. The 3D video extension 3D-HEVC is aimed at the encoded representation of both multiple views and associated depth maps. It involves changes in the underlying coding tool module. MV-HEVC currently only supports the Multiview Main Profile. In fact, in order to support HDR, it is necessary to define the Multiview Main 10 profile or the Stereo Main 10 profile for stereoscopic video.

[0091] Support for 360-degree video (3DoF)

[0092] The above solution can also be extended to support 360-degree video. The standardized compression scheme for 360-degree video is to first project it onto a 2D plane and then apply the compression scheme used for 2D video. The typical transfer workflow is as follows (reference

[10] ):

[0093] -Multi-camera array captures video and then applies image stitching to obtain spherical video

[0094] - For example, using projections (such as equirectangular (ERP) or cubemaps, etc.) to "unfold" the spherical video onto a 2D plane

[0095] -Use HEVC, AVC, VVC, etc. for video encoding, followed by packaging and delivery

[0096] - At the receiver, receive the 2D video and depacketize and decode it

[0097] - Project a 2D plane back onto a sphere given a specific viewpoint

[0098] - Rendering on the monitor

[0099] Therefore, from the perspective of encoding and delivery, 2D 360-degree video is no different from 2D flat video. For stereoscopic delivery of 360-degree HDR video, the above solution can be applied.

[0100] References

[0101] Each of these references is incorporated by reference in its entirety.

[0102] 1. US10,419,762, "Content-adaptive perceptual quantizer for high dynamic range images".

[0103] 2. US10,032,262, "Block-based content-adaptive reshaping for high dynamic range images".

[0104] 3. US10701375,"Encoding and decoding reversible,production-quality single-layer video signals."

[0105] 4. US10,264,287,"Inverse Luma / Chroma mappings with histogram transfer and approximation".

[0106] 5. US11,277,627,"High-fidelity full reference and high-efficiencyreduced reference encoding in end-to-end single-layer backward compatible encoding pipeline"

[0107] 6. U.S. Patent Application Publication No. 2022-0046245-A1, "Interpolation of reshaping function."

[0108] 7. WIPO PCT Patent Application Publication No. WO 2021 / 216607, "Reshaping functions for HDR imaging with continuity and reversibility constraints".

[0109] 8. GM. Su et al., U.S. patent application Ser. No. 17 / 630,901, filed Jan. 27, 2022, “Electro-optical transfer function conversion and signal legalization.”

[0110] 9.ITU-T Rec.H.265,"High efficiency video coding",ITU, version 08 / 2021.

[0111] 10. Reference: Y.Ye, "Recent trends and challenges in 360-degree video compression", ICME 2018, Hot3D talk.

[0112] Example Computer System Implementation

[0113] Embodiments of the present disclosure may be implemented using a computer system, a system configured with electronic circuit systems and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to image transformations, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the stereoscopic HDR video processing described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0114] Certain embodiments of the present disclosure include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to the stereoscopic HDR video processing as described above by executing software instructions in a program memory accessible to the processor. The present disclosure can also be provided in the form of a program product. The program product may include any tangible and non-transient medium carrying a set of computer-readable signals, the set of computer-readable signals including instructions that cause the data processor to perform the method of the present disclosure when executed by a data processor. According to the program product of the present disclosure, any of various tangible forms may be used. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0115] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents of the components (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the present disclosure.

[0116] Equivalents, extensions, alternatives and miscellaneous

[0117] Thus, example embodiments involving stereoscopic HDR video processing are described. In the foregoing description, embodiments of the present disclosure have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the sole and exclusive indication of the present disclosure and the applicant's inventive intent is the set of claims issued in specific form from this application, wherein such claim issuance includes any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages, or attributes that are not expressly cited in a claim should not limit the scope of such claim in any way. Therefore, the specification and drawings should be viewed in an illustrative and not restrictive sense.

Claims

1. A method for encoding stereoscopic high dynamic range (HDR) video, the method comprising: receiving a first view and a second view of a scene in a first codeword representation; applying a merging function to merge the first view and the second view into an input merged view in the first codeword representation; applying a reshaping process to the input merged view to generate a reshaped merged view in a second codeword representation and combiner metadata, wherein the combiner metadata allows a combiner function operating on the reshaped merged view to generate an output that approximates the input merged view in the first codeword representation; applying a partitioning function to the reshaped merged view to generate a first reshaped view and a second reshaped view in the second codeword representation; encoding the first reshaped view and the second reshaped view to generate an encoded bitstream; and combining the encoded bitstream with the combiner metadata to generate an encoded output.

2. The method of claim 1, wherein: The input merged view includes a side-by-side merge of the first view and the second view or a top-bottom merge of the first view and the second view.

3. The method of claim 2, wherein: A side-by-side merge is applied when the first view and the second view are in a letterbox format, and a top-bottom merge is applied when the first view and the second view are in a pillarbox format.

4. The method of claim 1, wherein: Encoding the first reshaped view and the second reshaped view comprises: The first reshaped view and the second reshaped view frames are packed into a single frame according to a frame packing format, and the single frame is compressed using a video encoder and supplemental enhancement information (SEI) messaging indicating the frame packing format.

5. The method of claim 1, wherein: Encoding the first reshaped view and the second reshaped view includes employing time interleaving.

6. The method of claim 1, wherein: Encoding the first reconstructed view and the second reconstructed view includes employing scalable video coding, multi-view coding, or 3D coding.

7. The method of claim 6, wherein: Encoding with Scalable Video Coding includes: For the second reshaped view, setting TemporalId equal to maxTemporalId; and For the first reshaped view, the TemporalId is set to be less than the maxTemporalId but greater than or equal to 0.

8. The method of claim 1, wherein: Encoding the first reshaped view and the second reshaped view includes merging the first reshaped view and the second reshaped view into a merged reshaped view having a merge format different from a merge format in the input merged view.

9. The method of claim 1, wherein: The remodeling process further comprises: Receiving a first SDR view and a second SDR view of the scene; applying the merging function to merge the first SDR view and the second SDR view into an SDR merged view; and The reshaping process is applied using both the SDR merged view and the input merged view to generate the reshaped merged view in the second codeword representation and the combiner metadata.

10. A method for decoding stereoscopic high dynamic range (HDR) video, the method comprising: receiving a bitstream comprising the reshaped coded data in a second codeword representation and combiner metadata; demultiplexing the bitstream to extract the reshaped coded data and the combiner metadata; decoding the reshaped encoded data to generate a reshaped merged view in the second codeword representation; applying a combiner function to the reshaped merged view to generate an output merged view in a first codeword representation based on the combiner metadata; as well as An output first view and an output second view in the first codeword representation are generated based on the output merged view.

11. The method of claim 10, wherein: Generating the output first view and the output second view includes: receiving a supplemental enhancement information (SEI) message indicating a packing format; and The output first view and the output second view are generated based on the SEI message.

12. The method of claim 10, wherein: Generating the output first view and the output second view includes: receiving a syntax element in the bitstream indicating a time interleaved coding format; and The output first view and the output second view are generated based on time interleaving.

13. The method of claim 12, wherein: If TemporalId is equal to maxTemporalId, extracting the output second view; and If the TemporalId is less than the maxTemporalId but greater than or equal to 0, the output first view is extracted.

14. An apparatus comprising a processor and configured to perform the method of any one of claims 1 to 13.

15. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the instructions being used to execute the method according to any one of claims 1 to 13 using one or more processors.

Images