Adaptive LUMA scaling

Abstract

An example apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: selecting a first sample value range; optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

Description

ADAPTIVE LUMA SCALING TECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to video and image processing and / or coding. Some examples and non-limiting embodiments relate generally to an input analyzer for adaptive luma scaling. BACKGROUND

[0002] It is known to process and render video in a multimedia system. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0004] FIG.1 shows a system pipeline for video coding for machines (VCM).

[0005] FIG.2 shows a designed coding pipeline.

[0006] FIG.3 is a block diagram showing a framework for training an input analyzer.

[0007] FIG.4 is a block diagram showing a framework for training an input analyzer with back- scaling prediction.

[0008] FIG.5 shows an encoder according to an embodiment.

[0009] FIG.6 shows a decoder according to an embodiment.

[0010] FIG.7 is an example apparatus configured to implement the examples described herein.

[0011] FIG.8 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein

[0012] FIG.9 is an example method based on the examples described herein.

[0013] FIG.10 is an example method based on the examples described herein.

[0014] FIG.11 is an example method based on the examples described herein.

[0015] FIG.12 is an example method based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0016] Some embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments may be shown. Indeed, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms ‘data,’ ‘content,’ ‘information,’ and similar terms may be used interchangeably to refer to data capable of being transmitted, received and / or stored in accordance with embodiments of the present invention. Thus, use of any such terms should not be taken to limit the spirit and scope of embodiments.

[0017] Neural networks

[0018] A neural network (NN) may be described as a computation graph consisting of several layers of computation. Each layer may consist of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0019] In some neural networks, such as convolutional neural networks for image classification, initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, whereas intermediate layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc.

[0020] Neural networks are being utilized in an ever-increasing number of applications for many different types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0021] One property of neural nets (and other machine learning tools) is that they are able to learn properties from input data, e.g., in supervised way or in unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0022] In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class orcategory index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output’s error, also referred to as the loss or loss function. Examples of losses are mean squared error, cross-entropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network’s output, i.e., to gradually decrease the loss, by means of gradient descent technique. In one example, at each training iteration, gradients of the loss function with respect to one or more weights or parameters of the NN are computed, for example by backpropagation technique; the computed gradients are then used by an optimization routine, such as Adam or Stochastic Gradient Descent (SGD) to obtain an update to the one or more weights or parameters.

[0023] The terms “model”, “neural network”, “neural net” and “network” are described and used herein interchangeably, and also the weights of neural networks are sometimes referred to herein as learnable parameters or simply as parameters.

[0024] Training a neural network is an optimization process, but the final goal may be different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following things:

[0025] – If the network is learning at all – in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.

[0026] – If the network is learning to generalize – in this case, also the validation set error needs to decrease and to be not too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model may be in the regime of overfitting. This means that the model has just memorized the training set’s properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

[0027] Basic convolution operation

[0028] A 2D convolution layer consists of a weight tensor ^ of size ^^^^ × ^^^ × ^^ × ^^,and takes input tensor ^ of size ^^^ × ^^^ × ^^^ and output a tensor ^ of size ^^^^ × ^^^^ × ^^^^.In the convolution weight tensor, each slice of size ^^^ × ^^ × ^^ may be referred to as “filter”.

[0029] Convolution operations in higher dimension use the same principles.

[0030] Video / image coding

[0031] Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0032] Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0033] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0034] In temporal inter prediction, the sources of prediction are previously decoded pictures in the same scalable layer. In intra block copy (IBC; a.k.a. intra-block-copy prediction), prediction may be applied similarly to temporal inter prediction but the reference picture is the current picture and only previously decoded samples can be referred in the prediction process. Inter-layer or inter- view prediction may be applied similarly to temporal inter prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, interprediction may refer to temporal inter prediction only, while in other cases inter prediction may refer collectively to temporal inter prediction and any of intra block copy, inter-layer prediction, and inter- view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction, temporal inter prediction, or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0035] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0036] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0037] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0038] In typical video codecs the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal referencepictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0039] In typical video codecs the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0040] Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor λ to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area: C = D + λR

[0041] In the equation C = D + λR above, C is the Lagrangian cost to be minimized, D is the image distortion (e.g. Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0042] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards,and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0043] Video Coding for Machines (VCM)

[0044] Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, i.e. consuming / watching the decoded images or videos. Recently, with the advent of machine learning, especially deep learning, there is a rising number of machines (i.e., autonomous agents) that analyze data independently from humans and that may even take decisions based on the analysis results without human intervention. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. For example, such analysis tasks may be performed by neural networks.

[0045] It is likely that the device where the analysis takes place has multiple “machines” or neural networks (NNs). These multiple machines may be used in a certain combination which is for example determined by an orchestrator sub-system. The multiple machines may be used for example in succession, based on the output of the previously used machine, and / or in parallel. For example, a video may be analyzed by one machine (NN) for detecting pedestrians, by another machine (another NN) for detecting cars, and by another machine (another NN) for estimating the depth of all the pixels in the frames.

[0046] Example use cases and applications are self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart TV and smart advertisement, person re- identification, smart traffic monitoring, drones, etc. In addition to image and video data, automatic analysis and processing is increasingly been performed for other types of data, such as audio, speech, text.

[0047] Compressing (and decompressing) data where the end user comprises machines (e.g., neural networks) is commonly referred to as compression or coding for machines. In the case ofvideo data, it is referred to as video compression or coding for machines (VCM).

[0048] Compressing for machines may differ from compressing for humans for example with respect to the algorithms and technology used in the codec, or the training losses used to train any neural network components of the codec, or the evaluation methodology of codecs.

[0049] It is to be understood that, when considering the case of coding for machines, we use the term “receiver-side” or “decoder-side” to refer to the physical or abstract entity or device which contains one or more machines, and runs these one or more machines on some encoded and eventually decoded video representation which is encoded by another physical or abstract entity or device, the “encoder-side device”.

[0050] The encoded video data may be stored into a memory device, for example. as a file. The stored file may later be provided to another device.

[0051] Alternatively, the encoded video data may be streamed from one device to another.

[0052] FIG.1 is a general illustration of a pipeline of Video Coding for Machines. A VCM encoder 104 encodes the input video 102 into a bitstream 106. A bitrate 110 may be computed 108 from the bitstream 106, as a measure of the size of the bitstream. A VCM decoder 112 decodes the bitstream 106 that was produced by the VCM encoder 104. The output of the VCM decoder is referred in the figure as “Decoded data for machines” 114. This data may be considered as the decoded or reconstructed video. However, in some implementations of this pipeline, this data may not have same or similar characteristics as the original video 102 which was input to the VCM encoder 104. For example, this data 114 may not be easily understandable by a human by simply rendering the data onto a screen. The output 114 of VCM decoder 112 is then input to one or more task neural networks (116, 118, 120, 122). In FIG. 1, for the sake of illustrating that there may be any number of task-NNs, there are three example task-NNs (116, 118, 120), and a non-specified one (Task-NN X 122). One goal of VCM may be to obtain a low bitrate while guaranteeing that the task- NNs still perform well in terms of the evaluation metric associated to each task.

[0053] As shown in FIG.1, the performances of the tasks are evaluated (132, 134, 136, 138) to determine task performances (142, 144, 146, 148).

[0054] It is to be understood that, in some cases, the VCM decoder may not be present. In one example, the machines are run directly on the bitstream. In some other cases, the VCM decoder may comprise only a lossless decoding stage, and the lossless decoded data is provided as input to the machines. In yet some other cases, the VCM decoder may comprise a lossless decoding stage following by a dequantization operation, and the loss-decoded and dequantized data is provided asinput to the machines.

[0055] When a conventional video encoder, such as a H.266 / VVC encoder, is used as a VCM encoder, one or more of the following approaches may be used to adapt the encoding to be suitable to machine analysis tasks:

[0056] – One or more regions of interest (ROIs) may be detected. An ROI detection method may be used. For example, ROI detection may be performed using a task NN, such as an object detection NN. In some cases, ROI boundaries of a group of pictures or an intra period may be spatially overlaid and rectangular areas may be formed to cover the ROI boundaries. The detected ROIs (or rectangular areas, likewise) may be used in one or more of the following ways: The quantization parameter (QP) may be adjusted spatially in a manner that ROIs are encoded using finer quantization step size(s) than other regions. For example, QP may be adjusted CTU-wise; The video is preprocessed to contain only the ROIs, while the other areas are replaced by one or more constant values or removed; The video is preprocessed so that the areas outside the ROIs are blurred or filtered; A grid is formed in a manner that a single grid cell covers a ROI. Grid rows or grid columns that contain no ROIs are downsampled as preprocessing to encoding.

[0057] – Quantization parameter of the highest temporal sublayer(s) is increased (i.e. coarser quantization is used) when compared to practices for human watchable video.

[0058] – The original video is temporally downsampled as preprocessing prior to encoding. A frame rate upsampling method may be used as postprocessing subsequent to decoding, if machine analysis at the original frame rate is desired.

[0059] – A filter is used to preprocess the input to the conventional encoder. The filter may be a machine learning based filter, such as a convolutional neural network.

[0060] It is to be understood that, in the context of video coding for machines, the terms “machine vision”, “machine vision task”, “machine task”, “machine analysis”, “machine analysis task”, “computer vision”, “computer vision task”, "task network" and “task” may be used interchangeably.

[0061] Also, it is to be understood that, in the context of video coding for machines, the terms “machine consumption” and “machine analysis” may be used interchangeably.

[0062] Encoder Optimization Information SEI message

[0063] Standardization is ongoing for specifying the Encoder Optimization Information(EOI) Supplemental Enhancement Information (SEI) message.

[0064] The following is an example syntax table of an encoder optimization information SEI message and associated semantics: encoder_optimization_info(payloadSize ) { Descriptor eoi_cancel_flag u(1) if( !eoi_cancel_flag ) { eoi_persistence_flag u(1) eoi_for_human_viewing_idc u(2) eoi_for_machine_analysis_idc u(2) eoi_reserved_zero_2bits u(2) eoi_type u(16) if( EoiObjectBasedFlag ) { eoi_object_based_idc u(16) if( eoi_object_based_idc & 0x02 ) { eoi_quant_threshold_delta ue(v) if( eoi_quant_threshold_delta > 0 ) eoi_pic_quant_object_flag u(1) } } if( EoiTemporalResamplingFlag ) { eoi_temporal_resampling_type_flag u(1) eoi_num_int_pics ue(v) if( eoi_temporal_resampling_type_flag && eoi_num_int_pics > 0 ) eoi_src_pic_flag u(1) } if( EoiSpatialResamplingFlag ) { eoi_orig_pic_dimensions_flag u(1) if( eoi_orig_pic_dimensions_flag ) { eoi_orig_pic_width u(16) eoi_orig_pic_height u(16) } else eoi_spatial_resampling_type_flag u(1) }if( EoiPrivacyProtectionFlag ) { eoi_privacy_protection_method_idc u(16) eoi_privacy_info_type u(8) } } }

[0065] The encoder optimization information SEI message is used to indicate if the video has been optimized for human viewing or machine analysis and which types of optimization have been applied in pre-processing or encoding.

[0066] Use of this SEI message requires the definition of the following variable: A quantization parameter value PicQuant.

[0067] eoi_cancel_flag equal to 1 specifies that the persistence of the encoder optimization information SEI message included in any previous PU in output order is cancelled. eoi_cancel_flag equal to 0 indicates that information on optimization that has been applied in pre-processing or encoding follows.

[0068] eoi_persistence_flag specifies the persistence of the optimization information provided in this SEI message. eoi_persistence_flag equal to 0 specifies that the optimization information applies for the current picture only. eoi_persistence_flag equal to 1 specifies that the optimization information applies for the current picture and all subsequent pictures of the current layer in output order until one or more of the following conditions are true:

[0069] – A new CLVS of the current layer begins.

[0070] – The bitstream ends.

[0071] – A picture in the current layer associated with an encoder optimization information SEI message is output that follows the current picture in output order.

[0072] eoi_for_human_viewing_idc equal to 3 specifies that purposes for the applied optimization include human viewing. eoi_for_human_viewing_idc equal to 2 specifies that the video is suitable but not specifically optimized for human viewing. eoi_for_huma_viewing_idc equal to 1 specifies that the video is unsuitable for human viewing. eoi_for_human_viewing_idc equal to 0 specifies that it is unknown if the video is suitable for human viewing.

[0073] eoi_for_machine_analysis_idc equal to 3 specifies that purposes for the appliedoptimization include machine analysis. eoi_for_machine_analysis_idc equal to 2 specifies that the video is suitable but not specifically optimized for machine analysis. eoi_for_machine_analysis_idc equal to 1 specifies that the video is unsuitable for machine analysis. eoi_for_machine_analysis_idc equal to 0 specifies that it is unknown if the video is suitable for machine analysis.

[0074] It is a requirement of bitstream conformance that the value of eoi_for_human_viewing_idc and eoi_for_machine_analysis_idc shall not be both equal to 1.

[0075] eoi_reserved_zero_2bits shall be equal to 0. Values greater than 0 for eoi_reserved_zero_2bits are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. Decoders conforming to this version of this Specification shall allow any value of eoi_reserved_zero_2bits in the range of 0 to 3, inclusive.

[0076] eoi_type indicates the types of optimization method as specified in Table x1 where ( eoi_type & bitMask ) not equal to 0 indicates that the optimization type with the bitMask value in Table x1 has been applied. When eoi_type is greater than 0 and ( eoi_type & bitMask ) is equal to 0, the optimization type with the bitMask value has not been applied. When eoi_type is equal to 0, optimization as determined by the application has been used. Table x1 – Definition of eoi_type bitMask Interpretation Object-based optimization; the pictures for which this SEI message persists have been pre-processed or encoded so that detected objects in the pictures are 0x01 optimized with respect to other parts of the pictures for the indicated optimization purposes 0x02 Temporal resampling optimization 0x04 Spatial resampling optimization 0x08 Temporal quality optimization in a manner that quality fluctuates temporally Spatial quality optimization; the pictures for which this SEI message persists have been pre-processed or encoded to reduce unnecessary information or improve the 0x10 quality of necessary information.(e.g. to reduce the amount of noise and remove speckles at the picture-level) Privacy protection optimization; the pictures for which this SEI message persists have been pre-processed or encoded to protect personal information. (e.g. removal 0x20 or replacing of personal identifiable information, pseudonymization, anonymization)

[0077] The variables EoiObjectBasedFlag, EoiTemporalResamplingFlag, EoiSpatialResamplingFlag, EoiTemporalQualityFlag, EoiSpatialQualityFlag, and EoiPrivacyProtectionFlag, specifying whether eoi_type indicates the type of the optimization to include object-based optimization, temporal resampling optimization, spatial resampling optimization, temporal quality optimization, spatial quality optimization, and privacy protection optimization, respectively, are derived as follows: EoiObjectBasedFlag = ( ( eoi_type & 0x01 ) > 0 ) ? 1 : 0 EoiTemporalResamplingFlag = ( ( eoi_type & 0x02 ) > 0 ) ? 1 : 0 EoiSpatialResamplingFlag = ( ( eoi_type & 0x04 ) > 0 ) ? 1 : 0 EoiTemporalQualityFlag = ( ( eoi_type & 0x08 ) > 0 ) ? 1 : 0 EoiSpatialQualityFlag = ( ( eoi_type & 0x10 ) > 0 ) ? 1 : 0 EoiPrivacyProtectionFlag = ( ( eoi_type & 0x20 ) > 0 ) ? 1 : 0

[0078] For example, when certain highest temporal sublayers have been encoded with such coarse quantization that human viewers perceive the quality fluctuation annoying, but machine task performance is not compromised, eoi_for_human_viewing_flag and eoi_for_machine_analaysis_flag can be set equal to 0 and 1, respectively, and eoi_type can be set equal to a value that causes EoiTemporalQualityFlag to be equal to 1.

[0079] When eoi_persistence_flag is equal to 0, it is a requirement of bitstream conformance that EoiTemporalResamplingFlag shall be equal to 0 and EoiTemporalQualityFlag shall be equal to 0.

[0080] eoi_object_based_idc, when present, indicates the type of object-based optimization as specified in Table x2, where ( eoi_object_based_idc & bitMask ) not equal to 0 indicates that the object-based optimization type associated with the bitMask value in Table x2 has been applied. When eoi_object_based_idc is greater than 0 and ( eoi_object_based_idc & bitMask ) is equal to 0, the object-based optimization type associated with the bitMask value has not been applied. When eoi_object_based_idc is equal to 0, an application-defined type of object-based optimization has been applied. The value of eoi_object_based_idc shall be in the range of 0 to 31, inclusive, in bitstreams conforming to this version of this Specification. Values of 32 to 65535, inclusive, for eoi_object_based_idc are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. When the value of eoi_object_based_idc is in the range of 32 to 65535, inclusive, decoders conforming to this version of this Specification shall ignore eoi_object_based_idc.Table x2 – Definition of eoi_object_based_idc bitMask Interpretation 0x01 Areas outside the detected objects have been blurred prior to encoding. Areas outside the detected objects have been encoded with coarser 0x02 transform-domain quantization than the quantization used for the detected objects. Areas outside the detected objects have been overwritten with a constant 0x04 sample value. Areas outside the detected objects have been overwritten in some form 0x08 but not with a constant sample value. Areas in the objects have been treated differently based on the object size. 0x10 For example, objects are pre-sorted in size and larger objects are coded with coarser quality than smaller objects during encoding.

[0081] eoi_quant_threshold_delta equal to 0 specifies that the quantization parameter difference between any area outside detected objects and any area that includes one or more detected objects is unknown or unspecified. eoi_quant_threshold_delta greater than 0 is used to indicate the quantization parameter threshold determining areas classified to be outside the detected objects or to include one or more detected objects, depending on the value of eoi_pic_quant_object_flag as specified below.

[0082] eoi_pic_quant_object_flag equal to 1 specifies that areas coded with the quantization parameter value PicQuant represent areas that include one or more detected objects. eoi_pic_quant_object_flag equal to 0 specifies that areas coded with the quantization parameter value PicQuant represent areas outside the detected objects.

[0083] When eoi_pic_quant_object_flag is equal to 1 and eoi__quant_threshold_delta is greater than 0, areas with quantization parameter greater than or equal to PicQuant + eoi_quant_threshold_delta represent areas outside the detected objects. When eoi_pic_quant_object_flag is equal to 0 and eoi_quant_threshold_delta is greater than 0, areas with quantization parameter less than or equal to PicQuant − eoi_quant_threshold_delta represent areas that include one or more detected objects.

[0084] eoi_temporal_resampling_type_flag equal to 0 specifies that the temporal resampling optimization is a subsampling operation. eoi_temporal_resampling_type_flag equal to 1 specifies that the temporal resampling optimization is an upsampling operation.

[0085] eoi_num_int_pics greater than 0 indicates that the count of pictures that the encoding system excluded between each pair of coded pictures in output order (when eoi_temporal_resampling_type_flag is equal to 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag is equal to 1) within the persistence of this SEI message is constant. When eoi_temporal_resampling_type_flag is equal to 0 and eoi_num_int_pics is greater than 0, eoi_num_int_pics specifies the count of pictures that the encoding system excluded between each pair of coded pictures in output order. When eoi_temporal_resampling_type_flag is equal to 1 and eoi_num_int_pics is greater than 0, eoi_num_int_pics specifies the count of pictures that the encoding system added between each pair of source pictures for encoding.

[0086] eoi_num_int_pics equal to 0 indicates that the count of pictures that the encoding system excluded between each pair of coded pictures in output order (when eoi_temporal_resampling_type_flag is equal to 0) or added between each pair of source pictures for encoding (when eoi_temporal_resampling_type_flag is equal to 1) within the persistence of this SEI message is unknown or varying.

[0087] The value of eoi_num_int_pics shall be in the range of 0 to 63, inclusive.

[0088] eoi_src_pic_flag equal to 1 specifies that the picture in the same access unit that contains the EOI SEI message is a source picture for temporal upsampling optimization. eoi_src_pic_flag equal to 0 provides no such indication.

[0089] eoi_orig_pic_dimensions_flag equal to 1 specifies that the eoi_orig_pic_width and eoi_orig_pic_height syntax elements are present. eoi_orig_pic_dimensions_flag equal to 0 specifies that the eoi_orig_pic_width and eoi_orig_pic_height are not present.

[0090] eoi_orig_pic_width and eoi_orig_pic_height, when present, indicate the width and height, respectively, of the original source picture in units of luma samples.

[0091] eoi_spatial_resampling_type_flag equal to 0 specifies that the spatial resampling optimization is a subsampling operation. eoi_spatial_resampling_type_flag equal to 1 specifies that the spatial resampling optimization is an up-sampling operation.

[0092] eoi_privacy_protection_method_idc, when present, indicates the method / algorithm that was used to apply privacy protection optimization. eoi_privacy_protection_method_idc greater than 0 and ( eoi_privacy_protection_method_idc & bitMask ) not equal to 0 indicates that the method / algorithm with the bitMask value in Table x3 has been used to apply privacy protection method. When eoi_privacy_protection_method_idc is equal to 0, the method / algorithm used for privacy protection is unknown or determined by the application.

[0093] The value of eoi_privacy_protection_method_idc shall be in the range of 0 to 15, inclusive, in bitstreams conforming to this version of this Specification. Values of 16 to 255, inclusive, for eoi_privacy_protection_method_idc are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. When the value of eoi_privacy_protection_method_idc is in the range of 16 to 255, inclusive, decoders conforming to this version of this Specification shall ignore eoi_privacy_protection_method_idc. Table x3 – Definition of eoi_privacy_protection_method_idc bitMask Interpretation Blurring; personal information is blurred to make it 0x01 unidentifiable. Replacing; personal information is replaced with 0x02 something different from the original to make it unidentifiable. Masking; personal information is masked so that it cannot 0x04 be identified Pixelation; personal information is pixelated to make it 0x08 undiscernible

[0094] eoi_privacy_info_type, when present, indicates the types of protected information as specified in Table x4 where eoi_privacy_info_type is greater than 0 and ( eoi_privacy_info_type & bitMask ) not equal to 0 indicates that the information type with the bitMask value in Table x4 has been protected. When eoi_privacy_info_type is equal to 0, an application-defined type of information has been protected. The value of eoi_privacy_info_type shall be in the range of 0 to 7, inclusive, in bitstreams conforming to this version of this Specification. Values of 8 to 255, inclusive, for eoi_privacy_info_type are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. When the value of eoi_privacy_info_type is in the range of 8 to 255, inclusive, decoders conforming to this version of this Specification shall ignore eoi_privacy_info_type. Table x4 – Definition of eoi_privacy_info_type bitMask Interpretation Information that identifies a person is protected. For example, the 0x01 face of the person.Information that can identify vehicles is protected. For example, the 0x02 license plate of the vehicle. Information that can infer locations is protected. For example text or 0x04 images on signs.

[0095] Truncating bit depth in video coding for machine tasks

[0096] In JVET-AG0178, a method of truncating the least significant bit in luma sample arrays prior to encoding was presented. The truncation was performed by shifting the luma sample values by 1 bit position to the right. It was asserted that the coding efficiency for object detection and object tracking tasks was improved. JVET-AG0178 tested the performance without left-shifting the decoded luma sample values as well as with left-shifting the decoded luma samples by 1 bit position.

[0097] In JVET-AH0115, it was proposed to indicate in the encoder optimization information (EOI) SEI message when bit depth truncation has been applied. In case it has been applied, three syntax elements were proposed to be included in the EOI SEI message, indicating the count of bit positions that sample values in Y, Cb, and Cr sample arrays, respectively, were right-shifted prior to encoding.

[0098] In an example comparison to bit depth truncation, some embodiments describe a finer- grained method for sample value range compaction. Moreover, bit depth truncation scales the sample values towards 0, whereas embodiments described herein allow scaling with an additive offset, which may be used, e.g., for compacting a sample value range towards its midpoint. Furthermore, the embodiments enable to control the back-scaling ratio.

[0099] Bitdepth range information SEI

[0100] An example implementation described in JVET-AI0182 allocates bits of an original source video into one or more bit depth ranges, with each range coded separately. Prior solutions are limited to two ranges, containing MSBs or LSBs.

[0101] The example implementation enables the ability to code multiple ranges within the same coded picture, by using subpictures or constituent rectangles to represent a range. Prior solutions required that each range be coded in a separate picture, in some cases an auxiliary picture. The invention allows ranges to be coded in single layer pictures or multiple layer pictures in an access unit.

[0102] The example implementation also includes signaling for each range a bit offset toindicate the position of the significant bits of a range in the coded sample. When multiple ranges are included within the same coded picture, this allows the ranges representing more significant bits to be less impacted by compression errors by placing them in high order bit positions, when a quantization parameter is applied to the entire picture. The example implementation allows use of lossy coding and does not require lossless coding.

[0103] The example implementation defines a process to reconstruct a target picture of the same bit-depth as the encoder source picture. Rounding offsets and clipping operations are used in the defined process, according to the signaled parameters, in order to be robust to compression errors. Prior solutions do not define the use of rounding offsets.

[0104] Tone mapping SEI message

[0105] The tone mapping SEI message specified in HEVC provides information to enable remapping of the color samples of the output decoded pictures for customization to particular display environments. The remapping process maps coded sample values in the RGB color space to target sample values. The mappings are expressed either in the luma or RGB color space domain and should be applied to the luma component or to each RGB component produced by color space conversion of the decoded image accordingly.

[0106] The tone mapping SEI message enables bit depth conversion by including the target bit depth indication in the target_bit_depth syntax element.

[0107] The tone mapping SEI message includes the tone_map_model_id syntax element, which indicates the model utilized for mapping the coded data into the target_bit_depth range. A tone_map_model_id of 0 corresponds to a linear mapping with clipping; a tone_map_model_id of 1 corresponds to a sigmoidal mapping; a tone_map_model_id of 2 corresponds to a user-defined table mapping, and a tone_map_model_id of 3 corresponds to a piece-wise linear mapping, tone_map_model_id of 4 corresponds to luminance dynamic range information.

[0108] The tone_map_model_id of 0 is controlled by min_value and max_value syntax elements, for which the semantics is specified as follows: min_value specifies the RGB sample value that maps to the minimum value in the bit depth indicated by target_bit_depth. It is used in combination with the max_value parameter. All sample values in the decoded picture that are less than or equal to min_value, after conversion to RGB as necessary, are mapped to this minimum value in the target_bit_depth representation. max_value specifies the RGB sample value that maps to the maximum value in the bit depth indicated by target_bit_depth. It is used in combination with the min_value parameter. All sample values in the decoded picture that are greater than or equal tomax_value, after conversion to RGB as necessary, are mapped to this maximum value in the target_bit_depth representation.

[0109] In the tone mapping SEI message, the mapping is indicated in the RGB domain rather than for luma or YUV as in this invention report. Moreover, the tone_map_model_id of 0 does not include an offset term in the mapping as proposed in this invention report. Furthermore, we are not aware that the tone mapping SEI message would have been applied to video coding targeted for machines.

[0110] Enhanced asymmetric stereoscopic video coding

[0111] An example implementation proposes reducing the luma dynamic range of one view of stereoscopic video in comparison with the other view and displaying such video on a stereoscopic display, such as a display with polarizing filter that causes alternating sample rows to be assigned to different views, which can be experienced with polarized glasses. A method according to this implementation achieved simultaneously a monoscopic viewing experience when not wearing 3D glasses for viewing and a stereoscopic 3D viewing experience with 3D glasses. The implementation also describes the signaling as follows:

[0112] According to an example, a contrast / brightness scaling indication may be included in various structures of the bitstream, such as a sequence parameter set, a picture parameter set, a video usability information (VUI) structure, a supplemental enhancement information (SEI) message, a sequence header, a group of pictures header, a picture header, or a slice header. If contrast / brightness scaling is required in the decoding loop, contrast / brightness scaling indications should be included in a structure that is required to be decoded entirely (thus excluding VUI and SEI messages).

[0113] Various proposed embodiments aim, for example, at better compression for video that is consumed by a machine vision task. To be more exact, the embodiments aim at a rate- distortion improvement when the distortion is measured by a metric reflecting a machine vision task accuracy.

[0114] Following are some example observations made from a large-scale simulation related to luma sample value range compaction for machine consumption that:

[0115] Instead of right-shifting a luma sample value or multiplying a luma sample value by scaling factor in the range of (0, 1), in some cases it is beneficial to perform middle compacting where an offset is added to the compacted sequence ^ℓafter it has been down-scaled by a scaling factor in the range of (0, 1). This added offsetthat sequences on which middle compactinghave been applied utilize luma values that are in the middle of the full luma range, e.g., centered around the middle value of the full luma range.

[0116] Instead of omitting back-scaling in the decoding system or perform back-scaling back to the source sample value range, in some cases it is beneficial perform back-scaling to a value range that is in between the compacted value range and the source value range. In the simulations, half back-scaling was explored, which yields a value range in the middle of the compacted and source value ranges.

[0117] Some embodiments described herein take advantage of the features described above.

[0118] In an embodiment, an example encoder indicates, in or along a bitstream:

[0119] – selecting a first sample value range;

[0120] – optimizing a video by reducing a source sample value range to the first sample value range; and

[0121] signaling following in or along a bitstream:

[0122] – information that the video has been optimized by reducing the source sample value range to the first sample value range; and

[0123] – information indicative of the first sample value range.

[0124] In an embodiment, the encoder selects the first sample value range so that its minimum value is greater than 0.

[0125] In an embodiment, the encoder selects the first sample value range so that its midpoint is the midmost sample value of the source sample value range.

[0126] In an embodiment, the encoder linearly scales the source sample value range to the first sample value range. In other words, the mapping from the source sample value range to the first sample value range is linear.

[0127] In an embodiment, the encoder indicates, in or along the bitstream, information indicative of a second sample value range to which the video is intended to be back-scaled in a decoding system.

[0128] In an embodiment, the encoder selects the second sample value range so that it is neither the source sample value range nor the first sample value range.

[0129] In an embodiment, an example decoder performs:

[0130] – decoding, from or along a bitstream, that the video has been optimized by reducing a source sample value range to a first sample value range;

[0131] – decoding, from or along the bitstream, or infers the first sample value range;

[0132] – decoding, from or along the bitstream, or infers a second sample value range to which the video is intended to be back-scaled in a decoding system;

[0133] – back-scaling a decoded sample value within the first sample value range linearly to the second sample value range; and

[0134] wherein: – a minimum value of the first sample value range, is greater than 0; and / or – the second sample value range is neither the source sample value range nor the first sample value range.

[0135] In an embodiment, decoder may clip the back-scaled sample values when the back- scaled sample values are out of the range of the second sample value range.

[0136] Clipping value z to the value range of x to y, inclusive, may be performed with the following Clip3 function: x; z < xClip3( x, y, z ) = ^ y ; z > yz ; otherwisemay be described with reference to the term range scaling ratio, which may be defined as firstAbsRange ÷ sourceAbsRange, where the firstAbsRange is the absolute difference between the maximum and minimum values of the first sample value range; and sourceAbsRange is the absolute difference between the maximum and minimum values of the source sample value range.

[0138] Some embodiments may be described with reference to the term range back-scaling ratio or back-scaling ratio, which may be defined as secondAbsRange ÷ firstAbsRange, where secondAbsRange is the absolute difference between the maximum and minimum values of the second sample value range and firstAbsRange is defined like above.

[0139] Some embodiments may be described with reference to the term back-scaling factor, which may be defined as secondAbsRange ÷ sourceAbsRange = backScalingRatio * scalingRatio, where backScalingRatio is the back-scaling ratio and scalingRatio is the range scaling ratio. When back-scaling factor is equal to 1, the back-scaling recovers the source sample value range. When back-scaling factor is less than 1, the second sample value range is narrower than the source sample value range. When back-scaling factor is greater than 1, the second sample value range is wider than the source sample value range.

[0140] Encoder operation for middle compacting

[0141] Referring again to FIG.6, it illustrates an example pipeline of the proposed luma- compacting pre-processing technique.

[0142] Let ^ be a video.

[0143] When the video ^ has a different bit-depth than what the encoder is configured to use, the video ^ may be pre-processed so that it is scaled to the bit-depth configured for theencoder. For example, when the encoder is configured to the bit-depth ^ = 10 and the video ^ hasa lower bit depth, the video ^ may be pre-processed by bit-shifting samples to the left prior to the luma down-scaling.

[0144] The video ^ is composed of pixels ^ = ^^^, ^^ , ^^^ ∈ ^, where ^^, ^^, and ^^represent the luma and chroma channels in the ^′^!^"colorspace.

[0145] The video ^ is first processed by the luma-compacting. Let ℓ ∈ [0; 1] be the luma-compacting factor. The luma-compacting step includes multiplying the luma ^^, of each pixel ^ within the video ^ by the luma-compacting factor ℓ: ∀^ = ^^^ , ^^ , ^^^ ∈ ^,^) = *^ℓ ℓ ℓ^ , ^^ , ^^ + = ^⌊ℓ- + ℓ^^⌉,  ^^ ,  ^^^,

[0146] where ^) = *^ℓ , ^ℓ , ^ℓ^ ^ ^ + ∈ ^ℓ is the pixel which belongs to the luma-compactedvideo ^ℓ, ⌊⋅⌉ is an operator that rounds to the nearest integer, and ℓ- is the luma offset.

[0147] The luma offset ℓ- allows for controlling the value towards which the luma component pℓ3 gets compacted. It is to be noted that the proposed luma-compacting essentially affect the distribution of luma pixel values ^^ℓin the video ^ℓ.

[0148] As example embodiments, two different potential luma-compacting methods are considered, namely zero-compacting and middle-compacting. Zero-compacting compacts the luma pℓ3 towards zero by not using any offset: ℓ- = 0,

[0149] On the other hand, middle-compacting compacts the luma ^^ℓtowards the middle of the luma range: 1− ℓℓ- = ^262 − 1^,

[0150] where ^ is the bit-depth of the video ^. Depending on if zero-compacting or middle- compacting is considered, the luma offset ℓ- value is any of the two defined above.

[0151] In an example embodiment, the luma offset ℓ- value could be any value that ensures that the luma down-scaled component ^^ℓstay in the valid range of a video with a bit-depth ^, i.e.,^ℓ ∈ [0; 26^ − 1]. Consequently, the bounds for the luma offset value are defined asℓ- ∈ [0; ^1 − ℓ^^26 − 1^] in the general case.

[0152] In another example embodiment, a clipping operation may be applied to the luma down-scaled component ^^ℓto ensure the values stay in the valid range of a video with a bit-depth^, e.g., ^ℓ^ ∈ [0; 26 − 1].

[0153] The examples described herein are directed to adaptive selection of luma scaling and backscaling factors. With the example embodiments described herein, higher compression gains than with an approach of using constant luma scaling and backscaling factors may be achieved.

[0154] The examples described herein aim at better compression for images and video that is consumed by a machine vision task. To be more exact, the examples described herein aim at achieving a lower rate-distortion cost, where the rate is a measurement of the size of a bitstream that represents an encoded image or video and the distortion is measured by a metric reflecting a machinevision task accuracy of a decoded image or video, where the decoded image or video is obtained by decoding the encoded image or video (e.g., the bitstream), and where the encoded image or video is obtained by encoding an input image or video, where the encoding and / or the decoding are performed by using at least some of the proposed embodiments.

[0155] In particular, the examples described herein are related to a luma range down-scaling method, also referred to as luma down-scaling for simplicity. Luma down-scaling is an operation where the value range of luma samples in a source video or image is scaled or converted to a narrower value range during compression. This operation may be performed at encoder side, for example as a pre-processing operation. On the other hand, luma up-scaling (also referred to as luma back-scaling, or luma range back-scaling) may be defined as an operation where the down-scaled luma value range is scaled or converted partially or fully back to the original range of the source in decompression. This operation may be performed at decoder side, for example as a post-processing operation after decoding.

[0156] When the decoded and post-processed video is input to machine analysis tasks, the luma range scaling method has shown to significantly enhance coding efficiency for machine analysis. However, it has been observed that the selection of the luma down-scaling factor is critical for the method efficiency. For optimal results the selection of the down-scaling factor should be adaptive depending on the content and characteristics of the input image or video frames. Furthermore, the selection of down-scaling factor is affected by the applied quantization parameter (QP) during compression. In addition, similar observations can be made for the back-scaling factor after reconstruction / decoding. There are indications that the optimal back-scaling factor is related to the down-scaling factor, QP, and the content and characteristics of the input.

[0157] Described herein is a method for encoding comprising: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; encoding the converted source picture to a coded picture; and signaling following in or along a bitstream: a first information indicating that at least the coded picture has been optimized by converting the source sample value range to the first sample value range; and a second information indicating the first sample value range.

[0158] In an embodiment, the output of the neural network inference additionally comprises information indicative of a second sample value range to which a decoded picture resulting fromdecoding of the coded picture is intended to be back-scaled in a decoding system.

[0159] In an embodiment, the method for encoding additionally comprises signaling, in or along the bitstream, a third information for indicating the second sample value range.

[0160] In an embodiment, a value range of the decoded picture (e.g., a luma range) may be back-scaled to two or more ranges, where the two or more ranges may be optimal for respective two or more machine analysis tasks or for respective two or more categories of machine analysis tasks. To this end, the encoder may signal information that is indicative of respective two or more sample value ranges, or may signal information that is indicative of one sample value range from which respective two or more sample value ranges may be derived by the decoder. For example, an image classification task may perform better when its input comprises a first luma range and an object tracking task may perform better when its input comprises a second luma range.

[0161] Described herein is a deep learning-based (e.g., neural network based) luma range predictor (Input analyzer) module and its corresponding training pipeline. The Input analyzer module is designed to analyze source image / video frame and predict an optimal luma down-scaling factor to provide limited luma range that maximizes rate-distortion performance for machine analysis applications, when the output of the input analyzer is encoded and decoded for example by an image or video codec. Optionally, the module can predict a back-scaling factor to determine the optimal luma range for the reconstructed image to maximize machine analysis accuracy. This is based on the coding pipeline shown in FIG.2.

[0162] It is to be understood that some embodiments may be realized without an input analyzer 201. For example, some embodiments may use a pre-defined luma down-scaling factor and / or a pre-defined luma back-scaling factor. In another example, some embodiments may use a lookup table or a function which selects a luma down-scaling factor and / or a luma back-scaling factor based on one or more inputs, such as the quantization parameter value.

[0163] There ^ 202 represents the input image / video source, ℓ 204 represents the luma scaling information, which includes the down-scaling factor and optionally back-scaling factor. The ℓ6206 is back-scaling factor (which may be generated by luma down-scaling 205), which can be into the bitstream and may be derived based on the luma scaling information ℓ 204. “Luma down-scaling” 205 performs the down-scaling of a luma component of the input image / video source 202. The luma down-scaling 205 may be interchangeably referred to as luma sample value range compaction, luma range compaction, luma dynamic range compaction, luma channel compaction, luma component compaction, and in such terms compaction may interchangeably be called scaling, down-scaling, or compacting. ^ℓ208 is the down-scaled input, which is provided as input to anencoder 210, the encoder 210 outputs a bitstream 212 that represents an encoded image / video, thebitstream 212 is decoded by a decoder 214 to obtain a reconstructed down-scaled image / video ^7 ℓ216 and the back-scaling factor ℓ6 206, ^7 ℓ 216 is input to a “luma back-scaling” module 220 thatscales back the output of the decoder 214 based on the back-scaling factor ℓ6 218 and ^7is aback-scaled reconstructed image that be input to one or more machine vision / analysis tasks 224 such as image classification. The luma back-scaling 220 may bereferred to as luma up-scaling, luma dynamic range up-scaling, luma range up-scaling, luma channel up-scaling, luma component up-scaling, and in such terms up-scaling may interchangeably be called restoration or expansion.

[0164] In an embodiment, the luma back-scaling factor 206 may be comprised in an output of an input analyzer 201 and provided to the encoder 210 to be encoded into the bitstream 212.

[0165] In its basic implementation, the input analyzer 201 generates a luma down-scaling factor that is applied uniformly to all luma values of the input. This down-scaling factor, or a value derived from this down-scaling factor, is transmitted within the bitstream 212 to the decoder side to indicate the value range used for source optimization and enable potential back-scaling operations based on that information. In an enhanced variant, input analyzer 201 outputs additionally a predicted back-scaling factor, which is transmitted to the decoder side as the ℓ6206.

[0166] In an embodiment, the input analyzer or another module may indicate to the decoder or to a post-processing module whether a reconstructed picture (e.g., a decoded image / video) is to be back-scaled. In an additional embodiment, it may indicate which components of the reconstructed picture is to be back-scaled. In another additional embodiment, it may indicate one or more regions where back-scaling is to be applied.

[0167] In an embodiment, luma back-scaling may be performed when a criterion is satisfied for at least one or more pictures of the reconstructed video, such as when a confidence score is lower than a predetermined threshold, where the criterion is based on a confidence score of one or more machine vision tasks that take as input the at least one or more pictures.

[0168] In an embodiment, the luma down-scaling factor and the luma back-scaling factor may be comprised in a Supplemental Enhancement Information (SEI) message, such as an Encoder Optimization Information (EOI) SEI message.

[0169] More in general, the EOI SEI message may comprise one or more of the following indications or syntax elements: Whether the encoder has performed range down-scaling; For which components (e.g., luma, chroma) the encoder has performed range down-scaling; One or more rangedown-scaling factors. Each of these may be the actual value, or an indicator or index of the actual value. For example, an encoder may indicate an index associated to an actual value of a down-scaling factor, where the index indexes a predetermined look-up table; Whether the decoder is to perform range back-scaling; For which components (e.g., luma, chroma) the decoder is to perform range back-scaling; One or more range back-scaling factors. Each of these may be the actual value, or an indicator or index of the actual value. For example, an encoder may indicate an index associated to an actual value of a back-scaling factor, where the index indexes a predetermined look-up table; One or more updates to respective one or more predetermined back-scaling factors.

[0170] In an embodiment for training an input analyzer, a forward pass for generating a loss signal comprises: an input analyzer under training; luma down-scaling; reconstruction of a decoded picture; luma back-scaling; estimation of a bitrate (or bitrate estimation) of a coded luma down- scaled picture; derivation of a distortion; wherein the loss is derived based on a weighted sum of the bitrate and the distortion.

[0171] In an embodiment for training an input analyzer, a backward pass comprises propagating gradients of the loss with respect to one or more parameters of the input analyzer backwards to modify the one or more parameters of the input analyzer under training.

[0172] In an embodiment, the reconstruction of a decoded picture from a luma down-scaled picture comprises using a reconstructed picture having a quantization parameter targeted in training and a luma sample value range matching to that provided by the input analyzer under training to derive a residual signal; adding the residual signal to the luma down-scaled picture.

[0173] Thus, the benefit of the described process for the reconstruction of a decoded picture is that encoding and decoding need not be performed for each pass and that end-to-end training can be performed, since the reconstruction step is differentiable.

[0174] In an embodiment, the bitrate estimation is performed using a neural network as a neural network inference, which provides an estimate of bits per pixel, count of bits, or any similar metric.

[0175] It is to be understood that the embodiments for training are not limited to luma range adaptation but may be similarly applied to any pre-processing prior to encoding and / or post- processing subsequent to decoding, wherein the pre-processing and / or post-processing comprise neural-network-based adaptation of the signal to be encoded and / or the signal output by the decoding, respectively.

[0176] To train the Input analyzer in an end-to-end manner, the training framework shown inFIG. 3 was designed for the basic implementation, where the presence and effect of the codec is simulated. Trainer 300 implements the training framework shown in FIG.3.

[0177] Whereas the training framework shown in FIG. 4 was designed for the enhanced variant with back-scaling prediction. Trainer with back-scaling prediction 400 implements the training framework shown in FIG.4.

[0178] The herein described training framework facilitates joint rate-distortion loss optimization. During the training phase, the Input analyzer module's parameters (or at least some of them) are updated through gradient descent optimization, while all other modules' parameters may remain fixed. Inputs for the Input analyzer 201 include the source image 302 / 402 and QP 304 / 404. The bit rate loss (BR_loss) 306 / 406 is computed by the Overfitted BPP estimator module 308 / 408, which estimates the bits per pixel (bpp) of the encoded signal. This deep learning-based (e.g., neural network based) bpp estimator 308 / 408 is pre-trained on the same dataset as the Input analyzer 201 and is intentionally overfitted to provide accurate bitrate loss estimation during Input analyzer 201 training. The offline dataset 310 / 410 was created by encoding the training dataset using combination of luma down-scaling factors (0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0) and QPs (22, 27, 32, 37, 42, 47). During the training process, the reconstructed image corresponding to the predicted scaling factor and input QP is retrieved from storage, and compression artifacts in the form of residuals are added to the downscaled input. This allows accurate distortion loss (D_loss) 312 / 412 computation based on the output of the TaskNNs / DNNs module 314 / 414. The TaskNNs / DNNs module 314 / 414 incorporates any differentiable machine analysis task or feature extractor. Compatible models may include object detection networks (e.g., RetinaNet, YOLO, Faster R-CNN), image segmentation architectures (e.g., U-Net), or backbone networks (e.g., ResNet, VGG16).

[0179] In the basic implementation, the Luma downscaling module 316 / 416 applies the Scaling factor 318 / 418 predicted by the input analyzer 201 by multiplying all luma values by this factor 318 / 418. Subsequently, in the Luma backscale module 320 / 420, complete luma range reversion is achieved by multiplying all luma values by the inverse of the scaling factor 318 / 418 (1 / Scaling factor). In the enhanced variant implementation, the Input analyzer 201 additionally predicts the up-scaling (e.g. back-scaling) factor 422 that is any value between 0 and 1.0. The zero indicates no back-scaling and the 1.0 stands for complete luma range reversion.

[0180] In the training phase, QP-wise lambda (λQP ) 324 / 424 is applied to optimize the rate- distortion balance for the total loss: Ltotal= BR_loss 306 / 406 × λQP324 / 424 + D_loss 312 / 412. In an example, QP-wise lambda values may be experimentally selected based on training data.

[0181] In an embodiment, there may be two or more input analyzers that are associated withrespective two or more QPs or QP-ranges, where each of the two or more input analyzers is trained by using data associated or coded with a corresponding QP or QP range.

[0182] Indications for the sample value ranges used in encoding optimization and controlling the back-scaling

[0183] In an embodiment, it is indicated in, or decoded from, an SEI message that the applied sample value range of luma and / or chroma video content has been optimized.

[0184] In different embodiments, the SEI message may be, but may not be limited to be, an encoder optimization information (EOI) SEI message or a bit-depth range information SEI message.

[0185] In the following, embodiments are described in relation to the encoder optimization information SEI message, but they may likewise be implemented in any other SEI message, such as the bit-depth range information SEI message.

[0186] In an example embodiment, a bit mask position for the eoi_type in the encoder optimization information SEI message is defined for the sample value range optimization. For example, without loss of generality, the value 0x40 may be used for indicating the sample value range optimization and may be defined as follows: Sample value range optimization; the pictures for which this SEI message persists 0x40 have been pre-processed so that the sample value range of one or more of the color components has been compacted.

[0187] In another example embodiment, a bit mask position for the eoi_type in the encoder optimization information SEI message is defined for the sample value range optimization. For example, without loss of generality, the value 0x40 may be used for indicating the sample value range optimization and may be defined as follows: Luma range adaptation optimization; the pictures for which this SEI message persists have been pre-processed or encoded to adapt the average luma computed on the whole 0x40 picture by decreasing or increasing its level (e.g. to reduce the amount of energy consumption at the display)

[0188] In yet another example embodiment, the value 0x40 may be used for indicating the sample value range optimization and may be defined as follows, wherein modification of the samplevalue range may comprise compacting, expansion, and / or shifting: Sample value range optimization; the pictures for which this SEI message persists 0x40 have been pre-processed so that the sample value range of one or more of the color components has been modified.

[0189] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, one or more syntax elements further characterizing the sample value range optimization may be present in the EOI SEI message.

[0190] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_dr_expansion_flag or eoi_ratio_luma_sign_flag, indicating whether sample value range has been compacted or expanded is present in the EOI SEI message.

[0191] In an example embodiment, the semantics of eoi_dr_expansion_flag are specified as follows: When eoi_dr_expansion_flag is equal to 0, the sample value range has been compacted. When eoi_dr_expansion_flag is equal to 1, the sample value range has been expanded.

[0192] In an example embodiment, the semantics of eoi_ratio_luma_sign_flag are specified as follows: eoi_ratio_luma_sign_flag specifies whether the adaptation of the luma is an increase or a decrease of its average value. If eoi_ratio_luma_sign_flag equals 0, the adaptation results in an increase of the average luma. If eoi_ratio_luma_sign_flag equals 1, the adaptation results in a decrease of the average luma.

[0193] In an embodiment, the variables ratioSign, which is equal to −1 when the luma range adaptation has resulted in a decrease of average luma values and equal to 1 when the luma range adaptation has resulted in an increase of average luma values, lumaRatioPercentage, indicating the average luma value resulting from the luma range adaptation in relation to the average luma value of the respective source pictures in percentage, and lumaRatioMult, indicating the ratio of the average luma value resulting from the luma range adaptation relative to the average luma value of the respective source pictures, are derived as follows: ratioSign = 1 − 2 * eoi_ratio_luma_sign_flag lumaRatioPercentage = 100 + ratioSign * eoi_ratio_luma_value lumaRatioMult = lumaRatioPercentage ÷ 100

[0194] In an embodiment, the values of eoi_ratio_luma_value and eoi_ratio_luma_sign_flagare limited to be such that the value of lumaRatioPercentage is not equal to 0.

[0195] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_dr_type or eoi_luma_adaptation_idc, indicates the type and / or method of the sample value range optimization that has been applied. Values of eoi_dr_type or eoi_luma_adaptation_idc may include, but may not be limited to, one or more of the following: – Linear sample value range optimization. – Piece-wise linear sample value range optimization. – Sample value range optimization for optimizing the rate-distortion performance for one or more machine analysis tasks, for example, as described in various embodiments. – Sample value attenuation for display power saving. For example, per-pixel brightness scaling has been used in pre-processing and / or encoding to achieve energy saving on a display device. Brightness scaling or per-pixel sample value attenuation may have been applied in a manner that the perceived quality of an attenuated image may be indistinguishable from or close to the perceived quality of the source image. – Sample value attenuation for converting a higher dynamic range to a lower dynamic range or a higher bit depth to a lower bit depth. For example, a high-dynamic-range (HDR) source image has been converted as part of pre-processing or encoding to be suitable for displaying on a standard-dynamic-range (SDR) display. – Sample value magnification for converting lower dynamic range to a higher dynamic range or a lower bit depth to a higher bit depth. For example, a standard-dynamic-range (SDR) source image has been converted as part of pre-processing or encoding to be suitable for displaying on a high-dynamic-range (HDR) display.

[0196] In an embodiment, one or more syntax elements depending on the value of type of the sample value range optimization may be present in the EOI SEI message. For example, when the type indicates sample value attenuation performed for display power saving, the type of the display for which the optimization has been performed may be indicated. Some example of types of the display may include, but may not be limited to, transmissive-pixel and emissive-pixel displays.

[0197] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_dr_method_idc or eoi_luma_adaptation_idc,indicates a method that has been applied for sample value range optimization. The value of eoi_dr_method_idc may be indicative of a specified algorithm or process. An encoder may select the value of eoi_dr_method_idc among a pre-defined set of values. In an embodiment, the pre- defined set of values may include, but may not be limited to, a value that indicates linear samplevalue scaling, e.g., scaling luma sample value ^^ to ⌊ℓ- + ℓ^^⌉ as described earlier. In anembodiment, the pre-defined set of values may include, but may not be limited to, a first value indicating the zero-compacting and a second value indicating the middle-compacting as described earlier. A decoder may decode the value of eoi_dr_method_idc with reference to the pre-defined set of values to determine the method that have been used for sample value range optimization.

[0198] In an embodiment, the semantics of eoi_luma_adaptation_idc are specified as follows: eoi_luma_adaptation_idc specifies the type or method of luma range adaptation. eoi_luma_adaptation_idc equal to 0 specifies that the luma value range has been adapted for display power saving. eoi_luma_adaptation_idc equal to 1 specifies that luma sample values have been multiplied by lumaRatioMult derived below as a part of the pre-processing prior to encoding.

[0199] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_spatial_variation_flag, indicates whether the sample value range optimization has been applied in a spatially adaptive manner or in a spatially invariant manner. For example, eoi_spatial_variation_flag equal to 0 indicates that the sample value range optimization has been applied in a spatially invariant manner, and eoi_spatial_variation_flag equal to 1 indicates the sample value range optimization may include spatial adaptation.

[0200] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_temporal_variation_flag, indicates whether the sample value range optimization has been applied in a temporally adaptive manner or in a temporally invariant manner. For example, eoi_temporal_variation_flag equal to 0 indicates that the sample value range optimization has been applied in a temporally invariant manner, and eoi_temporal_variation_flag equal to 1 indicates the sample value range optimization may include temporal adaptation.

[0201] In some embodiments, color components for which the sample value range has been optimized are indicated in, or decoded from, one or more syntax element(s) of the SEI message. For example, the SEI message may include a flag, without loss of generality called eoi_dr_dec_chroma_flag. eoi_dr_dec_chroma_flag equal to 0 indicates that the luma sample value range has been optimized, whereas chroma sample arrays are not present or the chroma sample value range has not been optimized. eoi_dr_dec_chroma_flag equal to 1 indicates that both the lumasample value range and the chroma sample value range (for both Cb and Cr) have been optimized.

[0202] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_dr_bit_depth_flag, indicates whether the sample value range optimization has been performed as bit-depth modification (such as bit-depth truncation) or as sample value range modification. For example, eoi_dr_bit_depth_flag equal to 0 indicates that the sample value range has been modified, and eoi_dr_bit_depth_flag equal to 1 indicates that the bit-depth has been modified.

[0203] In an embodiment, information controlling back-scaling of optimized sample value ranges is indicated in, or decoded from, the SEI message. One or more syntax elements may be indicative of each piece of the information, or one or more syntax elements may comprise a piece of information. The information may comprise, but may not be limited to, one or more of the following: – Indication that back-scaling is not to be applied in a decoding system; – Indication that back-scaling is not to be applied in a decoding system, when decoded video content is for machine consumption; – Indication that back-scaling is to be performed to recover the range indicated by the video usability information (VUI) in or along the bitstream; and / or – Indication the back-scaling is controlled by syntax element(s) indicative of a range of sample values of the back-scaled video.

[0204] In an embodiment, information indicating back-scaling performed when optimizing sample value ranges is indicated in, or decoded from, the SEI message. One or more syntax elements may be indicative of each piece of the information, or one or more syntax elements may comprise a piece of information. The information may comprise, but may not be limited to, one or more of the following: – Indication that back-scaling was applied in an encoding system; – Indication that back-scaling was applied in an encoding system, when decoded video content was optimized for machine consumption; – Indication that back-scaling was performed to recover the range indicated by the video usability information (VUI) in or along the bitstream; and / or– Indication the back-scaling was performed as indicated by syntax element(s) indicative of a range of sample values of the back-scaled video.

[0205] Syntax element(s) indicative of a range of sample values may include, but may not be limited to, any of the following: – a bit depth used for deriving the range of sample values; e.g., a range of 0 to 255, inclusive, may be derived for a full range of 8 bits per sample and a range of 0 to 127, inclusive, ma be derived for a full range of 7 bits per sample; – a minimum value and a maximum value of the range of sample values, for example, i) a sample value of a pre-defined point of the range, such as a sample value of the mid-point of the range or a minimum value of the range, and ii) a syntax element indicative of the absolute difference of the endpoint sample values of the range; or – a range scaling ratio and a sample value of a pre-defined point of the range, for example, when mapping a particular sample value of a first range to the second range, a range scaling ratio of 1.5 may indicate that the absolute difference between the particular sample value and the minimum value of a first range is scaled by multiplying the absolute difference by 1.5 and then added to the minimum value of a second range.

[0206] In an embodiment, a range of sample values for the indicated color components of the decoded video is indicated in, or decoded from, the SEI message.

[0207] In an embodiment, when the back-scaling is controlled by syntax element(s) indicative of a range of sample values of the back-scaled video, a range of sample values for the indicated color components of the back-scaled video is indicated in, or decoded from, the SEI message.

[0208] In an example embodiment, a syntax element, without loss of generality called eoi_dr_backscale_idc, is encoded in, or decoded from, the SEI message. eoi_dr_backscale_idc equal to 0 indicates that back-scaling is not to be applied in a decoding system. eoi_dr_backscale_idc equal to 1 indicates that back-scaling is to be applied according to the sample value range inferred from the VUI in or along the bitstream. eoi_dr_backscale_idc equal to 2 indicates that back-scaling is to be applied according to the syntax elements indicative of the range of sample values in the back- scaled video.

[0209] In some embodiments, the following syntax of the EOI SEI message may be used:encoder_optimization_info(payloadSize ) { Descriptor eoi_cancel_flag u(1) if( !eoi_cancel_flag ) { eoi_persistence_flag u(1) eoi_for_human_viewing_idc u(2) eoi_for_machine_analysis_idc u(2) eoi_type u(16) if( EoiObjectBasedFlag ) eoi_object_based_idc ue(v) if( EoiTemporalResamplingFlag ) { eoi_temporal_resampling_type_flag u(1) eoi_num_int_pics ue(v) } if(EoiSpatialResamplingFlag) eoi_spatial_resampling_type_flag u(1) if( EoiPrivacyProtectionFlag ) { eoi_privacy_protection_type_idc u(4) eoi_privacy_protected_info_type u(8) } if( EoiDynamicRangeFlag ) { eoi_dr_dec_chroma_flag u(1) eoi_dr_backscale_idc u(2) if( eoi_dr_dec_chroma_flag | | eoi_dr_backscale_idc = = 2 ) for( i = 0; i <= eoi_dr_dec_chroma_flag; i++ ) eoi_dr_bit_depth_minus8[ i ] ue(v) for( i = 0; i <= eoi_dr_dec_chroma_flag; i++ ) { eoi_dr_dec_min_val[ i ] u(v) eoi_dr_dec_max_val[ i ] u(v) } if( eoi_dr_backscale_idc = = 2 ) for( i = 0; i <= eoi_dr_dec_chroma_flag; i++ ) { eoi_dr_out_min_val[ i ] u(v) eoi_dr_out_max_val[ i ] u(v) }} } }

[0210] eoi_dr_bit_depth_minus8[ i ] + 8 indicates the bit-depth of luma samples (for i equal to 0) and chroma samples (for i equal to 1) and the length of eoi_dr_dec_min_val[ i ], eoi_dr_dec_max_val[ i ], eoi_dr_out_min_val[ i ] (when present), and eoi_dr_out_max_val[ i ] (when present) in bits.

[0211] eoi_dr_dec_min_val[ i ] indicates the minimum luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) that may be present in the decoded sample arrays.

[0212] eoi_dr_dec_max_val[ i ] indicates the maximum luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) that may be present in the decoded sample arrays.

[0213] eoi_dr_out_min_val[ i ] indicates the luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) corresponding to eoi_dr_dec_min_val[ i ].

[0214] eoi_dr_out_max_val[ i ] indicates the luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) corresponding to eoi_dr_dec_max_val[ i ].

[0215] Decoder operations for back-scaling

[0216] In an embodiment, a decoding system determines that back-scaling is to be applied. The determination may be based on, but may not be limited to, one or more of the following: - Decoding from the SEI message that back-scaling is intended to be applied; - Decoding from the SEI message that back-scaling is intended to be applied for machine consumption and providing the back-scaled video to a machine task; or - Selecting the video to be back-scaled based on a machine task being performed in the decoding system.

[0217] When a decoding system has determined that back-scaling is to be applied, it may determine a sample value range to be used for back-scaled video. The determination may be based on, but may not be limited to, one or more of the following: - Decoding from the SEI message information indicative of the sample value range of the back-scaled video;- Decoding from the SEI message information indicative of the range scaling ratio, such as information indicative of the first sample value range (e.g., the sample value range in use in the decoded video) and the source sample value range. The back-scaling ratio may be determined based on the range scaling ratio; o For example, for a selected low value of range scaling ratio, the back-scaling ratio may be determined to be the multiplicative inverse of the range scaling ratio, and for a selected high value of range scaling ratio, the back-scaling ratio may be determined to be 1 (no back-scaling). The back-scaling factor may be a monotonically non-increasing function of the range scaling ratio. - Selecting the back-scaling ratio based on the machine task to be applied; - Selecting the back-scaling ratio based on the spatial resolution; or o For example, for the back-scaling factor may be a monotonically non-increasing function of the range scaling ratio.

[0218] In an embodiment, a decoder decodes from an SEI message as presented above that back-scaling is intended to be applied and information indicative of the sample value range for the back-scaled video and performs the operations in the following paragraphs. It is to be understood that the implementation may be realized similarly with other syntax.

[0219] When eoi_dr_backscale_idc is equal to 1, the use of this SEI message requires the definition of the following variables: VideoFullRangeFlag as specified in ITU-T H.273 | ISO / IEC 23091-2, available from [https: / / www.iso.org / obp / ui / #iso:std:iso-iec:23091:-2:en or https: / / www.itu.int / rec / T-REC-H.273 / en (last accessed on August 20, 2024)].[ i ], specifying the luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) corresponding to eoi_dr_dec_min_val[ i ], and eoiDrOutMaxVal[ i ], specifying the luma sample value (for i equal to 0) or chroma sample value (for i equal to 1) corresponding to eoi_dr_dec_max_val[ i ], are derived as follows: for( i = 0; i <= eoi_dr_dec_chroma_flag; i++ ) { if( eoi_dr_backscale_idc == 0 ) { eoiDrOutMinVal[ i ] = eoi_dr_dec_min_val[ i ] eoiDrOutMaxVal[ i ] = eoi_dr_out_max_val[ i ] } else if( eoi_dr_backscale_idc == 1 ) { if( VideoFullRangeFlag ) {eoiDrOutMinVal[ i ] = 0 eoiDrOutMaxVal[ i ] = ( 1 << ( eoi_dr_bit_depth_minus8[ i ] + 8 ) ) - 1 } else if( i == 0 ) { mult = ( 1 << eoi_dr_bit_depth_minus8[ i ] ) eoiDrOutMinVal[ i ] = mult * 16 eoiDrOutMaxVal[ i ] = mul * 235 } else { / * VideoFullRangeFlag =00 && i == 1 * / mult = ( 1 << eoi_dr_bit_depth_minus8[ i ] ) eoiDrOutMinVal[ i ] = mult * 16 eoiDrOutMaxVal[ i ] = mult * 240 } } else if( eoi_dr_backscale_idc == 2 ) { eoiDrOutMinVal[ i ] = eoi_dr_out_min_val[ i ] eoiDrOutMaxVal[ i ] = eoi_dr_out_max_val[ i ] } }

[0221] In an embodiment, instead of decoding from an SEI message, a decoder concludes that that back-scaling is intended to be applied and concludes a sample value range for the back- scaled video. For example, the back-scaled sample value range may be pre-determined for the type of the machine task or the task network. Similarly, the back-scaled sample value range(s) may be represented by eoiDrOutMinVal[ i ]and eoiDrOutMaxVal[ i ] (i equal to 0 for luma and i equal to 1 for chroma).

[0222] In an embodiment, a luma sample value (i equal to 0) or a chroma sample value (i equal to 1), denoted as decSampleVal, in a decoded sample array is back-scaled to become outSampleVal as follows: decAbsDiff = eoi_dr_dec_max_val[ i ] - eoi_dr_dec_min_val[ i ] outAbsDiff = eoiDrOutMaxVal[ i ] - eoiDrOutMinVal[ i ] outSampleVal = eoiDrOutMinVal[ i ] + Round( ( decSampleVal - eoi_dr_dec_min_val[ i ] ) * outAbsDiff ÷ decAbsDiff ) where ‘÷’ is used to denote division in mathematical equations where no truncation or rounding is intended, and ‘Round’ is a rounding function to the closest integer value.

[0223] Encoder side bitrate control

[0224] Bitrate of the compression is reduced or increased when the input data is scaled to a smaller or larger first value range, respectively.

[0225] In an embodiment, an encoder selects a range scaling ratio to be applied, and to keep the total bitrate approximately unchanged (compared to that achieved with using the source sample value range), modifies the encoding QP value according to the range scaling ratio. For example, when the first range is half of the input range, QP values may be increased by 6 units. As a general rule, QP value can be modified using the following equation: QP2 = QP + 6 * log2(decAbsDiff / inAbsDiff)

[0226] When log2 in the logarithm function in the basis of 2, and decAbsDiff and inAbsDiff are the first range value and the input range value, respectively.

[0227] Additional embodiments

[0228] In an embodiment, an encoder may signal, in or along a bitstream, to a decoder, and the decoder may receive, from or along the bitstream, information indicating one or more first parameters and / or one or more second parameters, wherein the one or more first parameters may be used to determine the first range and the one or more second parameters may be used to determine the second range, and wherein each of the one or more first parameters may indicate a value of the first range out of two or more available values and each of the one or more second parameters may indicate a value of the second range out of two or more available values. In an example, a decoder may comprise a first look-up table with N predefined or predetermined values or sets of values for the first range and a second look-up table with M predefined or predefined values or sets of values for the second range; an encoder may signal to the decoder information indicative of a first index and a second index, where the first index is used to retrieve a first set of values of the first range, the second index is used to retrieve a second set of values of the second range, and M and N are whole numbers.

[0229] Generalizations

[0230] Embodiments may be similarly realized when an encoder expands a sample value range rather than compacts it. A sample value range expansion may be applied, for example, when the source video is detected to be dark or occupy a narrow sample value range.

[0231] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_backscale_ratio_value, indicates whether and by which factor the decoded luma sample values were scaled back in the encoding system whenderiving performance metrics (e.g., mAP) used for optimizing the luma range.

[0232] In an embodiment, the semantics of eoi_backscale_ratio_value are specified as follows: eoi_backscale_ratio_value indicates whether and how the decoded luma sample values were scaled back when performing the optimization for luma range adaptation. eoi_backscale_ratio_value equal to 0 indicates that the decoded luma sample values were not scaled back when performing the optimization for luma range adaptation. eoi_backscale_ratio_value greater than 0 indicates that the decoded sample values were multiplied by lumaBackscaleMult for the performed luma range adaptation optimization, where lumaBackscaleMult was derived as follows: lumaBackscaleMult = eoi_backscale_ratio_value ÷ lumaRatioPercentage

[0233] In an embodiment, luma range optimization for machine analysis is performed so that the back-scaled decoded luma sample arrays are used as input to one or more machine analysis tasks that produce task accuracy metrics, which are used for luma range optimization.

[0234] In an embodiment, a decoding system uses lumaBackscaleMult to multiply the decoded luma sample values to produce back-scaled luma sample arrays, which may be used as input to a machine analysis task.

[0235] In an embodiment, when sample value range optimization is indicated in the EOI SEI message, a syntax element, hereafter referred to as eoi_backscale_ratio_value, indicates whether and by which factor the decoded luma sample values should be scaled back.

[0236] In an embodiment, the semantics of eoi_backscale_ratio_value are specified as follows: eoi_backscale_ratio_value indicates whether and how the decoded luma sample values should be scaled. eoi_backscale_ratio_value equal to 0 indicates that the decoded luma sample values should not be modified. eoi_backscale_ratio_value greater than 0 is used to derive the value of lumaBackscaleMult that should be used to multiply decoded luma sample values as follows: lumaBackscaleMult = eoi_backscale_ratio_value ÷ lumaRatioPercentage

[0237] In an embodiment, a decoder decodes eoi_backscale_ratio_value from the EOI SEI message, derives lumaBackscaleMult, back-scales decoded luma sample values by multiplying them with lumaBackscaleMult, and uses a luma sample array with back-scaled luma sample values as input to a machine analysis task.

[0238] In some embodiments, the following syntax of the EOI SEI message may be used:encoder_optimization_info(payloadSize ) { Descriptor eoi_cancel_flag u(1) if( !eoi_cancel_flag ) { eoi_persistence_flag u(1) eoi_for_human_viewing_idc u(2) eoi_for_machine_analysis_idc u(2) eoi_type u(16) if( EoiObjectBasedFlag ) { eoi_object_based_idc ue(v) if( eoi_object_based_idc & 0x02 ) { eoi_quant_threshold_delta ue(v) if( eoi_quant_threshold_delta > 0 ) eoi_pic_quant_object_flag u(1) } } if( EoiTemporalResamplingFlag ) { eoi_temporal_resampling_type_flag u(1) eoi_num_int_pics ue(v) } if( EoiSpatialResamplingFlag ) { eoi_orig_pic_dimensions_flag u(1) if( eoi_orig_pic_dimensions_flag ) { eoi_orig_pic_width u(16) eoi_orig_pic_height u(16) } else eoi_spatial_resampling_type_flag u(1) } if( EoiPrivacyProtectionFlag ) { eoi_privacy_protection_method_idc u(4) eoi_privacy_info_type u(8) } if( EoiLumaAdaptationFlag ) { eoi_luma_adaptation_idc ue(v) eoi_ratio_luma_value u(7) eoi_ratio_luma_sign_flag u(1)if( eoi_luma_adaptation_idc = = 0 ) eoi_display_model u(4) if( eoi_luma_adaptation_idc = = 1 ) eoi_backscale_ratio_value u(7) } } }

[0239] FIG.5 shows an encoder 500 according to an embodiment. FIG.5 illustrates an image to be encoded (In), a predicted representation of an image block (P′n), a prediction error signal (Dn), a reconstructed prediction error signal (D′n), a preliminary reconstructed image (I′n), a final reconstructed image (R′n), a transform (T) and inverse transform (T−1), a quantization (Q) and inverse quantization (Q−1), entropy encoding (E), a reference frame memory (RFM), inter prediction (Pinter), intra prediction (Pintra), mode selection (MS) and filtering (F).

[0240] FIG.6 shows a decoder 600 according to an embodiment. FIG.6 illustrates a predicted representation of an image block (P′n), a reconstructed prediction error signal (D′n), a preliminary reconstructed image (I′n), a final reconstructed image (R′n), an inverse transform (T−1), an inverse quantization (Q−1), an entropy decoding (E-1), a reference frame memory (RFM), a prediction (either inter or intra) (P), and filtering (F).

[0241] Referring to FIG.5, the output of E comprises coded video data. Then, signaling 502 is encoded as an SEI message and multiplexed with the coded video data by a multiplexer 518 to form the final bitstream 520. Encoder 500 receives down-scaled input 501 as input.

[0242] Referring to FIG.6, one or more SEI messages are decoded first with SEI processing 610. R′nis the one or more reconstructed frame(s). Decoder 600 generates reconstructed down- scaled image / video 614 when receiving bitstream 520 as input.

[0243] A video encoder transforms the input video into a compressed representation suited for storage / transmission and a video decoder decompresses the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0244] A video encoder may encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be codedin a specified manner). Secondly the prediction error, e.g., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0245] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0246] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, e.g., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0247] An intra picture may be defined as a coded picture that is decoded using intra prediction only, or in other words, does not make use of inter prediction in decoding. An intra picture may be interchangeably called an intra frame.

[0248] An inter picture may be defined as a coded picture whose decoding may include intra prediction and inter prediction. An inter picture may be interchangeably called an inter frame.

[0249] FIG. 7 is an example apparatus 700, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 700 comprises at least one processor 702 (e.g., an FPGA and / or CPU and / or GPU), one or more memories 704 including computer program code 705, the computer program code 705 having instructions to carry out the methods described herein, wherein the at least one memory 704 and the computer program code 705 are configured to, with the at least one processor 702, cause the apparatus 700 to implement circuitry, a process, component, module, or function (implemented with control module 706) to implement the examples described herein.

[0250] Apparatus 700 may be a smartphone, personal digital device or assistant, smart television, laptop, pad, tablet, head-mounted display (HMD), or other user device or terminal device. The memory 704 may be a non-transitory memory, a transitory memory, a volatile memory (e.g. RAM), or a non-volatile memory (e.g., ROM).

[0251] Optionally included input analyzer 730 implements the examples described hereinrelated to the input analyzer for adaptive luma scaling. Optionally included input analyzer training implements the examples described herein for training the input analyzer for adaptive luma scaling.

[0252] The apparatus 700 includes a display and / or I / O interface 708, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, etc. The apparatus 700 includes one or more communication e.g. network (N / W) interfaces (I / F(s)) 710. The communication I / F(s) 710 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique including via one or more links 724. The communication I / F(s) 710 may comprise one or more transmitters or one or more receivers.

[0253] The transceiver 716 comprises one or more transmitters 718 and one or more receivers 720. The transceiver 716 and / or communication I / F(s) 710 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas 714 used for communication over wireless link 726.

[0254] The control module 706 of the apparatus 700 comprises one of or both parts 706-1 and / or 706-2, which may be implemented in a number of ways. The control module 706 may be implemented in hardware as control module 706-1, such as being implemented as part of the one or more processors 702. The control module 706-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 706 may be implemented as control module 706-2, which is implemented as computer program code (having corresponding instructions) 705 and is executed by the one or more processors 702. For instance, the one or more memories 704 store instructions that, when executed by the one or more processors 702, cause the apparatus 700 to perform one or more of the operations as described herein. Furthermore, the one or more processors 702, one or more memories 704, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0255] The apparatus 700 to implement the functionality of control 706 may correspond to any of the apparatuses depicted herein. Alternatively, apparatus 700 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 700 may be part of a self- organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0256] The apparatus 700 may also be distributed throughout the network including withinand between apparatus 700 and any network element (such as a base station and / or terminal device and / or user equipment).

[0257] Interface 712 enables data communication and signaling between the various items of apparatus 700, as shown in FIG.7. For example, the interface 712 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g. instructions) 705, including control 706 may comprise object-oriented software configured to pass data or messages between objects within computer program code 705. Computer program code (e.g. instructions) 705, including control 706 may comprise procedural, functional, or scripting code. The apparatus 700 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 700 may at least partially reside in a common housing 728, or a subset of the various components of apparatus 700 may at least partially be located in different housings, which different housings may include housing 728.

[0258] FIG. 8 shows a schematic representation of non-volatile memory media 800a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 800b (e.g. universal serial bus (USB) memory stick) and 800c (e.g. cloud storage for downloading instructions and / or parameters 802 or receiving emailed instructions and / or parameters 802) storing instructions and / or parameters 802 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein. Instructions and / or parameters 802 may represent or correspond to a non-transitory computer readable medium.

[0259] FIG. 9 is an example method 900 based on the examples described herein. At 910, the method includes receiving a source picture comprising a sample array with sample values within a source sample value range. At 920, the method includes using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range. At 930, the method includes modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range. At 940, the method includes encoding the converted source picture to a coded picture in or along a bitstream. Method 900 may be performed with input analyzer 201 or apparatus 700.

[0260] FIG.10 is an example method 1000 based on the examples described herein. At 1010, the method includes training a neural network during an offline or development phase using one or more training iterations until a stopping criterion is satisfied, wherein at least one of the one or more training iterations performed during the offline or development phase comprises a forward pass anda backward pass. At 1020, the method includes wherein the forward pass comprises: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and generating a loss, wherein the loss is based on a weighted sum of a bitrate estimate and a distortion, wherein the bitrate estimate and the distortion are computed based at least on the output of the neural network inference or data derived thereof. At 1030, the method includes wherein the backward pass comprises: computing one or more gradients of the loss with respect to one or more gradients for respective one or more parameters of the neural network, wherein the one or more gradients may be computed for example by a back-propagation algorithm; and updating at least one value of the one or more parameters of the neural network, based at least on the respective one or more gradients. Method 1000 may be performed with trainer 300, trainer with back-scaling prediction 400, or apparatus 700.

[0261] FIG.11 is an example method 1100 based on the examples described herein. At 1110, the method includes selecting a first sample value range. At 1120, the method includes optimizing a video by reducing a source sample value range to the first sample value range. At 1130, the method includes signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

[0262] The method may further include signaling, in or along the bitstream, a third information for indicating a second sample value range to which the video is intended to be back- scaled in a decoding system.

[0263] In an embodiment, the second sample value range is selected such that the second sample value range is different from the source sample value range and the first sample value range.

[0264] The method may be performed with an encoding apparatus, such as the encoder 500, apparatuses depicted in FIG.3 and FIG.4, or the apparatus 700.

[0265] FIG.12 is an example method 1200 based on the example described herein. At 1210, the method includes decoding a first information, from or along a bitstream, that a video has been optimized by reducing a source sample value range to a first sample value range. At 1220, the method includes decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range. At 1230, the method includes decoding the video based on the first information and the first sample value range.

[0266] The method may further include decoding, from or along the bitstream, a second sample value range comprised in a third information or inferring the second sample value range to which the video is intended to be back-scaled in the method; and back-scaling a decoded sample value within the first sample value range linearly to the second sample value range.

[0267] The method may be performed with an decoding apparatus, such as the decoder 600, apparatuses depicted in FIG.3 and FIG.4, or the apparatus 700.

[0268] The following examples are provided and described herein.

[0269] Example 1. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive a source picture comprising a sample array with sample values within a source sample value range; use the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modify the source picture to a converted source picture by converting the source sample value range to the first sample value range; and encode the converted source picture to a coded picture in or along a bitstream.

[0270] Example 2. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: signal, in or along the bitstream, information indicating that at least the coded picture has been optimized by converting the source sample value range to the first sample value range.

[0271] Example 3. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: signal, in or along the bitstream, information indicating the first sample value range.

[0272] Example 4. The apparatus of example 1, wherein the output of the neural network inference additionally comprises information indicative of a second sample value range to which a decoded picture resulting from decoding of the coded picture is intended to be back-scaled in a decoding system.

[0273] Example 5. The apparatus of example 4, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: signal, in or along the bitstream, information indicating the second sample value range.

[0274] Example 6. The apparatus of example 1, wherein the first sample value range comprises a range for luma sample values.

[0275] Example 7. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: transmit, in or along the bitstream, an indication that a reconstructed picture is to be back-scaled.

[0276] Example 8. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: transmit, in or along the bitstream, an indication of which one or more components of a reconstructed picture is to be back-scaled.

[0277] Example 9. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: transmit, in or along the bitstream, an indication of which one or more regions of a reconstructed picture is to be back-scaled.

[0278] Example 10. The apparatus of example 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to train a neural network to perform the neural network inference using: a forward pass to generate a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion, and a backward pass comprising propagating the loss backwards.

[0279] Example 11. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: train a neural network during an offline or development phase using one or more training iterations until a stopping criterion is satisfied, wherein at least one of the one or more training iterations performed during the offline or development phase comprises a forward pass and a backward pass; wherein the forward pass comprises: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and generating a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion; wherein the backward pass comprises: propagating the loss backwards; and updating at least one value of at least one parameter of the neural network, based at least on the loss.

[0280] Example 12. The apparatus of example 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: compute a bit rate loss using an overfitted bits per pixel estimator that estimates bits per pixel of a signal comprising an encoding of the source picture; wherein the bitrate estimation comprises thebit rate loss.

[0281] Example 13. The apparatus of example 12, wherein the overfitted bits per pixel estimator receives as input as least one quantization parameter to compute the bit rate loss.

[0282] Example 14. The apparatus of example 11, wherein the at least one value of the at least one parameter of the neural network is updated using gradient descent.

[0283] Example 15. The apparatus of example 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: compute a distortion loss using an output of a task neural network or deep neural network; wherein a reconstructed image corresponding to a predicted scaling factor and an input quantization parameter is retrieved from storage, and compression artifacts in the form of residuals are added to a downscaled input; wherein the distortion comprises the distortion loss.

[0284] Example 16. The apparatus of example 11, wherein: a luma downscaling module applies a scaling factor predicted by the neural network by multiplying all luma values by the scaling factor, and a luma backcale module multiplies all the luma values by an inverse of the scaling factor.

[0285] Example 17. The apparatus of example 11, wherein the neural network predicts an up- scaling or back-scaling factor that is any value between 0 and 1.0, wherein a value of 0 for the up- scaling or back-scaling factor indicates no back-scaling and a value of 1.0 for the up-scaling or back- scaling factor indicates complete luma range reversion.

[0286] Example 18. The apparatus of example 11, wherein the loss is generated as the bitrate estimation multiplied with a quantization parameter lambda, added to the distortion, wherein the quantization parameter lambda is multiplied with the bitrate estimation for rate-distortion balance for the generated loss.

[0287] Example 19. The apparatus of example 11, wherein two or more neural networks including the neural network are associated with respective two or more quantization parameters quantization parameter ranges, where each of the two or more neural networks is trained by using data associated or coded with a corresponding quantization parameter or quantization parameter range.

[0288] Example 20. A method including: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting thesource sample value range to the first sample value range; and encoding the converted source picture to a coded picture in or along a bitstream.

[0289] Example 21. The method of example 20, further including: signaling, in or along the bitstream, information indicating that at least the coded picture has been optimized by converting the source sample value range to the first sample value range.

[0290] Example 22. The method of example 20, further including: signaling, in or along the bitstream, information indicating the first sample value range.

[0291] Example 23. The method of example 20, wherein the output of the neural network inference additionally comprises information indicative of a second sample value range to which a decoded picture resulting from decoding of the coded picture is intended to be back-scaled in a decoding system.

[0292] Example 24. The method of example 23, further including: signaling, in or along the bitstream, information indicating the second sample value range.

[0293] Example 25. The method of example 20, wherein the first sample value range comprises a range for luma sample values.

[0294] Example 26. The method of example 20, further including: transmitting, in or along the bitstream, an indication that a reconstructed picture is to be back-scaled.

[0295] Example 27. The method of example 20, further including: transmitting, in or along the bitstream, an indication of which one or more components of a reconstructed picture is to be back-scaled.

[0296] Example 28. The method of example 20, further including: transmitting, in or along the bitstream, an indication of which one or more regions of a reconstructed picture is to be back- scaled.

[0297] Example 29. The method of example 20, further including training a neural network to perform the neural network inference using: a forward pass to generate a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion, and a backward pass comprising propagating the loss backwards.

[0298] Example 30. A method including: training a neural network during an offline or development phase using one or more training iterations until a stopping criterion is satisfied, wherein at least one of the one or more training iterations performed during the offline ordevelopment phase comprises a forward pass and a backward pass; wherein the forward pass comprises: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and generating a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion; wherein the backward pass comprises: propagating the loss backwards; and updating at least one value of at least one parameter of the neural network, based at least on the loss.

[0299] Example 31. The method of example 30, further including: computing a bit rate loss using an overfitted bits per pixel estimator that estimates bits per pixel of a signal comprising an encoding of the source picture; wherein the bitrate estimation comprises the bit rate loss.

[0300] Example 32. The method of example 31, wherein the overfitted bits per pixel estimator receives as input as least one quantization parameter to compute the bit rate loss.

[0301] Example 33. The method of example 30, wherein the at least one value of the at least one parameter of the neural network is updated using gradient descent.

[0302] Example 34. The method of example 30, further including: computing a distortion loss using an output of a task neural network or deep neural network; wherein a reconstructed image corresponding to a predicted scaling factor and an input quantization parameter is retrieved from storage, and compression artifacts in the form of residuals are added to a downscaled input; wherein the distortion comprises the distortion loss.

[0303] Example 35. The method of example 30, wherein: a luma downscaling module applies a scaling factor predicted by the neural network by multiplying all luma values by the scaling factor, and a luma backcale module multiplies all the luma values by an inverse of the scaling factor.

[0304] Example 36. The method of example 30, wherein the neural network predicts an up- scaling or back-scaling factor that is any value between 0 and 1.0, wherein a value of 0 for the up- scaling or back-scaling factor indicates no back-scaling and a value of 1.0 for the up-scaling or back- scaling factor indicates complete luma range reversion.

[0305] Example 37. The method of example 30, wherein the loss is generated as the bitrate estimation multiplied with a quantization parameter lambda, added to the distortion, wherein the quantization parameter lambda is multiplied with the bitrate estimation for rate-distortion balance for the generated loss.

[0306] Example 38. The method of example 30, wherein two or more neural networks including the neural network are associated with respective two or more quantization parameters quantization parameter ranges, where each of the two or more neural networks is trained by using data associated or coded with a corresponding quantization parameter or quantization parameter range.

[0307] Example 39. An apparatus including: means for receiving a source picture comprising a sample array with sample values within a source sample value range; means for using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; means for modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and means for encoding the converted source picture to a coded picture in or along a bitstream.

[0308] Example 40. An apparatus including: means for training a neural network during an offline or development phase using one or more training iterations until a stopping criterion is satisfied, wherein at least one of the one or more training iterations performed during the offline or development phase comprises a forward pass and a backward pass; wherein the forward pass comprises: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and generating a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion; wherein the backward pass comprises: propagating the loss backwards; and updating at least one value of at least one parameter of the neural network, based at least on the loss.

[0309] Example 41. A computer readable medium including instructions stored thereon for performing at least the following: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and encoding the converted source picture to a coded picture in or along a bitstream.

[0310] Example 42. A computer readable medium including instructions stored thereon for performing the methods as described in any of examples 21 to 29.

[0311] Example 43. The computer readable medium of example 41 or example 42, whereinthe computer readable medium comprises a non-transitory computer readable medium.

[0312] Example 44. A computer readable medium including instructions stored thereon for performing at least the following: training a neural network during an offline or development phase using one or more training iterations until a stopping criterion is satisfied, wherein at least one of the one or more training iterations performed during the offline or development phase comprises a forward pass and a backward pass; wherein the forward pass comprises: receiving a source picture comprising a sample array with sample values within a source sample value range; using the source picture as input to a neural network inference, wherein an output of the neural network inference is indicative of a first sample value range; modifying the source picture to a converted source picture by converting the source sample value range to the first sample value range; and generating a loss, wherein the loss is based on a weighted sum of a bitrate estimation and a distortion; wherein the backward pass comprises: propagating the loss backwards; and updating at least one value of at least one parameter of the neural network, based at least on the loss.

[0313] Example 45. A computer readable medium including instructions stored thereon for performing the methods as described in any of examples 31 to 38.

[0314] Example 46. The computer readable medium of example 44 or example 45, wherein the computer readable medium comprises a non-transitory computer readable medium.

[0315] Example 47: An apparatus comprising at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to selecting a first sample value range optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

[0316] Example 48: The apparatus of example 47, wherein the apparatus selects the first sample value range such that a minimum value of the first sample value range is greater than zero.

[0317] Example 49: The apparatus of example 47, wherein the apparatus selects the first sample value range such that a midpoint value of the first sample value range is equal to a midmost sample value of the source sample value range.

[0318] Example 50: The apparatus of any of the examples 47 to 49, wherein the apparatus is further caused to perform: linearly scaling the source sample value range to the first sample value range.

[0319] Example 51: The apparatus of any of the examples 47 to 50, wherein the apparatus is further caused to perform: signaling, in or along the bitstream, a third information for indicating a second sample value range to which the video is intended to be back-scaled in a decoding system.

[0320] Example 52: The apparatus of example 5, wherein apparatus selects the second sample value range such that the second sample value range is different from the source sample value range and the first sample value range.

[0321] Example 53: The apparatus of any of the examples 47 to 52, wherein the first sample value range comprises a luma sample value range, and wherein for optimizing the video, the apparatus is further caused to perform: luma-compacting by multiplying luma samples within the video by a luma-compacting factor.

[0322] Example 54: The apparatus of example 53, wherein the apparatus is caused to perform using a zero-compacting method or a middle-compacting method for performing the luma- compacting wherein when the zero-compacting method is used, the zero-compacting method compacts the luma samples in the video towards a zero; and wherein when the middle-compacting is used, the middle-compacting compacts the luma samples in the video towards the middle of a luma range of the video.

[0323] Example 55: The apparatus of any of the examples 47 to 54, wherein the first information, the second information, and / or the third information is signaled by using a supplemental enhancement information (SEI) message.

[0324] Example 56: The apparatus of example 55, wherein the supplemental enhancement information message comprises an encoder optimization information SEI message or a bit-depth range information SEI message.

[0325] Example 57: The apparatus of example 55 or 56, wherein the apparatus is further caused to perform including one or more syntax elements in the SEI message; and setting a value of a bit mask position of a first syntax element and / or values of the one or more syntax elements for indicating range optimization for the first sample value range.

[0326] Example 58: The apparatus of example 57, wherein the one or more syntax elements further indicate a type of the range optimization that has been applied to the first sample value range.

[0327] Example 59: The apparatus of example 58, wherein the values of the one or more syntax elements includes one or more of following sample value range optimization for optimizing a rate-distortion performance for one or more machine analysis tasks sample value attenuationperformed for display power saving sample value attenuation for converting a higher dynamic range to a lower dynamic range or a higher bit depth to a lower bit depth; or sample value magnification for converting the lower dynamic range to the higher dynamic range or the lower bit depth to the higher bit depth.

[0328] Example 60: The apparatus of example 59, wherein when the one or more syntax elements indicate sample value attenuation performed for the display power saving, the one or more syntax elements further indicate a type of a display for which the optimization has been performed.

[0329] Example 61: The apparatus of any of the example 57 to 60, wherein the apparatus is further caused to perform: defining a second syntax element for indicating a method that has been applied for the range optimization.

[0330] Example 62: The apparatus of example 61, wherein one or more values of the second syntax element indicates an algorithm or a process used for range optimization.

[0331] Example 63: The apparatus of any of the example 61 or 62, wherein the apparatus is further caused to perform: selecting the one or more values for the second syntax among a pre- defined set of values.

[0332] Example 64: The apparatus of example 63, wherein the pre-defined set of values comprise a first value indicating a linear sample value scaling a second value indicating the zero- compacting and a third value indicating the middle-compacting; and wherein the one or more values of the second syntax element with reference to the pre-defined set of values are intended to be used by the decoder to determine the method used for the range optimization.

[0333] Example 65: The apparatus of any of the examples 57 or 64, wherein the one or more syntax elements indicates luma and / or chroma video components for which the first sample value range has been optimized.

[0334] Example 66: The apparatus of example 65 wherein the one or more syntax elements comprise a flag, and wherein the flag equal to a first value indicates that the luma sample value range for the video has been optimized and chroma sample arrays are not present or a chroma sample value range is not optimized; and the flag equal to a second value indicates that the luma sample value range and the chroma sample value range have been optimized.

[0335] Example 67: The apparatus of any of the examples 55 to 66, wherein the third information comprises information controlling back-scaling of the first sample value.

[0336] Example 68: The apparatus of example 67, wherein the one or more syntax elements further indicate one or more of the following that the back-scaling is not to be applied in the decoding system that the back-scaling is not to be applied in the decoding system, when decoded video content is for machine consumption that the back-scaling is to be performed to recover the range indicated by a video usability information; or that the back-scaling is controlled by the one or more syntax elements indicative of the second sample value range of a back-scaled video.

[0337] Example 69: The apparatus of example 68, wherein the one or more syntax elements indicative of the second sample value range of the back-scaled video comprises one or more of the following a minimum value and a maximum value of the second sample value range a sample value of a pre-defined point of the second sample value range; an absolute difference of endpoint sample values of the second sample value range; or a range scaling ratio of the pre-defined point of the second sample value range.

[0338] Example 70: The apparatus of example 69, the sample value of the pre-defined point comprises: a sample value of a mid-point of the second sample value range or a minimum value of the second sample value range.

[0339] Example 71: The apparatus of any of the examples 66 to 70, wherein the one or more syntax elements comprise one or more of following a first minimum value field for indicating a minimum luma sample value or a minimum chroma sample value a first maximum value field for indicating a maximum luma sample value or a maximum chroma sample value present in a decoded sample array a second minimum value field for indicating the luma sample value or the chroma sample value corresponding to the first minimum value field a second maximum value field for indicating the luma sample value or the chroma sample value corresponding to the first maximum value field; or a depth field for indicating a bit-depth of luma samples and chroma samples and when present a length in bits of the first minimum value field, the first maximum value field, the second minimum value field, the second maximum value field.

[0340] Example 72: The apparatus of any of the examples 47 to 71, wherein the apparatus is further caused to perform decreasing a bitrate of compression when an input data or the video is scaled to the first sample value range when the first sample value range is smaller than the source sample value range; or increasing the bitrate of the compression when the input data or the video is scaled to the first value range when the first value range is larger than the source sample value range.

[0341] Example 73: The apparatus of example 72, wherein the apparatus is further caused to perform selecting a range scaling ratio to be applied to a quantization parameter; and modifying the quantization parameter based at least on the range scaling ratio to keep the bitrate similar orsubstantially similar to a bitrate that is achieved by using the source sample value range.

[0342] Example 74: The apparatus of any of the examples 47 to 73, wherein the second information comprises one or more first parameters, wherein the one or more first parameters are intended to be used to determine the first sample value range, and wherein the one or more first parameters indicate a value of the first sample value range out of two or more available values.

[0343] Example 75: The apparatus of any of the examples 51 to 74, wherein the third information comprises one or more second parameters, and wherein the one or more second parameters are indented to be used to determine the second sample value range, and wherein the one or more second parameters indicate a value of the second sample value range out of the two or more available values.

[0344] Example 76: An apparatus comprising at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to decoding a first information, from or along a bitstream, that a video has been optimized by reducing a source sample value range to a first sample value range decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

[0345] Example 77: The apparatus of example 76, wherein the apparatus is further caused to perform decoding, from or along the bitstream, a second sample value range comprised in a third information or inferring the second sample value range to which the video is intended to be back- scaled in the apparatus; and back-scaling a decoded sample value within the first sample value range linearly to the second sample value range.

[0346] Example 78: The apparatus of example 77, wherein the apparatus is further caused to perform clipping the back-scaled decoded sample value when the back-scaled sample value is out of the range of the second sample value range.

[0347] Example 79: The apparatus of any of the examples 76 to 78, wherein a minimum value of the first sample value range is greater than 0; and / or the second sample value range is different from a source sample value range and the first sample value range.

[0348] Example 80: The apparatus of any of the examples 76 to 79, wherein the apparatus is further caused to perform: determining whether the back-scaling is to be performed based on one or more of the following decoding from an SEI message that back-scaling is intended to be applied decoding from the SEI message that back-scaling is intended to be applied for machine consumption and providing the back-scaled video to a machine task; or selecting the video to be back-scaled basedon the machine task being performed in the apparatus.

[0349] Example 81: The apparatus of example 80, wherein when the apparatus determines that back-scaling is to be applied, the apparatus is further caused to perform: determining the second sample value range to be used for back-scaling the video based on one or more of the following decoding from the SEI message information indicative of the second sample value range of the back- scaled video decoding from the SEI message information indicative of a range scaling ratio indicative of the first sample value range and the source sample value range, wherein a back-scaling ratio is determined based on the range scaling ratio; or selecting the back-scaling ratio based on the machine task to be applied or a spatial resolution.

[0350] Example 82: The apparatus of example 77, wherein the apparatus is further caused perform concluding that back-scaling is intended to be performed for the video; and concluding a second sample value range for the back-scaled video.

[0351] Example 83: The apparatus of any of the example 76 to 82, wherein the apparatus is further caused to perform receiving following from or along a bitstream the first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and the second information indicating the first sample value range.

[0352] Example 84: The apparatus of any of the examples 76 to 83, wherein the first sample value range has been selected such that a minimum value of the first sample value range is greater than zero.

[0353] Example 85: The apparatus of any of the examples 76 to 83, wherein the first sample value range has been selected such that a midpoint value of the first sample value range is equal to a midmost sample value of the source sample value range.

[0354] Example 86: The apparatus of any of the examples 76 to 85, wherein the source sample value range has been linearly scaled to the first sample value range.

[0355] Example 87: The apparatus of any of the examples 77 to 86, wherein the apparatus is further caused to perform: receiving, in or along the bitstream, the third information.

[0356] Example 88: The apparatus of any of the examples 77 to 87, wherein the second sample value range is selected such that the second sample value range is different from the source sample value range and the first sample value range.

[0357] Example 89: The apparatus of any of the examples 76 to 88, wherein the first samplevalue range comprises a luma sample value range, and wherein optimizing the video comprises: luma-compacting by multiplying luma samples within the video by a luma-compacting factor.

[0358] Example 90: The apparatus of example 89, wherein luma compacting comprises: using a zero-compacting method or a middle-compacting method for performing the luma- compacting wherein when the zero-compacting method is used, the zero-compacting method compacts the luma samples in the video towards a zero; and wherein when the middle-compacting method is used, the middle-compacting method compacts the luma samples in the video towards the middle of a luma range of the video.

[0359] Example 91: The apparatus of any of the examples 76 to 90, wherein the first information, the second information, and / or the third information is received via a supplemental enhancement information (SEI) message.

[0360] Example 92: The apparatus of example 91, wherein the supplemental enhancement information message comprises an encoder optimization information SEI message or a bit-depth range information SEI message.

[0361] Example 93: The apparatus of example 91 or 92, wherein the apparatus is further caused to perform receiving one or more syntax elements in the SEI message; and wherein a value of a bit mask position of a first syntax element and / or values of the one or more syntax elements is set for indicating range optimization for the first sample value range.

[0362] Example 94: The apparatus of example 93, wherein the one or more syntax elements further indicate a type of the range optimization that has been applied to the first sample value range.

[0363] Example 95: The apparatus of example 94, wherein the values of the one or more syntax elements includes one or more of following sample value range optimization for optimizing a rate-distortion performance for one or more machine analysis tasks sample value attenuation performed for display power saving sample value attenuation for converting a higher dynamic range to a lower dynamic range or a higher bit depth to a lower bit depth; or sample value magnification for converting the lower dynamic range to the higher dynamic range or the lower bit depth to the higher bit depth.

[0364] Example 96: The apparatus of example 95, wherein when the one or more syntax elements indicate sample value attenuation performed for the display power saving, the one or more syntax elements further indicate a type of a display for which the optimization has been performed.

[0365] Example 97: The apparatus of any of the example 93 to 96, wherein the apparatus isfurther caused to perform: receiving a second syntax element indicating a method that has been applied for the range optimization.

[0366] Example 98: The apparatus of example 97, wherein one or more values of the second syntax element indicates an algorithm or a process used for range optimization.

[0367] Example 99: The apparatus of any of the examples 97 or 98, wherein the one or more values for the second syntax are selected among a pre-defined set of values.

[0368] Example 100: The apparatus of example 99, wherein the pre-defined set of values comprise a first value indicating a linear sample value scaling a second value indicating the zero- compacting and a third value indicating the middle-compacting; and wherein the one or more values of the second syntax element with reference to the pre-defined set of values are intended to be used by the decoder to determine the method used for the range optimization.

[0369] Example 101: The apparatus of any of the examples 93 or 100, wherein the one or more syntax elements indicate luma and / or chroma video components for which the first sample value range has been optimized.

[0370] Example 102: The apparatus of example 101 wherein the one or more syntax elements comprise a flag, and wherein the flag equal to a first value indicates that the luma sample value range for the video has been optimized and chroma sample arrays are not present or a chroma sample value range is not optimized; and the flag equal to a second value indicates that the luma sample value range and the chroma sample value range have been optimized.

[0371] Example 103: The apparatus of any of the examples 91 to 102, wherein the third information comprises information controlling back-scaling of the first sample value.

[0372] Example 104: The apparatus of example 103, wherein the one or more syntax elements further indicate one or more of the following that the back-scaling is not to be applied in the decoding system that the back-scaling is not to be applied in the decoding system, when decoded video content is for machine consumption that the back-scaling is to be performed to recover the range indicated by a video usability information; or that the back-scaling is controlled by the one or more syntax elements indicative of the second sample value range of a back-scaled video.

[0373] Example 105: The apparatus of example 104, wherein the one or more syntax elements indicative of the second sample value range of the back-scaled video comprises one or more of the following a minimum value and a maximum value of the second sample value range a sample value of a pre-defined point of the second sample value range; an absolute difference ofendpoint sample values of the second sample value range; or a range scaling ratio of the pre-defined point of the second sample value range.

[0374] Example 106: The apparatus of example 105, the sample value of the pre-defined point comprises: a sample value of a mid-point of the second sample value range or a minimum value of the second sample value range.

[0375] Example 107: The apparatus of any of the examples 102 to 106, wherein the one or more syntax elements comprise one or more of following a first minimum value field for indicating a minimum luma sample value or a minimum chroma sample value a first maximum value field for indicating a maximum luma sample value or a maximum chroma sample value present in a decoded sample array a second minimum value field for indicating the luma sample value or the chroma sample value corresponding to the first minimum value field a second maximum value field for indicating the luma sample value or the chroma sample value corresponding to the first maximum value field; or a depth field for indicating a bit-depth of luma samples and chroma samples and when present a length in bits of the first minimum value field, the first maximum value field, the second minimum value field, the second maximum value field.

[0376] Example 108: The apparatus of any of the examples 76 to 107, wherein a bitrate of compression decreased when an input data or the video is scaled to the first sample value range when the first sample value range is smaller than the source sample value range; or the bitrate of the compression is increased when the input data or the video is scaled to the first value range when the first value range is larger than the source sample value range.

[0377] Example 109: The apparatus of any of the examples 76 to 108, wherein the second information comprises one or more first parameters, wherein the one or more first parameters are intended to be used to determine the first sample value range, and wherein the one or more first parameters indicate a value of the first sample value range out of two or more available values.

[0378] Example 110: The apparatus of any of the examples 87 to 109, wherein the third information comprises one or more second parameters, and wherein the one or more second parameters are indented to be used to determine the second sample value range, and wherein the one or more second parameters indicate a value of the second sample value range out of the two or more available values.

[0379] Example 111: A method comprising selecting a first sample value range optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream a first information indicating that the video has been optimized byreducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

[0380] Example 112: A method comprising: decoding a first information, from or along a bitstream, that a video has been optimized by reducing a source sample value range to a first sample value range decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

[0381] Example 113: An apparatus comprising means for selecting a first sample value range means for optimizing a video by reducing a source sample value range to the first sample value range; and means for signaling following in or along a bitstream a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

[0382] Example 114: An apparatus comprising means for decoding a first information, from or along a bitstream, that a video has been means for optimized by reducing a source sample value range to a first sample value range means for decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

[0383] Example 115: A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform selecting a first sample value range optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range

[0384] Example 116: The computer readable medium of the examples 115, wherein the computer readable medium comprises a non-transitory computer readable medium.

[0385] Example 117: A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform decoding a first information, from or along a bitstream, that a video has been means for optimized by reducing a source sample value range to a first sample value range decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

[0386] Example 118: The computer readable medium of example 117, wherein the computerreadable medium comprises a non-transitory computer readable medium.

[0387] In some embodiments, certain ones of the operations above may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the operations above may be performed in any order and in any combination.

[0388] In the above, some example embodiments have been described with the help of syntax of the bitstream. It needs to be understood, however, that the corresponding structure and / or computer program may reside at the encoder for generating the bitstream and / or at the decoder for decoding the bitstream.

[0389] In the above, where example embodiments have been described with reference to an encoder, it needs to be understood that the resulting bitstream and the decoder have corresponding elements in them. Likewise, where example embodiments have been described with reference to a decoder, it needs to be understood that the encoder has structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0390] In the above, some example embodiments have been described with the help of certain syntax elements the bitstream. It needs to be understood, however, that embodiments may be similarly realized with different sets of syntax elements that partly or fully cover the semantics of one or more syntax elements described in example embodiments.

[0391] In the above, some embodiments may have been described in relation to the term quantization parameter. It is to be understood that embodiments may be similarly realized with reference to any similar term, such as quantizer, or control variable that defines or determines fidelity of the (de)coded video or image signal. In the above, some embodiments assume that the lower the quantization parameter value is, the better fidelity is achieved. It is to be understood that embodiments may be similarly realized with a control variable that has an opposite way of mapping values, i.e., the higher the control variable value is, the better fidelity is achieved. For example, embodiments may be realized with JPEG compression where the higher the quality factor is, the better fidelity is achieved.

[0392] In the above, some embodiments may have been described with reference to SEI message(s), such as an encoder optimization information SEI message. It needs to be understood, however, that embodiments may similarly be realized with any similar structures or data units, such as a metadata open bitstream unit (OBU), as specified in AV1 or AV2, and / or ITU-T T.35 metadata. Specific SEI message syntax structures have been presented in some example embodiments, but itneeds to be understood that embodiments generally apply to any SEI messages or any syntax structures that may have at least partly a similar intent as the intent of those specific SEI messages.

[0393] In the above, some embodiments may have been described with reference to syntax that may be present within a video or image bitstream, such as SEI message(s). It is to be understood that embodiments may be similarly realized with syntax that may be present along a video or image bitstream, such as but not limited to any of the following: -Metadata included in a container file that comprises or references to the video or image bitstream. For example, a sample group of the ISO base media file format may be defined for encoder optimization information or luma range adaptation information, where a sample group description entry comprises syntax elements similar to those described in embodiments. -Metadata included in a communication protocol header, such as a header extension of the IETF Real-Time Transport Protocol (RTP) or a payload header of an RTP packet, wherein the communication protocol is used to convey the video or image bitstream. -Metadata included in a signaling protocol, such as the IETF Session Description Protocol (SDP) or the ISO / IEC 23009-1 Media Presentation Description (MPD), which is used to indicate information of an associated media, which may comprise the video or image bitstream.

[0394] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential / parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0395] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0396] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signalprocessor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0397] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0398] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using e.g. a hyphen, dash (-), or number (or abbreviations having a character may be the same with a character removed), and may be case insensitive): 2D two-dimensional ASIC application specific integrated circuit CLVS coded layer video sequence CPU central processing unit CTU coding tree unit DCT Discrete Cosine Transform eoi, EOI encoder optimization information FPGA field programmable gate array GPU graphics processing unit H.2xx family of video coding standards (e.g. H.263, H.264, H.265, H.266, H.274) HEVC high efficiency video coding HMD head-mounted display idc indicator, indication IEC International Electrotechnical CommissionIETF Internet Engineering Task Force I / F interface I / O input / output ISO International Organization for Standardization ITU-T International Telecommunication Union Telecommunication Standardization Sector JPEG Joint Photographic Experts Group MPD media presentation description NAL network abstraction layer NN neural network N / W network OBU open bitstream unit orig original QP quantization parameter RAM random access memory RFM reference frame memory ROI region of interest ROM read only memory RTP real-time transport protocol SDP session description protocol SEI supplemental enhancement information SON self-organizing / optimizing network src source T.35 ITU-T Recommendation that specifies a mechanism to register metadata structures that are identified by a country code, a terminal provider code, and a terminal provider oriented code u(n) unsigned integer using n bits (e.g. u(1)) ue(v) unsigned integer 0-th order Exp-Golomb-coded syntax element with the left bit first. UI user interface USB universal serial bus VSEI versatile supplemental enhancement information VVC versatile video coding VCM video coding for machines

Claims

CLAIMS What is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: selecting a first sample value range; optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

2. The apparatus of claim 1, wherein the apparatus is further caused to perform: linearly scaling the source sample value range to the first sample value range.

3. The apparatus of any of the claims 1 or 2, wherein the apparatus is further caused to perform: signaling, in or along the bitstream, a third information for indicating a second sample value range to which the video is intended to be back-scaled in a decoding system.

4. The apparatus of any of the previous claims, wherein the first sample value range comprises a luma sample value range, and wherein for optimizing the video, the apparatus is further caused to perform: luma-compacting by multiplying luma samples within the video by a luma- compacting factor.

5. The apparatus of any of the previous claims, wherein the first information, the second information, and / or the third information is signaled by using a supplemental enhancement information (SEI) message.

6. The apparatus of claim 5, wherein the supplemental enhancement information messagecomprises an encoder optimization information SEI message or a bit-depth range information SEI message.

7. The apparatus of claim 5 or 6, wherein the apparatus is further caused to perform: including one or more syntax elements in the SEI message; and setting a value of a bit mask position of a first syntax element and / or values of the one or more syntax elements for indicating range optimization for the first sample value range.

8. The apparatus of claim 7, wherein the one or more syntax elements further indicate a type of the range optimization that has been applied to the first sample value range.

9. The apparatus of claim 8, wherein the values of the one or more syntax elements includes one or more of following: sample value range optimization for optimizing a rate-distortion performance for one or more machine analysis tasks; sample value attenuation performed for display power saving; sample value attenuation for converting a higher dynamic range to a lower dynamic range or a higher bit depth to a lower bit depth; or sample value magnification for converting the lower dynamic range to the higher dynamic range or the lower bit depth to the higher bit depth.

10. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: decoding a first information, from or along a bitstream, that a video has been optimized by reducing a source sample value range to a first sample value range; decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; anddecoding the video based on the first information and the first sample value range.

11. The apparatus of claim 10, wherein the apparatus is further caused to perform: decoding, from or along the bitstream, a second sample value range comprised in a third information or inferring the second sample value range to which the video is intended to be back-scaled in the apparatus; and back-scaling a decoded sample value within the first sample value range linearly to the second sample value range.

12. The apparatus of any of the claims 10 or 11, wherein the apparatus is further caused to perform: determining whether the back-scaling is to be performed based on one or more of the following: decoding from an SEI message that back-scaling is intended to be applied; decoding from the SEI message that back-scaling is intended to be applied for machine consumption and providing the back-scaled video to a machine task; or selecting the video to be back-scaled based on the machine task being performed in the apparatus.

13. The apparatus of claim 12, wherein when the apparatus determines that back-scaling is to be applied, the apparatus is further caused to perform: determining the second sample value range to be used for back-scaling the video based on one or more of the following: decoding from the SEI message information indicative of the second sample value range of the back-scaled video; decoding from the SEI message information indicative of a range scaling ratio indicative of the first sample value range and the source sample value range, wherein a back-scaling ratio is determined based on the range scaling ratio; or selecting the back-scaling ratio based on the machine task to be applied or a spatial resolution.

14. The apparatus of any of the claim 10 to 13, wherein the apparatus is further caused to perform receiving following from or along the bitstream:the first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and the second information indicating the first sample value range.

15. The apparatus of any of the claims 10 to 14, wherein the source sample value range has been linearly scaled to the first sample value range.

16. The apparatus of any of the claims 11 to 15, wherein the apparatus is further caused to perform: receiving, in or along the bitstream, the third information.

17. The apparatus of any of the claims 10 to 16, wherein the first sample value range comprises a luma sample value range, and wherein optimizing the video comprises luma-compacting by multiplying luma samples within the video by a luma-compacting factor.

18. The apparatus of claim 17, wherein the luma-compacting comprises: using a zero-compacting method or a middle-compacting method for performing the luma-compacting; wherein when the zero-compacting method is used, the zero-compacting method compacts the luma samples in the video towards a zero; and wherein when the middle-compacting method is used, the middle-compacting method compacts the luma samples in the video towards the middle of a luma range of the video.

19. The apparatus of any of the claims 10 to 18, wherein the first information, the second information, and / or the third information is received via a supplemental enhancement information (SEI) message.

20. The apparatus of claim 19, wherein the supplemental enhancement information message comprises an encoder optimization information SEI message or a bit-depth range information SEI message.

21. The apparatus of claim 19 or 20, wherein the apparatus is further caused to perform: receiving one or more syntax elements in the SEI message; and wherein a value of a bit mask position of a first syntax element and / or values of the one or more syntax elements is set for indicating range optimization for the first sample value range.

22. The apparatus of claim 21, wherein the one or more syntax elements further indicate a type of the range optimization that has been applied to the first sample value range.

23. The apparatus of claim 22, wherein the values of the one or more syntax elements includes one or more of following: sample value range optimization for optimizing a rate-distortion performance for one or more machine analysis tasks; sample value attenuation performed for display power saving; sample value attenuation for converting a higher dynamic range to a lower dynamic range or a higher bit depth to a lower bit depth; or sample value magnification for converting the lower dynamic range to a higher dynamic range or the lower bit depth to the higher bit depth.

24. A method comprising: selecting a first sample value range; optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

25. A method comprising: decoding a first information, from or along a bitstream, that a video has been optimized by reducing a source sample value range to a first sample value range; decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

26. An apparatus comprising: means for selecting a first sample value range; means for optimizing a video by reducing a source sample value range to the first sample value range; and means for signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

27. An apparatus comprising: means for decoding a first information, from or along a bitstream, that a video has been means for optimized by reducing a source sample value range to a first sample value range; means for decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and means for decoding the video based on the first information and the first sample value range.

28. A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform: selecting a first sample value range; optimizing a video by reducing a source sample value range to the first sample value range; and signaling following in or along a bitstream: a first information indicating that the video has been optimized by reducing the source sample value range to the first sample value range; and a second information indicating the first sample value range.

29. A computer readable medium comprising program instructions that, when executed by anapparatus, cause the apparatus to perform: decoding a first information, from or along a bitstream, that a video has been means for optimized by reducing a source sample value range to a first sample value range; decoding, from or along the bitstream, the first sample value range comprised in a second information or inferring the first sample value range; and decoding the video based on the first information and the first sample value range.

Images