Method for converting an image and associated device

Abstract

A method for converting an input image (Iin) having a first dynamic range (Δ1), into an output image (Iout), having a second dynamic range (Δ2) distinct from the first dynamic range, the input image being represented by an input luminance component (Yin) comprising first input pixel values, and at least one input chrominance component (Cbin, Crin) comprising second input pixel values, the output image being represented by an output luminance component (Yout) comprising first output pixel values, and at least one output chrominance component (Cbout, Crout) comprising second output pixels values, the method comprising the steps of: a) determining at least one statistical value (sin) associated with the input image, based on at least part of the first input pixel values, b) determining each first output pixel value based on a corresponding first input pixel value and said at least one statistical value, c) applying on first input nodes (115, 116, 117) of an artificial neural network (CNN1), the second input pixel values, respectively, and applying on at least one second node of the artificial neural network, the at least one statistical value, the artificial neural network being configured to provide on respective output nodes (111,112), the second output pixel values. A corresponding device (1) for converting an input image into an output image is also described.

Description

[0001] Method for converting an image and associated device

[0002] Technical field of the invention

[0003] The invention relates to the field of image processing.

[0004] More particularly, the invention relates to a method for luminance dynamic range conversion of an image, and an associated device.

[0005] State of the art

[0006] Image processing devices have been proposed for converting an input image having a first dynamic range (for instance a “Standard Dynamic Range” or SDR) into an output image having a second dynamic range (for instance a “High Dynamic Range” or HDR) that is distinct from the first dynamic range. Such a conversion is generally called “tone expansion” or “luminance expansion”. It has also been proposed to perform the conversion the other way round, a conversion generally called “inverse tone mapping” or “luminance mapping”.

[0007] In such an image processing device, a mapping unit is provided for transforming an input luminance value associated with a pixel of the input image into an output luminance value associated with the corresponding pixel in the output image.

[0008] Usually, the mapping unit is configured to determine tone expansion parameters based on an analytical processing, for example calculation of statistics being typical of the input image.

[0009] It is also known to use, in the mapping unit, a neural network which provides, as output, luminance mapped values. Such a solution is described for instance in the article “HDR image reconstruction from a single exposure using deep CNNs”, de G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, in ACM Trans. Graph., vol. 36, n° 6, 2017, and in the article “ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content” de D. Marnerides, J. Hatchett, and K. Debattista, in Computer Graphics Forum, vol. 37, n° 2, 2018.

[0010] However, when applied to high resolution images, especially ultra-high- resolution images (UHD), the volume of necessary features and layers, as well as the volume of necessary training data becomes cumbersome and hinders their real-time performances.

[0011] Summary of the invention i In this context, the invention provides a method for converting an input image having a first dynamic range, into an output image, having a second dynamic range distinct from the first dynamic range, the input image being represented by an input luminance component comprising first input pixel values, and at least one input chrominance component comprising second input pixel values, the output image being represented by an output luminance component comprising first output pixel values, and at least one output chrominance component comprising second output pixels values, the method comprising the steps of: a) determining at least one statistical value associated with the input image, based on at least part of the first input pixel values, b) determining each first output pixel value based on a corresponding first input pixel value and said at least one statistical value, c) applying on first input nodes of an artificial neural network, the second input pixel values, respectively, and applying on at least one second node of the artificial neural network, the at least one statistical value, the artificial neural network being configured to provide on respective output nodes, the second output pixel values.

[0012] Determining the output luminance component on a pixel-by-pixel, which can thus be performed e.g. by conventional means, and does not therefore require the involvement of a neural network. This allows for a reduced number of parameters and features required for the artificial neural network used in step c). The complexity of the method thus remains relatively low, even when the conversion follows a complex processing scheme. This method is therefore compatible with real time applications, even with input images in high or even ultra-high definition.

[0013] In addition, using the at least one statistical value associated with the input image while determining the first input pixel value, allows to take into account the input image as a whole, in an efficient manner. Likewise, while determining the at least output chrominance component, by applying the at least one statistical value to at least one second node, the artificial neural network then takes into account the input image as a whole. This reduces the complexity of the artificial neural network, and benefits real time application. In some embodiments, as described below the method further comprises applying on third input nodes of the artificial neural network, at least part of the first input pixel values.

[0014] For instance, a number of third input nodes is equal to a number of first input pixel values. Said otherwise, all available pixel values for the input luminance component are applied to the artificial neural network.

[0015] However, in another embodiment, the method comprises applying on third input nodes of the artificial neural network a plurality of sub-sampled values, respectively, the sub-sampled values being determined based on the first input pixel values by subsampling the input luminance component. This allows a reduction in parameters and overall complexity of the artificial neural network.

[0016] Likewise, in an embodiment, a number of first input nodes can be equal to a number of second input pixel values. Therefore, all available second input pixel values are fed into the artificial neural network, and are processed simultaneously. For instance, if the number of second input pixel values is smaller than a number of first input pixel values, due to an encoding scheme, all the available second input pixel values would still be processed at once by the artificial Neural Network.

[0017] Furthermore, in a possible embodiment of the method, in order to accomplish step b), for at least one first output pixel value, determining the first output value includes applying a function of at least two variables to the corresponding first input pixel value and the at least one statistical value. Hence, the pixel value would be determined through traditional real-time computation, using the input value to calculate an output value.

[0018] But, in order to reduce computation time, for at least one first output pixel value, determining the first output value includes reading the first output pixel value associated with the corresponding first input pixel value and said at least one statistical value in a look-up table. Content of the look-up table would have been calculated beforehand, and the content (i.e. the first output pixel values) is accessed through a direct addressing process.

[0019] For instance, such a look-up table stores possible first output pixel values, each possible first output pixel value corresponding to a given possible first input pixel value and a given possible value for the at least one statistical value. In the case where one statistical value is retrieved in step a), i.e. the at least one statistical value corresponds to one statistical value, then, the look-up-table is a two- dimensional look-up-table.

[0020] In a possible embodiment, the at least one statistical value is a central tendency value of the first input pixel values. For instance, this central tendency value can be the mean or median of the first input pixel values.

[0021] Furthermore, the artificial neural network can be a convolutional Neural Network. This specific type of artificial neural can advantageously be implement on certain hardware component, which would increase the overall efficiency of the method. For instance, computational time and / or energy consumption can be enhanced by using a convolutional Neural Network.

[0022] In a specific embodiment, the convolutional Neural Network comprises an input layer, five convolution layers corresponding to hidden layers, and an output layer, each convolution layer being following by an activation function.

[0023] To further reduce computational time and / or energy consumption, neuron weights of the artificial neural network are quantized to 16-bit integer values, instead of being float values.

[0024] The invention also provides an image converting device for converting an input image having a first dynamic range into an output image having a second dynamic range distinct from the first dynamic range, the image converting device comprising:

[0025] • a first processing module, configured to determine at least one statistical value associated with the input image based on first input pixel values associated with an input luminance component of the input image, and to provide, pixel by pixel, first output pixel values associated with an output luminance component of the output image, each single pixel value among the first output pixel values being determined based on a corresponding first input pixel value and the at least one statistical value,

[0026] • a second processing module configured to apply on first input nodes of an artificial neural network, second input pixel values associated with at least one input chrominance component of the input image, respectively, and on at least one second node of the artificial neural network, the at least one statistical value, the artificial neural network being configured to provide on respective output nodes, second output pixel values corresponding to at least one output chrominance component of the output image.

[0027] For instance, the first processing module is a Central Processing Unit, which is also able to run other tasks of the image converting device.

[0028] The second processing module can be a processing module optimized for machine learning. Such hardware components are particularly efficient for running operations and tasks related to the implementation of a machine learning algorithm, in term of speed and energy consumption.

[0029] In particular, the processing module optimized for machine learning is a Neural Processing Unit. These components, generally referred to as NPU are efficient for parallel processing, and are typically more efficient for running convolutional neural network, wherein there is no connected layers.

[0030] Alternatively, the processing module optimized for machine learning can be a Graphical Processing Unit, known as a GPU. Indeed, the Graphical Processing Unit, typically used for graphic rendering, is optimized for parallel processing, which may benefit implementations of machine learning algorithms.

[0031] In a possible embodiment, the image converting device can include a System on Chip. In this case, both the first processing module and the second processing module are comprised in the System on Chip.

[0032] In another possible embodiment, the image converting device can instead include a Field Programmable Gate Array, in this case, the first processing module and the second processing module are comprised in the Field Programmable Gate Array.

[0033] The image converting device may also include a memory, which stores a lookup table containing possible first output pixel values, each possible first output pixel value corresponding to a given possible first input pixel value and a given possible value for the at least one statistical value, the lookup table being directly addressable by the first processing module.

[0034] Detailed description of embodiments of the invention

[0035] Other possible features of the invention will appear upon reading the description below, made with reference to the appended drawings where:

[0036] - Figure 1 represents an example of an image converting device according to the invention. - Figure 2 represents a second possible embodiment for the image converting device from figure 1 .

[0037] - Figure 3 illustrates an example of a look-up table stored in a memory unit included in the image converting device of Figure 1 or Figure 2.

[0038] - Figure 4 shows an example of artificial neural network implemented into the image converting device of Figure 1 or Figure 2.

[0039] - Figure 5 shows the main steps of a method of converting an input image into an output image according to a possible embodiment of the invention.

[0040] Image converting device

[0041] A first possible embodiment of an image converting device 1 according to the invention is represented on Figure 1 .

[0042] This image converting device 1 may be implemented in practice by an electronic device, including a processor and a memory. The memory is storing program code instructions adapted to perform the operation and functions of the modules described below, when the concerned program code instructions are executed by the processor.

[0043] In particular, in the first embodiment, the image converting device 1 may be implemented on a System on a chip, also referred to as SoC.

[0044] Such a System on a chip is an integrated circuit and comprises at least one microprocessor, and a memory, suitable to perform processing tasks.

[0045] The memory stores program code instructions adapted to perform operations and functions described below when the program code instructions are executed by the microprocessor(s) in order to accomplish the processing tasks.

[0046] In the first embodiment, the SoC on which the image converting device 1 is implemented, comprises several processing units or modules. Each of these processing units is optimized for a certain type of task. For instance, the image converting device 1 here comprises at least a processing unit optimized for tasks related to artificial intelligence, like machine learning and / or deep learning. The image converting device 1 also comprise at least a processing unit optimized for other type of tasks, i.e., not necessarily related to artificial intelligence. This at least one processing unit may for instance, corresponds to the microprocessor of the SoC.

[0047] The processing unit optimized for tasks related to machine leaning is for instance a processing unit optimized for handling mathematical computations, like tensor computations, convolutions, etc. Furthermore, such a processing unit optimized for tasks related to machine learning can be configured as a manycore processor, to be able to handle high level of parallelization. For instance, the processing unit optimized for tasks related to machine learning may be optimized for low energy consumption regarding tasks as described above.

[0048] Such a processing unit optimized for task related to machine learning can be a Graphic Processing Unit (usually referred to as a GPU), or a Neural Processing Unit (i.e. an NPU), also known as an Al accelerator. It can otherwise refer to a Field-Programmable Gate Array (FPGA).

[0049] In the first embodiment, the image converting device 1 considered is a SoC with a split architecture comprising both a CPU and an NPU. For instance, the SoC can be a VideoSmart™ VS680 SoC from synaptics®. Hence, part of the processing tasks can be performed on the CPU while another part of the processing tasks can be performed on the NPU. Then, the CPU and the NPU would each be one of the processing units or processing modules included in the image converting device 1 . More specifically, the CPU is comprised in a first processing module (10), while the NPU is comprised in a second processing module (11 ).

[0050] The NPU is a processing unit enhanced for tasks related to deep learning, in order to reduce computation time and / or energy consumption. Here, we consider the example of an NPU optimized for tasks performed using a Neural Network, in particular, tasks performed using a Convolutional neural Network (CNN).

[0051] As it will be apparent from the following description, the image converting device 1 is designed to convert an input image hn having a first dynamic range Ai (for instance a standard dynamic range or SDR) into an output image lout having a second dynamic range A2 (for instance a high dynamic range or HDR) that is distinct from the first dynamic range A1.

[0052] Here, we can consider that the second dynamic range A2 is larger than the first dynamic range A1. Such a process of converting an input image hn having a first dynamic range A1 into an output image lout having a second dynamic range A2 larger than the first dynamic range A1 is generally referred to as “tone expansion", “luminance mapping", or “tone mapping".

[0053] As an alternative, the image converting device 1 can be used to provide the opposite conversion, thus converting an input image n having a high dynamic range into an output image lout with a standard dynamic range. Such a process of conversion is generally referred as “inverse tone mapping".

[0054] The input image is described by an input set of pixels. For instance, this input set of pixels can correspond to pixels arranged according to a two-dimensional grid, or array, which is intended to be displayed. Dimensions or sizes of the two- dimensional array are defined according to the display resolution of the input image lin.

[0055] Here, the input image hn corresponds to a full High-Definition (Full-HD) image, thus, the display resolution is equal to 1080 by 1920 pixels (generally written as 1080x1920). Alternatively, the input image hn can also correspond to an Ultra-High- Definition (UHD) image, in this case, the display resolution would be equal to 3840x2160 pixels.

[0056] In the present invention, the input image n is represented by a plurality of color components from the YCbCr color space. Alternatively, another color space could be used to describe the input image hn, for instance the YUV color space, or the RGB color space. For the latter, suitable color space coordinate conversions are required before implementing the method described below.

[0057] The input image hn is represented by an input luminance componentn, comprising first input pixel values, and at least one input chrominance component; Cbin, Crincomprising second input pixel values.

[0058] In particular, in the first embodiment, the input image hn is represented by two input chrominance components Cbin, Cnn, a first input chrominance component Cbin comprising second input pixel values, and a second input chrominance component Cnncomprising third input pixel values.

[0059] For clarity purpose, the first input pixel values, the second input pixel values, and the third input pixel values are referred to as the input luminance pixel values, the first input chrominance pixel values and the second input chrominance pixel values, respectively.

[0060] These input pixel values are associated to pixels of the input set of pixels. For instance, following a first encoding scheme, each pixel of the input image hn can be represented by an input luminance componentn, a first input chrominance component Cbin and a second input chrominance component Cnn. Then, each pixel is matched to three pixel values, one from the input luminance pixel values, one from the first input chrominance pixel values, and one from the second input chrominance pixel values. This first encoding scheme is for instance, the encoding scheme known as 4:4:4. According to this first encoding scheme, a number of input luminance pixel values, first input chrominance pixel values, and second input chrominance pixel values are all equal to a number of pixels in the input set of pixels. In this first encoding scheme, sizes of the input set of pixels are equal to sizes of the input pixel values, and all corresponds to array of 1080 by 1920 pixels, if the input image hn is in Full-HD for instance.

[0061] Alternatively, other encoding schemes may be implemented, wherein certain color components are encoded to lower resolutions. Usually, the luminance component is kept at its full resolution.

[0062] For instance, the input image hn can be received by the image converting device 1 encoded in a scheme known as 4:2:2. In this encoding scheme, considering the number of input luminance pixel values is equal to a number of pixels in the input set of pixels, for every block of 2 pixels of the input set of pixels, there is only 1 corresponding pixel value for the at least one input chrominance component. Here, there is only 1 corresponding pixels for the first input chrominance pixel values and the second input chrominance pixel values.

[0063] Likewise, the input image n can be received by the image converting device 1 encoded in a scheme known as 4:2:0. In this encoding scheme, considering the number of input luminance pixel values is equal to a number of pixels in the input set of pixels, for every block of 4 pixels of the input set of pixels, there is only 1 corresponding pixel value for the at least one input chrominance component. Here, there is only 1 corresponding pixels for the first input chrominance pixel values and the second input chrominance pixel values. In particular, if the input image hn is a full-HD image of dimensions 1080x1920, after a 4:2:0 encoding scheme has been applied by the reduction module 13, the first input pixel values (i.e. input luminance component Yin) remain as a 1080x1920 array, while the first input chrominance pixel values and the second input chrominance pixel values are transformed each into a 540x960 array.

[0064] If the at least one chrominance input component is received in an encoding scheme different from the first encoding scheme (4:4:4), it is referred to as a reduced input color component Cbin2, Cnn2, as illustrated on Figure 2.

[0065] According to the first embodiment hereby described, pixels of the input set of pixels are indexed by the letter I; and pixel values of the input luminance component Yin and the (two) input chrominance components Cbin, Cnn, associated to a given pixel i of the input image hnare respectively noted Yn(i), Cbin(i), Cnn(i).

[0066] If the RGB color space is used to represent the input image hn, then, three color components Rin, Gin, Bin (namely a red component Rin, a green component Gin and a blue component Bin) would be used instead. Conversions between the RGB and yCbCr color spaces are well-known.

[0067] As detailed below, the image converting device 1 includes two processing modules, a first processing module 10 and a second processing module 11 , in order to determine the output image lout on which tone mapping, or inverse tone mapping has been performed. In particular, the image converting device 1 aims at determining output color components Yout, Cbout, Crout representative of the output image lout, based on the color componentsn, Cbin, Cnnof the input image n.

[0068] Indeed, like the input image hn, the output image lout is described by an output set of pixels. The output set of pixels, usually, an array of pixel, has dimensions depending on the spatial resolution of the output image lout. Here, we consider an output image lout having a same size as the input image hn.

[0069] The output image lout is also represented by a plurality of color components from the yCbCr color space, i.e. an output luminance component Yout and at least one output chrominance component Cbout, Crout. In the present embodiment, pixels of the output set of pixels are associated with an output luminance component Yout, a first output chrominance component Cbout and a second output chrominance component Crout.

[0070] Each color component comprises output pixels values. In particular, the output luminance component Yout comprises first output pixel values, and the at least one chrominance comprises second output pixel values. In the first embodiment specifically, the first output chrominance component Cbout comprises the second output pixel values, while the second output chrominance component Crout comprises third output pixel values.

[0071] For clarity purpose, the first output pixel values, the second output pixel values, and the third output pixel values are referred to as the output luminance pixel values, the first output chrominance pixel values and the second output chrominance pixel values, respectively. The encoding scheme used describes how the output pixel values are associated with pixels from the output set of pixels. Here, we consider that the output image lout follows the same encoding scheme as the input image hn, i.e. the 4:4:4 encoding scheme. Each pixel from the output set of pixels is matched to a distinct pixel value from each of the output luminance pixel values, the first output chrominance pixel values and the second output chrominance pixel values.

[0072] Pixel values of the output luminance component and the (two) output chrominance components associated to a given pixel i of the output image lout hn are respectively noted Yout(i), Crout(i), Cbout(i).

[0073] In the first embodiment, the first processing module 10 corresponds to the CPU, while the second processing module 11 correspond to the NPU. Both these processing units are included in the System on a Chip.

[0074] First processing module 10: Statistical sub-module 101

[0075] The first processing module 10 includes a statistical sub-module 101 , configured to determine at least one statistical value Sin associated with the input image n. More particularly, the statistical value Sin is determined at least based on the input luminance pixel values. Said otherwise, the at least one statistical value Sin is determined according to the input luminance component Yin of the input image hn, for instance, by taking into consideration first input pixel values of all pixels of the input set of pixels. This at least one statistical value acts for instance like a global representation of the input luminance across the input image hn.

[0076] The statistical value Sin may be a measure of central tendency of (all) the input luminance pixel values. For instance, in the first embodiment, the statistical value Sin may be an average value of (all) the first input pixel values (defining the input luminance componentnof the input image hn in the present embodiment).

[0077] The statistical value Sin can also be a median value of (all) the first pixel values (defining the luminance componentnof the input image hn in the present embodiment).

[0078] In other embodiments, the statistical sub-module 101 may determine a plurality of statistical value Sin, for instance values counting the respective numbers of pixels of the input image hn associated respectively with different (predetermined) luminance ranges (thus defining a histogram of luminance pixel values of the input image lin hn). First processing module 10: direct addressing sub-module 102

[0079] The first processing module 10 includes a direct addressing sub-module 102. The direct addressing sub-module 102 is configured to access a precalculated lookup table (LUT) stored in a memory. The memorized look-up table is accessed through a direct addressing process, in order to retrieve a first output pixel value, corresponding to the output luminance component Yout.

[0080] The pre-calculated look-up table (LUT) accessible to the direct addressing sub-module 102 is constructed to store possible first output pixel values (output luminance pixel values), for corresponding first input pixel values (input luminance pixel value) and each possible value for the at least one statistical value Sin. The possible first output pixel value is determined according to an analytical function, when calculating the look-up table (LUT).

[0081] This analytical function establishes a relationship between an input luminance pixel value (i.e. a possible first input pixel value) and an output luminance pixel value (i.e. a possible first output pixel value) for a given pixel. Examples of such analytical functions for dynamic range conversion can be found in European Patent application EP3839876A1 , in PCT application WO2021 / 123284A1 , or in PCT application WO2022 / 136896.

[0082] Here, the analytical function for a given pixel has at least two arguments, the first input pixel value (associated to the input luminance component Yin), and the at least one statistical value Sin returned by the statistical sub-module 101 .

[0083] In the first embodiment, wherein a single statistical value Sin is determined by the statistical sub-module 101 , for instance, the mean or the median, the analytical function has two variables (or two arguments) for a given pixel from the input set of pixels. Such an analytical function can be written as: fy : Yin(i), Sin Yout(i), wherein fy is the analytical function, Yn(i) is a given possible input luminance pixel value, Sin is a possible value for the statistical value Sin, and Yout(i) is the corresponding possible output luminance pixel value.

[0084] Then, the pre-calculated look-up table is a two-dimensional look-up table, constructed by applying the aforementioned analytical function fy to combinations of all possible values for an input luminance pixel value and all possible values for the statistical value Sin. An example for thus 2D look-up table is illustrated on Figure 3. For instance, if the input image is encoded in 10 bits, then, possible pixel values for an input luminance pixel value range from 0 to 1023. Similarly, if the statistical value Sin retrieved by the statistical sub-module 101 corresponds to either the mean or the median of the input luminance component Yin, possible values for the statistical value Sin range from 0 to 1023.

[0085] The pre-calculated and memorized look-up table thus span 1024 times 1024 possibles values for the first output pixel value.

[0086] The output luminance pixel values are determined on a pixel-per-pixel basis. Said otherwise, the output luminance pixel value for each pixel from the input set of pixels is obtained one after the other, by addressing the 2D look-up table.

[0087] To this end, as represented in Figure 1 , the image converting device 1 comprises a sweeping sub-module 103 configured to sequentially ( / .e. successively) apply the (various) first pixel values Yn(i) to the direct addressing sub-module 102. While a given input image iS processed, the sweeping sub-module 103 sequentially provides a first input pixel value Yn(i) to the direct addressing sub-module 102, while the statistical sub-module 101 applies ( / .e. keeps applying) the determined statistical value Sin as a second input to the direct addressing sub-module.

[0088] In another possible embodiment, the direct addressing sub-module 102 can be replaced by a computation sub-module, which would apply the analytical function to an input first pixel value and to a statistical value Sin in order to compute a corresponding output first pixel value. The computation would occur in real time, for each pixel value from the first input pixel values.

[0089] Therefore, according to a first embodiment of the invention, the statistical value Sin and the first output pixel values, corresponding to the output luminance component Yout of the output image lout are determined using classical or regular computation means, through the CPU.

[0090] The second processing module 11

[0091] The second processing module 11 , which comprises an NPU in the first embodiment, is configured to implement an artificial neural network.

[0092] In particular, the NPU is configured to implement a convolutional Neural Network CNN1 in an optimized manner, i.e. reducing computational time and / or energy consumption. The convolutional Neural Network CNN1 is tasked to provide on respective output nodes 111 , 112, of an output layer 111 , the second output pixel values associated with the at least one chrominance component.

[0093] When provided with first input pixel values (i.e. input luminance pixel values), second input pixel values (i.e. first input chrominance pixel values and / or second input chrominance pixel values), and the at least one statistical value Sin, the convolutional Neural Network CNN1 is trained to predict corresponding second output pixel values (i.e. first output chrominance pixel values and / or second output chrominance pixel values).

[0094] In the first embodiment, we take advantage of the split architecture (meaning comprising both a CPU and an NPU), provided by the System on a Chip on which the image converting device 1 is implemented. Indeed, part of the processing required to produce an output image lout with a different dynamic range from an input image n is performed using traditional computational means on the CPU, while another part of the processing, wherein an artificial Neural Network is needed, is performed on the NPU. Thus, the available computational resources are optimized. Indeed, the NPU is usually less solicited than the CPU.

[0095] The artificial neural network, for instance, the convolutional Neural Network CNN1 , aims at predicting second output pixel values, i.e. a list of pixel values corresponding to the at least one output chrominance component Cbout, Crout. Said otherwise, the prediction is performed using pixel values, for instance, the input luminance pixel values and pixel values for the at least one chrominance output component Cbout, Crout. In particular, pixel values of both the first input chrominance component Cbinand the second input chrominance component Cnnare used for the prediction. Only the at least one output chrominance component Cbout, Crout is predicted using the convolutional Neural Network CNN1 .

[0096] This advantageously reduces the complexity, number of required features and layers, and allows for real-time processing of input images hn into output images lout.

[0097] In another possible embodiment, only pixel values corresponding to the at least one chrominance output component Cbout, Crout are applied to the artificial neural network.

[0098] In the first possible embodiment, the convolutional Neural Network CNN1 , is schematized on Figure 1 and further described in Figure 4. Usually, the convolutional Neural Network CNN1 , comprises at least three layers, including an input layer 113, at least one hidden layer 114 and the output layer 111 , in a feed-forward architecture.

[0099] For instance, the convolutional Neural Network CNN1 can comprise 5 hidden layers 114.

[0100] The input layer 113 includes input nodes 115, namely first input nodes 115, and at least one second input node 116. Here, we consider that the first input nodes 115 receive second input pixel values (i.e. corresponding the at least one input chrominance components Cbin, Cnn). For instance, we can consider several second input nodes 116, each receiving the same at least one statistical value sin.

[0101] Additionally, the input layer 113 can also include third input nodes 117 as represented on Figure 1 , which receive first input pixel values (i.e. corresponding to the input luminance component Yin).

[0102] In the present embodiment, the first input nodes 115 receive the second input pixel values corresponding to both the first and second input chrominance pixel values, the second input nodes 116 each receive the statistical value Sin (either the mean or the median), and the third input nodes receive the input luminance pixel values.

[0103] The hidden layers 114, for instance, the 5 of them, each comprises a plurality of neurons. Each neuron is configured to produce an output value based on respective values of nodes or neurons from a previous layer.

[0104] In particular, in a convolutional Neural Network CNN1 , the hidden layers 114 comprise at least one convolutional layer. In presently described example, all five hidden layers 114 are convolutional layers.

[0105] Hence, in the present embodiment, each neuron of the hidden layers 114 computes a convolution between values provided by a given set of neurons and a convolution tensor. The obtained result is referred to as a convolution result. The convolution tensor is also referred to as a kernel or a filter. Examples of convolution tensors are represented in Figure 4.

[0106] The kernel of each convolutional layers is described by parameters, such as weights W and bias B. The weights W and bias B involved in each convolution layer are obtained through a preliminary training phase, which is detailed below.

[0107] Each neuron of the hidden layers 114 then produces an output value, by applying an activation function on the convolution result. The activation function is for instance a Rectified Linear Unit activation function

[0108] (or ReLU activation function).

[0109] The output layer 110 includes neurons, forming output nodes 111 , 112 of the artificial Neural Network. For instance, the convolutional Neural Network CNN1 comprises several output nodes 111 , 112. These output nodes 111 , 112 produce second output pixel values corresponding to the at least one output chrominance component Cbout, Crout.

[0110] In the first embodiment, as two output chrominance components Cbout, Crout, are considered, i.e. a first chrominance output component Cbout and a second chrominance output component Crout, the output layer 111 comprises first and second output nodes 111 , 112. Each of these two output nodes 111 , 112 produces second output pixel values, based on values output from neurons of the hidden layers 114.

[0111] For instance, the first output nodes 111 provide the second output pixel values, related to the first output chrominance component Cbout. Said otherwise, the first output nodes 111 compute the first output chrominance pixel values. The second output nodes 112 then provide the third output pixel values, corresponding to the second output chrominance component Crout. Equivalently, the second output nodes 112 compute the second output chrominance pixel values.

[0112] Each output node 111 , 112 of the output layer 110 produces an output value (i.e. a value of the concerned output node), based on the values output from neurons of the hidden layers 114. For instance, here, given that the convolutional Neural Network CNN1 comprises 5 hidden layers 114, all of which are convolution layers, each output node corresponds to a convolution between a kernel (i.e. convolution tensor 118) and nodes from the previous convolution layer.

[0113] The description of the convolutional neural network CNN1 provided here is meant for illustrative purposes, and further characteristics of the convolutional neural network CNN1 are described below.

[0114] Additionally, in order to advantageously decrease computation time on NPU type hardware, parameters (such as weights W and bias B) involved in the hidden layers 114 of the convolutional Neural Network CNN1 can be quantized from floats to 16 bitintegers.

[0115] Assembling module In order to construct the output image lout, once the output color components Yout, Cbout, Crout, are determined through their respective unit, the image converting device 1 also includes an assembling module 12. This assembling module 12 is configured to receive the output pixel values associated with the output color components Yout, Crout, Cbout and to construct the output image lout.

[0116] In practice, the assembling module 12 may simply store received all first output pixel values Yout(i), in the memory of the image converting device 1 following the order used by the sweeping sub-module 103 ( / .e. a raster scan order).

[0117] Then, the assembling module 12 may receive and store the second output pixel values Cbout, Crout, as returned by the second processing module 11. For instance, in the first embodiment, the second output pixel values Cbout, Crout are returned by the second processing module 11 shaped as a tensor.

[0118] The obtained output image lout can then be displayed on a screen of the image converting device 1 , or, as a variation, transmitted to an external electronic device (using a communication circuit of the image converting device 1 ).

[0119] Said differently, the image converting device 1 may be implemented in an electronic device, such as a display device including a screen suitable for displaying the output image lout.

[0120] As a variation however, the image converting device 1 may be an electronic device with no display, but possibly with a communication circuit for transmitting the output color component Yout, Crout, Cbout (i.e. their pixel values) representing the output image lout to an external electronic device (that may include a screen suitable for displaying the output image lout).

[0121] Second embodiment: with reduction module

[0122] In a second embodiment, represented on Figure 2, the image converting device 1 further comprises a reduction module 13.

[0123] This reduction module 13 is configured to reduce a number of parameters and / or features required for the convolutional Neural Network CNN1 , by sub-sampling the input luminance componentn. Thus, the reduction module 13 is connected to an input of the second processing module 11 , in order to apply input sub-sampled pixel values to the input layer 113 of the convolutional Neural Network CNN1. These reduced pixel values correspond to reduced input luminance componentn2.

[0124] For instance, if the input image has been received encoded using a 4:2:0 scheme, the input luminance component Yin can also be subsampled in order to match the sizes of the first input chrominance pixel values and the second input chrominance pixel values. Thus, the sub-sampling of the input luminance pixel values reduces the number of pixel value provided to the second processing module 11 accordingly.

[0125] For instance, in the case where the input image hnfollows a 4:2:0 encoding scheme, the pixel value of every two input luminance pixel values can be feed to the input layer 113 of the convolutional Neural Network CNN1 . Hence, the first input pixel values are here arranged as a 540x940 array.

[0126] Training of the CNN

[0127] In order to perform the required task, which is to perform luminance mapping or inverse luminance mapping (i.e. conversion of dynamic range), the artificial neural network goes through a preliminary training phase.

[0128] In particular, in the first embodiment where the artificial neural network is the convolutional neural network of Figure 4, parameters of the hidden layers 114, for instance, the 5 convolutional layers, must be determined. Here, weights W and Bias B of the convolution tensor 118 are acquired through the preliminary training.

[0129] Preliminary trainings of the convolutional neural network CNN1 are well described in the state of the art. Here, the convolutional networks CNN1 is specifically trained to return the at least one chrominance output pixel values, for instance, the first output chrominance pixel values and the second output chrominance pixel values, after a dynamic range conversion has been applied.

[0130] To do so, the convolutional neural network CNN1 is trained on an input set of training images, wherein the at least one statistical value, the second input pixel values, and possibly, the first input pixel values, are all varied through a range of values in a organized manner. The input set of training images are converted into a set of reference images, by applying an analytical function. This analytical function is meant to perform the dynamic range conversion through classical computation mean (i.e. not relying on artificial intelligence). Example of such functions can be found in European patent application EP3839876A1 or in PCT application WO2021 / 123 284 A1 , or in PCT application WO2022 / 136896.

[0131] In the training phase, output images obtained through the convolution neural network are compared to the set of reference images. The comparison is done through a loss function or cost function, commonly used in such trainings, and parameters and / or features of the convolution neural network CNN1 are readjusted accordingly. For instance, the loss function is here a perceptual (possibly color) difference metric, between the output image and a reference image from the set of reference images.

[0132] According to a possible embodiment, the perceptual difference metric used is the AEITP color difference metric as described in the ITU-R BT.2124 recommendation. In an alternative implementation, a color difference metric such as the CIEDE2000 color difference may be used (see e.g. “The development of the CIE 2000 colour-difference formula: CIEDE2000", by M. R. Luo, G. Cui, and B. Rigg in Color Res. Appl., 2001 ). In yet another implementation, metrics such as the HDR-VDP metric or the HDR-VDP- 2 metric could be used (see in this respect the article “HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions", by R. Mantiuk, K.J. Kim, A.G. Rempel & W. Heidrich in ACM Transactions on Graphics, Volume 30, Issue 4, Article No. 40, pp 1 -14).

[0133] Training thus makes it possible for the artificial neural network CNN1 to perform the same conversion as through classical computation means as taught in cited documents, i.e. European patent application EP3839876A1 or in PCT application WO2021 / 123 284 A1 , or in PCT application WO2022 / 136896.

[0134] Method for converting an input image into an output image

[0135] Figure 5 shows the main steps of a method for converting an input image n into an output image lout. This method can be performed using the image converting device 1 previously described.

[0136] As previously described, the input image hn is represented by input color components, which are associated to pixels from the input set of pixels. Here, the input image hn is represented by color components from a chosen color space, e.g. YCbCr, and can be described using an input luminance component Yin and at least one input chrominance component Cbin, Cnn. For instance, two input chrominance components Cbin, Crincan be used to describe the input image n, a first input chrominance component Cbin, and a second input chrominance component Cnn.

[0137] Pixels from the input set of pixels are matched with pixel values, one from the first input pixel values, and at least one from the second input pixel values.

[0138] For instance, each pixel corresponds to three pixel values, one from the input luminance pixel values, one from the first input chrominance pixel values, and one from the second input chrominance pixel values. The method illustrated in Figure 5 comprises a step S1 of determining at least one statistical value Sin associated with the input image . This statistical value is based on at least part of the input luminance pixel values. For instance, all the input luminance pixel values can be accounted for in this step S1 .

[0139] Furthermore, it is also considered that a single statistical value Sin is determined through this step S1 . For instance, the statistical value Sin can be a measure of central tendency, such as the average of the input luminance pixel values. Alternatively, it can be the median of the input luminance pixel values.

[0140] As already mentioned, in other possible embodiments, several statistical values Sin may be determined by the statistical sub-module 101 in this step S1. These statistical values Sin may define a histogram characteristic of the pixel values in the input luminance pixel values.

[0141] Then, the method goes on with a step S2 of determining for each input luminance pixel value, a corresponding output luminance pixel according to the input luminance pixel value and to the at least one statistical value sin.

[0142] This step is implemented by the direct addressing sub-module 102, and consists for instance, in addressing using the given input luminance pixel value and the statistical value Sin a corresponding value in a two-dimensional look-up table (2D-LUT).

[0143] Thus, the output luminance pixel value is determined on a pixel-to-pixel basis, each input being matched to the corresponding output. The output luminance pixel values are pre-calculated and are stored in the look-up table. Output values are computed through a classical computation method, using the analytical function fyfor instance. Construction of such a look-up table has been described previously.

[0144] Usage of a direct addressing sub-module 102, wherein data are stored in a memory, allows for a time- and / or computation efficient method to retrieve the output luminance pixel values after a dynamic range conversion, without compromising on image quality.

[0145] In order to retrieve the pixel values for the other output color components, for instance, for the two output chrominance components Cbout, Crout, the aforementioned convolutional Neural Network CNN1 is advantageously used. Indeed, the relationship between chrominance components before and after dynamic range conversion is usually described using complex analytical functions. Such complex analytical functions are computationally heavy to implement, requiring numerous resources if classical computation means are used.

[0146] However, in the proposed invention, this difficulty is countered by using an artificial neural network, for instance a convolutional Neural Network CNN1 , suitable to implement complex analytical function, with possibility of real time performances, even on high (or ultra-high) definition input images .

[0147] Therefore, the method comprises a step S3 of applying on first input nodes 115 of an artificial neural network (here the convolutional Neural Network CNN1 ), the second input pixel values, respectively, and applying on at least one second node of the artificial neural network, the at least one statistical value Sin, the artificial neural network being configured to provide on respective output nodes, the second output pixel values.

[0148] Description for such an artificial neural network, in particular, a convolution neural network has been provided before. In particular, this convolutional Neural Network CNN1 has been trained in a dedicated training phase, in order to accomplish the required dynamic range conversion.

[0149] For instance, in the first embodiment of the method, in order to compute output chrominance pixel values for the first and second output chrominance component Cbout, Crout, all input chrominance pixel values (i.e. all second input pixel values) are applied to first input nodes 115 of the convolutional Neural Network CNN1 , a statistical value Sin is applied on second nodes 116, and all input luminance pixel values (i.e. all first input pixel values) are applied to third input nodes 117. These input nodes 115, 116, 117 are comprised in the input layer 113 of the convolutional Neural Network CNN1.

[0150] An equivalent description of how the convolution neural network CNN1 is implemented in the first embodiment of the method, is to consider that the input layer 113 receives a tensor as an input.

[0151] This input tensor has a shape and contains the previously listed elements applied to the input nodes. For instance, according to the first embodiment, the input tensor has a shape: (number of channel)x(number of inputs)x(height of input)x(width of input). Thus, the input tensor has here a shape 1x4x1920x1080. This input tensor 1130 is represented on Figure 4. The input tensor 1130 then passes through the hidden layers 114, which corresponds for instance, to 5 convolutional layers. Each of the hidden layers 114 take an output tensor from a previous layer, convolves it with a kernel, and apply a ReLU activation function to the convolution result in order to compute an output tensor, which is passed to the next layer of the convolutional Neural Network CNN1 .

[0152] Parameters of each hidden layers 114, for instance, bias B and / or weights W associated with each layer are determined through the aforementioned training phase.

[0153] Here, a first convolutional layer has a convolution tensor 118 of shape 22x4x1x1 , a second convolutional layer has a convolution tensor 118 of shape 10x22x1 x1 , a third convolutional layer has a convolution tensor 118 of shape 10x10x1x1 , a fourth convolutional layer has a convolution tensor of shape 6x10x1x1 , and a first convolutional layer has a convolution tensor 118 of shape 2x6x1 x1 .

[0154] A bias B of suitable shape is also added to the convolution result obtained through each convolution.

[0155] Each of these convolutional layers is followed by an ReLU activation function.

[0156] Given this architecture of the convolutional neural network CNN1 , for a pixel i of the input set of pixels, there are four input values, i.e. a statistical value, a luminance input pixel value, a first input chrominance pixel value and a second input chrominance pixel value. Then, through the convolution by, for instance the convolution tensor 118 of shape 22x4x1 x1 , these (here 4) input values are transformed into (here 22) features associated the pixel i. Through the hidden layers 114, these features are convolved into other features, but still associated to the pixel i. Said otherwise, physical dimensions of the input image is kept throughout the hidden layers 114 of the convolutional neural network CNN1. In the output layer, for each pixel i, the last convolution transforms the features from the previous hidden layer 114 into two pixel values (corresponding to two features), one corresponding to the first output chrominance pixel value for pixel i, and one corresponding to the second output chrominance pixel value for pixel i. Thus, in the convolutional Neural Network CNN1 , input values corresponding to the various pixels are processed in parallel, by applying the same series of convolution tensors in parallel to each of the sets of input values relating to a pixel in order to obtain in parallel the (here 2) output values relating to each pixel. Other convolutional Neural Network CNN1 architecture are possible in other embodiment of the invention, whilst taking advantages of the hardware optimization, provided by the NPU for instance.

[0157] The artificial neural network, once properly trained, then provide in this step, on respective output nodes 111 , 112, the second output pixel values, as computed through the hidden layers 114. Here, these second output pixel values correspond to the pixel values associated with the first and second output chrominance component Cbout, Crout of the output image lout, i.e. once the dynamic range of the input image lin has been converted.

[0158] Equivalently, given the artificial neural network described in the first embodiment is a convolutional Neural Network CNN1 , an output provided by the output layer 111 of this convolutional Neural Network CNN1 can be described as an output tensor 1100.

[0159] Here, for instance, given the convolutional Neural Network CNN1 described in the first embodiment and its input tensor 1130, the output tensor 1100 has a shape: (number of channel) x (number of output) x (height of output) x (width of output), i.e. 1x2x1920x1080. Only the at least one output chrominance pixel values are returned by the convolutional Neural Network CNN1 . The two output chrominance components correspond to the two outputs of the output tensor 1100.

[0160] According to the first embodiment of the method for converting an input image lin into an output image lout with a distinct dynamic range, the luminance output pixel values (i.e. first output pixel values) are determined are determined on a pixel-to-pixel basis with a look-up table LUT, whilst the chrominance output pixel values (i.e. second output pixel values) are determined all at once, as the output tensor 1100 provided by the artificial neural network.

[0161] Alternatively, in this step of the proposed method, reduced pixel values can be fed to the artificial network, thanks to the reduction module 13 as described previously. This allows a further improvement of computation time and / or energy required for running the artificial network.

[0162] For instance, third input nodes 117 can receive a sub-sampled set of first input pixel values. In a second embodiment of the method described herein, and applied on the image converting device 1 illustrated in Figure 2, an initial input image is considered to be encoded in 4:2:0, with display resolution 1920x1080 (i.e. full-HD). Then, the input luminance pixel values, the first input chrominance pixel values and the reduced second input chrominance pixel values have dimensions of 1920x1080 pixels, 540x960 pixels and 540x960 pixels for respectively.

[0163] Further reduction can be applied by the reduction module 13, to match the dimensions of the different input color components. A sub-sampled set of pixel values for the input luminance component Yin is provided to the convolutional neural network CNN1. For instance, one in two pixels can be discarded. Thus, a reduced input luminance componentn2 can applied to the input layer 113, the reduced input luminance pixel values having dimensions 540x960 pixels.

[0164] Shape of the input tensor 1130 is modified accordingly, as well as the shape of the convolution tensor 118.

[0165] Overall, the described method allows for a computational efficient and / or energy efficient luminance mapping (i.e. dynamic range conversion), while preserving the resulting image quality, even for high resolution input images hn.

[0166] Numerical references image converting device 1 input image n output image lout first dynamic range Ai second dynamic range A2 input luminance componentnfirst input chrominance component Cbin second input chrominance component Cnnfirst processing module 10 second processing module 11 output luminance component Yout first output chrominance component Cbout a second output chrominance component Crout statistical sub-module 101 direct addressing sub-module 102 sweeping sub-module 103 output nodes 111 , 112 output layer 110 input layer 113 hidden layer 114 first input nodes 115 second input nodes 116 third input nodes 117 convolution tensor 118

[0167] Weights W

[0168] Bias B Assembling module 12 reduction module 13 input color components Yin2, Cbin2, Cnn2

[0169] Step S1

[0170] Step S2 Step S3 input tensor 1130 output tensor 1100

Claims

CLAIMS1 . A method for converting an input image ( ) having a first dynamic range (Ai), into an output image (lout), having a second dynamic range (A2) distinct from the first dynamic range (A1), said input image ( ) being represented by an input luminance component (Yin) comprising first input pixel values, and at least one input chrominance component (Cbin, Cnn) comprising second input pixel values, said output image (lout) being represented by an output luminance component (Yout) comprising first output pixel values, and at least one output chrominance component (Cbout, Crout) comprising second output pixels values, the method comprising the steps of: a) determining at least one statistical value (Sin) associated with the input image ( ), based on at least part of the first input pixel values, b) determining each first output pixel value based on a corresponding first input pixel value and said at least one statistical value (Sin), c) applying on first input nodes (115, 116, 117) of an artificial neural network (CNN1 ), the second input pixel values, respectively, and applying on at least one second node (116) of the artificial neural network (CNN1 ), the at least one statistical value (Sin), the artificial neural network (CNN1 ) being configured to provide on respective output nodes (111 ,112), the second output pixel values.

2. The method according to claim 1 , wherein step c) further comprises applying on third input nodes (117) of the artificial neural network (CNN1 ), at least part of the first input pixel values.

3. The method according to claim 2, wherein a number of third input nodes (117) is equal to a number of first input pixel values.

4. The method according to claim 1 , wherein step c) further comprises applying on third input nodes (117) of the artificial neural network (CNN1 ), a plurality of subsampled values, respectively, said sub-sampled values being determined based on the first input pixel values by subsampling the input luminance component (Yn).

5. The method according to any one of claims 1 to 4, wherein a number of first input nodes (115) is equal to a number of second input pixel values.

6. The method according to any one of claims 1 to 5, wherein for at least one first output pixel value, determining said first output value includes applying a function of at least two variables to the corresponding first input pixel value and the at least one statistical value (Sin).

7. The method according to any one of claims 1 to 6, wherein for at least one first output pixel value, determining said first output value includes reading said first output pixel value associated with the corresponding first input pixel value and said at least one statistical value (Sin) in a look-up table (LUT).

8. The method according to claim 7, wherein the look-up table stores possible first output pixel values, each possible first output pixel value corresponding to a given possible first input pixel value and a given possible value for the at least one statistical value (Sin).

9. The method according to any one of claims 7 to 8, wherein the at least one statistical value (Sin) corresponds to one statistical value (Sin), and wherein said look-up-table is a two-dimensional look-up-table (LUT).

10. The method according to any one of claims 1 to 9, wherein the at least one statistical value (Sin) is a central tendency value of the first input pixel values.

11. The method according to any one of claim 1 to 10, wherein the artificial neural network is a convolutional Neural Network (CNN1 ).

12. The method according to claim 11 , wherein the convolutional Neural Network (CNN1 ) comprises an input layer (113), five convolution layers corresponding to hidden layers (114), and an output layer (111 ), each convolution layer being following by an activation function.

13. The method according to any one of claims 1 to 12, wherein neuron weights of the artificial neural network (CNN1 ) are quantized to 16-bit integer values.

14. An image converting device (1 ) for converting an input image (Ln) having a first dynamic range (Ai) into an output image (lout) having a second dynamic range (A2) distinct from the first dynamic range (A1), the image converting device (1 ) comprising: a) a first processing module (10), configured to determine at least one statistical value (Sin) associated with the input image (Ln) based on first inputpixel values associated with an input luminance component (Yin) of the input image ( ), and to provide, pixel by pixel, first output pixel values associated with an output luminance component (Yout) of the output image (lout), each single pixel value among the first output pixel values being determined based on a corresponding first input pixel value and the at least one statistical value (Sin), b) a second processing module (11 ) configured to apply on first input nodes (115) of an artificial neural network (CNN1 ), second input pixel values associated with at least one input chrominance component (Cbin, Cnn) of the input image ( ), respectively, and on at least one second node (116) of said artificial neural network (CNN1 ), the at least one statistical value (Sin), said artificial neural network (CNN1 ) being configured to provide on respective output nodes, second output pixel values corresponding to at least one output chrominance component (Cbout,Crout) of the output image (lout).

15. The image converting device (1 ) according to claim 14, wherein the first processing module (10) is a Central Processing Unit.

16. The image converting device (1 ) according to any one of claims 14 to 15, wherein the second processing module (11 ) is a processing module optimized for machine learning.

17. The image converting device (1 ) according to claim 16, wherein the processing module optimized for machine learning is a Neural Processing Unit.

18. The image converting device (1 ) according to claim 16, wherein the processing module optimized for machine learning is a Graphical Processing Unit.

19. The image converting device (1 ) according to any one of claims 14 to 17, wherein the first processing module (10) and the second processing module (11 ) are comprised in a System on Chip.

20. The image converting device (1 ) according to any one of claims 14 to 17, wherein the first processing module (10) and the second processing module (11 ) are comprised in a Field Programmable Gate Array.The image converting device (1 ) according to any one of claims 14 to 20, wherein a memory stores a lookup table (LUT) containing possible first output pixel values, each possible first output pixel value corresponding to a given possible first input pixel value and a given possible value for the at least one statistical value (Sin), the lookup table (LUT) being directly addressable by the first processing module (10).

Images