Method and apparatus for image energy reduction based on reversible neural network

Abstract

A method and apparatus allow for deriving a reduced energy image from an original image, which is self-contained and reversible, which means that the original image can be reconstructed without any auxiliary information, but only based on the reduced energy image. The difference between the original image and the energy-perceived image is that the value of the luminance or color component of the energy-perceived image is globally lower than the value of the luminance or color component of the input image. The proposed solution is based on a reversible energy aware network (InvEAN). Forward mapping the InvEAN to the input original image determines an energy aware image, allowing energy consumption at the display device to be reduced while maximizing QoE. Reverse mapping of the InvEAN to the energy-aware image will allow reconstruction of the original image. Both operations depend on the same InvEAN reversible neural network trained on the image data set according to a set of losses.

Description

Technical Field

[0001] At least one of the present embodiments generally relates to reducing energy consumption when displaying an image on a display device, and more particularly to methods and devices based on invertible neural networks that allow generation of a self-contained invertible energy-aware image from an original image and reconstruction of the original image from such a self-contained invertible energy-aware image. Background Art

[0002] Reducing the energy consumption of electronic devices has become a requirement not only for electronic device manufacturers, but also to minimize environmental impact and contribute to the emergence of a sustainable display industry. The increase in display resolution from SD to HD, then to 4K, and in the near future to 8K and beyond, as well as the introduction of high dynamic range imaging, has led to a corresponding increase in the energy demands of display devices. This is inconsistent with the global demand for reducing energy consumption given the vast number of devices equipped with displays (e.g., TVs, mobile phones, tablets, etc.). In fact, displays are the most significant energy consumer for consumer electronic devices, whether battery-powered (e.g., smartphones, tablets, head-mounted displays, automotive displays) or non-battery-powered (e.g., televisions, advertising display panels).

[0003] Different display technologies have been developed in recent years. Although modern displays consume energy in a more controlled and efficient manner than older displays, they are still the most significant energy consumer in the video chain.

[0004] Organic light-emitting diodes (OLEDs) are an example of a display technology that is becoming increasingly popular due to its many advantages over previous technologies, such as thin-film transistor liquid crystal displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays consist of individual LEDs that act as image pixels. Therefore, OLED power consumption is highly correlated with image content, and the power consumption for a given input image can be estimated by considering the values ​​of the displayed image pixels.

[0005] Since the display of images is the most significant source of energy consumption for display devices, a reduction in the overall energy consumption of such devices can be achieved by displaying images that require less energy to display (in other words, displaying images with reduced brightness values ​​or reduced color component values, in other words, images that are dimmed (i.e., darkened) compared to the original image). Such images are hereinafter referred to as "energy-aware images." This transformation into energy-aware images should be achieved while preserving as much visual similarity to the original image as possible to avoid hindering the user experience.

[0006] Different methods can be used to produce energy-aware images. These methods can be divided into four categories. The first category of methods is based on histograms or lookup tables, which reduce energy by, for example, manipulating the histogram. The number of empty bins can be reduced or modified in an energy-efficient manner. The second category relies on brightness-based or color-based transformations of the original image. The resulting color / brightness values ​​are selected so that they consume less energy when displayed and according to another criterion, for example, a just noticeable difference (JND) model based on brightness and / or contrast. The third category of methods requires side information such as saliency maps, object quality metrics, depth information, or gaze tracking to, for example, apply region-based image modifications that reduce energy requirements. The fourth and more recent category of methods leverages the power of deep networks. For this specific task, the network is trained in a completely unsupervised manner since there is no ground truth. The deep network is then used to maximize the quality of the output image under the constraint of a prior target power saving rate R.

[0007] Reducing the energy consumption of display devices is often addressed using dimming maps that indicate how each pixel's brightness should be modified on a pixel-by-pixel basis. Unfortunately, this requires transmitting such dimming maps entirely as auxiliary information along the video chain, for example using metadata. This can be expensive in terms of bandwidth and convenience, can be difficult in certain use cases, and the auxiliary information can also be corrupted. It is often more efficient and practical not to have to transmit any auxiliary information at all. Summary of the Invention

[0008] The embodiments described below have been designed with the foregoing in mind and introduce the concept of a self-contained reversible energy-aware image. The energy-aware image is obtained by mapping the original image into the energy-aware image using a reversible energy-aware network in a forward manner. The proposed solution also allows the exact original image to be reconstructed without significant loss by using the same reversible energy-aware network in a reverse manner on the energy-aware image.

[0009] A first aspect of at least one embodiment relates to a method comprising: obtaining an input image; determining a modified image of the input image by using a reversible neural network in a forward manner to map the input image into the modified image by modifying the values ​​of the luminance or color components of the input image, wherein the modified image is self-contained and reversible; and providing the modified image. In a variation of the first aspect, the modification of the values ​​of the luminance or color components is a global reduction of the sum of the values ​​of the luminance or color components of the entire image. The variation of the first aspect further comprises obtaining the modified image, reconstructing the input image by using the same reversible neural network in a reverse manner to map the modified image into a reconstructed image, and providing the reconstructed image.

[0010] A second aspect of at least one embodiment relates to a device comprising a processor configured to: obtain an input image; determine a modified image of the input image by using a reversible neural network in a forward manner to map the input image into the modified image by modifying the values ​​of the luminance or color components of the input image, wherein the modified image is self-contained and reversible; and provide the modified image. In a variation of the second aspect, the modification of the values ​​of the luminance or color components is a global reduction of the sum of the values ​​of the luminance or color components of the entire image. The variation of the second aspect further includes obtaining the modified image, reconstructing the input image by using the same reversible neural network in a reverse manner to map the modified image into a reconstructed image, and providing the reconstructed image.

[0011] In a variation of the first or second aspect, a reversible neural network is trained according to a loss function that is a linear combination of a set of loss functions, including: a mean absolute error loss function that characterizes the difference in brightness between an input image and a modified image; a mean absolute error loss function that characterizes the difference in brightness between an input image and a reconstructed image; a power loss function that characterizes the difference in power between an input image and a modified image; a structural similarity index metric loss function that characterizes the difference between an input image and a modified image; and a structural similarity index metric loss function that characterizes the difference between an input image and a reconstructed image.

[0012] A third aspect of at least one embodiment relates to a method comprising: obtaining a first image; generating a second image by mapping the first image into the second image using a reversible neural network in an inverse manner; and providing the second image, wherein the reversible neural network has been trained according to a loss function and wherein a sum of values ​​of brightness or color components of the second image is higher than a sum of values ​​of brightness or color components of the first image.

[0013] A fourth aspect of at least one embodiment relates to an apparatus comprising a processor configured to: obtain a first image; generate a second image by mapping the first image into the second image using a reversible neural network in an inverse manner; and provide the second image, wherein the reversible neural network has been trained according to a loss function and wherein a sum of values ​​of brightness or color components of the second image is higher than a sum of values ​​of brightness or color components of the first image.

[0014] A fifth aspect of at least one embodiment relates to a computer program comprising program code instructions executable by a processor, the computer program implementing at least the steps of the method according to the first aspect or one of its variants or according to the third aspect.

[0015] A sixth aspect of at least one embodiment relates to a non-transitory computer-readable medium comprising program code instructions executable by a processor, the computer program product implementing at least the steps of the method according to the first aspect or one of its variants or according to the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram illustrating an example of an environment including a display device in which various aspects and embodiments are implemented is illustrated.

[0017] Figure 2 Illustrated is an example of a distribution network based on a reversible energy-aware network for delivering images in accordance with at least one embodiment.

[0018] Figure 3 An example of an overall system architecture using a reversible energy-aware network is illustrated in accordance with at least one embodiment.

[0019] Figure 4 An example of the internal architecture of a reversible energy-aware network in accordance with at least one embodiment is illustrated.

[0020] Figure 5 An example process for training a reversible energy-aware network model is illustrated in accordance with at least one embodiment.

[0021] Figure 6 Illustrated is an example process for generating an energy-aware image from an original image through forward mapping of a reversible energy-aware network in accordance with at least one embodiment.

[0022] Figure 7 Illustrated is an example process for reconstructing an original image from an energy-aware image through inverse mapping of a reversible energy-aware network in accordance with at least one embodiment.

[0023] Figure 8 Target performance for different versions of a reversible energy-aware network is illustrated in accordance with at least one embodiment.

[0024] Figure 9 Different versions of reversible energy-aware networks according to embodiments are illustrated.

[0025] Figure 10 Target performance of a selected version of a reversible energy-aware network is illustrated as a function of the energy reduction rate of the no-reference quality metric NIQE on a BSDS dataset in accordance with at least one embodiment.

[0026] Figure 11 The impact of iterative applications of a reversible energy-aware network in terms of PSNR and SSIM metrics is illustrated in accordance with at least one embodiment.

[0027] Figure 12 Illustrated is an example of the impact of JPEG compression on reversible energy-aware networks in accordance with at least one embodiment.

[0028] Figure 13 Illustrated are examples of actual energy reduction rates observed when using a reversible energy-aware network in accordance with at least one embodiment. DETAILED DESCRIPTION

[0029] Figure 1 A block diagram of an example of an environment including a display device in which various aspects and embodiments are implemented is illustrated. In the depicted environment, a user interacts with a display device 100 that is connected to a data provider 180 through a communication network 150 .

[0030] The display device 100 includes a processor 101. The processor 101 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor can perform data processing, such as Figure 6 Pixel value reduction process 600.

[0031] The processor 101 may be coupled to an input unit 102 configured to transmit user interactions. Various types of input and modalities may be used for this purpose. A physical keypad or a touch-sensitive surface are typical examples of inputs suitable for this purpose, but voice control may also be used. Furthermore, the input unit may also include a digital camera capable of capturing still pictures or videos in two dimensions, or a more complex sensor capable of determining depth information in addition to pictures or videos and thus capable of capturing a full 3D representation.

[0032] The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Various types of displays may be used for this purpose, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to render sound data, which will be converted into audio waves by an adapted transducer, such as a speaker.

[0033] The processor 101 may be coupled to a communication interface 105 configured to exchange data with an external device. The communication preferably uses a wireless communication standard to provide mobility of the display device, such as cellular (eg, LTE), Wi-Fi, and the like.

[0034] The processor 101 can access information from and store data in the memory 106, which can include various types of memory, including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or any other type of memory storage device. In an embodiment, the processor 101 can access information from and store data in a memory that is not physically located on the device, such as on a server, a home computer, or other device.

[0035] The processor 101 may receive power from the power supply 108 and may be configured to distribute power to and / or control power to other components in the device 100. The power supply may be any suitable device for powering the device. As an example, the power supply may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like.

[0036] Although the figures depict the processor 101 and other elements 102 to 108 as separate components, it will be understood that these elements may be integrated together in an electronic package or chip. It will be understood that the display device 100 may include any sub-combination of the elements described herein while remaining consistent with the embodiments described below. The processor 101 may be further coupled to Figure 1 Other peripheral devices or units not depicted in the figure may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripheral devices may include a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like. For example, the processor 101 may be coupled to a positioning unit that is configured to locate the display device in its environment. The positioning unit may integrate a GPS chipset that provides a longitude and latitude location of the current location of the display device, and may also integrate other motion sensors that provide positioning services, such as an accelerometer and / or an electronic compass.

[0037] In at least one embodiment, the processor 101 of the display device 100 is configured to display an image on the display unit 103 according to the embodiments described further below. In a first variant embodiment, the image 190 is obtained from the data provider 180 via the communication network 150. In a second variant embodiment, the image is obtained from the memory 106, which is stored, for example, after being captured by the input unit 102.

[0038] Typical examples of device 100 are smart phones, tablet computers, laptop computers, external monitors, head-mounted displays, televisions, video projectors, computer screens, vehicles (e.g., for control and / or entertainment systems of cars, airplanes, ships, etc.), advertising display panels, medical monitors, etc. However, any device or combination of devices providing similar functionality can be used as display device 1101 while still complying with the principles of the present disclosure. In at least one embodiment, the device does not include a display unit, but rather prepares data for display so that another device such as a screen can perform display. Examples of such devices are set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0039] The communication network 150 preferably uses communication standards to provide interoperability between content providers and display devices. Such communication standards can be wireless, such as cellular (e.g., LTE) communication, Wi-Fi communication, and the like to ensure the mobility of the display device. Cable, satellite, or terrestrial digital television broadcast communication can also be used for the communication network 150 as well as broadband television communication. Such digital television standards can be based on well-established standards (such as DVB, ATSC, or the like). General network standards, such as Ethernet-based standards, can also be used.

[0040] The design of the proposed solution has been driven by several requirements and constraints. First, the solution should preferably preserve as much of the visual quality of the energy-aware image as possible while reducing its energy consumption on the display. Second, the solution should be reversible, so that the original image can be reconstructed from the energy-aware image. Third, the reconstruction process should not require any auxiliary information, such as metadata associated with the energy-aware image.

[0041] The following embodiment describes a new method for deriving an energy-reduced image from an original image. The energy-reduced image is self-contained and reversible, meaning that the original image can be reconstructed based solely on the energy-reduced image without any auxiliary information. The difference between the original image and the energy-aware image is that the values ​​of the luminance or color components of the energy-aware image are globally lower than those of the input image. In other words, the energy-aware image is, for example, globally dimmed compared to the original image, and as a result of this dimming, it requires less energy for its display on the screen. Other techniques can be used to modify the images while providing the same result. However, despite the modifications, the visual similarity between the images remains very high, ensuring a satisfactory quality of experience for the viewer.

[0042] To this end, a reversible energy-aware network (InvEAN) based on an invertible neural network (INN) framework has been designed. Performing forward mapping of the original image using the INN determines an energy-aware image that allows reducing energy consumption at the display device while maximizing visual similarity.

[0043] The following embodiments also describe a novel method for reconstructing an original image from an energy-reduced version of the original image. In effect, using an invertible energy-aware network to perform the reverse mapping of an energy-aware image (also known as inverse mapping) allows reconstruction of the original image with maximum similarity. Both operations rely on the same InvEAN reversible neural network, but are used in either forward or reverse mapping mode.

[0044] The described embodiments offer several advantages over conventional methods for reducing the brightness of images. The proposed method uses a reversible neural network to create a fully reversible method. For example, it may rely on a deep learning network. The reversible neural network targets a specific energy reduction rate. However, no auxiliary information is required to retrieve the original image. Furthermore, it can be applied iteratively to achieve higher target reduction rates.

[0045] The embodiments are described herein as image-based solutions. However, the same principles can be applied to other types of media, such as video, immersive 3D scenes, 360° images or videos, point clouds, or other image-based media. For example, in the case of video, simple frame-by-frame processing can be envisioned, which is enhanced by further temporal filtering of the frame-by-frame energy-aware images generated by InvEAN.

[0046] Figure 2 An example of a distribution network based on an Invertible Energy Aware Network for delivering images according to at least one embodiment is illustrated. In such a distribution network 200, there are different uses of InvEAN, which involve different types of devices and configurations.

[0047] The encoding device 220 transforms the original image 201 into an energy-aware image 202 by performing a forward mapping operation 221 of the original image 201 using the reversible energy-aware network 290. Such a transformation is performed, for example, by Figure 1 The energy-aware image 202 is then displayed in a conventional manner (e.g., by Figure 1 The image is distributed over a communication network 150) and does not require any auxiliary information or metadata to guide the reconstruction.

[0048] A first display device 230 receives the energy-aware image 202 and directly displays it 203. A second display device 240 receives the same energy-aware image 202, reconstructs the original image by performing inverse mapping 241 of the energy-aware image 202 using a reversible energy-aware network 290, and displays the reconstructed original image 204. As a result, the energy consumption of the second identical device 240 will be higher than that of the first device: first, because the displayed image itself is less energy-aware, and second, because the reconstruction process consumes energy to restore the original image.

[0049] The third display device 250 receives the original image 201 and transforms it into an energy-aware image 205 by performing a forward mapping operation 251 of the original image 201 using the reversible energy-aware network 290. The energy-aware image 205 is generated and can be displayed.

[0050] Fourth display device 260 receives original image 201 and transforms it into enhanced image 206 by performing an inverse mapping operation 261 on original image 201 using reversible energy-aware network 290. Enhanced image 206 is generated and can be displayed. Unlike the energy-aware images generated by devices 220 and 250, enhanced image 206 may require more energy to display. In fact, for such an image, the sum of the values ​​of the luminance or color components of enhanced image 206 is higher than the sum of the values ​​of the luminance or color components of original image 201.

[0051] Figure 3 An example of an overall system architecture using a reversible energy sensing network according to at least one embodiment is shown. The problem to be solved is as follows. (W is the width of the image and H is the height of the image), the goal is to determine the energy-aware image , making I * The estimated power consumption of is less than the original power consumption given a reduction rate called R, while having the highest possible quality. For this purpose, a reversible (energy-aware) neural network 310 is defined, which allows learning such , where Θ is a set of trainable parameters of the network. The method provides a reversible process that allows the generation of an energy-aware image 302 from an original image 301, and the reconstruction of the original original image 303 from the energy-aware image 302. The process of generating the energy-aware image uses a forward mapping 330, while the reconstruction from the energy-aware image uses a reverse mapping 360. Both operations use the same reversible energy-aware network 310.

[0052] The proposed InvEAN network 310 operates on the luminance component and is modulated by the chrominance components of the image, which are extracted using conventional RGB to YUV functions 320, 350. The output is a modified version of the luminance such that the RGB color image is reconstructed due to the luminance ratios as follows: in, C i represents the input pixel at spatial position i ( r i , g i , b i ). Y i and These operations are depicted in the figure by elements 340 and 370.

[0053] Based on the following marked I The original image 301, hereinafter marked as I fwd The corresponding energy-aware image 302 and the image hereinafter labeled I bwd The corresponding reconstructed image 303 is used to train the InvEAN model according to the loss function 380.

[0054] The loss function is the basis for optimizing training according to two main objectives. First, the InvEAN network (hereinafter labeled f Θ ) should allow obtaining energy-aware images, and second, running in reverse (performing reverse mapping) the InvEAN network (hereafter labeled ) should allow retrieval of the original image. This can be expressed as: Learning can be constrained so that images I fwd Keep as close as possible to the input image I , but with power constraints. In other words, the forward loss can be expressed as: For the backward pass, the goal is to retrieve the original image with the maximum similarity. In other words, the backward loss can be expressed as: Both the forward loss and the backward loss use the mean absolute error (MAE), which characterizes the difference in brightness between the original image and the reconstructed image for all pixels in the image. The loss can be expressed as: in Y is the original image and is the reconstructed image, N is the total number of pixels in the image, and i are the spatial coordinates of the pixel.

[0055] For all pixels of an image, the power loss characterizes the difference in power between the original image and the corresponding modified image. This loss can be expressed as: in: Where N is the number of pixels in the image.

[0056] The Structural Similarity Index metric (SSIM) loss characterizes the difference between the input image and the corresponding modified image. This metric is based on three comparison measures (i.e., brightness, contrast, and structure) and relies on the local mean, local variance, and local covariance. The loss is given by one minus the SSIM value and can be expressed as: SSIM is a well-known full-reference quality metric proposed in Wang, Zhou et al., “Image quality assessment: from errorvisibility to structural similarity”, IEEE transactions on image processing 13.4 (2004): 600-612. SSIM is in the range [0, 1], where 1 indicates the maximum value.

[0057] Finally, the total loss minimized during training is expressed as: The loss is described above as being based on luminance, but can also be based on color components.

[0058] The proposed architecture of InvEAN 310 is inspired by the reversible U-shaped network proposed in Zhao et al. Invertible image decolorization.IEEE Transactions on Image Processing, 30:6081–6095, 2021, and is Figure 4 Further described in .

[0059] Figure 4 An example of the internal architecture of a reversible energy-aware network according to at least one embodiment is illustrated. The architecture 400 is based on successive layers of Haar wavelet transform (downsampling 401 , 403 and upsampling 405 , 407 ) and conditional bilateral affine coupling layers 402 , 404 and 406 .

[0060] The same architecture and model are used for both forward and inverse mapping. The difference between forward and inverse mapping lies in the formulation of the bilateral affine coupling layer: layer 440 is used for forward mapping, while layer 441 is used for inverse mapping.

[0061] The binary Haar wavelet transform is used firstly to split the image into low-frequency and high-frequency wavelet subbands (in the left part 401, 403) in a downsampling manner, and secondly to concatenate the subbands together into a single channel (in the right part 405, 407) in an upsampling manner. In the proposed architecture, one level of wavelet transform provides one low-frequency subband and three high-frequency subbands. Consider the input feature , the binary Haar transform provides , where H is the height of the image in pixels, W is the width of the image in pixels, and C is the number of color components of the image (usually 3 color components). In other words, the "lower" version of the Haar transform is used to transform N input channels into 4*N output channels with a resolution four times smaller (the total number of pixels does not change), and the "upper" version of the Haar transform is used to transform 4*N input channels into N channels with a resolution four times larger.

[0062] There are several advantages to using such a transform. First, the Haar transform is simple, efficient, and completely reversible. Second, using the Haar transform is an elegant way to reduce computational complexity and has low memory requirements.

[0063] The affine coupling layer is a key component of InvEAN as it defines a bijective mapping between its input distribution and its output distribution, allowing reconstruction. To promote both structure preservation and energy reduction, the four subband channels from the Haar transform are split into a low-frequency channel in the affine coupling layer, and the three high-frequency channels are combined into a single layer channel.

[0064] In forward affine coupling 440, as described above, the input u Divided into two parts[ u LF , u HF ], and by c The adjusted affine transformation is applied as: where ⊙ represents element-wise multiplication. T j and S j Corresponding to the shift (transformation) and scaling coefficients of an arbitrary network, respectively. These two coefficients are learned independently using five layers of densely connected convolutional blocks. The conditional version of the coupling block is achieved by combining the data c Connect to subnet T j and S j To obtain.

[0065] In reverse affine coupling 441, the reverse path is simply calculated as: in, Represents element-wise division. T j and S j are the same scaling and shifting coefficients as used for the forward mapping.

[0066] The above equations for the affine couplings 440 and 441 use conventional notation, where u is the input and v is the output. These letters have nothing to do with the chrominance elements of the image.

[0067] Thus, the affine coupling used in both forward and reverse mapping is a two-sided affine coupling. Furthermore, the affine coupling is modulated by the color component, making it a modulated two-sided affine coupling. In another embodiment, the affine coupling is not modulated by the color component.

[0068] This adjustment is the adjustment data c To Subnet T j and S j The simple concatenation of the inputs does not prevent the proposed architecture from being reversible. In fact, the key point of the affine coupling layer is that the sub-network T j and S j Never inverted.

[0069] Reconciliation datac For example, directly based on the chrominance components U and V, or based on characteristics extracted from the chrominance components.

[0070] As introduced above, the architecture is based on a series of layers. Different cardinality of these layers has been experimented with, e.g. Figure 8 and Figure 9 As shown in .

[0071] The only difference between the forward and inverse mapping of InvEAN is the use of forward affine coupling 440 or inverse affine coupling 441, and the use of energy-aware images for inverse mapping instead of the original images for forward mapping. The learning parameters of the model are the same.

[0072] Figure 5 An example process for training an InvEAN model according to at least one embodiment is illustrated. Such a process 500 is implemented, for example, by the processor 101 of the device 100. In step 510, the processor obtains a set of images or a set of patches to use as a training dataset. In step 520, the parameters of the lightweight deep learning model are learned by iterating (step 521) on the images or patches of the dataset while minimizing the above-mentioned loss function. This results in a trained model of the InvEAN, which can be further provided to another device or unit in step 530 for generating energy-aware images from the original images, such as Figure 6 or for reconstructing the original image from the energy-aware image, as described in Figure 7 Described in .

[0073] Figure 6 An example process for generating an energy-aware image from an original image by forward mapping of a reversible energy-aware network is illustrated in accordance with at least one embodiment. Such a process 600 is, for example, performed by Figure 1 The processor 101 of the device 100 or the Figure 2 The device 220 or 250 is implemented. In step 610, an original input image is obtained. In step 620, an energy-aware image is generated from the original input image by performing forward mapping of the InvEAN. In step 630, the energy-aware image is provided, for example, for transmission via a communication network ( Figure 2 device 220) for further display or directly displayed on the screen ( Figure 2 The display of energy-aware images allows for reduced energy consumption of the display device compared to displaying the original input image, while maintaining a satisfactory quality of experience.

[0074] Figure 7An example process for reconstructing an original image from an energy-aware image by reverse mapping of a reversible energy-aware network is illustrated in accordance with at least one embodiment. Such a process 700 is, for example, performed by Figure 1 The processor 101 of the device 100 or the Figure 2 The energy-aware image is implemented by the device 240. In step 710, an energy-aware image is obtained. In step 720, an image is reconstructed from the energy-aware image by performing inverse mapping of the InvEAN. In step 730, the energy-aware image is provided, for example, for direct display on a screen. Compared to displaying the energy-aware image, displaying the reconstructed image increases the energy consumption of the display device, but restores the original image with the highest possible fidelity.

[0075] Figure 8 The target performance of different versions of the Reversible Energy-Aware Network according to at least one embodiment is illustrated. The InvEAN architecture and process have been evaluated on the BSDS dataset, which consists of 300 images, 200 training images, 40 validation images, and 60 images for testing. The images of this dataset have a resolution of 481×321 pixels, in either landscape or portrait format. The images have been cropped into patches of 128×128 pixels, which have undergone random data augmentation (i.e., horizontal flip, vertical flip, and 90-degree rotation), resulting in a total of 8,000 patches of images (40 epochs with 200 patches each). The network was trained using the following parameters: ADAM solver, learning rate of 0.001, weight decay of 0.0001, and batch size of 8. The coefficients of the linear combination of the loss functions were empirically set to { λ 1=1, λ 2= ​​1, λ 3= 4, λ 4= 1, λ 5= 1}.

[0076] The proposed InvEAN method is implemented in PyTorch. To evaluate different architectural variants of InvEAN, different numbers of down-sampling and up-sampling reversible Haar wavelet transform and conditional bilateral affine coupling layers are used. For each of these layers, the chrominance data is conditioned c Needs to be rescaled to fit the feature map resolution. Rescaling is performed via bicubic interpolation.

[0077] The scaling and shift coefficients in each coupled layer are learned thanks to five layers of densely connected convolutional blocks. Each convolutional filter has a size of 3×3 with a padding of 1, followed by a leaky ReLU activation layer with a negative slope equal to 0.2. The number of intermediate channels in these convolutional blocks is fixed to 32.

[0078] During training, in order to account for quantization effects, a quantization operation is performed after the forward pass or before doing the backward pass. This simulates a real application where energy-aware images are stored in an 8-bit unsigned integer format.

[0079] exist Figure 8 In

[15] , the following three full-reference conventional quality metrics are used: PSNR, SSIM, and VIF. The table shows the average quality scores of these three metrics calculated on three different datasets: the BSDS dataset consisting of 60 images (top part of the table), the Kodak24 dataset consisting of 24 images (middle part of the table), and the VOC2021 dataset consisting of 400 images (lower part of the table).

[0080] Different variants of InvEAN were evaluated to highlight the impact of the number of downsampling operations, the number of reversible blocks between downsampling operations, and scaling. D[x] I[a,b] represents: D = number of downsampling (upsampling) operations; I = number of reversible blocks per downsampling (upsampling) level. For example, D[2] I[1,1] describes an architecture with two downsampling (upsampling) operations and one reversible block per downsampling (upsampling) level. C indicates the use of conditional bilateral affine coupling layers. The bold line indicates the optimal trade-off between quality and architectural complexity. Different reduction rates, varying between 10%, 20%, 40%, and 60%, were also evaluated.

[0081] The best trade-off between target quality and architectural complexity is the configuration D[2] I[1,1]+C (which is highlighted in bold text), which corresponds to Figure 4 The architecture shown in , where each level of spatial resolution has two down-(up-)sampling operations and a conditional bilateral affine coupling layer. The conditioning of the bilateral affine coupling layer allows for global performance improvements. This is particularly evident for the VIF metric. Overall, the D[2] I[1,1]+C architecture provides the best compromise on average, although better results can be observed for specific cases and reduction rates.

[0082] For the BSDS dataset, the table also presents the performance of the proposed method when performing the inverse mapping (i.e., the reverse pass). The results indicate that the proposed InvEAN method successfully restores the original image with high quality, regardless of the reduction rate. The average PSNR value is 38 dB, and the average SSIM is 0.98. The average VIF score varies slightly, ranging from 0.94 to 0.9, representing the lowest to highest reduction rates, respectively.

[0083] Furthermore, for the three full-reference target quality metrics, the proposed method outperforms the linear scaling based method. Similar trends are observed when considering the R-ACE network proposed by Nugroho and Ruan in “R-ACE network for OLED image power saving” in 2022 IEEE 4th Global Conference on Life Sciences and Technologies (LifeTech), pages 284–285, IEEE, 2022, although not as significant as the previous one, and with the exception of the score obtained at R = 10%.

[0084] The performance of the proposed method is compared with linear scaling and R-ACE networks on the BSDS, Kodak24, and VOC2012 datasets, respectively. The figures show that the proposed method provides significantly better results than linear scaling and R-ACE networks. Furthermore, the proposed method demonstrates good generalization capabilities.

[0085] Figure 9 Different versions of a reversible energy-aware network according to an embodiment are illustrated. In the figure, a rectangle labeled "H" represents a Haar wavelet function, and a rectangle labeled "A" represents a bilateral affine coupling.

[0086] InvEAN 901 is a simple InvEAN with a single Haar wavelet function and a single two-sided affine coupling. It corresponds to Figure 8 The first row uses D[1] I[1].

[0087] InvEAN 902 corresponds to Figure 8 The third row uses D[2] I[1,1].

[0088] InvEAN 903 corresponds to Figure 8 The fifth row uses D[2] I[2,2].

[0089] Figure 10 The target performance of selected versions of a reversible energy-aware network according to at least one embodiment is shown as a function of the energy reduction rate (NIQE) on the BSDS dataset using the no-reference quality metric. With this quality metric, higher is better. When using the D[2] I[1,1]+C architecture, the proposed method provides better results than both linear scaling and R-ACE methods. It is also interesting to note that the average NIQE score remains constant for different reduction rates.

[0090] In summary, the proposed method outperforms simple linear scaling and provides competitive performance compared to the R-ACE method.

[0091] Figure 11 The effect of iterative application of an Invertible Energy-Aware Network (IAN) on PSNR and SSIM metrics, according to at least one embodiment, is illustrated. In practice, iteratively performing the InvEAN forward mapping operation is one solution for producing energy-aware images with lower energy. Instead of using different networks trained for different energy reduction rates, a network trained for 10% energy reduction can be iteratively applied several times to obtain energy-aware images with greater than 10% reduction.

[0092] Formally, the iterative process for the forward pass and the backward pass corresponds to: The figure presents the distortion with respect to the energy reduction curves (PSNR for curve 1101 and SSIM for curve 1102) when considering 1, 2, 5, and 10 iterations of a network trained for a 10% energy reduction rate and running the iterations several times. One iteration corresponds to the baseline method. The figure also shows four control points obtained by the InvEAN network, which was trained for energy reduction rates of 10%, 20%, 40%, and 60%, respectively. The two energy reduction / PSNR curves 1101 (i.e., one for independent inference of energy-aware images and the other for the iterative process) are very similar, indicating that the two methods perform similarly. A similar trend is not observed when considering the average SSIM score in curve 1102. In fact, the expected advantage is observed when inferring energy-aware images with a dedicated network. However, the average SSIM score does not drop significantly, indicating that the iterative method is a good candidate for adjusting the energy reduction target. Thus, a network trained for an energy reduction rate of 10% can be used in an iterative manner with 2 to 5 iterations for target energy reduction rates of, for example, between 10% and approximately 35%.

[0093] Figure 12Illustrated are examples of the impact of JPEG compression on an invertible energy-aware network in accordance with at least one embodiment. The impact is evaluated in two directions: forward on the first row and reverse on the second row. The first row is obtained by computing the PSNR / SSIM metric between the original image and JPEG compressed versions of the energy-aware image with different quality factors. The second row is obtained by computing the PSNR / SSIM metric between the original image and a reconstructed image obtained by applying the inverse InvEAN network to the compressed energy-aware image to recover the input image and determine the PSNR and SSIM scores. For these measurements, the reduction rate was set to 20% and the Kodak24 dataset was used. As expected, the quality of both the forward and reverse images degrades with JPEG compression. It is interesting to note that despite not being completely robust to JPEG compression, the performance of the InvEAN network remains consistent with the amount of compression. More importantly, the quality of the reversed images does not collapse at all.

[0094] Figure 13 An example of actual energy reduction rates observed when using a reversible energy-aware network according to at least one embodiment is illustrated. The energy consumption of a test image and its corresponding version processed by the proposed method was measured on a SONY OLED KD-55AF9 display (55 inches, HD). Figure 13 The actual energy reduction rates observed for different energy reduction targets are presented, along with those for linear scaling (left), the R-ACE network (center), and the proposed method (right). As expected, the actual energy reduction rate increases with the energy reduction target. The three tested methods perform similarly. However, it is noted that the actual energy reduction is much smaller than the target energy consumption. This difference may be due to the energy model we used. The model assumes a linear relationship between image brightness and power consumption. This assumption is valid for RGB OLED screens, but not for other technologies (such as RGBW OLED screens, which is the screen we used for testing). Therefore, it will be necessary to revisit the energy model in order to have a more refined control over the final achieved energy consumption.

[0095] In terms of complexity, the number of trainable parameters is 806,184. For an input image of size 128×128 and without any optimization, the number of FLOPS is equal to 5.297 GFlops. The experiments were conducted using dedicated infrastructure with a carbon efficiency of 0.1 kgCO₂eq / kWh. A total of 50 hours of computation were performed on A100-type PCIe 40 / 80GB hardware (250W TDP). Total emissions were estimated to be 1.25 kgCO₂eq, 0% of which were directly offset.

[0096] Although the different embodiments have been described separately, any combination of the embodiments can be performed together while complying with the principles of this disclosure.The terms "method" and "process" are used interchangeably.

[0097] Reference to "one embodiment" or "an embodiment" or "one implementation" or "implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout the specification are not necessarily all referring to the same embodiment.

[0098] Furthermore, this application or its claims may refer to "determining" various pieces of information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from memory.

[0099] Furthermore, this application or its claims may involve "obtaining" various pieces of information. As with "accessing," obtaining is intended to be a broad term. Obtaining information may include, for example, one or more of: receiving information, accessing information, or retrieving information (e.g., from a memory or optical media storage device). Furthermore, "obtaining" is often involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0100] It is to be understood that, for example, in the context of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of..." is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the context of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed option and the second-listed option (A and B), or only the first-listed option and the third-listed option (A and C), or only the second-listed option and the third-listed option (B and C), or all three options (A, B, and C). This can be extended to as many options as listed, as will be readily apparent to those of ordinary skill in this and related arts.

Claims

1. A method comprising: Get the input image; determining a modified image of the input image by using a reversible neural network in a forward manner to map the input image into the modified image by modifying values ​​of brightness or color components of the input image, wherein the modified image is self-contained and reversible; and Provide modified image. 2 . The method according to claim 1 , wherein the modification of the value of the brightness or color component is a global reduction of the sum of the values ​​of the brightness or color components of the entire image.

3. A method comprising obtaining a first image; generating a second image by mapping the first image into the second image using a reversible neural network in a reverse manner; and providing a second image, The modification of the value of the brightness or color component is a global increase of the sum of the values ​​of the brightness or color components of the entire image.

4. The method according to any one of the preceding claims, wherein the reversible neural network has been trained according to a combined loss function, the combined loss function being a linear combination of a set of loss functions comprising: The mean absolute error loss function that characterizes the difference in brightness between the input image and the modified image; The mean absolute error loss function that characterizes the difference in brightness between the input image and the reconstructed image; a power loss function that characterizes the difference in power between the input image and the modified image; A structural similarity index metric loss function that characterizes the difference between the input image and the modified image; as well as The structural similarity index metric loss function characterizes the difference between the input image and the reconstructed image.

5. The method according to any one of the preceding claims, wherein the architecture of the reversible neural network comprises a series of layers, the series of layers comprising at least: Haar wavelet downsampling layer; Bilateral affine coupling; as well as Haar wavelet upsampling layer.

6. The method of claim 5, wherein the Haar wavelet downsampling layer transforms the input into one low-frequency sub-band and three high-frequency sub-bands.

7. The method according to any one of claims 5 or 6, wherein the bilateral affine coupling combines three high frequency sub-bands together.

8. The method according to any one of claims 5 to 7, wherein the bilateral affine coupling performs operations on the low frequency sub-band and the high frequency combined sub-band based on scaling and shifting operations of a deep learning network.

9. The method according to any one of claims 5 to 8, wherein the two-sided affine coupling is modulated by color components.

10. An apparatus comprising a processor, the processor being configured to: Get the input image; determining a modified image of the input image by using a reversible neural network in a forward manner to map the input image into the modified image by modifying values ​​of brightness or color components of the input image, wherein the modified image is self-contained and reversible; and Provide modified image. 11 . The apparatus of claim 10 , wherein the modification of the value of the luminance or color component is a global reduction of the sum of the values ​​of the luminance or color components of the entire image.

12. An apparatus comprising a processor, the processor being configured to: obtaining a first image; generating a second image by mapping the first image into the second image using a reversible neural network in a reverse manner; and providing a second image, The modification of the value of the brightness or color component is a global increase of the sum of the values ​​of the brightness or color components of the entire image.

13. The apparatus according to any one of claims 10 to 12, wherein the reversible neural network has been trained according to a loss function, the loss function being a linear combination of a set of loss functions comprising: The mean absolute error loss function that characterizes the difference in brightness between the input image and the modified image; The mean absolute error loss function that characterizes the difference in brightness between the input image and the reconstructed image; a power loss function that characterizes the difference in power between the input image and the modified image; A structural similarity index metric loss function that characterizes the difference between the input image and the modified image; as well as The structural similarity index metric loss function characterizes the difference between the input image and the reconstructed image.

14. The apparatus according to any one of claims 10 to 13, wherein the architecture of the reversible neural network comprises a series of layers, the series of layers comprising at least: Haar wavelet downsampling layer; Bilateral affine coupling; as well as Haar wavelet upsampling layer.

15. The apparatus of claim 14, wherein the Haar wavelet downsampling layer transforms the input into one low-frequency sub-band and three high-frequency sub-bands.

16. The apparatus according to any one of claims 14 or 15, wherein the bilateral affine coupling combines three high frequency sub-bands together.

17. The apparatus according to any one of claims 14 to 16, wherein the bilateral affine coupling performs some operations on the low frequency sub-band and the high frequency combined sub-band based on scaling and shifting operations of a deep learning network.

18. The apparatus according to any one of claims 14 to 17, wherein the two-sided affine coupling is modulated by a color component.

19. A computer program comprising program code instructions for implementing the method according to any one of claims 1 to 9 when executed by a processor.

20. A non-transitory computer-readable medium comprising program code instructions for implementing the method according to any one of claims 1 to 9 when executed by a processor.

Images