Method and device for reducing display energy by using spatially alternating complementary colors

Abstract

The method and device enable reducing the energy or power consumption required to render an image by replacing pairs of adjacent pixels in an image with pixels of spatially alternating complementary colors that require less energy to display, in other words, by setting the colors of pairs of adjacent pixels in an image to pairs of spatially alternating complementary colors that require less energy to display. Such a solution exploits the spatial fusion property of the human visual system, which perceives complementary adjacent pixels as a single pixel. The spatially alternating complementary colors are selected to be more parsimonious than a single color in terms of the power consumption required to render the color. This combination doubles the search space dimension for energy reduction from three to six. The technique used to replace pixels with adjacent pixels of spatially alternating complementary colors is either pixel doubling, pixel skipping, or pixel averaging. The association between colors and corresponding spatially alternating complementary colors can be stored in a lookup table. These principles can be used for images or videos containing a series of images.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to European Patent Application Publication No. 22306997.2, filed December 22, 2022, which is incorporated herein by reference in its entirety.

[0002] Technical Field At least one of the present embodiments relates generally to reducing energy consumption in display devices, and more particularly to methods and devices for reducing the energy required to render an image by setting the colors of pairs of adjacent pixels of the image to pairs of spatially alternating complementary colors that require less energy to display. [Background technology]

[0003] background Reducing the energy consumption of electronic devices has become a requirement not only for electronic device manufacturers but also for limiting environmental impact as much as possible and contributing to the emergence of a sustainable display industry. The increase in display resolution from SD to HD, then to 4K, and in the near future to 8K and beyond, as well as the introduction of high dynamic range imaging, has correspondingly increased the energy requirements of display devices. This conflicts with the global need to reduce energy consumption, given that many devices (i.e., televisions, mobile phones, tablets, etc.) contain displays. In fact, displays are the most significant source of energy consumption in consumer electronics devices, whether battery-powered (e.g., smartphones, tablets, head-mounted displays, car display screens) or not (e.g., television sets, advertising display panels).

[0004] Various display technologies have been developed in recent years, and although modern displays consume energy in a more controllable and efficient manner than older displays, they are still the most important energy consumer in the video chain.

[0005] Organic light-emitting diodes (OLEDs) are an example of a display technology that is becoming increasingly popular due to many advantages over previous technologies such as thin-film transistor liquid crystal displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays consist of individual LEDs as image pixels. Therefore, OLED power consumption is highly correlated with the image content, and the power consumption for a given input image can be estimated by considering the values ​​of the image pixels being displayed.

[0006] Although OLED displays consume energy in a more controllable and efficient manner, they still remain the most important energy consumer in the video chain. Various techniques have been developed to reduce the energy required to display an image on a display device. To date, most solutions to the problem of reducing the energy required to display several image pulses have focused on slightly modifying the brightness and / or color of the colors of each frame of the pulse. Therefore, these solutions limit the number of dimensions to search to find a more energy-saving image. That is, for one pixel color, these solutions propose another pixel color, limiting the search space dimensions to three color channels. Summary of the Invention [Means for solving the problem]

[0007] overview The embodiments described below are designed with the above in mind and describe methods and devices for reducing the energy (i.e., power consumption) required to render an image by setting the colors of pairs of adjacent pixels in the image to pairs of spatially alternating complementary colors that require less energy to display. Such a solution exploits the spatial fusion property of the human visual system, which perceives complementary adjacent pixels as a single pixel. The spatially alternating complementary colors are selected to be more parsimonious than a single color in terms of the power consumption required to render the color. This combination doubles the search space dimension for energy reduction from three to six. The techniques used to replace pixels with adjacent pixels of spatially alternating complementary colors are either pixel doubling, pixel skipping, or pixel averaging. The association between colors and corresponding spatially alternating complementary colors can be stored in a lookup table. These principles can be used for images or videos containing a series of images.

[0008] A first aspect of at least one embodiment relates to a method that includes determining pairs of alternating complementary colors based on input colors of pixels, wherein the average color of the pairs of alternating complementary colors is the same as or perceptually similar to the input color, and the sum of the energy consumed by displaying the pairs of pixels having the alternating complementary colors is less than twice the energy consumed by displaying the pixels having the input color. A variation of the first aspect further includes iterating multiple times to generate sets of pairs of alternating complementary colors and selecting, from the set of pairs of alternating complementary colors, a pair of alternating complementary colors that has the lowest energy consumption when displayed. A further variation of the first aspect includes iterating over a set of input colors that includes the colors of all pixels of the input image or a subset of all pixels of the input image, or iterating over all possible color valences of a selected color space or a subset of all possible color valences of a selected color space. A further variation of the first aspect includes storing an association between the set of input colors and the corresponding determined pairs of alternating complementary colors.

[0009] A second aspect of at least one embodiment relates to a method including: obtaining a pair of adjacent pixels of an image; obtaining alternating complementary color pairs based on the colors of the adjacent pixel pairs according to the first aspect; and setting the color of a first pixel of the adjacent pixel pair to the first color of the spatially alternating complementary color pair and setting the color of a second pixel of the adjacent pixel pair to the second color of the spatially alternating complementary color pair. A variation of the first aspect further includes converting a single pixel into an adjacent pixel pair by adding a new pixel to double the width or height of the image, the adjacent pixel pair being a horizontal or vertical pair. A variation of the first aspect further includes skipping one pixel of the adjacent pixel pair, the spatially alternating complementary color pair being based on the color of the non-skipped pixel of the adjacent pixel pair. A variation of the first aspect further includes selecting the spatially alternating complementary color pair based on an average color between the colors of the first pixel and the second pixel of the adjacent pixel pair.

[0010] A third aspect of at least one embodiment relates to a method comprising obtaining pairs of adjacent pixels of an image or video, obtaining pairs of alternating complementary colors according to the first aspect, and setting colors of the pair of adjacent pixels of the image or video to spatially alternating complementary colors if the pixels are spatially contained within a region of interest selected according to a criterion, wherein a first pixel of the pair of adjacent pixels is set to a first color of the pair of alternating complementary colors and a second pixel of the pair of adjacent pixels is set to a second color of the pair of alternating complementary colors, and wherein the selection criterion is a spatiotemporal just noticeable difference map, or a motion field, or a saliency map, or based on eye tracking, or based on attention modeling, or a meta-database.

[0011] A fourth aspect of at least one embodiment relates to a device comprising one or more processors configured to determine pairs of alternating complementary colors based on an input color of a pixel for an input color in an image, wherein the average color of the alternating complementary color pair is the same color as or a perceptually similar color to the input color, and wherein the sum of the energy consumed by displaying the pairs of pixels having the alternating complementary colors is less than twice the energy consumed by displaying the pixels having the input color. A variation of the first aspect further includes selecting a color valence for a first color of the alternating complementary color pair according to a selection criterion, and determining a second color of the spatially alternating complementary color pair based on the selected first color valence and the input color, wherein the selection criterion is a maximum color difference from the input color, or that the color is a saturated color, or that the color is a grayscale color, or that the color has the same luminance as the input color. A variation of the first aspect further includes determining the color pairs by repeating multiple times to generate a set of spatially alternating complementary color pairs, and further including selecting the spatially alternating complementary color pair from the set of spatially alternating complementary color pairs that has the lowest energy consumption, wherein the iterations are performed over a set of input colors that includes the colors of all pixels of the input image or a subset of all pixels of the input image, or a set of input colors that includes all possible color valences of a selected color space or a subset of all possible color valences of a selected color space, and associations between the set of input colors and the corresponding determined spatially alternating complementary color pairs are stored in a lookup table using the input colors as indexes.

[0012] A fifth aspect of at least one embodiment relates to a device comprising one or more processors configured to acquire pairs of adjacent pixels of an image, acquire alternating complementary color pairs based on the colors of the pairs of adjacent pixels described in the first aspect, set the colors of the pairs of adjacent pixels of the image or video to spatially alternating complementary colors, set a first pixel of the pair of adjacent pixels to the first color of the pair of alternating complementary colors, and set the color of a second pixel of the pair of adjacent pixels to the second color of the pair of alternating complementary colors. A variation of the first aspect further includes converting a single pixel into a pair of adjacent pixels by adding a new pixel to double the width or height of the image, the pair of adjacent pixels being a horizontal or vertical pair. A variation of the first aspect further includes skipping one pixel of the pair of adjacent pixels, the alternating complementary color pair being based on the color of the non-skipped pixel of the pair of adjacent pixels. A variation of the first aspect further includes selecting the pairs of alternating complementary colors based on an average color between the colors of the first and second pixels of the pair of adjacent pixels.

[0013] A sixth aspect of at least one embodiment relates to a device comprising a processor configured to acquire pairs of adjacent pixels of an image or video, acquire pairs of spatially alternating complementary colors as described in the first aspect, and set colors of the pair of adjacent pixels of the image or video to the spatially alternating complementary colors if the pixels are spatially contained within a region of interest selected according to a criterion, wherein a first pixel of the pair of adjacent pixels is set to a first color of the spatially alternating complementary color pair and a second pixel of the pair of adjacent pixels is set to a second color of the spatially alternating complementary color pair, and wherein the selection criterion is a spatiotemporal just noticeable difference map, or a motion field, or a saliency map, or is based on eye tracking, or is based on attention modeling, or is a meta-database.

[0014] A seventh aspect of at least one embodiment relates to a computer program comprising program code instructions executable by a processor, the computer program performing at least the steps of the method according to the first aspect.

[0015] An eighth aspect of at least one embodiment relates to a non-transitory computer readable medium comprising program code instructions executable by a processor, the computer program product performing at least the steps of the method of the first aspect. [Brief explanation of the drawings]

[0016] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] FIG. 1 illustrates a block diagram of an example display device in which various aspects and embodiments may be implemented. [Figure 2A] 1 shows the normalized response spectrum (or spectral sensitivity function) of human cones. [Figure 2B] An example of a contrast sensitivity function (CSF) is shown below. [Figure 3] 1 illustrates an example of decomposing color into spatially alternating complementary colors, according to an embodiment. [Figure 4] 1 illustrates an example of a process for reducing energy consumption in pixels of an image using spatially alternating complementary colors, according to an embodiment. [Figure 5] 1 illustrates an example of a process for establishing spatially alternating complementary color candidate pairs according to a first embodiment. [Figure 6] 10 illustrates an example process for establishing candidate pairs of spatially alternating complementary colors in a color space that provides a color transform and an inverse color transform, according to a second embodiment. [Figure 7] 1 illustrates an example of replacing pairs of adjacent pixels with pairs of spatially alternating complementary color pixels based on resolution enhancement, according to one embodiment. [Figure 8A] 1 illustrates an example of replacing adjacent pixels with spatially alternating complementary color pixels based on pixel skipping, according to one embodiment. [Figure 8B] 1 illustrates an example of replacing adjacent pixels with spatially alternating complementary color pixels based on pixel skipping, according to one embodiment. [Figure 9]10 illustrates an example of color replacement by spatially alternating complementary color replacement based on pixel averaging, according to one embodiment. [Figure 10] 1 illustrates an example process for generating a lookup table for spatially alternating complementary colors, according to an embodiment, and an example process for modifying an image using the lookup table, according to an embodiment. [Figure 11] 10A-10C illustrate two examples of spatially alternating complementary color process deployments according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description 1 illustrates a block diagram of an example of a display device in which various aspects and embodiments may be implemented. In the illustrated environment, a user interacts with a display device 100 that is connected to a data provider 180 via a communications network 150.

[0018] Display device 100 includes processor 101. Processor 101 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor may perform data processing such as process 400 of Figure 4, process 500 of Figure 5, or process 600 of Figure 6, and related embodiments thereof, operating in a uniform color space.

[0019] The processor 101 may be coupled to an input unit 102 configured to communicate user interactions. For that purpose, multiple types of input and modalities may be used. A physical keypad or a touch-sensitive surface are typical examples of inputs adapted for this application, although voice control may also be used. In addition, the input unit may also include a digital camera capable of capturing still pictures or videos in two dimensions, or a more complex sensor capable of determining depth information in addition to pictures or videos, and thus capturing a full 3D representation.

[0020] The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. For that purpose, several types of displays can be used, such as an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to convert audio data into audio waves through an adapted transducer, for example a speaker.

[0021] The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses a wireless communication standard that provides mobility for the display device, such as cellular (e.g., LTE) communication, Wi-Fi communication, or the like.

[0022] Processor 101 may access information from or store data in memory 106. Memory 106 may include multiple types of memory, including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or any other type of memory storage device. In embodiments, processor 101 may access information from or store data in memory that is not physically located on the device, such as on a server, a home computer, or another device.

[0023] The processor 101 may receive power from a power source 108 and may be configured to provide power to and / or control other components within the device 100. The power source may be any suitable device for powering the device. By way of example, the power source may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0024] While the figures depict the processor 101 and the other elements 102-108 as separate components, it should be understood that these elements may be integrated together in an electronic package or chip. It should be understood that the display device 100 may include any subcombination of the elements described herein while remaining consistent with the embodiments described below. The processor 101 may further be coupled to other peripheral devices or units not shown in FIG. 1 that may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, peripheral devices may include a universal serial bus (USB) port, a vibration device, a television receiver / transmitter, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, etc. For example, the processor 101 may be coupled to a localization unit configured to locate the display device within its environment. The location unit may integrate not only a GPS chipset that provides longitude and latitude position relative to the current location of the display device, but also other motion sensors such as an accelerometer and / or e-compass that provide location services.

[0025] Typical examples of display device 100 include smartphones, tablets, laptops, external monitors, head-mounted displays, television sets, video projectors, computer screens, water vehicles (e.g., control and / or entertainment systems for automobiles, airplanes, boats, etc.), advertising display panels, medical monitors, etc. However, any device or component of a device providing similar functionality may be used as display device 100 while conforming to the principles of the present disclosure. In at least one embodiment, the device does not include a display unit but prepares data for display so that another device, such as a screen, can perform the display. Examples of such devices include set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0026] The following embodiments describe a method for modifying pairs of color pixels to reduce the energy required to display the modified pixels on a display device while maintaining as much visual similarity and experience quality as possible with the original pair of pixels. The embodiments utilize the principles of visual and spatial fusion and propose to set the colors of pairs of adjacent pixels in an image to a spatially alternating complementary color pair that requires less energy to display. The spatially alternating complementary color pair is selected so that the average color of the color pair is perceptually similar / identical to the input color or the average of the input colors, and the energy of the color pair is less than the energy of the input color. In this specification, the term "color energy" should be understood as the energy required to render a pixel of a color. Different types of spatial substitution are described: pixel doubling, pixel skipping, and pixel averaging.

[0027] Prior art solutions propose selecting a color that consumes less energy, for example, by reducing the overall brightness of a pixel or by lowering the levels of some color components. Using color pairs has the advantage of versatility, where the selection of the paired color is based on two parameters: energy consumption and creating a color that is visually identical to the first color. This method broadens the search space for lower-energy colors, thus increasing the probability of finding a combination that reduces energy more significantly. This method also provides images with a better quality of experience, i.e., images with better similarity compared to traditional solutions.

[0028] FIG. 2A shows the normalized response spectrum (or spectral sensitivity function) of human cones. Electromagnetic radiation is characterized by its wavelength (or frequency) and its intensity. The range of wavelengths that humans can perceive is approximately 380 nm to 780 nm. Wavelengths within this range are known as "visible light." Color perception is based on the varying sensitivities of different cells in the retina (color receptors, i.e., cones and rods) to different wavelengths of light. Human observers have three types of color receptors, called cone cells. This results in trichromatic color vision, and cones are typically classified according to the wavelengths of their spectral sensitivity peaks, i.e., short wavelength (S), mid wavelength (M), and long wavelength (L), as shown in FIG. 2A, or simply according to the primary colors around which these peaks are centered, i.e., blue, green, or red.

[0029] According to the trichromatic theory, the color of a light spectrum perceived by a human observer can be characterized by three single scalar values. From a mathematical perspective, this first stage of human vision can be compared to that of a triple kernel energy calculation process. If si(λ) is the wavelength response of a given light spectrum, and l(λ), m(λ), and s(λ) are the spectral sensitivity functions of the L, M, and S cones, respectively, then Equation 1 defines Li, Mi, and Si, which are the three scalar values ​​that characterize the color of spectrum si(λ) as seen by a human observer.

number

[0030] Although the spectrum of light reaching the eye from a given direction determines the color sensation in that direction, there are still many possible spectral combinations that result in the same color sensation. In colorimetry, the term "metamerism" refers to the creation of the same apparent color from light signals with different spectral power distributions. Color spectra created in this way are called metameric spectra. Based on equation (1), the mathematical definition of metamerism is:

number

number

[0031] The embodiments described herein are designed to benefit from the visual fusion of the human visual system, more particularly from the integration of visual excitation from corresponding retinal images of adjacent pixels into a single visual field.

[0032] The term "spatial resolution" refers to the distance between independent measurements, or the physical dimensions that represent the pixels of an image. It is thus the distance between two adjacent pixels of a displayed image.

[0033] The visual acuity of the human eye limits the spatial resolution that the visual system can process. According to various studies, the human visual system can distinguish a spatial difference of approximately 0.6 arc minutes. Since 1' x π / (60 x 180) = 0.0002909 rad, 0.6 arc minutes = 0.0001745328 rad. Beyond a certain distance, two adjacent pixels cannot be resolved and are perceived as a single pixel. The luminous power from various sub-pixels is summed, which gives the apparent continuity of the image as seen on the screen. This is the concept of spatial fusion that will be used in the embodiments described below.

[0034] For example, the typical viewing distance for a mobile phone screen is 25-30 cm. Therefore, if the distance between two point-sized light sources is less than 0.044-0.052 mm, they will appear as a single light source. For a television screen, if the distance between two point-sized light sources is less than 0.52 mm, they will appear as a single light source.

[0035] Figure 2B shows an example of a contrast sensitivity function (CSF). The CSF is a more comprehensive assessment of visual function than visual acuity because it relates the visibility of a spatial pattern to both its size and contrast; it determines only the smallest resolvable pattern size. The CSF depends on luminance and visual field size.

[0036] These principles are used to determine pairs of colors that, when spatially combined, are perceived by a human observer as distinct (single, stable) colors. Indeed, the technical effect used in the present invention relies on the visual fusion property of the human visual system, i.e., the integration of visual excitations from corresponding retinal images of complementary adjacent pixels into a single visual perception. In other words, when visualizing a display of spatially alternating complementary colors, the human visual system perceives a single corresponding color that visually has the same perceptual properties.

[0037] Therefore, the advanced principle of the present invention can be viewed as adding dimension to an image signal by replacing color with two visually complementary colors, and using this added dimension to minimize energy consumption. This principle is referred to herein as spatially alternating complementary colors (SACC). Two adjacent pixels of spatially alternating complementary colors are perceived by a user as a single pixel.

[0038] The term "visual fusion" relates to fusion between colors, and the term "spatial fusion" relates to fusion between pixels. These terms are used interchangeably herein, as the embodiments relate to fusion of colors of spatially adjacent pixels.

[0039] Different embodiments propose different solutions for setting the colors of adjacent pixel pairs in an original image to spatially alternating complementary color pairs. At least one embodiment is based on pixel doubling (which can also be understood as pixel division). In other words, the number of pixels in the input image is doubled to create the necessary adjacent pixels, either in width only, height only, or both dimensions, and a new "duplicated" pixel is added, forming the additional half of the adjacent pixel pair. A pixel in the original image is replaced with two pixels of the same color (hence the division concept). The adjacent pixel pair (original pixel and duplicated pixel) is then replaced with a spatially alternating complementary color pixel pair; in other words, a spatially alternating complementary color pair is assigned to the adjacent pixel. If the original content sent to the display is at a lower resolution than the display's resolution, some internal upsampling is usually performed within the display itself. In such cases, this upsampling can be replaced with this first embodiment, which is based on pixel doubling, which implicitly uses the upsampled resolution.

[0040] At least one embodiment is based on pixel skipping, in which one pixel out of two is freed by canceling the content originally displayed on it, and the spatially alternating complementary color pairs are determined based only on the color of the first pixel of each adjacent pair of pixels in the original image, without further consideration of the color of the second pixel of the original pair, thereby allowing the colors of the original pair of adjacent pixels to be set by the spatially alternating complementary color pairs, i.e., the first pixel of the original pair is assigned to the first color of the spatially alternating complementary color pair, and the second pixel of the original pair is assigned to the second color of the spatially alternating complementary color pair.

[0041] At least one embodiment is based on pixel averaging, in which spatially alternating complementary color pairs are determined based on an average color calculated from the colors of a first pixel and a second pixel of an adjacent pair of pixels in the original image. The first pixel of the original adjacent pair is assigned the first color of the spatially alternating complementary color pair, and the second pixel of the original adjacent pair is assigned the second color of the spatially alternating complementary color pair. Unlike the second embodiment based on pixel skipping, this embodiment considers the colors of all pixels in the original image or video.

[0042] For all three embodiments, different arrangements of adjacent pixels may be used, for example, based on a striped pattern, a mosaic pattern, or a random pattern, as will be further described in connection with FIG.

[0043] FIG. 3 illustrates an example of decomposing color into spatially alternating complementary colors according to an embodiment. In this diagram, row 300 corresponds to an extraction from the original image and represents a row consisting of pixels 301-306. The three numbers within each block correspond to the color of the corresponding pixel, represented as RGB values ​​expressed using 8-bit depth. For example, the first pixel is defined by the following values ​​for the pixel's color components: 147 is the value for red, 107 is the value for green, and 0 is the value for blue. This results in a brown pixel. The colors of the other pixels are medium gray for the second pixel 302, dark blue for the third pixel 303, dark magenta for the fourth pixel 304, reddish-brown for the fifth pixel 305, and light green for the sixth pixel 306, respectively.

[0044] The figure shows an embodiment based on horizontal pixel doubling. To do this, pixels 301-306 need to be duplicated into row 310, where, for example, pixel 301 is duplicated into pixels 301' and 301". In this embodiment, the adjacent pixels whose colors are swapped are horizontally adjacent, in other words, in an adjacent pixel pair, pixels 301' and 301"; the next pair are 302' and 302", and so on up to pair 306' and 306".

[0045] Row 310 shows a set of pairs of pixels (301A, 301B through 306A, 306B) with spatially alternating complementary colors that are used to replace original pixels 301-306. As with row 300, the values ​​within the blocks represent the pixel's color. Due to spatial fusion in the human visual system, the spatial arrangement of green pixel 301A and red pixel 301B is perceived by a human observer as a brown pixel similar to pixel 301, or more generally, the combination of pixels 301' and 301''. A complete example is described below in conjunction with FIG. 7. In other embodiments, for example, when image resolution cannot be increased, other techniques such as pixel skipping or pixel averaging are used. Examples of such methods are described below in conjunction with FIGS. 8A, 8B, and 9.

[0046] 4 shows an example of a process for reducing energy consumption at pixels of an image using spatially alternating complementary colors, according to an embodiment. Process 400 may be performed, for example, by processor 101 of device 100 of FIG. 1. In at least one embodiment, process 400 is iterated over a set of colors that includes the colors of all pixels of the input image. In another embodiment, process 400 is iterated over a set of colors that includes all possible color values ​​according to a selected color space. In other embodiments, iteration is performed over a subset of pixels or a subset of the color space.

[0047] This process is a general process in which some of the steps contain some variations that are necessary to implement the three embodiments introduced above.

[0048] In step 410, the processor obtains the colors C1 and C2 of the pair of adjacent pixels p1 and p2. In the case of an embodiment based on pixel doubling, before this step, the processor must first create a new adjacent pixel p2 with respect to pixel p1, for example by doubling the width of the image. In this case, pixel p2 is created as a copy of pixel p1, in other words, its color C2 is the same as color C1. In the embodiment based on pixel skipping, the color of pixel p2 is replaced with the color of p1. In the embodiment based on pixel averaging, the color of pixel p1 is replaced with the average color between the colors of p1 and p2.

[0049] In step 420, the processor calculates spatially alternating complementary colors C corresponding to colors C1 and C2. A , C B The spatially alternating complementary color pairs are selected based on two constraints. The first constraint relates to the quality of the experience and is the color C. A and C B The neighboring pixel p A and p B The second constraint is related to reducing the energy required for display and ensures that the combination of pixels p1 and p2 of colors C1 and C2 is perceptually similar to the combination of pixels p1 and p2 of colors C1 and C2. A and C B pixel p A and p B is less than the energy required to display pixels p1 and p2 of colors C1 and C2. Step 420 ensures that the energy required to display pixels p1 and p2 of the first color C1 and C2 is less than the energy required to display pixels p1 and p2 of colors C1 and C2. A and then selecting a suitable second color C according to perceptual similarity and energy reduction constraints. B This results in a definition of a color pair corresponding to the input colors C1 and C2. Step 420 is repeated (415) multiple times to determine a set of candidate pairs of alternating complementary colors {C AB} to determine

[0050] In at least one embodiment, spatially alternating complementary colors CA , C B The pair of IN is determined based on the

[0051] In step 430, the processor selects candidate pairs C as spatially alternating color pairs. A , C B Select one of the colors C IN In at least one embodiment, the processor selects the candidate pair having the lowest energy consumption.

[0052] In step 440, the processor, according to one of the techniques presented herein, divides adjacent pixels p1 and p2 of colors C1 and C2 into pixels of color C1 and C2. A and C B Two adjacent pixels p A and p B In other words, the colors of the adjacent pixels p1 and p2 are replaced by the color C A and C B is set to

[0053] Regarding color similarity, the verification of the first constraint is based on comparing the average of the color pair cA, cB with the color cIN. The average of the color pair is calculated in a display color space, a standard color space, or a color space representing human color vision. Examples of display color spaces are sRGB and AdobeRGB. Examples of standard color spaces (also known as measurement color spaces) are CIEXYZ and CIELUV. Examples of color spaces representing human color vision (also known as uniform color spaces) are CIELab, IPT, OKLab, and OSA-UCS. Thus, the resulting display colors of the processed image in terms of human perception are the desired colors similar to the source image, but energy consumption is reduced thanks to the appropriate selection of the (cA, cB) color pair according to the display color power model.

[0054] Regarding energy consumption, in embodiments using frame doubling, the verification of the second constraint is A and c B Two temporally consecutive pixels pA and p B is based on comparing the energy of each pixel p of color c with that of the pixel p of color c, where A and p B The duration of display of pixel p is half the duration of pixel p. Energy expressed throughout this specification is based on a display color power model. A simple example of such a model is based on the sum of the RGB pixel values ​​raised to the power of gamma, where gamma is between 1.8 and 2.3, e.g., 2.2. For example, pixel 302 in Figure 3, with RGB values ​​of 127, 141, 141, is (127 2.2 +141 2.2 +141 2.2 )=149480. The first pixel 302A in the temporally consecutive pixels with RGB values ​​of 147, 147, 0 is represented by a color power value of 1 / 2×(147 2.2 +147 2.2 +0 2.2 )=29420. The second of the temporally consecutive pixels, 302B, is represented by a color power value of 103042. As shown in Table 1, the energy comparison leads to the conclusion that it would be more efficient to replace pixel 302 (with a color power value of 149480) with the paired combination of temporally consecutive pixels 302A and 302B (with a therefore lower combined color power value of 132462).

[0055] [Table 1]

[0056] The colors presented to a human observer are defined by the display response to an RGB triplet, e.g., sRGB or BT-709, or the response defined in other display color spaces implemented in a display instance or model with given parameter adjustments (brightness, contrast, color temperature, etc.). A color space {C} is chosen in which color arithmetic operations are realized. A display with colors represented in {C} has a color gamut {G} that represents a complete subset of the colors the display can render. In the case of an image or video, the colors C1 and C2 of each pair of pixels p1 and p2 are represented as a color C A and C B is replaced by C A and C B and color C so that the spatial combination of C gives a visual color perception close to either C1 or the average of C1 and C2. A and C B is such that the energy consumption for displaying the pair of colors C1 and C2 is less than the energy consumption for displaying the pair of colors C1 and C2.

[0057] The repetition of step 420 generates a set of candidate pairs of spatially alternating complementary colors {C AB}. From this set, a preferred candidate pair (e.g., the candidate pair with the lowest energy consumption) for a given input may be selected, thus generating an association between the input color and pairs of spatially alternating complementary colors. In at least one embodiment, the association is stored in a lookup table. This allows for faster implementation by eliminating the need for the display device to perform all iterations of step 420 again for each image. Thus, at least one embodiment includes steps 410, 420, 430 being repeated across all possible colors in the color gamut, thus leading to the creation of the lookup table, while another embodiment only includes steps 410, 430, and 440 being repeated across all pairs of pixels in the input image, leading to a modified image that consumes less energy to display.

[0058] The input of the lookup table is a single input color. In embodiments based on pixel doubling, this input color is the color of the first pixel of an adjacent pair of pixels. In embodiments based on pixel dropping, this input color can be the color of either the first pixel or the second pixel of the adjacent pair of pixels, depending on the pixel being dropped (i.e., not necessarily the first pixel). In embodiments based on pixel averaging, this input color is the average color between the colors of the first and second pixels of the adjacent pair of pixels.

[0059] 5 shows an example of a process for establishing candidate pairs of spatially alternating complementary colors according to a first embodiment. Process 500 is performed, for example, by processor 101 of device 100 of FIG. 1 and corresponds to step 420 of FIG. 4. This is described here according to an embodiment based on pixel averaging. The process is performed for a pair of adjacent input pixels p1 and p2, whose colors are represented by input color triplets R1G1B1 and R2G2B2, respectively.

[0060] In step 510, the processor calculates the energy consumption P of the pair of adjacent input pixels p1 and p2 according to the selected color power consumption model. IN Determine. P IN =P(R1G1B1)+P(R2G2B2)

[0061] The power consumption model is display dependent. For OLED displays, the power consumption model can follow a color model, including the RGBW case where a white LED supports the RGB LED in each physical pixel. The color model for RGBW displays is described by Murdoch et al. in "Perfecting the color reproduction of RGBW OLED" proc. 30th International Congress of Imaging Science. This model can be extended with appropriate parameters to represent the power per pixel.

[0062] In step 520, the processor calculates the average color point C between p1 and p2, which corresponds to the average between the input color triplets R1G1B1 and R2G2B2 in the color gamut {G} of the color space {C}. IN The color space of this drawing is selected from among a display color space, a standard color space, or a human visual color space. In embodiments based on pixel doubling or pixel dropping, C IN The averaging calculation can be omitted since only a single color is considered, so that is the color of the selected pixel (i.e., the one that has been doubled or not dropped).

[0063] In step 530, the processor samples the color space {C} in the color gamut {G} to find a set of candidate colors {C A}. Different criteria may be used to determine the sampling space. In at least one embodiment, sampling is determined by C IN A maximum color difference criterion is used to restrict the color space around C. In various embodiments, the set of candidates may be saturated colors, or grayscale colors (i.e., a subset of gray shades), or C. IN or a combination of these colors. In at least one embodiment, the entire color gamut space is searched, and thus the set of candidates is the complete set of possible values. In at least another embodiment, a subset of the color gamut space is selected, for example, using a smaller color resolution. In another embodiment, several randomly selected candidates are used.

[0064] The process then iterates through steps 540-590 to find a set of candidate colors for the first pixel of the neighboring pixels {C A}. In step 540, the processor constructs a set of candidate color pairs for each color in the selected color C A , the second color C for the second pixel in the adjacent pixel B C B =2 C IN -CA or, alternatively, 2·C IN =(C A +C B ) is determined.

[0065] This allows C IN is the color C in color space {C} A and C B Since it is the average of A and C B The spatial combination of adjacent pixels of color C IN This ensures that the pixels look similar to the original pixel.

[0066] In step 550, the processor B is included in the gamut {G} of the color space {C}. In fact, if a color is outside the gamut, it cannot be displayed, and as a result, C A and C B Subpixel combination is color C IN If the color is out of gamut, it will not result in correct color pairing for adjacent pixels of spatially alternating complementary colors, so the selected C A The iteration stops for a value of . In this case, the process has a value of C A If the next value of remains, the iteration begins again at step 540 .

[0067] In step 560, the processor A , C B The corresponding triplet R of the pair A G A B A , R B G B B B Determine.

[0068] In step 570, the processor A , C B The energy consumption of the pixel combination P ABSince the power consumption model of a display device is not necessarily known, the energy consumption to display a color can be simply approximated by the sum of the gamma powers of its RGB values, as shown in Table 1. This is evaluated as follows for the selected color power consumption model: P AB =P(R A G A B A )+P(R B G B B B )

[0069] In step 580, the processor AB <P IN In fact, a candidate pair of adjacent pixel colors is considered only if it results in some energy reduction. In other words, a candidate pair of alternating complementary colors is considered if the sum of the energies of the pair of pixels of alternating complementary colors is less than the sum of the energies of the original pair of pixels. If not, the candidate pair is discarded and the process continues until there is no candidate pair of C in the set. A If the next value of remains, the iteration begins again at step 540 .

[0070] In step 590, the processor selects candidate pair C A , C B Let us denote the set of candidate pairs {C AB}.

[0071] In step 430, the processor selects candidate pairs C as spatially alternating complementary colors. A , C B to replace the colors C1 and C2 of pixels p1 and p2. In at least one variant, the processor selects one of the candidate pairs {C AB}, the position of the candidate pair with the lowest energy consumption is identified as follows: xmin=argmin{P(R A G A B A )+P(R B G B B B )}

[0072] As a result, the processor selects

number

[0073] In at least one embodiment, the selection of the best candidate pair is done directly in step 590. The process does not use a list of candidates but works with a single candidate. The new candidate provided in step 580 is compared with the previously selected candidate, and then the one with the lowest energy consumption is kept as the selected candidate (starting with the first candidate by default).

[0074] In at least one embodiment, the input color C is IN and its best color pair substitution C Axmin , C Bxmin The correspondence between is stored in the lookup table introduced with respect to Figure 4. By providing the input colors to the lookup table, it is possible to immediately obtain the corresponding color pair without having to perform process 400 of Figure 4 again.

[0075] In at least one embodiment, a mathematical minimization method, such as least-squares minimization, is used to replace steps 520-590 to find the color pair C1 and C2 and their best color pair replacement C Axmin , C Bxmin Find the correspondence between

[0076] In at least one embodiment, an additional step is added between steps 520 and 530 to check that the pair of adjacent pixels p1 and p2 spatially belongs to a subset of the image to be processed, e.g., belongs to a region in the image that has the greatest ability to mask artifacts. Such a region or mask may be given, for example, by a spatiotemporal just noticeable difference (JND) map, a motion field, a saliency map, etc. If the pair of adjacent pixels p1 and p2 does not belong to this region or mask, then the pair of adjacent pixels p1 and p2 is not considered for color pair replacement.

[0077] In at least one embodiment, the triplet {R IN G IN B IN} are ordered according to their energy consumption, so that the triplets are processed in descending order from most consuming to least consuming. In such a case, a threshold may be defined that corresponds to the overall energy reduction to be achieved. Once a certain number of triplets have been processed and this threshold is reached, the overall process is stopped. Other ordering criteria may be defined, such as determining which RGB combinations on the display consume the most energy and can be replaced with the greatest effect. Substitution Power Ratio

number

number

[0078] 6 shows an example of a process for establishing candidate pairs of spatially alternating complementary colors in a color space that provides a color transformation and an inverse color transformation according to a second embodiment. Process 600 is performed, for example, by processor 101 of device 100 of FIG. 1 and corresponds to step 420 of FIG. 4. The difference from the first embodiment is that the operations are performed in a color space that provides a color transformation and an inverse color transformation. Examples of such color spaces are the CIE XYZ color space or the OKLab color space used herein.

[0079] When using CIE XYZ, a color is defined by a triplet of coordinates in XYZ space, where Y is luminance, Z is pseudo-equal to B, and X is a blend of the three RGB curves. First, a compress / expand transform is applied (also called "gamma correction" or "companding"), i.e., each RGB value is raised to the power of a gamma value (which depends on the display). Next, a forward color transform FT(RGB) allows the XYZ coordinates to be calculated from the RGB values, and an inverse color transform IT(XYZ) applies the inverse operation, i.e., the RGB values ​​can be calculated from the XYZ coordinates, followed by an inverse companding operation.

[0080] Process 600 is performed for a pair of adjacent input pixels p1 and p2 whose colors are represented by input color triplets R1G1B1 and R2G2B2, respectively.

[0081] In step 610, the processor calculates the energy consumption P for the pair of adjacent input pixels p1 and p2 according to the selected color power consumption model. IN Determine. P IN =P(R1G1B1)+P(R2G2B2)

[0082] In step 620, the processor calculates the average color point C between p1 and p2, which corresponds to the average of the input color triplets R1G1B1 and R2G2B2 in the color gamut {G} of the color space {C}. IN In embodiments based on pixel doubling or pixel dropping, C INThe averaging calculation can be omitted since only a single color is considered, so that is the color of the selected pixel (i.e., the one that is being doubled or not dropped).

[0083] In step 630, the processor IN G IN B IN Applying a forward color transform from RGB to XYZ to the triplet yields X in XYZ space. IN Y IN Z IN The forward color transform is display dependent. An example of a color transform is the sRGBtoXYZ matrix, whose white point corresponds to the CIE standard illuminant D65. In step 640, the processor samples the color space {C} within the color gamut {G} to obtain a set of candidate colors {C} for adjacent pixels of spatially alternating complementary colors. A}. Determine the coordinate X of the candidate color. A Y A Z A may verify a specific condition. In a first variant, a selected C A X so that the color is a part of a gray shade (i.e., a grayscale color). A =Y A =Z A In a second variant, X A coordinate, Y A Coordinate or Z A One of the coordinates is selected C A In a third variant, the selected C A Y so that the color has no or minimal variation in luminance compared to the input color, and is therefore closer to isoluminance. A =Y IN In a fourth variant, Y is adjusted to minimize the variation in luminance compared to the input color (quasi-isoluminance). A =α Y IN , where α∈[0.8,1.2].

[0084] The process then iterates through steps 650-690 to determine a set of candidate colors for spatially alternating complementary adjacent pixels {C A}. In step 650, the processor constructs a set of candidate color pairs for each color in the selected color C A For coordinate X B Y B Z B Second color C B In color space, the color is C IN Regarding C A In the chosen color space, this is done simply using the following calculation:

number

[0085] In step 655, the processor B is included in the gamut {G} of the color space {C}. In fact, if a color is outside the gamut, it cannot be displayed, and as a result, C A and C B Subpixel combination is color C IN If the color is out of gamut, it will not result in the correct color pairing for temporally consecutive pixels, so the selected C A The iteration stops for a value of . In this case, the process has a value of C A If the next value of remains, the iteration begins again at step 650 .

[0086] In step 660, the processor A , C B The corresponding triplet R of the pair A G A B A , R B G B B B Inverse transformation R A G A B A =IT(X A Y A Z A ) and RB G B B B =IT(X B Y B Z B In step 670, the processor determines the color C A , C B , which is evaluated using the selected color power consumption model, for example, as follows: P AB =P(R A G A B A )+P(R B G B B B )

[0087] In step 680, the processor AB <P IN In fact, a candidate pair of adjacent pixel colors is considered only if it results in some energy reduction. In other words, a candidate pair of alternating complementary colors is considered if the sum of the energies of the pair of pixels of alternating complementary colors is less than the sum of the energies of the original pair of pixels. If not, the candidate pair is discarded and the process continues until there is no candidate pair of C in the set. A If the next value of remains, the iteration begins again at step 650 .

[0088] In step 690, the processor selects candidate pair C A , C B Let us denote the set of candidate pairs {C AB}.

[0089] In step 430, the processor selects candidate pairs C as spatially alternating complementary color pairs. A , C B to replace the color of the pair of adjacent pixels p1 and p2. Several variations and embodiments described in connection with Figure 5 for selecting one pair from a set of candidate pairs also apply here, including building a look-up table that stores associations between input colors and pairs of spatially alternating complementary colors.

[0090] The third embodiment is based on the same process as described in FIG. 6, with the difference that the color space selected is a uniform color space (e.g., CIELab, IPT, OKLab, etc.). A uniform color space is constructed so that the same geometric distance (2-distance) everywhere in the color space reflects the same amount of perceptual color difference. In the following description, the CIELab color space is selected. In this color space, color is represented by three values ​​L, a, and b, where L is the perceived lightness, and a and b are the four inherent opponent colors of human vision, namely red, green, blue, and yellow. In this embodiment, some modifications are required to adapt some of the steps of process 600 to a uniform color space. In step 630, the processor calculates R IN G IN B IN Applying the forward color transformation RGB to CIELab to the triplet yields L in the CIELab color space. IN a IN b IN The forward color transform conventionally involves a gamma operation. In step 640, the processor samples the color space {C} within the color gamut {G} to obtain a set of spatially alternating complementary neighboring pixel candidate colors {C A} is determined. The coordinates of the candidate color L A a A b A may verify a specific condition. In a first variant, a selected C A a so that the color is a part of a gray shade (i.e., a grayscale color). A =b A In a second variant, L A a A b A and its cylindrical version L A C A h A L IN a IN b IN In a third variation, the selected C A L so that the color has no or minimal variation in luminance compared to the input color, and is therefore closer to isoluminance.A =L IN In a fourth variant, L is adjusted to minimize the variation in luminance compared to the input color (quasi-isoluminant). A =α L IN , where α∈[0.8,1.2]. In step 650, the processor A For coordinate L B a B B B Second color C B and the color is C IN Regarding C A In a uniform color space, this is done simply using the following calculation:

number

[0091] In step 660, the processor A , C B The corresponding triplet R of the pair A G A B A , R B G B B B Inverse transformation R A G A B A =IT(X A Y A Z A ) and R B G B B B =IT(X B Y B Z B ) The other steps of process 600 remain the same. The variations and embodiments described in relation to Figures 5 and 6 also apply to the third embodiment.

[0092] 7 illustrates an example of replacing pairs of adjacent pixels with pairs of spatially alternating complementary color pixels based on resolution enhancement, according to one embodiment. In the figure, array 700 represents an example of an original (i.e., before modification) image to be displayed, here including three rows of four pixels each (to simplify the drawing). Each pixel is represented by a rectangle containing color values ​​according to the colors defined in FIG. 3. For example, pixel 701 in the upper left corner is a brown pixel.

[0093] In a first variant of such an embodiment, the width is doubled compared to the original image: for example, if the input image has a resolution of 1920 x 1080 pixels, the modified image, in which the original pixels are replaced by spatially alternating complementary color pixels, has a resolution of 3840 x 1080 pixels.

[0094] Array 710 represents an image based on the original image of array 700, modified according to this first variant. The processor first inserts additional columns 711, 712, 713, and 714 to create pixel duplication. The processor then selects a first pair of adjacent pixels, e.g., pixels 715 and 716. From the color (301) of the first pixel 715 of the first pair of adjacent pixels, the processor determines a pair of spatially alternating complementary colors (301A, 301B), e.g., using a lookup table, as described above. These colors are then used to set the colors of the adjacent pair of pixels. As a result, the color of pixel 715 is set to 301A, and the color of pixel 716 is set to 301B. The process is repeated for all pairs of adjacent pixels, and the result is a modified image 710 that, when displayed on a screen, appears similar to original image 700 but requires less energy to display.

[0095] In a second variant of such an embodiment, the height is doubled compared to the original image: for example, if the input image has a resolution of 1920 x 1080 pixels, the modified image, in which the original pixels are replaced by spatially alternating complementary color pixels, has a resolution of 1920 x 2160 pixels.

[0096] Array 720 represents the image modified according to the second variant. The process is the same as described in the previous paragraph, but adapted vertically. The insertion is done vertically, thus rows 721, 722, 723 are inserted. The selection of adjacent pixel pairs is now vertical. The process results in a modified image in which the color of the first pixel in the first column is set to 301A and 301B, respectively.

[0097] In a third variant of such an embodiment, both the width and height are doubled compared to the original image. For example, if the input image has a resolution of 1920 x 1080 pixels, the modified image, in which the original pixels are replaced by spatially alternating complementary color pixels, has a resolution of 3840 x 2160 pixels. In other words, each pixel is replaced by a set of four pixels consisting of two spatially alternating complementary colors. In this case, the second constraint verification related to energy consumption must be adapted accordingly, i.e., by comparing the total power of the four replacement pixels with four times the power of the original pixel.

[0098] Array 730 represents an image modified according to the third variant, which combines row and column insertions as described in the first two variants. As a result, original pixel 701 of array 700 is replaced by a set of four pixels 731, 732, 733, 734, having colors 301A, 301B, 301B, and 301A, respectively. It is sufficient to determine the pair of spatially alternating complementary colors corresponding to color 301 of original pixel 701 only once. As a result, the pairs (301A, 301B) applied to the four pixels 731, 732, 733, 734 are tessellated (i.e., the order between the colors of the pair is changed for the second row), providing a good distribution of colors.

[0099] The first and second variants can be implemented internally in the display panel by physically doubling the number of pixels in one direction, but without providing access to the additional pixels to the outside world.

[0100] The third variant is more commonly used. In fact, the resolution of currently available content is often inferior to the capabilities of the display device. It is quite common for full HD content (1920x1080) to be displayed on a UHD-capable (3840x2160) or 4K-capable (4096x2160) device. Therefore, this technique can be considered a simple upscaling function that provides additional pixels for upscaling while also reducing the energy required to display the upscaled image.

[0101] In these embodiments, the first pixel of a pixel pair is always replaced with the first color of the spatially alternating complementary color pair. In at least one embodiment, an interleave is introduced between rows (e.g., columns) with respect to the order of selection of color pairs. In a first row (e.g., column), the color of the first pixel of a pixel pair is replaced with the first color of the spatially alternating complementary color pair, and the color of the second pixel of the pixel pair is replaced with the second color of the spatially alternating complementary color pair, while in a second row (e.g., column), the color of the first pixel of a pixel pair is replaced with the second color of the spatially alternating complementary color pair, and the color of the second pixel of the pixel pair is replaced with the first color of the spatially alternating complementary color pair.

[0102] 8A and 8B show examples of replacing adjacent pixels with spatially alternating complementary color pixels based on pixel skipping, according to an embodiment. This method may be used, for example, when the image resolution cannot be increased, typically when the image to be displayed has the same resolution as the display. In FIG. 8A, array 800 represents an image containing 6 rows and 8 columns of pixels, with each pixel represented by a rectangle containing color values ​​according to the colors defined in FIG. 3.

[0103] This embodiment is based on skipping (dropping) one column out of two, as shown in array 810, where the dropped pixel is represented by an "X" symbol. In this case, the pair of adjacent pixels is a horizontal pair. For example, adjacent pixel pair 813 includes pixels 811 and 812, and the value of pixel 812 is skipped, i.e., not considered for determining the spatially alternating complementary color pair. In at least one embodiment, the skipped pixel is replaced with a copy of the adjacent pixel. In other embodiments, effectively no action is taken on the pixel's value because it is simply not considered by the above-described process.

[0104] The pixels of array 810 are then used by one of the algorithms described above for determining spatially alternating complementary pixels, resulting in array 820. This array 820 illustrates an embodiment in which the color of the first pixel of the first horizontal pair of pixels (in the upper left corner, whose original color was 301) is replaced with color 301A, which corresponds to the first color of the spatially alternating complementary pair determined in terms of color value 301, while the color of the second pixel of the first pair of pixels (whose original value was 302, but has been dropped) is replaced with color 301B, which corresponds to the second color of the spatially alternating complementary pair determined in terms of color value 301.

[0105] In this scheme, the first pixel of a pixel pair is always replaced with the first color of a spatially alternating complementary pair. A different scheme, shown in array 830, introduces an interleaving between rows in terms of the order of color pair selection. In the first row, the color of the first pixel of a pixel pair is replaced with the first color of a spatially alternating complementary pair, and the color of the second pixel of the pixel pair is replaced with the second color of the spatially alternating complementary pair, while in the second row, the color of the first pixel of a pixel pair is replaced with the second color of the spatially alternating complementary pair, and the color of the second pixel of the pixel pair is replaced with the first color of the spatially alternating complementary pair. This is particularly interesting in uniform areas. This can be seen in the upper right corner, where such a solution results in a good (mosaic-style) diffusion of colors 306A and 306B in the area that originally had uniform color 306.

[0106] FIG. 8B illustrates various arrangements for skipping pixels according to one pixel-skipping-based embodiment. Array 840 represents an image containing three rows and four columns of pixels, with each pixel represented by a different number to distinguish it from the others for better understanding of the arrangements. In each arrangement, dropped pixels are represented by an "X" symbol, and pairs of adjacent pixels are represented by dashed rectangles. In at least one embodiment, the color of the non-dropped pixels is copied to the dropped pixels to provide the necessary input for process 500 of FIG. 5 and process 600 of FIG. 6.

[0107] Arrays 841 and 842 represent images in which pairs of adjacent pixels are selected horizontally: in array 841, the first (left) pixel of the pair is skipped, and in array 842, the second (right) pixel of the pair is skipped, resulting in two arrangements of the same shape but different color values.

[0108] Arrays 843 and 844 represent images in which pairs of adjacent pixels are selected vertically. In array 843, the first (top) pixel of the pair is skipped, and in array 844, the second (bottom) pixel of the pair is skipped. This results in two arrangements of the same shape but different color values. These arrays also show an example of what happens when there is an odd number of pixels in one direction. In this example, the last pair includes the pixel in the last row and the pixel above it.

[0109] Array 845 represents an image in which pairs of adjacent pixels are selected horizontally. In this arrangement, the skipped pixels within the pairs alternate from one row to the other. For example, in the first row of this array, the second (right) pixel of the pair is skipped (hence pixels 872 and 874), in the next row it is the first (left) pixel of the pair that is skipped (hence pixels 881 and 883), and so on. Array 846 shows a similar configuration, but done vertically.

[0110] Arrays 846 and 847 represent an image in which pairs of adjacent pixels are selected horizontally according to the colors of the pairs. In array 846, the skipped pixel is the pixel with the highest energy in the pair. For example, assuming pixel 871 requires less energy than pixel 872, this pixel is kept and pixel 872 is skipped. In array 847, the skipped pixel is the pixel with the lowest energy in the pair. As a result, pixel 872 is kept and pixel 871 is skipped. Other criteria may be used to determine which pixels to skip.

[0111] In at least one embodiment, the processor generates a plurality of images using the different arrangements described above according to a selection of the different embodiments, calculates a corresponding total energy for each of the corresponding modified images, and then selects the image with the lowest energy as the modified image.

[0112] 9 shows an example of color replacement by spatially alternating complementary color replacement based on pixel averaging, according to one embodiment. This method may be used, for example, when the display resolution cannot be doubled. It is based on averaging the colors of pairs of adjacent pixels. Compared to the pixel skipping technique discussed in connection with FIG. 8, this allows all pixels of the image to be considered.

[0113] Array 900 represents an image containing 6 rows and 8 columns of pixels, with each pixel represented by a rectangle containing a color value according to the colors defined in Figure 3. In this embodiment, the possible arrangements of adjacent pixel pairs are limited to a choice of either horizontal or vertical orientation. Arrays 910, 920, and 930 illustrate a situation where a horizontal pairing arrangement is selected, as indicated by the dashed rectangles representing adjacent pixel pairs.

[0114] In this case, an average color is calculated between adjacent pairs of pixels, for example, the first two pixels in the first row of array 900 (color values ​​301 and 302), to yield an average color (value 312) as shown in array 910. This average value is used according to process 500 of Figure 5 or process 600 of Figure 6 to determine the color pair (312A, 312B) shown in array 920, thus providing a modified image that resembles the original image but requires less energy to display.

[0115] In an alternative embodiment, the order of the color pairs is swapped row by row, as shown in array 930. In a first row, the color of the first pixel of an adjacent pixel pair is replaced with the first color of the spatially alternating complementary pair (312A), while in a second row, the color of the first pixel of an adjacent pixel pair is replaced with the second color of the spatially alternating complementary pair (331B), and so on.

[0116] 10 illustrates an example process for generating a lookup table for spatially alternating complementary colors, and an example process for modifying an image using the lookup table, according to an embodiment, which are implemented based on the embodiments described above.

[0117] Process 1000 is directed to generating a lookup table of spatially alternating complementary colors. Step 1010 is repeated multiple times. In at least one embodiment, the iteration occurs over all possible colors in the color gamut. In at least one other embodiment, the iteration occurs only over a subsampled color space. In at least one other embodiment, the iteration occurs over all colors in a given image set. Other embodiments may use other color subsets. This includes steps 1020-1050. In step 1020, an input color is obtained. In step 1030, a set of spatially alternating complementary color pairs corresponding to the input color is determined, for example, using one of the embodiments described above. In step 1040, one of the pairs in the set is selected, for example, the pair that provides the best energy performance. In step 1050, the associations between the input color and the spatially alternating complementary color pairs are stored in a lookup table. At the end of process 1000, the lookup table contains a set of associations between the input color and the spatially alternating complementary color pairs.

[0118] Process 1001 aims to modify an image using a lookup table. In step 1060, an input image is obtained. In step 1070, a pair of adjacent pixels is selected, and in step 1080, a pair of spatially alternating complementary colors corresponding to a color based on the pair of adjacent pixels is obtained from the lookup table. In step 1090, the colors of the pair of adjacent pixels are set to the pair of spatially alternating complementary colors. Steps 1070, 1080, and 1090 are then repeated for all pairs of adjacent pixels.

[0119] FIG. 11 shows two examples of deployment of a spatially alternating complementary color process, according to an embodiment. In at least one embodiment, device 1101 is a display device such as that described in FIG. 1. In this case, processor 101 of device 1101 is configured to acquire an input image or video 1100, process it through the SACC process as described above, and then display it on display unit 103 of FIG. 1. In other words, processor 101 of device 1001 is configured to acquire an input image or video 1100 and determine a modified image or video to be displayed using the acquired spatially alternating complementary colors by using a lookup table, resulting in an image that reduces energy consumption of the display device when compared to displaying the original input image. The lookup table may be acquired from a data provider via a communications network and / or from the device's internal memory. Image or video 1100 may be acquired from a data provider via a communications network, for example, from the device's internal memory where it was captured by an input unit and then stored. Typical examples of device 1101 are smartphones, tablets, laptops, external monitors, head-mounted displays, television sets, video projectors, computer screens, water vehicles (e.g., control and / or entertainment systems for automobiles, airplanes, boats, etc.), advertising display panels, medical monitors, etc. However, any device or component of a device that provides similar functionality may be used as display device 1101 while conforming to the principles of the present disclosure.

[0120] In at least one embodiment, the device 1102 does not include a display unit, but prepares data for display so that another device 1103, such as a screen, can perform the display. In this case, a processor in the device performs the SACC process described herein to generate a new image or video 1110 that is perceptually similar to the original video but requires less energy when displayed. This modified video is then provided to the display device 1103 for presentation to a human viewer. Examples of such devices 1102 include set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0121] Light generation in display devices, including mobile phones and televisions, is expensive. Reducing the amount of light generated is desirable, as it helps reduce the amount of energy needed to operate the display. The benefits of this are twofold: it reduces the burden on the climate and extends the battery life of mobile devices. Compared to other methods that aim to reduce energy consumption for the same reasons, the proposed method, by its configuration, ensures that the light emitted by each pixel is generated by a combination of subpixels, minimizing energy consumption on average. This means that it is more versatile and therefore more efficient than operating on a single pixel.

[0122] Although different embodiments have been described separately, any combination of the embodiments can be made while respecting the principles of the present disclosure.

[0123] References to "one embodiment" or "one embodiment" or "one implementation" or "one embodiment," as well as other variations, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in one embodiment" or "in one embodiment" or "in one embodiment" in various places throughout this specification, as well as any other variations, are not necessarily all referring to the same embodiment.

[0124] Additionally, the application or its claims may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0125] Additionally, the application or its claims may refer to "obtaining" various information. Obtaining, like "accessing," is intended to be a broad term. Obtaining information may include, for example, one or more of receiving information, accessing information, or retrieving information (e.g., from memory or optical media storage). Furthermore, "obtaining" typically involves in some way an operation such as, for example, storing information, processing information, transmitting information, moving information, replicating information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0126] Additionally, the terms "image" and "frame" are used interchangeably herein and both are used to refer to a set of pixels arranged, for example, in a two-dimensional array. A sequence of multiple images or frames arranged according to a temporal order is conventionally referred to as a "video."

[0127] It should be understood that the use of any of the following terms " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B), for example, in the case of "A / B," "A and / or B," and "at least one of A and B." As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This may be extended to all items listed, as would be readily apparent to one of ordinary skill in the art.

Claims

1. For the input color of a pixel, determining pairs of alternating complementary colors based on the input colors of the pixels; an average color of the alternating complementary color pairs is the same color as or a perceptually similar color to the input color; determining that the sum of the energy consumed by displaying pairs of pixels having alternating complementary colors is less than twice the energy consumed by displaying pixels having said input colors; A method comprising:

2. 10. The method of claim 1, further comprising: selecting a color valence in a color space for a first color of the pair of alternating complementary colors according to a selection criterion; and determining a second color of the pair of alternating complementary colors based on the selected first color valence and the input color.

3. The method of claim 2 , wherein the selection criterion is a maximum color difference from the input color.

4. The method of claim 2 , wherein the selection criterion is that the color is a saturated color.

5. The method of claim 2 , wherein the selection criterion is that the color is a grayscale color.

6. The method of claim 2 , wherein the selection criterion is that the color has the same luminance as the input color.

7. 7. The method of claim 1, wherein the color space is an XYZ color space and the second color of the alternating complementary pair is selected to be symmetrical to the first color of the alternating complementary pair with respect to the input color.

8. 7. The method of claim 1, wherein the color space is a uniform color space and the second color of the alternating complementary pair is selected to be symmetrical to the first color of the alternating complementary pair with respect to the input color.

9. 9. The method of claim 1, wherein the determining the color pairs is repeated a number of times to generate a set of alternating complementary color pairs, and further comprising selecting the alternating complementary color pair from the set of alternating complementary color pairs that has the lowest energy consumption.

10. The method of claim 9 further iterating over a set of input colors comprising the colors of all pixels of the input image or a subset of all pixels of the input image.

11. 10. The method of claim 9, further iterating over a set of input colors comprising all possible color values ​​in the selected color space or a subset of all possible color values ​​in the selected color space.

12. 12. The method of claim 10 or 11, further comprising storing an association between the set of input colors and the corresponding determined pairs of alternating complementary colors.

13. The method of claim 12 , wherein the associations are stored in a look-up table using the input colors as indexes.

14. obtaining pairs of adjacent pixels of the image; Obtaining pairs of alternating complementary colors based on the colors of the pairs of adjacent pixels according to any one of claims 1 to 13; setting the color of a first pixel of said pair of adjacent pixels to said first color of said pair of alternating complementary colors and setting the color of a second pixel of said pair of adjacent pixels to said second color of said pair of alternating complementary colors; A method comprising:

15. 15. The method of claim 14, further comprising, before the first step of method 14, adding new pixels to double the width of the image, converting single pixels into pairs of adjacent pixels, the pairs of adjacent pixels being horizontal pairs.

16. 15. The method of claim 14, further comprising, before the first step of method 14, adding new pixels to double the height of the image, converting single pixels into pairs of adjacent pixels, the pairs of adjacent pixels being vertical pairs.

17. 15. The method of claim 14, further comprising, before the first step of method 14, adding new pixels to double the width of the image and double the height of the image, converting single pixels into two pairs of adjacent pixels.

18. The method of claim 17 , wherein the two pairs of adjacent pixels are horizontal pairs of pixels.

19. The method of claim 17 , wherein the two pairs of adjacent pixels are vertical pairs of pixels.

20. The method of claim 14 , wherein the second pixel of the pair of adjacent pixels is skipped, and the alternating complementary color pairs are based on the color of the first pixel of the pair of adjacent pixels.

21. 15. The method of claim 14, wherein the first pixel of the pair of adjacent pixels is skipped and the alternating complementary color pairs are based on the color of the second pixel of the pair of adjacent pixels.

22. The method of claim 14 , wherein the alternating pairs of complementary colors are based on an average color between the colors of the first pixel and the second pixel of the pair of adjacent pixels.

23. Obtaining pairs of adjacent pixels of an image of the video; Obtaining pairs of alternating complementary colors according to any one of claims 1 to 13; setting colors of the pair of adjacent pixels of the image of the video to spatially alternating complementary colors if the pixel is spatially contained within a region of interest selected according to a criterion, wherein a first pixel of the pair of adjacent pixels is set to a first color of the pair of alternating complementary colors and a second pixel of the pair of adjacent pixels is set to a second color of the pair of alternating complementary colors; A method comprising:

24. The method of claim 23 , wherein the selection criteria is based on a spatiotemporal just noticeable difference map.

25. The method of claim 23 , wherein the selection criteria is based on a motion field.

26. The method of claim 23 , wherein the selection criteria is based on a saliency map.

27. The method of claim 23 , wherein the selection criteria is based on eye tracking.

28. The method of claim 23 , wherein the selection criteria is based on attention modeling.

29. 24. The method of claim 23, wherein the selection criteria is a metadata database.

30. For the input color of a pixel, determining pairs of alternating complementary colors based on the input colors of the pixels; an average color of the alternating complementary color pairs is the same color as or a perceptually similar color to the input color; the sum of the energy consumed by displaying pairs of pixels having alternating complementary colors is less than twice the energy consumed by displaying pixels having said input colors; 10. A device comprising: one or more processors configured to:

31. 31. The device of claim 30, further comprising: selecting a color valence in a color space for a first color of the pair of alternating complementary colors according to a selection criterion; and determining a second color of the pair of alternating complementary colors based on the selected first color valence and the input color.

32. 32. The device of claim 31, wherein the selection criterion is a maximum color difference from the input color.

33. 32. The device of claim 31, wherein the selection criterion is that the color is saturated.

34. 32. The device of claim 31, wherein the selection criterion is that the color is a grayscale color.

35. 32. The device of claim 31, wherein the selection criterion is that the color has the same luminance as the input color.

36. 36. A device according to any one of claims 30 to 35, wherein the colour space is an XYZ colour space and the second colour of the alternating complementary pair is selected to be symmetrical to the first colour of the alternating complementary pair with respect to the input colour.

37. 36. A device according to any one of claims 30 to 35, wherein the colour space is a uniform colour space and the second colour of the alternating complementary pair is selected to be symmetrical to the first colour of the alternating complementary pair with respect to the input colour.

38. 38. The device of claim 30, wherein the determining the color pairs is repeated a number of times to generate a set of alternating complementary color pairs, and further comprising selecting the alternating complementary color pair from the set of alternating complementary color pairs that has the lowest energy consumption.

39. 39. The device of claim 38, further iterating over a set of input colors that includes colors of all pixels of the input image or a subset of all pixels of the input image.

40. 39. The device of claim 38, further iterating over a set of input colors comprising all possible color values ​​in a selected color space or a subset of all possible color values ​​in a selected color space.

41. 41. A device according to claim 39 or 40, further comprising storing an association between the set of input colours and the corresponding determined pairs of alternating complementary colours.

42. 42. The device of claim 41, wherein the associations are stored in a lookup table using the input colors as indexes.

43. Obtaining pairs of adjacent pixels of the image; Obtaining pairs of alternating complementary colors based on the colors of the pairs of adjacent pixels according to any one of claims 30 to 42; setting the color of a first pixel of the pair of adjacent pixels to the first color of the pair of alternating complementary colors and setting the color of a second pixel of the pair of adjacent pixels to the second color of the pair of alternating complementary colors; 10. A device comprising: one or more processors configured to:

44. 44. The device of claim 43, wherein the method further comprises, before the first step of method 43, adding new pixels to double the width of the image, converting single pixels into pairs of adjacent pixels, the pairs of adjacent pixels being horizontal pairs.

45. 44. The device of claim 43, wherein the method further comprises, before the first step of method 43, adding new pixels to double the height of the image, converting single pixels into pairs of adjacent pixels, the pairs of adjacent pixels being vertical pairs.

46. 44. The device of claim 43, wherein the method further comprises, before the first step of method 43, adding new pixels to double the width of the image and double the height of the image, converting single pixels into two pairs of adjacent pixels.

47. 47. The device of claim 46, wherein the two pairs of adjacent pixels are horizontal pairs of pixels.

48. 47. The device of claim 46, wherein the two pairs of adjacent pixels are vertical pairs of pixels.

49. 44. The device of claim 43, wherein the second pixel of the pair of adjacent pixels is skipped, and the alternating complementary color pairs are based on the color of the first pixel of the pair of adjacent pixels.

50. 44. The device of claim 43, wherein the first pixel of the pair of adjacent pixels is skipped and the alternating complementary color pairs are based on the color of the second pixel of the pair of adjacent pixels.

51. 44. The device of claim 43, wherein the spatially alternating pairs of complementary colors are based on an average color between colors of the first pixel and the second pixel of the pair of adjacent pixels.

52. 1. A device comprising one or more processors, Obtaining pairs of adjacent pixels in the image of the video; obtaining pairs of alternating complementary colors according to any one of claims 30 to 42, setting colors of the pair of adjacent pixels of the image of the video to spatially alternating complementary colors if the pixel is spatially contained within a region of interest selected according to a criterion, wherein a first pixel of the pair of adjacent pixels is set to a first color of the pair of alternating complementary colors and a second pixel of the pair of adjacent pixels is set to a second color of the pair of alternating complementary colors; device.

53. 53. The device of claim 52, wherein the selection criteria is based on a spatiotemporal just noticeable difference map.

54. 53. The device of claim 52, wherein the selection criteria is based on a motion field.

55. 53. The device of claim 52, wherein the selection criteria is based on a saliency map.

56. 53. The device of claim 52, wherein the selection criteria is based on eye tracking.

57. 53. The device of claim 52, wherein the selection criteria is based on attention modeling.

58. 53. The device of claim 52, wherein the selection criteria is a metadata database.

59. 59. The device of any one of claims 30 to 58, wherein the device is selected in the set comprising a smartphone, a tablet, a laptop, an external monitor, a head mounted display, a television set, a video projector, a computer screen, a vehicle control system, a vehicle entertainment system, an advertising display panel, a medical monitor.

60. A computer program comprising program code instructions for performing the method of any one of claims 1 to 29 when the computer program is executed by a processor.

61. A non-transitory computer readable medium comprising program code instructions for performing the method of any one of claims 1 to 29 when executed by a processor.

Images