Method and device for reducing flicker of consecutive pixels of temporally alternating complementary colors - Patent Application 20070122999

Abstract

The method and device enable reducing the energy required to render an image by replacing pixels of the image with temporally consecutive pixels of alternating complementary colors that require less energy to display and are selected according to a polarity constraint. This solution exploits the flicker fusion property of the human visual system, which allows selecting complementary colors that can visually have the same perceptual properties as corresponding monochrome colors. The alternating complementary colors are selected to be more economical in terms of the energy consumption required to render the colors. This replacement is performed by either frame doubling, frame skipping, or frame averaging. The concept of consecutive pixels is temporal. In other words, when using frame doubling, one pixel is replaced by two temporally consecutive pixels of alternating complementary colors, with the replacement pixels having half the duration. When using frame skipping or averaging, two temporally consecutive pixels are replaced by two temporally consecutive pixels of alternating complementary colors, with the replacement pixels having the same duration. The polarity constraint enables a reduction in perceived flicker. The association between colors and corresponding alternating complementary colors can be obtained from a lookup table. These principles can be used for videos containing an image or a sequence of images.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to European Patent Application No. 22306994.9, filed December 22, 2022, which is incorporated herein by reference in its entirety.

[0002] At least one of the present embodiments relates generally to energy consumption in display devices, and more specifically to reducing the energy required to render an image by replacing pixels of the image with successive pixels of temporally alternating complementary colors selected according to a polarity constraint to reduce perceived flicker. [Background technology]

[0003] Reducing the energy consumption of electronic devices has become a requirement not only for electronic device manufacturers but also for those who want to limit their environmental impact as much as possible and contribute to the emergence of a sustainable display industry. The increase in display resolution from SD to HD, then to 4K, and in the near future to 8K and beyond, as well as the introduction of high dynamic range imaging, has resulted in a corresponding increase in the energy requirements of display devices. Given the vast number of devices that have displays (i.e., TVs, mobile phones, tablets, etc.), this is inconsistent with the global need to reduce energy consumption. In fact, displays are the most significant source of energy consumption for consumer electronic devices, whether they are battery-powered (e.g., smartphones, tablets, head-mounted displays, automotive display screens) or not (e.g., televisions, advertising display panels).

[0004] In recent years, various display technologies have been developed, and although modern displays consume energy in a more controllable and efficient manner than older displays, they are still the most significant source of energy consumption in video circuits.

[0005] Organic Light Emitting Diodes (OLEDs) are an example of a display technology that is gaining popularity due to a number of advantages over older technologies such as Thin-Film Transistor Liquid Crystal Displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays consist of individual LEDs as image pixels. OLED energy consumption is therefore highly correlated to image content, and the energy consumption of a given input image can be estimated by considering the values ​​of the image pixels being displayed.

[0006] Although OLED displays consume energy in a more controllable and efficient manner, they are still the most significant energy consumer in video circuits. Various techniques have been developed to reduce the energy required to display an image on a display device. To date, most solutions to the problem of reducing the energy required to display several image pulses have focused on modifying the color of each frame of the pulse by a small amount, either in brightness and / or color. Therefore, these solutions limit the number of dimensions searched to find an image that saves more energy. For pixel color, a separate pixel color is proposed, limiting the search space dimension to three color channels. Summary of the Invention

[0007] The embodiments described below are designed with the above in mind and describe methods and devices that enable reducing the energy required to render an image (i.e., power consumption) by replacing pixels of the image with temporally consecutive pixels of alternating complementary colors that require less energy to display and are selected according to a polarity constraint. This solution takes advantage of the flicker fusion properties of the human visual system, which allows selecting complementary colors that may visually have the same perceptual properties as corresponding monochrome colors. The alternating complementary colors are selected to be more economical in terms of the energy consumption required to render the colors. This replacement is performed by either frame doubling, frame skipping, or frame averaging. The concept of consecutive pixels is temporal. In other words, when using frame doubling, a pixel is replaced by two temporally consecutive pixels of alternating complementary colors, with the replacement pixels having half the duration. When using frame skipping or averaging, two temporally consecutive pixels are replaced by two temporally consecutive pixels of alternating complementary colors, with the replacement pixels having the same duration. The polarity constraint enables a reduction in perceived flicker. The association between a color and its corresponding alternating complementary color can be obtained from a look-up table.These principles can be used for an image or a video containing a sequence of images.

[0008] A first aspect is directed to a method that includes iterating over pixels of an image of a video, the iteration including, for a pixel, obtaining an alternating complementary color pair based on a color of the pixel; replacing the pixel of the image of the video with a pair of temporally consecutive pixels based on polarity information, wherein if the polarity information has a first value, setting the color of the first temporally consecutive pixels to a first color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to a second color of the alternating complementary color pair; otherwise, setting the color of the first temporally consecutive pixels to the second color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to the first color of the alternating complementary color pair; and inverting the polarity information for a next pixel.

[0009] A second aspect is directed to a device comprising one or more processors configured to iterate over pixels of an image of a video, the iteration including, for a pixel, obtaining an alternating complementary color pair based on a color of the pixel; replacing the pixel of the image of the video with a pair of temporally consecutive pixels based on polarity information, wherein if the polarity information has a first value, setting the color of the first temporally consecutive pixels to a first color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to a second color of the alternating complementary color pair; otherwise, setting the color of the first temporally consecutive pixels to the second color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to the first color of the alternating complementary color pair; and inverting the polarity information for a next pixel.

[0010] A third aspect of at least one embodiment is directed to a computer program comprising program code instructions executable by a processor, the computer program performing at least the steps of the method of the first aspect.

[0011] A fourth aspect of at least one embodiment is directed to a non-transitory computer-readable medium comprising program code instructions executable by a processor, the computer program product performing at least the steps of the method of the first aspect. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a block diagram of an example display device in which various aspects and embodiments may be implemented. [Figure 2A] 1 shows the normalized response spectrum (or spectral sensitivity function) of the human cones. [Figure 2B] Temporal contrast sensitivity functions for various adaptation fields are shown. [Figure 2C] 1 shows modulation sensitivity as a function of frequency for luminance and color flicker. [Figure 3] 1 illustrates an example of separating a color into alternating complementary colors, according to some embodiments. [Figure 4]1 illustrates an example of a process for reducing energy consumption of pixels in an image using alternating complementary colors, according to some embodiments. [Figure 5] 1 illustrates an example of a process for establishing candidate pairs of alternating complementary colors according to a first embodiment. [Figure 6] 10 illustrates an example process for establishing candidate pairs of alternating complementary colors according to a second embodiment in a color space that provides color transformations and inverse color transformations. [Figure 7] 10 illustrates an example of pixel replacement with temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame doubling. [Figure 8] 1 illustrates an example of pixel replacement with temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame skipping. [Figure 9] 1 illustrates an example of pixel replacement with temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame averaging. [Figure 10] 1 illustrates an example process for generating a lookup table of alternating complementary colors, according to some embodiments, and an example process for modifying an image using the lookup table, according to some embodiments. [Figure 11] 10A-10C illustrate two example arrangements for an alternating complementary color process, according to some embodiments. [Figure 12] 10 shows an example of pixel replacement with alternating complementary color pixels according to an alternative embodiment based on frequency doubling and using opposite polarity for adjacent blocks. [Figure 13] 10 shows an example of a block for pixel replacement with alternating complementary color pixels according to an alternative embodiment using opposite polarity for adjacent blocks with unequal block sizes. [Figure 14] 1 illustrates an example of a process for reducing energy consumption of pixels in an image using alternating complementary colors of opposite polarity, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] 1 illustrates a block diagram of an example of a display device in which various aspects and embodiments may be implemented. In the illustrated environment, a user interacts with a display device 100 that is connected to a data provider 180 via a communications network 150.

[0014] Display device 100 includes processor 101. Processor 101 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor may perform data processing such as process 400 of FIG. 4, process 500 of FIG. 5, or process 600 of FIG. 6, and related embodiments thereof, operating in a uniform color space.

[0015] The processor 101 may be coupled to an input unit 102 configured to convey user interactions. Multiple types of input and modalities may be used for that purpose. A physical keypad or a touch-sensitive surface are typical examples of inputs adapted for this use, although voice control may also be used. Furthermore, the input unit may comprise a digital camera capable of capturing two-dimensional still images or video, or a more complex sensor capable of determining depth information in addition to the image or video, thus capturing a full 3D representation.

[0016] The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. For that purpose, several types of displays can be used, such as an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to convert audio data into audio waves via a suitable transducer, for example a loudspeaker.

[0017] The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses a wireless communication standard to provide mobility for the display device, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc.

[0018] Processor 101 may access information from and store data in memory 106, which may comprise multiple types of memory, such as random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or any other type of memory storage device. In some embodiments, processor 101 may access information from and store data in memory that is not physically located on the device, such as a server, a home computer, or another device.

[0019] The processor 101 may receive energy from the power source 108 and may be configured to distribute and / or control the energy to other components within the device 100. The energy source may be any suitable device for providing energy to the device. By way of example, the energy source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0020] While the figure depicts the processor 101 and the other elements 102-108 as separate components, it will be understood that these elements may be integrated into an electronic package or chip. It will be understood that the display device 100 may include any subcombination of the elements described herein while remaining consistent with the embodiments described below. The processor 101 may be further coupled to other peripherals or units not shown in FIG. 1 , which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals may include a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, etc. For example, the processor 101 may be coupled to a location determination unit configured to determine the location of the display device within its environment. The location unit may integrate a GPS chipset that provides longitude and latitude position relative to the current location of the display device, but may also integrate other motion sensors such as an accelerometer and / or electronic compass that provide location services.

[0021] Typical examples of display device 100 are smartphones, tablets, laptops, external monitors, head-mounted displays, television sets, video projectors, computer screens, vehicles (e.g., control and / or entertainment systems for cars, airplanes, boats, etc.), advertising display panels, medical monitors, etc. However, any device or composition of devices providing similar functionality may be used as display device 100 and still conform to the principles of the present disclosure. In at least one embodiment, a device does not include a display unit but prepares data for display so that another device, such as a screen, can perform the display. Examples of such devices are set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0022] The following embodiments describe methods for modifying image light pulses for color pixels while maintaining as much visual similarity and perceived quality as possible with the original light pulses and reducing the energy required to display the modified image pulses on a display device. More generally, some embodiments are based on using temporal modulation to determine a color pair to replace an input color, where the average color of the color pair is perceptually similar / identical to the input color and the energy of the color pair is less than twice the energy of the input color. The term "color energy" should be understood herein as the energy required to render a color pixel. Different types of temporal modulation are described: frequency doubling, frame skipping, and frame averaging.

[0023] Compared with the state-of-the-art technique of selecting different colors on a single frame, which consumes less energy, using color pairs has the advantage of versatility, and the color pair selection is based on two parameters: energy consumption and visual matching with the original color. This method increases the search space of lower-energy colors, and therefore increases the probability of finding a combination that reduces energy more significantly.

[0024] Figure 2A shows the normalized response spectrum (or spectral sensitivity function) of human cones. Electromagnetic radiation is characterized by its wavelength (or frequency) and its intensity. The range of wavelengths perceptible by humans is approximately 380 nm to 780 nm. When wavelengths fall within this range, the light is known as "visible light." Color perception is based on the varying sensitivity of different cells in the retina (color receptors: cones and rods) to different wavelengths of light. Human observers have three types of color receptors known as cone cells. This gives us three-color color vision. Cones are typically labeled according to the wavelength of their peak spectral sensitivity (short (S), medium (M), and long (L)), or simply according to the primary color (blue, green, or red) around which their peak is centered, as shown in Figure 2A.

[0025] Trichromatic color theory teaches that the color perceived by a human observer in a light spectrum can be characterized by three single scalar values. From a mathematical perspective, this first step in human vision can be compared to that of a triple kernel energy calculation process. If si(λ) is the wavelength response of a given light spectrum, and l(λ), m(λ), and s(λ) are the spectral sensitivity functions of the L cone, M cone, and S cone, respectively, then Equation 1 below defines Li, Mi, and Si. These are the three scalar values ​​that characterize the color of spectrum si(λ) as seen by a human observer.

[0026]

number

[0027] The spectrum of light reaching the eye from a given direction determines the color sensation in that direction, but there are many more possible spectral combinations that result in the same color sensation. In colorimetry, the term metamerism refers to the matching of the same apparent color of light signals with different spectral power distributions. Such matching color spectra are called metameric spectra. Based on equation (1), the mathematical definition of metamerism is:

[0028]

number

[0029]

number

[0030] Figure 2B shows the temporal contrast sensitivity function (CSF) for various adaptation fields. In the spatial domain, spatial vision can be characterized by the contrast sensitivity function (CSF). To thoroughly investigate the visual system's response to flicker, the temporal contrast sensitivity function (TSF) or De Lange function can be plotted (De Lange, 1958). The TSF plots how flicker varies with contrast (and vice versa). In this figure, the area above the curve represents the region where a human observer does not perceive flicker, while the area below the curve represents the region where flicker is perceived. The eye appears to be most sensitive to flicker frequencies between 15 and 20 Hz at high luminance (photopic vision). At photopic light levels, less than 1% contrast is required to detect the stimulus, and the high-temporal-frequency cutoff is near 60 Hz. At low light levels, maximum contrast is approximately 20%, and the high-temporal-frequency cutoff is approximately 15 Hz. Maximum contrast is required to detect high-frequency flicker. Temporal resolution is not as efficient at low light levels (scotopic vision).

[0031] Figure 2C shows modulation sensitivity as a function of frequency for luminance and color flicker. In this figure, luminance levels are measured in trolands (td), which characterize retinal illuminance. This figure was obtained from a psychovision study in a typical application of heterochromatic flicker photometry (HFP). Participants viewed a stimulus that rapidly alternated in time between two lights of different colors. They then had to adjust the intensity (i.e., the amplitude of the light spectrum) of one of the two lights to minimize the perception of flicker caused by the alternating lights. The left figure relates to luminance flicker, and the right figure relates to color flicker. HFP has long been the standard psychophysical method for finding isoluminant colors.

[0032] The principles illustrated in Figures 2A, 2B, and 2C are used to determine pairs of colors that, when combined in time, are perceived by a human observer as a distinct (single, stable) color. The technical effect used in this invention relies on temporal psychovisual modulation and the existence of a maximum cutoff frequency in the human eye's flicker sensitivity. Therefore, the high-level principle of this invention can be thought of as adding dimension to an image signal by temporally replicating each pixel into two visually complementary, temporally consecutive pixels, and using this added dimension to minimize the equivalent energy consumption of the pixel. This principle is referred to herein as alternating complementary color (ACC). Two temporally consecutive pixels are perceived by a user as a single pixel if the alternation between these pixels is faster than the flicker fusion frequency.

[0033] The typical flicker fusion frequency is approximately 50Hz to 60Hz, depending on the retinal illumination. However, sensitivity to flicker in isoluminant situations is less (20Hz to 30Hz) than in situations where the luminance changes between two images of a pair. Furthermore, by adding the specific condition that the luminance difference between the two colors is small, flicker due to the alternation of the two colors is reduced. This isoluminant condition can be used to limit the visibility of flicker when mixed with an alternating configuration of primary colors.

[0034] FIG. 3 illustrates an example of decomposing color into alternating complementary colors, according to some embodiments. In this diagram, line 300 represents a series of pixels 301-306. The three numbers within each block correspond to the color of the corresponding pixel, represented by RGB values ​​expressed using 8-bit depth. For example, the first pixel is defined by pixel color component values ​​of 147 for red, 107 for green, and 0 for blue. This results in a brown pixel. The colors of the other pixels are: second pixel 302 is medium gray; third pixel 303 is dark navy blue; fourth pixel 304 is dark magenta; fifth pixel 305 is reddish-brown; and sixth pixel 306 is bright green, respectively.

[0035] Line 310 shows a set of temporally consecutive pixel pairs (301A, 301B) through (306A, 306B). These temporally consecutive pixel pairs correspond to alternating complementary colors that can be used to replace original pixels 301 through 306. As with line 300, the values ​​within the blocks represent the colors of the temporally consecutive pixels. In at least one embodiment, the temporally consecutive pixels are half the duration of the original pixels. In other words, a first image frequency (e.g., 60 Hz) is doubled to a second image frequency (e.g., 120 Hz), and the input image is decomposed into a pair of output images displayed at the second image frequency. For example, pixel 301, displayed in the input image at a frequency of 60 Hz, can be replaced by consecutive pixels 301A (a green pixel) and 301B (a red pixel), displayed at a global frequency of 120 Hz. The green and red consecutive pixels are perceived as brown pixels by a human observer due to heterochromatic flicker fusion. A complete example is described below in connection with Figure 7. In other embodiments, for example when doubling the display frequency is not possible, other techniques such as frame skipping or frame averaging are used. Examples of such methods are described below in connection with Figures 8 and 9.

[0036] 4 shows an example of a process for reducing energy consumption of pixels of an image using alternating complementary colors, according to some embodiments. Process 400 may be performed, for example, by processor 101 of device 100 of FIG. 1. In at least one embodiment, process 400 is iterated over a color set that includes the colors of all pixels of the input image. In another embodiment, process 400 is iterated over a color set that includes all possible color values ​​according to a selected color space. In other embodiments, iteration is performed over a subset of pixels or a subset of the color space.

[0037] In step 410, the processor calculates the color c of pixel p. IN Get.

[0038] In step 420, the processor IN The alternating complementary color pair c corresponding to A ,cB The color pair is selected based on two constraints. The first constraint is related to the quality of experience, and the color c A and c B Temporally consecutive pixels p A and p B But color c IN The second constraint relates to reducing the energy required for display. In embodiments using frame doubling, color c A and c B Temporally consecutive pixels p A and p B The energy required to display two temporally modulated half-cycles of color c IN In other embodiments using frame skipping or frame averaging, the second constraint is verified differently, as described further below. Step 420 ensures that the energy required to display a first color c is lower than the energy required to display a period of pixels p. A according to certain criteria as described in further embodiments, and then selecting a suitable second color c according to similarity constraints and energy reduction constraints. B and determining the input color c IN Step 420 results in the definition of color pairs corresponding to the set of candidate pairs of alternating complementary colors {C AB} are iterated multiple times (415).

[0039] In step 430, the processor selects one of the candidate pairs, C A ,C B , color C IN As a pair of temporally consecutive pixel colors to replace the pixel of , for example, select the candidate pair with the lowest energy consumption.

[0040] In step 440, the processor modulates the color c according to one of the time modulation techniques presented herein. IN pixel p, color c A and c B Two temporally consecutive pixels p A and p BReplace by

[0041] Regarding color similarity, the first constraint is verified by checking the color pair c A ,c B The average of color c IN The average of the color pairs is calculated in a color space, a standard color space, or a color space that represents human color vision. Examples of color spaces are sRGB, AdobeRGB. Examples of standard color spaces (also known as measurement color spaces) are CIEXYZ, CIELUV. Examples of color spaces that represent human color vision (also known as uniform color spaces) are CIELab, IPT, OKLab, OSA-UCS. The resulting colors of the display of the processed image with respect to human perception are therefore the desired colors as in the source image, while the energy consumption is calculated based on the color pair (c) depending on the display color power model. A ,c B ) is reduced thanks to the appropriate selection of

[0042] Regarding energy consumption, in embodiments using frame doubling, the verification of the second constraint is A and c B Two temporally consecutive pixels p A and p B is based on comparing each pixel p of color c with the corresponding pixel p A and p B The display period of pixel p is half the duration of pixel p. Energy expressed throughout this specification is based on a display color power model. A simple example of such a model is based on the sum of the RGB pixel values ​​raised to the power of gamma, where gamma is between 1.8 and 2.3, e.g., 2.2. For example, pixel 302 in Figure 3, with RGB values ​​of 127, 141, 141, would be (127 2.2 +141 2.2 +141 2.2 The first pixel 302A in the temporal sequence, which has RGB values ​​of 147, 147, 0, is represented by a color power value of ½×(147 2.2 +147 2.2 +0 2.2) = 29420. The second of the temporally consecutive pixels, 302B, is represented by a color power value of 103042. As shown in Table 1, by comparing the energies, it is concluded that it would be more efficient to replace pixel 302 (color power value of 149480) with the pair of temporally consecutive pixels (302A and 302B) combined (total color power value of 132462, therefore lower).

[0043] [Table 1]

[0044] In another embodiment using frame skipping, the second constraint related to energy consumption is verified by comparing the color power values ​​of the original pixel and the pixel skipped by the determined pair of replacement pixels (302A and 302B as described above).

[0045] In another embodiment using frame averaging, the second constraint related to energy consumption is verified by comparing the color power values ​​of the two pixels averaged by the pair of replacement pixels (302A and 302B as described above) determined for the averaged pixel.

[0046] The colors presented to a human observer are defined by the display response to an RGB triplet, e.g., sRGB or BT-709, or the response defined in any other color space implemented in the display instance or model, with given parameter adjustments (brightness, contrast, color temperature, etc.). A color space {C} is selected and color arithmetic operations are performed. A display whose colors are represented in {C} has a color gamut {G} that represents the complete subset of colors the display can render. For an image or video, each pixel color pulse C IN is C A Subpulse and C B The resulting visual appearance is C A and C B The combination of time and INGives color perception, C A and C B The energy consumption of C IN lower than the original consumption of

[0047] The iterations in step 420 generate a set of candidate pairs of temporally alternating complementary colors {C AB} is created. From this set, a preferred candidate pair (e.g., the pair with the lowest energy consumption) can be selected for a given input, thus creating an association between the input color and pairs of temporally alternating complementary colors. In at least one embodiment, this association is stored in a lookup table. This prevents the display device from having to perform all of the iterations of step 420 again for each image, allowing for faster implementation. Thus, at least one embodiment includes steps 410, 420, 430, for example, being iterated over all possible colors in the color gamut, thus creating the lookup table, while another embodiment only includes steps 410, 430, and 440 being iterated over all pixel pairs in the input image, resulting in a modified image that consumes less energy to display. Exemplary embodiments are shown in FIGS. 10 and 11.

[0048] 5 shows an example of a process for establishing candidate pairs of alternating complementary colors according to the first embodiment. Process 500 may be performed, for example, by processor 101 of device 100 of FIG. 1 and corresponds to step 420 of FIG. 4. This process begins with an input color triplet R IN G IN B IN In step 510, the processor operates on an input pixel p whose color is represented by the energy consumption P of the input color triplet according to the selected color power consumption model. INThe power consumption model is display dependent. For OLED displays, the power consumption model may follow the color model, such as in the RGBW case where a white LED supports an RGB LED per physical pixel. The color model for RGBW displays is given in Murdoch et al., "Perfecting the color reproduction of RGBW OLED," proc. 30th International Congress of Imaging Science. This model can be extended with appropriate parameters to represent the power per pixel. In step 520, the processor determines the power consumption of the R color in the gamut {G} of the color space {C}. IN G IN B IN Color point C corresponding to the triplet IN The color space of this diagram is selected from among a color space, a standard color space, or a human visual color space. In step 530, the processor samples a color space {C} within the color gamut {G} to determine a set of candidate colors {C A Different criteria can be used to determine the sampling space. In at least one embodiment, a maximum color distance criterion is used to determine the sampling space C IN In various embodiments, the candidate set is limited to saturated colors, or grayscale colors (i.e., part of the gray ramp), or C IN or a combination of these colors. In at least one embodiment, the entire color gamut space is searched, and thus the candidate set is the full set of possible values. In at least another embodiment, a subset of the color gamut space is selected, for example, using a smaller color resolution. In another embodiment, several randomly selected candidates are used.

[0049] The process then repeats through steps 540-590 to select a set of candidate colors {C A In step 540, the processor constructs a set of candidate color pairs for each color in the selected color CA , the second color C of the second pixel of the temporally consecutive pixels B but, C B =2.C IN -C A As it is, Or alternatively, 2.C. IN =(C A +C B ) is determined.

[0050] This results in color C A and C B The combination of consecutive pixels in time is color C IN This is because the pixel is guaranteed to look the same as the pixel in C. IN is the color C in color space {C} A and C B In step 550, the processor B is contained in the gamut {G} of the color space {C}. In fact, if a color is outside the gamut, it cannot be displayed, and as a result, A Subpulse and C B The sub-pulse combination is color C IN If the color is out of gamut, the iteration continues with the selected value C A This is because the correct color pairing is not available for consecutive pixels in time. In this case, given the remaining values ​​in the set, the process stops at C A The iteration resumes at step 540 for the next value of .

[0051] In step 560, the processor calculates the color pair C A ,C B For the corresponding triplet R A G A B A ,R B G B B B Determine.

[0052] In step 570, the processor A , C BThe energy consumption of the pixel combination P AB Since the power consumption model of the display device is not necessarily known, the energy consumption to display a color can be simply approximated by the power gamma by the sum of its RGB values, as shown in Table 1. In embodiments using frame doubling, this is evaluated over a half-period using the selected color power consumption model as follows:

[0053]

number

[0054] In step 580, the processor AB <P IN In fact, candidate pairs of colors of temporally consecutive pixels are considered only if they result in some energy reduction. If not, the candidate pair is discarded, and with the remaining values ​​in the set, the process A The iteration resumes at step 540 for the next value of P. A +P B <2.P IN It can also be formulated as P A , P B , P IN are color C A , C B , C IN In other words, in embodiments using frame doubling, a candidate pair of alternating complementary colors is considered when the sum of the energy of the pair of alternating complementary colors is lower than twice the energy of the input color. In step 590, the processor A ,C B Let us denote the set of candidate pairs {C AB}.

[0055] In step 430, the processor selects candidate pair C A ,C B One of the colors C INIn at least one variation, the processor selects candidate pairs {C AB The position of the candidate pair with the lowest energy consumption in the list of { is identified as follows:

[0056]

number

[0057] As a result, the processor

[0058]

number

[0059] In at least one embodiment, color C IN Correspondence between colors and their best color pair permutation C Axmin ,C Bxmin are stored in the lookup table so that subsequent modifications can be made very efficiently. By providing an input color to the lookup table, the corresponding color pair can be obtained immediately without having to re-execute process 400 of Figure 4 and process 500 of Figure 5.

[0060] In at least one embodiment, a mathematical minimization method, such as least-squares minimization, is used to replace steps 520-590 to determine color C IN Matching and its best color pair permutation C Axmin ,C Bxmin Find out.

[0061] In at least one embodiment, an additional step is added between steps 520 and 530 to generate the triplet R IN G IN B IN Verify that the triplet R spatially belongs to the subset of the image we want to process, e.g., belongs to a region in the image that has the highest ability to mask artifacts. Such a region or mask can be given, for example, by a spatiotemporal just noticeable difference (JND) map, a motion field, a saliency map, etc. IN G IN B IN If does not belong to this region or mask, then this color is not considered for color pair permutation.

[0062] In at least one embodiment, the triplet {R IN G IN B IN} are ordered according to their energy consumption so that they are processed in descending order from most to least consuming. In such a case, a threshold may be defined corresponding to the global energy reduction to be achieved. When this threshold is reached for a given number of processed triplets, the global process stops. Other ordering criteria can be defined, such as determining the RGB combinations that consume the most energy and can be replaced with the greatest effect on the display. The map in color space is then calculated as the replacement power ratio.

[0063]

number

[0064] 6 shows an example of a process for establishing candidate pairs of alternating complementary colors according to the second embodiment in a color space that provides a color transform and an inverse color transform. Process 600 is performed, for example, by processor 101 of device 100 of FIG. 1 and corresponds to step 420 of FIG. 4. The difference from the first embodiment is that the operations are performed in a color space that provides a color transform and an inverse color transform. Examples of such color spaces are the CIE XYZ color space or the OKLab color space used herein.

[0065] When using CIE XYZ, a color is defined by a triplet of coordinates in XYZ space, where Y is luminance, Z is quasi-equal to B, and X is a blend of the three RGB curves. A compressing / expanding transformation is conventionally applied (also called "applying gamma" or "companding"), i.e., raising each RGB value to the power of a gamma value, which depends on the display (not shown). A forward color transform FT(RGB) then makes it possible to calculate the XYZ coordinates from the RGB values, and then an inverse operation is applied by an inverse color transform IT(XYZ), i.e., it makes it possible to calculate the RGB values ​​from the XYZ coordinates, also followed by an inverse companding operation.

[0066] This process uses the input color triplet R IN G IN B IN In step 610, the processor operates on an input pixel p whose color is represented by the energy consumption P of the input color triplet according to the selected color power consumption model. IN In step 620, the processor determines R in the color gamut {G} of the color space {C}. IN G IN B IN Color point C corresponding to the triplet IN In step 630, the processor performs a forward color transformation from RGB to XYZ as R IN G IN B IN Applying it to a triplet, we get the X in XYZ space. IN Y IN Z INThe forward color transform is display dependent. An example of a color transform is the sRGBtoXYZ matrix, where the white point corresponds to the CIE standard illuminant D65. In step 640, the processor samples the color space {C} within the color gamut {G} to obtain a set of candidate colors {C A}. Determine the coordinate X of the candidate color. A Y A Z A may verify certain conditions. In a first variant, the selected color C A is part of the gray ramp (i.e., a grayscale color), A =Y A =Z A In a second variant, the selected color C A X is a saturated color A Coordinate or Y A Coordinate or Z A In the third variant, one of the coordinates is set to zero. A has no or minimal variation in luminance compared to the input color and is therefore close to isoluminant. A =Y IN In a fourth variant, Y is adjusted to minimize the variation in luminance compared to the input color (quasi-isoluminance). A =α.Y IN and α∈[0.8,1.2].

[0067] The process then iterates through steps 650-690 to generate a set of candidate colors for temporally consecutive pixels {C A In step 650, the processor constructs a set of candidate color pairs for each color in the selected color C A For a color, the color is IN Regarding C A symmetrical with respect to the coordinate X B Y B Z B Second color C B In the chosen color space, this is easily done using the following calculation:

[0068]

number

[0069] In step 655, the processor B is contained in the gamut {G} of the color space {C}. In fact, if a color is outside the gamut, it cannot be displayed, and as a result, A Subpulse and C B The sub-pulse combination is color C IN If the color is out of gamut, the iteration continues with the selected value C A This is because the correct color pairing is not available for consecutive pixels in time. In this case, given the remaining values ​​in the set, the process stops at C A The iteration resumes at step 650 for the next value of .

[0070] In step 660, the processor calculates the inverse transform R A G A B A =IT(X A Y A Z A ) and R B G B B B =IT(X B Y B Z B ) by applying color pairs C A ,C B For the corresponding triplet R A G A B A , R B G B B B In step 670, the processor determines color C A , C B In embodiments using frame doubling, this is evaluated over each half period using the selected color power consumption model as follows:

[0071]

number

[0072] In step 680, the processor AB <P IN In fact, candidate pairs of colors of temporally consecutive pixels are considered only if they result in some energy reduction. If not, the candidate pair is discarded, and with the remaining values ​​in the set, the process A The iteration resumes at step 650 for the next value of P. In embodiments using frame doubling, the test at step 680 is A +P B <2.P IN It can also be formulated as P A , P B , P IN are color C A , C B , C IN In other words, in embodiments using frame doubling, a candidate pair of alternating complementary colors is considered when the sum of the energy of the pair of alternating complementary colors is lower than twice the energy of the input color.

[0073] In step 690, the processor selects candidate pair C. A ,C B Let us denote the set of candidate pairs {C AB}.

[0074] In step 430, the processor selects candidate pair C A ,C B One of the colors C IN 5 as a pair of colors of temporally consecutive pixels that will replace the pixels of . The variations and embodiments described in the context of FIG. 5 for selecting a pair from a set of candidate pairs also apply here.

[0075] The third embodiment is based on the same process as described in FIG. 6, except that the color space selected is a uniform color space (e.g., CIELab, IPT, OKLab, or other). Uniform color spaces are constructed so that the same geometric distance (point-to-point distance) reflects the same amount of perceptual color difference in any color space. For the following discussion, the CIELab color space is chosen. In this color space, color is represented by three values: L for perceived lightness, a and b for the four inherent opponent colors of human vision (red, green, blue, and yellow). In this embodiment, some modifications are required to adapt some of the steps of process 600 to a uniform color space. In step 630, the processor performs a forward color transformation from RGB to CIELab using R IN G IN B IN Applying it to the triplet, we get L in CIELab space. IN a IN b IN The forward color transformation conventionally involves a gamma operation. In step 640, the processor samples the color space {C} within the color gamut {G} to obtain a set of candidate colors for temporally consecutive pixels {C A} is determined. The coordinates of the candidate color L A a A b A may verify certain conditions. In a first variant, the selected color C A is part of the gray ramp (i.e., a grayscale color), A =b A In the second variant, L A a A b A and its cylindrical version L A C A h A The same luminance as L IN a IN b IN In a third variant, the selected color C is set to the hue of the selected color C, but with maximum saturation. A L is chosen so that there is no or minimal variation in luminance compared to the input color and is therefore close to isoluminance. A =L INIn a fourth variant, L is adjusted to minimize the variation in luminance compared to the input color (quasi-isoluminance). A =α.L IN and α∈[0.8,1.2]. In step 650, the processor A Regarding the color, C IN Regarding C A symmetrical with respect to the coordinate L B a B b B Second color C B In a uniform color space, this is easily done using the following calculation:

[0076]

number

[0077] In step 660, the processor calculates the inverse transform R A G A B A =IT(X A Y A Z A ) and R B G B B B =IT(X B Y B Z B ) by applying color pairs C A ,C B For the corresponding triplet R A G A B A , R B G B B B The other steps of process 600 are identical. The variations and embodiments described in the context of Figure 5 for selecting one pair from a set of candidate pairs also apply to the third embodiment.

[0078] 7 shows an example of pixel replacement by temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame doubling. In this embodiment, the display frequency is doubled compared to the original image frequency. For example, if a sequence of images is intended to be displayed at 50 Hz, the display frequency is doubled to display a sequence of modified images at 100 Hz, allowing the original pixels to be replaced by alternating complementary color pixels, thus reducing the energy consumption of the display while maintaining the quality of experience.

[0079] In the figure, line 700 represents the temporal sequence of the original (i.e., unmodified) image to be displayed (in this case, including three images 701, 702, and 703). These images are displayed for periods t1, t2, and t3, respectively. In the example of a 50 Hz display frequency, the duration of these periods is 20 ms. To simplify the drawing, images 701, 702, and 703 each consist of two rows of three pixels. The pixels are represented here by numbered blocks. The numbers identify the pixel colors with reference to the colors introduced in FIG. 3. For example, the first line of image 701 consists of pixels 301, 302, and 303. Thus, the first pixel 301 of this line is brown with RGB values ​​of 147, 107, 0, the second pixel 302 is dark navy blue with RGB values ​​of 127, 141, 141, and the third pixel 303 is dark magenta with RGB values ​​of 82, 108, 160. In the second line, the three pixels are dark navy blue, brown, and dark magenta, respectively.

[0080] Line 710 represents a temporal sequence of displayed modified images, including images 711, 712, 713, 714, and 715. Each of these images is displayed for half the duration compared to line 700, corresponding to the frequency doubling. Thus, the original 50 Hz display frequency of line 700 is doubled to 100 Hz in line 710, and periods t1A, t1B, t2A, t2B, and t3A are 10 ms long. Twice as many images are displayed in line 710 compared to line 700. This allows for the insertion of intermediate images to introduce alternating complementary color pixels, thus reducing energy consumption when displaying the images.

[0081] For each pixel in original image 701, a color pair is determined as described above in connection with FIG. 5 or 6. This color pair is used to define a first pixel of the first color of the color pair in image 711 and a second pixel of the second color in image 712, and these two pixels are displayed consecutively at twice the frequency of the expected display of pixel 301. For example, brown pixel 301 in image 701 is replaced by green pixel 301A in image 711 and red pixel 301B in image 712. These red- and green-substituted pixels are displayed half the time as the original brown pixel. As described above, thanks to the human visual system, these pixels are perceived by a human observer as having brown pixel 301, while requiring less energy to display them.

[0082] A frame doubling scheme for pixel replacement with alternating complementary color pixels when applied to a sequence of images, in other words, video, is presented in Figure 7. However, the same principles apply when displaying a single still image (e.g., a text editing application on a computer screen content, a composition screen on a tablet, an email application on a smartphone, a still image on an advertising screen, etc.). In this case, Figure 7 is limited to elements relating to image 701 (the single image being displayed) and images 711 and 712. Instead of displaying image 701 at a given frequency as in the past, images 711 and 712 are displayed alternately at twice the frequency.

[0083] FIG. 8 shows an example of pixel replacement with temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame skipping. This method can be used, for example, when the display frequency cannot be doubled. It is based on skipping one of two images in a sequence of images, deriving a color pair from the colors of the remaining images, and then replacing the original sequence of images with a sequence of images that successively includes an image with a pixel of the first color of the color pair and an image with a pixel of the second color of the color pair. Another way to describe this embodiment is to replace a pair of temporally consecutive pixels at the same location in two consecutive images with another pair of temporally consecutive pixels at the same location in two consecutive images, where the second pair is perceptually similar to the first pair but requires less energy to display. The color of the new pair is selected based on the color of the first pixel of the first pair.

[0084] This is shown in the figure, where line 800 indicates the temporal sequence of original images 801, 802, 803, 804, and 805. Line 810 indicates the temporal succession of displayed energy-reduced images 811, 812, 813, 814, and 815. Images 811 and 812 are obtained from original image 801 as described above by determining color pairs that are perceptually similar to the colors of original image 801 but require reduced energy to display. For example, color 301 of first pixel p1 of first line in image 801 is processed as described above to determine color pair 301A, 301B. These colors are used in first image 811 and second image 812 of the sequence, respectively. Images 802 and 804 are discarded so that their original pixels (e.g., first pixel p2 of the first line) are not considered at all in the displayed image.

[0085] In this embodiment, the constraints related to reducing the energy required to display the color c A and c B Temporally consecutive pixels p A and p BThe energy required to display the original color c IN In fact, in such an embodiment, the energy required to display the original pixel p and the skipped pixel is less than the energy required to display the temporally consecutive pixel p. A and p B replaces two original pixels that are less than half the size of the original pixel but of the same duration (the original pixel is p1, and the skipped pixel is p2).

[0086] FIG. 9 illustrates an example of pixel replacement with temporally consecutive pixels of alternating complementary colors according to an embodiment based on frame averaging. This method can be used, for example, when the display frequency cannot be doubled. It is based on averaging the pixel colors of two consecutive frames and replacing these frames with two frames containing alternating complementary colors determined based on the averaged colors. Another way to describe this embodiment is to replace a pair of temporally consecutive pixels at the same location in two consecutive images with another pair of temporally consecutive pixels at the same location in two consecutive images, where the second pair is perceptually similar to the first pair but requires less energy to display. The colors of the new pair are selected based on the average of the colors of the first pair. Compared to the frame-skipping technique described in connection with FIG. 8, this technique allows for consideration of all pixels of all images in the original sequence of images. As a result, modified images 901 and 902 depend on images 801 and 802 of FIG. 8.

[0087] Line 900 shows the temporal sequence of energy-reduced images 901, 902, 903, 904, 905 obtained by processing images 801, 802, 803, 804, 805 of Figure 8. In this method, the processor first averages the colors of the pixels from images 801 and 802 of Figure 8. For example, a first pixel p1 (of color 301) of a first line in a first image is averaged with a first pixel p2 (of color 302) of a first line in a second image. In a preferred embodiment, the averaging is performed by converting the color value RGB of the pixel in a uniform color space, e.g., CIELab, calculating the average of the two pixels, and converting the result back to RGB color space. This results in the average color value 312 of this first pixel of the first line. The alternating complementary colors c of the respective values ​​312A and 312B are A and c B The pair of colors p is determined based on the average color value 312 using the same method as described above. These colors are calculated for the first pixel p of the first line of the first image 901. A and the first image p of the first line of the second image 902 B Used for.

[0088] In this embodiment, the constraint related to reducing the energy required to display the color c A and c B Temporally consecutive pixels p A and p B This is verified here by determining that the energy required to display the pixel p1 is less than the energy required to display the original pair of pixels p1 and p2. Indeed, in such an embodiment, the energy required to display the pixel p1 is less than the energy required to display the original pair of pixels p2. A and p B replaces two original pixels that are less than half the size of the original pixel but of the same duration.

[0089] Both frame skipping and frame averaging methods may result in the loss of some of the original signal, which may affect the visual quality of the modified image because the spatial or temporal resolution is affected. To improve the quality of the signal, some spatial or temporal filtering may be used.

[0090] One improvement is to apply the alternating complementary color process only to uniform regions, leaving images where high spatial frequencies (i.e., edges) are present unchanged. In at least one embodiment, for each pair of consecutive images, the processor detects edges using a contour filter (e.g., Canny edge detector, Gaussian difference), optionally dilates these contours, and saves the contour zones as mask (M1, M2). Two parameters are required: the dilation size and the threshold above / below which the mask is binarized. Mask (M1, M2) is then inverted to save the non-contour zones.

[0091]

number

[0092]

number

[0093]

number

[0094]

number

[0095]

number

[0096] As a result, ACC is only applied to the more homogeneous regions. The temporal information from (C1, C2) is preserved and transferred directly to the (C3, C4) subframes based on high spatial frequencies.

[0097] The above enhancement techniques can be applied in three temporal modes, including frequency doubling, frame skipping, and frame averaging.

[0098] Prediction of regions of interest (e.g., using eye tracking, attention modeling, meta-databases, etc.) can also be used to define which regions are retained as original regions and which regions the ACC process is applied to, as shown in Table 2.

[0099] [Table 2]

[0100] 10 illustrates an example process for generating a lookup table of alternating complementary colors, according to some embodiments, and an example process for modifying an image using the lookup table, according to some embodiments, which are implemented based on the embodiments described above.

[0101] Process 1000 aims to generate a lookup table of alternating complementary colors. Step 1010 is repeated multiple times. In at least one embodiment, the iteration is performed over all possible colors in the color gamut. In at least one other embodiment, the iteration is performed only over a subsampled color space. In at least one other embodiment, the iteration is performed over all colors in a given image set. Other embodiments may use other color subsets. This includes steps 1020-1050. In step 1020, an input color is obtained. In step 1030, a set of pairs of alternating complementary colors corresponding to the input color is determined, for example, using one of the embodiments described above. In step 1040, one of the pairs in the set is selected, for example, the pair that provides the best energy performance. In step 1050, the associations between the input colors and the pairs of alternating complementary colors are stored in a lookup table. At the end of process 1000, the lookup table contains a set of associations between input colors and pairs of alternating complementary colors.

[0102] Process 1001 aims to modify an image using a lookup table. In step 1060, an input image is obtained. In step 1070, a pixel is selected, and in step 1080, a pair of temporally alternating complementary colors corresponding to the pixel's color is obtained from the lookup table. In step 1090, the colors of pairs of temporally consecutive pixels are set to the alternating complementary color pair. Steps 1070, 1080, and 1090 are then repeated for all pairs of adjacent pixels.

[0103] FIG. 11 shows two example arrangements for an alternating complementary color process, according to some embodiments. In at least one embodiment, device 1101 is a display device such as the display device described in FIG. 1. In this case, processor 101 of device 1101 is configured to acquire an input image or video 1100 and display it on display unit 103 of FIG. 1 after processing by the ACC process as described above. In other words, processor 101 of device 1101 is configured to acquire an input image or video 1100 and determine a modified image or video to be displayed using spatially alternating complementary colors by using a lookup table (LUT) that results in an image that reduces energy consumption of the display device compared to displaying the original input image. The lookup table may be acquired from a data provider via a communications network and / or from the device's internal memory. Image or video 1100 may be acquired from a data provider via a communications network, from the device's internal memory, or stored after capture by an input unit, for example. Typical examples of device 1101 are smartphones, tablets, laptops, external monitors, head-mounted displays, television sets, video projectors, computer screens, vehicles (e.g., control and / or entertainment systems for cars, airplanes, boats, etc.), advertising display panels, medical monitors, etc. However, any device or arrangement of devices providing similar functionality may be used as display device 1101 and still conform to the principles of the present disclosure.

[0104] In at least one embodiment, device 1102 does not include a display unit, but prepares data for display so that another device 1103, such as a screen, can perform the display. In this case, a processor in the device performs the ACC process described herein to generate a new image or video 1110 that is perceptually similar to the original video but requires less energy when displayed. This modified video is then provided to display device 1103 for presentation to a human viewer. Examples of such devices 1102 are set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0105] Light generation in display devices such as mobile phones and televisions is expensive. Reducing the amount of light generated is desirable, as it helps reduce the amount of energy required to operate the display. The benefits of this are twofold: reduced pressure on the climate and increased battery life in mobile devices. Compared to other methods that aim to reduce energy consumption for the same reasons, the proposed method, by its structure, ensures that the light emitted by each pixel is generated by a combination of two light pulses that, on average, minimizes energy consumption. This means that it is more versatile and therefore more efficient than acting on a single light pulse at a single pixel.

[0106] However, the introduction of the above techniques may result in the perception of flicker, which can reduce the quality of experience. Flicker is perceived when a light source emits unstably or when the luminance changes rapidly. This is the case for displays using the above-mentioned alternating complementary color processing method. A first technique for reducing flicker is to apply an equal luminance constraint when selecting complementary colors. Furthermore, because temporal contrast sensitivity decreases with increasing frequency, this also allows for attenuation of the perceived flicker. However, some perception may remain, which can be further attenuated through the embodiments described below.

[0107] In at least one embodiment, flicker is reduced by relying on spatial averaging (perceptual averaging) of colors at the eye level. This can be done by processing colors using blocks (i.e., regions of an image that cover an area) with inverted polarity. The concept of opposite polarity is referred to herein as "C" in pairs of alternating complementary colors. B thenC A " and the opposite "C A thenC B " should be understood as choosing "C". Flicker will be naturally attenuated by the spatial averaging performed by the human eye between the opposite polarities of the blocks. The time average of the color will be the same as the energy will be reduced, but the flicker effect will be reduced. Therefore, C A and C B Depending on the order of selection between (in other words, "C A thenC B " or "C B thenC A "), the two colors do not play exactly the same role in a complementary pair. Using different selection orders for adjacent blocks helps to reshape the flicker. The selection order is, for example, C for the first block of pixels in the first image. A and the complementary second image of this first block is C B and using C B and in the complementary second image of this adjacent block, C A In this case, flicker will be reduced due to the visual spatial averaging between adjacent blocks with opposite polarity.

[0108] 12 shows examples of pixel replacement with alternating complementary color pixels according to variant embodiments based on frequency doubling and using opposite polarity for adjacent blocks. While these examples are based on frequency doubling embodiments, the principle of opposite polarity for adjacent blocks applies equally to frame skipping and frame averaging embodiments, with the difference that intermediate images exist between images 1211 and 1212 and after image 1212 (such as images 1222 and 1224). In frame skipping embodiments, the pixels of these additional images are skipped as described above. In frame averaging embodiments, the pixels of these additional images are averaged with the pixels of images 1211 and 1212 as described above. Furthermore, the principle of opposite polarity can be used for different sizes, for example, from 1×1 pixels to N×M pixels (where N or M is defined and fixed or variable, and N or M is up to columns or rows).

[0109] Similar to FIG. 7, line 1210 represents a series of images 1211 and 1212, each consisting of two lines of three pixels to be displayed and numbers 301, 302, 303, 304, and 305 identifying the colors of the different pixels. Similar to FIG. 7, line 1220 represents an exemplary embodiment using frequency doubling with alternating complementary colors but without opposite polarity, with images 1221 and 1223 using the first color of the color pair (identified by suffix A in the figure) and images 1222 and 1224 using the second color of the color pair (identified by suffix B in the figure). These first two lines are presented here for comparison with lines 1230-1260, which represent a different, alternative embodiment using opposite polarity for various block sizes and shapes. For each of these lines 1230-1260, four images are displayed in place of the series of images 1211 and 1212. The modifications relative to the embodiment using the frequency-doubling ACC method without the reverse polarity of line 1220 are represented by highlighting the numbers in italics.

[0110] In a first variant shown in line 1230, the block size is 1x1 pixel, resulting in opposite polarity relative to each other pixel. Furthermore, each line starts with the opposite polarity relative to the previous line, resulting in a checkerboard pattern of opposite polarity. This is the variant that provides the best results, as the patterning is at the finest size and is the same in both the vertical and horizontal directions.

[0111] In another variation, shown at line 1240, the block size is one column of the image. In another variation, shown at line 1250, the block size is one row of the image.

[0112] In another alternative embodiment, the block size is non-square. In another alternative embodiment, the block size is non-rectangular. This is shown in line 1260 by using an "L-shape", although any other shape may be used.

[0113] In another variant, the block size depends on the content of the image, with homogeneous regions containing small blocks (eg, as small as 1x1 pixel) and more structured regions containing larger blocks.

[0114] FIG. 13 shows an example of blocks for pixel replacement with alternating complementary color pixels according to a variant embodiment using opposite polarity for adjacent blocks with unequal block sizes. In this case, image 1300 is decomposed into smaller blocks at the periphery, where peripheral vision may occur, making them more susceptible to flicker. More specifically, blocks 1310 and 1311 at the left and right boundaries of the image use block sizes of 1×1 pixels with opposite polarity (thus similar to the first variant of FIG. 11 ), while blocks 1320 and 1321 are 2×2 in size, blocks 1330 and 1331 are 4×4 in size, and the block size in the central region 1340 of the image is 8×8. While this diagram shows block size variations horizontally, the same principle applies vertically, resulting in a similar diagram rotated 90°. The two block size variations can be combined to result in concentric blocks of different sizes.

[0115] FIG. 14 shows an example of a process for reducing energy consumption of pixels of an image using alternating complementary colors of opposite polarity, according to some embodiments. Process 1400 may be performed, for example, by processor 101 of device 100 of FIG. 1. Process 1400 takes as input an image including a set of blocks defined according to one of the embodiments or variations identified above. Step 1410 may be repeated for all blocks that make up the input image. This step 1410 includes steps 1420 and 1430. In step 1420, the processor obtains a block of the image. Step 1430 is then repeated for the pixels of the block.

[0116] In step 1440, color c IN In step 1450, the processor obtains pixel p having color c IN The alternating complementary color pair c corresponding to A ,c B This is done using one of the embodiments described above, for example using a look-up table that associates colors with corresponding pairs of alternating complementary colors.

[0117] In step 1460, the polarity information is checked. The polarity is represented by information common to the entire process 1400, and this information indicates whether the polarity is A or B at any time. The information can be implemented by a simple Boolean variable (e.g., true means the polarity is A, false means the polarity is B) or any other type of variable (e.g., enumeration, integer, string, etc.). The polarity information is set to a default value when the process 1400 starts (e.g., A as shown in lines 1230-1260 of FIG. 12). If the result of this check is that the polarity information is equal to A, the processor executes step 1470, where it divides pixel p into a pair of temporally consecutive pixels p A and p B Substitute with p A Change the color of A Set it to p B Change the color of B Otherwise, if the polarity information is equal to B, the processor executes step 1480 and sets pixel p to a pair of temporally consecutive pixels p B and p A Substitute with p B Change the color of B Set it to p A Change the color of A After replacing all pixels in the block with consecutive pixels of alternating complementary colors, the processor, in step 1490, reverses the polarity before processing the next block. In other words, if the polarity information was A, it becomes B, and vice versa. If the implementation of the polarity information is based on Boolean values, this is simply done by inverting the value of the variable (i.e., taking the two's complement). This process is repeated again for the next block until all blocks have been processed.

[0118] Once all the blocks have been processed, the processor can display the successive images using one of the temporal techniques selected between frequency doubling, frame skipping, or frame averaging, which makes it possible to provide opposite polarity to alternating complementary colors, thus reducing the perception of flicker by a human observer, as illustrated in the variant embodiment shown in Figure 12.

[0119] Thanks to the above-described embodiments, the selection of alternating complementary colors becomes more efficient for a given display, reduces flicker and artifacts, and therefore the principle of consecutive pixels of alternating complementary colors is applicable to more use cases and system configurations.

[0120] In addition, reducing flicker perception may allow for stronger pixel modulation, possibly with some luminance modulation. This means that more freedom may be provided in determining the color pair set, resulting in a larger set of pairs including color pairs that further reduce energy consumption. Selecting such pairs may allow for further reduction in energy consumption.

[0121] Although different embodiments have been described separately, the embodiments may be combined in any manner while respecting the principles of the present disclosure.

[0122] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," or "in one implementation" or "in an implementation," as well as other variations thereof, appearing in various places throughout this specification are not necessarily all referring to the same embodiment.

[0123] Additionally, this application or its claims may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0124] Also, in this application or its claims, various pieces of information may be referred to as "obtaining." Obtaining, like "accessing," is intended to be a broad term. Obtaining information may include, for example, one or more of receiving information, accessing information, or retrieving information (e.g., from memory or optical media storage). Furthermore, "obtaining" typically involves, in some manner, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0125] Furthermore, the terms "image" and "frame" are used interchangeably herein and are both used to refer to a set of pixels arranged, for example, in a two-dimensional array. A sequence of images or frames arranged in a temporal order is conventionally referred to as a "video."

[0126] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be applied regardless of the number of items listed, as will be readily apparent to one of ordinary skill in the art.

Claims

1. 1. A method comprising iterating over pixels of an image of a video, the iteration comprising, for the pixels: obtaining pairs of alternating complementary colors based on the colors of the pixels; replacing the pixels of the image of the video by pairs of temporally consecutive pixels based on polarity information, wherein if the polarity information has a first value, setting the color of the first temporally consecutive pixels to a first color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to a second color of the alternating complementary color pair; otherwise, setting the color of the first temporally consecutive pixels to the second color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to the first color of the alternating complementary color pair; inverting the polarity information of a next pixel.

2. 2. The method of claim 1, wherein the alternating complementary color pair is obtained from a lookup table based on the color of the pixel, the lookup table including a set of associations between colors and alternating complementary color pairs, an average color of the alternating complementary color pair being perceptually identical to the color, and a sum of the energies of the alternating complementary color pair being less than twice the energy of the color.

3. The method of claim 2 , wherein the lookup table is constructed by iterating over multiple possible values ​​of an input color.

4. 4. The method of claim 3, wherein the lookup table is constructed by selecting color values ​​of a first color of the alternating complementary color pair according to a selection criterion and determining a second color of the alternating complementary color pair based on the selected first color value, such that an average color of the alternating complementary color pair is perceptually similar to an input color and the sum of the energies of the alternating complementary color pair is less than twice the energy of the input color.

5. 5. The method of claim 4, wherein the alternating complementary color pairs are selected from a set of alternating complementary color pairs, the set being obtained by multiple iterations over a set of color values ​​of the first colors of the pairs, and further comprising selecting the pair having the lowest energy.

6. The method of claim 5 , wherein the selection criteria is based on a maximum color distance from the input color.

7. The method of claim 5 , wherein the selection criterion is that the first color is a saturated color.

8. The method of claim 5 , wherein the selection criterion is that the first color is a grayscale color.

9. The method of claim 5 , wherein the selection criterion is that the first color has the same luminance as the input color.

10. 10. The method of claim 4, wherein the color space is an XYZ color space and the second color of the alternating complementary color pair is selected to be symmetrical to the first color of the alternating complementary color pair with respect to the input color.

11. 10. The method of claim 4, wherein the color space is a uniform color space and the second color of the alternating complementary color pair is selected to be symmetrical to the first color of the alternating complementary color pair with respect to the input color.

12. The method according to any one of claims 1 to 11, further comprising an additional step, prior to said replacing step, in which said pixel is verified to be spatially contained in a region of interest selected according to a criterion.

13. The method of claim 12 , wherein the selection criteria is based on a spatiotemporal least noticeable difference map.

14. The method of claim 12 , wherein the selection criteria is based on a motion field.

15. The method of claim 12 , wherein the selection criteria is based on a saliency map.

16. The method of claim 12 , wherein the selection criteria is based on eye tracking.

17. The method of claim 12 , wherein the selection criteria is based on attention modeling.

18. The method of claim 12 , wherein the selection criteria is a metadata database.

19. 19. The method of any one of claims 1 to 18, further comprising displaying the temporally successive pixels using frequency doubling so that the temporally successive pixels have half the duration compared to pixels of the image of the video.

20. The method of any one of claims 1 to 18, wherein the temporally consecutive pixels are displayed using frame skipping.

21. 19. The method of any one of claims 1 to 18, wherein the temporally consecutive pixels are displayed using frame averaging, and the pairs of alternating complementary colors are selected based on an average color between a color of a pixel of a first frame and a color of a pixel of a second frame.

22. 1. A device comprising one or more processors configured to iterate over pixels of an image of a video, the iteration comprising: obtaining pairs of alternating complementary colors based on the colors of the pixels; replacing the pixels of the image of the video by pairs of temporally consecutive pixels based on polarity information, wherein if the polarity information has a first value, setting the color of the first temporally consecutive pixels to a first color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to a second color of the alternating complementary color pair; otherwise, setting the color of the first temporally consecutive pixels to the second color of the alternating complementary color pair and setting the color of the second temporally consecutive pixels to the first color of the alternating complementary color pair; and reversing the polarity information of a next pixel.

23. 23. The device of claim 22, wherein the alternating complementary color pairs are obtained from a lookup table based on the color of the pixel, the lookup table including a set of associations between colors and alternating complementary color pairs, an average color of the alternating complementary color pairs being perceptually identical to the color, and a sum of the energies of the alternating complementary color pairs being less than twice the energy of the color.

24. 24. The device of claim 23, wherein the lookup table is constructed by iterating over multiple possible values ​​of an input color.

25. 25. The device of claim 24, wherein the lookup table is constructed by selecting color values ​​of a first color of the alternating complementary color pair according to a selection criterion and determining a second color of the alternating complementary color pair based on the selected first color value, such that an average color of the alternating complementary color pair is perceptually similar to an input color and the sum of the energies of the alternating complementary color pair is less than twice the energy of the input color.

26. 26. The device of claim 25, wherein the pairs of alternating complementary colors are selected from a set of pairs of alternating complementary colors, the set being obtained by multiple iterations over a set of color values ​​for the first colors of the pairs, and further comprising selecting the pair having the lowest energy.

27. 27. The device of claim 26, wherein the selection criteria is based on a maximum color distance from the input color.

28. 27. The device of claim 26, wherein the selection criterion is that the first color is a saturated color.

29. 27. The device of claim 26, wherein the selection criterion is that the first color is a grayscale color.

30. 27. The device of claim 26, wherein the selection criterion is that the first color has the same luminance as the input color.

31. 31. A device according to any one of claims 22 to 30, wherein the colour space is the XYZ colour space and the second colour of the alternating complementary pair is selected to be symmetrical to the first colour of the alternating complementary pair with respect to the input colour.

32. 31. A device according to any one of claims 22 to 30, wherein the colour space is a uniform colour space and the second colour of the alternating complementary pair is selected to be symmetrical to the first colour of the alternating complementary pair with respect to the input colour.

33. A device according to any one of claims 22 to 32, further comprising an additional step, prior to said replacing step, in which it is verified that said pixel is spatially contained in a region of interest selected according to a criterion.

34. The device of claim 33 , wherein the selection criteria is based on a spatiotemporal least noticeable difference map.

35. The device of claim 33 , wherein the selection criteria is based on a motion field.

36. The device of claim 33 , wherein the selection criteria is based on a saliency map.

37. The device of claim 33 , wherein the selection criteria is based on eye tracking.

38. The device of claim 33 , wherein the selection criteria is based on attention modeling.

39. 34. The device of claim 33, wherein the selection criteria is a metadata database.

40. 40. A device according to any one of claims 22 to 39, further comprising displaying the temporally successive pixels using frequency doubling so that the temporally successive pixels have half the duration compared to pixels of the image of the video.

41. A device according to any one of claims 22 to 39, wherein the temporally consecutive pixels are displayed using frame skipping.

42. 40. A device as claimed in any one of claims 22 to 39, wherein the temporally successive pixels are displayed using frame averaging, and the pairs of alternating complementary colours are selected based on an average colour between the colour of a pixel in a first frame and the colour of a pixel in a second frame.

43. 43. The device of any one of claims 22 to 42, wherein the device is selected from the set comprising a smartphone, a tablet, a laptop, an external monitor, a head mounted display, a television, a video projector, a computer screen, a vehicle control system, a vehicle entertainment system, an advertising display panel, a medical monitor.

44. A computer program comprising program code instructions for performing the method according to any one of claims 1 to 21 when the computer program is executed by a processor.

45. A non-transitory computer readable medium comprising program code instructions for performing the method of any one of claims 1 to 21 when executed by a processor.

Images