Method and apparatus for pixel color replacement in complemental color based video

Abstract

A method and apparatus proposes modifying the color of pixels of a video to reduce the energy required to display the video. This is accomplished by determining a pair of replacement complementary colors for a pair of co-located pixels of successive frames of the video and replacing the colors of the pair of co-located pixels with the pair of replacement complementary colors. The pair of replacement complementary colors is selected to provide energy reduction while ensuring perceptual similarity in order to provide a satisfactory quality of experience. Different embodiments propose different solutions to apply a replacement process to different arrangements of frames and pixels.

Description

[0001] Cross-references to related applications This application claims priority to European Application No. 23306641.4, filed on 29 September 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure is in the field of video content distribution, and more specifically, at least one embodiment relates to reducing the power consumption for displaying video by using complementary colors to replace the colors of a pair of colocalized pixels in successive frames of a video. Background Technology

[0003] Reducing the energy consumption of electronic devices has become a requirement not only for manufacturers but also for minimizing environmental impact and contributing to a sustainable display industry. The increase in display resolution, from SD to HD, then to 4K, and soon to 8K and higher, along with the introduction of high dynamic range imaging, has led to a corresponding increase in the energy requirements of display devices. The sheer number of devices known to have displays (i.e., TVs, mobile phones, tablets, etc.) contradicts the global need to reduce energy consumption. In fact, for consumer electronics devices, whether battery-powered (e.g., smartphones, tablets, head-mounted displays, automotive displays) or non-battery-powered (e.g., televisions, advertising display panels), displays are the most significant source of energy consumption.

[0004] In recent years, various display technologies have been developed. Although modern displays consume energy in a more controlled and efficient manner compared to older displays, they remain the most significant energy source in the video chain.

[0005] For backlit displays, their energy consumption is mainly determined by the intensity of the backlight.

[0006] Organic light-emitting diodes (OLEDs) are an example of display technology that is finding increasing use compared to previous technologies, such as thin-film transistor liquid crystal displays (TFT-LCDs), due to their numerous advantages. Instead of using uniform backlighting, OLED displays, as well as mini-LEDs, consist of individual pixels that directly emit image light. OLED power consumption is therefore highly dependent on the image content, and the power consumption for a given input image can be estimated by considering the value of the displayed image pixels.

[0007] Therefore, the following is interesting: carefully studying energy-sensitive images or videos, i.e., images or videos that require less energy, especially when displayed on OLED displays in consumer electronics devices. Techniques for reducing energy consumption include the use of Spatial Alternating Complementary Color (SACC), as described in, for example, International Patent Application WO2024132682A1, or Temporal Alternating Complementary Color (TACC), as described in, for example, International Patent Application WO2024132680A1. What is common in the techniques described in both applications is that they involve reducing the energy requirements of the display device, with the goal of maintaining, for example, the Quality of Experience (QoE) based on contrast, brightness, temporal smoothness, or color level by utilizing the principle of alternating complementary colors. Summary of the Invention

[0008] Considering the foregoing, the embodiments described below have been designed and introduce a process for modifying video to reduce its energy consumption while maintaining perceptual similarity. This process can be applied to a pair of colocalized pixels in successive frames of a video, and the pair of replacement colors is symmetrical about the average color of the pair of pixels, based on replacing the original colors of the pair of colocalized pixels with a pair of replacement colors. The pair of replacement colors can be selected such that the sum of energy consumed by displaying the pair of pixels with the replacement colors is less than the sum of energy consumed by displaying the pair of pixels with the original colors. The pair of replacement colors can be selected to be perceptually individually similar to or close to the pair of original colors. This process is named Color Replacement for Energy-Reduced Videos, hereinafter abbreviated as CRV.

[0009] The first aspect relates to a method comprising: obtaining a first pair of colors for a pair of pixels, wherein a first pixel of the pair of pixels is included in a first image of a video and a second pixel of the pair of pixels is included in a second image of the video, and wherein the first pixel and the second pixel are co-located in the images; determining an average color between the colors in the first pair of colors; determining a second pair of colors, wherein the colors in the second pair of colors are symmetrical about the average color, and the sum of energy consumed by displaying a pair of pixels having the second pair of colors is less than the sum of energy consumed by displaying a pair of pixels having the first pair of colors; and replacing the first pair of colors of the pair of pixels with the second pair of replacement colors.

[0010] The second aspect relates to a method comprising: obtaining visual content; iterating over successive frames of the visual content, iterating over pixel pairs located in the successive frames, and for a pair of iterated pixels: determining a second pair of colors for the pair of iterated pixels according to the first aspect; modifying the visual content by replacing the colors of the pair of iterated pixels with the second pair of colors; and providing the modified visual content.

[0011] The third aspect relates to an apparatus comprising a processor configured to: obtain a first pair of colors for a pair of pixels, wherein a first pixel of the pair of pixels is included in a first image of a video and a second pixel of the pair of pixels is included in a second image of the video, and wherein the first pixel and the second pixel are co-located in the image; determine an average color between the colors in the first pair of colors; determine a second pair of colors, wherein the colors in the second pair of colors are symmetrical about the average color, and the sum of energy consumed by displaying a pair of pixels having the second pair of colors is less than the sum of energy consumed by displaying a pair of pixels having the first pair of colors; and replace the first pair of colors of the pair of pixels with the second pair of replacement colors.

[0012] The fourth aspect relates to an apparatus comprising a processor configured to: acquire visual content; iterate over successive frames of the visual content, iterate over pairs of pixels located in the successive frames, and for a pair of iterated pixels: determine a second pair of colors for the pair of iterated pixels according to the first aspect; modify the visual content by replacing the colors of the pair of iterated pixels with the second pair of colors; and provide the modified visual content.

[0013] The fifth aspect relates to a computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to the first or second aspect.

[0014] The sixth aspect relates to a non-transitory computer-readable medium that stores executable program instructions to cause a computer to execute the instructions thereby performing the method according to the first or second aspect.

[0015] The above simplified summary of the invention is presented to provide a basic understanding of some aspects of this disclosure. This summary is not a broad overview of the subject matter. It is not intended to identify key / essential elements of the embodiments or to depict the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description provided below. Attached Figure Description

[0016] This disclosure can be better understood by considering the following detailed description taken in conjunction with the accompanying drawings, in which: Figure 1 The diagram illustrates an example block diagram of an architecture that includes means for implementing various aspects and embodiments.

[0017] Figure 2 The illustrations show two examples of the deployment of the CRV process according to an embodiment.

[0018] Figure 3 The illustration shows a first example of a process for modifying a video to reduce its energy consumption, according to an embodiment.

[0019] Figure 4 The illustration shows an example of numerical values ​​for replacing the complementary color of a pair of pixels using the CRV process according to an embodiment.

[0020] Figure 5 The illustration shows a second example of the CRV process according to an embodiment.

[0021] Figure 6 The illustration shows an example of replacing complementary colors according to an embodiment.

[0022] Figure 7 The illustration shows a third example of the CRV process according to an embodiment.

[0023] Figure 8 The illustration shows an example of replacing complementary color selection according to a third example of a CRV-based process in an embodiment.

[0024] Figure 9 The illustration shows a first example of the application of the CRV process on successive images according to an embodiment.

[0025] Figure 10 The illustration shows a second example of the application of the CRV process on successive images according to an embodiment.

[0026] Figure 11 The illustration shows a third example of the application of the CRV process on successive images according to an embodiment.

[0027] Figure 12 The illustration shows an example of a replacement complementary color selection where the energy gain is not maximized according to an embodiment.

[0028] Figure 13 The illustration shows an example of a process for modifying video using the CRV process according to an embodiment.

[0029] Figure 14 The illustration shows an example of a process for generating a correspondence table of alternative complementary colors according to an embodiment.

[0030] It should be understood that the accompanying drawings are used to illustrate examples of various aspects, features, and embodiments according to this disclosure, and are not necessarily the only possible configurations. In the various drawings, similar reference numerals always refer to the same or similar features. Detailed Implementation

[0031] SACC and TACC techniques can be applied frame-by-frame to successive images in a video. However, this frame-by-frame process modifies the color of a pixel using two other complementary colors. Therefore, it does not account for the fact that, unlike in still images, in video, different colors may be displayed at each moment and at each pixel location. In reality, when displaying video, frame-by-frame images may be subject to camera movement or motion, resulting in pixels of very different colors being displayed successively (i.e., at the same pixel location in two successive frames).

[0032] Furthermore, aside from reducing overall energy consumption by alternating complementary colors and conveying the exact same perceived color once fused by the viewer's visual system, the techniques described in the aforementioned application aim to achieve maximum gain without any constraint on alternating complementary colors. As a result, replacing a pixel color with two other colors may lead to the selection of a color that is perceptually very different from the original color.

[0033] The embodiments described herein propose using a pair of replacement colors in successive frames of a video sequence to replace the colors of a pair of pixels located at the same spatial location in the frame. The pair of replacement colors are perceptually individually similar to a pair of original colors (e.g., as close as possible in terms of distance in the color space) and will consume less energy when displayed on the screen.

[0034] In this context, perceptual similarity can be defined as the similarity of the low-level physical properties (contrast, brightness) of stimuli using objective measures (e.g., a set of XYZ measurements performed using a colorimeter or a set of corresponding Lab values) that correspond to the distribution of responses of neurons in the visual cortex.

[0035] Figure 1 The diagram illustrates an example block diagram of an architecture that includes means for implementing various aspects and embodiments. In the described environment, a user interacts with device 100, such as a television, which is connected to a server 180, for example, operated by a content provider. Server 180 delivers multimedia content 190, such as image-based video streams. In a video distribution system, multiple devices 100, 19x interact with multiple content providers and corresponding servers 180, 18x that deliver various multimedia contents 190, 19x. A single content provider may use multiple servers. The devices exchange data via a communication network 150.

[0036] Communication network 150 preferably uses communication standards to provide interoperability between content providers and devices. Such communication standards can be wireless communication standards, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc., to ensure device mobility. Cable, satellite, or terrestrial digital television broadcasting communications and broadband television communications can also be used in communication network 150. Such digital television standards can be based on mature standards, such as DVB, ATSC, etc. Common network standards can also be used, such as those based on Ethernet.

[0037] Device 100 includes processor 101. Processor 101 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor can perform data processing, such as... Figure 2 CRV process 212, 222 Figure 3 Process 300 Figure 5 Process 500 Figure 7 Process 700 Figure 13 Process 1300 or Figure 14 The process is 1400.

[0038] Processor 101 may be coupled to input unit 102, which is configured to convey user interaction. Various types of input and modalities can be used for this purpose. A physical keypad or touch-sensitive surface is a typical example of an input suitable for this use, but voice control can also be used. Additionally, the input unit may include: a digital camera capable of capturing still images or videos in two dimensions; or a more sophisticated sensor capable of determining depth information in addition to images or videos, and therefore capable of capturing a complete 3D representation.

[0039] Processor 101 may be coupled to display unit 103, which is configured to output visual data for display on a screen. Various types of displays can be used for this purpose, such as liquid crystal displays (LCDs) or organic light-emitting diode (OLED) display units. Processor 101 may also be coupled to audio unit 104, which is configured to convert sound data into audio waves via an adaptive transducer (such as, for example, a speaker).

[0040] The processor 101 may be coupled to a communication interface 105, which is configured to exchange data with an external device. Communication preferably uses a wireless communication standard to provide device mobility, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc.

[0041] Processor 101 can access information from and store data in memory 106. Memory 106 may include various types of memory, including random access memory (RAM), read-only memory (ROM), hard disk, subscriber identity module (SIM) card, memory stick, secure digital storage (SD) card, and any other type of memory storage device. In embodiments, processor 101 can access information from and store data in memory that is not physically located on the device (e.g., on a server, home computer, or another device).

[0042] Processor 101 is configured to execute an image energy reduction algorithm that modifies an input image to require less energy when used (e.g., displayed) compared to using the input image. Various techniques have been disclosed to provide this feature.

[0043] The processor 101 may receive power from the power supply 108 and may be configured to distribute and / or control power to other components in the device 100. The power supply may be any suitable device for powering the device. As an example, the power supply may include one or more dry cell battery packs (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0044] Although the figure depicts processor 101 and other components 102 through 108 as separate parts, it will be understood that these components may be integrated together in an electronic package or chip. It will be understood that device 100 may include any sub-combination of the components described herein while remaining consistent with the embodiments described below. Processor 101 may be further coupled to... Figure 1 Other peripheral devices or units not described herein may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, such peripheral devices may include a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, etc. For example, processor 101 may be coupled to a positioning unit configured to locate the device within its environment. The positioning unit may integrate a GPS chipset that provides the longitude and latitude of the device's current location, and may also integrate other motion sensors, such as accelerometers and / or electronic compasses, that provide positioning services.

[0045] At least one example of the embodiments may relate to an apparatus comprising the device as described herein and at least one of the following: (i) an antenna configured to receive a signal including data representing image information; (ii) a band limiter configured to limit the received signal to a frequency band including the data representing image information; and (iii) a display configured to display an image based on the image information.

[0046] At least one example of the embodiments may relate to a device as described herein, wherein the device includes one of a television, a television signal receiver, a set-top box, a gateway device, a mobile device, a cellular phone, a tablet computer, a computer, a laptop computer, or other electronic devices.

[0047] Figure 2 The illustrations show two examples of the deployment of the CRV process according to an embodiment.

[0048] In at least one embodiment, Figure 1 The device 100 is a display device 210. In other words, the display device 210 includes a display unit and a processor configured to implement a CRV process 212 according to embodiments described herein to generate a modified video 214 from an input video 201, which is then displayed on the display unit of the display device. The input video 201 can be obtained from a data provider via a communication network, or from the device's internal memory, for example, after being captured by the input unit and stored. Typical examples of the device 210 are smartphones, tablet computers, laptop computers, monitors, head-mounted displays, televisions, video projectors, computer screens, vehicles (e.g., control and / or entertainment systems for automobiles, airplanes, ships, etc.), advertising display panels, medical monitors, etc. However, any device or combination of devices providing similar functionality can be used as the display device 210 while still conforming to the principles of this disclosure.

[0049] In at least one embodiment, the principles described herein are implemented in device 220, which does not include a display unit but prepares data for display by another device 230, which includes a display unit. In this case, the processor of device 220 implements a CRV process 222 according to the embodiments described herein to generate a modified video 224 from the input video 201 and provides the modified video to the display device 230. Examples of such device 220 are set-top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc.

[0050] In combination Figure 2In both embodiments described, the CRV process can use correspondence table 216 or 226, which can be obtained from a data provider via a communication network and / or from the device's internal memory.

[0051] Figure 3 The illustration shows a first example of a process for modifying a video to reduce its energy consumption, according to an embodiment. The CRV process 300 is, for example, performed by… Figure 1 The processor 101 of the apparatus 100 is implemented. In step 310, the processor obtains a pair of pixels of the video to be modified. A first pixel is obtained from a first frame, and a second pixel is obtained from a second frame. The two pixels are co-located in the sense that they occupy the same location in the two frames. The two frames may be successive frames. In step 320, the processor determines a pair of replacement colors that are symmetrical about the average color between the colors of the first and second pixels. The pair of replacement colors is determined based on the distance to the colors of the two pixels, thereby reducing (e.g., minimizing) the sum of energy consumed by displaying the two pixels, while ensuring that the replacement complementary color of each pixel is similar to the original color of each pixel. In step 330, in the video, the processor replaces the colors of the pair of pixels with the determined pair of replacement colors. The CRV process 300 may be performed iteratively for all pixel pairs of all frame pairs of the video. For a certain color pair, there is no replacement pair that reduces the sum of energy, so the color pair is not replaced (e.g., a pair of black pixels), and in this case, step 330 is not performed.

[0052] Figure 4The illustration shows a numerical example of replacing the complementary color of a pair of pixels using the CRV process according to an embodiment. This example uses a very simplified 4x1 image, in other words, four pixels in a single row. Image 400 shows four pixels (401, 402, 403, 404) of a first frame, and image 410 shows four pixels (411, 412, 413, 414) of a second frame. The second frame is a successor frame to the first frame. In at least one embodiment, a successor frame refers to a frame that directly follows the first frame. In at least one embodiment, more than two successor frames may be used. In each pixel, the R, G, and B values ​​are shown vertically. Pixels 401 and 411 are co-located. When the CRV process is applied to a pair of images 400, 410, the CRV process can modify the color of the pixels as described in the embodiments herein, resulting in a pair of images 420, 430. For example, the first pixel 401 (RGB values ​​= 111, 109, 72) of the first image appears to be "olive green," and the first pixel 411 (RGB values ​​= 99, 127, 171) of the second image appears to be "gasoline blue." When these pixels are modified according to the CRV process, they are replaced with pixels 421 and 431, respectively. The first pixel 421 (RGB values ​​= 88, 108, 85) of the first image appears to be a greenish version of "olive green," and the first pixel 431 (RGB values ​​= 110, 122, 170) of the second image appears to be more purplish compared to "gasoline blue."

[0053] Figure 5 The illustration shows a second example of the CRV process according to an embodiment. The CRV process 500 is, for example, performed by... Figure 1 The processor 101 of the device 100 is implemented, and for example, corresponds to Figure 4 Step 430 of process 400. For a dual RGB triple unit... and Execute this process, The first image The color of the first pixel in the image, and It is the second image of the video. The color of the second pixel in the image. and It is an image of successive frames (or an intermediate image generated previously), and the first and second pixels are co-located (i.e., located at the same position in the image). Color space The color space selected for the operation is a color space with color transformation and inverse color transformation, such as the XYZ color space used in the description of the process. In this color space, the forward color transformation... Applying to RGB to calculate (CIE) XYZ, RGB XYZ, and inverse color transformation Applying the inverse operation XYZ RGB and XYZ are chosen because they reduce the computational load for finding the complementary color replacement. Uniform color spaces (such as CIELab, IPT, or OKLab) can also be selected. In this case, the operations described below must be adapted to the selected color space.

[0054] A uniform color space (UCS) is established such that the same geometric distance anywhere in the color space reflects the same amount of perceived color difference. In this case, a positive color transformation... Often applied to RGB to calculate (CIE) XYZ, then Lab or equivalent visual coordinates RGB are calculated. XYZ Lab. It can also be calculated directly from RGB: RGB. Lab. The inverse transformation operates similarly: RGB = .

[0055] In step 510, the processor determines the first image of the video according to the selected color power consumption model. The first pixel and the second image Energy consumption of input color triplets of colocalized pixels and This power consumption model is display-dependent. For OLED displays, this power consumption model can be adapted to include a color model, specifically the RGBW case, where the white LED supports RGB LEDs for each physical pixel. The color model for RGBW displays was presented by Murdoch et al. in "Perfecting the color reproduction of RGBW OLED," proc. 30th International Congress of Imaging Science. This model can be extended to have enough parameters to represent the power of each pixel.

[0056] In step 515, the processor in the color space Internal determination Two colored dots in and yes and The response of the triplet, and the color transformation is applied as follows to obtain coordinates and coordinates : = = In step 520, the processor is in the color gamut Internal color space Sampling is performed to determine a pair of candidate colors. So that of coordinates and of Check the following equations using coordinates: In other words, regarding and The midpoint between (average) ), and It is symmetrical.

[0057] Then, for a pair of candidate colors in the group Iteratively execute the remaining steps 525 to 530.

[0058] In step 525, the processor applies the inverse transform. and To obtain the corresponding RGB values: In step 530, the processor evaluates a pair of candidate colors from the set. Each The energy consumption of the triple unit is obtained to obtain the corresponding energy consumption. A list.

[0059] In step 535, the processor in the set Identify minimum power : And retain the corresponding color pairs This corresponds to the trajectory that identifies the minimum power in this series. That is, having power The set of color pairs : In step 540, the processor will consume energy The energy consumption of the dual-input color triplet + Compare them.

[0060] if Then, by utilizing the dual-color system with minimal energy consumption... To replace the aforementioned pair of colors It will not bring any gain in terms of energy consumption. In this case, in step 545, the dual-color It will not be replaced.

[0061] if Then, by utilizing the dual-color system with minimal energy consumption... To replace the aforementioned pair of colors This brings some gain in terms of energy consumption. In this case, in step 550, it makes... A pair of colors with the minimum distance Selected: because about Since they are symmetrical, minimizing the distance between these two pairs is equivalent to minimizing the distance between a color in the first pair and its corresponding color in the second pair.

[0062] In multiple color pairs If a pair is found, the first pair in the list of multiple pairs can be selected. In another embodiment, a random pair is selected. In yet another embodiment, a different criterion is used, for example... With maximum brightness, one pair is selected.

[0063] In step 555, the processor utilizes the selected pair of colors To replace the aforementioned pair of colors .

[0064] In the above embodiments, if CRV color selection is performed in the XYZ color space, and if the power is linearly related to the LED intensity, and further, if the target display is an RGBW display, then optional criteria and steps can be added before step 510 to filter out triplet groups that cannot perform CRV processing. and In this optional step, it will be determined that... and Whether it belongs to the color gamut is determined by four The three primary colors define different triangles, such as... Figure 6 As shown in the image. In fact, if... and Those belonging to the same triangle, and where power is linearly related to LED intensity, cannot achieve power gain. Adding this criterion will save processing load when it is impossible to find a replacement complementary color.

[0065] However, when performing a search for CRV colors in a uniform color space (such as Lab), even and Located in color gamut Within the same sub-triangle, alternative complementary colors that consume less power can also be found. In this case, choosing different triangles... and This optional step is not necessary.

[0066] Figure 6 The illustration shows an example of replacing complementary colors according to an embodiment. It illustrates the color space of an RGBW display. In addition to the conventional red, green, and blue LEDs, this display also includes a white LED. This allows for finer color adjustments and reduces energy consumption. The illustration shows the color gamut. The three distinct sub-triangles in the image, and for a pair of exemplary input colors This shows the pair of input colors. Determined average color, color pair The minimum power trajectory and making with A pair of colors that minimizes the distance and has the minimum power (i.e., belongs to the trajectory with the minimum power). .

[0067] Figure 7 The illustration shows a third example of the CRV process according to an embodiment. The CRV process 700, for example, is... Figure 1 The processor 101 of the device 100 is implemented, and for example, corresponds to Figure 4 Step 430 of process 400. For a dual RGB triple unit... and Execute this process, The first image The color of the first pixel in the image, and It is the second image of the video. The color of the second pixel in the image. and These are images of consecutive frames, and the first and second pixels are co-localized (i.e., located at the same position in the image). This third example of the CRV process is a fast version that utilizes a simplified set of operations to provide video-replaced complementary colors. It is based on a geometric method using the XYZ color space to identify the replacement complementary colors. In other color spaces (e.g., OKLab), projection is much more complex. The operation of this process is illustrated in... Figure 8 middle.

[0068] In step 710, the processor in the color space Internal determination Two colored dots in and yes and The response of the triplet, and the color transformation is applied as follows to obtain coordinates and coordinates : = = In step 720, from Calculate the midpoint : In step 730, The sub-triangle it belongs to was identified. Figure 8 In this context, this corresponds to the BGW sub-triangle.

[0069] In step 740, In the sub-triangle, A parallelogram is defined. This parallelogram is defined as follows: - Q is set to the white dot Q= - It is about The point symmetric to Q: - The two primary color lines of the identified sub-triangle and Identified (in) Figure 5-9 Above, green primary color line and blue primary color line ), - Established as parallel to of exist Projection on - Established as parallel to of exist Projection on The obtained parallelogram It is the trajectory of minimum power.

[0070] In step 750, They were respectively identified as In the closest An orthogonal projection onto the boundary of a parallelogram. The resulting pair of colors. These colors are considered the best compromise between energy reduction and perceived quality because they are closest to... And it has minimal power consumption.

[0071] Figure 8 The illustration shows an example of alternative complementary color selection based on a third example of a CRV-based process according to an embodiment. It allows for the illustration... Figure 7 The process involves 700 steps. It shows the mapping on the BGW sub-triangle. Colors, the midpoint between these colors, The two primary color lines of the parallelogram and the BGW sub-triangle and as well as color, The colors are respectively In the closest Orthogonal projections onto the boundary of a parallelogram.

[0072] The CRV process can be repeated to replace the colors of the entire video. According to the following description... Figures 9 to 11 The at least three different variant embodiments illustrated in the figures achieve this by applying the process to multiple successive images. In these figures, circles respectively represent the determination of a pair of alternative complementary colors from a pair of input colors. Figure 3 , 5 Or one of the processes 300, 500, or 700.

[0073] Figure 9 The illustration shows a first example of the application of the CRV process on successive images according to an embodiment. The vertical direction represents the time dimension. The rectangles in the left column represent successive images of the video to be modified, labeled Im1, Im2, Im3, Im4, Im5, Im6. For example, Im1 is the first image of the video, Im2 is the second image of the video, and so on. The rectangles in the right column represent successive images modified by the CRV process and are labeled Im1', Im2', Im3', Im4', Im5', Im6'. In this first example, the successive images are processed in pairs. The first pair, Im1 and Im2, is used as input to the CRV process, resulting in modified images Im1' and Im2'. This simple variant embodiment uses only half the number of processes relative to the number of images.

[0074] Figure 10The illustration shows a second example of the application of the CRV process on successive images according to an embodiment. The vertical direction represents the time dimension. The rectangles in the left column represent successive images of the video to be modified, labeled Im1, Im2, Im3, Im4, Im5, Im6. The rectangles in the right column represent successive images modified by the CRV process, and are labeled Im1', Im2', Im3', Im4', Im5', Im6'. The first pair, Im1 and Im2, are used as inputs to the CRV process, resulting in modified images Im1' and Im2'. Im1' is used as the final modified image Im1'. Im2' is used as input to the CRV process along with the next image Im3, resulting in modified images Im2' and Im3'. Im2' is used as the final modified image, Im3' is used as input to the CRV process along with the next image, and so on. Therefore, using this method, the modified image will depend on the previous image, the current image, and the next image. This allows for better image stitching. Additionally, because the color replacement process is applied to each image, this method offers increased energy savings compared to the first method.

[0075] Figure 11 The illustration shows a third example of the application of the CRV process on successive images according to an embodiment. The vertical direction represents the time dimension. The rectangles in the left column represent successive images of the video to be modified, labeled Im1, Im2, Im3, Im4, Im5, Im6. The rectangles in the right column represent successive images modified by the CRV process and are labeled Im1", Im2", Im3", Im4", Im5", Im6". The middle column represents the intermediate images labeled Im1', Im2', Im3', Im4', Im5', Im6', which are used to generate the modified images. The first pair, Im1 and Im2, is used as input to the CRV process, resulting in the intermediate images Im1' and Im2'. The second pair, Im3 and Im4, is used as input to the CRV process, resulting in the intermediate images Im3' and Im4'. In another step, the intermediate images Im2' and Im3' are used as input to the CRV process, resulting in modified images Im2'' and Im3'. Im1' (which is equal to Im1'), Im2'', and Im3'' are used as the final modified images to replace Im1, Im2, and Im3, respectively. The replacement of Im4 will use intermediate results from the next pair of input images Im5 and Im6, and so on. This third example allows for better image concatenation. Additionally, because the color replacement process is applied to each image, this method offers increased energy reduction compared to the first method.

[0076] exist Figure 9 , 10Each of the embodiments described in 11 or 2 can be cascaded to further improve energy reduction and color similarity by processing the obtained image Imi' once or multiple times.

[0077] Figure 12 The illustration shows an example of a replacement complementary color selection where the energy gain is not maximized according to an embodiment. In this example... The choice was made to further emphasize the trade-off between energy gain and cost. and The distance between them defines the quality constraint. In this example, the color is not projected onto the lines of the parallelogram, but is instead chosen to be closer to the original input color.

[0078] In at least one embodiment, this distance can be adjusted to introduce a tuning mechanism that selects a desired compromise between quality and energy reduction. For example, through graphical elements, the adjustment can be under the viewer's control, allowing selection of the intensity of modifications performed on the input video. For example, in the case of a mobile device, the adjustment can also be performed by the device based on the battery charge level.

[0079] Figure 13 The illustration shows an example of a process for modifying video using a CRV process according to an embodiment. Process 1300, for example, is performed by... Figure 1 The processor 101 of the device 100 is implemented.

[0080] In step 1310, the processor obtains visual content or a portion of visual content, such as a video or frame sequence (i.e., an image comprising a set of pixels).

[0081] In step 1330, the CRV process is applied to the visual content. Step 1330 is performed iteratively on multiple pixels of successive frame pairs of the video. In step 1332, the processor obtains the color. Pixel p1 and color Pixel p2. In step 1334, the processor, according to the above embodiment, targets the color. Determine a pair of complementary colors that have reduced energy consumption. In step 1336, using one of the above processes 300, 500, or 700, the processor utilizes the replacement complementary color as determined in step 1334. Use this to replace the colors of pixels p1 and p2.

[0082] In step 1350, the modified visual content is provided, for example, when process 1300 is... Figure 2 The device 210 is displayed on the screen when it is executed, or when process 1300 is performed by Figure 2 The device 220 is transmitted over the communication network during execution.

[0083] In at least one embodiment, step 1334 is based on Figure 3 The process 300. In at least one embodiment, step 1334 is based on Figure 5 The process 500. In at least one embodiment, step 1334 is based on Figure 7 The process 700. In at least one embodiment, step 1334 is based on the following description. Figure 14 The process 1400 generates a mapping table. This mapping table stores the association between a pair of input colors and a pair of replacing complementary colors. The mapping table can use the pair of input colors as an index, so that accessing the mapping table using the pair of input colors will provide the pair of replacing complementary colors.

[0084] In at least one embodiment, the correspondence table provides several pairs of alternative complementary colors corresponding to different levels of color similarity distance.

[0085] In at least one embodiment, only a subset of the entire image is processed, rather than the entire image. For this purpose, an additional step is added before step 932 to examine the pixels. Spatially, it belongs to a subset of the image to be processed, such as a region in the image with the highest ability to mask artifacts. This region or mask can be given by, for example, a just-perceptible difference (JND) map, motion field, saliency map, etc. If the pair of pixels... If a pixel does not belong to this region or mask, then color pair replacement will not be considered for that pixel, and therefore, process steps 1332 to 1336 will not be executed for that pixel pair. Other considerations can trigger the exclusion of a subset of pixels, such as the overlay of graphics on the image.

[0086] Figure 14 The illustration shows an example of a process for generating a correspondence table of alternative complementary colors according to an embodiment. Process 1400, for example, is performed by... Figure 1 The processor 101 of the device 100 is implemented. Step 1410 can be performed iteratively multiple times. In at least one embodiment, iteration is performed on all possible colors in the color gamut. In at least one other embodiment, iteration is performed only on the color space of the subsampled image. In at least one other embodiment, iteration is performed on all unique colors of a predetermined set of videos, a video, a predetermined set of images, an image, or a region of an image. Each iteration includes steps 1411, 1412, and 1413. In step 1411, the processor obtains a pair of input colors. In step 1412, for example, using Figure 3 Process 300 or Figure 5 process 500 or Figure 7 In process 700, a pair of replacement complementary colors corresponding to the pair of input colors are used. The pair of input colors is determined. In step 1013, the pair of input colors... and the selected pair of complementary colors The relationships between them are stored in a correspondence table. At the end of process 1400, the correspondence table includes a set of relationships between input color pairs and replacement complementary color pairs. Figure 2 In this context, components 216 and 226 are correspondences that can be obtained from process 1400.

[0087] The proposed CRV process uses two colors to search for two other alternative complementary colors each time. In terms of power, this allows for finding potentially higher gains. It also allows operation in linear color spaces (such as XYZ), which significantly reduces the processing load of searching for alternative complementary colors. Minimization is not required; direct computation of the minimum power trajectory and color pairs closer to the original color is possible. CRV is more efficient than TACC or SACC in the sense that it provides higher energy gain in the moving bands of an image. In the static bands, TACC and SACC provide the same results as CRV. It also allows for finding alternative complementary colors close to the original color, thus avoiding the use of frames with completely different colors to replace frames in a video. Unlike TACC, which inherently introduces flicker, CRV does not introduce flicker, which also means that the post-processing operations (equal brightness, pixel swapping) required to reduce flicker in TACC are no longer needed.

[0088] Although some parts of this description mention video, the embodiments are not limited to video and are based on the same principles described above, applicable to any type of visual media content, such as stereoscopic (3D) video and 360° immersive video.

[0089] Generally, one or more other examples of the embodiments can also provide a computer-readable storage medium, such as a non-volatile computer-readable storage medium, which stores instructions for encoding or decoding image information (such as video data) according to the methods or apparatus described herein. One or more embodiments can also provide a computer-readable storage medium that stores a bitstream generated according to the methods or apparatus described herein. One or more embodiments can also provide methods and apparatus for transmitting or receiving bitstreams or signals generated according to the methods or apparatus described herein.

[0090] Many examples of the embodiments described herein are specifically described, and often in a manner that may sound restrictive, at least for the purpose of illustrating individual characteristics. However, this is for the purpose of clarity of description and not to limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, embodiments, features, etc., can also be combined and interchanged with other embodiments, features, etc., described in earlier documents.

[0091] It is important to note that the syntax elements used here are descriptive terms. Therefore, the use of other syntax element names is not excluded.

[0092] When the accompanying diagram is presented as a flowchart, it also provides a block diagram of the corresponding device. Similarly, when the accompanying diagram is presented as a block diagram, it also provides a flowchart of the corresponding method / process.

[0093] Generally, the examples of embodiments, implementations, features, etc., described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features can be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, suitable hardware, software, and firmware. One or more examples of methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users. Furthermore, the term “processor” is used herein to broadly include various configurations of one or more processors.

[0094] The references to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” appearing in various places throughout this application, and any other variations thereof, do not necessarily all refer to the same embodiment.

[0095] Additionally, this application may involve "determining" various information segments. Determining information can include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0096] In addition, this application may involve "accessing" various information segments. Accessing information may include one or more of the following: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0097] Furthermore, this application may relate to "receiving" various information segments. Like "accessing," "receiving" is intended to be used in a broad sense. Receiving information can include one or more of the following: (e.g., accessing information or retrieving information from memory). Additionally, "receiving" is typically involved during operation in one manner or another, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0098] To be understood, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the above “ / ,” “and / or,” and “at least one of…” is intended to include: selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this phrase is intended to include: selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A, B, and C). It will be clear to those skilled in the art and related fields that this can be extended for any number of listed items.

[0099] It will be apparent to those skilled in the art that implementations can generate various signals, which are formatted to carry information, and these messages can be stored or transmitted, for example. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal can be formatted to carry a bitstream of the described embodiment. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding the data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal can be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal can be stored on a processor-readable medium.

[0100] Various embodiments are described herein. Features of these embodiments can be provided, individually or in any combination, across a variety of claim types and classes.

Claims

1. A method comprising: Obtain a first pair of colors for a pair of pixels, wherein the first pixel of the pair of pixels is included in a first image of a video and the second pixel of the pair of pixels is included in a second image of the video, and wherein the first pixel and the second pixel are co-located in the image; Determine the average color between the colors in the first pair of colors; A second pair of colors is determined, wherein the colors in the second pair are symmetrical about the average color, and the sum of the energy consumed by displaying a pair of pixels having the second pair of colors is less than the sum of the energy consumed by displaying a pair of pixels having the first pair of colors; and The first pair of colors of the pair of pixels is replaced by the second pair of replacement colors.

2. The method of claim 1, wherein the first color in the second pair of colors is determined based on the distance to the first color in the first pair of colors.

3. The method of any one of claims 1 or 2, wherein the first color in the first pair of colors is perceptually identical to the first color in the second pair of colors, and the second color in the first pair of colors is perceptually identical to the second color in the second pair of colors.

4. The method according to any one of claims 1 to 3, further comprising: Determine a set of color pairs with the minimum energy, and select the second pair of colors from the set of color pairs with the minimum energy.

5. The method of any one of claims 1 to 4, wherein the second image is directly appended to the first image.

6. The method according to any one of claims 1 to 5, further comprising: The method is executed iteratively for multiple first pairs of colors, and the association between the first pair of colors and the corresponding second pair of colors is stored.

7. The method of claim 6, wherein a pair of colors is used as an index, and the association is stored in a corresponding table.

8. A method comprising: Obtain visual content; The visual content is iterated over successive frames, and the pixel pairs located in the successive frames are iterated over. For each iterated pixel pair: According to the method of any one of claims 1 to 7, a second pair of colors is determined for the pair of iterative pixels; and The visual content is modified by replacing the colors of the iterative pair of pixels with the second pair of colors; and Provides modified visual content.

9. An apparatus comprising a processor, the processor being configured to: Obtain a first pair of colors for a pair of pixels, wherein the first pixel of the pair of pixels is included in a first image of a video and the second pixel of the pair of pixels is included in a second image of the video, and wherein the first pixel and the second pixel are co-located in the image; Determine the average color between the colors in the first pair of colors; A second pair of colors is determined, wherein the colors in the second pair are symmetrical about the average color, and the sum of the energy consumed by displaying a pair of pixels having the second pair of colors is less than the sum of the energy consumed by displaying a pair of pixels having the first pair of colors; and The first pair of colors of the pair of pixels is replaced by the second pair of replacement colors.

10. The device of claim 9, wherein the first color in the first pair of colors is perceptually identical to the first color in the second pair of colors, and the second color in the first pair of colors is perceptually identical to the second color in the second pair of colors.

11. The device of any one of claims 9 or 10, wherein the processor further comprises: The method is executed iteratively for multiple first pairs of colors, and the association between the first pair of colors and the corresponding second pair of colors is stored.

12. The device of claim 11, wherein a pair of colors is used as an index, and the association is stored in a correspondence table.

13. An apparatus comprising a processor, the processor being configured to: Obtain visual content; The visual content is iterated over successive frames, and the pixel pairs located in the successive frames are iterated over. For each iterated pixel pair: According to the method of any one of claims 9 to 12, a second pair of colors is determined for the pair of iterative pixels; and The visual content is modified by replacing the colors of the iterative pair of pixels with the second pair of colors; and Provides modified visual content.

14. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method as claimed in any one of claims 1 to 8.

15. A non-transitory computer-readable medium storing executable program instructions to cause a computer to execute the instructions thereby performing the method as claimed in any one of claims 1 to 8.

Images