Local dimming processing algorithm and correction system

Abstract

A method of providing full array local dimming for a display, the method comprising executing an image processing algorithm on a processor having instructions for determining a new pixel value for each of a plurality of pixels of an image, mapping the new pixel value to a previous pixel value for each of the plurality of pixels, bilinearly scaling the image for the zone, repeating the determining, mapping and scaling until an approximation is achieved, and compiling the iterated results into a data set. The method also includes dividing an image for display into a plurality of zones, each zone having at least one LED associated therewith, and the method also includes making an illumination decision from the data set, the illumination decision being for at least one LED associated with a zone of the plurality of zones.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This U.S. patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 199928, filed February 3, 2021, which is incorporated herein by reference.

[0002] Technical Field The technical field relates generally to displays, and more particularly to displays for vehicles.

[0003] Background technology Automobiles typically use displays to share information with vehicle occupants. In particular, the displays present information for use by the vehicle's driver. However, displays must function in a wide variety of ambient lighting conditions, ranging from bright, shining daylight to dark night. To adjust for different ambient lighting conditions, the brightness of the backlight for the display can be varied.

[0004] Organic light-emitting diode (OLED) displays are visually very appealing, especially in the automotive market, but are extremely expensive.

[0005] Another option is to provide a display with full-array local dimming (FALD), which consists of multiple arrays of light-emitting diodes (LEDs) across the entire back surface of the screen. These arrays darken parts of the screen that need to be darker, without affecting areas of the screen that need to be brighter. Local dimming using thin-film transistor (TFT) displays with direct, zoned backlighting, such as that provided by FALD displays, is a cheaper way to achieve similar performance.

[0006] To enable full array local dimming (FALD), the image must be tiled by indexing into zones. Each zone is quite large, even for relatively low-resolution displays. In a high-performance computing environment, the way to determine the state of each zone as on or off is simply to read every pixel in the entire image and determine whether it contains content. This places a significant burden on the processor and memory.

[0007] Even for low-pixel displays, the processing power required to provide FALD using traditional calculation methods can be computationally intensive. Determining the state of each zone as on or off simply involves reading every pixel in the entire image and determining whether it contains content. This places a significant burden on the processor and memory. As an example, consider 720px x 1920px x 60fps x 32bit = 248MB / sec. This is too much data to process in real time on a low-cost embedded system-on-chip (SoC) with other features. A common option is to develop and deploy a customer-specific integrated circuit (ASIC) to perform this calculation separately, but such an ASIC significantly increases the cost of the product.

[0008] This approach therefore involves a dedicated processor that increases costs and integration challenges that reduce its benefits over OLED.

[0009] Therefore, what is needed is an arrangement and method that provides a unique, low-cost approach to enabling local dimming without increasing hardware costs. More importantly, what is needed is a FALD display that reduces errors caused by approximation of pixel states in zones.

[0010] The discussion of the background art provided herein is intended to provide a general context for the present disclosure. To the extent provided in this background art section, the work of the presently named inventors, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are not admitted, expressly or impliedly, as prior art against the present disclosure.

[0011] overview One embodiment of a method for providing full array local dimming in a display comprises: determining a new pixel value for each of a plurality of pixels of the image; for each of the plurality of pixels, mapping a new pixel value to a previous pixel value; Bilinearly scaling the zone image, Repeat the determination, mapping and scaling until an approximation is achieved. Compiling the repeated results into a dataset, and executing on the processor an image processing algorithm having instructions for:

[0012] The method also includes dividing the image for display into a plurality of zones, each zone having at least one LED associated therewith.

[0013] The method also includes making, by the processor, an illumination decision from the dataset, where the illumination decision is for at least one LED associated with one of the plurality of zones.

[0014] Other embodiments of this aspect include corresponding computer systems, apparatus and computer programs recorded on one or more computer storage devices, each configured to perform the operations of this method.

[0015] Implementations may include one or more of the following features: The method may include making an illumination determination for each of a plurality of zones.

[0016] Dividing the image into a plurality of zones may further include dividing the image such that each of the plurality of zones has one LED associated therewith.

[0017] The method may include using luminescence data of pixels in one of the multiple zones to make an illumination determination.

[0018] The method may include determining that the illumination determination is "yes" if there is at least one pixel having a luminance within one of the plurality of zones.

[0019] The method may include illuminating at least one LED associated with one of the plurality of zones if the illumination determination is "yes."

[0020] The method may include converting the compiled data set into a YUV image format.

[0021] Dividing the image into a plurality of zones and making the illumination determination may further include using a first processor and using a second processor to execute an image analysis algorithm.

[0022] The method may include sending the data set from the second processor to the first processor prior to the illumination determination.

[0023] The method may include sending the data set from the second processor to the first processor before dividing the image into a plurality of zones.

[0024] The second processor may be one of a plurality of systems-on-chip.

[0025] The second processor may be coupled to at least one memory, the memory containing a look-up table of expansion curve values.

[0026] The determination, mapping and scaling may be repeated for three iterations.

[0027] Dividing the image into zones and making the illumination determination may be performed by a first processor, and executing the image analysis algorithm may be performed by a second processor.

[0028] Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0029] Another embodiment of a method for providing full-array local dimming in a display comprises dividing an image for the display having a plurality of pixels into a plurality of zones, each zone having at least one LED associated therewith.

[0030] This method is determining a new pixel value for each of the plurality of pixels; for each of the plurality of pixels, mapping a new pixel value to a previous pixel value; Bilinearly scaling the zone image, Repeat the determination, mapping and scaling until an approximation is achieved. Compiling the repeated results into a dataset, The method also includes executing an image processing algorithm on a processor having instructions for:

[0031] The method also includes making, by the processor, an illumination decision from the dataset, where the illumination decision is for at least one LED associated with one of the plurality of zones.

[0032] Other embodiments of this aspect include corresponding computer systems, apparatus and computer programs recorded on one or more computer storage devices, each configured to perform the operations of this method.

[0033] Implementations may include one or more of the following features.

[0034] The method may include making an illumination determination for each of a plurality of zones.

[0035] Dividing the image into a plurality of zones may further include dividing the image such that each of the plurality of zones has one LED associated therewith.

[0036] The method may include using luminescence data of pixels in one of the multiple zones to make an illumination decision, and determining that the illumination decision is "yes" if there is at least one pixel in the one of the multiple zones that has a luminance.

[0037] The method may include illuminating at least one LED associated with one of the plurality of zones if the illumination determination is "yes."

[0038] The method may include converting the compiled data set to a YUV image format. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0039] Other objects, features and characteristics of the present invention, as well as the method of operation and function of the associated elements of construction, combination of parts and economies of manufacture, will become more apparent from a study of the following detailed description and appended claims, taken in conjunction with the accompanying drawings, all of which form a part hereof. It should be understood that the detailed description and specific examples, while indicating preferred embodiments of the disclosure, are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

[0040] Other advantages of the disclosed subject matter will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. [Brief explanation of the drawings]

[0041] [Figure 1] FIG. 1 illustrates an example of a display having a FALD, according to one exemplary embodiment. [Figure 2] FIG. 1 illustrates a block diagram illustrating an apparatus for implementing FALD, according to one exemplary embodiment. [Figure 3] FIG. 10 illustrates a graph of application of an expansion curve for pixel scaling, according to one example embodiment. [Figure 4] FIG. 10 illustrates a graph of percentage of pixels on compared to scaling iterations according to one example embodiment. [Figure 5] FIG. 1 illustrates a first embodiment of a method for providing an image with full-array local dimming.

[0042] Detailed Description With reference to the figures, in which like numerals refer to like parts across multiple views, a display system 100 for a vehicle 10 and method for providing full array local dimming (FALD) is shown and described herein. This embodiment can provide dimming that includes a reduction in the scale of the light intensity provided by the array, which can include turning off an area completely or only partially reducing the light intensity on an area of ​​the display 200.

[0043] 1-5, there is shown a display system 100. The display system 100 includes a display 200, a first processor 120, a plurality of second processors 130, and at least one memory device 140. The display 200, the first processor 120, the at least one second processor 130, and the memory device have communication therebetween, which is indicated by the reference numeral 150.

[0044] Display 200 shows an image 210. A plurality of LEDs 112 may provide backlighting for display 200. Image 210 may be divided into a plurality of zones 220, each of which is made up of a plurality of pixels 230. The number of pixels 230 for a given image 210 and display 200 may vary depending on the overall size of the display, the number of LEDs for the display, or other desired factors. Each zone has at least one LED 112 associated with it.

[0045] In one embodiment, there is one LED 112 associated with each zone 220 .

[0046] For each of the plurality of zones 220, the display system 100 must determine, based on the state of the pixels in that zone 220 of the display 200, whether that zone should be backlit by at least one associated LED or not.

[0047] To provide the FALD to the image 210, an evaluation algorithm 310 is applied to each of the plurality of zones 220 to determine whether the associated at least one LED 112 should be turned on or off. To reduce the computational power required to evaluate the image 210 by the evaluation algorithm 310, an image processing algorithm 300 is applied prior to evaluation of the LEDs.

[0048] In one embodiment, the image processing algorithm 300 is applied before dividing the image 210 into multiple zones 220. Alternatively, the image processing algorithm 300 may be applied after the image has been divided into multiple zones 220. In this embodiment, an image processing algorithm as described herein is applied iteratively to each of the multiple zones 220. Regardless of the order, both the image processing algorithm 300 and the division of the image into multiple zones 220 occur before the LED evaluation algorithm 310.

[0049] The image processing algorithm 300 may be executed by the first processor 120, and the evaluation algorithm may be executed by the second processor 130. Alternatively, both the image processing algorithm and the evaluation algorithm may be executed by the second processor 130. Furthermore, either the first processor 120 or the second processor 130 may be a system-on-chip. There may be multiple first processors 120 and second processors 130. Thus, processing can be performed with industry-standard hardware acceleration that is supported at no additional cost in many embedded systems-on-chips (SoCs).

[0050] The image processors 120, 130 implement the image processing algorithm 300 using a look-up table (LUT) with an expansion curve. The old pixel values ​​may include the color associated with each pixel. The new pixel values ​​may be black and white. Thus, all pixels with color are converted to a white / on format, and others are converted to a black / off format.

[0051] The new pixel value is mapped to the old pixel value.

[0052] The image 210 is then bilinearly scaled by a scale factor of no more than 2. By scaling the image, the original pixel data is preserved. Thus, in one embodiment, the second processors 130 may be SoCs.

[0053] In one embodiment, the scale factor is two.

[0054] Scaling the data reduces the processing power required, but as the number of scaling iterations increases, so does the error measured in the reduction in the percent of zones that are on, so the lookup table applies a scaling curve to correct for this error.

[0055] The lookup table may be loaded into the processor 130 from at least one memory device 140 and stored in the processor 130 during the approximation process. Thus, iterative application of the use of the lookup table to apply the expansion curve can be used, as shown in Figures 3 and 4. The expansion curve as shown in Figure 3 corrects this error by downscaling.

[0056] Consider FIG. 4, which shows ×2 and ×3 iterations of a lookup table with application of an expansion curve. Even with high iteration scaling, the error caused by downscaling is reduced to near zero. As shown in FIG. 4, the percentage of LEDs evaluated to be turned on is compared to the number of scaling iterations that occur. After ×3 scaling iterations, the percentage of LEDs that are on becomes a substantially linear number. Therefore, further scaling iterations may not be necessary. Thus, the impact of scaling as a computation reduction technique becomes visible to the user.

[0057] Once the approximation process is complete, the information is compiled into a data set, here with an 8x downscale approximation. The end result of the iterative scaling and expansion curve process is highly compressed data. In one example, the downscale is an 8x downscale, as shown in Figure 2. This is a 2x downscale factor (2x) performed three times to obtain the 8x downscale factor. 3 ) In this example, the compression is up to 256x due to this process alone. This allows for a smart FALD implementation without requiring additional processing hardware.

[0058] For example, for a zone of approximately 80x80, the pixels may be (for example) mathematically compressed to 5x5 or the like, in which case the CPU or first processor can easily read the smaller amount of information and make lighting decisions based on this information.

[0059] Furthermore, a pixel color format with lower bandwidth consumption (fewer bytes per pixel) may be utilized for processing and final analysis of the compressed data by the CPU. The original image material for analysis is typically presented as RGB pixel data with four bytes per pixel for direct display feed. Built-in filter engines (described above) often support multiple color formats and "on-the-fly" color format conversion. After the first iteration, the data may be stored / read in a format with fewer bytes per pixel, such as YUV NV12 with two bytes per pixel. This significantly reduces system bandwidth consumption during analysis. YUV color formatting includes luma data (Y) and chroma data (UV). However, as described in more detail below, only luma data is used by the evaluation algorithm 310. Therefore, only one byte per pixel data is used. Because the scaled and transformed dataset is used only by the evaluation algorithm 310, the loss of color and chroma data is not necessary for further processing and may be discarded from the transformed dataset to save memory and processing.

[0060] FIG. 2 illustrates the reformatting of image 210 from RGB color formatting to YUV color formatting during the process, which may be included as an option to simplify software analysis. A conventional lookup table already converts pixels to black and white values ​​equivalent to on and off luminance data. Furthermore, as a simplification, chroma information (UV) may not be necessary for the evaluation algorithm. Therefore, potential loss of detail in chroma (UV) information may have no impact on the analysis, and such a format results in only a small loss of important luminance (Y) information. The final analysis results may be stored in planar or semi-planar YUV format.

[0061] Finally, the evaluation algorithm 310 looks at the luminance value (Y) of each of the multiple zones. If a luminance value is present and on (white), the evaluation algorithm 310 determines that at least one LED 112 in that zone 220 should be on. However, if a luminance value is absent, i.e., off (black), the evaluation algorithm determines that at least one LED in that zone should be off. Thus, the CPU only needs to read and process the important luminance (Y) information; it does not need to read the unimportant chroma (UV) information, which it can discard.

[0062] The image may be divided to have a specific number of LEDs per zone, e.g., one or more LEDs, or to provide zones such that each zone is centered around one LED. If there is one LED per zone, the illumination decision for that zone may be based on the luminance values ​​of all pixels within that zone. In other words, if any of the pixels in a zone have a luminance value, the luminance evaluation is "yes" / "on." However, if no pixels in a zone have a luminance value, the luminance evaluation is "no" / "off." In this example, the LED illuminating a particular pixel may not be located directly behind the pixel itself, but may be located close enough to provide a sufficient amount of backlight to illuminate the "yes" / "on" pixel. For example, in a 3x3 pixel zone, the LED may be located directly behind the center pixel. However, if any of the nine pixels have an associated luminance, the illumination decision is "yes" / "on." For the other eight pixels in that zone, the LED that illuminates that pixel is not located directly behind this pixel, but is located close enough to the centrally located LED to have sufficient brightness.

[0063] The level of sufficient brightness and the proximity that an LED can have to a particular pixel to provide that sufficient illumination may be determined for a particular display system 100 by one skilled in the art.

[0064] Alternatively, multiple zones may be determined by dividing the number of pixels across the entire image to form zones of uniform pixel size (e.g., each zone is 10x10 pixels). If multiple LEDs are present in a zone, the LED illumination decision may be the same for all LEDs. For example, if there are two or more LEDs in a zone, all LEDs may be turned on or all LEDs may not be turned on for that zone. The illumination decision may be to turn on all LEDs if there is at least one pixel with a brightness value. In this embodiment, some LEDs may be unnecessarily turned on. However, the overall processing power for making the illumination decision may be less than if the decision were made for each LED individually.

[0065] In one embodiment, there may be one luminance value per zone, regardless of the number of pixels associated with a particular zone. If any of the pixels have luminance, the zone will have an luminance value of ON. This is information that can be saved as part of the data set and converted to YUV format. Therefore, the location of the specific pixel within the zone that has the luminance value and requires illumination may not be known.

[0066] The amount of illumination provided by each LED 112 decreases as the distance from the LED 112 increases. Thus, a pixel directly above the center of an LED 112 has more illumination than a pixel further away from the LED 112. Depending on the division of the multiple zones 220, some pixels, for example, pixels at or near the periphery of a zone 220, will be less illuminated by the LEDs 112 than pixels closer to the center of the zone.

[0067] If a pixel having a brightness value, and therefore requiring illumination by an LED 112, is located near or near a zone, the amount of illumination provided by the associated LED 112 may be less, or even less than desired. Thus, it may be desirable to illuminate not only the zone 220, but also the LEDs 112 in adjacent zones 220. Thus, a further step in the illumination determination for each zone 220 may be to evaluate the brightness value of each of the adjacent zones 220 as well.

[0068] One skilled in the art can determine the number of zones 220, the size of the zones, and the number of LEDs 112 per zone for a particular display system 100 based on the overall display 100 size, display definition, number of LEDs, available processing power, and desired brightness values ​​for the display system 100.

[0069] A method 500 according to one embodiment is shown in Figure 5. A method for providing full-array local dimming on a display comprises dividing an image into zones by a first processor 502, applying an image processing algorithm using a second processor 504, which includes determining new pixel values ​​using a look-up table (LUT) 506, mapping the new pixel values ​​to old pixel values ​​508, and bilinearly scaling the image 510. The determining, mapping, and scaling are repeated until an approximation is achieved. The results are compiled into a dataset 512. The dataset is converted to a YUV image format and stored in memory 514.

[0070] As noted above, the method can be performed on one processor and the step of sending the data set from one processor to the other can be omitted.

[0071] Furthermore, after any number of iterations of determining new pixel values, mapping values, and scaling the image (including after just one iteration), the data set may be compiled and converted to YUV format for storage.

[0072] Furthermore, the total number of iterations performed and the approximation to be achieved may be determined to balance the data reduction and associated processing with the amount of error introduced by the scaling process. One skilled in the art can determine the desired number of iterations and approximation for a particular display system 100, including the number, size and speed of displays, processors, memory, etc.

[0073] The data set 512 is sent from the second processor 130 to the first processor 120, step 518. The method also includes executing, by the first processor, an evaluation algorithm to make an illumination decision for each of the plurality of zones from the data set, 520.

[0074] Thus, the evaluation algorithm may include making an illumination determination for each of a plurality of zones.

[0075] Additionally, lighting decisions are made based not only on the lighting value of the zone, but also on the lighting values ​​of each of the zones adjacent to the zone.

[0076] Finally, due to the reduced computational power required for the evaluation algorithm, the method can also be implemented at a frequency that is consistent with display system 100. For example, display system 100 may have a refresh rate of 60 Hz. The refresh rate of display system 100 may be selected in relation to multiple parameters, which may include, but are not limited to, the processing of the full-array local dimming method described herein.

[0077] The invention has been described herein in an illustrative manner, and it is to be understood that the terminology used is intended to be in the nature of description rather than of limitation. Obviously, many modifications and variations of the invention are possible in light of the above teachings. The invention may be practiced otherwise than as specifically described within the scope of the appended claims.

Claims

1. 1. A method for providing full array local dimming in a display, comprising: The method comprises dividing an image for the display having a plurality of pixels into a plurality of zones, each zone having at least one LED associated therewith; The method comprises: determining a new pixel value for each of a plurality of pixels of the image; for each of the plurality of pixels, mapping the new pixel value to a previous pixel value; bilinearly scaling an image of a zone to evaluate whether to turn on the at least one LED associated with that zone; repeating the determining, mapping and scaling until the percentage of LEDs estimated to be turned on remains constant; Compiling the repeated results into a dataset, executing on a processor an image processing algorithm having instructions for: The method comprises making, by the processor, a lighting decision from the dataset, the lighting decision for illuminating the at least one LED associated with one of the plurality of zones, the lighting decision being based on a luminance of the one of the plurality of zones and a zone adjacent to the one of the plurality of zones. method.

2. The method of claim 1 , further comprising making an illumination determination for each of the plurality of zones.

3. The method of claim 2 , wherein dividing the image into the plurality of zones further comprises dividing the image such that each of the plurality of zones has one LED associated therewith.

4. The method of claim 1 , further comprising using luminescence data of the pixels in one of the plurality of zones to make an illumination decision.

5. The method of claim 4 , further comprising determining that the illumination decision is “yes” if there is at least one pixel having a luminance within one of the plurality of zones.

6. The method of claim 5 , further comprising illuminating the at least one LED associated with a zone of the plurality of zones if the illumination decision is “yes.”

7. The method of claim 1 , further comprising converting the compiled data set into a YUV image format.

8. 2. The method of claim 1, wherein the determining of the new pixel value, the mapping, and the scaling are performed by a first processor, and the evaluating of whether to turn on the at least one LED is performed by a second processor.

9. The method of claim 8 , further comprising sending the data set from the second processor to the first processor prior to the illumination determination.

10. The method of claim 8 , further comprising sending the data set from the second processor to the first processor before dividing the image into a plurality of zones.

11. The method of claim 8 , wherein the second processor is one of a plurality of systems on a chip.

12. The method of claim 8 , wherein the second processor is coupled to at least one memory, the memory including a look-up table of expansion curve values.

Images