Local dimming processing algorithm and correction system

Abstract

A method for providing full-array local dimming to a display includes: executing an image processing algorithm with a processor having instructions for: determining a new pixel value for each of a plurality of pixels in an image; mapping the new pixel value to a previous pixel value for each of the plurality of pixels; bilinearly scaling the partitioned image; repeating the determination, mapping, and scaling until an approximation is reached; and compiling the repeated results into a dataset. The method further includes: dividing the image of the display into a plurality of partitions, each partition having at least one LED associated therewith; and making a lighting decision based on the dataset, wherein the lighting decision is directed to at least one LED associated with one of the plurality of partitions.

Description

[0001] Cross-references to related applications

[0002] This U.S. patent application claims the benefit of U.S. Provisional Application 63 / 199,928, filed February 3, 2021, which is incorporated herein by reference. Technical Field

[0003] The technical field broadly relates to displays, and more specifically, to displays for vehicles. Background Technology

[0004] Cars typically use displays to share information with vehicle occupants. In particular, displays show information for the driver. However, these displays must operate under a wide variety of ambient lighting conditions, ranging from bright daylight to complete darkness. To adapt to these varying lighting conditions, the brightness of the display's backlight varies.

[0005] Organic light-emitting diode (OLED) displays are visually appealing, but they are very expensive, especially in the automotive market.

[0006] Another option is to provide a display with full array local dimming (FALD), which consists of an array of multiple light-emitting diodes (LEDs) above the entire back of the screen. These arrays dim the parts of the screen that need to be darkened without affecting the areas that need to be brightened. Using a thin-film transistor (TFT) display for local dimming is a cheaper way to achieve similar performance, with localized direct backlighting provided by the FALD display.

[0007] To enable Full Array Local Dimming (FALD), the image must be tiled by indexing component partitions. Even for relatively low-resolution displays, each partition is quite large. In powerful computing environments, the method used to determine the state of each partition as on or off simply reads every pixel in the entire image and determines whether it contains content. This places a significant load on the processor and memory.

[0008] Even for low-pixel displays, examples of providing the processing power required for FALD using traditional computational methods would require high computing power. This method of determining the state of each partition as on or off simply reads every pixel in the entire image and determines whether it contains content. This places a heavy load on the processor and memory. Consider this example: 720px × 1920px × 60fps × 32 bits = 248 MB / s. This is too much data to process in real time on a low-cost embedded system-on-chip (SoC) with other functions. A common option is to develop and deploy a custom application-specific integrated circuit (ASIC) to perform this computation independently, but such an ASIC adds significant cost to the product.

[0009] Therefore, this approach requires a dedicated processor, which increases costs and integration challenges, thereby reducing the benefits relative to OLED.

[0010] Therefore, there is a need for innovative, low-cost methods and arrangements to achieve local dimming without increasing hardware costs. More importantly, there is a need for FALD displays that reduce errors caused by approximate pixel states.

[0011] The background description provided herein is for the purpose of providing a general overview of the context of this disclosure. To the extent described in this background section, the work of the currently named inventors and aspects of the description that may otherwise not conform to the prior art at the time of filing are neither explicitly nor implicitly considered to be prior art to this disclosure. Summary of the Invention

[0012] One embodiment of a method for providing full-array local dimming to a display includes executing an image processing algorithm with a processor having instructions for: determining a new pixel value for each of a plurality of pixels in an image; mapping the new pixel value to a previous pixel value for each of the plurality of pixels; bilinearly scaling the partitioned image; repeating the determination, mapping, and scaling until an approximation is reached; and compiling the repeated results into a dataset.

[0013] The method also includes dividing the image on the display into multiple partitions, each partition having at least one LED associated with it.

[0014] The method also includes using a processor to make lighting decisions based on the dataset, wherein the lighting decisions are directed at at least one LED associated with one of a plurality of partitions.

[0015] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods.

[0016] The implementation may include one or more of the following features. The method may include making lighting decisions for each of multiple zones.

[0017] Dividing an image into multiple partitions can also include dividing the image such that each of the multiple partitions has an associated led.

[0018] This method may include using brightness data of pixels in one of multiple partitions to make lighting decisions.

[0019] The method may include determining the lighting decision as yes when at least one pixel with brightness exists in one of a plurality of partitions.

[0020] The method may include, when the lighting decision is yes, at least one led associated with one of a plurality of zones.

[0021] This method may include converting the compiled dataset into a YUV image format.

[0022] Dividing the image into multiple partitions and making the lighting decision may also include using a first processor, and executing the image analysis algorithm using a second processor.

[0023] The method may include sending the dataset from the second processor to the first processor before the lighting decision is made.

[0024] The method may include sending the dataset from the second processor to the first processor before dividing the image into multiple partitions.

[0025] The second processor may be one of multiple system-on-a-chip.

[0026] The second processor may be connected to at least one memory, and said memory includes a lookup table for extended curve values.

[0027] The determination, mapping, and scaling can be repeated three times in an iteration.

[0028] Dividing an image into multiple partitions and making lighting decisions can be done using a first processor, while executing image analysis algorithms can be done using a second processor.

[0029] The implementation of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0030] Another embodiment of the method for providing full-array local dimming to a display includes dividing an image of a display having multiple pixels into multiple partitions, each partition having at least one LED associated with it.

[0031] The method also includes executing an image processing algorithm with a processor having instructions for: determining a new pixel value for each of a plurality of pixels; mapping the new pixel value to a previous pixel value for each of the plurality of pixels; bilinearly scaling the partitioned image; repeating the determination, mapping, and scaling until an approximation is reached; and compiling the repeated results into a dataset.

[0032] The method also includes using a processor to make lighting decisions based on the dataset, wherein the lighting decisions are directed at at least one LED associated with one of a plurality of partitions.

[0033] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods.

[0034] The implementation may include one or more of the following features.

[0035] This method may include making lighting decisions for each of multiple zones.

[0036] Dividing an image into multiple partitions can also include dividing the image such that each of the multiple partitions has an associated led.

[0037] The method may include making an illumination decision using luminance data of pixels in one of a plurality of partitions, and determining the illumination decision as yes when at least one pixel with luminance exists in one of the plurality of partitions.

[0038] The method may include, when the lighting decision is yes, at least one LED associated with one of a plurality of zones.

[0039] This method may include converting a compiled dataset into a YUV image format. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0040] Other objects, features, and characteristics of the invention, as well as the functionality of related elements, combinations of components, and economics of manufacture of the methods of operation and structures, will become more apparent upon consideration of the following detailed description and appended claims with reference to the accompanying drawings, all of which form part of this specification. It should be understood that while the detailed description and specific examples indicate preferred embodiments of the present disclosure, they are intended for illustrative purposes only and not for limiting the scope of the disclosure. Attached Figure Description

[0041] Other advantages of the disclosed subject matter will be readily understood, as they become even better when considered in conjunction with the accompanying drawings, by referring to the following detailed description, in which:

[0042] Figure 1 This is an example of a display with a FALD according to an exemplary embodiment;

[0043] Figure 2 This is a block diagram illustrating an apparatus for implementing FALD according to an exemplary embodiment;

[0044] Figure 4 This is a graphical illustration of an extended curve application for pixel scaling according to an exemplary embodiment;

[0045] Figure 3 It is a graphical illustration of the percentage of pixel opening compared to a scaling iteration, according to an exemplary embodiment; and

[0046] Figure 5 This is an illustration of a first embodiment of a method for providing an image with full-array local dimming. Detailed Implementation

[0047] Referring to the accompanying drawings, in which the same numbers denote the same parts in all the views, this document illustrates and describes a display system 100 for a vehicle 10 and a method for providing full array local dimming (FALD). These embodiments can provide dimming, including reducing the proportion of light intensity provided by the array, including completely turning off areas of the display 200 or only partially reducing the light intensity on those areas.

[0048] refer to Figure 1-5 The diagram illustrates a display system 100. The display system 100 includes a display 200, a first processor 120, a plurality of second processors 130, and at least one memory device 140. Communication exists between the display 200, the first processor 120, the at least one second processor 130, and the memory device, as shown in diagram 150.

[0049] Display 200 shows an image. Multiple LEDs can provide backlighting for display 200. The image can be divided into multiple zones, each consisting of multiple pixels. The number of pixels for a given image and display 200 can vary depending on the overall size of the display, the number of LEDs in the display, or other desired factors. Each zone has at least one LED associated with it.

[0050] In one embodiment, there is an LED associated with each partition.

[0051] For each of the multiple partitions, the display system 100 must determine whether the partition should or should not be illuminated by at least one associated LED backlight based on the state of the pixels within the partition of the display 200.

[0052] To provide FALD to the image, evaluation algorithm 310 is applied to each of the multiple partitions to determine whether at least one associated LED should be turned on or off. To reduce the computational power required to evaluate the image using evaluation algorithm 310, image processing algorithm 300 is applied prior to LED evaluation.

[0053] In one embodiment, image processing algorithm 300 is applied before the image is divided into multiple partitions. Alternatively, image processing algorithm 300 may be applied after the image is divided into multiple partitions. In this embodiment, the image processing algorithm as described herein is applied iteratively to each of the multiple partitions. Regardless of the order, image processing algorithm 300 and dividing the image into multiple partitions 22 both occur before LED evaluation algorithm 310.

[0054] Image processing algorithm 300 can be executed by first processor 120, and evaluation algorithm can be executed by second processor 130. Alternatively, both image processing algorithm and evaluation algorithm can be executed by second processor 130. Furthermore, the first or second processors 120, 130 can be systems-on-a-chip (SoCs). Multiple first and second processors 120, 130 can also exist. Therefore, processing can be performed using industry-standard hardware acceleration supported by many embedded SoCs without additional cost.

[0055] Image processors 120 and 130 implement image processor algorithm 300 using a lookup table (LUT) with an extended curve. Old pixel values ​​may include the color associated with each pixel. New pixel values ​​may be black and white. Therefore, any pixel with color is converted to white / on format, while other pixels are converted to black / off format.

[0056] The new pixel value is mapped to the old pixel value.

[0057] The image is then bilinearly scaled with a scaling factor no greater than 2. This scaling preserves the original pixel data. Therefore, in one embodiment, the second plurality of processors 130 may be a SoC (System-on-a-Chip).

[0058] In one embodiment, the scaling factor is 2.

[0059] While scaling the data reduces the required processing power, the measurement error increases as the number of scaling iterations increases and the percentage partitioning decreases. Therefore, lookup tables apply expansion curves to correct for this error.

[0060] The lookup table can be loaded from at least one memory device 140 into the processor 130 and stored there during the execution of the approximation process. Therefore, iterative applications using lookup tables with applied expansion curves can be utilized, such as... Figure 3 and4 As shown. Figure 3 As shown, the expansion curve corrects for the error caused by shrinkage.

[0061] consider Figure 4 The x2 and x3 iterations of the lookup table with an expansion curve were applied. Even for high iteration scaling, the error caused by scaling was reduced to near zero. Figure 4 As shown, the percentage of LEDs evaluated as on is compared to the number of scaling iterations that occurred. After x3 scaling iterations, the percentage of LEDs on becomes a substantially linear number. Therefore, further scaling iterations may not be necessary. Thus, the effect of scaling as a computational simplification technique becomes visible to the user.

[0062] Once the approximation process is complete, where here there is an 8-fold reduced approximation, the information is compiled into a dataset. The final result of the iterative scaling and curve expansion process is highly compressed data. For example... Figure 2 As shown, in one example, the reduction is 8X reduction. This is a 2x scaling factor, which has been iterated three times (2^3). 3 This process achieves an 8x reduction factor. In this example, the compression is ~256x solely due to this process. This enables a smart FALD implementation without requiring additional processing hardware.

[0063] For example, a partition of approximately 80×80 pixels can be computationally compressed to 5×5 or a similar size. The CPU or first processor can then easily read the smaller piece of information and make lighting decisions based on that information.

[0064] Furthermore, pixel color formats with lower bandwidth consumption (lower bytes per pixel) can be used for CPU-based processing of compressed data and final analysis. The raw image material used for analysis is typically represented as RGB pixel data, with 4 bytes per pixel used for direct display feed. Embedded filter engines (as described above) typically support multiple color formats and "on-the-fly" color format conversion. After the first iteration, data can be stored / read in a format with fewer bytes per pixel—e.g., YUV NV12 with 2 bytes per pixel. This significantly reduces system bandwidth consumption during analysis. YUV color formatting includes luminance data (Y) and chrominance data (UV). However, as explained in further detail below, only luminance data is used for evaluation algorithm 310. Therefore, only 1 byte is used per pixel. The scaled and transformed dataset is used only by evaluation algorithm 310, so the loss of color and chrominance data does not need to be used for further processing and can be discarded from the transformed dataset to save memory and processing time.

[0065] Figure 2The process illustrates reformatting the image from RGB color format to YUV color format during this procedure, which can optionally be included to simplify software analysis. The previous lookup table has already converted pixels into black and white values ​​equivalent to on and off luminance data. Furthermore, to simplify the problem, chromaticity information (UV) may not be necessary for the evaluation algorithm. Therefore, potential loss of detail in chromaticity (UV) information may not affect the analysis at all, and in this format, the loss of important luminance (Y) information is minimal. The final analysis results can be stored in planar or half-planar YUV format.

[0066] Finally, the evaluation algorithm 310 examines the luminance (Y) value for each of the multiple partitions. If a luminance value indicating "on" (white) exists, the evaluation algorithm 310 determines that at least one LED 120 in that partition should be on. However, if no luminance value indicating "off" (black) exists, the evaluation algorithm determines that at least one LED in that partition should be off. Therefore, the CPU may only read and process the important luminance (Y) information, while not needing to read and potentially discarding the less important chrominance (UV) information.

[0067] An image can be divided into sections, each with a certain number of LEDs (e.g., one or more), or sections can be provided such that each section is centered around an LED. If each section has one LED, the illumination decision for that section may be based on the luminance values ​​of all pixels within that section. In other words, if any pixel within the section has a luminance value, the luminance is evaluated as "yes / on". However, if no pixel within the section has a luminance value, the luminance is evaluated as "no / off". In this case, the LED illuminating a particular pixel may not be directly behind the pixel itself, but may be close enough to provide sufficient backlight to illuminate the "yes / on" pixel. For example, for a 3×3 pixel section, the LED could be directly behind the center pixel. However, if any one of the nine pixels has an associated luminance, the illumination decision is "yes / on". For the other eight pixels in the section, the LED illuminating that pixel is not directly behind the pixel, but is close enough to the center LED to have sufficient luminance.

[0068] For a given display system 100, those skilled in the art can determine the level of sufficient brightness and the proximity of LEDs that can come from specific pixels to provide sufficient illumination.

[0069] Alternatively, multiple partitions can be determined by dividing the image into a number of pixels to create partitions of uniform pixel size, for example, 10×10 pixels per partition for the entire image. If multiple LEDs exist in a partition, the LED brightness decision may be the same for all LEDs. For example, if more than one LED exists in a partition, all LEDs in that partition may be illuminated, or no LEDs may be illuminated. If at least one pixel has a brightness value, the illumination decision may illuminate all LEDs. This embodiment may result in some LEDs being unnecessarily illuminated. However, the overall processing power for making illumination decisions is less than making decisions individually for each LED.

[0070] In one embodiment, each partition can have a luminance value, regardless of the number of pixels associated with that particular partition. If any pixel has luminance, the luminance value for that partition will be on. This information can be saved as part of a dataset and converted to YUV format. Therefore, the location of one or more specific pixels(s) within a partition that have luminance values ​​and require illumination may not be known.

[0071] The amount of illumination provided by each LED decreases with increasing distance from the LED. Therefore, a pixel directly above the center of the LED will be brighter than a pixel further away. Depending on the division into multiple zones, for example, some pixels on or near the periphery of a zone may receive less illumination from the LED than pixels closer to the center of the zone.

[0072] If one or more pixels with luminance values ​​and therefore need to be illuminated by LEDs are located on or near a zone, the amount of illumination provided by the associated LEDs may be lower than or even less than expected. Therefore, it may be necessary to illuminate neighboring zones as well as the LEDs within that zone. Thus, an additional step in the lighting decision for each zone could be to evaluate the luminance values ​​of each neighboring zone.

[0073] Those skilled in the art will be able to determine the number of zones, the size of the zones, and the number of LEDs in each zone for a particular display system 100, including based on the size of the entire display 200, the display definition, the number of LEDs, the available processing power 120, 130, and the desired brightness value of the display system 100.

[0074] Figure 5A method 500 according to one embodiment is illustrated. The method of providing full-array local dimming to a display includes: dividing an image into multiple partitions 502 using a first processor; applying an image processing algorithm 504 using a second processor, including: determining new pixel values ​​using a lookup table (LUT) 506; mapping the new pixel values ​​to old pixel values ​​508; and bilinearly scaling the image 510. The determination, mapping, and scaling are repeated 514 until an approximation is reached. The result is compiled into a dataset 512. The dataset is converted to a YUV image format and saved to memory.

[0075] As previously mentioned, this method can be executed on a single processor and eliminates the step of sending a dataset from one processor to another.

[0076] Furthermore, after determining new pixel values, mapping values, and scaling the image any number of times, including after only one iteration, the dataset can be compiled and converted into the YUV format to be saved.

[0077] Furthermore, the total number of iterations performed and the approximation to be achieved can be determined to balance the reduction of data and related processing with the amount of error introduced by the scaling process. Those skilled in the art will be able to determine the desired number of iterations and approximations for a particular display system 100, including the number, size, and speed of the display, processor, memory, etc.

[0078] In step 518, dataset 512 is sent from second processor 130 to first processor 110. The method also includes executing an evaluation algorithm on the first processor to make lighting decisions 520 for each of the multiple partitions based on the dataset.

[0079] Therefore, the evaluation algorithm can also include making lighting decisions for each of the multiple partitions.

[0080] In addition, lighting decisions are also made based on the lighting values ​​of each adjacent partition.

[0081] Finally, due to the reduced computational power required for the evaluation algorithm, the method can also be executed at a frequency matched to the display system 100. For example, the display system 100 may have a refresh rate of 60 Hz. The refresh rate of the display system 100 can be selected for several parameters, which may include, but are not limited to, the processing of the full-array dimming method described herein.

[0082] The invention has been described herein in an illustrative manner, and it should be understood that the terminology used is intended to describe the nature of the words and not to limit them. Clearly, many modifications and variations of the invention are possible based on the foregoing teachings. The invention may be practiced in ways other than those specifically described within the scope of the appended claims.

Claims

1. A method for providing full-array local dimming to a display, comprising: The image processing algorithm is executed using a processor with instructions, the instructions being used to: For each of the multiple pixels in the image, a new pixel value is determined based on a lookup table, which is used to convert the pixel into a black and white value equivalent to on and off brightness data, and the lookup table applies an expansion curve to correct for errors caused by scaling. For each of a plurality of pixels, a new pixel value is mapped to a previous pixel value, the previous pixel value including the color associated with each pixel, and the new pixel value being a black and white value; Bilinear scaling of images; The iterative application of determination, mapping, and scaling is repeated until an approximation is reached, reducing the scaling error to near zero. The iterative application uses a lookup table for applying the expansion curve. Compile duplicate results into a dataset; and The image on the display is divided into multiple zones, each zone having at least one LED associated with it; The processor makes lighting decisions based on the dataset, wherein the lighting decisions are for at least one LED associated with one of a plurality of zones.

2. The method of claim 1, further comprising making lighting decisions for each of the plurality of zones.

3. The method of claim 2, wherein dividing the image into multiple partitions further comprises dividing the image such that each of the multiple partitions has an LED associated therewith.

4. The method of claim 1, further comprising using luminance data of pixels in one of the plurality of partitions to make lighting decisions.

5. The method of claim 4, further comprising determining the lighting decision as yes when at least one pixel having brightness exists in one of the plurality of partitions.

6. The method of claim 5, further comprising, when the lighting decision is yes, at least one LED associated with one of a plurality of zones.

7. The method of claim 1 further includes converting the compiled dataset into a YUV image format.

8. The method of claim 1, wherein a first processor is used to divide the image into multiple partitions and make the lighting decision, and a second processor is used to execute the image processing algorithm.

9. The method of claim 8, further comprising sending the dataset from the second processor to the first processor prior to the lighting decision.

10. The method of claim 8, further comprising sending the dataset from the second processor to the first processor before dividing the image into multiple partitions.

11. The method of claim 8, wherein the second processor is one of a plurality of systems-on-a-chip.

12. The method of claim 8, wherein the second processor is connected to at least one memory, and wherein the memory includes a lookup table for extended curve values.

13. The method of claim 1, wherein the determination, mapping, and scaling are repeated three times in iterations.

14. A method for providing full-array local dimming to a display, comprising: An image on a display with multiple pixels is divided into multiple partitions, each partition having at least one LED associated with it; The image processing algorithm is executed using a processor with instructions, the instructions being used to: For each of the multiple pixels, a new pixel value is determined based on a lookup table, which is used to convert the pixel into a black and white value equivalent to on and off brightness data, and the lookup table applies an expansion curve to correct for errors caused by scaling. For each of a plurality of pixels, a new pixel value is mapped to a previous pixel value, the previous pixel value including the color associated with each pixel, and the new pixel value being a black and white value; Bilinear scaling of the partitioned image; The iterative application of determination, mapping, and scaling is repeated until an approximation is reached, reducing the scaling error to near zero. The iterative application uses a lookup table for applying the expansion curve. Compile duplicate results into a dataset; and The processor makes lighting decisions based on the dataset, wherein the lighting decisions are for at least one LED associated with one of a plurality of zones.

15. The method of claim 14, further comprising making lighting decisions for each of the plurality of zones.

16. The method of claim 15, wherein dividing the image into a plurality of partitions further comprises dividing the image such that each of the plurality of partitions has an LED associated therewith.

17. The method of claim 14, further comprising using luminance data of pixels in one of the plurality of partitions to make an illumination decision, and determining the illumination decision as yes when at least one pixel having luminance exists in one of the plurality of partitions.

18. The method of claim 17, further comprising, when the lighting decision is yes, at least one LED associated with one of a plurality of zones.

19. The method of claim 14, further comprising converting the compiled dataset into a YUV image format.

20. The method of claim 14, wherein dividing the image into multiple partitions and making the lighting decision is performed using a first processor, and executing the image processing algorithm is performed using a second processor.

Images