Display system calibration

Abstract

This provides a method for calibrating a display system. [Solution] A test pattern for a display is presented. The test pattern includes multiple blobs. A camera is used to capture an image containing one or more of the multiple blobs in the presented test pattern. An exposure calibration parameter is calculated using the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern. The camera exposure of the display system is calibrated using the exposure calibration parameter. A display system configured to implement this method is also provided.

Description

Technical Field

[0001] This description generally relates to systems and methods for calibrating a display system, and more specifically, to systems and methods for automatically calibrating the exposure of a camera component of a display system using a single captured image.

Background Art

[0002] A display system typically includes one or more projectors, cameras, or display devices. The display system may be used to project videos and images onto various different surfaces. In a projector-camera system, structured light is used to determine information about the system. It is necessary for the structured light to be properly exposed within the camera image in order to appropriately determine information about the system.

[0003] The most common method of automatic exposure is iterative, for example, using a method such as hill climbing to determine the correct exposure of the system. In such a method, a photo is taken at an initial exposure, and that photo is analyzed to determine whether the exposure should be increased or decreased. A new photo is taken at the new exposure. Depending on this analysis, the new exposure may be higher or lower than the previous photo. The new photo is analyzed to determine whether the new exposure should be increased or decreased. New photos are repeatedly taken and analyzed until an appropriate exposure is found.

[0004] Another method of automatic exposure involves taking multiple photos at different exposures. The photos at different exposures are combined to generate camera images that are all different combinations of exposures. This process is called high dynamic range (HDR).

[0005] Because iterative approaches and HDR require capturing multiple images, they are slow. Furthermore, they place a heavy load on the network due to the increased number of images that require communication.

[0006] Therefore, the purpose of this disclosure is to eliminate or mitigate at least some of these shortcomings. [Overview of the Initiative]

[0007] According to one embodiment, a method for calibrating a display system is provided. A test pattern is presented for the display. The test pattern includes a plurality of blobs. An image containing one or more of the plurality of blobs in the displayed test pattern is captured using a camera. An exposure calibration parameter is calculated using the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern. The exposure of the display system is calibrated using the exposure calibration parameter.

[0008] According to another embodiment, a display system is provided comprising: a device for displaying a test pattern including a plurality of blobs; a camera for capturing an image including one or more of the plurality of blobs in the displayed test pattern; and a processor configured to calculate an exposure calibration parameter using the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern, and to calibrate the exposure of the display system using the exposure calibration parameter.

[0009] In some embodiments, if the image is underexposed, the maximum luminance level is measured from the captured image. In some embodiments, if the image is overexposed, the maximum luminance level is calculated as the theoretical maximum luminance level of the displayed light, based on the maximum luminance level measured from the captured image. The theoretical maximum luminance level may be calculated by measuring the mid-range luminance level, considering the reference value of light read by the camera, and extrapolating the mid-range luminance level to obtain the theoretical maximum luminance level.

[0010] In some embodiments, at least one of a plurality of blobs includes an inner region and an outer region surrounding the inner region, wherein the inner region has a luminance level near a first end of the luminance range, and the outer region has a luminance level near a second opposite end of the luminance range. The outer region may have a luminance level at the maximum level of the luminance range, and the inner region may have a luminance level at or near the minimum level of the luminance range. The blob may include a clear boundary between the inner and outer regions. The blob may include a slope that transitions between the inner and outer regions.

[0011] In some embodiments, the blob may further include a background region surrounding the outer region, the background region having a luminance level at the opposite end of the luminance range from the outer region. The blob may include a clear boundary between the background region and the outer region.

[0012] In some embodiments, the blobs may be arranged in an asymmetrical pattern, and blobs at different locations within the asymmetrical pattern may be displayed in different colors, the colors of which may be identified by the detection location of the blobs within the asymmetrical pattern.

[0013] In some embodiments, the blobs may have different shapes, and blobs having different shapes may be displayed in different colors, the colors of which may be identified by the detected shape of the blob.

[0014] In some embodiments, multiple local exposure calibration parameters may be calculated for a corresponding region, and a broad exposure calibration parameter may be calculated as a mixture of multiple local exposure calibration parameters. The ratio of the multiple local exposure calibration parameters to the broad exposure calibration parameter can be used to calculate the corresponding local threshold. [Brief explanation of the drawing]

[0015] Here, embodiments will be described for illustrative purposes only with reference to the following drawings.

[0016] [Figure 1] This is a block diagram of the display system. [Figure 2] This graph shows the ideal relationship between projected light and captured light. [Figure 3] This graph shows the relationship between projected light and captured light in the presence of ambient light. [Figure 4] This graph shows the relationship between projected light and captured light in an overexposed image. [Figure 5] This graph shows the relationship between projected light and captured light in an underexposed image. [Figure 6] This is a flowchart showing a method for calibrating a display system. [Figure 7] This is a block diagram of a display system that projects multiple blobs onto a surface. [Figure 8] Here are examples of different types of blobs that may be used. [Figure 9] This is an example of multiple blobs of different colors arranged in an asymmetrical pattern. [Figure 10] This is an example of multiple blobs of different colors, represented by different shapes. [Figure 11] These are exemplary images of multiple blobs at different exposures. [Modes for carrying out the invention]

[0017] For the sake of simplicity, the same reference numerals in the description refer to the same structures in the drawings. Referring to FIG. 1, a block diagram of a display system is schematically shown by reference numeral 100. The display system 100 includes a projector 102, a camera 104, a memory 106, and a processor 108. The display system 100 may also include a communication interface 110 and / or an input / output device (not shown).

[0018] The projector 102 is configured to project light or an image onto a surface 120. In some embodiments, the surface may be a screen, a wall, a table, or other flat or three-dimensional (3D) object. The projector 102 functions as an output and displays visual information. In some embodiments, the visual information output from the projector 102 may be dynamically updated based on the input received from the camera 104.

[0019] The camera 104 captures an image of the surface 120. The captured image includes the image projected onto the surface 120 by the projector 102. The captured image may also include the surrounding environment of the surface 120. The camera 104 can also capture a video. The camera 104 functions as the eyes of the display system 100, enabling it to monitor changes, detect motion, or record feedback from the user.

[0020] In some embodiments, processor 108 may be a central processing unit (CPU), a microcontroller, a processing core, or the like. Processor 108 can comprise a single processor 108 or multiple cooperative processors 108. In some embodiments, the functionality implemented by processor 108 may be implemented by one or more specially designated hardware and firmware components such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), etc. In some embodiments, processor 108 may be a dedicated processor that may be implemented via the dedicated logic circuitry of an ASIC or FPGA to improve the processing speed of the calibration operations described herein.

[0021] Processor 108 is communicatively coupled to memory 106. Memory 106 is a non-transitory computer-readable storage medium. In some embodiments, the memory can include a combination of volatile memory such as, for example, random access memory (RAM), and non-volatile memory such as, for example, read only memory (ROM), electrically erasable programmable read only memory (EEPROM), or flash memory.

[0022] Memory 106 stores data including applications, application data, configuration data, calibration data, etc. For example, a calibration application is stored in the memory. When executed by processor 108, this calibration application causes the processor to calibrate display system 100 as described in more detail below. The calibration application may be implemented as a stand-alone application or as part of a suite of separate applications.

[0023] The communication interface 110 is communicatively coupled to the processor 108. The communication interface 110 includes appropriate hardware such as transmitters, receivers, transceivers, and network interface controllers to facilitate communication between the display system 100 and other computing devices. Specific components of the communication interface 110 can be selected based on the type of communication medium and protocol used. In some embodiments, the camera 104 may not be an integrated unit with the projector 102. In such embodiments, the communication interface 110 is configured to facilitate the communication of images from the camera 104 to the processor 108 and memory 106.

[0024] Camera 104 is calibrated to a target exposure. In some embodiments, the information required from the captured image is binary, such as ON vs. OFF. In such embodiments, any exposure that can distinguish between ON and OFF in binary is sufficient. In some embodiments, the target exposure is the optimal exposure for obtaining as much information as possible from the captured image.

[0025] If the exposure setting of camera 104 results in too little light reaching the sensor, the resulting image will be underexposed. Underexposed images are dark and may lose detail in shadows and highlights. Conversely, if the exposure setting of camera 104 results in too much light reaching the sensor, the resulting image will be overexposed. Overexposed images are bright and may lose detail in highlights.

[0026] As described, camera 104 is calibrated to the target exposure. As explained above, in some embodiments, it is desirable to distinguish between shades of gray in the image. Therefore, the target exposure is set as high as possible while reducing the possibility of overexposing the image. This ensures that the image captured by camera 104 is not so overexposed that it cuts off data at the upper end of the brightness range of the projected light. In some embodiments, it is not necessary to distinguish between shades of gray, and binary information is desired. The target exposure can be set to a lower value, as long as the exposure is sufficient to distinguish between ON (projected light) and OFF (unprojected light). This lower value of the target exposure can improve the speed at which the image is captured. In some embodiments, it is desirable to see details in the shading of low light. The target exposure can be set to a higher value. This results in the upper end of the brightness range of the projected light being cut off, and the information contained therein is lost. Therefore, this approach should be considered if the information contained at the upper end of the brightness range of the projected light is of lower value than the information contained at the lower end of the brightness range of the projected light.

[0027] Referring to Figure 2, a graph illustrating the ideal relationship between the projected light from projector 102 and the captured light from camera 104 at ideal exposure is schematically shown by reference numeral 200. As shown, the projected light increases linearly from 0 to 100%. Similarly, the captured light from the camera increases linearly from 0 to 100%. As can be understood, in some embodiments, the projected light may not increase linearly but curvilinearly. In such embodiments, the curve may be known based on the projector model or can be determined from measurements, third-party sources, etc. As shown, when projector 102 is not projecting light, camera 104 does not read light. When projector 102 is projecting light at maximum brightness, camera 104 reads light at camera 104's maximum brightness.

[0028] The ideal relationship 200 assumes that there are no other light sources that may affect the amount of light read by the camera 104. In practice, this is often difficult to achieve because secondary ambient light sources are present. For example, natural light (sunlight, moonlight, starlight, and reflected light, etc.), artificial light (ceiling lights, wall lights, and diffuse lamps, etc.), and ambient light (streetlights, traffic lights, and other outdoor light sources, etc.) may also be captured by the camera 104. Therefore, even when the projector 102 is not projecting light, the camera 104 may capture a non-zero reference value of light. Referring to Figure 3, a graph illustrating the relationship between the projected light from the projector 102 and the light captured by the camera 104 in the presence of ambient light is schematically shown by the symbol 300. As shown, when the projector 102 is not projecting light, the camera 104 reads a reference value of light. When the projector 102 is projecting light at maximum brightness, the camera 104 reads the light at the camera 104's maximum brightness.

[0029] As explained above, if the exposure setting causes the image to be overexposed, the captured image may be cropped at the brighter end of the projected light. Referring to Figure 4, a graph illustrating the relationship between the projected light from the projector 102 and the captured light by the camera 104 when the image is overexposed is schematically shown by reference numeral 400. In the example shown, if the projector 102 is not projecting light, the camera 104 reads a value above the light reference value. Furthermore, if the projector 102 is projecting light only at approximately 40% of the projector 102's maximum brightness, the camera 104 reads the light at the camera 104's maximum brightness. Since the camera 104 reaches its maximum brightness at 40% of the projector's maximum brightness, it continues to do so across the entire range from 40% to 100% of the projector's maximum brightness. At 40% brightness, the camera 104 reads 100% brightness. From light projected from projector 102 at 40% brightness to light projected at 100% brightness, the additional light from projector 102 causes image artifacts, thus degrading the quality of the image captured by the camera.

[0030] Conversely, referring to Figure 5, a graph showing the relationship between the projected light from the projector 102 and the captured light from the camera 104 when the image is underexposed is schematically indicated by the reference numeral 500. In the example shown, when the projector 102 is not projecting light, the camera 104 reads a value below the non-zero reference value for light. When the projector 102 is projecting light at maximum brightness, the camera 104 reads a value below the camera 104's maximum brightness.

[0031] Referring to Figure 6, a flowchart illustrating a method for calibrating the display system 100 is schematically shown by reference numeral 600. In 602, a test pattern for the display is projected using a projector 102. The test pattern may be projected onto one or more different surfaces. The test pattern includes multiple blobs. In 604, an image is captured using a camera 104. The image includes one or more of the multiple blobs displayed on the test pattern. In 606, an exposure calibration parameter is calculated using the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern. In 608, the exposure of the camera 104 of the display system 100 is calibrated using the exposure calibration parameter. In some embodiments, the exposure is controlled by the amount of time the shutter of the camera 104 remains open.

[0032] Referring to Figure 7, a block diagram of the display system 100 for projecting an image is schematically shown by reference numeral 700. The projector 102 is configured to project a test pattern 702 onto one or more surfaces 120. Thus, the test pattern 702 includes features to facilitate the calibration of the projector 102 with respect to exposure, color, brightness, geometric arrangement, distortion, focus, color convergence, etc. Thus, the test pattern 702 is formed from a plurality of blobs 706. The blobs 706 may be ramps, squares, circles, other geometric shapes, or other suitable forms. Furthermore, each of the blobs 706 may have the same form as the other blobs 706, or the blobs 706 may have different forms.

[0033] Referring to Figure 8a, a linearly inclined blob is schematically indicated by reference numeral 800. In some embodiments, the linearly inclined blob 800 begins at one end 802 where the projected light is zero and increases linearly to the opposite end 804 where the projected light is 100%. However, it should be noted that when shown against a white background, it is difficult to determine where the linearly inclined blob 800 ends. Similarly, when shown against a black background, it is difficult to determine where the linearly inclined blob 800 begins. Therefore, in some embodiments, markers are placed at both ends of the linearly inclined blob 800. The markers can take the form of a predetermined symbol, letter, shape, set of dots, or other known identifier. The distance between the marker and the corresponding end of the linearly inclined blob 800 is known. Therefore, the markers can be used to identify the start and end of the linearly inclined blob 800 regardless of the background color. In some embodiments, the markers are projected using a moderate brightness level. In some embodiments, markers adjacent to one end 802 are projected with 100% or near-100% brightness, while markers adjacent to the opposite end 802 are projected with zero or near-zero brightness.

[0034] Referring to Figure 8b, a pyramidal blob is schematically represented by reference numeral 820. This pyramidal blob includes an inner region 822 and an outer region 824 surrounding the inner region 822. The inner region 822 has a luminance level near the first end of the luminance range. The outer region 824 has a luminance level near the second opposite end of the luminance range. In some embodiments, the outer region 824 has a luminance level at the maximum level of the luminance range, and the inner region 822 has a luminance level at or near the minimum level of the luminance range.

[0035] In the example shown, the inner region 822 has a luminance level of zero for the projected light, and the outer region 824 has a luminance level of 100% for the projected light. In the example shown, the pyramidal blob 820 is substantially square. Furthermore, in the example shown, the pyramidal blob 820 includes a sloping transition section 826 between the inner region 822 and the outer region 824.

[0036] In some embodiments, an inverted pyramidal blob 820 can be used, in which case the inner region 822 has a projected light with a brightness level of 100%, and the outer region 824 has a projected light with a brightness level of zero.

[0037] In some embodiments, the pyramidal blob 820 further includes a background region 828 surrounding the outer region 824. The background region 828 has a luminance level at the opposite end of the luminance range from the outer region. Thus, in the example shown, the background region 828 has a luminance level of zero for the projected light. When using the inverted form of the pyramidal blob 820, the background region 828 has a luminance level of 100% for the projected light.

[0038] The pyramidal blob 820 makes it easier to find the start and end of the slope transition 826 compared to the linear slope blob 800. Furthermore, it allows for the determination of information other than exposure. Because the pyramidal blob 820 is a smaller, more compact pattern, it allows for the determination of more local information.

[0039] Referring to Figure 8c, a donut-shaped blob is schematically represented by reference numeral 840. The donut blob 840 is similar to the pyramidal blob. However, the donut blob 840 has a clear boundary 842 between the inner region 822 and the outer region 824, rather than a sloping transition 826. Due to the absence of the sloping transition 826, the donut blob 840 may be more compact than the pyramidal blob 820. Furthermore, in the example shown, the inner region 822 has a brightness level that is close to zero but greater than zero when projected light.

[0040] In some embodiments, a pyramidal blob 820 or donut blob 840 having a brighter outer region 824 and a darker inner region 822 helps distinguish projected light from light from other external light sources that may appear in the image. For example, ambient light from a secondary light source, reflections, and sometimes camera noise may appear as noise blobs. For camera 104, a noise blob is bright on the inside and dark on the outside. A pyramidal and / or donut-shaped blob with a dark center is far less likely to be generated by noise. Therefore, using a dark center shape helps distinguish intentionally projected blobs from blobs caused by environmental noise.

[0041] Exposure calibration parameters can be calculated using the linearity of projected light. As mentioned above, in some cases, the projected light may be nonlinear. However, in these cases, the principle remains the same, although the calculation becomes slightly more complex depending on the properties of the curve. In the worst-case scenario, the curve can be calculated using the piecewise linear method.

[0042] When the camera captures an image containing one or more of several blobs, the image is processed. If the maximum brightness level determined from the captured blobs is lower than the maximum brightness level readable by the camera, the image is underexposed. The exposure calibration parameter (ECP) can be determined as the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern, as follows:

number

[0043] Next, the exposure of the display system can be calibrated as follows:

number

[0044] Referring to Figure 11a, exemplary images of multiple pyramidal blobs 820 are schematically shown by reference numeral 1100. This image shows what the camera 104 is expected to capture at the target exposure. In some embodiments, the brightness range read by the camera is in the range of 0 to 255. Thus, in some embodiments, the maximum brightness at the target exposure is read by the camera as 255, and the minimum brightness at the target exposure is read by the camera as 0.

[0045] Referring to Figure 11b, another exemplary image of multiple pyramidal blobs 820 is schematically shown by symbol 1130. In this example, image 1130 is an underexposed image at a target exposure 1100. In the example shown, the assumed input exposure is 100,000. However, the maximum brightness read by the camera is 195. The minimum brightness observed by the camera is 0. Therefore, it can be determined that image 1130 is underexposed. The exposure calibration parameters can be calculated as follows:

number

[0046] In this example, to reduce the possibility of overexposure in the image, the exposure is calculated based on a luminance of 250, rather than the maximum luminance of 255. The corrected exposure can be calculated as follows:

number

[0047] Under the adjusted exposure, the predicted new maximum brightness read by camera 104 is 195 * 1.28 = 249.6. Similarly, the new minimum brightness predicted to be read by camera 104 under the adjusted exposure is New Min = 0 * 1.28 = 0.

[0048] If the maximum brightness level determined from the captured blob is the maximum brightness level readable by the camera, the image may be overexposed. To determine if the image is overexposed, processor 108 compares the brightness levels captured by the camera to portions of the blob that are expected to be below the maximum brightness level. If these portions are also captured by camera 104 at the maximum brightness level, the image will be overexposed.

[0049] Determining exposure calibration parameters becomes somewhat more complex when the image is overexposed, because the maximum brightness level determined from the capture does not uniquely correspond to the maximum brightness level readable by the camera. Therefore, a theoretical maximum exposure level is calculated based on the maximum brightness level measured from the capture image. Using the theoretical maximum brightness level of the projected light, the exposure calibration parameters are determined as follows:

number

[0050] The theoretical maximum brightness level is calculated by measuring the mid-tone brightness level, considering the reference light value read by the camera, and extrapolating the mid-tone brightness level to obtain the theoretical maximum brightness level. The mid-tone brightness level is selected so that the mid-tone projection light value is not cropped in the captured image. The theoretical maximum exposure level can then be calculated as follows:

number

[0051] The above formula is derived from the geometry of the blob. For example, consider pyramidal blob 820 (the geometry of pyramidal blob 820, as well as the proportional areas of the outer region 824, the sloping transition region 826, and the inner region 822). In the case of overexposure, the outer region 824 is visible to the camera more widely than in its projected state. This is because a portion of the sloping transition region 826 is visible at 100% brightness rather than a lower brightness due to the overexposure.

[0052] Therefore, it is possible to determine how much of the gradient transition area 826 will appear as the outer region 824 to the camera. Thus, for example, if camera 104 starts reading 100% brightness at 80% of the projected brightness, the boundary between the outer region 824 and the gradient transition area 826 will appear at 80% rather than 100% on the gradient transition area 826. Thus, the gradient transition area 826 will be approximately 80% of the predicted size of the gradient transition area 826. This is called the gradient ratio. Thus, the theoretical camera reading of 100% projected light can be reformulated as follows:

number

[0053] Referring to Figure 11c, yet another exemplary image of multiple pyramidal blobs 820 is schematically shown by symbol 1160. In this example, image 1160 is an overexposed version of the image at a target exposure 1100. In the example shown, the assumed input exposure is 100,000. The maximum brightness read by the camera is 255. The minimum brightness observed by the camera is 20. The boundary between the outer region 824 and the slope transition region 826 appears at 64% instead of 100% on the slope transition region 826. Therefore, it can be determined that image 1130 is overexposed. The theoretical camera reading for 100% projection light can be calculated as follows:

number

[0054] Exposure calibration parameters can be calculated as follows:

number

[0055] The corrected exposure can be calculated as follows:

number

[0056] The new maximum brightness predicted to be read by camera 104 under the adjusted exposure is 255. Similarly, the new minimum brightness predicted to be read by camera 104 under the adjusted exposure is New Min=20 * 0.645 = 13.

[0057] The projected light may be projected onto different surfaces 704 at different angles. In addition, different regions of the surface 704 may have different reflective properties for many different reasons. Therefore, each local region of the image may have its own local target exposure calibration parameter. A broad target exposure calibration parameter can be calculated for the entire image. In some embodiments, the broad target exposure calibration parameter is a mixture of the local exposure calibration parameters for all local regions. In some embodiments, the local target exposure calibration parameters are averaged to obtain the broad target exposure calibration parameter. In some embodiments, some of the local target exposure calibration parameters are weighted more heavily than others. For example, the local target exposure calibration parameter at the center of the image may be weighted more heavily than the local target exposure calibration parameter at the edge of the image, or vice versa. To facilitate the determination of the broad target exposure calibration parameter, local regions can be identified by using patterns. Once identified, the local exposure calibration parameter can then be associated with its identified region of interest in the image. In some embodiments, the pattern is easily recognizable regardless of the initial exposure of the image. This may include using blobs of different colors, blobs with different geometric properties, asymmetrical blob arrangements, unique blob arrangements, or any combination thereof.

[0058] Once the exposure is calibrated, it is often useful to project and capture each primary color, typically red, green, and blue, separately to correct color convergence, color balance, and / or other projection characteristics. In some embodiments, the blobs 706 are arranged in an asymmetric pattern. Blobs at different positions within the asymmetric pattern are projected in different colors. The colors are identified by the detection position of the blobs 706 within the asymmetric pattern. Referring to Figure 9a, an example of an asymmetric pattern is schematically shown by the reference numeral 900. The shown asymmetric pattern 900 includes two red blobs 902 positioned to the left of a green blob 904 positioned below a blue blob 906. As can be understood, different combinations of blobs 706 can be used to generate different asymmetric patterns, as long as they are easily identifiable. The asymmetric pattern allows the display system 100, even with a monochrome camera, to determine which color corresponds to which blob. Referring to Figure 9b, a monochrome representation of the colors shown in Figure 9a is schematically shown by the reference numeral 950.

[0059] In some embodiments, the blob 706 has a different shape. Blobs 706 with different shapes are projected in different colors. The color is identified by the shape of the detected blob 706. Referring to Figure 10, examples of blobs with different shapes are schematically shown by reference numeral 1000. In the shown example 1000, the triangular blob 1002 is red, the square blob 1004 is green, and the circular blob 1006 is blue. As can be understood, a variety of shapes can be used as long as they are easily identifiable.

[0060] Thresholding can also benefit from the exposure calibration described above. Thresholding is a technique used in image processing to separate an object or region of interest from the background by converting an image to a binary format, where each pixel is assigned either a maximum or minimum value, typically representing black or white. This is often an initial step in processes such as image segmentation, object detection, and feature extraction.

[0061] Broad thresholding is a technique that applies a single threshold across the entire image. Pixels with brightness above the threshold are set to a certain color (usually white), and pixels below the threshold are set to a different color (usually black). Adaptive (local) thresholding is a technique that uses different thresholds for different areas of an image. This is particularly useful for images with uneven lighting or shading. Adaptive thresholding can handle changing lighting conditions within an image well by adjusting the threshold based on local pixel brightness.

[0062] Ideally, the threshold should be the midpoint between the minimum and maximum pixel values ​​predicted to be read by the camera in wide exposure. In good exposure, this is also 50% of the projected light. To determine the local threshold, the corresponding local target exposure calibration parameter is compared to the wide exposure calibration parameter used for the image. That is, the local threshold is calculated based on the ratio of the local target exposure calibration parameter to the wide exposure calibration parameter. Therefore, if the local target exposure calibration parameter is greater than the wide exposure calibration parameter, the projected light in that region is less than the wide projection light, so the threshold for threshold setting will be greater than 50% of the projected light. Conversely, if the local target exposure calibration parameter is smaller than the wide exposure calibration parameter, the projected light in that region is greater than the wide projection light, so the threshold for threshold setting will be less than 50% of the projected light. In this way, the camera readings of the behavior of the projected light in the region are predicted, so the local threshold can be calculated.

[0063] For example, consider that an exposure of 20 is ideal in a certain local area. That is, with an exposure of 20, 100% of the projected light corresponds to 100% of the light read by camera 104. However, due to the wide-area exposure calibration parameter, the exposure in the local area becomes 10. As a result, in the local area, 100% of the projected light corresponds to 50% of the light read by camera 104. Therefore, the local threshold is set to 25, rather than a threshold of 50.

[0064] Furthermore, the threshold can also be adjusted based on the camera reading at zero projection light. Specifically, the threshold can be adjusted to an intermediate point between the camera reading at zero projection light and the camera reading at 100% projection light. Refer again to the exemplary image shown in Figure 11b. In this example, the threshold (124) is calculated as half of the new maximum value (249) minus the new minimum value (0). Note that in this example, values ​​are truncated to expedite processing. In the example shown in Figure 11c, the threshold (121) is calculated as half of the new maximum value (255) minus the new minimum value (13). Similar to the calculation of exposure calibration parameters, this calculation assumes linearity. If either camera 104 or projector 102 is nonlinear, the calculation is adjusted for nonlinearity.

[0065] If the initial camera image is severely overexposed or underexposed and cannot detect Blob 706, another image can be taken. In this case, the camera exposure can be increased or decreased based on a predetermined constant. This constant may be specific to the type of camera, the camera's sensitivity, a typical use case, or any combination thereof. For example, if it is determined that the image is severely overexposed and cannot detect Blob 706, the camera exposure can be reduced, for example, to 1 / 16. Similarly, if the image is severely underexposed and cannot detect Blob 706, the camera exposure can be increased, for example, by 16. This coefficient can be varied based on experiments for each use case. For example, consider a brightness level of 12.5% ​​(1 / 8 of the maximum brightness level) for the inner region 822. If the captured image is severely overexposed and the camera reads the brightness level as 100%, it can be determined that the exposure is at least 8 times too high. Therefore, a coefficient of 16 is reasonable in this example.

[0066] Although the above description assumes that the display system uses projector 102, it should be understood that other devices (e.g., display panels using technologies such as light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), and quantum dot LEDs (QLEDs)) can be used to present images to the user.

[0067] The claims should not be limited by the embodiments described in the examples above, and the broadest interpretation consistent with the description as a whole should be given. The terms “a,” “an,” and “one” are defined as “at least one,” that is, unless otherwise specified, these terms do not exclude multiple items.

[0068] Terms such as “substantially,” “generally,” and “about” that modify the values, states, or features of the exemplary embodiments should be understood to mean that the values, states, or features are defined within the permissible range that is permitted for the proper operation of the embodiments for their intended use.

[0069] Unless otherwise specified, the terms “connected” and “joined,” as well as their derivatives and variations, refer to any direct or indirect structural or functional connection or joining between two or more elements. For example, the connection or joining between elements may be acoustic, mechanical, optical, electrical, thermal, logical, or any combination thereof.

[0070] The expression "based on" is intended to mean "based at least partially on," and therefore should not be interpreted restrictively, as it may also mean "based entirely on" or "based partially on." More specifically, the expression "based on" may also be understood to mean "dependent on," "representing," "indicating," "related to," or similar expressions.

[0071] "At least one" means one or more, and "multiple" means two or more. The term "and / or" describes related relationships between related objects and indicates that there may be three relationships. For example, A and / or B may indicate three cases: only A exists, both A and B exist, or only B exists. Here, A and B may be singular or plural. The symbol " / " usually indicates an "or" relationship between related objects. "At least one of the following items (elements)" or a similar expression indicates any combination of these items, including a single item (element), or any combination of multiple items (elements). For example, "at least one of A, B, or C" may include A, B, C, A and B, A and C, B and C, or A, B and C, and "at least one of A, B, and C" may also be understood to include A, B, C, A and B, A and C, B and C, or A, B and C. In addition, unless otherwise specified, the ordinal numbers such as "First" and "Second" in the embodiments of this application are used to distinguish between multiple subjects and are not used to limit the order, chronological order, priority, or importance of the multiple subjects.

[0072] Those skilled in the art should understand that embodiments of this application may be provided as methods, apparatus (or systems), computer-readable storage media, or computer program products. Accordingly, this application may take the form of hardware-only embodiments, software-only embodiments, or embodiments combining software and hardware. Furthermore, this application may take the form of computer program products implemented on one or more computer-readable storage media (including, but not limited to, disk memory, optical memory, etc.) containing computer-readable program code.

[0073] This application will be described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products described herein. It should be understood that computer program instructions may be used to implement each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams. Computer program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or another programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or another programmable data processing device generate a machine for implementing a particular function in one or more steps in a flowchart and / or one or more blocks in a block diagram.

[0074] Computer program instructions may instead be stored in computer-readable memory that can instruct the computer or another programmable data processing device to operate in a specific way, thereby generating artifacts that include instruction units. These instruction units implement specific functions in one or more steps in a flowchart and / or one or more blocks in a block diagram.

[0075] Computer program instructions may instead be loaded into a computer or another programmable data processing device, so that a series of operations and steps are executed on the computer or another programmable device, thereby generating the computer implementation process. Therefore, instructions executed on the computer or another programmable device provide steps for implementing a particular function in one or more steps in a flowchart and / or one or more blocks in a block diagram.

[0076] It will be apparent to those skilled in the art that various modifications and variations can be made to this application without departing from its scope. This application is intended to encompass these modifications and variations to the extent that they fall within the scope of protection defined by the following claims and their equivalents.

Claims

1. A method for calibrating a display system, wherein the method is A step of presenting a test pattern for display, wherein the test pattern includes a plurality of blobs, The steps include capturing an image using a camera that includes one or more of the multiple blobs in the presented test pattern, The steps include calculating an exposure calibration parameter using the ratio of the maximum brightness level readable by the camera to the maximum brightness level determined from one or more captured blobs in the test pattern, A method comprising the step of calibrating the exposure of the camera of the display system using the exposure calibration parameters.

2. The method according to claim 1, wherein if the image is underexposed, the maximum brightness level is measured from the captured image.

3. The method according to claim 1, wherein if the image is overexposed, the maximum brightness level is calculated as the theoretical maximum brightness level of the presented light based on the maximum brightness level measured from the captured image.

4. The method according to claim 3, wherein the theoretical maximum brightness level is calculated by measuring the intermediate brightness level, taking into account the reference value of light read by the camera, and extrapolating the intermediate brightness level to obtain the theoretical maximum brightness level.

5. The method according to claim 1, wherein at least one of the plurality of blobs includes an inner region and an outer region surrounding the inner region, the inner region having a luminance level near a first end of the luminance range, and the outer region having a luminance level near a second opposite end of the luminance range.

6. The method according to claim 5, wherein the outer region has a luminance level at the maximum level of the luminance range, and the inner region has a luminance level at or near the minimum level of the luminance range.

7. The method according to claim 5, wherein the blob includes a clear boundary between the inner region and the outer region.

8. The method according to claim 5, wherein the blob includes a slope that transitions between the inner region and the outer region.

9. The method according to claim 5, further comprising a background region surrounding the outer region, wherein the background region has a luminance level at the end opposite to the outer region in terms of luminance range.

10. The method according to claim 9, wherein the blob includes a clear boundary between the background region and the outer region.

11. The blobs are arranged in an asymmetrical pattern. Blobs at different positions within the aforementioned asymmetrical pattern are presented in different colors. The method according to claim 5, wherein the color is identified by the detection position of the blob in the asymmetric pattern.

12. The blob has a different shape, Blobs of different shapes are presented in different colors. The method according to claim 5, wherein the color is identified by the detected shape of the blob.

13. The method according to claim 1, wherein multiple local exposure calibration parameters are calculated for a corresponding area, and a wide-area exposure calibration parameter is calculated as a mixture of the multiple local exposure calibration parameters.

14. The method according to claim 13, wherein a corresponding local threshold is calculated using the ratio of the plurality of local exposure calibration parameters to the wide-area exposure calibration parameter.

15. Display system and, Camera and, A display system comprising a processor configured to implement the method described in any one of claims 1 to 14.

Images