Method, host and computer-readable storage medium for determining ambient light brightness

Abstract

Embodiments of the present application provide a method, a host and a computer readable storage medium for determining ambient light brightness. The method comprises: obtaining a first frame and a second frame, wherein the first frame comprises a plurality of first ROIs (Regions of Interest) and the second frame comprises a plurality of second ROIs, and each first ROI corresponds to a second ROI; in response to determining that the first ROIs comprise at least one specific ROI and at least one first candidate ROI satisfying a predetermined condition, obtaining at least one second candidate ROI in the second ROIs, wherein each second candidate ROI corresponds to the at least one specific ROI; and determining the ambient light brightness based on the at least one first candidate ROI and the at least one second candidate ROI.

Description

Technical Field

[0001] The present invention generally relates to an environment detection mechanism, and more particularly, to a method, a host, and a computer-readable storage medium for determining ambient light luminance. Background Art

[0002] Typically, head-mounted displays (HMDs) use additional light detectors (such as ambient light sensors or cameras) to detect ambient light levels and control the shutter transmittance and display brightness accordingly. However, installing additional light detectors results in low space efficiency and high costs. Summary of the Invention

[0003] Therefore, the present invention relates to a method for determining ambient light brightness, a host, and a computer-readable storage medium that can be used to solve the above technical problems.

[0004] An embodiment of the present invention provides a method for determining ambient light brightness, applicable to a host. The method includes: obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the first ROIs respectively correspond to the second ROIs; in response to determining that the first ROI includes at least one specific ROI that meets a predetermined condition and at least one first candidate ROI, obtaining at least one second candidate ROI from the second ROIs, wherein the at least one second candidate ROI respectively corresponds to the at least one specific ROI; and determining the ambient light brightness based on the at least one first candidate ROI and the at least one second candidate ROI.

[0005] An embodiment of the present invention provides a host comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to perform the following steps: obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the first ROIs correspond to the second ROIs respectively; in response to determining that the first ROI includes at least one specific ROI that satisfies a predetermined condition and at least one first candidate ROI, obtaining at least one second candidate ROI among the second ROIs, wherein the at least one second candidate ROI corresponds to the at least one specific ROI respectively; and determining ambient light brightness based on the at least one first candidate ROI and the at least one second candidate ROI.

[0006] An embodiment of the present invention provides a non-transitory computer-readable storage medium recording an executable computer program, which is loaded by a host to perform the following steps: obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the first ROIs respectively correspond to the second ROIs; in response to determining that the first ROI includes at least one specific ROI that satisfies a predetermined condition and at least one first candidate ROI, obtaining at least one second candidate ROI among the second ROIs, wherein the at least one second candidate ROI respectively corresponds to the at least one specific ROI; and determining ambient light brightness based on the at least one first candidate ROI and the at least one second candidate ROI. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

[0008] Figure 1 A schematic diagram illustrating a host according to an embodiment of the present invention is shown.

[0009] Figure 2 A flow chart of a method for determining ambient light brightness according to an embodiment of the present invention is shown.

[0010] Figure 3A FIG. 4 shows the corresponding relationship between different camera intensities and ambient light brightness according to an embodiment of the present invention.

[0011] Figure 3B FIG. 4 shows the corresponding relationship between different camera intensities and ambient light brightness according to an embodiment of the present invention.

[0012] Figure 3C Show the basis Figure 3A and Figure 3B The correspondence between different camera intensities and ambient light brightness.

[0013] Figure 4 FIG. 1 is a flowchart illustrating a mechanism for determining a specific ROI and a first candidate ROI according to an embodiment of the present invention.

[0014] Figure 5 A flowchart of determining the ambient light brightness based on the first candidate ROI and the second candidate ROI according to an embodiment of the present invention is shown.

[0015] Figure 6 An application scenario of determining ambient light brightness according to the first embodiment of the present invention is shown.

[0016] Figure 7 An application scenario of determining ambient light brightness according to the second embodiment of the present invention is shown.

[0017] Figure 8 A schematic diagram illustrating a host according to an embodiment of the present invention is shown.

[0018] [Explanation of Symbols]

[0019] 100, 800: host

[0020] 102: Storage Circuit

[0021] 104: Processor

[0022] 301, 302: Curve

[0023] 610, 710: First frame

[0024] 611, 612, 613, 614: First ROI

[0025] 620: Second frame

[0026] 621, 622, 623, 624: Second ROI

[0027] 711, 721: District

[0028] 720: Second frame

[0029] 811, 821: Display

[0030] 812, 822: M×N shutter transmittance controller

[0031] 813, 823: Light source

[0032] 814, 824: Timing controller

[0033] 830: Image buffer

[0034] 841, 842~84K: Tracking camera

[0035] S210, S220, S230, S410, S420, S430, S440, S450, S510, S520, S530, S540, S550: Steps DETAILED DESCRIPTION

[0036] Reference will now be made in detail to the presently preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0037] See also Figure 1 , Figure 1 A schematic diagram of a host according to an embodiment of the present invention is shown. In various embodiments, the host 100 can be implemented as any smart device and / or computer device.

[0038] The storage circuit 102 is one or a combination of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and records a plurality of modules executable by the processor 104 .

[0039] The processor 104 may be coupled to the memory circuit 102 and may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like.

[0040] In one embodiment, host 100 may be implemented as a tracking device capable of performing, for example, inside-out tracking and / or outside-in tracking. In one embodiment, the tracking device may be a wearable device such as a head-mounted display (HMD). In some embodiments, the HMD may be used to provide reality services (e.g., augmented reality (AR) services, virtual reality services, and / or similar services) by displaying corresponding visual content to the wearer, but the present invention is not limited thereto.

[0041] In one embodiment, the host 100 may be provided with one or more tracking cameras for capturing images at a frame rate (e.g., 60 frames per second (FPS)), wherein the images may be used to perform tracking functions such as inside-out tracking.

[0042] In one embodiment, the images captured by the tracking camera include a plurality of bright frames and a plurality of dark frames. In one embodiment, the bright frames and the dark frames are interwoven with each other. That is, the preceding and following images of a bright frame are both dark frames, and the preceding and following images of a dark frame are both bright frames.

[0043] In some embodiments, some of the light frames may be captured continuously. For example, the tracking camera may be configured to capture several light frames continuously after capturing one dark frame. In some embodiments, some of the dark frames may be captured continuously. For example, the tracking camera may be configured to capture several dark frames continuously after capturing one light frame. In some embodiments, the tracking camera may be configured to capture several dark frames continuously after capturing several light frames continuously. In one embodiment, the tracking camera may be configured to capture several light frames continuously after capturing several dark frames continuously, but the present invention is not limited thereto.

[0044] In one embodiment, when the tracking camera is configured to capture one or more bright frames, the corresponding image capture parameters used by the tracking camera (e.g., exposure time and / or gain that collectively form a first set of image capture parameters) are automatically determined to be appropriate values ​​so that the luma (i.e., the average value of grayscale values ​​in a specific region) of certain ROIs in the display of the host 100 (e.g., an HMD) is within a specific range. In this case, bright frames used to perform tracking (e.g., inside-out tracking) are less likely to have overexposure issues.

[0045] In one embodiment, when the tracking camera is configured to capture one or more dark frames, the corresponding image capture parameters used by the tracking camera (e.g., exposure time and / or gain that together form a second set of image capture parameters) are fixed to obtain appropriate feature images for performing tracking.

[0046] In one embodiment, a first image capture parameter set for capturing a bright frame may exemplarily include a first exposure time and a first gain, and a second image capture parameter set for capturing a dark frame may exemplarily include a second exposure time and a second gain. In one embodiment, the first image capture parameter set and the second image capture parameter set are exemplarily shown in Table 1 below.

[0047]

[0048] Table 1

[0049] In the example shown in Table 1, the first exposure time of the tracking camera used to capture bright frames ranges between a minimum exposure time (referred to as Exp(min)) and a maximum exposure time (referred to as Exp(max)). In one embodiment, Exp(min) may be several microseconds, and Exp(max) may be several milliseconds, but the present invention is not limited thereto. The first gain of the tracking camera used to capture bright frames ranges between a minimum gain (referred to as Gain(min)) and a maximum gain (referred to as Gain(max)). In one embodiment, Gain(min) may be 1x (i.e., maintained at a predetermined gain), and Gain(max) may be several times the predetermined gain (e.g., 16 times the predetermined gain (referred to as 16x)), but the present invention is not limited thereto. Additionally, in the example shown in Table 1, the second exposure time and second gain of the tracking camera used to capture dark frames are fixed at Exp(min) and Gain(min), respectively.

[0050] In an embodiment of the present invention, the ambient light level can be determined while maintaining the settings of the tracking camera (e.g., maintaining the frame rate, the number of dark / light frames, and image capture parameters). In this case, it is not necessary to install a light detector for ambient light level detection on the host 100, thereby improving the space efficiency and cost of the host 100.

[0051] From another perspective, the present invention can use the information originally used for tracking to estimate the ambient light brightness, thereby allowing the host 100 to determine the ambient light brightness without configuring a light detector.

[0052] In an embodiment of the present invention, the processor 104 may access the module stored in the storage circuit 102 to implement the method for determining the ambient light brightness provided in the present invention, which will be further discussed below.

[0053] See also Figure 2 , Figure 2 A flow chart of a method for determining ambient light brightness according to an embodiment of the present invention is shown. The method of this embodiment can be performed by Figure 1 The host 100 in the embodiment is used to execute the Figure 1 The components shown in Figure 2 Details of each step in .

[0054] In step S210, the processor 104 obtains a first frame (referred to as F1) and a second frame (referred to as F2). In one embodiment, the processor 104 obtains a plurality of first specific frames and a plurality of second specific frames. In some embodiments, the processor 104 may control a camera (e.g., a tracking camera) of the host 100 to capture the first specific frames and the second specific frames, but the present invention is not limited thereto.

[0055] In one embodiment, the first specific frame is a bright frame captured by the tracking camera based on a first set of image capture parameters, and the second specific frame is a dark frame captured by the tracking camera based on a second set of image capture parameters. In one embodiment, the processor 104 retrieves one of the first specific frames as the first frame F1 and retrieves one of the second specific frames as the second frame F2.

[0056] In an embodiment where the first specific frame is a light frame and the second specific frame is a dark frame, the processor 104 may retrieve one of the light frames as the first frame F1 and retrieve one of the dark frames as the second frame F2. To better illustrate the following concepts, the first specific frame will be assumed to be a light frame and the second specific frame will be assumed to be a dark frame, but the present invention is not limited thereto. In one embodiment, the first specific frame and the second specific frame are interleaved with each other (i.e., the light frame and the dark frame are interleaved with each other).

[0057] In one embodiment, the first specific frame and the second specific frame are captured based on the frame rate of the camera. For example, if the frame rate is 60 FPS, the camera will capture an image every 16.66 milliseconds, and this image may be one of the first specific frame or the second specific frame.

[0058] In one embodiment, the time difference between the first frame F1 and the second frame F2 may be less than a predetermined time threshold. That is, as long as the captured timing of a bright frame is sufficiently close to the captured timing of a dark frame, the processor 104 may determine the bright frame and the dark frame as the first frame F1 and the second frame F2, respectively.

[0059] In one embodiment, the time difference between the first frame F1 and the second frame F2 is equal to the inverse of the frame rate. In this case, the first frame F1 may be the previous image or the next image of the second frame F2.

[0060] In one embodiment, processor 104 determines that first frame F1 includes multiple first ROIs. In various embodiments, the first ROIs may be designed to include one or more pixels of first frame F1. In embodiments where each first ROI includes one pixel, if the size of first frame F1 is W×H (W is the width of first frame F1 and H is the height of first frame F1), processor 104 may determine that there are W×H first ROIs in first frame F1. In embodiments where each first ROI includes multiple pixels, the pixels in each first ROI and the position of each first ROI in first frame F1 may be determined based on designer requirements. In one embodiment, the first ROIs do not overlap.

[0061] In one embodiment, the processor 104 determines that the second frame F2 includes a plurality of second ROIs, wherein the first ROIs correspond to the second ROIs. For example, in an embodiment where each first ROI includes one pixel and the first frame F1 includes W×H first ROIs, each second ROI also includes one pixel of the second frame F2, and the second frame F2 is determined to include W×H second ROIs.

[0062] From another perspective, the first ROI in the first frame F1 corresponds to the second ROI in the second frame F2 in a one-to-one manner. Specifically, for one of the first ROIs having a portion of a pixel in the first frame F1, there will be one of the second ROIs having a corresponding portion of the pixel in the second frame F2.

[0063] In one embodiment, after determining a first ROI in first frame F1 and a second ROI in second frame F2, processor 104 determines whether the first ROI includes at least one specific ROI that satisfies a predetermined condition and at least one first candidate ROI. Before making this determination, the term "camera strength" proposed by the present invention should be introduced.

[0064] Specifically, in conventional methods for comparing the brightness difference between two image regions (e.g., a first ROI / a second ROI), the brightness of the image regions is used as a reference. By definition, the brightness of an image region is the average grayscale value of the pixels in that image region. If the image regions being compared were captured using the same set of image capture parameters and do not suffer from overexposure, brightness will be an appropriate reference for brightness comparison.

[0065] However, for image regions (eg, first ROI / second ROI) captured based on different sets of image capture parameters, luminance will not be suitable for brightness comparison. In this case, it would be more appropriate to perform brightness comparison based on camera intensity of each image region.

[0066] In one embodiment, the camera intensity of an image region may be obtained based on the brightness of the image region and a corresponding set of image capture parameters. In instances where the set of image capture parameters for the image region includes exposure time and gain, the camera intensity of the image region may be characterized by dividing the brightness of the image region by the product of the corresponding exposure time and the corresponding gain.

[0067] In one embodiment, if the image area under consideration includes only one pixel, its brightness will be the grayscale value of the pixel therein, and thus the camera intensity can be characterized by dividing the grayscale value of the pixel by the product of the corresponding exposure time (ms) and the corresponding gain, but the present invention is not limited thereto.

[0068] To better illustrate the concept of camera strength, the following Table 2 will be used as an example.

[0069]

[0070] Table 2

[0071] In scene 1 shown in Table 2, a first image area is captured based on corresponding image capture parameters, and the brightness and camera intensity of the first image area are shown in Table 2. In scene 2 shown in Table 2, a second image area (which captures the same environment as the first image area) is captured based on corresponding image capture parameters, and the brightness and camera intensity of the second image area are shown in Table 2.

[0072] As can be seen from Table 2, although the ambient light brightness of scene 1 is almost twice that of scene 2, the brightness of the first image area is the same as the brightness of the second image area, which means that when the first image area and the second image area are captured based on different image capture parameters, the brightness is not appropriate for performing brightness comparison.

[0073] However, the camera intensity of the first image region is twice that of the second image region, which accurately represents the difference between the light intensity of the first image region and the light intensity of the second image region. That is, when the first image region and the second image region are captured based on different image capture parameters, the camera intensity is more appropriate for brightness comparison.

[0074] In one embodiment, the corresponding relationship between different camera intensities and corresponding ambient light brightness can be measured by conducting experiments in advance.

[0075] See also Figure 3A , Figure 3A The corresponding relationship between different camera intensities and ambient light brightness according to an embodiment of the present invention is shown. Figure 3A In FIG. 3 , the camera intensity of the ROI in the bright frame is shown as curve 301 under different ambient light levels.

[0076] If you can Figure 3A As can be seen from FIG, when the ambient light brightness is relatively low (eg, less than 10,000 lux) and the resolution is also high, the corresponding relationship between the camera intensity and the ambient light brightness is linear.

[0077] However, when the ambient light brightness is relatively high (e.g., above 10,000 lux), the correspondence between camera intensity and ambient light brightness becomes nonlinear. Specifically, the reason why the correspondence becomes nonlinear when the ambient light brightness is high is because when capturing a bright frame, the corresponding image capture parameters are automatically determined to obtain the feature points required for inside-out tracking in the bright frame. In this case, if some objects with high light intensity (e.g., light sources such as fluorescent lamps and / or floodlights) are captured in the bright frame, the relevant image areas in the bright frame will be overexposed, and the actual brightness will not be achieved.

[0078] Therefore, when the ambient light brightness is relatively low (for example, below 10,000 lux), use Figure 3A From another perspective, when the ambient light brightness is relatively high (for example, higher than 10000 lux), using Figure 3A The camera intensity in the image will be inaccurate to estimate the corresponding ambient light brightness.

[0079] See also Figure 3B , Figure 3B The corresponding relationship between different camera intensities and ambient light brightness according to an embodiment of the present invention is shown. Figure 3B In FIG. 3 , the camera intensity of the ROI in the dark frame is shown as curve 302 under different ambient light levels.

[0080] If you can Figure 3B As can be seen from FIG, for almost all ambient light levels, the corresponding relationship between camera intensity and ambient light level is linear, but when the ambient light level is low (eg, below 10,000 lux), the resolution is relatively low.

[0081] Specifically, as can be seen from the enlarged view of the camera intensity portion corresponding to the lower ambient light brightness range (e.g., below 5000 lux) (shown in FIG. Figure 3B The lower left of the two adjacent data points ( Figure 3B The difference between the two images (shown as triangles in FIG) is large, which indicates that the resolution is relatively low when the ambient light brightness is low.

[0082] Therefore, when the ambient light brightness is relatively high (for example, above 10000 lux), use Figure 3B The camera intensity in the image will be more accurate in estimating the corresponding ambient light brightness. Figure 3A and Figure 3B Related observations can be obtained Figure 3C .

[0083] See also Figure 3C , Figure 3C Show the basis Figure 3A and Figure 3BThe correspondence between different camera intensities and ambient light brightness. Figure 3C , it can be determined that when the ambient light brightness estimated based on the camera intensity of a bright frame (ROI) is higher than a brightness threshold (e.g., 10,000 lux), the estimated ambient light brightness will be inaccurate due to the nonlinearity of curve 301. In this case, the ambient light brightness estimated based on the camera intensity of a dark frame (ROI) corresponding to the considered bright frame will be more accurate.

[0084] In one embodiment, based on Figure 3C Based on the observation in , the processor 104 may determine whether the first ROI includes at least one specific ROI that meets a predetermined condition and at least one first candidate ROI.

[0085] See also Figure 4 , Figure 4 FIG. 1 is a flowchart illustrating a mechanism for determining a specific ROI and a first candidate ROI according to an embodiment of the present invention.

[0086] In step S410, processor 104 determines a first camera intensity for each first ROI based on the first image capture parameter set and the first luminance of each first ROI. In one embodiment, processor 104 may determine the first camera intensity for each first ROI by dividing the first luminance of each first ROI by the product of the first exposure time and the first gain. For example, if the first luminance of a first ROI is L1 (i.e., the average grayscale value in the first ROI), the first exposure time is Exp1, and the first gain is G1, then the first camera intensity for the first ROI may be represented as L1 / (Exp1·G1), but the present invention is not limited thereto.

[0087] In step S420, the processor 104 may determine a first estimated brightness for each first ROI based on the first camera intensity of each first ROI. In one embodiment, the processor 104 may determine the first estimated brightness for each first ROI by searching a first lookup table for a result corresponding to the first camera intensity of each first ROI as the first estimated brightness for each first ROI.

[0088] In one embodiment, based on Figure 3A In this case, once a first camera intensity of a first ROI is obtained, the processor 104 can use Figure 3A The corresponding ambient light brightness in is used as the corresponding first estimated brightness. For example, according to Figure 3A , if the first camera intensity of a first ROI is 2000, the corresponding first estimated brightness will be about 6000. For another example, according to Figure 3A, if the first camera intensity of a first ROI is 6000, the corresponding first estimated brightness will be about 15000.

[0089] In step S430, processor 104 determines whether the first estimated brightness of one of the first ROIs exceeds a brightness threshold (e.g., 10,000 lux) or whether the first luminance of the one of the first ROIs reaches an upper luminance limit (e.g., 255). If the first estimated brightness of one of the first ROIs exceeds the brightness threshold or the first luminance of the one of the first ROIs reaches the upper luminance limit, processor 104 may execute step S440 to determine that the one of the first ROIs belongs to a specific ROI that meets a predetermined condition. Specifically, if the first estimated brightness of the first ROI under consideration exceeds the brightness threshold, it indicates that the accuracy of the first estimated brightness is low. Furthermore, if the first luminance of the first ROI under consideration reaches the upper luminance limit, it indicates that the first ROI under consideration has an overexposure problem, meaning that the first ROI under consideration is not suitable for estimating the corresponding ambient light brightness. In other words, processor 104 may determine that the portion of the first ROI whose first estimated brightness exceeds the brightness threshold is a specific ROI that meets the predetermined condition.

[0090] On the other hand, in response to determining that the first estimated brightness of one of the first ROIs is not greater than the brightness threshold and the first luminance of the one of the first ROIs does not reach the upper luminance limit, the processor 104 may execute step S450 to determine that the one of the first ROIs belongs to the first candidate ROI. In other words, the processor 104 may determine another portion of the first ROI whose first estimated brightness is not greater than the brightness threshold as the first candidate ROI, but the present invention is not limited thereto.

[0091] Re-reference Figure 2 In step S220, in response to determining that the first ROI includes at least one specific ROI that satisfies a predetermined condition and at least one first candidate ROI, processor 104 obtains at least one second candidate ROI from the second ROI, where each of the at least one second candidate ROI corresponds to the at least one specific ROI. In one embodiment, after obtaining the specific ROI from the first ROI, processor 104 determines the second candidate ROI based on the specific ROI. Because the first ROI in first frame F1 and the second ROI in second frame F2 have a one-to-one correspondence, processor 104 may determine the portion of the second ROI corresponding to the specific ROI in the first ROI as the second candidate ROI, but the present invention is not limited to this.

[0092] Next, in step S230 , the processor 104 determines the ambient light brightness based on the at least one first candidate ROI and the at least one second candidate ROI.

[0093] See also Figure 5 , Figure 5 A flowchart of determining the ambient light brightness based on the first candidate ROI and the second candidate ROI according to an embodiment of the present invention is shown.

[0094] In step S510, the processor 104 determines a first camera intensity for each first candidate ROI based on the first image capture parameter set and the first luminance of each first candidate ROI. In one embodiment, the processor 104 may determine the first camera intensity for each first candidate ROI by dividing the first luminance of each first candidate ROI by the product of the first exposure time and the first gain. For example, if the first luminance of a first candidate ROI is L1 (i.e., the average grayscale value in this first candidate ROI), the first exposure time is Exp1, and the first gain is G1, then the first camera intensity for this first candidate ROI may be represented by L1 / (Exp1·G1), but the present invention is not limited to this.

[0095] In step S520 , the processor 104 determines a first estimated brightness of each first candidate ROI based on the first camera intensity of each first candidate ROI.

[0096] In one embodiment, the processor 104 may determine the first estimated brightness of each first candidate ROI by searching a first lookup table for a result corresponding to the first camera intensity of each first candidate ROI as the first estimated brightness of each first candidate ROI.

[0097] In one embodiment, based on Figure 3A In this case, once the first camera intensity of a first candidate ROI is obtained, the processor 104 can use Figure 3A The corresponding ambient light brightness in is used as the corresponding first estimated brightness. For example, according to Figure 3A , if the first camera intensity of a first candidate ROI is 2000, the corresponding first estimated brightness will be about 6000.

[0098] In step S530, the processor 104 determines a second camera intensity for each second candidate ROI based on the second image capture parameter set and the second luminance of each second candidate ROI. In one embodiment, the processor 104 may determine the second camera intensity for each second candidate ROI by dividing the second luminance of each second candidate ROI by the product of the second exposure time (i.e., Exp(min)) and the second gain (Gain(min)). For example, if the second luminance of a second candidate ROI is L2 (i.e., the average grayscale value in this second candidate ROI), the second camera intensity for this second candidate ROI may be represented by L1 / (Exp(min)·Gain(min)), but the present invention is not limited to this.

[0099] In step S540 , the processor 104 determines a second estimated brightness of each second candidate ROI based on the second camera intensity of each second candidate ROI.

[0100] In one embodiment, the processor 104 may determine the second estimated brightness of each second candidate ROI by searching a second lookup table for a result corresponding to the second camera intensity of each second candidate ROI as the second estimated brightness of each second candidate ROI.

[0101] In one embodiment, based on Figure 3B In this case, once a second camera intensity of a second candidate ROI is obtained, the processor 104 can use Figure 3B The corresponding ambient light brightness in is used as the corresponding second estimated brightness. For example, according to Figure 3B , if the second camera intensity of a second candidate ROI is 4000, the corresponding second estimated brightness will be about 15100. For another example, according to Figure 3B , if the second camera intensity of a second candidate ROI is 10,000, the corresponding second estimated brightness will be about 40,000.

[0102] In step S550 , the processor 104 obtains the ambient light brightness based on the first estimated brightness of each first candidate ROI and the second estimated brightness of each second candidate ROI.

[0103] In one embodiment, the ambient light brightness in step S230 may be a first estimated brightness of one or more of the first candidate ROIs and / or a second estimated brightness of one or more of the second candidate ROIs.

[0104] In one embodiment, the ambient light brightness in step S230 may be the estimated ambient light brightness corresponding to the first frame F1 and / or the second frame F2. In this case, the processor 104 may obtain the ambient light brightness by taking the average of the first estimated brightness of each first candidate ROI and the second estimated brightness of each second candidate ROI.

[0105] As can be seen from the above description, embodiments of the present invention can determine ambient light levels by further utilizing information provided by bright and dark frames for implementing tracking functions (e.g., inside-out tracking). Therefore, there is no need to additionally provide a light detector on the host 100 (e.g., an HMD), thereby improving space efficiency while reducing costs.

[0106] In order to better understand the concept of the present invention, Figure 6 will be used as an example for discussion.

[0107] See also Figure 6 , Figure 6 The following illustrates an application scenario of determining ambient light brightness according to the first embodiment of the present invention. Figure 6 In the embodiment, the processor 104 obtains one of the bright frames as the first frame 610 and obtains one of the dark frames as the second frame 620. The first frame 610 may be an image before the second frame 620, but the present invention is not limited thereto. In the first embodiment, it is assumed that each of the first ROIs and each of the second ROIs includes a plurality of pixels.

[0108] exist Figure 6 , it is assumed that the processor 104 determines first ROIs 611 - 614 in the first frame 610 and determines second ROIs 621 - 624 in the second frame 620 , wherein the first ROIs 611 - 614 correspond one-to-one to the second ROIs 621 - 624 .

[0109] In one embodiment, the processor 104 may determine a first estimated brightness of each of the first ROIs 611 to 614 and determine whether there is an overexposure problem in each of the first ROIs 611 to 614. Figure 6 In the example, assuming that the first ROIs 611 and 612 correspond to objects with high light intensity (e.g., some light sources), the first estimated brightness of each of the first ROIs 611 and 612 will be higher than the brightness threshold, or the first luminance of each of the first ROIs 611 and 612 will reach the luminance upper limit. In this case, the processor 104 may determine the first ROIs 611 and 612 as specific ROIs that meet the predetermined condition.

[0110] In addition, assuming that the first ROIs 613 and 614 do not correspond to objects with high light intensity (e.g., some light sources), the first estimated brightness of each of the first ROIs 613 and 614 will not be higher than the brightness threshold, and the first luminance of each of the first ROIs 611 and 612 will not reach the upper luminance limit. In this case, the processor 104 may determine the first ROIs 613 and 614 as first candidate ROIs.

[0111] Since the first ROIs 611 and 612 are considered as specific ROIs that meet the predetermined conditions, the processor 104 can accordingly find the portions of the second ROIs 621 to 624 that correspond to the first ROIs 611 and 612 (ie, specific ROIs) as the second candidate ROIs. Figure 6 In the illustrated scenario, the processor 104 may determine the second ROIs 621 and 622 as second candidate ROIs corresponding to the first ROIs 611 and 612 (ie, specific ROIs).

[0112] The processor 104 may then determine the ambient light brightness based on the first candidate ROIs (i.e., first ROIs 613 and 614) and the second candidate ROIs (i.e., second ROIs 621 and 622). In one embodiment, the processor 104 may obtain the ambient light brightness by taking the average of the first estimated brightness of each first candidate ROI (i.e., first ROIs 613 and 614) and the second estimated brightness of each second candidate ROI (i.e., second ROIs 621 and 622).

[0113] exist Figure 6 , the size / position / shape of each first ROI 611 to 614 and each second ROI 621 to 624 are merely examples. In various embodiments, the size / position / shape of each first ROI and each second ROI may be arbitrarily determined based on the designer's requirements.

[0114] See also Figure 7 , Figure 7 The following illustrates an application scenario of determining ambient light brightness according to the second embodiment of the present invention. Figure 7 In the embodiment, the processor 104 obtains one of the bright frames as the first frame 710 and obtains one of the dark frames as the second frame 720 , wherein the first frame 710 may be an image before the second frame 720 , but the present invention is not limited thereto.

[0115] In the second embodiment, it is assumed that each first ROI and each second ROI includes one pixel, that is, each pixel in the first frame 710 is one of the first ROIs in the first frame 710 , and each pixel in the second frame 720 is one of the second ROIs in the second frame 720 .

[0116] exist Figure 7 In the example, it is assumed that based on the above teachings, pixels in region 711 (i.e., the first ROI) are determined to have an overexposure problem. In this case, the processor 104 may determine the pixels outside region 711 as the first candidate ROI, and determine the pixels within region 711 as a specific ROI that meets the predetermined condition.

[0117] Afterwards, the processor 104 may determine the pixels in region 721 of the second frame 720 (i.e., the second ROI) as the second candidate ROI corresponding to the specific ROI. Next, the processor 104 may determine the ambient light brightness based on the first candidate ROI (i.e., pixels outside region 711) and the second candidate ROI (i.e., pixels within region 721). In one embodiment, the processor 104 may obtain the ambient light brightness by averaging the first estimated brightness of each first candidate ROI and the second estimated brightness of each second candidate ROI.

[0118] In one embodiment, if the processor 104 determines that no specific ROI that meets the predetermined condition exists in the first frame F1, then all first ROIs in the first ROI do not have an overexposure problem (i.e., all first ROIs in the first ROI can be considered as first candidate ROIs). In this case, the processor 104 can determine the ambient light brightness based on the first ROI. In one embodiment, the processor 104 can obtain the ambient light brightness by taking the average of the first estimated brightness of each first ROI, but the present invention is not limited to this.

[0119] In an embodiment of the present invention, after obtaining the ambient light brightness, the processor 104 may accordingly implement a brightness control mechanism and several shutter control mechanisms, which will be discussed in detail below.

[0120] See also Figure 8 , Figure 8 A schematic diagram of a host according to an embodiment of the present invention is shown. Figure 8 In FIG. 8 , the host 800 includes a processor 104 , displays 811 , 821 , M×N shutter transmittance controllers 812 , 822 , light sources 813 , 823 , timing controllers 814 , 824 , an image buffer 830 , and tracking cameras 841 to 84K.

[0121] In one embodiment, the display 811 may be a left display of the host 800 (eg, HMD), and the light source 813 may be a light source of the left display. In one embodiment, the processor 104 may control the light source 813 by sending a control signal to the timing controller 814.

[0122] In one embodiment, the display 811 may correspond to a first shutter structure having M×N shutter zones (where M and N are integers). In one embodiment, the first shutter structure may be positioned between the user's left eye and the display 811, and the processor 104 may control the transmittance of each shutter zone by sending a control signal to the M×N shutter transmittance controller 812. In one embodiment, the size of the first shutter structure is the same as the size of the display 811. In one embodiment, if one of the shutter zones of the first shutter structure is adjusted to have a lower transmittance, light from the corresponding portion on the display 811 will be less perceptible to the user, i.e., the corresponding portion on the display 811 will appear (slightly) darker to the user.

[0123] In one embodiment, the display 821 may be a right display of the host 800 (eg, HMD), and the light source 823 may be a light source of the right display. In one embodiment, the processor 104 may control the light source 823 by sending a control signal to the timing controller 824 .

[0124] In one embodiment, the display 821 may correspond to a second shutter structure having M×N shutter zones (where M and N are integers). In one embodiment, the second shutter structure may be positioned between the user's right eye and the display 821, and the processor 104 may control the transmittance of each shutter zone by sending a control signal to the M×N shutter transmittance controller 822. In one embodiment, the size of the second shutter structure is the same as that of the display 821. In one embodiment, if one of the shutter zones of the second shutter structure is adjusted to have a lower transmittance, light from the corresponding portion on the display 821 will be less perceptible to the user, i.e., the corresponding portion on the display 821 will appear (slightly) darker to the user.

[0125] In one embodiment, the image buffer 830 (which may be implemented as part of the storage circuit 102 ) may be used to store images (eg, the first frame F1 and the second frame F2 ) for determining the ambient light brightness, but the invention is not limited thereto.

[0126] In one embodiment, the processor 104 may be coupled to the tracking cameras 841 to 84K, and each tracking camera may be configured to capture corresponding light and dark frames to perform tracking functions and determine ambient light levels, but the invention is not limited thereto.

[0127] In the third embodiment, after obtaining the ambient light brightness, the processor 104 may determine the target brightness of the displays 811 and 821 based on the ambient light brightness, and set the brightness of the displays 811 and 821 to the target brightness. In one embodiment, the process of determining the target brightness may refer to conventional techniques for determining the brightness of a display based on ambient light, and the details thereof will not be repeated herein.

[0128] In other embodiments where the host 800 includes only one display, the processor 104 may also determine a target brightness based on the ambient light level and set the brightness of the display accordingly.

[0129] In the fourth embodiment where each first ROI and each second ROI includes one pixel, the processor 104 may obtain a shutter area corresponding to a display (eg, display 811 and / or 821 ), and obtain a region brightness of each shutter area based on the first candidate ROI and the second candidate ROI.

[0130] In one embodiment, the processor 104 may determine a first estimated brightness for each first candidate ROI and a second estimated brightness for each second candidate ROI based on the teachings of the above embodiments. Next, the processor 104 may determine a zone brightness for each shutter zone by averaging the first estimated brightness for each first candidate ROI and the second estimated brightness for each second candidate ROI.

[0131] by Figure 7 Taking the illustrated scenario as an example, it is assumed that the content of first frame 710 is part of a bright frame and the content of second frame 720 is part of a dark frame, and the size and position of the shutter region under consideration are assumed to be the same as the portion of the bright / dark frame. That is, the shutter region under consideration corresponds to some of the first and second candidate ROIs. In this case, the processor 104 can determine the region brightness of the shutter region by averaging the first estimated brightness of each first candidate ROI corresponding to the shutter region (i.e., pixels outside region 711) and the second estimated brightness of each second candidate ROI corresponding to the shutter region (i.e., pixels within region 721).

[0132] Next, processor 104 determines a first target transmittance for each shutter zone based on the first contrast value (referred to as CR1) and the zone brightness of each shutter zone. In one embodiment, the first target transmittance of a shutter zone can be represented by "W255 / ((CR1-1)·ALIT)", where W255 is the brightness of the display (or the brightness corresponding to the highest grayscale value) and ALIT is the zone brightness of the shutter zone. In one embodiment, the first target transmittance of each shutter zone is constrained between an upper transmittance limit and a lower transmittance limit.

[0133] After determining the first target transmittance of each shutter zone, the processor 104 may set the first transmittance of each shutter zone as the corresponding first target transmittance.

[0134] As can be observed in the formula “W255 / ((CR1-1)·ALIT)”, the first target transmittance is negatively correlated with the corresponding zone brightness. That is, the higher the zone brightness of the shutter zone, the lower the corresponding first target transmittance will be, and vice versa.

[0135] Therefore, for some image areas corresponding to objects with high light intensity (e.g., light sources), the transmittance of the corresponding shutter areas will be determined to have a lower value, so that the light from these objects will not affect the user's visual experience. For example, when the processor 104 displays some visual content (e.g., AR objects) at positions corresponding to these objects, the processor 104 may adjust the shutter areas corresponding to these objects to have a lower transmittance, so that the user can see the visual content more clearly, but the present invention is not limited to this.

[0136] In a fifth embodiment, the processor 104 may obtain a specific shutter area corresponding to a portion of a display (e.g., displays 811 and / or 821). In one embodiment, the specific shutter area may correspond to a gazed region on the display. That is, the processor 104 may determine the shutter area currently gazed at by the user as the specific shutter area, but the present invention is not limited thereto.

[0137] Afterwards, the processor 104 obtains the brightness of the specific shutter area based on the first candidate ROI and the second candidate ROI. The details of obtaining the brightness of the specific shutter area can be referred to the teaching of obtaining the brightness of the shutter area above, which will not be repeated herein.

[0138] Next, the processor 104 determines a specific target transmittance for the specific shutter zone based on a specific contrast value (referred to as CR') and the zone brightness of the specific shutter zone. In the fifth embodiment, the specific contrast value may be a contrast value specific to a portion of the display (e.g., a contrast value specific to the user's gaze zone on the display) that may be higher than the first contrast value, but the present invention is not limited thereto.

[0139] In one embodiment, the specific target transmittance for a specific shutter zone can be represented by "W255 / ((CR'-1)·ALIT)", where W255 is the brightness of the display (or the brightness corresponding to the highest grayscale value) and ALIT is the zone brightness of the specific shutter zone. In one embodiment, the specific target transmittance for a specific shutter zone is constrained between an upper transmittance limit and a lower transmittance limit.

[0140] In the fifth embodiment, the processor 104 sets the second transmittance of the specific shutter area as the specific target transmittance.

[0141] Therefore, the processor 104 can specifically adjust the specific target transmittance of a specific shutter zone, thereby improving the user's visual experience. For example, the visual content that the user is looking at can be seen more clearly by the user, but the present invention is not limited thereto.

[0142] The present invention further provides a computer-readable storage medium for executing the method for determining ambient light brightness. The computer-readable storage medium comprises a plurality of program instructions (e.g., setup program instructions and deployment program instructions) implemented therein. These program instructions can be loaded into a host computer 100 or 800 and executed by the host computer 100 or 800 to perform the method for determining ambient light brightness and the aforementioned functions of the host computer 100 or 800.

[0143] In summary, embodiments of the present invention can further utilize the information provided by the bright and dark frames to perform tracking functions to determine ambient light brightness. Therefore, there is no need to provide an additional light detector on the host (e.g., HMD), thereby reducing costs and improving space efficiency.

[0144] In addition, embodiments of the present invention further provide a brightness control mechanism and several shutter control mechanisms that can improve the user's visual experience.

[0145] It will be apparent to those skilled in the art that various modifications and variations may be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, the present invention is intended to cover modifications and variations of the present invention as long as the modifications and variations fall within the scope of the following claims and their equivalents.

Claims

1. A method for determining ambient light brightness, applicable to a host, characterized in that: include: Obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the plurality of first ROIs respectively correspond to the plurality of second ROIs; In response to determining that the plurality of first regions of interest include at least one specific region of interest satisfying a predetermined condition and at least one first candidate region of interest, obtaining at least one second candidate region of interest among the plurality of second regions of interest, wherein the at least one second candidate region of interest corresponds to the at least one specific region of interest respectively; as well as determining the ambient light brightness based on the at least one first candidate region of interest and the at least one second candidate region of interest, wherein the first frame is captured based on a first set of image capture parameters, and the method further comprises: determining a first camera intensity for each of the first regions of interest based on the first image capture parameter set and a first luminance of each of the first regions of interest, wherein the first image capture parameter set includes a first exposure time and a first gain, and the first camera intensity for each of the first regions of interest is determined by dividing the first luminance of each of the first regions of interest by a product of the first exposure time and the first gain; determining a first estimated brightness of each of the first regions of interest based on the first camera intensity of each of the first regions of interest; In response to determining that the first estimated brightness of one of the plurality of first regions of interest is higher than a brightness threshold or the first luminance of the one of the plurality of first regions of interest reaches a luminance upper limit, determining that the one of the plurality of first regions of interest belongs to the at least one specific region of interest that satisfies the predetermined condition; and In response to determining that the first estimated brightness of one of the multiple first regions of interest is not higher than the brightness threshold and the first luminance of the one of the multiple first regions of interest does not reach the luminance upper limit, it is determined that the one of the multiple first regions of interest belongs to the at least one first candidate region of interest.

2. The method according to claim 1, further comprising: obtaining a plurality of first specific frames and a plurality of second specific frames, wherein the plurality of first specific frames correspond to a first image capture parameter set, and the plurality of second specific frames correspond to a second image capture parameter set; Retrieving one of the plurality of first specific frames as the first frame; as well as One of the plurality of second specific frames is retrieved as the second frame. The method according to claim 2 , wherein the plurality of first specific frames and the plurality of second specific frames are interleaved with each other. The method according to claim 2 , wherein a time difference between the first frame and the second frame is less than a predetermined time threshold. The method of claim 2 , wherein a time difference between the first frame and the second frame is equal to an inverse of a frame rate. 6 . The method of claim 2 , wherein the first set of image capture parameters comprises at least one automatically determined image capture parameter and the second set of image capture parameters comprises at least one fixed image capture parameter. The method of claim 1 , wherein each of the plurality of first regions of interest and each of the plurality of second regions of interest comprises one or more pixels. 8 . The method according to claim 1 , wherein the first estimated brightness of each of the first regions of interest is determined by searching a first lookup table for a result corresponding to the first camera intensity of each of the first regions of interest as the first estimated brightness of each of the first regions of interest.

9. The method of claim 1, wherein the first frame and the second frame are used to perform a tracking function.

10. The method of claim 1 , wherein the first frame is captured based on a first image capture parameter set, the second frame is captured based on a second image capture parameter set, and determining the ambient light brightness based on the at least one first candidate region of interest and the at least one second candidate region of interest comprises: determining a first camera intensity for each of the first candidate regions of interest based on the first set of image capture parameters and a first luminance of each of the first candidate regions of interest; determining a first estimated brightness of each of the first candidate regions of interest based on the first camera intensity of each of the first candidate regions of interest; determining a second camera intensity for each of the second candidate regions of interest based on the second set of image capture parameters and a second luminance of each of the second candidate regions of interest; determining a second estimated brightness of each of the second candidate regions of interest based on the second camera intensity of each of the second candidate regions of interest; as well as The ambient light brightness is obtained based on the first estimated brightness of each of the first candidate regions of interest and the second estimated brightness of each of the second candidate regions of interest. The method according to claim 10 , wherein the ambient light brightness is an average of the first estimated brightness of each first candidate ROI and the second estimated brightness of each second candidate ROI.

12. The method according to claim 1, further comprising: determining a target brightness of the display based on the ambient light brightness; as well as The brightness of the display is set to the target brightness.

13. The method according to claim 1, further comprising: obtaining a plurality of shutter zones corresponding to the display; Obtaining a region brightness of each of the shutter regions based on the at least one first candidate region of interest and the at least one second candidate region of interest; determining a first target transmittance of each shutter zone based on a first contrast value of each shutter zone and the zone brightness; as well as The first transmittance of each shutter area is set to the corresponding first target transmittance.

14. The method according to claim 13 , wherein each of the first candidate ROI and each of the second candidate ROI comprises one pixel, and the step of obtaining the zone brightness of each of the shutter zones based on the at least one first candidate ROI and the at least one second candidate ROI comprises: determining a first estimated brightness of each of the first candidate ROIs and a second estimated brightness of each of the second candidate ROIs; as well as The zone brightness of each of the shutter zones is determined by averaging the first estimated brightness of each of the first candidate ROIs corresponding to each of the shutter zones and the second estimated brightness of each of the second candidate ROIs corresponding to each of the shutter zones.

15. The method according to claim 1, further comprising: obtaining a specific shutter area corresponding to a portion of the display from among the plurality of shutter areas; Obtaining a region brightness of the specific shutter region based on the at least one first candidate region of interest and the at least one second candidate region of interest; determining a specific target transmittance of the specific shutter zone based on a specific contrast value of the specific shutter zone and the zone brightness; as well as The second transmittance of the specific shutter area is set as the specific target transmittance. The method of claim 15 , wherein the specific shutter zone corresponds to a gaze zone on the display.

17. A host comprising: a storage circuit for storing program codes; A processor is coupled to the memory circuit and accesses the program code to perform: Obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the plurality of first ROIs respectively correspond to the plurality of second ROIs; In response to determining that the plurality of first regions of interest include at least one specific region of interest satisfying a predetermined condition and at least one first candidate region of interest, obtaining at least one second candidate region of interest among the plurality of second regions of interest, wherein the at least one second candidate region of interest corresponds to the at least one specific region of interest respectively; as well as determining an ambient light intensity based on the at least one first candidate region of interest and the at least one second candidate region of interest, wherein the first frame is captured based on a first set of image capture parameters, and the processor further performs: determining a first camera intensity for each of the first regions of interest based on the first image capture parameter set and a first luminance of each of the first regions of interest, wherein the first image capture parameter set includes a first exposure time and a first gain, and the first camera intensity for each of the first regions of interest is determined by dividing the first luminance of each of the first regions of interest by a product of the first exposure time and the first gain; determining a first estimated brightness of each of the first regions of interest based on the first camera intensity of each of the first regions of interest; In response to determining that the first estimated brightness of one of the plurality of first regions of interest is higher than a brightness threshold or the first luminance of the one of the plurality of first regions of interest reaches a luminance upper limit, determining that the one of the plurality of first regions of interest belongs to the at least one specific region of interest that satisfies the predetermined condition; as well as In response to determining that the first estimated brightness of one of the multiple first regions of interest is not higher than the brightness threshold and the first luminance of the one of the multiple first regions of interest does not reach the luminance upper limit, it is determined that the one of the multiple first regions of interest belongs to the at least one first candidate region of interest.

18. A non-transitory computer-readable storage medium recording an executable computer program, wherein the executable computer program is loaded by a host to perform the following steps: Obtaining a first frame and a second frame, wherein the first frame includes a plurality of first regions of interest (ROIs), the second frame includes a plurality of second ROIs, and the plurality of first ROIs respectively correspond to the plurality of second ROIs; In response to determining that the plurality of first regions of interest include at least one specific region of interest satisfying a predetermined condition and at least one first candidate region of interest, obtaining at least one second candidate region of interest among the plurality of second regions of interest, wherein the at least one second candidate region of interest corresponds to the at least one specific region of interest respectively; as well as determining ambient light brightness based on the at least one first candidate region of interest and the at least one second candidate region of interest, wherein the first frame is captured based on a first set of image capture parameters, and the executable computer program is further loaded to perform the following steps: determining a first camera intensity for each of the first regions of interest based on the first image capture parameter set and a first luminance of each of the first regions of interest, wherein the first image capture parameter set includes a first exposure time and a first gain, and the first camera intensity for each of the first regions of interest is determined by dividing the first luminance of each of the first regions of interest by a product of the first exposure time and the first gain; determining a first estimated brightness of each of the first regions of interest based on the first camera intensity of each of the first regions of interest; In response to determining that the first estimated brightness of one of the plurality of first regions of interest is higher than a brightness threshold or the first luminance of the one of the plurality of first regions of interest reaches a luminance upper limit, determining that the one of the plurality of first regions of interest belongs to the at least one specific region of interest that satisfies the predetermined condition; as well as In response to determining that the first estimated brightness of one of the multiple first regions of interest is not higher than the brightness threshold and the first luminance of the one of the multiple first regions of interest does not reach the luminance upper limit, it is determined that the one of the multiple first regions of interest belongs to the at least one first candidate region of interest.

Images