Method and system for detecting and removing mura - Patents.com

Abstract

A method for detecting mura in a virtual image in a near-eye display (110, 410, NED) and a method for mura removal are provided. The method for mura detection includes acquiring a virtual image rendered on the near-eye display (110, 410, NED), extracting mura features (320) of the virtual image according to a mura type, and evaluating the mura degree of the virtual image based on the mura type. The method for mura removal includes acquiring mura features (320) of a first virtual image rendered on the near-eye display (110, 410, NED), calculating compensation coefficients based on the mura features (320), and adjusting grayscale values ​​of the near-eye display (110, 410, NED) based on the compensation coefficients to obtain a second virtual image.

Description

[Technical Field]

[0001] The present disclosure relates generally to microdisplay technology, and more particularly to a mura removal system and method for virtual images rendered in a near-eye display. [Background technology]

[0002] Micro light-emitting diode (LED) display panels have the advantages of small size, high refresh rate, and high brightness. However, due to the manufacturing process or long operating time, micro LED display panels suffer from unevenness, such as image retention, spots, bright or dark spots, or cloudiness, which reduces the quality level of the micro LED display panel. The unevenness of the display panel can usually be compensated for by an unevenness elimination method. The unevenness elimination method is used to eliminate the unevenness or improve the uniformity of the LED display panel. Conventionally, the unevenness of a micro LED display panel is compensated for directly by adjusting the grayscale value of the pixels in the micro LED display panel.

[0003] Near-eye displays can be provided as AR, VR, head-up / head-mounted, or other displays. Generally, near-eye displays typically include an image generator and an optical combiner that transmits the projected image from the image generator to the human eye. The projected image is a virtual image in front of the human eye. The image generator can be a microLED-based display, a liquid crystal on silicon (LCOS) display, or a digital light processing (DLP) display. The aforementioned mura (non-uniformity) of a microLED display panel can affect the quality of the final virtual image transmitted to the human eye. The source image exhibits pixel-to-pixel luminance and color variations, which are observed as non-uniformity in the distribution of luminance and / or chromaticity. Non-uniformity arising from the image generator also causes non-uniformity in the final rendered virtual image. The mura present as non-uniformity in the display can be observed by human vision. Therefore, mura is a fatal visual artifact for displays. Furthermore, compared with traditional displays, non-uniformity artifacts in near-eye displays are much more obvious due to their proximity to the human eye. However, for near-eye displays, there is a need for a way to detect mura and apply mura removal in the final rendered virtual image. Summary of the Invention

[0004] An embodiment of the present disclosure provides a method for detecting mura of a virtual image in a near-eye display, the method including: obtaining a virtual image rendered on the near-eye display, extracting mura features of the virtual image according to a mura type, and evaluating a mura degree of the virtual image based on the mura type.

[0005] Furthermore, an embodiment of the present disclosure also provides a method for removing mura from a virtual image rendered by a near-eye display, the method including: obtaining mura characteristics of a first virtual image rendered on the near-eye display; calculating a compensation coefficient based on the mura characteristics; and adjusting grayscale values ​​of the near-eye display based on the compensation coefficient to obtain a second virtual image.

[0006] An embodiment of the present disclosure further provides a system for detecting mura in a virtual image rendered on a near-eye display, the system comprising: an image generator configured to render a virtual image; an imager configured to acquire the virtual image; a positioner connected to the image generator and the imager and configured to control a relative position of the near-eye display and the imager; and a processor connected to the imager and configured to evaluate the mura degree of the virtual image.

[0007] An embodiment of the present disclosure further provides a system for removing mura from a virtual image rendered on a near-eye display, the system including: an image generator configured to render a first virtual image; an imager configured to acquire the virtual image; a positioner connected to the image generator and the imager and configured to control a relative position between the image generator and the imager; a mura extractor connected to the imager and configured to extract mura features from the first virtual image; a compensation calculator connected to the extractor and configured to calculate a compensation coefficient; and a driver connected to the compensation calculator and the image generator and configured to adjust a grayscale value of the image generator based on the compensation coefficient to obtain a second virtual image.

[0008] Embodiments and various aspects of the present disclosure are set forth in the following detailed description and accompanying drawings, in which the various features shown are not drawn to scale. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a schematic diagram of an exemplary mura removal system according to some embodiments of the present disclosure. [Figure 2] 1 shows a flowchart illustrating an exemplary method for mura removal according to some embodiments of the present disclosure. [Figure 3] 1 illustrates an exemplary mura removal process according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a schematic block diagram of an exemplary mura removal system according to some embodiments of the present disclosure. [Figure 5A] 1 illustrates various types of mura features according to some embodiments of the present disclosure. [Figure 5B] 1 illustrates various types of mura features according to some embodiments of the present disclosure. [Figure 5C] 1 illustrates various types of mura features according to some embodiments of the present disclosure. [Figure 6] 1 shows a flowchart illustrating an exemplary method for mura extraction according to some embodiments of the present disclosure. [Figure 7] 1 shows a flowchart illustrating an exemplary method for mura removal according to some embodiments of the present disclosure. [Figure 8] 1 illustrates an exemplary mura removal process according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Reference will now be made in detail to exemplary embodiments, some examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with aspects related to the present invention as set forth in the appended claims. Certain aspects of the present disclosure are described in further detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall control.

[0011] To improve image quality, mura elimination is needed for near-eye displays. Mura elimination refers to the process of eliminating / suppressing visual artifacts and achieving relative uniformity of brightness and / or color in a display.

[0012] In some embodiments, systems and methods are provided for detecting mura and performing mura removal for near-eye displays.

[0013] Mura refers to non-uniformity in luminance and / or chromaticity that is observed by the human eye as a visual artifact. Therefore, Mura features need to be extracted to evaluate and even suppress visual artifacts. Mura features can be extracted by analyzing profiles such as luminance scales, gradient boundaries / frequency domains, or grayscale histograms. Mura features can be identified into three types: corner Mura, cloud Mura, and global Mura. For evaluation, Mura can be classified into various levels according to human sensitivity to determine the dominant Mura type. For example, the human eye is more sensitive to corner effects than cloud effects, and more sensitive to cloud effects than global non-uniformity. Therefore, corner Mura can be determined as the dominant Mura type for evaluation. In some embodiments, Mura can be classified according to the Mura intensity of each Mura type. For example, if the Mura intensity of cloud Mura is greater than that of corner Mura, cloud Mura is determined as the dominant Mura. In some embodiments, the classification is defined by the user. For example, global non-uniformity Mura may be set by a user as the dominant Mura for a particular image. In some embodiments, one or more Mura types may be determined as the dominant Mura for evaluation. For example, both corner Mura and cloud Mura may be determined as dominant Mura, and thus both corner Mura and cloud Mura may be evaluated to obtain a final Mura score.

[0014] Based on the extracted and identified mura features, compensation can be performed by adjusting the matrix gray values ​​of an image generator (e.g., an image generator of a near-eye display) to achieve a relatively uniform distribution of the displayed image. The compensation coefficient can be calculated as the inverse of the variation in luminance and / or chromaticity by a baseline threshold. The baseline threshold can be determined by a matrix gray value histogram. In the histogram, the amount percentage distribution of all gray values ​​(e.g., 0 to 255) is calculated. The peak representing the maximum amount of gray value is extracted. For example, the maximum amount percentage is 0.2, and the corresponding gray value is 87. That is, the majority of the gray values ​​in this image are 87. Next, the majority gray value is determined as the baseline threshold for compensation. This is a histogram approach to determining the baseline for compensation. Note that not all mura features are considered to calculate the compensation coefficient. For example, the identified mura features are evaluated as candidates for calculating the compensation coefficient. The calculation of the compensation coefficient can be based on a mura feature selected from the candidates, i.e., the identified mura feature. The display's driving capability, with a sufficient range (e.g., an additional 100% grayscale adjustment space), can also be considered in the compensation tolerance for calculating the correction coefficients. The compensation tolerance here refers to the constraints on compensation during operation. For example, normal operation is 8 bits with a gray value range of 0 to 255. With one additional bit, the system can operate in a gray value range of 0 to 511 (9 bits) with a single compensation capability (100%). However, if the compensation coefficient is more than twice the normal range (e.g., 200 to 300%), it is outside the operating range (e.g., 9 bits), so the compensation coefficient is always cut off at the maximum compensation tolerance (100%) and cannot further adjust the grayscale as expected. In other words, the calculation of the correction coefficients takes into account the display's driving capability. After compensation, mura (i.e., non-uniformity) can be reexamined / reevaluated to determine whether the compensated image meets human sensitivity. This compensation process is referred to herein as the mura removal process.It should be noted that in this disclosure, the term compensation is equivalent to the term correction, and compensation factor means a correction factor. Mura in this specification also refers to local and global non-uniformity. Mura removal can be interpreted as correcting or compensating for non-uniformity.

[0015] FIG. 1 is a schematic diagram of an exemplary mura removal system 100 according to some embodiments of the present disclosure. As shown in FIG. 1, the system 100 is configured to detect mura / non-uniformity in a virtual image rendered on a near-eye display. The system 100 includes a near-eye display (NED) 110 for displaying an image in front of a human eye, an imager provided as an imaging module 120, a positioner provided as a positioning device 130, and a processor provided as a processing module 140. Furthermore, ambient light can be provided by an ambient light module 150. The near-eye display 110 can be provided as an AR, VR, head-up / head-mounted display, or other display. The positioning device 130 is configured to set an appropriate spatial relationship between the near-eye display (NED) 110 and the imaging module 120. For example, the positioning device 130 is configured to set the distance between the near-eye display 110 and the imaging module 120 to a range of 10 mm to 25 mm. Furthermore, the positioning device 130 can adjust the relative positions (e.g., distance and spatial position) between the near-eye display 110 and the imaging module 120. The imaging module 120 is configured to emulate a human eye to measure display optical characteristics and observe display performance. In some embodiments, the imaging module 120 can include an array light measurement device (LMD) 122 and a near-eye display (NED) lens 121. For example, the LMD 122 can be a colorimeter or an imaging camera such as a CCD (charge-coupled device) or CMOS (complementary metal-oxide semiconductor). The near-eye display (NED) lens 121 of the imaging module 120 has a front aperture with a small diameter of 1 mm to 6 mm. Therefore, the near-eye display (NED) lens 121 can provide a wide field of view (e.g., 60 to 180 degrees) forward, and the near-eye display lens 121 is configured to emulate a human eye for observing the near-eye display 110. The optical properties of the virtual image are measured by the imaging module 120 based on the positioning device 130 .

[0016] In some embodiments, the near-eye display 110 can include an image generation unit 111, also referred to herein as an image source, and a light combiner (not shown in FIG. 1 ), also referred to herein as image optics. The image generation unit 111 can be a microdisplay, such as a microLED, microOLED, LCOS, or DLP display, and can be configured to form a light engine together with an additional projector lens. The projected image from the light engine through the designed optics is transmitted to the human eye via the light combiner. The optics of the light combiner can be a reflective and / or diffractive optic, such as a freeform mirror / prism, a birdbath or cascade mirror, a grating coupler (waveguide), etc.

[0017] The processing module 140 is configured to perform mura analysis, mura feature extraction, compensation coefficient calculation, etc. In some embodiments, the processing module 140 may be included in a computer or server. In some embodiments, the processing module 140 may be deployed in a cloud, although not limited thereto.

[0018] In some embodiments, a driver (not shown in FIG. 1 ) provided as a driving module can further be provided to compensate the image generation unit 111 to remove mura from the virtual image to be displayed. The compensation coefficients are calculated in the processing module 140 and then transferred to the driving module. Thus, mura removal can be performed in the system 100. The driving system can be connected to communicate with the near-eye display 110, specifically with the image generation unit 111 of the near-eye display 110. For example, the driving module can be configured to adjust the grayscale value of the image generation unit 111. When a driving system including the functions of display driving and compensation (adjusting the gray values ​​in image processing) is integrated into the near-eye display, the compensation coefficient data from the processing module 140 can be transferred to the near-eye display system 110.

[0019] In some embodiments, for example, for AR applications, ambient light is provided from ambient light module 150. Ambient light module 150 is configured to generate a uniform light source with a corresponding color (e.g., D65) that can support measurements made under ambient light backgrounds and simulation of various conditions, such as daylight, outdoors, or indoors.

[0020] Figure 2 shows a flowchart illustrating an exemplary mura removal method 200 according to some embodiments of the present disclosure. Figure 3 illustrates an exemplary mura removal process 300 according to some embodiments of the present disclosure. Method 200 can be performed by mura removal system 100. With reference to Figures 2 and 3, mura removal method 200 includes steps 202-212.

[0021] In step 202, a raw virtual image is rendered on the near-eye display. Rendering, as used herein, refers to, but is not limited to, a process in generating a two-dimensional or three-dimensional image from a model by an application program. The raw virtual image 310 is rendered from a module, such as an AR module, in a near-eye display, such as near-eye display 110. The raw virtual image 310 is then generated by an image generation unit of the near-eye display and projected through a combiner optical system facing the human eye. The raw virtual image 310 can be captured by an imaging module, such as imaging module 120, for analysis. In practice, there may be multiple virtual images rendered on the display for mura / non-uniformity analysis under multiple test patterns with various gray values. In some embodiments, the raw virtual image is rendered under ambient lighting conditions, such as D65 daylight conditions, by an ambient light module (e.g., ambient light module 150 shown in FIG. 1).

[0022] In step 204, Mura features 320 are extracted from the raw virtual image 310. The Mura features 320 can be extracted by analyzing a profile of the raw virtual image 310. The profile can include a brightness scale, a gradient boundary / frequency domain, or a grayscale histogram. The Mura features can be further identified as corner Mura 321, cloud Mura 322, or global Mura (i.e., global non-uniformity).

[0023] A plain baseline is determined by histogram analysis in step 206. The plain baseline, which may also be referred to as a baseline threshold, may be determined by a matrix histogram.

[0024] In step 208, a compensation coefficient is calculated based on the mura features. The compensation coefficient may be calculated further based on a plain baseline and a self-definition (e.g., an average value in a local zone or an entire zone). The self-defined compensation target / basis may take into account an average value in a local zone or an entire zone. The compensation coefficient may be calculated by comparing the mura features with the plain baseline and further considering a compensation tolerance (e.g., a one-time compensation capacity). In some embodiments, not all mura features are used to calculate the compensation coefficient. For example, only the mura features identified in step 204 are evaluated as candidates for calculating the compensation coefficient.

[0025] In step 210, compensation coefficients 330 are applied to the image generator to adjust the grayscale values. In some embodiments, the compensation coefficients 330 are applied to the pixel pipeline of the image generator. For example, the compensation coefficients may be applied to the pixel pipeline of a micro LED display. Different pixels may correspond to different compensation coefficients. Thus, the image displayed by the micro LED display may be improved on a pixel-by-pixel basis.

[0026] In step 212, the degree of mura (non-uniformity) of the compensated virtual image rendered on the near-eye display is re-evaluated. The compensated virtual image is rendered to obtain an improved virtual image 340. The mura level of the improved virtual image 340 is reduced compared to the raw virtual image 310, and re-evaluation can be performed. During the evaluation, the mura can be classified into various levels according to human sensitivity, as described above. For example, the human eye is more sensitive to corner effects than cloud effects, and more sensitive to cloud effects than overall non-uniformity. Therefore, corner mura can be determined as the primary mura type to be evaluated.

[0027] Figure 4 is a schematic block diagram of an exemplary mura removal system 400 according to some embodiments of the present disclosure. The mura removal system 400 can be configured to perform the mura removal method 200 shown in Figure 2. As shown in Figure 4, the mura removal system 400 includes a near-eye display 410 for displaying an image for the human eye, an imager provided as an imaging module 420, a mura feature extractor provided as a mura feature extraction module 430, a compensation calculator provided as a compensation calculator module 440, a driver provided as a display driver module 450, and an evaluator provided as an evaluation module 460.

[0028] The near-eye display 410 can include an image generator (or image source) 411 and a light combiner (or image optics). The image generator 411 can be a microdisplay, such as a microLED (μLED) display, a microOLED, an LCOS, or a DLP display, and can be configured to form a light engine together with an additional projector lens. The projected image from the light engine through the designed optics is transmitted to the human eye via the light combiner. The optics can be a reflective and / or diffractive optics, such as a freeform mirror / prism, a birdbath or cascade mirror, a grating coupler (waveguide), etc.

[0029] The imaging module 420 may include an array light measurement device (LMD) and a near-eye display lens. For example, the LMD may be a colorimeter or a CCD / CMOS imaging camera. The near-eye display lens has a front aperture with a small diameter of 1 mm to 6 mm. The near-eye display lens is configured to provide a wide field of view (e.g., 60 to 180 degrees) forward and simulate the human eye for observing the near-eye display 410. Imaging data of the virtual image may be acquired from the virtual image by the imaging module 420. The imaging data may include brightness, chromaticity, grayscale values, etc. Note that a normal lens may also be used with a camera to obtain relative values ​​in measurements (e.g., uniformity).

[0030] The mura feature extraction module 430 is configured to analyze the rendered virtual image with respect to luminance and / or chromaticity or XYZ intensities of the multi-colored virtual image based on the acquired imaging data. The mura feature extraction module 430 is communicatively coupled to the imaging module 420 and extracts mura features from the virtual image captured by the imaging module 420.

[0031] The compensation calculation module 440 is communicatively connected to the mura feature extraction module 430 and configured to calculate a compensation coefficient. The compensation calculation module 440 can be configured to calculate a compensation coefficient based on the mura features extracted by the mura feature extraction module 430. The compensation coefficient can be calculated based on a plain baseline and a self-definition (e.g., an average value in a local zone or an entire zone). The compensation coefficient can be calculated by comparing the mura features with the plain baseline and further considering a compensation tolerance (e.g., a one-time compensation capacity). In some embodiments, not all mura features are used to calculate the compensation coefficient. For example, only identified mura features are evaluated as candidates for calculating the compensation coefficient.

[0032] The display driver module 450 is connected to communicate with the compensation calculation module 440 and is further connected to communicate with the near-eye display 410. The display driver module 450 is configured to adjust the grayscale values ​​of an image source, such as the image generator 411. In some embodiments, the display driver module 450 is configured to apply the compensation coefficients calculated by the compensation calculation module 440 to a pixel pipeline of the image generator 411 included in the near-eye display 410. For example, the compensation coefficients may be applied to a pixel pipeline of a micro LED display. Different pixels may correspond to different compensation coefficients.

[0033] The mura evaluation module 460 is communicatively connected to the imaging module 420 and configured to evaluate the mura degree (e.g., the degree of non-uniformity) after compensation. After compensation, the virtual image is re-rendered on the near-eye display 410 to obtain an improved virtual image, which can be captured by the imaging module 420. The mura evaluation module 460 can then evaluate the mura degree of the improved virtual image captured by the imaging module 420. In some embodiments, the mura evaluation module 460 can further be configured to classify the mura. The mura can be classified into various levels according to human sensitivity, and a dominant mura type can be determined. For example, as described above, the human eye is more sensitive to corner effects than cloud effects, and is more sensitive to cloud effects than global non-uniformity. Therefore, corner mura can be determined as the dominant mura for evaluation. In some embodiments, the mura can be classified according to the mura degree of each mura type. For example, if the Murasaki cloud has a greater Murasaki score than the Murasaki corner score, the Murasaki cloud is determined as the dominant Murasaki. In some embodiments, the classification is user-defined. For example, global Murasaki may be set by the user as the dominant Murasaki for a particular image. In some embodiments, one or more Murasaki types can be determined as the dominant Murasaki for evaluation. For example, both corner Murasaki and cloud Murasaki can be determined as dominant Murasaki, and then both corner Murasaki and cloud Murasaki can be evaluated to obtain a final Murasaki score.

[0034] Thus, evaluation may determine that the mura of enhanced virtual image 340 is reduced compared to the mura of raw virtual image 310.

[0035] In some embodiments, the mura removal system 400 further includes a pre-processing module (e.g., a pre-processor) connected to the imaging module 420 and the mura feature extraction module 430. The pre-processing module is configured to perform pre-processing on the raw virtual image before extracting the mura features. In some embodiments, the pre-processing removes adverse effects in the raw virtual image that degrade image quality, such as noise or distortion. In some embodiments, the pre-processing applies mapping / alignment from the virtual image pixel matrix to the source pixel matrix (image generator). For example, a 10,000×10,000 virtual image pixel matrix is ​​converted to a 640×480 source pixel matrix of the image generator. Optionally, the mapping / alignment can be performed by the compensation calculation module 440 for further gray value adjustment in the compensation process.

[0036] The connection between the aforementioned modules may be, but is not limited to, wired or wireless communication, for example, via the Internet, Bluetooth, etc. In some embodiments, the mura feature extraction module 430, the compensation calculation module 440, the display driving module 450, the mura evaluation module 460, and the pre-processing module may be, but is not limited to, integrated into a computer system or server, or may be deployed in the cloud.

[0037] In some embodiments, the virtual image rendered by the near-eye display 410 is acquired from the imaging module 420, which may directly exhibit some obvious artifacts. Figures 5A-5C respectively illustrate various types of mura features according to some embodiments of the present disclosure. Figure 5A illustrates distinct dark regions at each of the three corners of the image, referred to herein as corner features. Figure 5B illustrates a raw image converted into a pseudo-color image representing absolute / relative luminance distribution. As shown, cloud-like regions appearing to float above the global plane, constituting cloud-like features, are visible above the local regions depicted by dashed-dotted lines. Figure 5C illustrates a raw image plotted as a 3D surface to obtain a 3D image. As shown, in addition to the corner and cloud-like artifact zones, the 3D surface gradually attenuates in multiple directions, which is referred to as global mura. Therefore, non-uniformities on the global surface are observable. The gradual surface attenuation of the 3D image can be extracted as a mura feature. In the example of a waveguide, light may attenuate from one corner to the other. The human eye is sensitive to mura artifacts, so these are also non-uniformities that can occur in the virtual image.

[0038] To extract mura features in a rendered virtual image, a method for extracting mura features is provided. Figure 6 shows a flowchart illustrating an example mura extraction method 600 according to some embodiments of the present disclosure. Referring to Figure 6, the mura extraction method 600 includes steps 610-640.

[0039] In step 610, a live virtual image in the near-eye display is rendered. The live virtual image with mura is rendered from a model, such as an AR model. The live virtual image is then displayed by an image generator in the near-eye display and projected through a combiner optics in front of the human eye. The live virtual image can be captured by an imaging module, such as imaging module 120 or 420, for analysis. In some embodiments, the live virtual image is rendered under ambient lighting conditions, such as D65 daylight conditions.

[0040] In step 620, the raw virtual image is pre-processed to remove adverse effects such as noise or distortion. The raw virtual image can be captured by an imaging module (e.g., an array light measurement device (LMD) and a near-eye display lens) with a perfect white and / or gray test pattern. The captured raw virtual image is then pre-processed. In some embodiments, the pre-processing removes adverse effects that reduce image quality, such as noise or distortion, in the raw virtual image. Distortions, including those of the camera lens or NED optical module, are taken into account in this procedure.

[0041] In step 630, Mura features are extracted based on Mura types. Mura types include corner Mura, cloud Mura, and global Mura (or global non-uniformity). For different Mura types, Mura features can be extracted with different profiles. It is worth noting that the appearance of cloud Mura refers to an area of ​​non-uniformity. The area of ​​non-uniformity may be a spot or other description. Here, cloud Mura is used to describe a global area of ​​non-uniformity, and cloud Mura also refers to similar area forms such as spots.

[0042] More specifically, step 630 may further include steps 631 to 633.

[0043] In step 631, for corner mura, mura features are extracted with a brightness threshold profile. For example, the brightness profile of the image is compared with the brightness threshold profile. The threshold may be an upper / lower threshold. If the brightness of the corner exceeds (or falls below) the brightness threshold corresponding to the corner, e.g., the brightness of the corner is greater than (or less than) the brightness threshold corresponding to the corner, the corner mura feature is extracted.

[0044] In step 632, for the cloud-like mura, mura features are extracted in the spatial gradient profile or frequency domain. For example, the spatial gradient profile can be applied to extract the mura features. In some embodiments, a regional variation scale can be obtained according to the spatial gradient profile. The mura features can be extracted by comparing the regional variation scale with a preset threshold. In some embodiments, the captured image can be transformed into the frequency domain for further filtering in human contrast sensitivity perception to extract visual non-uniformity information.

[0045] In step 633, for global unevenness, the unevenness features are extracted in a global profile (e.g., a histogram), such as a grayscale histogram. The histogram can be applied as a profile to analyze the non-uniformity as a baseline. For example, through the image histogram, the maximum amount of gray value (i.e., peak) is extracted as a majority representation of the image, as a baseline analysis for the image uniformity. It should also be noted that gradients including scale and direction in the entire field of view can also be considered in the global non-uniformity analysis.

[0046] In step 640, the Mura degree is evaluated by considering the Mura type, Mura feature amount, and human perception. The Mura type can refer to corner Mura, cloud / spot Mura, or global Mura. Regarding Mura, the evaluation is performed not only on the type but also on the amount. The amount refers to the degree of non-uniformity, such as a percentage of the difference scale. For example, an 8% difference scale is more severe than a 5% difference scale. The human eye has relative sensitivity to non-uniformity. For example, if the difference / non-uniformity is less than 1%, the difference / non-uniformity is not obvious / visible to the human eye. After obtaining the Mura feature, an evaluation can be performed to determine the Mura degree in the near-eye display. Mura can be classified into various levels according to human sensitivity, and a major Mura type can be determined. For example, as described above, the human eye is more sensitive to corner effects than cloud effects, and more sensitive to cloud effects than global non-uniformity. Therefore, corner Mura can be determined as the major Mura for evaluation. In some embodiments, Mura can be classified according to the Mura intensity of each Mura type. For example, if the Mura intensity of cloud Mura is greater than the Mura intensity of corner Mura, cloud Mura is determined as the dominant Mura. In some embodiments, the classification is defined by the user. For example, global Mura can be set by the user as the dominant Mura for a particular image. In some embodiments, one or more Mura types can be determined as the dominant Mura for evaluation. For example, both corner Mura and cloud Mura can be determined as dominant Mura, and then both corner Mura and cloud Mura can be evaluated to determine the final Mura intensity. A luminance scale or an area size (e.g., 30% of the area of ​​the image) can be set as a threshold for evaluating Mura intensity. For example, for corner Mura, the luminance scale is set as the threshold. For cloud Mura, the area size is set as the threshold. In some embodiments, quantitative evaluation can be performed according to human perception. For example, if the luminance difference is less than 1%, this luminance difference is not obvious / invisible to the human eye.

[0047] Mura in multicolor virtual images rendered in near-eye displays is also considered. For each primary color channel, mura occurs on the image surface after the light spreads along its path. Image light is generated in a microdisplay, travels through the lens of a light engine / projector, further travels through a light combiner (such as a waveguide), and finally is received by the human eye, which refers to the light spread along its path. The mura features are the same as those described above and will not be repeated further in this specification. Furthermore, for three primary color channels (e.g., RGB), each color channel has its own mura type and scale. For a channel, the features can include one or more mura types. The scale of each channel can vary. For example, in the case of overall attenuation from one corner to the other three corners, the variation can be 30% to 80%. Therefore, in addition to brightness nonuniformity, serious problems such as color shift and white balance occur.

[0048] To solve these problems, a mura removal method is provided for color correction, depending on the doping ratio of the three main channels. The doping is performed in the display driver.

[0049] In some embodiments, a method for mura removal is provided. Figure 7 shows a flowchart illustrating an exemplary mura removal method 700 according to some embodiments of the present disclosure. Figure 8 illustrates an exemplary mura removal process 800 according to some embodiments of the present disclosure. Referring to Figures 7 and 8, the mura removal method 700 includes steps 702-712.

[0050] In step 702, a raw virtual image 810 in the near-eye display is rendered and acquired under a test pattern. The test pattern may be a solid test pattern with various gray values ​​(e.g., 63, 127, 255, etc.). Multiple test patterns with various gray values ​​and colors may be applied to the measurement. Each test pattern may be one or more colors, such as an R / G / B pattern and / or a white image. It is worth noting that a partial on / off pixel test pattern can be directly used for the measurement instead of a solid pattern. That is, multiple partial on / off patterns can be used for the measurement and finally combined into one solid image. The raw virtual image is displayed by the image generation unit of the near-eye display and projected through a combiner optical system in front of the human eye. The raw virtual image can be captured by an imaging module for analysis. In some embodiments, the raw virtual image is rendered under ambient lighting conditions, such as D65 daylight conditions.

[0051] In some embodiments, a multi-color virtual image is acquired under a white test pattern.

[0052] In step 704, three-primary color (R, G, B) and multi-color virtual images are acquired (820). The image can include a three-primary color image of RGB, which is a red, green, and blue image. Each primary color virtual image corresponds to one channel. Therefore, three monochromatic virtual images can be captured with three color test patterns to obtain imaging data for the monochromatic virtual image. A multi-color (white) virtual image can also be obtained by this procedure.

[0053] In step 706, Mura features for each color channel are extracted (830). For each monochromatic virtual image, Mura features are extracted. Mura features can be extracted based on Mura types, such as corner Mura, cloud Mura, and global Mura. More specifically, for corner Mura, Mura features are extracted based on a brightness threshold profile. For cloud Mura, Mura features are extracted based on a spatial gradient profile or frequency domain. For global Mura, Mura features are extracted based on a global profile (e.g., a histogram), such as a grayscale histogram. Further details regarding the extraction of Mura features are described above with reference to the method 600 described above and will not be repeated here.

[0054] In step 708, compensation coefficients for each color channel are calculated by considering mura characteristics in terms of both luminance and chromaticity nonuniformity. The compensation coefficients for each channel can be calculated according to the individual virtual images rendered. In some embodiments, the individual differences among the three channels are considered to calculate the compensation coefficients. For example, a color shift is calculated based on the difference between a monochromatic virtual image and an overall white virtual image in the entire field of view. The overall white virtual image is formed by superimposing all the monochromatic virtual images. In some embodiments, the compensation coefficients are calculated according to the doping ratios among the three channels. The doping ratios refer to the ratios among the three main channels: red, green, and blue. Therefore, the compensation coefficients can be further used to correct color and / or white balance by adjusting the doping ratio and overall tone of each pixel in the matrix.

[0055] In step 710, compensation coefficients for the image generator are applied to the three main channels (840). For example, the grayscale values ​​of the virtual image in the image generator (e.g., a micro LED display, a micro display) are adjusted according to the correction coefficients for each channel.

[0056] After compensation in step 712, the mura degree of the multi-color virtual image is re-evaluated (850). After compensation is applied to the raw virtual image image data, the rendered virtual image is re-evaluated to determine whether the non-uniformity / mura has been successfully removed. In some embodiments, the mura evaluation can be performed in both luminance and color. Because large-scale mura has been suppressed, the human eye should not perceive any remaining mura.

[0057] In some embodiments, a non-transitory computer-readable storage medium containing instructions is also provided, which may be executed by a device to perform the above-described methods. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, or any other flash memory, NVRAM, cache, registers, any other memory chip or cartridge, and network versions thereof. A device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0058] It should be noted that relational terms such as "first" and "second" are used herein merely to distinguish one entity or operation from another and do not require or imply an actual relationship or order between those entities or operations. Furthermore, the words "comprising," "having," "containing," and "including," as well as other similar forms, are intended to be equivalent in meaning and open-ended in that the one or more items referred to by any one of these words is not meant to be an exhaustive list of such one or more items, nor is it meant to be limited to only the listed one or more items.

[0059] As used herein, the term "or" includes all possible combinations unless otherwise specified or impractical. For example, if it is stated that a database may include A or B, then the database may include A, or B, or A and B, unless otherwise specified or impractical. As a second example, if it is stated that a database may include A, B, or C, then the database may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C, unless otherwise specified or impractical.

[0060] It should be understood that the above-described embodiments can be realized by hardware, or software (program code), or a combination of hardware and software. If realized by software, it may be stored in the above-described computer-readable medium. When executed by a processor, the software can perform the methods of the present disclosure. The computing units and other functional units described in this disclosure can be realized by hardware, or software, or a combination of hardware and software. Those skilled in the art will also understand that multiple modules / units among the above-described modules / units may be combined into one module / unit, or that each of the above-described modules / units may be further divided into multiple sub-modules / sub-units.

[0061] In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Certain adaptations and modifications can be made to the described embodiments. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the invention being indicated by the following claims. It is also intended that the order of steps depicted in the figures is for illustrative purposes only and is not intended to be limiting to the particular order of steps. Thus, one skilled in the art will recognize that these steps may be performed in different orders while performing the same method.

[0062] The drawings and specification disclose illustrative embodiments. However, many variations and modifications to these embodiments are possible. Thus, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. 1. A method for detecting mura of a virtual image in a near-eye display, comprising: obtaining the virtual image rendered on the near-eye display; Extracting mura characteristics of the virtual image according to the mura type; evaluating the unevenness degree of the virtual image based on the unevenness type; A method comprising:

2. The method of claim 1 , wherein the mura type comprises corner mura, cloud mura, or global mura.

3. Extracting the unevenness feature of the virtual image according to the unevenness type includes: extracting the unevenness feature based on a brightness threshold profile when the unevenness type is the corner unevenness; The method of claim 2 , comprising:

4. Extracting the unevenness feature of the virtual image according to the unevenness type includes: If the mura type is the cloud-like mura, extracting the mura feature based on a spatial gradient profile or a frequency domain. The method of claim 2 , comprising:

5. Extracting the unevenness feature of the virtual image according to the unevenness type includes: extracting the unevenness features based on an overall profile when the unevenness type is the overall unevenness; The method of claim 2 , comprising:

6. Before extracting the mura features of the virtual image according to the mura type, the method includes: Converting the virtual image into a pseudo-color image representing the absolute or relative luminance distribution. The method of claim 2 further comprising:

7. Before extracting mura features of the virtual image based on the mura type, the method further comprises: Plotting the virtual image as a 3D surface to obtain a 3D image. The method of claim 2 further comprising:

8. Evaluating the unevenness of the virtual image based on the unevenness type includes: determining one or more dominant mura types of the virtual image; evaluating the mura degree of the virtual image based on one or more predetermined thresholds corresponding to the one or more major mura types; The method of any one of claims 2 to 7, further comprising:

9. The method of claim 8 , wherein the corner mura is determined as the dominant mura type.

10. The one or more predetermined thresholds are: a brightness scale threshold value corresponding to said corner mura; or The region size threshold corresponding to the cloud-like unevenness 10. The method of claim 9, comprising:

11. The method of claim 10 , wherein the predetermined threshold value for the luminance scale threshold is 30% and the predetermined threshold value for the region size threshold is 30%.

12. Obtaining the virtual image rendered on the near-eye display includes: acquiring the virtual image under a full test pattern or a plurality of partial test patterns; further comprising The method according to any one of the preceding claims, wherein the partial test patterns have different grey values ​​and colours.

13. The method of claim 12 , wherein the test pattern is a full white test pattern.

14. The method of claim 12 , wherein the test pattern is a full gray test pattern.

15. The method of claim 12 , wherein the virtual image is rendered under ambient lighting conditions.

16. Before extracting the mura feature of the virtual image according to the mura type, preprocessing the virtual image by removing one or more of noise or distortion; The method of any one of claims 1 to 15, further comprising:

17. 1. A method for mura removal of a virtual image rendered by a near-eye display, comprising: acquiring a mura characteristic of a first virtual image rendered on the near-eye display; calculating a compensation coefficient based on the mura characteristics; adjusting the grayscale value of the near-eye display based on the compensation coefficient to obtain a second virtual image; A method comprising:

18. Evaluating the unevenness of the second virtual image.

20. The method of claim 17, further comprising:

19. Acquiring the mura feature of the first virtual image includes: Obtaining monochromatic virtual images of different colors; acquiring the mura characteristics of each of the monochromatic virtual images; 19. The method of claim 17 or 18, further comprising:

20. Calculating the compensation coefficient based on the mura characteristics includes: calculating the compensation coefficient based on the mura characteristics of each monochromatic virtual image; 20. The method of claim 19, further comprising:

21. Calculating the compensation coefficient based on the mura characteristics includes: applying a mapping from the pixel matrix of the first virtual image to a source pixel matrix; 21. The method of claim 20, further comprising:

22. Calculating the compensation coefficient based on the mura characteristics includes: determining a color shift based on a difference between the monochromatic virtual images on the overall virtual image; determining the color shift as the unevenness characteristic; 20. The method of claim 19, further comprising:

23. 23. The method of claim 22, wherein the difference is a doping ratio between the monochromatic virtual images.

24. Evaluating the unevenness of the second virtual image includes: evaluating the unevenness of the second virtual image in luminance and color; The method of any one of claims 18 to 23, further comprising:

25. Acquiring the mura feature of the first virtual image includes: obtaining a first multi-color virtual image rendered on the near-eye display; acquiring the mura characteristics of the multi-color virtual image; The method of any one of claims 17 to 24, further comprising:

26. 26. The method of claim 25, wherein the first multi-color virtual image is acquired under a white test pattern.

27. Acquiring the mura feature of the first virtual image includes: obtaining a monochromatic virtual image of three primary colors (red, green, and blue) rendered on the near-eye display; acquiring the mura characteristics of the monochrome image of each color; The method of any one of claims 17 to 24, further comprising:

28. 28. The method of claim 27, wherein the three primary color (red, green, and blue) monochromatic virtual images are acquired under red, green, and blue test patterns, respectively.

29. 1. A system for detecting mura in a virtual image rendered on a near-eye display, comprising: an image generator configured to render a virtual image; an imager configured to acquire the virtual image; a positioner connected to the image generator and the imager and configured to control a relative position of the near-eye display and the imager; a processor coupled to the imager and configured to evaluate unevenness of the virtual image; A system comprising:

30. The processor: extracting unevenness features of the virtual image according to the unevenness type; Evaluating the unevenness of the virtual image based on the unevenness type 30. The system of claim 29, further configured to:

31. 30. The system of claim 29, wherein the mura type comprises corner mura, cloud mura, or global mura.

32. 32. The system of claim 31, wherein the processor is further configured to, in response to the mura feature being the corner mura, extract the mura feature based on a brightness threshold profile.

33. 32. The system of claim 31, wherein the processor is further configured, in response to the mura feature being the mura cloud, to extract the mura feature based on a spatial gradient profile or a frequency domain.

34. 32. The system of claim 31, wherein the processor is further configured to, in response to the mura feature being the global mura, extract the mura feature based on a global profile.

35. The processor: Converting the virtual image into a pseudo-color image representing the absolute or relative luminance distribution 32. The system of claim 31 further configured to:

36. The processor: The virtual image is plotted as a 3D surface to obtain a 3D image.

32. The system of claim 31 further configured to:

37. The processor: determining one or more dominant mura types of the virtual image; Evaluating the mura degree of the virtual image based on one or more predetermined thresholds corresponding to the one or more major mura types. The system of any one of claims 31 to 36, further configured to:

38. 38. The system of claim 37, wherein the corner mura is determined as the dominant mura type.

39. The one or more predetermined thresholds are: a brightness scale threshold value corresponding to said corner mura; or The region size threshold corresponding to the cloud-like unevenness 39. The system of claim 38, comprising:

40. 40. The system of claim 39, wherein the predetermined threshold for the luminance scale threshold is 30% and the predetermined threshold for the region size threshold is 30%.

41. The system of any one of claims 29 to 40, wherein the processor is further configured to perform pre-processing on the virtual image.

42. 42. The system of claim 41, wherein the preprocessing is to remove one or more of noise or distortion.

43. The system of any one of claims 29 to 42, wherein the imager is further configured to acquire the virtual image under a full white test pattern.

44. The system of any one of claims 29 to 42, wherein the imager is further configured to acquire the virtual image under a full gray test pattern.

45. The imager includes: a near-eye display lens configured to simulate a human eye to obtain the virtual image; a light measurement device configured to measure the virtual image; The system of any one of claims 29 to 44, comprising:

46. 46. ​​The system of claim 45, wherein the light measurement device comprises a colorimeter or an imaging camera.

47. 46. ​​The system of claim 45, wherein the positioner is further configured to determine a spatial relationship between the image generating portion and the light measurement device.

48. 48. The system of any one of claims 29 to 47, wherein the image generator is further configured to render the virtual image under ambient lighting conditions.

49. 49. The system of any one of claims 29 to 48, wherein the image generator is one of a micro LED based display, a Liquid Crystal on Silicon (LCOS) display, or a Digital Light Processing (DLP) display.

50. 50. The system of any one of claims 29 to 49, wherein the near-eye display is one of an augmented reality display, a virtual reality display, a head-up display, or a head-mounted display.

51. 51. The system of claim 50, wherein the near-eye display comprises the image generator and an optical combiner configured to project the virtual image from the image generator to a human eye.

52. 1. A system for mura removal of a virtual image rendered on a near-eye display, comprising: an image generator configured to render a first virtual image; an imager configured to acquire the first virtual image; a positioner connected to the image generating unit and the imager and configured to control a relative position between the image generating unit and the imager; a mura feature extractor connected to the imager and configured to extract mura features of the first virtual image; a compensation calculation unit connected to the mura feature extraction unit and configured to calculate a compensation coefficient; a driver connected to the compensation calculation unit and the image generation unit, the driver configured to adjust a grayscale value of the image generation unit based on the compensation coefficient to obtain a second virtual image; A system comprising:

53. a preprocessor configured to perform preprocessing on the first virtual image; 53. The system of claim 52, further comprising:

54. 54. The system of claim 53, wherein the preprocessing is to remove one or more of noise or distortion.

55. 53. The system of claim 52, wherein the pre-processor is further configured to map a pixel matrix of the first virtual image to a source pixel matrix of the image generator.

56. the imager is further configured to acquire a monochromatic virtual image of a different color for each individual channel of the first virtual image; The system of any one of claims 52 to 55, wherein the mura feature extractor is further configured to obtain the mura feature for each of the differently colored monochromatic virtual images.

57. 57. The system of claim 56, wherein the compensation calculation unit is configured to calculate the compensation coefficient based on the mura characteristics of each of the different color monochromatic virtual images.

58. The compensation calculation unit determining a color shift based on a difference between the single-color virtual images on the overall virtual image; The color shift is determined as the unevenness characteristic.

58. The system of claim 57, further configured to:

59. 59. The system of claim 58, wherein the difference is a doping ratio between the monochromatic virtual images.

60. 53. The system of claim 52, wherein the compensation calculator is further configured to map a pixel matrix of the first virtual image to a source pixel matrix of the image generator.

61. the imager is further configured to acquire a first multi-color virtual image rendered by the image generator; The system of any one of claims 47 to 60, wherein the mura feature extractor is further configured to obtain the mura features of the virtual image.

62. 62. The system of claim 61, wherein the first multi-color virtual image is acquired under a white test pattern.

63. 62. The system of claim 61, wherein the imager is further configured to acquire a monochromatic virtual image of three primary colors (red, green, and blue) rendered by the image generator.

64. 64. The system of claim 63, wherein the three primary color (red, green, and blue) monochromatic virtual images are acquired under red, green, and blue test patterns, respectively.

65. 65. The system of any one of claims 52 to 64, wherein the image generator is further configured to render the first virtual image under ambient lighting conditions.

66. The system of any one of claims 52 to 65, wherein the mura feature extraction unit is further configured to extract the mura features of the first virtual image according to a mura type, the mura type including corner mura, cloud-like mura, or overall mura.

67. The unevenness feature extraction unit extracting the unevenness feature based on a brightness threshold profile in response to the unevenness feature being the corner unevenness; 67. The system of claim 66, further configured to:

68. The unevenness feature extraction unit extracting the mura feature based on a spatial gradient profile or a frequency domain in response to the mura feature being the cloud-like mura feature; 67. The system of claim 66, further configured to:

69. The unevenness feature extracting unit extracts the unevenness feature based on an overall profile in response to the unevenness feature being the overall unevenness.

67. The system of claim 66, further configured to:

70. The unevenness feature extraction unit Converting the virtual image into a pseudo-color image representing the absolute or relative luminance distribution 67. The system of claim 66, further configured to:

71. The unevenness feature extraction unit The virtual image is plotted as a 3D surface to obtain a 3D image.

67. The system of claim 66, further configured to:

72. The system of any one of claims 66 to 71, further comprising an evaluation unit configured to evaluate the first virtual image and the second virtual image based on the mura type.

73. The evaluation unit determining one or more dominant mura types of the first virtual image; Evaluating the mura level of the second virtual image based on one or more predetermined thresholds corresponding to the one or more major mura types.

73. The system of claim 72, further configured to:

74. 74. The system of claim 73, wherein the corner mura is determined as the dominant mura type.

75. The one or more predetermined thresholds are: a brightness scale threshold corresponding to said corner mura; and The region size threshold corresponding to the cloud-like unevenness 74. The system of claim 73, comprising:

76. 76. The system of claim 75, wherein the predetermined threshold for the luminance scale threshold is 30% and the predetermined threshold for the region size threshold is 30%.

77. The imager includes: a near-eye display lens configured to simulate a human eye to obtain the virtual image; a light measurement device configured to measure the virtual image; 78. The system of any one of claims 52 to 77, comprising:

78. 78. The system of claim 77, wherein the light measurement device comprises a colorimeter or an imaging camera.

79. 79. The system of claim 78, wherein the positioner is further configured to determine a spatial relationship between the image generating unit and the light measurement device.

80. 80. The system of any one of claims 52 to 79, wherein the image generator is one of a micro LED based display, a Liquid Crystal on Silicon (LCOS) display, or a Digital Light Processing (DLP) display.

81. 81. The system of any one of claims 52 to 80, wherein the near-eye display is one of an augmented reality display, a virtual reality display, a head-up display, or a head-mounted display.

82. 82. The system of claim 81 , wherein the near-eye display comprises the image generator and an optical combiner configured to project the virtual image from the image generator to a human eye.

Images