Method and system for virtual image compensation and evaluation

Abstract

A method for image compensation of a virtual image displayed by a near-eye display based on a micro display projector, the method comprising: acquiring a virtual image displayed by the near-eye display, where the virtual image is formed by a source image emitted from the micro display projector; preprocessing the image data of the virtual image to obtain preprocessed image data; obtaining a relationship between the source image and the virtual image; determining an image baseline value of the virtual image; and obtaining a compensation factor matrix comprising a compensation factor for each pixel in the source image based on the relationship and the image baseline value.

Description

Technical Field

[0001] The present disclosure relates generally to microdisplay technology and, more particularly, to a method and system for virtual image compensation and evaluation. Background Art

[0002] Near-eye displays can be provided as augmented reality (AR) displays, virtual reality (VR) displays, head-up displays / head-mounted displays, or other displays. In general, near-eye displays typically include an image generator and an optical combiner that transmits the projected image from the image generator to the human eye. The optical combiner is a set of reflective and / or diffractive optical devices, such as free-form mirrors / prisms, Birdbaths, or cascaded mirrors, and / or grating couplers (waveguides). In addition, the projected image is a virtual image in front of the human eye. The image generator can be a micro-LED-based display, an LCOS (liquid crystal on silicon) display, or a DLP (digital light processing) display. The virtual image is presented to the human eye from the image generator and the optical combiner.

[0003] Uniformity is a key performance indicator for displays, which is used to evaluate image quality. It usually refers to defects in the display matrix and is also called non-uniformity. Non-uniformity (also known as Mura) includes changes in the global distribution and in local areas. For near-eye displays such as AR / VR, visual artifacts such as mottled spots, bright spots, or black spots, or a cloud-like appearance can also be observed on the virtual image presented in the display system. In the virtual image presented in the AR / VR display, non-uniformity can be shown in brightness and / or chromaticity. Compared with traditional displays, non-uniform artifacts are more obvious due to the proximity to the human eye. Therefore, a method for improving virtual image quality is needed. Summary of the invention

[0004] The embodiment of the present disclosure provides a method for compensating a virtual image displayed by a near-eye display based on a micro-display projector. The method includes: acquiring a virtual image displayed by the near-eye display, wherein the virtual image is formed by a source image emitted from the micro-display projector; preprocessing image data of the virtual image to obtain preprocessed image data; acquiring a relationship between the source image and the virtual image; determining an image baseline value of the virtual image; and based on the relationship and the image baseline value, obtaining a compensation factor matrix including a compensation factor for each pixel in the source image.

[0005] The embodiments of the present disclosure also provide a method for evaluating the compensation quality of a virtual image displayed by a near-eye display based on a micro-display projector. The method includes: acquiring a first virtual image displayed by the near-eye display, wherein the virtual image is formed by a source image emitted from the micro-display projector; preprocessing the image data of the first virtual image to obtain preprocessed image data; acquiring a relationship between the source image and the virtual image; determining an image baseline value of the first virtual image; evaluating the non-uniformity of the first virtual image; based on the relationship and the image baseline value, obtaining a compensation factor matrix including a compensation factor for each pixel in the source image; based on the compensation factor matrix, adjusting the image data of each pixel of the first virtual image, and displaying a second virtual image; re-evaluating the non-uniformity of the second virtual image; and comparing the non-uniformity of the second virtual image with the non-uniformity of the first virtual image.

[0006] The embodiments of the present disclosure further provide a device, comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions so that the device performs the above method for compensating a virtual image displayed by a near-eye display based on a micro-display projector.

[0007] The embodiments of the present disclosure further provide a device, comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions so that the device performs the above method for evaluating the compensation quality of a virtual image displayed by a near-eye display based on a micro-display projector. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments and various aspects of the present disclosure are presented in the following detailed description and accompanying drawings.The various features shown in the drawings are not drawn to scale.

[0009] Figure 1 A framework of a homogenization method for improving image quality according to some embodiments of the present disclosure is shown.

[0010] Figure 2 A flow chart illustrating an exemplary compensation method according to some embodiments of the present disclosure is shown.

[0011] Figure 3 (a) Figure 3 (b) and Figure 3 (c) shows examples of an all-white test pattern, a colored captured virtual image, and a pseudo-color brightness distribution image according to some embodiments of the present disclosure.

[0012] Figure 4A flow chart illustrating an exemplary pre-processing method according to some embodiments of the present disclosure is shown.

[0013] Figure 5 Shown are exemplary regions of interest (ROIs) determined from image data according to some embodiments of the present disclosure.

[0014] Figure 6 An exemplary image after distortion correction is shown according to some embodiments of the present disclosure.

[0015] Figure 7 An example of pixel registration from a virtual image to a source image using a mapping ratio is shown according to some embodiments of the present disclosure.

[0016] Figure 8 Examples of image histograms according to some embodiments of the present disclosure are shown.

[0017] Fig. 9 Examples of false-colored generated images with compensation according to some embodiments of the present disclosure are shown.

[0018] Fig.10 Another flow chart of an exemplary image compensation method according to some embodiments of the present disclosure is shown.

[0019] Fig.11 Examples of false-color image non-uniformity are shown according to some embodiments of the present disclosure.

[0020] Fig.12 A flow chart of an exemplary method for re-evaluating non-uniformity of an updated virtual image according to some embodiments of the present disclosure is presented.

[0021] Fig.13 (a) and Fig.13 (b) shows exemplary uniformity before and after compensation according to some embodiments of the present disclosure.

[0022] Fig.14 (a) and Fig.14 (b) shows exemplary brightness distribution before and after compensation according to some embodiments of the present disclosure.

[0023] Fig.15 Nine point uniformity is shown before and after compensation according to some embodiments of the present disclosure.

[0024] Fig.16 is a schematic diagram of an exemplary system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings. The following description is with reference to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements, unless otherwise indicated. The implementations set forth in the following description of the exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of devices and methods consistent with the aspects listed in the appended claims and related to the present invention. Specific aspects of the present disclosure are described in more detail below. If there is a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0026] By developing and integrating a uniformization (also known as demura) algorithm into the display driver system, non-uniformity can be compensated to improve image quality. Demura refers to the process used to eliminate / suppress visual artifacts and achieve relative uniformity of brightness and / or color in a display.

[0027] According to the present disclosure, compensation methods and systems are provided for improving uniformity in near-eye displays.

[0028] Figure 1 The framework of a homogenization method for improving image quality according to some embodiments of the present disclosure is shown. Figure 1 , a rendered virtual image 110 displayed by a near eye display (NED) is acquired by an imaging light measurement device (LMD). After the virtual image is pre-processed (including a registration process) 120, the uniformity of the virtual image is characterized 130 for compensation calculation by comparing with a baseline 131 to obtain non-uniformity 132. Compensation factors for the pixel matrix are generated 140 taking into account the non-uniformity matrix and the target matrix. Finally, the grayscale values ​​of the pixel matrix are adjusted 150 according to the compensation factors for each pixel of the image generator to obtain a rendered virtual image 160 with compensation. The rendered virtual image 160 with compensation can be re-evaluated. Finally, the rendered virtual image 160 with compensation is compared 170 with the rendered virtual image 110 without compensation to determine the uniformity improvement quality (e.g., NU (non-uniformity) ≤ 10%).

[0029] Figure 2 A flow chart illustrating an exemplary compensation method 200 is shown according to some embodiments of the present disclosure. Figure 2 , method 200 includes steps 202 to 216.

[0030] At step 202, a virtual image displayed by a near-eye display (NED) is acquired. The virtual image is presented by the NED and displayed to the human eye by a micro-display projector of the NED. The virtual image is formed by a source image emitted from the micro-display projector and transmitted in front of the human eye. In order to characterize the non-uniformity of the virtual image for further compensation calculation, the virtual image is captured by an imaging LMD (light measurement device). In some embodiments, the LMD may be a colorimeter or an imaging camera, such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor). The grayscale and / or brightness distribution of the virtual image is obtained in the full field of view of the virtual image. Therefore, the grayscale value and / or brightness value (also referred to as image data) of each pixel of the virtual image is obtained. A full white test pattern may be applied in the measurement. In some embodiments, the source image is a full white image (e.g., a full white test pattern), and the virtual image is also a full white image. Therefore, based on the full white test pattern, the compensation factor for the compensation model may be more accurately calculated. In some embodiments, the source image includes a plurality of partially enabled (partial-on) patterns, rather than a full pattern. The plurality of partially enabled patterns are stacked together to form a full white pattern. For example, three partially enabled patterns are presented to the NED in sequence. Finally, a full screen virtual image is obtained. In some embodiments, one or more images with various grayscales / brightnesses can be presented in the NED. According to some embodiments of the present disclosure, Figure 3 (a) shows an example of a full white test pattern, Figure 3 (b) shows the captured virtual image in color, and Figure 3 (c) shows a pseudo-color intensity image. Figure 3 (a) Figure 3 (b) and Figure 3 (c), Virtual image captured by a 2D colorimeter with a NED lens.

[0031] At step 204, the image data of the virtual image is preprocessed to obtain preprocessed image data. The image data includes grayscale values ​​and / or brightness values ​​of the image. Figure 4 A flow chart illustrating an exemplary preprocessing method 400 is shown according to some embodiments of the present disclosure. Figure 4 , the preprocessing method 400 includes steps 402 to 406.

[0032] At step 402, a region of interest (ROI) in the virtual image is extracted. Image data in the ROI of the virtual image is preprocessed. In some embodiments, the ROI may be determined by a preset threshold. Figure 5 An exemplary ROI 510 determined from image data according to some embodiments of the present disclosure is shown. Figure 5, based on a predefined threshold, a ROI for the full field of view of the virtual image is determined 510. In some embodiments, the ROI is determined by comparing the average value of the image data with a threshold, and the average image data value in the region of interest is not less than the threshold. In some embodiments, the ROI is determined by comparing the value of the image data for each pixel with a threshold. For example, the ROI can be determined according to Equation 1:

[0033] (Equation 1)

[0034] Among them, the threshold Can be set based on the image histogram, and The value of the image data representing the pixel. For example, the threshold is set to a grayscale value that is less than 10% of the entire grayscale level (255), for example, the threshold is set to 225.

[0035] In some embodiments, the virtual image is divided into a ROI and a dark area around the ROI (e.g., reference Figure 5 , District 520).

[0036] At step 404, noise points are excluded from the ROI. In some embodiments, noise points are excluded from the ROI by evaluating the emission area and the background area of ​​the virtual image.

[0037] At step 406, distortion correction is performed on the ROI. In some embodiments, preprocessing further includes image distortion correction. Due to the LMD lens and the DUT (device under test) module, the captured virtual image is distorted. In order to obtain accurate distribution data, the captured image needs to be distortion-free, that is, the distortion is corrected by remapping the geometric pixel matrix. Typically, distortion (e.g., barrel distortion) is observed, and an inverse transformation is applied accordingly to correct the distortion. In some embodiments, the distortion can be corrected by Equation 2-1 and Equation 2-2.

[0038] (Equation 2-1)

[0039] (Equation 2-2)

[0040] in, is the coordinate of the pixel after distortion correction, corresponding to the original coordinate The term r represents the distance of the pixel from the center of the virtual image. The terms k1, k2, k3 are coefficients of the distortion parameters. In some embodiments, tangential distortion may also be corrected. Figure 6 An exemplary image after distortion correction is shown according to some embodiments of the present disclosure.

[0041] After step 204, the relationship between the source image and the virtual image is obtained. Return to reference Figure 2 , the relationship between the source image and the virtual image can be obtained through the following steps 206 and 208.

[0042] At step 206, the mapping ratio between the source image and the virtual image is calculated. In order to complete the compensation by the image generator, pixel registration is performed to map the captured virtual image to the matrix array of the image generator. In some embodiments, each pixel of the image generator / source can be extracted by evaluating the mapping ratio and the full field size of the virtual image. In some embodiments, the pixels are identified by image processing such as morphology and feature extraction. The position of the pixel can be determined by morphological image processing (e.g., dilation / erosion, etc.).

[0043] Since the virtual image is captured by a higher resolution imaging LMD, the virtual image is much larger than the source image. For example, the mapping ratio between the virtual image and the source image is 3 or 5. Figure 7 An example of pixel registration from a virtual image to a source image at a mapping ratio of 5 according to some embodiments of the present disclosure is shown. Each unit area 710 (shown as a cross) represents an extracted pixel of the source image. In some embodiments, the mapping ratio is determined by the full field size of the virtual image, the full field size of the source image, the dimension of the virtual image, and the dimension of the source image. For example, the mapping ratio is calculated by Equations 3-1 to 3-3:

[0044] (Equation 3-1)

[0045] (Equation 3-2)

[0046] (Equation 3-3)

[0047] Wherein, R is the mapping ratio, D1 is the size of the imaging LMD used to acquire the virtual image, FOV1 is the effective field of view of the imaging LMD, D2 is the effective emission area of ​​the micro light emitting array in the micro display projector, and FOV2 is the effective field of view of the micro display projector. In some embodiments, the micro display projector includes a micro display panel and a lens. The micro display panel includes a micro light emitting array that can form an effective emission area. For example, the micro display panel is a micro inorganic LED (light emitting diode) display panel, a micro OLED (organic light emitting diode) display panel, or a micro LCD (liquid crystal display) display panel.

[0048] At step 208, a source image data matrix is ​​calculated based on the preprocessed image data and the mapping ratio. The source image data matrix includes the same pixel dimensions of the source image. In some embodiments, the source image data matrix is ​​obtained by Equation 4.

[0049] (Equation 4)

[0050] in, is the source image data, R is the mapping ratio, and M1 is the preprocessed image data matrix composed of the preprocessed image data.

[0051] At step 210, an image baseline value is determined. In order to characterize image uniformity and perform compensation, an image baseline value needs to be set for the overall global representation and compensation target. Therefore, an image baseline value is determined for the entire virtual image. In some embodiments, the image baseline value can be determined by analyzing an image histogram (e.g., the proportion of pixels corresponding to each grayscale value). The grayscale distribution in the entire image is taken into account in the histogram method. Figure 8 An example of an image histogram according to some embodiments of the present disclosure is shown. Figure 8 , showing the number of pixels proportional (i.e., vertical axis) corresponding to each gray value (i.e., horizontal axis). In some embodiments, the image baseline value is determined by the maximum ratio (i.e., peak value) of the pixels. In some embodiments, the baseline value is determined by calculating the average gray value of all pixels. For example, the baseline value can be obtained by Equation 5.

[0052] (Equation 5)

[0053] in, is the baseline value, n is the number of pixels, and The gray value of each pixel.

[0054] At step 212, a compensation factor matrix is ​​calculated based on the source image data matrix and the image baseline value. The compensation factor matrix includes a compensation factor for each pixel in the source image. In some embodiments, the compensation factor matrix for each pixel can be obtained by Equation 6.

[0055] (Equation 6)

[0056] in, represents the compensation factor matrix for the image generator (e.g., ). is the baseline matrix consistent with the image baseline value.

[0057] The compensation factor can be positive or negative. A positive compensation factor means that the original pixel value is pulled up to the baseline value by increasing. A negative compensation factor means that the original pixel value is pulled down to the baseline value by decreasing.

[0058] Therefore, a compensation model (e.g., a compensation factor matrix) for improving image quality is established. The compensation model can be used to further perform compensation for non-uniformity. In some embodiments, the compensation model (e.g., a compensation factor matrix) can be stored in a hard disk of the display system, or in a memory of the micro display panel, so that compensation can be performed on the entire display system.

[0059] In some embodiments, the compensation method 200 may further include step 214 .

[0060] At step 214, image data for each pixel of the virtual image is adjusted based on the compensation factor matrix. In some embodiments, image data for each pixel in the source image is adjusted based on the compensation factor matrix. The source image is an image transmitted by the micro LEDs of the display device to form the virtual image.

[0061] In some embodiments, the adjusted image data for each pixel in the source image is obtained by Equation 7:

[0062] (Equation 7)

[0063] in, is an adjusted image data matrix comprising adjusted image data for each pixel, is a source image data matrix comprising image data for each pixel in the source image, and is the compensation factor matrix.

[0064] In some embodiments, the image data includes a grayscale value for each pixel, for example, the original grayscale value of the pixel is 128. The corresponding compensation factor is 0.2. Therefore, the grayscale value after compensation is 153.6. Using a rounding function (e.g., round or ceil / floor), the grayscale value after compensation is 154. In some embodiments, the compensation capability depends on the display drive system. In some embodiments, the grayscale value after compensation usually overflows the original grayscale value range (e.g., 0 to 255). Therefore, the grayscale value range after compensation includes the original grayscale value range and the extended grayscale value range. For example, the grayscale value range after compensation is a range of 0 to 511 (e.g., with 9 bits). In some embodiments, the grayscale values ​​of some individual pixels still exceed the grayscale value range after compensation. The grayscale values ​​that exceed the grayscale value range after compensation can be truncated at the boundary (e.g., at 0 or at 511). Fig. 9An example of a generated image with compensation in false color according to some embodiments of the present disclosure is shown. Fig. 9 ,and Figure 3 (c) In comparison, the image with compensation is adjusted with grayscale values ​​and the uniformity is improved.

[0065] In some embodiments, the image data includes a brightness value of each pixel. The adjusted image data of each pixel in the source image is obtained by adjusting the brightness value of each pixel.

[0066] In some embodiments, the image data for the micro-luminescent array is adjusted based on the compensation factor matrix. In this example, the grayscale value of each pixel is adjusted to obtain an updated virtual image.

[0067] Therefore, in some embodiments, method 200 further includes step 216 to display the updated virtual image.

[0068] Fig.10 Another flow chart of an exemplary image compensation method 200 according to some embodiments of the present disclosure is shown. Fig.10 As shown in , in order to check the improvement quality of the compensation method, after step 210 , the method 200 further includes step 211 .

[0069] At step 211, the non-uniformity of the virtual image is evaluated. Based on the image baseline value, the non-uniformity of the virtual image can be evaluated. The non-uniformity can be calculated according to Equation 8:

[0070] (Equation 8)

[0071] in, Indicates the non-uniformity of the image. is the source image data matrix, and is the baseline matrix consistent with the image baseline value. Fig.11 An example of image non-uniformity in false color according to some embodiments of the present disclosure is shown. In some embodiments, the non-uniformity can be evaluated directly before mapping the virtual image to the source image matrix.

[0072] In some embodiments, method 200 may further include step 218 .

[0073] At step 218 , the non-uniformity of the updated virtual image is re-evaluated and compared with the non-uniformity of the original virtual image (ie, the non-uniformity evaluated in step 211 ). Fig.10 The other steps in the reference are the same as above. Figure 2 Those described are the same and will not be repeated here.

[0074] Fig.12 1 shows a flow chart of an exemplary method 1200 for re-evaluating the non-uniformity of an updated virtual image according to some embodiments of the present disclosure. Fig.12 As shown in , re-evaluating the non-uniformity of the updated virtual image includes steps 1202 to 1210 .

[0075] At step 1202, a plurality of uniformity zones distributed in the updated virtual image are determined. For example, the plurality of zones may be determined as nine uniformity zones uniformly distributed around the virtual image.

[0076] At step 1204, the brightness values ​​of the plurality of regions are summed. It is noted that the brightness value may be represented by a grayscale value.

[0077] At step 1206, the average brightness value L of the updated virtual image is calculated. 平均 In some embodiments, the average brightness value of the updated virtual image is obtained by Equation 9:

[0078] (Equation 9)

[0079] Wherein, S is the sum of the brightness values ​​of the plurality of regions, and N is the number of regions, for example, N is equal to 9.

[0080] At step 1208, a uniformity value for each of the plurality of zones is obtained. In some embodiments, the uniformity value U for each of the plurality of zones is n From equation 10 we get:

[0081] (Equation 10)

[0082] in, is the uniformity value at zone n, is the brightness value of zone n, and is the average brightness value of the updated virtual image. In this example, n is in the range of 1 to 9. Therefore, the method for re-evaluating the non-uniformity of the updated virtual image can be used to quantitatively evaluate the global uniformity.

[0083] In some embodiments, method 1200 further comprises step 1210, using To calculate the non-uniformity (NU) of the updated virtual image, is the maximum value among the previous uniformity calculations among the n regions.

[0084] In some embodiments, the plurality of regions are determined in the original virtual image, and uniformity values ​​are calculated for the regions in the original virtual image. The non-uniformity of each region of the updated virtual image is then compared to the non-uniformity of the same region of the original virtual image.

[0085] In some embodiments, after comparing the non-uniformity of the original virtual image and the non-uniformity of the updated virtual image, a virtual image with higher uniformity is finally displayed.

[0086] Fig.13 (a) and Fig.13 (b) shows an exemplary uniformity comparison before and after compensation according to some embodiments of the present disclosure. Fig.13 (a) shows the captured virtual image before compensation, and Fig.13 (b) shows the captured virtual image after compensation. Fig.14 (a) and Fig.14 (b) shows exemplary corresponding brightness distributions before and after compensation according to some embodiments of the present disclosure. Fig.14 (a) shows the brightness distribution before compensation, and Fig.14 (b) shows the brightness distribution after compensation. Fig.13 (a) Fig.13 (b) Fig.14 (a) and Fig.14 As shown in (b), the uniformity and brightness distribution are significantly improved. Fig.15 Nine point uniformity before and after compensation according to some embodiments of the present disclosure is shown. Fig.15 , the uniformity values ​​at nine zones are plotted. Fig.15 It is shown that the fluctuations in uniformity in the image distribution have been significantly mitigated and are close to ideal smoothness (ie, uniformity value equal to 1). After compensation, the image quality has been significantly improved.

[0087] By using the homogenization / demura algorithm, the image quality of the virtual image presented in the NED can be significantly improved and visual artifacts can be effectively eliminated after compensation.

[0088] Fig.16 is a schematic diagram of an exemplary system 1600 according to some embodiments of the present disclosure. Fig.16As shown in , a system 1600 is provided for improving the uniformity of a virtual image presented in a near-eye display, and the above-mentioned compensation method 200 can be performed. The system 1600 includes a near-eye display (NED) 1610 for displaying an image in front of a person's eyes, an imager provided as an imaging module 1620, a positioner provided as a positioning device 1630, and a processor provided as a processing module 1640. Additionally, ambient light can be provided by an ambient light module 1650. The near-eye display 1610 can be provided as an AR (augmented reality) display, a VR (virtual reality) display, a head-up display / head-mounted display, or other displays. The positioning device 1630 is provided for setting an appropriate spatial relationship between the near-eye display (NED) 1610 and the imaging module 1620. For example, the positioning device 1630 is configured to set the distance between the near-eye display 1610 and the imaging module 1620 to be within a range of 10 mm to 25 mm. The positioning device 1630 can further adjust the relative position (e.g., distance and spatial position) of the near-eye display 1610 and the imaging module 1620. The imaging module 1620 is configured to simulate the human eye to measure display optical characteristics and to observe display performance. In some embodiments, the imaging module 1620 may include an array light measurement device (LMD) 1622 and a near-eye display (NED) lens 1621. For example, the LMD 1622 may be a colorimeter or an imaging camera such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor). The near-eye display (NED) lens 1621 of the imaging module 1620 is provided with a front aperture having a small diameter of 1 mm to 6 mm. Therefore, the near-eye display (NED) lens 1621 can provide a wide field of view (e.g., 60 degrees to 180 degrees) in front, and the near-eye display lens 1621 is configured to simulate the human eye to observe the near-eye display 1610. The optical properties of the virtual image are measured by the imaging module 1620 based on the positioning device 1630.

[0089] In some embodiments, the near-eye display 1610 may include an image generator 1611 (also referred to herein as an image source) and an optical combiner (also referred to herein as an image optics) ( Fig.16). The image generator 1611 may be a microdisplay such as a microLED, microOLED, LCOS, or DLP display, and may be configured to form a light engine with an additional projector lens. In some embodiments, the microdisplay projector includes a microdisplay panel and a plurality of lenses. The microdisplay panel includes a microlight-emitting array that can form an effective emission area. For example, the microdisplay panel is a microinorganic LED display panel, a microOLED display panel, or a microLCD display panel. The projected image from the light engine is transmitted to the human eye through an optical combiner through designed optical devices. The optical devices of the optical combiner may be reflective and / or diffractive optical devices, such as free-form reflectors / prisms, Birdbaths, or cascaded reflectors, grating couplers (waveguides), etc.

[0090] The processing module 1640 is configured to calculate compensation factors and evaluate uniformity / non-uniformity, etc. In some embodiments, the processing module 1640 may be included in a computer or server. In some embodiments, the processing module 1640 may be deployed in the cloud, which is not limited herein.

[0091] In some embodiments, a driver ( Fig.16 1610). A compensation factor (not shown) may be further provided for compensating the image generator 1611. The compensation factor is calculated in the processing module 1640 and then transmitted to the driving module. Therefore, using the system 1600, a compensation method can be performed. The driving system can be coupled to communicate with the near-eye display 1610, specifically with the image generator 1611 of the near-eye display 1610. For example, the driving module can be configured to adjust the grayscale value of the image generator 1611. When the driving system including the display driving and compensation functions (grayscale value adjustment in image processing) is integrated in the near-eye display, the data of the compensation factor from the processing module 1640 can be transmitted to the near-eye display system 1610.

[0092] In some embodiments, such as for AR applications, ambient light is provided from ambient light module 1650. Ambient light module 1650 is configured to generate a uniform light source with a corresponding color (such as D65) that can support measurements in an ambient light background and simulation of various scenes such as daylight, outdoors, or indoors.

[0093] In some embodiments, a non-transient computer-readable storage medium including instructions is also provided, and the instructions can be executed by the device to perform the above method. Common forms of non-transient media include: for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a tape or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a hole pattern, a RAM, a PROM, and an EPROM, a FLASH-EPROM or any other flash memory, an NVRAM, a cache, a register, any other storage chip or storage box, and their networked versions. The device may include one or more processors (CPU), an input / output interface, a network interface, and / or a memory.

[0094] It should be noted that relational terms, such as "first" and "second", herein are used only to distinguish an entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. In addition, the words "comprising", "having", "containing", and "including" and other similar forms are intended to be equivalent in meaning and are open-ended, and one or more items following any of these words are not meant to be an exhaustive list of such one or more items, or to be limited to the listed one or more items.

[0095] As used herein, unless expressly stated otherwise, the term "or" encompasses all possible combinations unless not feasible. For example, if it is stated that a database may include A or B, then unless expressly stated otherwise or not feasible, the database may include A, or B, or A and B. As a second example, if it is stated that a database may include A, B, or C, then unless expressly stated otherwise or not feasible, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0096] It is understood that the above embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above computer-readable medium. The software can perform the disclosed method when executed by a processor. The computing unit and other functional units described in this disclosure can be implemented by hardware, or software, or a combination of hardware and software. It will also be understood by those of ordinary skill in the art that a plurality of the above modules / units can be combined into one module / unit, and each of the above modules / units can be further divided into a plurality of submodules / subunits.

[0097] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain changes and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art in view of the description and practice of the invention disclosed herein. The description and examples are intended to be considered as merely exemplary, and the true scope and spirit of the invention are indicated by the following claims. The order of steps shown in the accompanying drawings is also intended to be for illustrative purposes only and is not intended to be limited to any particular order of steps. Therefore, it will be appreciated by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0098] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are employed, they are used only in a general and descriptive sense and not for the purpose of limitation.

Claims

1. A method for compensating a virtual image displayed by a near-eye display based on a micro-display projector, the method comprising: acquiring a virtual image displayed by the near-eye display, wherein the virtual image is formed by a source image emitted from the micro-display projector; preprocessing the image data of the virtual image to obtain preprocessed image data; Acquire the relationship between the source image and the virtual image; determining an image baseline value of the virtual image; and Based on the relationship and the image baseline value, a compensation factor matrix including a compensation factor for each pixel in the source image is obtained.

2. The method according to claim 1, wherein: Acquiring the relationship between the source image and the virtual image further includes: calculating a mapping ratio between the source image and the virtual image; calculating a source image data matrix based on the preprocessed image data and the mapping ratio, wherein the source image data matrix comprises the same pixel dimensions of the source image; and The source image data matrix is ​​determined to represent a relationship between the source image and the virtual image.

3. The method according to claim 1, wherein: The virtual image is acquired by an image light measuring device.

4. The method according to claim 1, wherein: The source image is a full white image.

5. The method according to claim 4, wherein: The source image includes a plurality of partially enabled patterns, and the partially enabled patterns are stacked to form the full white image.

6. The method according to claim 1, wherein: Preprocessing the image data of the virtual image to obtain preprocessed image data further comprises: determining a region of interest in the virtual image; and Image data in the region of interest is processed.

7. The method according to claim 6, wherein: The region of interest is determined by a preset threshold, wherein an average image data value in the region of interest is not less than the preset threshold.

8. The method according to claim 7, wherein: The image data value of each pixel in the region of interest is not less than the preset threshold.

9. The method according to claim 7, wherein: The virtual image is divided into the region of interest and a dark region surrounding the region of interest.

10. The method according to claim 6, wherein: Processing the image data in the region of interest further comprises: excluding noise from the region of interest; and The virtual image is corrected for distortion.

11. The method according to claim 10, wherein: The distortion of the virtual image is corrected by the following equation: ； ； in, is the coordinate of the pixel in the virtual image after distortion correction, corresponding to the original coordinate ; r represents the distance from the pixel to the center of the virtual image; and k1, k2, k3 are coefficients of distortion parameters.

12. The method according to claim 2, wherein: The mapping ratio is determined by the full field size of the virtual image, the full field size of the source image, the dimension of the virtual image, and the dimension of the source image.

13. The method according to claim 12, wherein: The mapping ratio is calculated by the following equation: R = R1 / R2, where R1 = D1 / FOV1, R2 = D2 / FOV2, where R is the mapping ratio, D1 is the size of an imaging light measurement device (LMD) used to acquire the virtual image, FOV1 is the effective field of view of the imaging LMD, D2 is the effective emission area of ​​the micro light emitting array in the micro display projector, and FOV2 is the effective field of view of the micro display projector.

14. The method according to claim 1, wherein: Determining the image baseline value of the virtual image further comprises: Acquiring an image histogram of the virtual image data; and The maximum ratio value in the image histogram is determined as the image baseline value.

15. The method according to claim 1, wherein: Determining the image baseline value of the virtual image further comprises: Calculate the average grayscale value of all pixels in the virtual image; and The average gray value is determined as the image baseline value.

16. The method according to claim 2, wherein: The source image data matrix is ​​obtained by the following equation: ， in, is the source image data, R is the mapping ratio, and M1 is the preprocessed image data matrix of the preprocessed image data.

17. The method according to claim 16, wherein: The compensation factor matrix is ​​obtained by the following equation: ； in, is the compensation factor matrix, and is the baseline matrix of the image baseline values.

18. The method of claim 2, further comprising: Image data for each pixel of the virtual image is adjusted based on the compensation factor matrix.

19. The method according to claim 18, wherein: Adjusting the image data of each pixel of the virtual image based on the compensation factor matrix includes: Image data for each pixel in the source image is adjusted based on the compensation factor matrix.

20. The method according to claim 19, wherein: The image data for each pixel in the source image is adjusted by the following equation: ； in, is an adjusted image data matrix comprising adjusted image data for each pixel, is the source image data matrix, and is the compensation factor matrix.

21. The method according to claim 18, wherein: The micro display projector includes a micro display panel and a lens, and the micro display panel includes a micro light emitting array, and adjusting the image data of each pixel of the virtual image based on the compensation factor matrix includes: Image data for each pixel of the micro-light emitting array is adjusted based on the compensation factor matrix.

22. The method according to any one of claims 1 to 21, wherein: The image data includes grayscale values ​​or brightness values.

23. The method of any one of claims 18 to 22, further comprising displaying the updated virtual image.

24. The method according to any one of claims 1 to 20, wherein: The micro display projector includes a micro display panel and a lens, and the micro display panel includes a micro light emitting array.

25. The method according to claim 24, wherein: The micro display panel is one of the following: a micro inorganic LED (light emitting diode) display panel, a micro OLED (organic light emitting diode) display panel, a DLP (digital light processing) display panel, or a micro LCD (liquid crystal display) display panel.

26. The method according to any one of claims 1 to 25, wherein: The near-eye display is one of an augmented reality display, a virtual reality display, a head-up display or a head-mounted display.

27. A method for evaluating compensation of a virtual image displayed by a near-eye display based on a micro-display projector, the method comprising: acquiring a first virtual image displayed by the near-eye display, wherein the virtual image is formed by a source image emitted from the micro-display projector; preprocessing the image data of the first virtual image to obtain preprocessed image data; Acquire the relationship between the source image and the virtual image; determining an image baseline value of the first virtual image; evaluating a non-uniformity of the first virtual image; obtaining a compensation factor matrix including a compensation factor for each pixel in the source image based on the relationship and the image baseline value; adjusting image data of each pixel of the first virtual image based on the compensation factor matrix, and displaying a second virtual image; re-evaluating the non-uniformity of the second virtual image; and The non-uniformity of the second virtual image is compared to the non-uniformity of the first virtual image.

28. The method according to claim 27, wherein: Acquiring the relationship between the source image and the virtual image further includes: calculating a mapping ratio between the source image and the virtual image; and calculating a source image data matrix based on the preprocessed image data and the mapping ratio, wherein the source image data matrix comprises the same pixel dimensions of the source image; and The source image data matrix is ​​determined to represent a relationship between the source image and the virtual image.

29. The method according to claim 27, wherein: The non-uniformity of the first virtual image is calculated by the following equation: ； in, is the non-uniformity of the first virtual image, is the source image data, and is a baseline matrix consistent with the image baseline value.

30. The method according to any one of claims 27 to 29, wherein: Re-evaluating the non-uniformity of the second virtual image further comprises: determining a plurality of uniformity regions distributed in the second virtual image; summing the brightness values ​​of the plurality of regions; The average brightness value of the second virtual image is calculated based on the following equation: ， wherein S is the sum of the brightness values ​​of the plurality of regions, and N is the number of the regions; and The uniformity value of each of the plurality of zones is obtained by the following equation: ， in, is the uniformity value at zone n, is the brightness value of the region n, and is the average brightness value of the second virtual image.

31. The method of claim 30, further comprising: pass to evaluate the non-uniformity of the second virtual image, wherein NU is the non-uniformity of the second virtual image, and is the maximum value of the uniformity value among the N regions.

32. The method according to claim 30 or 31, wherein: Evaluating the non-uniformity of the first virtual image further comprises: determining the plurality of regions in the first virtual image; and calculating a uniformity value for each of the plurality of regions in the first virtual image; and Comparing the non-uniformity of the second virtual image with the non-uniformity of the first virtual image further comprises: The non-uniformity of each of the plurality of regions of the second virtual image is compared to the non-uniformity of the same region of the plurality of regions of the first virtual image.

33. The method of claim 27, wherein: Adjusting the image data of each pixel of the virtual image based on the compensation factor matrix includes: Image data for each pixel in the source image is adjusted based on the compensation factor matrix.

34. The method of claim 27, wherein: The micro display projector includes a micro display panel and a lens, and the micro display panel includes a micro light emitting array, and adjusting the image data of each pixel of the virtual image based on the compensation factor matrix includes: Image data for each pixel of the micro-light emitting array is adjusted based on the compensation factor matrix.

35. An apparatus for compensating a virtual image displayed by a near-eye display based on a micro-display projector, the apparatus comprising: a memory configured to store instructions; as well as One or more processors configured to execute the instructions to cause the apparatus to perform the method according to any one of claims 1 to 26.

36. An apparatus for evaluating compensation of a virtual image displayed by a near-eye display based on a micro-display projector, the apparatus comprising: a memory configured to store instructions; as well as One or more processors configured to execute the instructions to cause the apparatus to perform the method according to any one of claims 27 to 34.

Images