Near-eye display device and image calibration method

Abstract

The present disclosure provides a near-eye display device and an image calibration method, the near-eye display device comprising: a projection module, generating a projection light ray of a projection image; a beam splitter, for transmitting at least part of the projection light ray in a calibration mode, and reflecting at least part of the projection light ray in a viewing mode for viewing; an image generation module, for converting the transmitted at least part of the projection light ray into a re-imaged image in the calibration mode; a controller, for obtaining an image calibration parameter based on a reference image corresponding to the re-imaged image and the projection image in the calibration mode, calibrating the re-imaged image based on the image calibration parameter, and updating the projection image with the calibrated re-imaged image until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and switching to the viewing mode. The near-eye display device can be calibrated in real time on the spot dynamically.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of near-eye display, and particularly relates to a near-eye display device and an image calibration method. BACKGROUND

[0002] A video see-through head-mounted projection module is a device that displays a real scene captured by a camera in real time and virtual content in a head-mounted projection module. The picture observed by the user through the projection module is a composite image of the real scene captured by the camera and the superimposed virtual content, thereby realizing the visual effect of augmented reality (AR), virtual reality (VR) or mixed reality (MR).

[0003] As a core hardware device of near-eye display, the display accuracy and stability of the video see-through head-mounted projection module directly affect the user experience. In order to eliminate the optical system aberration, assembly error and long-term use drift, in the related technology, the calibration parameters are fixed in the device before the near-eye display device is shipped by using precise optical equipment (such as a standard calibration plate) in a laboratory environment for offline calibration. However, this kind of calibration method cannot dynamically adjust the calibration parameters, and cannot cope with the drift caused by the change of the environment (such as temperature and humidity) or the long-term use of the device in actual use, resulting in poor calibration effect.

[0004] Therefore, there is an urgent need for a near-eye display device that can realize in-situ dynamic real-time calibration. SUMMARY

[0005] The present disclosure provides a near-eye display device and an image calibration method, which can realize in-situ dynamic real-time calibration of the near-eye display device.

[0006] The technical solution of the present disclosure is implemented as follows:

[0007] In a first aspect, the present disclosure provides a near-eye display device, comprising: a projection module, generating a projection light ray of a projection image; a beam splitter, configured to transmit at least part of the projection light ray in a calibration mode, and reflect at least part of the projection light ray in a viewing mode for viewing; an image generation module, configured to convert the transmitted at least part of the projection light ray into a re-imaged image in the calibration mode; and a controller, configured to obtain an image calibration parameter based on a reference image corresponding to the re-imaged image and the projection image in the calibration mode, calibrate the re-imaged image based on the image calibration parameter, and update the projection image using the calibrated re-imaged image, until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and then switch to the viewing mode.

[0008] In a second aspect, the disclosure provides an image calibration method, which comprises: in a calibration mode, obtaining an image calibration parameter based on a reference image corresponding to a re-imaged image and a projection image, the re-imaged image being obtained by converting at least part of the projection light transmitted by the beam splitter through the image generation module, the projection light being generated by the projection module based on the projection image; calibrating the re-imaged image based on the image calibration parameter; and updating the projection image using the calibrated re-imaged image until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and switching to a viewing mode.

[0009] In a third aspect, the disclosure provides a computer-readable storage medium having stored thereon a program or instructions, which, when executed by a processor, implement the steps of the image calibration method according to the second aspect.

[0010] In a fourth aspect, the disclosure provides a computer program product, wherein the computer program product comprises a computer program or instructions, and when the computer program product is executed on a processor, the processor executes the computer program or instructions to implement the steps of the image calibration method according to the second aspect.

[0011] In a fifth aspect, the disclosure provides a chip comprising a processor and a communication interface, the communication interface being coupled to the processor, and the processor being configured to execute a program or instructions to implement the steps of the image calibration method according to the second aspect.

[0012] The disclosure provides a near-eye display device and an image calibration method. If the difference between the re-imaged image and the reference image is less than or equal to the difference threshold, it indicates that the near-eye display device has a small error and does not need to be calibrated. If the difference between the re-imaged image and the reference image is greater than the difference threshold, it indicates that the image output by the projection module in the current working environment needs to be calibrated by the image calibration parameter. Therefore, the calibration mode is entered, the image calibration parameter is determined according to the difference between the re-imaged image and the reference image, and the projection image is updated using the calibrated re-imaged image, i.e., the controller sends the calibrated re-imaged image to the projection module, the projection module continues to project the calibrated re-imaged image, and the corresponding re-imaged image is generated again. When the difference between the final re-imaged image and the reference image is less than or equal to the difference threshold, it indicates that the calibration is completed, and the near-eye display device can enter the viewing mode. In the subsequent viewing mode, the determined image calibration parameter can be used to calibrate the image to be displayed. The disclosure realizes the function of real-time calibration of the light near-eye display device (only integrated with the beam splitter) in the field by the dynamic beam splitting of the beam splitter, the real-time imaging of the image generation module, and the iterative processing of the controller. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1A structural schematic diagram of a near-eye display device provided by the present disclosure.

[0014] Figure 2 A structural schematic diagram of a near-eye display device provided by the present disclosure, in which a reference image is an image corresponding to ambient light.

[0015] Figure 3 A structural schematic diagram of a beam splitter provided by the present disclosure.

[0016] Figure 4 A structural schematic diagram of another beam splitter provided by the present disclosure.

[0017] Figure 5 A light path schematic diagram of a near-eye display device in a viewing mode provided by the present disclosure.

[0018] Figure 6 A light path schematic diagram of a near-eye display device in a calibration mode provided by the present disclosure.

[0019] Figure 7 A structural schematic diagram of another near-eye display device provided by the present disclosure.

[0020] Figure 8 A flow schematic diagram of an image calibration method provided by the present disclosure.

[0021] Figure 9 A flow schematic diagram of another image calibration method provided by the present disclosure.

[0022] Figure 10 A flow schematic diagram of yet another image calibration method provided by the present disclosure. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present disclosure will be clearly described below with reference to the drawings in the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present disclosure.

[0024] The terms “first”, “second”, and the like in the specification of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the present disclosure can be implemented in an order other than that illustrated or described herein, and the objects distinguished by “first”, “second”, and the like are generally of a kind and are not limited in number, for example, the first object can be one or more.

[0025] Video see-through head-mounted display is a technology that captures the external environment in real time through a camera (objective lens), combines sensor (such as gyroscope, accelerometer) to obtain the user's head posture, and accurately superimposes virtual content (such as 3D model, information label) into the real scene. The picture seen by the user is a mixed image of "reality + virtual" after digital processing.

[0026] Since the user relies entirely on the real-time video captured by the camera, the virtual content is superimposed into the video through digital image processing technology. Video see-through can dynamically adjust the real scene (such as brightness, contrast, distortion correction), and is easier to realize virtual-real occlusion relationship processing. Therefore, video see-through technology is widely used in near-eye display devices.

[0027] In order to ensure high-precision image display and user experience, existing video see-through head-mounted displays usually need to be accurately calibrated to eliminate optical system aberration, component assembly error, and drift that may occur during long-term use. Optical system aberration is the deviation between actual imaging and ideal imaging when light passes through optical elements due to design or manufacturing defects. Aberration directly affects the geometric accuracy, color consistency, and image quality of virtual-real fusion, causing the user to perceive blurred virtual objects, edge artifacts, and color distortion, which destroys the sense of immersion. Assembly error is the physical position deviation of optical elements and sensors during manufacturing and assembly due to process limitations, causing the user to perceive virtual object position deviation, jitter, or proportion misadjustment (such as AR navigation arrow deviating from the actual road). Long-term use drift is the slow shift of parameters caused by environmental changes, material aging, or mechanical stress accumulation during system operation.

[0028] To address the above errors, traditional video see-through performs offline calibration before leaving the factory, which requires the use of high-precision equipment (such as laser interferometer, standard calibration plate) to calibrate system parameters in the laboratory. However, this method cannot correct drift or environmental changes (such as temperature fluctuations) during use, cannot perform dynamic adjustment, and can only be periodically returned to the factory for maintenance. External devices can also be used for calibration, such as relying on external cameras, projectors, or calibration targets (such as checkerboard patterns) for in-situ calibration. However, this method not only increases hardware costs, but also has high operation complexity. Since near-eye display devices require lightweight, it is difficult to integrate them into near-eye display devices.

[0029] Based on the above problems, the present disclosure aims to provide a lightweight near-eye display device that can realize in-situ real-time calibration, such as Figure 1As shown, the near-eye display device comprises: a projection module 10 for generating projection light rays corresponding to a projection image; a beam splitter 20 for, in a calibration mode, transmitting at least part of the projection light rays, and, in a viewing mode, reflecting at least part of the projection light rays for viewing; an image generation module 30 for, in the calibration mode, converting the transmitted at least part of the projection light rays into a re-imaged image; and a controller 40 for, in the calibration mode, obtaining image calibration parameters based on a reference image corresponding to the projection image and the re-imaged image, calibrating the re-imaged image based on the image calibration parameters, and updating the projection image with the calibrated re-imaged image until a difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and switching to the viewing mode.

[0030] The projection module 10 is typically a high-resolution micro display such as an Organic Light Emitting Diode (OLED), a Micro Light-Emitting Diode (Micro-LED), or a Liquid Crystal on Silicon (LCoS).

[0031] The projection image displayed by the projection module 10 corresponds to the projection light rays generated thereby. The reference image corresponding to the projection image is a standard image that the projection image is ultimately expected to display, and in an ideal case where the near-eye display device has no error, the projection image is identical to the reference image. The reference image can be, for example, a checkerboard, a dot pattern, a specific geometric pattern, a specific color image, or an image of an external environment captured by the near-eye display device. If the reference image is an image of an external environment captured by the near-eye display device, the image of the external environment can be pre-stored or acquired in real time. It should be noted that if an image of an external environment captured by the near-eye display device is used as the reference image, a scene with rich feature information needs to be photographed, such as a building or a fence with many straight lines or edges, so that the reference image contains rich features and is convenient for subsequent comparison with the re-imaged image.

[0032] The beam splitter 20 achieves light splitting by coating technology (such as dielectric film or metal film) and can control the transmission and reflection ratio of light passing through. The beam splitter 20 can be a polarizing beam splitter, a dichroic beam splitter, or the like, which can achieve proportional transmission and reflection of incoming light rays, and the present disclosure does not limit the material and implementation principle of the beam splitter 20.

[0033] In combination Figure 1 In the calibration mode, the projection light rays emitted by the projection module 10 (represented by three solid arrows in the figure pointing from the projection module 10) are transmitted in full by the beam splitter 20 to the image generation module 30, and the transmitted projection light rays are represented by dashed arrows in the figure. At this time Figure 1part of the projection light is reflected by the beamsplitter 20; another case is that part of the projection light is transmitted by the beamsplitter 20 to the image generation module 30, and is reflected (indicated by the dotted arrow in the figure). Figure 1 part of the projection light is reflected by the beamsplitter 20; another case is that part of the projection light is transmitted by the beamsplitter 20 to the image generation module 30, and is reflected (indicated by the dotted arrow in the figure).

[0034] In the calibration mode, if part of the projection light is reflected by the beamsplitter 20, the part of the projection light can be used to preview the effect of the calibration in real time; in the viewing mode, if part of the projection light is transmitted by the beamsplitter 20, the part of the projection light can be used to monitor whether the near-eye display device needs to be calibrated in real time.

[0035] The function of the image generation module 30 is to convert the light signal into a digital image, i.e., an image sensor (such as a Complementary Metal Oxide Semiconductor, CMOS or Charge Coupled Device, CCD). In the calibration mode, the image generation module 30 captures the projection light transmitted by the beamsplitter 20, and generates a re-imaged image, i.e., the actual imaging of the projection image displayed by the projection module 10 after the optical link.

[0036] The controller 40 has the ability to communicate with a remote server, and can perform different operations according to the storage location of the reference image: in the case that the reference image is stored in the remote server, the controller 40 sends the calibration image to the remote server; in the case that the reference image is stored in the near-eye display device, the controller 40 sends the reference image and the re-imaged image to the remote server together; the remote server analyzes the difference between the re-imaged image and the reference image, determines whether the difference is greater than a difference threshold, in the case that the difference is greater than the difference threshold, generates image calibration parameters, and sends them back to the controller 40, the controller 40 controls the near-eye display device to calibrate the output image according to the received image calibration parameters.

[0037] In some implementations, based on the re-imaged image and the reference image, the image calibration parameters are obtained, including performing feature extraction of the same scale on the re-imaged image and the reference image, to obtain re-imaged image features corresponding to the re-imaged image, and reference image features corresponding to the reference image; according to the feature difference between the re-imaged image features and the reference image features, at least one of the geometric calibration parameters, the color calibration parameters, the brightness calibration parameters, and the contrast calibration parameters is determined.

[0038] Specifically, the same scale feature extraction is performed on the re-imaged image (actually captured after projection) and the reference image (theoretical target of the projected image), such as: geometric features, edge extraction, corner point, shape contour, etc., for detecting distortion or deviation of the optical system; color features, analyzing color distribution (such as RGB histogram, color temperature) to calibrate color difference; brightness and contrast features, statistics of pixel brightness mean and variance, adjusting the dynamic range of the display device. Feature extraction can be realized by traditional algorithms (such as SIFT, HOG) or deep learning models (such as convolutional neural network). Comparing the features of the re-imaged image and the reference image, the geometric calibration parameters for correcting optical distortion (such as barrel distortion, pincushion distortion) or projection angle deviation, the color calibration parameters for adjusting color gamut mapping or compensating the influence of ambient light on color, the brightness calibration parameters and contrast calibration parameters for optimizing display brightness and contrast to adapt to different environmental light conditions are obtained. Thus, through comprehensive calibration of geometric, color, brightness, contrast, etc. parameters, the consistency of the projected image and the reference image is ensured, the display precision is improved, and the calculation of image calibration parameters is based on the quantitative analysis of feature differences, avoiding full-image pixel-by-pixel comparison and reducing the consumption of computing resources.

[0039] Specifically, the re-imaged image and the reference image are respectively pre-processed, Gaussian filtering or non-local mean denoising is adopted to reduce image noise, histogram equalization or brightness scaling is used to unify the dynamic range of the two images (normalization processing); feature points of the two images are extracted by feature extraction algorithm or deep learning; the two images are registered by RANSAC or homography matrix to establish pixel-level correspondence; geometric deviation, color deviation, brightness deviation, contrast deviation, etc. of the two images are analyzed to generate pre-distortion matrix, color correction matrix and brightness, contrast compensation parameters, which are integrated into display matrix, i.e. image calibration parameters. The obtained image calibration parameters are used to correct the image output by the projection module 10 in real time, the re-imaged image is continuously collected, and the difference is analyzed until the difference between the re-imaged image and the reference image is less than the difference threshold, and the final image calibration parameters are obtained.

[0040] In some embodiments, for the difference determination between the reference image and the re-imaged image, specifically: based on the re-imaged image features and the reference image features, the feature similarity between the re-imaged image and the reference image is determined; in the case that the feature similarity is less than or equal to the similarity threshold, it is determined that the difference between the re-imaged image and the reference image is less than or equal to the difference threshold.

[0041] The controller 40 calculates the feature similarity (for example, using the SSIM structural similarity index, cosine similarity or a self-defined weighted score) through the matching degree of the reimaging image feature and the reference image feature, and the specific feature similarity determination strategy is not limited, for example, the feature similarity score can be obtained by weighted sum of the geometric feature difference, color feature difference, brightness feature difference and contrast feature difference. When the feature similarity is less than or equal to a preset threshold (for example, SSIM < 0.95), it indicates that the reimaging image and the reference image are too different, and the calibration needs to be continued. When the feature similarity exceeds the threshold, it is determined that the calibration is completed, and the viewing mode is automatically switched. The similarity threshold setting clearly defines the calibration termination condition, avoids subjective judgment errors, improves the calibration reliability, and realizes the closed-loop process of "calibration-update-re-calibration" based on the real-time feedback of the feature similarity, without manual intervention.

[0042] It should be noted that, in combination with Figure 1 The purpose of calibration is to make the image corresponding to the reflected part of the projection light (the image seen by the human eye) consistent with the expected reference image (the difference is less than the difference threshold). Since the light entering the human eye cannot be directly measured, the image corresponding to the transmitted part of the projection light is the same as the image corresponding to the reflected part of the projection light, so the image corresponding to the transmitted part of the projection light (i.e., the reimaging image) is collected by the image generation module 30 as the image seen by the human eye. By comparing the difference between the reimaging image and the reference image, the difference between the image seen by the human eye and the expected reference image can be determined.

[0043] In the embodiments of the present disclosure, if the difference between the reimaging image and the reference image is less than or equal to the difference threshold, it indicates that the near-eye display device has a small error, and the image does not need to be calibrated; if the difference between the reimaging image and the reference image is greater than the difference threshold, it indicates that the image output by the projection module in the current working environment needs to be calibrated by the image calibration parameter, so the calibration mode is entered, the image calibration parameter is determined according to the difference between the reimaging image and the reference image, and the reimaging image after calibration is used to update the projection image, that is, the controller sends the reimaging image after calibration to the projection module, the projection module continues to project the reimaging image after calibration to generate the corresponding reimaging image again, until the difference between the final reimaging image and the reference image is less than or equal to the difference threshold, which indicates that the calibration is completed, and the near-eye display device can enter the viewing mode. The determined image calibration parameter can be used to calibrate the image to be displayed in the subsequent viewing mode. The present disclosure realizes the lightweight (only integrated with a beam splitter) real-time calibration of the near-eye display device through the dynamic beam splitting of the beam splitter, the real-time imaging of the image generation module and the iterative processing of the controller.

[0044] In some embodiments, the reference image is an image of the external environment captured by the near-eye display device, in particular, as shown in Figure 2 the beamsplitter 20 is also configured to reflect at least part of the external environment light to the image generation module 30; and the image generation module 30 is also configured to convert at least part of the external environment light into the reference image. Figure 2 The external environment light is shown by the black arrows.

[0045] The beamsplitter 20 can reflect part of the external environment light to the image generation module 30, or can reflect all of the external environment light to the image generation module 30, according to a preset beamsplitting ratio. The preset beamsplitting ratio is different for different materials of the beamsplitter 20. For example, the preset beamsplitting ratio is 9:1, i.e., 90% of the projected light is reflected and 10% of the projected light is transmitted through the beamsplitter 20; or the preset beamsplitting ratio is 1:1, i.e., 50% of the projected light is reflected and 50% of the projected light is transmitted through the beamsplitter 20.

[0046] The external environment light is converted into the reference image. The image generation module 30 time-multiplexes the light, i.e., after entering the calibration mode, the image generation module 30 first converts the external environment light into the reference image and stores the reference image in the controller 40. The real-time captured environment image is directly used as a reference. The environment for capturing the reference image is exactly the same as the environment for using the near-eye display device, so that new errors caused by changes in the environment (e.g., fluctuations in light and temperature) are avoided. The reference image is generated based on actual environment light, rather than idealized preset values, so that the calibration parameters are more suitable for real use scenarios. Real-time changes in the external environment light are continuously fed back to the reference image, and the controller 40 iteratively calibrates based on the reference image, so that display distortion caused by sudden changes in the environment is avoided.

[0047] In some embodiments, as shown in Figure 3 the beamsplitter 20 includes a plurality of layers (two layers are shown in Figure 3 ) of dielectric reflection film 201 and light-transmitting optical substrate 202; the dielectric reflection film 201 is configured to reflect and transmit the incoming projected light according to a preset beamsplitting ratio; and the light-transmitting optical substrate 202 is configured to support the plurality of layers of dielectric reflection film 201.

[0048] The dielectric reflection film 201 is formed by alternately depositing high-refractive-index and low-refractive-index optical thin films (e.g., silicon dioxide and titanium dioxide). The beam splitter 20 uses a composite structure composed of two dielectric reflective films 201 to selectively reflect and transmit light of specific wavelengths through interference effects, controlling the beam splitting ratio (e.g., a reflection-to-transmission ratio of 9:1). The beam splitting ratios of the two dielectric reflective films 201 of the beam splitter 20 can be the same or different; that is, the beam splitting ratios of the front and rear surfaces of the beam splitter 20 can be the same or different. The transparent optical substrate 202 is the transparent support substrate of the beam splitter 20, typically made of a high-transmittance, low-birefringence material (such as BK7 glass, fused silica, or polycarbonate). In some feasible implementations, the transparent optical substrate 202 is... Figure 3 The curved substrate shown can converge the light transmitted through beam splitter 20 ( Figure 2 The projected light emitted from the projection module 10 (indicated by the dashed arrow in the middle).

[0049] Due to the presence of the multilayer dielectric reflective film 201, the beam splitter 20 reflects and transmits the incoming light in a certain ratio. That is, in viewing mode, when the projected light emitted by the projection module 10 passes through the beam splitter 20, a portion of the projected light (e.g., 95%) is reflected into the human eye; and a portion of the projected light (e.g., 5%) is transmitted into the image generation module 30.

[0050] In calibration mode, ambient light is reflected (e.g., 95%) by the beam splitter 20 and enters the image generation module 30, while a portion is transmitted through the beam splitter 20 into the human eye. The image generation module 30 converts the reflected ambient light into a reference image, and the controller 40 sends the reference image to the projection module 10. In calibration mode, the brightness of the projected light from the projection module 10 (e.g., the display brightness of the projection module 10 is adjusted to the maximum) is greater than the brightness of the projected light in viewing mode, so that the image generation module 30 collects more projected light transmitted through the beam splitter 20.

[0051] In this embodiment, the beam splitter 20 can reflect and transmit incoming light according to a preset beam splitting ratio, enabling users to use the near-eye display device normally in viewing mode; at the same time, in calibration mode, it enables on-site real-time calibration of the near-eye display device.

[0052] In some embodiments, such as Figure 4 As shown, the beam splitter 20 also includes an electrochromic layer 203, which has different reflection and transmission ratios under different voltages.

[0053] The controller 40 is also configured to apply a first voltage to the beam splitter 20 in viewing mode, wherein the reflectivity of the beam splitter 20 is greater than its transmittance under the first voltage; and, in calibration mode, apply a second voltage to the beam splitter 20 after sending a projected image to the projection module 10, wherein the transmittance of the beam splitter 20 is greater than its reflectivity under the second voltage.

[0054] The electrochromic layer 203 is a thin film structure composed of electrochromic material, which can undergo reversible electrochemical reactions when an external voltage is applied, thereby changing the reflectivity and transmissivity. The transmissivity and reflectivity can be precisely controlled by adjusting the voltage size and polarity.

[0055] In some implementable manners, the beamsplitter 20 has a transmissivity of 0 at the first voltage and a reflectivity of 0 at the second voltage. In the viewing mode, the beamsplitter 20 is required to reflect all the incoming light, as shown in FIG. 4, which is a propagation path of light in the viewing mode. The ambient light is totally reflected by the beamsplitter 20 to the image generation module 30, and the projection light generated by the projection module 10 is also totally reflected by the beamsplitter 20 into the human eye. Figure 5

[0056] As shown in FIG. 5, which is a propagation path of light in the calibration mode. At the first voltage, the beamsplitter 20 totally reflects the ambient light to the image generation module 30, the image generation module 30 generates a reference image and sends it to the projection module 10 via the controller 40, the controller 40 applies the second voltage to the beamsplitter 20 and stops collecting the ambient light, at the second voltage, the beamsplitter 20 totally transmits the projection light generated by the projection module 10 when displaying the reference image to the image generation module 30, and the image generation module 30 generates a re-imaged image according to the total projection light. Figure 6

[0057] It should be noted that in the viewing mode, although the brightness of the projection module 10 is not high, and the transmissivity of the beamsplitter 20 is very low, even without the electrochromic layer 203, the transmitted part of the projection light of the projection module 10 has little effect on the ambient light of the image generation module 30. However, in order to make the imaging more accurate, in the viewing mode, the electrochromic layer 203 included in the beamsplitter 20 works at the first voltage, and the transmissivity is 0, so that the projection light of the projection module 10 is totally reflected, thereby avoiding the influence of the transmitted part of the projection light on the imaging of the image generation module 30.

[0058] In the embodiments of the present disclosure, the electrochromic layer 203 is added to the beamsplitter 20, so that in the viewing mode and the calibration mode, the light passing through the beamsplitter 20 does not interfere with each other, avoiding the reference image and the re-imaged image being disturbed by irrelevant light, so that the final calibration result is more accurate.

[0059] In some embodiments, in the calibration mode, the reference image is a monochromatic image; the beamsplitter 20 is configured to transmit light corresponding to a wavelength of the monochromatic image in the calibration mode; and the controller 40 is configured to obtain the image calibration parameter based on the re-imaged image and the reference image, including: obtaining an image calibration parameter corresponding to the wavelength of the monochromatic image based on the re-imaged image and the reference image. ​​

[0060] Different wavelengths of light are significantly different in sensitivity to geometric deviations (such as distortion, displacement), color deviations (color shift, dispersion), brightness deviations (response nonlinearity), and contrast deviations (signal-to-noise ratio) in the optical system. Through wavelength calibration, complex errors can be decomposed into independent parameters, and precise segmented optimization can be achieved. For example, short wavelengths (blue light, 450 nm) are significantly affected by diffraction effects and are easily affected by the curvature of optical elements, and are suitable for adjusting geometric distortion; medium wavelengths (green light, 550 nm) are sensitive to green light in the image generation module 30, and are easy to expose to brightness gradient, and are suitable for adjusting brightness uniformity error; long wavelengths (red light, 650 nm) have strong penetration and are easily affected by coating dispersion, and are suitable for adjusting color deviation; wide spectrum (full band), multi-wavelength superposition can evaluate the nonlinear response of the image generation module 30, and is suitable for adjusting contrast error.

[0061] Specifically, the controller 40 generates a set of monochromatic images, such as a blue light checkerboard pattern (calibrate geometric distortion), a green light uniform gray image (calibrate brightness uniformity), and a red light pure color filling image (calibrate color channel gain), each image corresponding to a specific wavelength. The projection module 10 displays each wavelength monochromatic image in turn, the beam splitter 20 transmits the projected light to the image generation module 30, and the image generation module 30 captures the transmitted projected light to generate a reimaging image corresponding to the wavelength. The reimaging image of a wavelength is compared with the corresponding reference image (the monochromatic image generated by the controller 40, such as the blue light checkerboard pattern displayed by the projection module 10 and then transmitted by the beam splitter 20, and the reimaging image generated by the image generation module 30, the corresponding reference image is the blue light checkerboard pattern), and the partial image calibration parameters corresponding to the wavelength are obtained. After each reference image is compared and analyzed with the corresponding reimaging image, the final complete image calibration parameters are obtained.

[0062] In the embodiments of the present disclosure, through the closed-loop design of wavelength projection-sensing-analysis, complex full-link errors are decomposed into wavelength-sensitive independent parameters, and precise segmented calibration of geometric, color, brightness, and other deviations is achieved.

[0063] In some embodiments, as shown in Figure 7 The near-eye display device further includes a light ray collection module 50 and / or an image optimization module 60. The light ray collection module 50 is configured to collect ambient light rays from a predetermined area of the near-eye display device and converge the ambient light rays. The image optimization module 60 is configured to receive at least part of the projected light rays reflected by the beam splitter 20 and project the at least part of the projected light rays to the human eye.

[0064] The light collection module 50 can be an objective lens. According to different focal lengths, the light collection module 50 can collect external environment light of a preset area at different distances from the near-eye display device. The shorter the focal length, the higher the magnification of the light collection module 50, and the wider the field of view, which can collect more light, but the working distance (the farthest distance from the near-eye display device that can be collected) will be shorter. That is, a short focal length usually corresponds to a high magnification, a small field of view, and low image brightness, and a long focal length usually corresponds to a low magnification, a large field of view, and high image brightness.

[0065] In order to adapt to the near-eye display device to different distances of the external scene can be adapted more accurate image, in the calibration mode, change the focal length of the light collection module 50, to collect the light of the preset area at different distances from the near-eye display device, so as to obtain reference images at different focal lengths, based on the difference of reference images at different focal lengths, finally obtain an image calibration parameter corresponding to each focal length. In this way, in the viewing mode, if the focal length of the light collection module 50 is adjusted, the image calibration parameter corresponding to the focal length can be used to calibrate the image corresponding to the collected external environment light, so that the final calibration effect is more accurate.

[0066] Since the change of the focal length affects the brightness value of the image, in some realizable ways, the corresponding relationship between the focal length and the brightness can be stored in advance. In the viewing mode, the controller 40 can also adjust the brightness of the calibrated projection image according to the brightness corresponding to the focal length after calibrating the projection image through the image calibration parameter.

[0067] It should be noted that in the calibration mode, in order to avoid the external environment light reflected by the beam splitter 20 into the image generation module 30, affecting the generation of the re-imaging image, therefore, in the calibration mode, after obtaining the reference image, the external environment light needs to be shielded, such as: if the near-eye device includes a light collection module 50, after obtaining the reference image, the light collection module 50 is shielded (such as closing the objective lens cover).

[0068] The image optimization module 60 can be an eyepiece, used to receive at least part of the projection light reflected by the beam splitter 20 to project at least part of the projection light to the human eye. The image optimization module 60 further magnifies and adjusts the focal length of the image corresponding to the part of the projection light reflected by the beam splitter 20, so that the observer can see clearer details.

[0069] The near-eye display device provided by the present disclosure has the following technical effects:

[0070] The spectroscope switches the transmission / reflection ratio in two modes, so that the same light path can project a display image (viewing mode) and transmit part of the light to an image sensor to generate a re-imaging image (calibration mode), thereby realizing hardware lightweight without additional sensors or mechanical structures.

[0071] The errors such as geometric distortion, chromatic aberration, and uneven brightness have wavelength sensitivity, and full-band calibration is prone to parameter coupling errors, so single-color image wavelength calibration is adopted to optimize different deviations.

[0072] The multiplexing spectroscope 20 collects real-time ambient light as a dynamic reference image. In the calibration mode, the spectroscope 20 reflects the ambient light to generate a reference image and transmits the projection light to generate a re-imaging image, ensuring that the reference image is consistent with the current environment. In addition, the electrochromic layer 203 controls the transmission / reflection ratio of the spectroscope 20 (e.g., the transmittance is greater than 90% at the second voltage), isolating the mutual interference of ambient light and projection light and improving the robustness of calibration.

[0073] Pixel-by-pixel comparison of the full image requires a large amount of computation and is difficult to meet real-time requirements. Therefore, based on the quantification calibration of feature differences, combined with closed-loop iterative optimization, the key differences between the re-imaging image and the reference image are extracted through features such as edges, color temperature, and brightness distribution (traditional algorithms or CNN), the data amount is reduced, and the pre-distortion matrix and color correction matrix are calculated based on the feature differences, the projection output is adjusted accordingly, and the controller 40 continuously compares the updated re-imaging image and the reference image until the feature similarity (e.g., SSIM) meets the standard, and the calibration is automatically terminated.

[0074] The near-eye display device needs to adapt to different observation distances (e.g., virtual-real fusion scenarios of AR glasses), and the light collection module dynamically adjusts the focal length to generate a multi-distance reference image library, and stores corresponding geometric / luminance parameters for different focal lengths (e.g., high distortion compensation for short focal length). In the viewing mode, the pre-stored parameters are called according to the current focal length, and the ambient light is dynamically fine-tuned to ensure consistency in multiple scenarios.

[0075] As shown in FIG. 8, Figure 8 The present disclosure provides an image calibration method, which is exemplarily described below with the controller 40 as an execution subject. The method can include the following steps 801-805.

[0076] In step 801, in the calibration mode, based on the reference image corresponding to the re-imaging image and the projection image, the image calibration parameters are obtained.

[0077] The re-imaging image is obtained by converting at least part of the projection light transmitted by the spectroscope through the image generation module, and the projection light is generated by the projection module based on the projection image.

[0078] In step 802, the re-imaging image is calibrated based on the image calibration parameters.

[0079] In step 803, the projected image is updated using the calibrated re-imaging image.

[0080] In step 804, it is determined whether the difference between the re-image and the reference image is less than or equal to the difference threshold.

[0081] If yes, proceed to step 805; otherwise, proceed to steps 801 through 804.

[0082] In step 805, switch to viewing mode.

[0083] In some embodiments, combined with Figure 8 ,like Figure 9 As shown, in step 801 above, in calibration mode, image calibration parameters are obtained based on the re-imaging image and the reference image, including the following steps 801a and 801b.

[0084] In step 801a, in calibration mode, features of the same scale are extracted from the re-imaging image and the reference image to obtain the re-imaging image features corresponding to the re-imaging image and the reference image features corresponding to the reference image.

[0085] In step 801b, at least one of the geometric calibration parameter, color calibration parameter, brightness calibration parameter, and contrast calibration parameter is determined based on the feature differences between the re-image features and the reference image features.

[0086] In some embodiments, combined with Figure 9 ,like Figure 10 As shown, step 804 above can be implemented through steps 804a and 804b below.

[0087] In step 804a, the feature similarity between the re-image and the reference image is determined based on the features of the re-image and the reference image.

[0088] In step 804b, it is determined whether the feature similarity is less than or equal to the similarity threshold.

[0089] If yes, proceed to step 805; otherwise, proceed to steps 801 through 804.

[0090] In step 805, switch to viewing mode.

[0091] In this embodiment of the present disclosure, the effect of the image calibration method provided in the present disclosure can be referred to the technical effect of the controller 40 in the near-eye display device. To avoid repetition, it will not be described again here.

[0092] The present disclosure also provides a computer readable storage medium storing at least one instruction for being executed by a processor to implement the image calibration method according to any of the above embodiments.

[0093] The present disclosure also provides a computer program product comprising computer instructions stored in a computer readable storage medium; a processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the image calibration method according to any of the above embodiments.

[0094] The present disclosure also provides a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute programs or instructions to implement the processes of the above image calibration method embodiments and achieve the same technical effects. For the sake of brevity, the same will not be repeated here.

[0095] It should be understood that the chip mentioned in the present disclosure can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0096] In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device, server and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0097] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0098] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit.

[0099] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0100] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the present disclosure can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0101] It should be noted that the technical solutions described in the present disclosure can be combined arbitrarily without conflict.

[0102] The above is merely specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure.

Claims

1. A near-eye display device, characterized in that, The near-eye display device includes: The projection module generates the projected light rays for projecting the image; A beam splitter is used to transmit at least a portion of the projected light in calibration mode and to reflect at least a portion of the projected light in viewing mode for viewing. An image generation module is used to convert at least a portion of the transmitted projected light rays into a re-imaging image in calibration mode; The controller is configured to, in calibration mode, obtain image calibration parameters based on the reference image corresponding to the re-image and the projected image, calibrate the re-image based on the image calibration parameters, and update the projected image using the calibrated re-image until the difference between the re-image and the reference image is less than or equal to a difference threshold, and then switch to viewing mode. The beam splitter is also used to reflect at least a portion of the ambient light to the image generation module; The image generation module is also used to convert at least a portion of the ambient light into the reference image.

2. The near-eye display device according to claim 1, characterized in that, The beam splitter includes: a multilayer dielectric reflective film and a transparent optical substrate; The multilayer dielectric reflective film is used to reflect and transmit incoming projected light according to a preset beam splitting ratio; The light-transmitting optical substrate is used to support the multilayer dielectric reflective film.

3. The near-eye display device according to claim 2, characterized in that, The brightness of the projected light in the calibration mode is greater than the brightness of the projected light in the viewing mode.

4. The near-eye display device according to claim 2, characterized in that, The beam splitter further includes an electrochromic layer, wherein the electrochromic layer has different reflectance and transmittance under different voltages; The controller is further configured to apply a first voltage to the beam splitter in viewing mode, wherein the reflectivity of the beam splitter is greater than its transmittance under the first voltage; Furthermore, in calibration mode, after sending a projected image to the projection module, a second voltage is applied to the beam splitter, wherein the transmittance of the beam splitter under the second voltage is greater than its reflectance.

5. The near-eye display device according to claim 1, characterized in that, In calibration mode, the reference image is a monochrome image; The beam splitter is used to transmit light corresponding to the wavelength of the monochromatic image in calibration mode; The step of obtaining the image calibration parameters based on the re-image and the reference image includes: Image calibration parameters corresponding to the wavelength of the monochrome image are obtained based on the re-imaging image and the reference image.

6. The near-eye display device according to any one of claims 2 to 4, characterized in that, The light-transmitting optical substrate is a curved substrate to converge the projected light rays transmitted through the beam splitter.

7. The near-eye display device according to claim 1, characterized in that, The step of obtaining the image calibration parameters based on the re-image and the reference image includes: The same scale features are extracted from the re-image and the reference image to obtain the re-image features corresponding to the re-image and the reference image features corresponding to the reference image. Based on the feature differences between the re-image features and the reference image features, at least one of the geometric calibration parameters, color calibration parameters, brightness calibration parameters, and contrast calibration parameters is determined.

8. The near-eye display device according to claim 7, characterized in that, The controller is configured to determine the feature similarity between the re-image and the reference image based on the features of the re-image and the features of the reference image; If the feature similarity is less than or equal to a similarity threshold, the difference between the re-image and the reference image is determined to be less than or equal to a difference threshold.

9. The near-eye display device according to any one of claims 1 to 5, characterized in that, The near-eye display device further includes: an image optimization module; The image optimization module is used to receive at least a portion of the projected light reflected by the beam splitter, so as to project the at least a portion of the projected light onto the human eye.

10. The near-eye display device according to any one of claims 1 to 4, characterized in that, The near-eye display device further includes: a light acquisition module; The light acquisition module is used to collect and converge ambient light from a preset area of ​​the near-eye display device.

11. An image calibration method, characterized in that, The image calibration method, applied to any one of claims 1 to 10, comprises: In calibration mode, image calibration parameters are obtained based on the reference image corresponding to the re-imaging image and the projection image. The re-imaging image is obtained by converting at least a portion of the projected light transmitted by the beam splitter through the image generation module. The projected light is the light generated by the projection module based on the projection image. The re-imaging image is calibrated based on the image calibration parameters; The projected image is updated using the calibrated re-image until the difference between the re-image and the reference image is less than or equal to a difference threshold, at which point the viewing mode is switched.

12. The image calibration method according to claim 11, characterized in that, The step of obtaining the image calibration parameters based on the re-image and the reference image includes: The same scale features are extracted from the re-image and the reference image to obtain the re-image features corresponding to the re-image and the reference image features corresponding to the reference image. Based on the feature differences between the re-image features and the reference image features, at least one of the geometric calibration parameters, color calibration parameters, brightness calibration parameters, and contrast calibration parameters is determined.

13. The image calibration method according to claim 12, characterized in that, The image calibration method further includes: Based on the features of the re-image and the features of the reference image, the feature similarity between the re-image and the reference image is determined; If the feature similarity is less than or equal to a similarity threshold, the difference between the re-image and the reference image is determined to be less than or equal to a difference threshold.

Images