Near-to-eye display device and image calibration method

Abstract

The invention provides a near-eye display device and an image calibration method, and the near-eye display device comprises a projection module which generates projection light of a projection image; the spectroscope is used for transmitting at least part of the projection light rays in a calibration mode and reflecting at least part of the projection light rays for watching in a watching mode; the image generation module is used for converting at least part of transmitted projection light into a re-imaging image in a calibration mode; and the controller is used for acquiring an image calibration parameter based on the reimaging image and a reference image corresponding to the projection image in a calibration mode, calibrating the reimaging image based on the image calibration parameter, and updating the projection image by using the calibrated reimaging image. And when the difference between the re-imaging image and the reference image is smaller than or equal to a difference threshold value, switching to a viewing mode. The near-to-eye display device can be dynamically calibrated in real time on the spot.

Description

Technical Field

[0001] The present disclosure relates to the field of near-eye display technology, and in particular to a near-eye display device and an image calibration method. Background Art

[0002] A video see-through head-mounted projection module uses a camera to capture the external real-world scene in real time, fuses it with virtual content, and displays it on the head-mounted projection module. The image the user observes through the projection module is a composite image of the real-world scene captured by the camera and the superimposed virtual content, thus achieving the visual effects of augmented reality (AR), virtual reality (VR), or mixed reality (MR).

[0003] As the core hardware device for near-eye displays, the display accuracy and stability of video see-through head-mounted projection modules directly impact the user experience. To eliminate optical system aberrations, assembly errors, and long-term drift, related technologies perform offline calibration of near-eye display devices using precision optical equipment (such as standard calibration plates) in a laboratory environment before shipment, and then embed the calibration parameters into the device. However, this calibration method cannot dynamically adjust the calibration parameters and cannot address environmental changes (such as temperature and humidity) or drift caused by long-term device use during actual user experience, resulting in poor calibration results.

[0004] Therefore, there is an urgent need for a near-eye display device that can achieve on-site dynamic real-time calibration. Summary of the Invention

[0005] The present disclosure provides a near-eye display device and an image calibration method, which can realize on-site dynamic real-time calibration of the near-eye display device.

[0006] The technical solution of the present disclosure is achieved as follows: In a first aspect, the present disclosure provides a near-eye display device, which includes: a projection module for generating projection light of a projected image; a spectrometer for transmitting at least part of the projection light in a calibration mode, and reflecting at least part of the projection light in a viewing mode for viewing; an image generation module for converting at least part of the transmitted projection light into a re-imaged image in the calibration mode; a controller for obtaining image calibration parameters based on a reference image corresponding to the re-imaged image and the projected image in the calibration mode, calibrating the re-imaged image based on the image calibration parameters, and updating the projected image using the calibrated re-imaged image until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, thereby switching to the viewing mode.

[0007] In a second aspect, the present disclosure provides an image calibration method, which includes: in a calibration mode, obtaining image calibration parameters based on a re-imaged image and a reference image corresponding to a projected image, wherein the re-imaged image is obtained by converting at least a portion of the projection light transmitted by a spectroscope through an image generation module, and the projection light is the light generated by the projection module based on the projection image; calibrating the re-imaged image based on the image calibration parameters; and updating the projected image using the calibrated re-imaged image until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and then switching to a viewing mode.

[0008] In a third aspect, the present disclosure provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the steps of the image calibration method described in the second aspect are implemented.

[0009] In a fourth aspect, the present disclosure provides a computer program product, wherein the computer program product includes a computer program or instructions, and when the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the image calibration method as described in the second aspect.

[0010] In a fifth aspect, the present disclosure provides a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the image calibration method as described in the second aspect.

[0011] The present disclosure provides a near-eye display device and image calibration method. If the difference between a re-imaged image and a reference image is less than or equal to a difference threshold, the near-eye display device has a small error and does not require image calibration. If the difference between the re-imaged image and the reference image is greater than the difference threshold, the image output by the projection module under the current working environment needs to be calibrated using image calibration parameters. Therefore, a calibration mode is entered, image calibration parameters are determined based on the difference between the re-imaged image and the reference image, and the projected image is updated using the calibrated re-imaged image. Specifically, a controller sends the calibrated re-imaged image to the projection module, which continues to project the calibrated re-imaged image and regenerates the corresponding re-imaged image until the difference between the final re-imaged image and the reference image is less than or equal to the difference threshold. This indicates that calibration is complete, and the near-eye display device can enter a viewing mode. In subsequent viewing modes, the determined image calibration parameters can be used to calibrate the image to be displayed. The present disclosure implements a lightweight (only the spectrometer is integrated) on-site real-time calibration function for a near-eye display device through dynamic spectrometer splitting, real-time imaging by an image generation module, and iterative processing by a controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A schematic structural diagram of a near-eye display device provided by the present disclosure.

[0013] Figure 2 A schematic structural diagram of a near-eye display device provided by the present disclosure, in which a reference image is an image corresponding to external ambient light.

[0014] Figure 3 A schematic structural diagram of a spectrometer provided in the present disclosure.

[0015] Figure 4 A schematic structural diagram of another spectrometer provided in the present disclosure.

[0016] Figure 5 Schematic diagram of the optical path of the near-eye display device in the viewing mode provided by the present disclosure.

[0017] Figure 6 Schematic diagram of the optical path of the near-eye display device in the calibration mode provided by the present invention.

[0018] Figure 7 A schematic structural diagram of another near-eye display device provided by the present disclosure.

[0019] Figure 8 A flowchart of an image calibration method provided by the present disclosure is provided.

[0020] Figure 9 A schematic flow chart of another image calibration method provided by the present disclosure.

[0021] Figure 10 A flowchart of another image calibration method provided by the present disclosure is provided. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in this disclosure to clearly describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this disclosure.

[0023] The terms "first," "second," and the like in the specification of the present disclosure are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the present disclosure can be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class of objects, and do not limit the number of objects. For example, the first object may be one or more.

[0024] The video see-through head-mounted display uses a camera (objective lens) to capture the external environment in real time, combines sensors (such as gyroscopes and accelerometers) to obtain the user's head posture, and accurately superimposes virtual content (such as 3D models and information tags) onto the real scene. The picture the user sees is a digitally processed "reality + virtual" hybrid image.

[0025] Because users rely entirely on real-time video footage captured by cameras, virtual content is overlaid onto the video using digital image processing technology. Video see-through technology can dynamically adjust the real scene (such as brightness, contrast, and distortion correction) and more easily handles the relationship between virtual and real occlusion. Therefore, video see-through technology is widely used in near-eye display devices.

[0026] To ensure high-precision image display and user experience, existing video see-through head-mounted displays typically require precise calibration to eliminate optical system aberrations, component assembly errors, and drift that may occur during long-term use. Optical system aberrations are the deviations between actual and ideal imaging caused by design or manufacturing defects when light passes through optical components. Aberrations directly affect the geometric accuracy, color consistency, and image quality of virtual-reality fusion, causing users to perceive virtual objects as blurry, with edge artifacts and color distortion, thus destroying the sense of immersion. Assembly errors are physical position deviations caused by process limitations during the manufacturing and assembly of optical components and sensors, causing users to perceive virtual objects as offset, jitter, or disproportionately positioned (e.g., AR navigation arrows deviating from the actual road). Long-term drift is the slow shift of system parameters during operation due to environmental changes, material aging, or accumulated mechanical stress.

[0027] To address these errors, traditional video fluoroscopy undergoes offline calibration before shipment. This requires calibrating system parameters in the laboratory using high-precision equipment (such as laser interferometers and standard calibration plates). However, this method cannot correct for drift during use or environmental changes (such as temperature fluctuations), and cannot be dynamically adjusted, requiring regular factory maintenance. Calibration can also be assisted by external equipment, such as relying on external cameras, projectors, or calibration targets (such as checkerboard patterns) for in-situ calibration. However, this method not only increases hardware costs and is highly complex, but also makes integration into near-eye display devices difficult due to their lightweight requirements.

[0028] Based on the above problems, the present disclosure aims to provide a lightweight near-eye display device that can achieve on-site real-time calibration, such as Figure 1As shown, the near-eye display device includes: a projection module 10 for generating projection light for a projected image; a beam splitter 20 for transmitting at least part of the projection light in a calibration mode, and reflecting at least part of the projection light in a viewing mode for viewing; an image generation module 30 for converting at least part of the transmitted projection light into a re-imaged image in a calibration mode; a controller 40 for obtaining image calibration parameters based on a reference image corresponding to the re-imaged image and the projected image in the calibration mode, calibrating the re-imaged image based on the image calibration parameters, and updating the projected image using the calibrated re-imaged image until the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and then switching to the viewing mode.

[0029] The projection module 10 is typically a high-resolution micro display screen, such as an organic light-emitting diode (OLED), a micro light-emitting diode (Micro-LED), or a liquid crystal on silicon (LCoS).

[0030] The light generated corresponding to the projected image displayed by the projection module 10 is the projected light. The reference image corresponding to the projected image is the standard image that the projected image is ultimately expected to display. Under the ideal condition that the near-eye display device is error-free, the projected image is the same as the reference image. The reference image can be, for example, a checkerboard, a dot pattern, a specific geometric figure, a specific color image, or an image of the external environment captured by the near-eye display device. If the reference image is an image of the external environment captured by the near-eye display device, the image of the external environment can be either pre-stored or acquired in real time. It should be noted that if the image of the external environment captured by the near-eye display device is used as a reference image, it is necessary to shoot a scene with rich feature information, such as buildings and fences with many straight lines or edges, so that the reference image contains rich features, which is convenient for subsequent comparison with the re-imaged image.

[0031] Beam splitter 20 uses coating technology (such as dielectric or metal coatings) to split light beams, controlling the ratio of transmission and reflection of light as it passes through. Beam splitter 20 can be a polarizing beam splitter, a dichroic beam splitter, or any other device capable of proportionally transmitting and reflecting incoming light. This disclosure does not limit the material or implementation principle of beam splitter 20.

[0032] Combine Figure 1 In the calibration mode, the projection light emitted by the projection module 10 (indicated by three solid arrows pointing from the projection module 10 in the figure) is completely transmitted by the beam splitter 20 to the image generation module 30. The transmitted projection light is indicated by the dotted arrow in the figure. Figure 1In the viewing mode, the projection light emitted by the projection module 10 is partially reflected by the beam splitter 20 to the image generation module 30, and the reflected projection light is indicated by the dotted arrow. Figure 1 There is no part of the projected light indicated by the dotted arrow; another situation is that part of the projected light is transmitted to the image generation module 30 by the beam splitter 20 and part is reflected (indicated by the dot-dash arrow in the figure).

[0033] In the calibration mode, if the beam splitter 20 reflects part of the projected light, this part of the projected light can be used to preview the calibration effect in real time; in the viewing mode, if the beam splitter 20 transmits part of the projected light, this part of the projected light can be used to monitor in real time whether the near-eye display device needs to be calibrated.

[0034] Image generation module 30 converts optical signals into digital images, representing an image sensor (e.g., a Complementary Metal Oxide Semiconductor (CMOS) or a Charge Coupled Device (CCD)). In calibration mode, image generation module 30 captures the projected light transmitted by beam splitter 20 and generates a re-imaged image, which is the actual image of the projected image displayed by projection module 10 after passing through the optical link.

[0035] The controller 40 has the ability to communicate with a remote server and can perform different operations according to the storage location of the reference image: when the reference image is stored on the remote server, the controller 40 sends the calibration image to the remote server; when the reference image is stored on the near-eye display device, the controller 40 sends the reference image and the re-imaged image together to the remote server; the remote server analyzes the difference between the re-imaged image and the reference image, determines whether the difference is greater than the difference threshold, and generates image calibration parameters if the difference is greater than the difference threshold and sends them back to the controller 40. The controller 40 controls the near-eye display device to calibrate the output image based on the received image calibration parameters.

[0036] In some implementations, image calibration parameters are obtained based on the re-imaged image and the reference image, including performing feature extraction on the re-imaged image and the reference image at the same scale to obtain re-imaged image features corresponding to the re-imaged image and reference image features corresponding to the reference image; and determining at least one of geometric calibration parameters, color calibration parameters, brightness calibration parameters, and contrast calibration parameters based on feature differences between the re-imaged image features and the reference image features.

[0037] Specifically, features of the same scale are extracted from the re-image (the image captured after actual projection) and the reference image (the theoretical target of the projected image). These features include: geometric features (such as edges, corners, and shape outlines) to detect optical system distortion or offset; color features (such as RGB histograms and color temperature) to analyze color distribution (such as RGB histograms and color temperature) to calibrate color differences; and brightness and contrast features (such as pixel brightness mean and variance) to adjust the dynamic range of the display device. Feature extraction can be achieved using traditional algorithms (such as SIFT and HOG) or deep learning models (such as convolutional neural networks). The re-image features are compared with the reference image features to obtain geometric calibration parameters for correcting optical distortion (such as barrel distortion and pincushion distortion) or projection angle deviation; color calibration parameters for adjusting color gamut mapping or compensating for the effects of ambient light on color; and brightness and contrast calibration parameters for optimizing display brightness and contrast to adapt to varying ambient lighting conditions. Through comprehensive calibration of parameters such as geometry, color, brightness, and contrast, the projected image is ensured to be consistent with the reference image, thereby improving display accuracy. The calculation of image calibration parameters is based on quantitative analysis of feature differences, avoiding pixel-by-pixel comparison of the entire image and reducing computing resource consumption.

[0038] Specifically, the reimaged image and the reference image are preprocessed separately. Gaussian filtering or non-local mean denoising is used to reduce image noise. Histogram equalization or brightness scaling is used to unify the dynamic range of the two images (normalization). Feature points of the two images are extracted using a feature extraction algorithm or deep learning. The two images are registered using RANSAC or a homography matrix to establish a pixel-level correspondence. The geometric deviation, color deviation, brightness deviation, and contrast deviation of the two images are analyzed to generate a pre-distortion matrix, a color correction matrix, and brightness and contrast compensation parameters, which are integrated into a display matrix, i.e., image calibration parameters. The obtained image calibration parameters are used to perform real-time correction on the image output by the projection module 10. The reimaged image is continuously acquired and the difference is analyzed until the difference between the reimaged image and the reference image is less than a difference threshold. Finally, the final image calibration parameters are obtained.

[0039] In some embodiments, the difference between the reference image and the re-imaged image is determined specifically as follows: based on the re-imaged image features and the reference image features, the feature similarity between the re-imaged image and the reference image is determined; when the feature similarity is less than or equal to the similarity threshold, the difference between the re-imaged image and the reference image is determined to be less than or equal to the difference threshold.

[0040] The controller 40 calculates the feature similarity (for example, using the SSIM structural similarity index, cosine similarity, or a custom weighted score) by comparing the degree of match between the features of the re-imaged image and the features of the reference image. The specific feature similarity determination strategy is not limited. For example, the feature similarity score can be obtained by weighted summing of geometric feature differences, color feature differences, brightness feature differences, and contrast feature differences. When the feature similarity is less than or equal to a preset threshold (such as SSIM < 0.95), it indicates that the difference between the re-imaged image and the reference image is too large, and further iterative calibration is required. When the feature similarity exceeds the threshold, the calibration is determined to be complete and the mode is automatically switched to viewing mode. By setting a clear calibration termination condition through the similarity threshold, subjective judgment errors are avoided, calibration reliability is improved, and based on real-time feedback of feature similarity, a closed-loop process of "calibration-update-recalibration" is achieved without the need for manual intervention.

[0041] It should be noted that, combined with Figure 1 The purpose of calibration is to ensure that the image corresponding to the portion of the projected light reflected by the projection module 10 (the image seen by the human eye) is consistent with the desired reference image (the difference is less than a difference threshold). Since the light entering the human eye cannot be directly measured, the image corresponding to the portion of the projected light transmitted through the beam splitter 20 is the same as the image corresponding to the portion of the projected light reflected. Therefore, the image corresponding to the portion of the projected light transmitted by the projection module 10 (i.e., the re-imaged image) is collected by the image generation module 30 as the image seen by the human eye. By comparing the difference between the re-imaged image and the reference image, the difference between the image seen by the human eye and the desired reference image can be determined.

[0042] In an embodiment of the present disclosure, if the difference between the re-imaged image and the reference image is less than or equal to a difference threshold, the near-eye display device has a small error and does not require image calibration. If the difference between the re-imaged image and the reference image is greater than the difference threshold, the image output by the projection module in the current working environment needs to be calibrated using image calibration parameters. Therefore, a calibration mode is entered, image calibration parameters are determined based on the difference between the re-imaged image and the reference image, and the projected image is updated using the calibrated re-imaged image. Specifically, the controller sends the calibrated re-imaged image to the projection module, which continues to project the calibrated re-imaged image to regenerate a corresponding re-imaged image until the difference between the final re-imaged image and the reference image is less than or equal to the difference threshold. This indicates that calibration is complete, and the near-eye display device can enter a viewing mode. In subsequent viewing modes, the determined image calibration parameters can be used to calibrate the image to be displayed. The present disclosure achieves lightweight (only the spectrometer is integrated) on-site real-time calibration of a near-eye display device through dynamic spectrometer splitting, real-time imaging by the image generation module, and iterative processing by the controller.

[0043] In some embodiments, the reference image is an image of the external environment captured by the near-eye display device, specifically, Figure 2 As shown, the beam splitter 20 is further used to reflect at least part of the external ambient light to the image generation module 30; the image generation module 30 is further used to convert at least part of the external ambient light into a reference image. Figure 2 The thick black arrow in the middle indicates the external ambient light.

[0044] The beam splitter 20 can reflect a portion of the ambient light toward the image generation module 30, or it can reflect all of the ambient light toward the image generation module 30, depending on the preset splitting ratio. Different materials of the beam splitter 20 correspond to different preset splitting ratios. For example, a preset splitting ratio of 9:1 means that 90% of the projected light is reflected and 10% is transmitted by the beam splitter 20. A preset splitting ratio of 1:1 means that 50% of the projected light is reflected and 50% is transmitted by the beam splitter 20.

[0045] The reference image is obtained by converting the corresponding ambient light. The image generation module 30 collects light in a time-sharing multiplexed manner. That is, after entering the calibration mode, the image generation module 30 first converts the ambient light into a reference image and stores it in the controller 40. The real-time collected ambient image is directly used as a benchmark. The collection environment of the reference image is exactly the same as the current operating environment of the near-eye display device, avoiding the introduction of new errors due to environmental changes (such as light fluctuations and temperature fluctuations). In addition, the reference image is generated based on actual ambient light rather than idealized preset values, making the calibration parameters more suitable for real-world usage scenarios. The real-time changes in the ambient light will be continuously fed back to the reference image, and the controller 40 iteratively calibrates accordingly to avoid display distortion caused by sudden changes in the environment.

[0046] In some embodiments, as Figure 3 As shown, the spectroscope 20 includes: multiple layers ( Figure 3 , two layers are shown) a dielectric reflective film 201 and a light-transmitting optical substrate 202; the dielectric reflective film 201 is used to reflect and transmit the incoming projection light according to a preset splitting ratio; the light-transmitting optical substrate 202 is used to support the multilayer dielectric reflective film 201.

[0047] The dielectric reflective film 201 is formed by alternating deposition of high refractive index and low refractive index optical films (such as ) is a composite structure composed of a plurality of layers of dielectric reflective films 201 and a plurality of layers of dielectric reflective films 202. The dielectric reflective films 201 of the beam splitter 20 selectively reflect and transmit light of a specific wavelength through interference effects, and control the splitting ratio (e.g., the ratio of reflection to transmission is 9:1). The splitting ratios of the two layers of dielectric reflective films 201 of the beam splitter 20 can be the same or different, that is, the splitting ratios of the front surface and the back surface of the beam splitter 20 can be the same or different. The transparent optical substrate 202 is the transparent supporting base of the beam splitter 20 and is usually made of a high-transmittance, low-birefringence material (e.g., BK7 glass, fused quartz, polycarbonate). In some achievable embodiments, the transparent optical substrate 202 is Figure 3 The curved substrate shown in FIG. 1 can converge the light transmitted by the beam splitter 20 ( Figure 2 The projection light (light indicated by the dotted arrow in the figure) is emitted from the projection module 10.

[0048] Due to the presence of the multilayer dielectric reflective film 201, the beam splitter 20 reflects and transmits incoming light in a specific ratio. That is, in viewing mode, when the projection light emitted by the projection module 10 passes through the beam splitter 20, a portion (e.g., 95%) of the projection light is reflected and enters the human eye, while a portion (e.g., 5%) of the projection light is transmitted and enters the image generation module 30.

[0049] In the calibration mode, the external ambient light is reflected by the beam splitter 20 (e.g., 95%) and enters the image generation module 30, and part of it is transmitted through the beam splitter 20 and enters the human eye; the image generation module 30 converts the reflected external ambient light into a reference image, and the controller 40 sends the reference image to the projection module 10. The brightness of the projection light projected by the projection module 10 in the calibration mode (e.g., the display brightness of the projection module 10 is adjusted to the maximum) is greater than the brightness of the projection light in the viewing mode, so that the image generation module 30 collects more projection light transmitted by the beam splitter 20.

[0050] In the disclosed embodiment, the spectrometer 20 can reflect and transmit incoming light according to a preset spectroscopic ratio, thereby enabling the user to use the near-eye display device normally in viewing mode; at the same time, in calibration mode, on-site real-time calibration of the near-eye display device is achieved.

[0051] In some embodiments, as Figure 4 As shown, the spectroscope 20 further includes an electrochromic layer 203 . The reflectance and transmittance ratios of the electrochromic layer 203 are different under different voltages.

[0052] The controller 40 is further configured to apply a first voltage to the beam splitter 20 in a viewing mode, wherein the reflectivity of the beam splitter 20 at the first voltage is greater than the transmittance; and, in a calibration mode, apply a second voltage to the beam splitter 20 after sending a projected image to the projection module 10, wherein the transmittance of the beam splitter 20 at the second voltage is greater than the reflectivity.

[0053] The electrochromic layer 203 is a thin film structure made of electrochromic materials that undergoes a reversible electrochemical reaction when an external voltage is applied, thereby changing the reflectivity and transmittance. The transmittance and reflectivity can be precisely controlled by adjusting the voltage and polarity.

[0054] In some implementations, the transmittance of the spectroscope 20 is 0 at the first voltage, and the reflectance is 0 at the second voltage. In the viewing mode, the spectroscope 20 is required to reflect all incoming light, such as Figure 5 The figure shows the propagation path of light in the viewing mode. The external ambient light is completely reflected by the beam splitter 20 to the image generation module 30, and the projection light generated by the projection module 10 is also completely reflected by the beam splitter 20 and enters the human eye.

[0055] like Figure 6 Figure 2 shows the light propagation path in calibration mode. Under a first voltage, the beam splitter 20 reflects all ambient light to the image generation module 30. The image generation module 30 generates a reference image and transmits it to the projection module 10 via the controller 40. The controller 40 applies a second voltage to the beam splitter 20 and stops collecting ambient light. Under a second voltage, the beam splitter 20 transmits all projection light generated by the projection module 10 when displaying the reference image to the image generation module 30. The image generation module 30 generates a re-imaged image based on all the projection light.

[0056] It should be noted that in viewing mode, although the brightness of the projection module 10 is not high and the transmittance of the beam splitter 20 is very low, even without the electrochromic layer 203, the partially transmitted projection light from the projection module 10 has essentially no effect on the ambient light of the image generation module 30. However, to achieve more accurate imaging, in viewing mode, the electrochromic layer 203 included in the beam splitter 20 operates at a first voltage with a transmittance of zero, and the projection light from the projection module 10 is completely reflected, thereby preventing the partially transmitted projection light from affecting the image generated by the image generation module 30.

[0057] In the embodiment of the present disclosure, an electrochromic layer 203 is added to the spectroscope 20 so that the light passing through the spectroscope 20 in the viewing mode and the calibration mode will not affect each other, thereby preventing the reference image and the re-imaged image from being interfered with by irrelevant light, making the final calibration result more accurate.

[0058] In some embodiments, in the calibration mode, the reference image is a monochrome image; the spectrometer 20 is used to transmit light corresponding to the wavelength of the monochrome image in the calibration mode; in the above-mentioned controller 40, image calibration parameters are obtained based on the re-imaged image and the reference image, including: image calibration parameters corresponding to the wavelength of the monochrome image are obtained based on the re-imaged image and the reference image.

[0059] Different wavelengths of light exhibit significant differences in their sensitivity to geometric deviations (such as distortion and displacement), color deviations (color cast and dispersion), brightness deviations (response nonlinearity), and contrast deviations (signal-to-noise ratio) in optical systems. Wavelength-specific calibration can decompose complex errors into independent parameters, enabling precise segmented optimization. For example, short wavelengths (blue light, 450nm) exhibit significant diffraction effects and are susceptible to the curvature of optical components, making them suitable for adjusting geometric distortion. Medium wavelengths (green light, 550nm) are sensitive to green light in the image generation module 30, which easily exposes brightness gradients and is therefore suitable for adjusting brightness uniformity errors. Long wavelengths (red light, 650nm) have strong penetrability and are susceptible to coating dispersion, making them suitable for adjusting color deviations. Wide-spectrum (full-band) multi-wavelength superposition allows for the evaluation of the nonlinear response of the image generation module 30 and is therefore suitable for adjusting contrast errors.

[0060] Specifically, the controller 40 generates a set of monochromatic images, such as a blue checkerboard pattern (to calibrate geometric distortion), a green uniform grayscale image (to calibrate brightness uniformity), and a red pure color fill image (to calibrate color channel gain). Each image corresponds to a specific wavelength. The projection module 10 sequentially displays the monochromatic images of each wavelength. The beam splitter 20 transmits the projected light to the image generation module 30, which captures the transmitted projected light and generates a re-imaged image of the corresponding wavelength. The re-imaged image of a particular wavelength is compared with a corresponding reference image (a monochromatic image generated by the controller 40, such as a blue checkerboard pattern, displayed by the projection module 10, then passed through the beam splitter 20, and then generated by the image generation module 30; the corresponding reference image is the blue checkerboard pattern). The partial image calibration parameters corresponding to that wavelength are obtained. After comparing and analyzing each reference image with the corresponding re-imaged image, the final complete image calibration parameters are obtained.

[0061] In the disclosed embodiment, a closed-loop design of wavelength-divided projection-sensing-analysis is used to decompose complex full-link errors into independent wavelength-sensitive parameters, thereby achieving precise segmented calibration of deviations such as geometry, color, and brightness.

[0062] In some embodiments, as Figure 7 As shown, the near-eye display device also includes: a light collection module 50, and / or an image optimization module 60; the light collection module 50 is used to collect external ambient light from a preset area of ​​the near-eye display device and converge it; the image optimization module 60 is used to receive at least part of the projected light reflected by the spectrometer 20 to project at least part of the projected light to the human eye.

[0063] The light collection module 50 can be an objective lens. Depending on the focal length, the light collection module 50 can collect ambient light from a predetermined area at different distances from the near-eye display device. The shorter the focal length, the higher the magnification of the light collection module 50. Simultaneously, its viewing angle is wider, allowing it to collect light from a wider range of areas. However, the working distance (the maximum distance from the near-eye display device that can be collected) is also shorter. Specifically, a short focal length typically corresponds to a high magnification, a small field of view, and lower image brightness, while a long focal length typically corresponds to a low magnification, a large field of view, and higher image brightness.

[0064] To enable the near-eye display device to produce more accurate images for external scenes at varying distances, the focal length of the light collection module 50 is modified in calibration mode to capture light from a preset area at varying distances from the near-eye display device, thereby obtaining reference images at varying focal lengths. Based on the differences in the reference images at varying focal lengths, image calibration parameters corresponding to each focal length are ultimately obtained. Thus, in viewing mode, if the focal length of the light collection module 50 is adjusted, the image corresponding to the captured ambient light can be calibrated using the image calibration parameters corresponding to that focal length, resulting in a more accurate calibration result.

[0065] Since the change in focal length affects the brightness value of the image, in some feasible methods, the correspondence between focal length and brightness can be pre-stored. In the viewing mode, the controller 40 can also adjust the brightness of the calibrated projected image according to the brightness corresponding to the focal length after calibrating the projected image using the image calibration parameters.

[0066] It should be noted that, in the calibration mode, in order to prevent the external ambient light from being reflected by the spectrometer 20 and entering the image generation module 30, thereby affecting the generation of the re-imaged image, in the calibration mode, after obtaining the reference image, the external ambient light needs to be blocked. For example, if the near-eye device includes a light collection module 50, after obtaining the reference image, the light collection module 50 will be blocked (such as closing the objective lens cover).

[0067] Image optimization module 60 may be an eyepiece configured to receive at least a portion of the projected light reflected by beam splitter 20 and direct the at least a portion of the projected light to the human eye. Image optimization module 60 further magnifies and adjusts the focus of the image corresponding to the portion of the projected light reflected by beam splitter 20, enabling the observer to see clearer details.

[0068] The near-eye display device provided by the present disclosure has the following technical effects: The beam splitter switches the transmission / reflection ratio between two modes, allowing the same optical path to project the display image (viewing mode) and transmit part of the light to the image sensor to generate a re-imaged image (calibration mode). This eliminates the need for additional sensors or mechanical structures, achieving lightweight hardware.

[0069] Errors such as geometric distortion, chromatic aberration, and uneven brightness are wavelength-sensitive, and full-band calibration can easily lead to parameter coupling errors. Therefore, monochromatic image wavelength calibration is used to optimize different deviations in a targeted manner.

[0070] The multiplexed spectroscope 20 collects real-time ambient light as a dynamic reference image. In calibration mode, the spectroscope 20 reflects ambient light to generate a reference image and transmits projected light to generate a re-imaged image, ensuring that the reference image is consistent with the current environment. In addition, the electrochromic layer 203 controls the transmittance / reflection ratio of the spectroscope 20 through voltage (e.g., transmittance > 90% at the second voltage), isolating the mutual interference between ambient light and projected light, thereby improving calibration robustness.

[0071] Comparing the entire image pixel by pixel requires a large amount of computation and is difficult to meet real-time requirements. Therefore, quantitative calibration based on feature differences is combined with closed-loop iterative optimization. Through features such as edges, color temperature, and brightness distribution (traditional algorithms or CNN), the key differences between the re-imaged image and the reference image are extracted to reduce the amount of data. The pre-distortion matrix, color correction matrix, etc. are calculated based on the feature differences, and the projection output is adjusted in a targeted manner. The controller 40 continuously compares the updated re-imaged image with the reference image until the feature similarity (such as SSIM) meets the standard, thereby automatically terminating the calibration.

[0072] Near-eye display devices need to adapt to different viewing distances (such as the virtual-reality fusion scenarios of AR glasses). The light acquisition module dynamically adjusts the focal length, generates a multi-distance reference image library, and stores corresponding geometric / brightness parameters for different focal lengths (such as high distortion compensation for short focal lengths). In viewing mode, pre-stored parameters are called according to the current focal length, and dynamic fine-tuning is combined with ambient light to ensure display consistency in multiple scenes.

[0073] like Figure 8 As shown, the present disclosure provides an image calibration method, and the image calibration method provided by the present disclosure is exemplarily described below by taking the controller 40 as an example. The method may include the following steps 801 to 805.

[0074] In step 801 , in a calibration mode, image calibration parameters are acquired based on a re-imaged image and a reference image corresponding to the projected image.

[0075] The re-imaged image is obtained by converting at least a portion of the projection light transmitted by the beam splitter through the image generation module, and the projection light is the light generated by the projection module based on the projection image.

[0076] In step 802, the re-imaged image is calibrated based on image calibration parameters.

[0077] In step 803, the projection image is updated using the calibrated re-imaged image.

[0078] In step 804 , it is determined whether the difference between the re-imaged image and the reference image is less than or equal to a difference threshold.

[0079] If yes, execute step 805; otherwise, execute steps 801 to 804.

[0080] In step 805, switch to viewing mode.

[0081] In some embodiments, combined Figure 8 ,like Figure 9 As shown, in the above step 801 , in the calibration mode, image calibration parameters are obtained based on the re-imaged image and the reference image, including the following steps 801 a and 801 b.

[0082] In step 801a, in a calibration mode, feature extraction is performed on the re-imaged image and the reference image at the same scale to obtain re-imaged image features corresponding to the re-imaged image and reference image features corresponding to the reference image.

[0083] In step 801b, at least one of a geometric calibration parameter, a color calibration parameter, a brightness calibration parameter, and a contrast calibration parameter is determined based on the feature difference between the re-imaged image feature and the reference image feature.

[0084] In some embodiments, combined Figure 9 ,like Figure 10 As shown, the above step 804 can be specifically implemented through the following steps 804a and 804b.

[0085] In step 804a, feature similarity between the re-imaged image and the reference image is determined based on the re-imaged image features and the reference image features.

[0086] In step 804b, it is determined whether the feature similarity is less than or equal to a similarity threshold.

[0087] If yes, execute step 805; otherwise, execute steps 801 to 804.

[0088] In step 805, switch to viewing mode.

[0089] In the embodiment of the present disclosure, the effect of the image calibration method provided by the present disclosure can refer to the technical effect of the controller 40 in the above-mentioned near-eye display device, and will not be repeated here to avoid repetition.

[0090] The present disclosure further provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is configured to be executed by a processor to implement the image calibration method described in the above embodiments.

[0091] The present disclosure also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes to implement the image calibration method described in each of the above embodiments.

[0092] An embodiment of the present disclosure further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image calibration method embodiment, and can achieve the same technical effects. To avoid repetition, they are not described here.

[0093] It should be understood that the chip mentioned in the embodiments of the present disclosure can also be called a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0094] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, servers and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0098] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in this disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0099] It should be noted that the technical solutions described in this disclosure can be combined arbitrarily without conflict.

[0100] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present disclosure, and they should all be covered by the protection scope of the present disclosure.

Claims

1. A near-eye display device, characterized in that: The near-eye display device comprises: A projection module, generating projection light for projecting an image; a beam splitter for transmitting at least a portion of the projection light in a calibration mode and reflecting at least a portion of the projection light in a viewing mode for viewing; an image generation module for converting at least a portion of the transmitted projection light into a re-imaged image in a calibration mode; A controller is configured to, in a calibration mode, obtain image calibration parameters based on the re-imaged image and a reference image corresponding to the projected image, calibrate the re-imaged image based on the image calibration parameters, and update the projected image using the calibrated re-imaged image until a difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and then switch to a viewing mode.

2. The near-eye display device according to claim 1, wherein: The beam splitter is further configured to reflect at least a portion of the ambient light to the image generation module; The image generation module is further configured to convert at least a portion of the external ambient light into the reference image.

3. The near-eye display device according to claim 2, wherein: The spectroscope comprises: a multi-layer dielectric reflective film and a light-transmitting optical substrate; The multilayer dielectric reflective film is used to reflect and transmit the incoming projection light according to a preset splitting ratio; The light-transmitting optical substrate is used to support the multi-layer dielectric reflective film.

4. The near-eye display device according to claim 3, wherein: The brightness of the projection light projected by the projection module in the calibration mode is greater than the brightness of the projection light projected in the viewing mode.

5. The near-eye display device according to claim 3, wherein: The spectroscope further includes: an electrochromic layer, wherein the reflectance and transmittance of the electrochromic layer are different at different voltages; The controller is further configured to apply a first voltage to the beam splitter in a viewing mode, wherein the reflectivity of the beam splitter under the first voltage is greater than the transmittance; Furthermore, in a calibration mode, after the projection image is sent to the projection module, a second voltage is applied to the beam splitter, and the transmittance of the beam splitter under the second voltage is greater than the reflectance.

6. The near-eye display device according to claim 1, wherein: In the calibration mode, the reference image is a monochrome image; The beam splitter is used to transmit light corresponding to the wavelength of the monochromatic image in a calibration mode; The acquiring the image calibration parameters based on the re-imaged image and the reference image includes: Image calibration parameters corresponding to the wavelength of the monochromatic image are acquired based on the re-imaged image and the reference image.

7. The near-eye display device according to any one of claims 3 to 5, characterized in that: The light-transmitting optical substrate is a curved substrate for converging the projection light transmitted through the beam splitter.

8. The near-eye display device according to claim 1, wherein: The acquiring the image calibration parameters based on the re-imaged image and the reference image includes: performing feature extraction of the same scale on the re-imaged image and the reference image to obtain re-imaged image features corresponding to the re-imaged image and reference image features corresponding to the reference image; At least one of a geometric calibration parameter, a color calibration parameter, a brightness calibration parameter, and a contrast calibration parameter is determined based on a characteristic difference between the re-imaged image characteristic and the reference image characteristic.

9. The near-eye display device according to claim 8, wherein: The controller is configured to determine a feature similarity between the re-imaged image and the reference image based on the re-imaged image feature and the reference image feature; In a case where the feature similarity is less than or equal to a similarity threshold, it is determined that the difference between the re-imaged image and the reference image is less than or equal to a difference threshold.

10. The near-eye display device according to any one of claims 1 to 6, characterized in that: The near-eye display device further includes: an image optimization module; The image optimization module is used to receive at least part of the projection light reflected by the beam splitter, so as to project the at least part of the projection light to the human eye.

11. The near-eye display device according to any one of claims 2 to 5, characterized in that: The near-eye display device further includes: a light collection module; The light collection module is used to collect and converge the external ambient light within a preset area of ​​the near-eye display device.

12. An image calibration method, characterized in that: Applied to the near-eye display device according to any one of claims 1 to 11, the image calibration method comprises: In a calibration mode, image calibration parameters are obtained based on the re-imaged image and a reference image corresponding to the projected image, wherein the re-imaged image is obtained by converting at least a portion of the projection light transmitted by the beam splitter by an image generation module, and the projection light is light generated by the projection module based on the projection image; calibrating the re-imaged image based on the image calibration parameters; The projected image is updated using the calibrated re-imaged image until a difference between the re-imaged image and the reference image is less than or equal to a difference threshold, and then the viewing mode is switched.

13. The image calibration method according to claim 12, wherein: The acquiring the image calibration parameters based on the re-imaged image and the reference image includes: performing feature extraction of the same scale on the re-imaged image and the reference image to obtain re-imaged image features corresponding to the re-imaged image and reference image features corresponding to the reference image; At least one of a geometric calibration parameter, a color calibration parameter, a brightness calibration parameter, and a contrast calibration parameter is determined based on a characteristic difference between the re-imaged image characteristic and the reference image characteristic.

14. The image calibration method according to claim 13, wherein: The image calibration method further includes: determining a feature similarity between the re-imaged image and the reference image based on the re-imaged image features and the reference image features; In a case where the feature similarity is less than or equal to a similarity threshold, it is determined that the difference between the re-imaged image and the reference image is less than or equal to a difference threshold.

Images