Display method, device, equipment and storage medium of near-eye display device

Abstract

The application relates to the technical field of near-eye display, and provides a display method and device of a near-eye display equipment, equipment and a storage medium, which comprises the following steps: acquiring first energy distribution information of a to-be-displayed image of the near-eye display equipment; based on a preset optical model of the near-eye display equipment, second energy distribution information of a predicted artifact corresponding to the to-be-displayed image is determined according to the first energy distribution information; third energy distribution information of a target image mask corresponding to the to-be-displayed image is determined according to the first energy distribution information and the second energy distribution information; the near-eye display equipment is controlled to display the to-be-displayed image according to the first energy distribution information, and the near-eye display equipment is controlled to display the target image mask according to the third energy distribution information, so that the target image mask is displayed as background layer bottom noise of the to-be-displayed image, the contrast sensitivity of the human eye to the predicted artifact is reduced, the predicted artifact is masked, and therefore the display effect of the near-eye display equipment is improved.

Description

Technical Field

[0001] This application relates to the field of near-eye display technology, and in particular to a display method, apparatus, device and storage medium for a near-eye display device. Background Technology

[0002] Currently, for near-eye display devices, such as augmented reality (AR) glasses using diffractive waveguides (especially monochromatic resin waveguides), due to the sub-diffraction orders of the gratings included in the diffractive waveguide, residual reflections on the front surface of the lens, and internal stress of the waveguide material, the virtual image (such as an icon) displayed by AR glasses often produces a dim reflection around the main image (usually below or to the side), which is the physical artifact corresponding to the main image.

[0003] In related technologies, physical coating (AR coating) is generally used to reduce the occurrence of physical artifacts. However, physical coating cannot eliminate reflection, and physical artifacts, a diffraction byproduct, are determined by the physical lattice characteristics of the diffractive waveguide, making them difficult to reduce or even eliminate through physical coating.

[0004] Due to the existence of physical artifacts, when displaying images on near-eye display devices, if a high-brightness monochromatic optical engine (such as Micro-LED) is used to display the corresponding image, the faint physical artifacts become extremely glaring against a dark background, which can easily lead to poor display effects of near-eye display devices, and consequently, poor visual experience for users. Summary of the Invention

[0005] The main objective of this application is to provide a display method, apparatus, device, and storage medium for a near-eye display device, aiming to solve the technical problem of poor display effect of near-eye display devices due to the presence of physical artifacts.

[0006] In a first aspect, this application provides a display method for a near-eye display device, comprising: Obtain the first energy distribution information of the image to be displayed by the near-eye display device; Based on the preset optical model of the near-eye display device, the second energy distribution information of the predicted artifacts corresponding to the image to be displayed is determined according to the first energy distribution information; Based on the first energy distribution information and the second energy distribution information, the third energy distribution information of the target image mask corresponding to the image to be displayed is determined; Based on the first energy distribution information, the near-eye display device is controlled to display the image to be displayed, and based on the third energy distribution information, the near-eye display device is controlled to display the target image mask, so that the target image mask is displayed as the background noise of the image to be displayed.

[0007] Secondly, this application provides a display device for a near-eye display device, the display device comprising: An energy distribution acquisition module is used to acquire first energy distribution information of the image to be displayed by the near-eye display device; The first energy distribution determination module is used to determine the second energy distribution information of the predicted artifact corresponding to the image to be displayed based on the first energy distribution information and a preset optical model of the near-eye display device. The second energy distribution determination module is used to determine the third energy distribution information of the target image mask corresponding to the image to be displayed based on the first energy distribution information and the second energy distribution information. The display module is configured to control the near-eye display device to display the image to be displayed based on the first energy distribution information, and to control the near-eye display device to display the target image mask based on the third energy distribution information, so that the target image mask is displayed as the background noise of the image to be displayed.

[0008] Thirdly, this application provides a near-eye display device, which includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, when executing the computer program, implement the steps of the display method for the near-eye display device as described above.

[0009] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the display method for a near-eye display device as described above.

[0010] Having obtained the first energy distribution information of the image to be displayed by the near-eye display device, the first energy distribution information can be used by the near-eye display device, combined with its preset optical model, to predict the second energy distribution information of the predicted artifacts corresponding to the image to be displayed. The first and second energy distribution information can be used by the near-eye display device to determine the third energy distribution information of the target image mask corresponding to the image to be displayed. The third energy distribution information can be used by the near-eye display device to display the target image mask. When the third energy distribution information is determined based on the first and second energy distribution information, the target image mask can be displayed as the background noise of the image to be displayed, thereby reducing the human eye's sensitivity to the contrast of the predicted artifacts and masking them. When the target image mask can mask the predicted artifacts, it can reduce or even eliminate the predicted artifacts perceived by the human eye, thus improving the display effect of the near-eye display device and enhancing the visual experience of the human eye with the image displayed by the near-eye display device. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of a display method for a near-eye display device provided in an embodiment of this application; Figure 2 This is a schematic block diagram of a display device for a near-eye display apparatus provided in an embodiment of this application; Figure 3 This is a schematic block diagram of a near-eye display device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] This application provides a display method, apparatus, device, and storage medium for a near-eye display device. The display method for this near-eye display device can be applied to near-eye display devices. Near-eye display devices may include AR glasses, virtual reality (VR) glasses, AR headsets, VR headsets, etc., and are not limited thereto.

[0014] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0015] Please see Figure 1 , Figure 1This is a schematic flowchart illustrating a display method for a near-eye display device according to an embodiment of this application. It should be noted that the display method for a near-eye display device provided in this embodiment can be used in near-eye display devices.

[0016] like Figure 1 As shown, the display method of the near-eye display device includes steps S101 to S104.

[0017] S101. Obtain the first energy distribution information of the image to be displayed by the near-eye display device.

[0018] For example, the image to be displayed in a near-eye display device can also be referred to as the main image of the near-eye display device. The image to be displayed may include multiple pixels to be displayed.

[0019] In some implementations, to improve the comprehensiveness of the prediction of predicted artifacts corresponding to the image to be displayed, all preset pixels included in the image to be displayed can be determined as the pixels to be displayed, for subsequent calculation of the predicted artifacts corresponding to the image to be displayed. Predicted artifacts are the physical artifacts that may be affected by the diffraction effect of near-eye display devices, predicted before the actual display of the image to be displayed.

[0020] In some implementations, to simplify computation, a subset of preset pixels can be selected from all preset pixels in the image to be displayed as the pixels to be displayed, for subsequent calculation of the predicted artifacts corresponding to the image to be displayed.

[0021] For example, when the image to be displayed includes multiple preset image objects, some of the preset image objects can be selected as the target image objects of the image to be displayed. The preset image objects include at least one of the icons in the image to be displayed, the text in the image to be displayed, etc., and are not limited thereto.

[0022] Based on the center pixel of the target image object, determine the pixels to be displayed in the image to be displayed. The center pixel can be used to indicate the geometric center pixel or the brightness center pixel of the pixel region where the target image object is located in the image to be displayed.

[0023] When the center pixel is used to indicate the geometric center pixel of the pixel region where the target image object is located in the image to be displayed, the pixel coordinates of all preset pixels in the pixel region where the target image object is located can be averaged to obtain the average value of the corresponding pixel coordinates. This average value of the pixel coordinates is then rounded down to obtain the rounded average value of the pixel coordinates. This rounded average value of the pixel coordinates can be used to determine the pixel coordinates of the geometric center pixel of the pixel region.

[0024] The pixel coordinates of each preset pixel in the pixel region where the target image object is located in the image to be displayed are represented as follows: Taking a region with N preset pixels as an example, averaging the pixel coordinates of all preset pixels in the pixel region can be expressed as:

[0025] in, Used to indicate the average value of the pixel coordinates of each preset pixel in a pixel region.

[0026] right By performing rounding, the pixel coordinates of the geometric center pixel of the pixel region can be determined.

[0027] When the center pixel is used to indicate the brightness center pixel of the pixel region where the target image object is located in the image to be displayed, the brightness of all preset pixels in the pixel region where the target image object is located can be obtained. Then, using the brightness of all preset pixels in the pixel region as weights, a weighted average is calculated on the pixel coordinates of each preset pixel in the pixel region to obtain the corresponding weighted average pixel coordinates. This weighted average pixel coordinates is then rounded to obtain the rounded weighted average pixel coordinates. The rounded weighted average can be used to determine the pixel coordinates of the brightness center pixel of the pixel region.

[0028] The pixel coordinates of each preset pixel in the pixel region where the target image object is located in the image to be displayed are represented as follows: The brightness of each preset pixel is represented as For example, using the brightness of all preset pixels in a pixel region as weights, the weighted average of the pixel coordinates of each preset pixel in the pixel region can be expressed as:

[0029] in, It is used to indicate the weighted average of the pixel coordinates of each pixel in a pixel region.

[0030] right By rounding down, the pixel coordinates of the brightness center pixel can be determined.

[0031] For example, the first energy distribution information includes at least a first position of the pixel to be displayed in the image to be displayed and a first energy intensity of the pixel to be displayed. The first position may include the pixel coordinates of the pixel to be displayed. The first energy intensity may include the first luminance of the pixel to be displayed. However, it is not limited to this; the first energy distribution information may also include a first luminous area of ​​the pixel to be displayed. The first luminous area can be used to indicate the area of ​​the effective luminous region occupied by the pixel on the near-eye display device when it is lit. No limitation is imposed here.

[0032] In some implementations, selecting a portion of preset image objects from a plurality of preset image objects as target image objects for an image to be displayed includes: obtaining a preset energy intensity of the preset image objects; and determining the preset image objects as target image objects when the preset energy intensity is greater than or equal to a preset energy intensity threshold.

[0033] The preset energy intensity of a preset image object can be determined based on at least one of the average, maximum, median, and mode of the pixel energy intensity of each preset pixel included in the preset image object.

[0034] Taking the preset energy intensity including the brightness of a preset image object as an example, the brightness of the preset image object can be determined based on at least one of the average brightness, maximum brightness, median brightness, and mode brightness of all preset pixels included in the preset image object. Of course, it is not limited to this, and no restriction is made here.

[0035] Having obtained the first energy distribution information of the image to be displayed by the near-eye display device, this first energy distribution information can be used to subsequently predict the prediction artifacts corresponding to the image to be displayed. Accordingly, during the process of obtaining the first energy distribution information of the image to be displayed, a portion of preset pixels can be selected from all preset pixels included in the image to be displayed as the pixels to be displayed. These pixels can be used to subsequently predict the prediction artifacts corresponding to the image to be displayed, which helps to reduce the amount of calculation required to predict the prediction artifacts corresponding to the image to be displayed.

[0036] S102. Based on the preset optical model of the near-eye display device, determine the second energy distribution information of the predicted artifact corresponding to the image to be displayed according to the first energy distribution information.

[0037] For example, a near-eye display device may be pre-built with a preset optical model of the near-eye display device.

[0038] In some implementations, the near-eye display device is controlled to display a preset calibration image; the near-eye display device is scanned across the entire field of view using a preset scanning device to obtain an energy distribution image corresponding to the preset calibration image, the energy distribution image being used to characterize the mapping relationship between the preset calibration image and the preset artifacts corresponding to the preset calibration image; and a preset optical model of the near-eye display device is established based on the energy distribution image.

[0039] For example, a near-eye display device may include a display module. The display module may include a diffractive waveguide. Due to the presence of the diffractive waveguide in the near-eye display device, the diffraction effect of the diffractive waveguide will cause a preset artifact corresponding to the preset calibration image when the near-eye display device displays the preset calibration image. The preset artifact is the physical artifact generated by the diffraction effect of the diffractive waveguide when the preset calibration image is actually displayed.

[0040] For example, a preset calibration image may include multiple preset calibration pixels. When a near-eye display device displays a preset calibration image, each preset calibration pixel may have a corresponding calibration pixel artifact; that is, each calibration pixel artifact is caused by its corresponding preset calibration pixel. The calibration pixel artifacts corresponding to all the preset calibration pixels included in the preset calibration image can form the preset artifact corresponding to the preset calibration image.

[0041] For example, when controlling a near-eye display device to display a preset calibration image, the near-eye display device can be scanned across the entire field of view using a preset scanning device in a standard laboratory environment to obtain the energy distribution map corresponding to the preset calibration image.

[0042] Preset scanning devices include 3D scanners, industrial cameras, LiDAR, etc. These devices can be used to perform full-field scanning of near-eye display devices. They can establish a communication connection with the near-eye display device. No restrictions are imposed here.

[0043] The energy distribution image can record the energy distribution information of preset calibration pixels included in the preset calibration image. The energy distribution information of the preset calibration pixels includes at least the location and energy intensity of the preset calibration pixels. Furthermore, the energy distribution image can also record the energy distribution information of calibration pixel artifacts generated at the location of the preset calibration pixels. The energy distribution information of the calibration pixel artifacts includes at least the location and energy intensity of the calibration pixel artifacts.

[0044] For example, an energy distribution image can record the positions of calibration pixel artifacts generated at different locations of preset calibration pixels in a preset calibration image. Therefore, the energy distribution image can characterize the positional mapping relationship between preset calibration pixels and their corresponding calibration pixel artifacts in the preset calibration image. Correspondingly, the energy distribution image can record the energy intensity of the calibration pixel artifacts generated at different locations of preset calibration pixels in the preset calibration image. Therefore, the energy distribution image can characterize the energy intensity mapping relationship between preset calibration pixels and their corresponding calibration pixel artifacts in the preset calibration image.

[0045] Of course, this is not limited to this. For example, the energy distribution image can also record the displacement vector, diffusion radius, energy attenuation coefficient, etc., of the calibration pixel artifacts generated at different locations of the preset calibration pixels in the preset calibration image. The displacement vector is equivalent to the positional offset of the calibration pixel artifact relative to the corresponding preset calibration pixel. The diffusion radius is equivalent to the radius by which the calibration pixel artifact diffuses from its location to other locations. The energy attenuation coefficient is equivalent to the change in energy intensity as the calibration pixel artifact diffuses from its location to other locations. For example, the energy distribution image can also record the area of ​​the calibration pixel artifacts generated at different locations of the preset calibration pixels in the preset calibration image. Therefore, the energy distribution image can characterize the area mapping relationship between the preset calibration pixels in the preset calibration image and the corresponding calibration pixel artifacts. This is also not limited.

[0046] Near-eye display devices can combine the energy distribution information of preset calibration pixels with the energy distribution information of the corresponding calibration pixel artifacts to characterize the mapping relationship between the preset calibration image and the preset artifacts corresponding to the preset calibration image. In other words, the energy distribution image can be used to characterize the mapping relationship between the preset calibration image and the preset artifacts corresponding to the preset calibration image. For example, near-eye display devices can solidify the energy distribution image into a mathematical model to establish a preset optical model for the near-eye display device.

[0047] Taking the energy distribution information of the preset calibration pixel, including the position and energy intensity of the preset calibration pixel, as an example, the energy distribution information of the calibration pixel artifact caused by the preset calibration pixel includes the position and energy intensity of the calibration pixel artifact.

[0048] The preset calibration pixels include calibration pixel 1, calibration pixel 2, ..., calibration pixel n. The position of calibration pixel 1 is position P1, the position of calibration pixel 2 is position P2, and the position of calibration pixel n is position Pn. The energy intensity of the calibration pixels is luminance I1, the energy intensity of calibration pixel 2 is luminance I2, and the energy intensity of calibration pixel n is luminance In.

[0049] The calibration pixel artifact caused by calibration pixel 1 is denoted as Calibration Pixel Artifact 1, the calibration pixel artifact caused by calibration pixel 2 is denoted as Calibration Pixel Artifact 2, and so on, with the calibration pixel artifact caused by calibration pixel n being denoted as Calibration Pixel Artifact n. The position of Calibration Pixel Artifact 1 is denoted as position Pg1, the position of Calibration Pixel Artifact 2 as Pg2, and the position of Calibration Pixel Artifact n as position Pgn. The energy intensity of the calibration pixel artifact is denoted as luminance Ig1, the energy intensity of the calibration pixel artifact is denoted as luminance Ig2, and the energy intensity of the calibration pixel artifact n as luminance Ign.

[0050] The mapping relationship between the preset calibration image and the preset artifact corresponding to the preset calibration image can be expressed as: P1→Pg1, P2→Pg2, Pn→Pgn; I1→Ig1, I2→Ig2, In→Ign.

[0051] It should be noted that when a near-eye display device displays a preset calibration image or an image to be displayed, for the same location of the calibration pixel and the image to be displayed, the calibration pixel artifact caused by the calibration pixel can be the same as the predicted pixel artifact caused by the image to be displayed. Therefore, the mapping relationship between the preset calibration image and the preset artifact corresponding to the preset calibration image can also be used as the mapping relationship between the image to be displayed and the predicted artifact corresponding to the image to be displayed. Thus, the mapping relationship between the preset calibration image represented by the energy distribution image and the preset artifact corresponding to the preset calibration image can be solidified into a mathematical model. This mathematical model can cover the mapping relationships between the images to be displayed and their corresponding predicted artifacts, such as P1→Pg1, P2→Pg2, Pn→Pgn; I1→Ig1, I2→Ig2, In→Ign, etc.

[0052] The preset optical model can be stored in the near-eye display device, such as in the non-volatile memory of the near-eye display device. Of course, the preset optical model is not limited to this. The preset optical model can also be determined by combining the energy distribution image and at least one of the following: the grating diffraction order distribution model corresponding to the diffraction waveguide included in the near-eye display device, and the lens reflection energy model.

[0053] Based on this, by establishing a preset optical model for the near-eye display device, the preset optical model can be used to predict the predicted artifacts corresponding to the image to be displayed in real time, which is beneficial to improving the convenience and accuracy of determining the predicted artifacts corresponding to the image to be displayed.

[0054] When a near-eye display device stores a preset optical model, the near-eye display device can input the first energy distribution information of the image to be displayed into the preset optical model. Then, the preset optical model can output the second energy distribution information of the predicted artifacts corresponding to the image to be displayed based on the first energy distribution information.

[0055] For example, the first energy distribution information includes at least the first position of the pixel to be displayed in the image to be displayed and the first energy intensity of the pixel to be displayed.

[0056] In some implementations, based on the first position of the pixel to be displayed, a second position corresponding to the first position is determined from the mapping relationship represented by a preset optical model, the second position representing the position of the predicted pixel artifact caused by the pixel to be displayed; based on the first energy intensity of the pixel to be displayed, a second energy intensity corresponding to the first energy intensity is determined from the mapping relationship represented by the preset optical model, the second energy intensity representing the energy intensity of the predicted pixel artifact caused by the pixel to be displayed; based on the second position and second energy intensity of the predicted pixel artifacts caused by each of the multiple pixels to be displayed in the image to be displayed, the second energy distribution information of the predicted artifacts corresponding to the image to be displayed is determined.

[0057] Since both the image to be displayed and the preset calibration image are images that the near-eye display device needs to display or actually displays, the image to be displayed can also be used as the preset calibration image, and the preset calibration image can also be used as the image to be displayed. Therefore, the preset optical model can also be used to characterize the mapping relationship between the image to be displayed and the predicted artifacts corresponding to the image to be displayed.

[0058] For example, if the mapping relationship represented by the preset optical model includes the positional mapping relationship between preset calibration pixels and calibration pixel artifacts, the mapping relationship represented by the preset optical model also includes the positional mapping relationship between the pixel to be displayed and the predicted pixel artifact caused by the pixel to be displayed. Based on this, if the first position of the pixel to be displayed in the image to be displayed is obtained, the second position corresponding to the first position can be determined from the positional mapping relationship, and the second position represents the position of the predicted pixel artifact corresponding to the pixel to be displayed.

[0059] Taking the first position of the pixel to be displayed as P_main and the second position of the predicted pixel artifact corresponding to the pixel to be displayed as P_ghost as an example, the position mapping relationship can be expressed as P_ghost = P_main + v, where v is used to indicate the displacement vector of the second position relative to the first position. When determining the displacement vector of the second position relative to the first position, the change in the incident angle of the image to be displayed in the field of view can be considered to improve the robustness of determining the second position.

[0060] For example, if the mapping relationship represented by the preset optical model includes the energy intensity relationship between preset calibration pixels and calibration pixel artifacts, the mapping relationship represented by the preset optical model also includes the energy intensity mapping relationship between the image to be displayed and the corresponding predicted pixel artifacts. Based on this, if the first energy intensity of the pixels to be displayed in the image to be displayed is obtained, the second energy intensity corresponding to the first energy intensity can be determined from the energy intensity mapping relationship. The second energy intensity represents the energy intensity of the predicted pixel artifact corresponding to the pixel to be displayed.

[0061] When the image to be displayed includes multiple pixels, the predicted pixel artifacts corresponding to each pixel to be displayed together form the predicted artifacts corresponding to the image to be displayed. Then, the near-eye display device can determine the energy intensity of the predicted artifacts corresponding to the image to be displayed based on the second position and the second energy intensity of the predicted pixel artifacts.

[0062] Of course, this is not the only possibility. For example, based on the first luminous area of ​​the pixel to be displayed, a second luminous area corresponding to the first luminous area is determined from the mapping relationship represented by a preset optical model. The second luminous area represents the luminous area of ​​the predicted pixel artifact caused by the pixel to be displayed. No further limitations are imposed here.

[0063] Based on the preset optical model of the near-eye display device, and according to the first energy distribution information, the second energy distribution information is used to determine the second energy distribution information of the predicted artifact corresponding to the image to be displayed. The second energy distribution information can then be combined with the first energy distribution information to determine the third energy distribution information of the target image mask corresponding to the image to be displayed.

[0064] S103. Based on the first energy distribution information and the second energy distribution information, determine the third energy distribution information of the target image mask corresponding to the image to be displayed.

[0065] Given the first energy distribution information and the second energy distribution information, the near-eye display device can combine the first energy distribution information and the second energy distribution information to determine the third energy distribution information of the target image mask corresponding to the image to be displayed.

[0066] For example, the third energy distribution information includes at least the third location and the third energy intensity of the target image mask.

[0067] In some implementations, a third energy intensity of the target image mask is determined based on a first energy intensity included in the first energy distribution information, wherein the third energy intensity is positively correlated with the first energy intensity.

[0068] For example, a first energy intensity can be used to indicate the first brightness of the pixel to be displayed. A third energy intensity can be used to indicate the grayscale value of the target image mask.

[0069] Taking the first energy intensity as L_main and the third energy intensity as G_comp as an example, the third energy intensity can be expressed as G_comp = L_main × k(1 + a), where k indicates the first energy attenuation coefficient included in the first energy distribution information, and a indicates the preset visual overmodulation coefficient. For example, the visual overmodulation coefficient can be set to be greater than or equal to 5% and less than or equal to 10%, but it is not limited to this. The visual overmodulation coefficient can be preset, set by the user, or adjusted by the user; there are no restrictions here.

[0070] When the third energy intensity is positively correlated with the first energy intensity, the greater the first energy intensity, the greater the third energy intensity will be, and the smaller the first energy intensity, the smaller the third energy intensity will be.

[0071] In some implementations, the third position of the target image mask is determined based on the second position included in the second energy distribution information.

[0072] Based on the consideration of using a target image mask to change the noise distribution of the region where the predicted artifact is located, and taking advantage of the visual masking effect of the human eye and the principle of spatial frequency submersion, the predicted artifact is made "invisible" at the perception level, that is, the human eye's perception of the predicted artifact is reduced or even eliminated. The near-eye display device needs to set the target image mask at the second position of each predicted pixel artifact included in the predicted artifact, so that the target image mask can cover the corresponding predicted pixel artifact. Then, the third position of the target image mask can be determined according to the second position included in the second energy distribution information.

[0073] Taking the second position of the predicted pixel artifact in the predicted artifact as P_ghost as an example, the third position of the target image mask can also be represented as P_ghost.

[0074] Accordingly, since the second position of the predicted pixel artifact in the prediction artifact is determined based on the first position of the corresponding pixel to be displayed in the image to be displayed, the third position of the target image mask is essentially determined indirectly based on the first position of the corresponding pixel to be displayed. Therefore, when the first position of the pixel to be displayed changes, the second position of the corresponding predicted pixel artifact in the prediction artifact also changes, and consequently, the third position of the corresponding target image mask also changes, thus achieving pixel-level dynamic tracking.

[0075] Of course, the third energy distribution information is not limited to this. For example, the third energy distribution information may also include the third luminous area of ​​the target image mask. The third luminous area of ​​the target image mask is determined based on the second luminous area included in the second energy distribution information.

[0076] Based on the consideration of using a target image mask to change the noise distribution of the region where the predicted artifact is located, and taking advantage of the visual masking effect of the human eye and the principle of spatial frequency submersion, so that the predicted artifact can be "invisible" at the perception level, the near-eye display device needs to set a corresponding target image mask at the second position of each predicted pixel artifact included in the predicted artifact, and set a third luminous area of ​​the corresponding target image mask according to the second luminous area of ​​each predicted pixel artifact. Then, the third luminous area of ​​the target image mask can be determined according to the second luminous area included in the second energy distribution information.

[0077] For example, the third luminous area of ​​the target image mask is greater than the second luminous area of ​​the corresponding predicted pixel artifact, and the ratio between the third luminous area and the second luminous area is a preset multiple.

[0078] For example, the preset multiplier can be greater than or equal to 1.1 and less than or equal to 1.3. Of course, the preset multiplier is not limited to this; it can be pre-set, or it can be set or adjusted by the user. There are no restrictions here.

[0079] Even if the first position of the pixel to be displayed in the image is slightly offset due to installation tolerance during the installation of the near-eye display device, resulting in a slight offset between the actual artifact of the image to be displayed and the corresponding predicted artifact, the target image mask can still cover the actual artifact of the image to be displayed during the display.

[0080] When the third energy distribution information of the target image mask corresponding to the image to be displayed is determined based on the first energy distribution information and the second energy distribution information, the third energy distribution information can be used by the near-eye display device to subsequently display the target image mask.

[0081] S104. Based on the first energy distribution information, control the near-eye display device to display the image to be displayed, and based on the third energy distribution information, control the near-eye display device to display the target image mask, so that the target image mask is displayed as the background noise of the image to be displayed.

[0082] For example, given the first energy distribution information, the near-eye display device can display an image to be displayed based on the first energy distribution information. Taking the first energy distribution information as including the first position and first energy intensity of the pixel to be displayed, the near-eye display device can display a pixel with the first energy intensity at the first position. Furthermore, the multiple pixels displayed by the near-eye display device can form an image to be displayed, thereby enabling the near-eye display device to display the image.

[0083] Given the third energy distribution information, the near-eye display device can display a target image mask based on the third energy distribution information.

[0084] For example, at least at the third position, a target image mask with a third energy intensity is displayed.

[0085] Of course, it is not limited to this. For example, when displaying a target image mask, a near-eye display device can also enlarge the third luminous area of ​​the target image mask so that the target image mask can more comprehensively cover the corresponding predicted artifacts.

[0086] For example, when displaying a target image mask, a near-eye display device can also filter the target image mask so that the edges of the target image mask present a preset shape.

[0087] For example, the preset shape may include a Gaussian distribution. Near-eye display devices can apply Gaussian filtering to the target image mask so that the edges of the target image mask have a Gaussian distribution.

[0088] When the edges of the target image mask are presented in a preset shape, the edges of the target image mask can reduce the human eye's capture of the sharp edges of the corresponding predicted artifacts, thereby improving the compensation effect of the target image mask on the predicted artifacts.

[0089] When the near-eye display device displays an image to be displayed based on the first energy distribution information, and displays a target image mask based on the third energy distribution information, the target image mask can be superimposed on the background layer of the image to be displayed as a compensation for the noise floor of the image to be displayed. Prediction artifacts caused by the image to be displayed can be superimposed on the compensation noise floor. Utilizing the Weber-Fechner Law, when the energy intensity of the background layer of the image to be displayed increases, the contrast perception sensitivity of the human eye to prediction artifacts of the same energy decreases by more than 70%. Based on this, the target image mask can transform prediction artifacts from independent interference objects into part of the background noise in the background layer of the image to be displayed, thereby reducing or even eliminating prediction artifacts perceived by the human eye, thus improving the display effect of the near-eye display device.

[0090] In some implementations, in response to a change in at least one of the ambient energy distribution information of the environment where the near-eye display device is located and the device temperature information of the near-eye display device, the third energy distribution information is adjusted to obtain the adjusted third energy distribution information.

[0091] Based on the adjusted third energy distribution information, the near-eye display device is controlled to display the target image mask.

[0092] Near-eye display devices can sense the ambient energy distribution information of their surroundings using an ambient light sensor. When this ambient energy distribution changes, to improve the display effect while maintaining user immersion, the near-eye display device can adjust its third energy distribution information accordingly, resulting in an adjusted third energy distribution. Based on this adjusted third energy distribution information, the near-eye display device can then display the target image mask.

[0093] For example, ambient energy distribution information includes ambient energy intensity. Ambient energy intensity can be used to indicate the ambient brightness of the environment in which the near-eye display device is located.

[0094] For example, in response to changes in environmental energy intensity, the third energy intensity included in the third energy distribution information is adjusted to obtain the adjusted third energy intensity, which is positively correlated with changes in environmental energy intensity.

[0095] For example, when the ambient energy intensity increases, the near-eye display device can increase the third energy intensity included in the third energy distribution information so that the target image mask displayed according to the adjusted third energy intensity can be adapted to the environment after the ambient energy intensity increases. That is, the target image mask can still reduce or even eliminate the prediction artifacts perceived by the human eye, which is beneficial to improving the display effect of the near-eye display device.

[0096] For example, when the ambient energy intensity decreases, the near-eye display device can reduce the third energy intensity included in the third energy distribution information so that the target image mask displayed subsequently can be adapted to the environment after the ambient energy intensity decreases, thereby preventing the background layer of the image to be displayed from being too bright and affecting the user's visual immersion in the displayed image.

[0097] The increase in energy intensity in response to an increase in ambient energy intensity can be preset or set or adjusted by the user. No restrictions are imposed here.

[0098] The magnitude of the energy intensity reduction when the third energy intensity decreases in response to a lower ambient energy intensity can be preset or set or adjusted by the user. No restrictions are imposed here.

[0099] In one exemplary embodiment, the mapping relationship between the ambient energy intensity and the third energy intensity can be pre-tested to ensure that the third energy intensity is adapted to the ambient energy intensity. Accordingly, when the ambient energy intensity changes, the third energy intensity can be adjusted based on the mapping relationship between the ambient energy intensity and the third energy intensity to ensure that the adjusted third energy intensity is adapted to the changed ambient energy intensity. However, this is not a limitation and is not intended to restrict the scope of the invention.

[0100] For example, in response to changes in device temperature information, the third position included in the third energy distribution information is adjusted to obtain an adjusted third position, which is positively correlated with changes in device temperature information.

[0101] The materials used in the fabrication of near-eye display devices may deform with temperature. For example, in a near-eye display device that includes a display module, and the display module includes an optical waveguide, the device temperature information may include the optical waveguide temperature. The materials used in the fabrication of the optical waveguide may include resin materials. The resin materials will deform with changes in the temperature of the optical waveguide.

[0102] Therefore, in order to make the target image mask adaptable to the deformation of the fabrication material of the near-eye display device, the near-eye display device can adjust the third position, which includes the third energy distribution information, in response to changes in device temperature information, to obtain the adjusted third position. The adjusted third position is positively correlated with changes in device temperature information.

[0103] For example, when the device temperature information increases, the near-eye display device can increase the third position included in the third energy distribution information, so that the target image mask displayed subsequently based on the adjusted third position can adapt to the deformation of the near-eye display device after the device temperature information increases. This allows the target image mask to still reduce or even eliminate the prediction artifacts perceived by the human eye, which is beneficial to improving the display effect of the near-eye display device.

[0104] For example, when the device temperature information decreases, the near-eye display device can reduce the third position included in the third energy distribution information, so that the target image mask displayed subsequently based on the adjusted third position can adapt to the deformation of the near-eye display device after the device temperature information decreases. This allows the target image mask to still reduce or even eliminate the prediction artifacts perceived by the human eye, which is beneficial to improving the display effect of the near-eye display device.

[0105] The magnitude of the position increase when responding to an increase in device temperature can be preset, or it can be set or adjusted by the user. No restrictions are imposed here.

[0106] The magnitude of the position reduction when responding to a decrease in device temperature information to the reduction of the third position can be preset, or it can be set or adjusted by the user. There are no restrictions here.

[0107] In one exemplary embodiment, the mapping relationship between the device temperature information and the third position can be pre-tested to ensure that the third position is adapted to the device temperature information. Accordingly, when the device temperature information changes, the third position can be adjusted based on the mapping relationship between the device temperature information and the third position so that the adjusted third position is adapted to the changed device temperature information. However, this is not a limitation and is not intended to restrict the scope of the invention.

[0108] Therefore, adjusting the third energy distribution information in response to changes in at least one of the environmental energy distribution information of the near-eye display device and the device temperature information of the near-eye display device can improve the robustness of the target image mask under different environments and temperatures, thereby improving the display effect of the image to be displayed.

[0109] Please see Figure 2 , Figure 2 This is a schematic block diagram of a display device for a near-eye display device according to an embodiment of this application. The display device can be configured in a near-eye display device to perform the aforementioned display method of the near-eye display device. The near-eye display device may include AR glasses, VR glasses, AR helmets, VR helmets, etc., and is not limited thereto.

[0110] like Figure 2 As shown, the display device includes an energy distribution acquisition module 110, a first energy distribution determination module 120, a second energy distribution determination module 130, and a display module 140.

[0111] The energy distribution acquisition module 110 is used to acquire the first energy distribution information of the image to be displayed by the near-eye display device; The first energy distribution determination module 120 is used to determine the second energy distribution information of the predicted artifact corresponding to the image to be displayed based on the first energy distribution information and a preset optical model of the near-eye display device. The second energy distribution determination module 130 is used to determine the third energy distribution information of the target image mask corresponding to the image to be displayed based on the first energy distribution information and the second energy distribution information. Display module 140 is configured to control the near-eye display device to display the image to be displayed based on the first energy distribution information, and to control the near-eye display device to display the target image mask based on the third energy distribution information, so that the target image mask is displayed as the background noise of the image to be displayed.

[0112] For example, the first energy distribution information includes at least the first position of the pixel to be displayed in the image to be displayed and the first energy intensity of the pixel to be displayed; The first energy distribution determination module 120 includes a first location determination submodule, a first energy intensity determination submodule, and a first energy distribution determination submodule.

[0113] The first position determination submodule is used to determine the second position corresponding to the first position from the mapping relationship represented by the preset optical model based on the first position of the pixel to be displayed. The second position represents the position of the predicted pixel artifact caused by the pixel to be displayed. The first energy intensity determination submodule is used to determine the second energy intensity corresponding to the first energy intensity from the mapping relationship represented by the preset optical model based on the first energy intensity of the pixel to be displayed. The second energy intensity represents the energy intensity of the predicted pixel artifact caused by the pixel to be displayed. The first energy distribution determination submodule is used to determine the second energy distribution information of the predicted artifacts corresponding to the image to be displayed based on the second position and second energy intensity of the predicted pixel artifacts caused by each of the plurality of pixels to be displayed in the image to be displayed.

[0114] For example, the second energy distribution determination module 130 includes a second energy intensity determination submodule, a second position determination submodule, and a second energy distribution determination submodule.

[0115] The second energy intensity determination submodule is used to determine the third energy intensity of the target image mask based on the first energy intensity included in the first energy distribution information, wherein the third energy intensity is positively correlated with the first energy intensity; The second position determination submodule is used to determine the third position of the target image mask based on the second position included in the second energy distribution information; The second energy distribution determination submodule is used to determine the third energy distribution information of the target image mask based on the third energy intensity and the third position.

[0116] For example, display module 140 includes a first display submodule.

[0117] A first display submodule is configured to display a target image mask having a third energy intensity at least at the third position.

[0118] For example, the display device also includes an energy distribution adjustment submodule.

[0119] An energy distribution adjustment submodule is used to adjust the third energy distribution information in response to a change in at least one of the environmental energy distribution information of the environment where the near-eye display device is located and the device temperature information of the near-eye display device, so as to obtain the adjusted third energy distribution information. Display module 140 includes a second display submodule.

[0120] The second display submodule is used to control the near-eye display device to display the target image mask according to the adjusted third energy distribution information.

[0121] For example, the energy distribution adjustment submodule includes an energy intensity adjustment submodule.

[0122] An energy intensity adjustment submodule is used to adjust the third energy intensity included in the third energy distribution information in response to changes in the ambient energy intensity, so as to obtain an adjusted third energy intensity. The adjusted third energy intensity is positively correlated with changes in the ambient energy intensity. The energy distribution adjustment submodule includes the position adjustment submodule.

[0123] The position adjustment submodule is used to adjust the third position included in the third energy distribution information in response to the change of the device temperature information, so as to obtain the adjusted third position, which is positively correlated with the change of the device temperature information.

[0124] For example, the display device also includes a calibration image display submodule, a scanning submodule, and a modeling submodule.

[0125] The image display submodule is used to control the near-eye display device to display a preset calibration image; The scanning submodule is used to perform a full-field scan of the near-eye display device using a preset scanning device to obtain an energy distribution image corresponding to the preset calibration image. The energy distribution image is used to characterize the mapping relationship between the preset calibration image and the preset artifact corresponding to the preset calibration image. The modeling submodule is used to establish a preset optical model of the near-eye display device based on the energy distribution image.

[0126] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0127] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0128] For example, the above-described method and apparatus can be implemented as a computer program that can run on a near-eye display device to display an image to be displayed and a target image mask corresponding to the image to be displayed. For example, the near-eye display device may include VR glasses, AR glasses, VR headsets, AR headsets, etc., and is not limited thereto.

[0129] Please see Figure 3 , Figure 3 This is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of this application.

[0130] like Figure 3 As shown, the near-eye display device includes a memory and a processor. The memory and processor can be connected via a system bus, and the memory may include a storage medium and internal memory.

[0131] The storage medium can store the operating system and computer programs. When the computer program is executed, it enables the processor to perform any display method of the near-eye display device.

[0132] The processor provides computing and control capabilities to support the operation of the entire near-eye display device.

[0133] Internal memory provides an environment for the execution of computer programs stored in storage media. When these computer programs are executed by the processor, the processor can execute any display method of a near-eye display device.

[0134] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the near-eye display device to which the present application is applied. A specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0136] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, perform the following steps: Obtain the first energy distribution information of the image to be displayed by the near-eye display device; Based on the preset optical model of the near-eye display device, the second energy distribution information of the predicted artifacts corresponding to the image to be displayed is determined according to the first energy distribution information; Based on the first energy distribution information and the second energy distribution information, the third energy distribution information of the target image mask corresponding to the image to be displayed is determined; Based on the first energy distribution information, the near-eye display device is controlled to display the image to be displayed, and based on the third energy distribution information, the near-eye display device is controlled to display the target image mask, so that the target image mask is displayed as the background noise of the image to be displayed.

[0137] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the near-eye display device described above can be referred to the corresponding process in the aforementioned near-eye display device display method embodiments, and will not be repeated here.

[0138] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to in various embodiments of the display method of the near-eye display device of this application.

[0139] The computer-readable storage medium can be an internal storage unit of the near-eye display device described in the foregoing embodiments, such as the hard disk or memory of the near-eye display device. Alternatively, the computer-readable storage medium can be an external storage device of the near-eye display device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the near-eye display device.

[0140] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A display method for a near-eye display device, characterized in that, include: Obtain the first energy distribution information of the image to be displayed by the near-eye display device; Based on the preset optical model of the near-eye display device, the second energy distribution information of the predicted artifacts corresponding to the image to be displayed is determined according to the first energy distribution information; Based on the first energy distribution information and the second energy distribution information, the third energy distribution information of the target image mask corresponding to the image to be displayed is determined; Based on the first energy distribution information, the near-eye display device is controlled to display the image to be displayed, and based on the third energy distribution information, the near-eye display device is controlled to display the target image mask, so that the target image mask is displayed as the background noise of the image to be displayed.

2. The display method according to claim 1, characterized in that, The first energy distribution information includes at least the first position of the pixel to be displayed in the image to be displayed and the first energy intensity of the pixel to be displayed; The method of determining the second energy distribution information of the predicted artifacts corresponding to the image to be displayed based on the preset optical model of the near-eye display device and according to the first energy distribution information includes: Based on the first position of the pixel to be displayed, a second position corresponding to the first position is determined from the mapping relationship represented by the preset optical model, and the second position represents the position of the predicted pixel artifact caused by the pixel to be displayed; Based on the first energy intensity of the pixel to be displayed, a second energy intensity corresponding to the first energy intensity is determined from the mapping relationship represented by the preset optical model. The second energy intensity represents the energy intensity of the predicted pixel artifact caused by the pixel to be displayed. The second energy distribution information of the predicted artifacts corresponding to the image to be displayed is determined based on the second position and second energy intensity of the predicted pixel artifacts caused by each of the plurality of pixels to be displayed in the image to be displayed.

3. The display method according to claim 2, characterized in that, The step of determining the third energy distribution information of the target image mask corresponding to the image to be displayed based on the first energy distribution information and the second energy distribution information includes: Based on the first energy intensity included in the first energy distribution information, the third energy intensity of the target image mask is determined, and the third energy intensity is positively correlated with the first energy intensity; Based on the second position included in the second energy distribution information, the third position of the target image mask is determined; The third energy distribution information of the target image mask is determined based on the third energy intensity and the third position.

4. The display method according to claim 3, characterized in that, The step of controlling the near-eye display device to display the target image mask based on the third energy distribution information includes: At least at the third position, a target image mask with a third energy intensity is displayed.

5. The display method according to claim 1, characterized in that, After determining the third energy distribution information of the target image mask corresponding to the image to be displayed based on the first energy distribution information and the second energy distribution information, the method further includes: In response to a change in at least one of the ambient energy distribution information of the environment where the near-eye display device is located and the device temperature information of the near-eye display device, the third energy distribution information is adjusted to obtain the adjusted third energy distribution information; The step of controlling the near-eye display device to display the target image mask based on the third energy distribution information includes: Based on the adjusted third energy distribution information, the near-eye display device is controlled to display the target image mask.

6. The display method according to claim 5, characterized in that, The environmental energy distribution information includes environmental energy intensity; the adjustment of the third energy distribution information in response to changes in the environmental energy distribution information to obtain adjusted third energy distribution information includes: In response to the change in the ambient energy intensity, the third energy intensity included in the third energy distribution information is adjusted to obtain the adjusted third energy intensity, which is positively correlated with the change in the ambient energy intensity. The adjustment of the third energy distribution information in response to changes in the device temperature information, to obtain adjusted third energy distribution information, includes: In response to the change in the device temperature information, the third position included in the third energy distribution information is adjusted to obtain the adjusted third position, which is positively correlated with the change in the device temperature information.

7. The display method according to any one of claims 1 to 6, characterized in that, The display method further includes: Control the near-eye display device to display a preset calibration image; The near-eye display device is scanned across the entire field of view using a preset scanning device to obtain an energy distribution image corresponding to the preset calibration image. The energy distribution image is used to characterize the mapping relationship between the preset calibration image and the preset artifacts corresponding to the preset calibration image. Based on the energy distribution image, a preset optical model of the near-eye display device is established.

8. A display device for a near-eye display system, characterized in that, The display device includes: An energy distribution acquisition module is used to acquire first energy distribution information of the image to be displayed by the near-eye display device; The first energy distribution determination module is used to determine the second energy distribution information of the predicted artifact corresponding to the image to be displayed based on the first energy distribution information and a preset optical model of the near-eye display device. The second energy distribution determination module is used to determine the third energy distribution information of the target image mask corresponding to the image to be displayed based on the first energy distribution information and the second energy distribution information. The display module is configured to control the near-eye display device to display the image to be displayed based on the first energy distribution information, and to control the near-eye display device to display the target image mask based on the third energy distribution information, so that the target image mask is displayed as the background noise of the image to be displayed.

9. A near-eye display device, characterized in that, The near-eye display device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, when executing the computer program, implement the steps of the display method of the near-eye display device as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the display method of the near-eye display device as described in any one of claims 1 to 7.

Images