Image display method, apparatus, AR display system, and electronic device

Abstract

The application relates to an image display method and device, an AR display system and an electronic device, and is applied to an AR display system, the AR display system comprising a diffractive optical waveguide. The image display method comprises the following steps: acquiring an image to be displayed; inputting the image to be displayed into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is generated according to at least a projection inverse function; the projection inverse function is used to indicate an image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive optical waveguide; and generating an image display result based on the target compensation image. Through the application, the problem of low accuracy of image display is solved, and a display optimization method of a high-precision AR display system is realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image display methods, apparatus, AR display systems and electronic devices. Background Technology

[0002] Augmented Reality (AR) display systems based on diffractive waveguides, such as AR glasses, use RGB light sources emitted by the display to propagate through the waveguides. Due to the inconsistent transmission paths and energy losses of the three colors of light in the waveguides, the composite image of the three colors of light transmitted to the human eye will have uneven brightness and color, resulting in a display quality that cannot meet people's visual needs. The low-quality display effect can easily cause eye fatigue. These quality problems are inherent to the curves of diffractive waveguides, and current designs that rely solely on diffractive waveguides cannot solve the problems of uneven brightness and color in low-quality displays.

[0003] In related technologies, methods to improve the image display effect of AR display systems typically consider the device perspective, improving the transmission optical efficiency parameters of waveguides to balance the transmission loss of RGB light. However, unavoidable processing errors exist in the actual fabrication of diffractive waveguides, causing discrepancies between the actual effect and the design effect, failing to fundamentally solve the problem. Alternatively, software-based methods can be used to calibrate and compensate for the brightness of the displayed image. However, these methods all rely on statistical prior assumptions. If the actual ambient lighting and displayed image do not conform to these assumptions, the correction methods will fail, resulting in low image display accuracy.

[0004] Currently, no effective solution has been proposed to address the issue of low image display accuracy in related technologies. Summary of the Invention

[0005] This application provides an image display method, apparatus, AR display system, and electronic device to at least address the problem of low image display accuracy in related technologies.

[0006] In a first aspect, embodiments of this application provide an image display method applied to an AR display system, the AR display system including a diffractive waveguide, the method comprising:

[0007] Get the image to be displayed;

[0008] The image to be displayed is input into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide;

[0009] The image display result is generated based on the target compensation image.

[0010] In some embodiments, prior to acquiring the image to be displayed, the method further includes:

[0011] Obtain the image to be trained and the ground truth image corresponding to the image to be trained;

[0012] The image to be trained is input into a neural network model containing the inverse projection function to obtain a network-predicted image; wherein the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different;

[0013] The loss function is obtained based on the network predicted image and the ground truth image. The neural network model is then optimized based on the loss function to obtain the model optimization parameters.

[0014] The image conversion model is generated after iterative training based on the optimized parameters of the model.

[0015] In some embodiments, acquiring the image to be displayed includes:

[0016] Obtain the original image;

[0017] An eye-tracking algorithm is used to obtain the gaze point of the original image, the gaze region in the original image is obtained based on the gaze point, and the gaze region is determined as the image to be displayed.

[0018] In some embodiments, the AR display system further includes a microdisplay; the generation of the image display result based on the target compensation image includes:

[0019] Obtain a physical brightness mapping model that matches the microdisplay;

[0020] The target compensation image is processed by adjusting the pixel brightness using the physical brightness mapping model to obtain the target adjusted image;

[0021] The image display result is generated based on the target adjusted image.

[0022] In some embodiments, obtaining a physical brightness mapping model that matches the microdisplay includes:

[0023] Obtain the current-physical brightness calibration result of the microdisplay, and obtain the brightness loss correction result based on the current-physical brightness calibration result;

[0024] Based on the brightness loss correction results, the image-physical brightness calibration results of the microdisplay are obtained, and the physical brightness mapping model is generated based on the image-physical brightness calibration results.

[0025] In some embodiments, the inverse projection function includes hardware device optical parameters, hardware device optical functions, and the surface bidirectional reflection distribution function of the diffractive waveguide.

[0026] Secondly, embodiments of this application provide an image display device applied to an AR display system, the AR display system including a diffractive waveguide, and the device including: an acquisition module, a compensation module, and a generation module;

[0027] The acquisition module is used to acquire the image to be displayed;

[0028] The compensation module is used to input the image to be displayed into a pre-trained image conversion model to output a target compensated image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensated image, and the image mapping relationship is at least associated with the diffractive waveguide;

[0029] The generation module is used to generate an image display result based on the target compensation image.

[0030] Thirdly, embodiments of this application provide an AR display system, the system comprising: a diffractive waveguide and a main control device;

[0031] The main control device is used to acquire the image to be displayed;

[0032] The main control device inputs the image to be displayed into a pre-trained image conversion model to output a target compensation image; wherein, the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide;

[0033] The main control device generates an image display result based on the target compensation image.

[0034] Fourthly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image display method as described in the first aspect above.

[0035] Fifthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the image display method as described in the first aspect above.

[0036] Compared to related technologies, the image display method, apparatus, AR display system, and electronic device provided in this application are applied to an AR display system, which includes a diffractive waveguide. The method includes: acquiring an image to be displayed; inputting the image to be displayed into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is trained and generated at least based on an inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide; generating an image display result based on the target compensation image, thus solving the problem of low image display accuracy and realizing a high-precision AR display system display optimization method.

[0037] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a hardware structure block diagram of a terminal for an image display method according to an embodiment of this application;

[0040] Figure 2 This is a flowchart of an image display method according to an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of an AR glasses displaying an image according to an embodiment of this application;

[0042] Figure 4 This is a schematic diagram illustrating the training of an image conversion model according to an embodiment of this application;

[0043] Figure 5 This is a preferred flowchart of an image display method according to an embodiment of this application;

[0044] Figure 6 This is a structural block diagram of an image display device according to an embodiment of this application;

[0045] Figure 7 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0047] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for an image display method according to an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0050] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the gesture recognition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0052] This embodiment provides an image display method applied to an AR display system, which includes a diffractive waveguide. Figure 2 This is a flowchart of an image display method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0053] Step S220: Obtain the image to be displayed.

[0054] The image to be displayed refers to the true image input to the AR display system for display. Specifically, the AR display system mainly includes a diffractive waveguide and a microdisplay. Taking AR glasses using a diffractive waveguide as an example, Figure 3 This is a schematic diagram of an AR glasses displaying an image according to an embodiment of this application, such as... Figure 3 As shown, the AR glasses need to display a true image projected by the optical engine; however, the image seen by the human eye, as a microdisplay within the digital signal input terminal, corresponds to the distorted, uncompensated image captured by the image acquisition device, resulting in problems such as uneven brightness and color, and distortion. It is understood that this image acquisition device can be, but is not limited to, various binocular cameras, PTZ cameras, or other devices used for image acquisition.

[0055] Step S240: The image to be displayed is input into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide.

[0056] In some embodiments, the aforementioned inverse projection function includes hardware device optical parameters, hardware device optics, and the surface bidirectional reflection distribution function of the diffracting waveguide.

[0057] Specifically, the image display process of the above AR display system can be abstracted into an imaging model obtained by modeling the imaging of the hardware and the propagation of light, as shown in Equation 1:

[0058]

[0059] Where x represents the input image of the aforementioned microdisplay; p represents the geometric optical intrinsic and extrinsic parameters of the aforementioned optomechanical system; g represents the global illumination irradiance of the aforementioned AR display system; s represents the spectral reflectance characteristics of the diffractive waveguide; c represents the composite intrinsic and extrinsic parameters of the aforementioned image acquisition device; π p (x, p) represents the composite geometric projection and radiative transfer function of the optical engine; π s Represents the bidirectional reflection distribution function of the diffracting waveguide surface; π c Represents the composite geometric projection and radiative transfer function of the image acquisition device; This refers to the output image displayed by the AR display system. In other words, in this embodiment, the hardware devices mentioned in the optical parameters and optical functions of the aforementioned hardware devices include optical engines, diffractive waveguides, image acquisition devices, and projectors, etc.; corresponding to Formula 1 above, the optical parameters of the hardware devices include parameters such as p, c, g, and s, and the optical functions of the hardware devices include π. p (x, p), π c Therefore, in this embodiment, the imaging and light propagation of the hardware device in the AR display system are modeled based on the above theory, and then abstracted into the imaging model shown in Formula 1, so that the subsequent modeling process based on the imaging model can optimize the display quality at the hardware and software system level.

[0060] It should be noted that the image in Formula 1 above... This includes various issues related to the image quality of optomechanical projection displays based on diffractive waveguides; and by inverting the imaging model shown in Equation 1, the inverse projection function can be obtained, which in turn yields an optomechanical projection input image x. * That is, the compensation image of the input image x, then using this supplementary image x * By compensating for various issues in the image quality of diffractive waveguide optical engines and projection displays, it becomes possible to ensure that the images captured by the image acquisition device are consistent with the images perceived by the viewer of the AR display system. However, since Equation 1 is a non-explicit expression, its inverse problem is difficult to solve using conventional numerical optimization algorithms. Furthermore, for each diffractive waveguide AR display system with different hardware parameters, there is generally a fixed functional mapping relationship between the uncompensated output image and the distorted output image due to the influence of display quality. This functional mapping relationship is consistent with the image mapping relationship between the image to be displayed and the target compensation image. Moreover, influenced by at least one of the hardware devices in the AR display system, such as the diffractive waveguide and the optomechanical system, the corresponding image mapping relationship differs between AR display systems with different hardware devices; that is, the image mapping relationship is related to the diffractive waveguide and other hardware devices. Therefore, deep learning algorithms can be used to train the image conversion model with a large number of image samples to obtain a nonlinear fitting solution to this inverse problem. This solves the problem of not being able to directly solve the image mapping relationship between the image to be displayed and the target compensation image using conventional numerical algorithms. Thus, by inputting the image to be displayed into the fully trained image conversion model, the target compensation image x, which matches the nonlinear fitting solution, can be output. * It is understandable that, according to the inverse projection function shown in Formula 2 below, the above target compensation image x is input. * This can compensate for various problems in the image quality of optomechanical projection displays based on diffractive waveguides, thereby improving image quality; Formula 2 is shown below:

[0061] π c (πs (π p (x * Formula 2: p), g), s), c) = x

[0062] Step S260: Generate an image display result based on the target compensation image.

[0063] After the target compensation image is obtained through step S240, the target compensation image can be processed by the AR display system to generate an image display result so that the viewer of the AR display system can view it with their eyes.

[0064] In related technologies, the input and output luminance and chrominance values ​​are usually measured and obtained. Based on these measurements, a look-up table (LUT) correction calculation or a transformation matrix calculation is performed. However, such methods only compensate for the display quality in specific scenarios and cannot adapt to the displayed image content, resulting in low accuracy of image display.

[0065] In this embodiment, through steps S220 to S260, the image to be displayed is input into a pre-trained image conversion model to obtain a target compensated image and then displayed. This enables image compensation adapted to the content of the displayed image to be achieved based on the image conversion model generated by the inverse projection function. An end-to-end diffractive waveguide AR display system image quality correction model is established, and the display image quality is optimized at the hardware and software system level. This greatly improves the problem of uneven brightness and color of the displayed image in the AR display system, avoids the distortion of the displayed image caused by physical loss when light propagates in the diffractive waveguide, solves the problem of low image display accuracy, and realizes a high-precision display optimization method for the AR display system.

[0066] In some embodiments, before acquiring the image to be displayed, the image display method further includes the following steps:

[0067] Step S211: Obtain the image to be trained and the ground truth image corresponding to the image to be trained.

[0068] Specifically, images displayed by the AR display system can be pre-collected in a dark room using an image acquisition device to obtain an image training set; the image training set includes the images collected by the image acquisition device, i.e., the images to be trained, as well as the ground truth images displayed on the microdisplay of the AR display system.

[0069] Step S212: Input the image to be trained into a neural network model containing the inverse projection function to obtain the network prediction image; wherein, the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different.

[0070] The aforementioned inverse projection function can be constructed from the first and second optical feature networks to realize the inverse optomechanical projection process. Both the first and second optical feature networks can employ geometric correction networks, photometric compensation networks, or radiometric correction networks, and other optical feature networks besides the first and second can also be used to construct the inverse optomechanical projection process, as long as the optical correction angles of each optical feature network are different. It is understood that each of the aforementioned optical feature networks can be set as a UNet network or a Generative Adversarial Network (GAN) neural network model, and the initial values ​​of each optical feature network can be randomly set. By using optical feature networks with different optical correction angles, the inverse optomechanical projection process can be trained from different aspects. Compared to training from various optical correction angles using the same neural network, this scheme can greatly alleviate the pressure on the correction network, effectively improve the accuracy and efficiency of training, and thus improve the accuracy and efficiency of image display.

[0071] Step S213: Obtain the loss function based on the network-predicted image and the ground truth image; optimize the neural network model based on the loss function to obtain the model optimization parameters; generate the iteratively trained image conversion model based on the model optimization parameters.

[0072] Specifically, Figure 4 This is a schematic diagram illustrating the training of an image conversion model according to an embodiment of this application, as shown below. Figure 4 As shown, during the model training phase, the uncompensated image captured by the image acquisition device is input as the training image to the inverse optomechanical projection process composed of the aforementioned optical feature networks; wherein, this inverse optomechanical projection process can be expressed as follows: The inverse optomechanical projection process outputs the image predicted by the network. A loss function, typically L1 or L2 loss, is established between the predicted and ground truth images. The neural network model is then optimized based on this loss function to obtain its optimized parameters, which in turn determine the optimized parameters of the neural network model for the inverse optomechanical projection process. By iteratively training the neural network model with a large number of image samples, an image conversion model with increasingly accurate parameters can be trained, further improving the processing accuracy of the image conversion model.

[0073] Through steps S211 to S213, the network predicts the image by inputting the image to be trained into a neural network model containing the inverse projection function, and optimizes the loss function obtained from the network predicts the image and the ground truth image, thereby improving the accuracy and efficiency of image display.

[0074] In some embodiments, the acquisition of the image to be displayed further includes the following steps: acquiring an original image; using an eye-tracking algorithm to acquire a gaze point for the original image, acquiring a gaze region in the original image based on the gaze point, and determining the gaze region as the image to be displayed. The original image refers to the complete ground truth image input to the AR display system to be displayed. Eye tracking is the process of measuring eye movement; the most important event in eye tracking is determining where a human or animal is looking, i.e., the gaze point. For example, an eye-tracking algorithm based on the pupil-corneal reflex method can be used to calculate the pupil center position and corneal center position, and the gaze point can be calculated based on these positions. Then, based on the calculation result of the gaze point, a gaze region viewed by the human eye can be cropped from the original image, and this gaze region can be used as the image to be displayed, in order to obtain a compensated image for a local area of ​​the original image.

[0075] In related technologies, the gaze point is obtained by combining the acquired measurement values ​​with eye-tracking technology, and calibration parameters are selected based on the gaze point's attention area. However, this method still does not consider the content of the displayed image and cannot adapt to the diversity of displayed image content. This application, through the above embodiments, utilizes an eye-tracking algorithm to obtain the gaze point area, thereby achieving precise selection of calibration parameters, reducing the model's computational load from seconds to milliseconds, effectively accelerating model computation, and enabling the above method to be applied in embedded systems with demanding computational speed requirements.

[0076] In some embodiments, the AR display system further includes a microdisplay; the generation of the image display result based on the target compensation image further includes the following steps:

[0077] Step S261: Obtain the physical brightness mapping model that matches the microdisplay.

[0078] It should be noted that Formula 2 above can compensate for various problems in the optical-mechanical projection display image quality of diffractive waveguides. However, the current digital representation of images is based on an 8-bit depth. Prioritizing the brightness adjustment range of microdisplays, it is necessary to map and transform the digital representation of the image to expand its brightness adjustment range based on the physical brightness adjustment range of the microdisplay. Therefore, the physical brightness mapping model can be obtained by physically calibrating the selected microdisplay.

[0079] In some embodiments, obtaining the physical brightness mapping model matching the microdisplay further includes the following steps: obtaining the current-physical brightness calibration result of the microdisplay, and obtaining a brightness loss correction result based on the current-physical brightness calibration result; obtaining the image-physical brightness calibration result of the microdisplay based on the brightness loss correction result, and generating the physical brightness mapping model based on the image-physical brightness calibration result. Specifically, the relationship curve between the driving current and physical brightness of the microdisplay is calibrated to obtain the current-physical brightness calibration result, so as to correct the system brightness loss, thereby improving the accuracy of the physical brightness mapping model construction through brightness loss correction. Then, the relationship between the values ​​in the grayscale range of the image after system correction and the physical brightness is calibrated to obtain the above-mentioned image-physical brightness calibration result, and finally, the physical brightness-image value mapping model of each pixel position of the microdisplay in the system is obtained.

[0080] Step S262: Use the physical brightness mapping model to perform pixel brightness adjustment processing on the target compensation image to obtain the target adjustment image; generate the image display result based on the target adjustment image.

[0081] Specifically, the above-mentioned physical brightness mapping model is used to calculate the brightness adjustment value of each display pixel on the microdisplay, and the pixel brightness of the microdisplay is adjusted to obtain the target adjusted image after brightness and color correction, thereby generating the above-mentioned image display result.

[0082] In related technologies, the output image of the diffracted waveguide display is typically acquired by a camera and paired with the input image to train a deep learning model. This method only considers the image perspective and does not take into account the physical brightness of the display device, resulting in low image display accuracy. However, the embodiments of this application, through steps S261 to S262, establish a physical brightness mapping model by combining the physical brightness characteristics of the micro-display device, such as adjustment range and typical brightness. This physical brightness mapping model is then used to adjust the pixel brightness of the target compensation image, resulting in better brightness and color uniformity in the image observed by the human eye. This further improves the display quality of the AR display system and enhances the accuracy of image display.

[0083] The embodiments of the present invention will be described in detail below with reference to practical application scenarios. Figure 5 This is a preferred flowchart of an image display method according to an embodiment of this application, such as... Figure 5 As shown, the image display method includes the following steps:

[0084] Step S501: Obtain the input original image, and use eye-tracking technology to select a region from the original image to obtain the image to be displayed.

[0085] Step S502: The image to be displayed is compensated using an image conversion model to obtain the target compensated image.

[0086] Step S503: Input the target compensation image into the physical brightness mapping model for pixel brightness adjustment processing to obtain the target adjustment image.

[0087] Step S504: Input the target adjustment image into the microdisplay.

[0088] In step S505, the light emitted by the microdisplay based on the target adjusted image is transmitted through the projection optics system and the diffraction waveguide for viewing by the human eye of the AR display system.

[0089] Through the above steps S501 to S505, the calibration of the microdisplay's physical brightness and current parameters, the deep learning modeling of the waveguide imaging model, the image correspondence, and the eye-tracking acquisition of the gaze point are comprehensively processed to achieve display quality compensation that is adapted to the display image content and the diversity of gaze points. This improves the display quality of the AR display system based on diffractive waveguides and can improve the problem of brightness and color uniformity.

[0090] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0091] This embodiment also provides an image display device applied to an AR display system, which includes a diffractive waveguide. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0092] Figure 6 This is a structural block diagram of an image display device according to an embodiment of this application, such as... Figure 6As shown, the device includes: an acquisition module 62, a compensation module 64, and a generation module 66; the acquisition module 62 is used to acquire an image to be displayed; the compensation module 64 is used to input the image to be displayed into a pre-trained image conversion model to output a target compensated image; wherein, the image conversion model is trained and generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensated image, and the image mapping relationship is at least associated with the diffractive waveguide; the generation module 66 is used to generate an image display result based on the target compensated image.

[0093] Through the above embodiments, the image to be displayed is input into a pre-trained image conversion model through the compensation module 64 to obtain the target compensated image, and the image display result is generated and displayed through the generation module 66. In this way, image compensation adapted to the content of the displayed image can be realized based on the image conversion model generated by the inverse projection function. This avoids the distortion of the displayed image caused by physical loss when light propagates in the diffraction waveguide, solves the problem of low image display accuracy, and realizes a display optimization device for a high-precision AR display system.

[0094] In some embodiments, the image display device further includes a training module; the training module is used to acquire a training image and a ground truth image corresponding to the training image; the training module inputs the training image into a neural network model containing the inverse projection function to obtain a network prediction image; wherein the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different; the training module obtains a loss function based on the network prediction image and the ground truth image, optimizes the neural network model based on the loss function, and obtains model optimization parameters; the training module generates the iteratively trained image conversion model based on the model optimization parameters.

[0095] In some embodiments, the acquisition module 62 is further configured to acquire an original image; the acquisition module 62 uses an eye-tracking algorithm to acquire a gaze point for the original image, acquires a gaze region in the original image based on the gaze point, and determines the gaze region as the image to be displayed.

[0096] In some embodiments, the AR display system further includes a microdisplay; the generation module 66 is also used to obtain a physical brightness mapping model that matches the microdisplay; the generation module 66 uses the physical brightness mapping model to perform pixel brightness adjustment processing on the target compensation image to obtain a target adjustment image; the generation module 66 generates the image display result based on the target adjustment image.

[0097] In some embodiments, the generation module 66 is further configured to obtain the current-physical brightness calibration result of the microdisplay, and obtain the brightness loss correction result based on the current-physical brightness calibration result; the generation module 66 obtains the image-physical brightness calibration result of the microdisplay based on the brightness loss correction result, and generates the physical brightness mapping model based on the image-physical brightness calibration result.

[0098] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0099] This embodiment also provides an AR display system, which includes: a diffractive waveguide and a main control device; the main control device is used to acquire an image to be displayed; the main control device inputs the image to be displayed into a pre-trained image conversion model to output a target compensation image; wherein, the image conversion model is trained and generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide; the main control device generates an image display result based on the target compensation image.

[0100] Through the above embodiments, the image to be displayed is input into a pre-trained image conversion model by the main control device to obtain the target compensated image and display it. In this way, the image compensation adapted to the content of the displayed image can be realized based on the image conversion model generated by the inverse projection function. This avoids the distortion of the displayed image caused by the physical loss that occurs when light propagates in the diffraction waveguide, solves the problem of low image display accuracy, and realizes a display optimization system for a high-precision AR display system.

[0101] In some embodiments, the main control device is further configured to acquire a training image and a ground truth image corresponding to the training image; the main control device inputs the training image into a neural network model containing the inverse projection function to obtain a network prediction image; wherein the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different; the main control device obtains a loss function based on the network prediction image and the ground truth image, optimizes the neural network model based on the loss function, and obtains model optimization parameters; the main control device generates the iteratively trained image conversion model based on the model optimization parameters.

[0102] In some embodiments, the main control device is further configured to acquire an original image; the main control device uses an eye-tracking algorithm to acquire a gaze point for the original image, acquires a gaze region in the original image based on the gaze point, and determines the gaze region as the image to be displayed.

[0103] In some embodiments, the AR display system further includes a microdisplay; the main control device is also used to acquire a physical brightness mapping model that matches the microdisplay; the main control device uses the physical brightness mapping model to perform pixel brightness adjustment processing on the target compensation image to obtain a target adjustment image; the main control device generates the image display result based on the target adjustment image.

[0104] In some embodiments, the main control device is further configured to obtain the current-physical brightness calibration result of the microdisplay and obtain the brightness loss correction result based on the current-physical brightness calibration result; the main control device obtains the image-physical brightness calibration result of the microdisplay based on the brightness loss correction result and generates the physical brightness mapping model based on the image-physical brightness calibration result.

[0105] In some embodiments, the aforementioned inverse projection function includes hardware device optical parameters, hardware device optical functions, and the surface bidirectional reflection distribution function of the diffracting waveguide.

[0106] In some embodiments, a computer device is provided, which may be a terminal. Figure 7 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application, such as... Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image display method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0107] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0108] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0109] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0110] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0111] S1, Obtain the image to be displayed.

[0112] S2, the image to be displayed is input into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide.

[0113] S3, Generate the image display result based on the target compensation image.

[0114] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0115] Furthermore, in conjunction with the image display methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the image display methods in the above embodiments.

[0116] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0117] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image display method, characterized in that, Applied to an AR display system, the AR display system including a diffractive waveguide, the method includes: Obtain the image to be trained and the ground truth image corresponding to the image to be trained; The image to be trained is input into a neural network model containing an inverse projection function to obtain a network-predicted image; wherein, the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different; the inverse projection function includes hardware device optical parameters, hardware device optical functions, and the surface bidirectional reflection distribution function of the diffractive waveguide; The loss function is obtained based on the network predicted image and the ground truth image. The neural network model is then optimized based on the loss function to obtain the model optimization parameters. Based on the optimized parameters of the model, an iteratively trained image conversion model is generated; Get the image to be displayed; The image to be displayed is input into a pre-trained image conversion model to output a target compensation image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide; The image display result is generated based on the target compensation image.

2. The image display method according to claim 1, characterized in that, The process of acquiring the image to be displayed includes: Obtain the original image; An eye-tracking algorithm is used to obtain the gaze point of the original image, the gaze region in the original image is obtained based on the gaze point, and the gaze region is determined as the image to be displayed.

3. The image display method according to claim 1, characterized in that, The AR display system further includes a microdisplay; the image display result generated based on the target compensation image includes: Obtain a physical brightness mapping model that matches the microdisplay; The target compensation image is processed by adjusting the pixel brightness using the physical brightness mapping model to obtain the target adjusted image; The image display result is generated based on the target adjusted image.

4. The image display method according to claim 3, characterized in that, The process of obtaining a physical brightness mapping model that matches the microdisplay includes: Obtain the current-physical brightness calibration result of the microdisplay, and obtain the brightness loss correction result based on the current-physical brightness calibration result; Based on the brightness loss correction results, the image-physical brightness calibration results of the microdisplay are obtained, and the physical brightness mapping model is generated based on the image-physical brightness calibration results.

5. An image display device, characterized in that, The device is applied to an AR display system, which includes a diffractive waveguide, and comprises a training module, an acquisition module, a compensation module, and a generation module. The training module is used to acquire the image to be trained and the ground truth image corresponding to the image to be trained. The training module inputs the image to be trained into a neural network model containing an inverse projection function to obtain a network-predicted image; wherein, the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different; the inverse projection function includes hardware device optical parameters, hardware device optical functions, and the surface bidirectional reflection distribution function of the diffractive waveguide; The training module obtains a loss function based on the network-predicted image and the ground truth image, and optimizes the neural network model based on the loss function to obtain model optimization parameters. The training module generates an iteratively trained image conversion model based on the model optimization parameters. The acquisition module is used to acquire the image to be displayed; The compensation module is used to input the image to be displayed into a pre-trained image conversion model to output a target compensated image; wherein the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensated image, and the image mapping relationship is at least associated with the diffractive waveguide; The generation module is used to generate an image display result based on the target compensation image.

6. An AR display system, characterized in that, The system includes: a diffractive waveguide and a main control device; The main control device is used to acquire the image to be trained and the ground truth image corresponding to the image to be trained; The main control device inputs the image to be trained into a neural network model containing an inverse projection function to obtain a network-predicted image; wherein, the inverse projection function includes a first optical feature network and a second optical feature network, and the optical correction angles between the first optical feature network and the second optical feature network are different; the inverse projection function includes hardware device optical parameters, hardware device optical functions, and the surface bidirectional reflection distribution function of the diffractive waveguide; The main control device obtains a loss function based on the network prediction image and the ground image, and optimizes the neural network model based on the loss function to obtain model optimization parameters. The main control device generates an iteratively trained image conversion model based on the model optimization parameters. The main control device is used to acquire the image to be displayed; The main control device inputs the image to be displayed into a pre-trained image conversion model to output a target compensation image; wherein, the image conversion model is generated at least based on the inverse projection function; the inverse projection function is used to indicate the image mapping relationship between the image to be displayed and the target compensation image, and the image mapping relationship is at least associated with the diffractive waveguide; The main control device generates an image display result based on the target compensation image.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the image display method according to any one of claims 1 to 4.

8. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the image display method according to any one of claims 1 to 4 when it is run.

Images