Image display method for head-mounted display, construction system, and computing control unit

Abstract

The application discloses an image display method, a construction system and a computing control unit of a head-mounted display, and is applied to the image field. The first to-be-displayed image is corrected through a pre-correction network model, the visual quality of the image displayed by the first display module is improved, geometric distortion, spherical aberration, field curvature and other defects of the image seen by the user are reduced, the visual fatigue of the user is reduced, and the immersion of the VR experience is improved. Moreover, in the process in which the user uses the head-mounted display, training data is collected in real time online, and the pre-correction network model is trained based on the collected training data, the training process of the pre-correction network model is simplified, and the training efficiency of the pre-correction network model is improved. Moreover, the training data in various actual use scenes can be continuously used in an online mode, the types of the training data used for training the pre-correction network model are enriched, and the accuracy of the pre-correction network model is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing, specifically to an image display method, construction system, and computing control unit for a head-mounted display. Background Technology

[0002] With the introduction of low-cost virtual reality (VR) devices, VR technology is rapidly changing the way we work, interact, and socialize. Among them, head-mounted displays (HMDs) are one of the most well-known VR devices, providing users with a vivid and immersive visual experience through high-performance head tracking technology and real-time rendering based on graphics processing units (GPUs).

[0003] However, current head-mounted displays are often limited by size, making it impossible to use multi-lens elements to improve visual acuity and reduce aberrations. As a result, the visual quality of existing head-mounted displays is often low, and the images seen by users may be significantly distorted. Summary of the Invention

[0004] This application provides an image display method, construction system, and computing control unit for a head-mounted display, which can solve the problem that the visual quality of existing head-mounted displays is often low, and the images seen by the user may be significantly distorted. The technical solution includes:

[0005] On one hand, a method for displaying images using a head-mounted display is provided. The head-mounted display integrates a pre-calibration network model, an analog display network model, and a first display module. The method includes:

[0006] The head-mounted display is controlled to be in a first image display mode, the first image display mode including:

[0007] Get the first image to be displayed;

[0008] The first image to be displayed is input into the pre-correction network model to obtain the pre-corrected image;

[0009] The pre-corrected image is input into the first display module for display.

[0010] The pre-corrected image is input into the simulation display network model to obtain the first predicted image;

[0011] The parameters of the pre-correction network model are adjusted based on the difference between the first predicted image and the first image to be displayed.

[0012] Optionally, the method further includes:

[0013] The calibration of the simulated display network model is triggered based on preset conditions;

[0014] The pre-corrected image is input into the simulated display network model to obtain the first predicted image, including:

[0015] The pre-corrected image is input into the most recently calibrated simulation display network model to obtain the first predicted image.

[0016] Optionally, the calibration of the simulated display network model can be triggered based on preset conditions, including:

[0017] When preset conditions are met, the head-mounted display will switch to the second image display mode, which includes:

[0018] Obtain the second image to be displayed;

[0019] The second image to be displayed is input into the first display module for display.

[0020] The current simulation display network model is calibrated;

[0021] The method also includes:

[0022] After the current analog display network model has been calibrated, the head-mounted display will be switched to the first image display mode.

[0023] Optionally, the preset conditions include at least one of the following: the number of times the current simulation display network model has been used reaches a first threshold; the difference between the first predicted image and the first image to be displayed is greater than a second threshold.

[0024] Optionally, the head-mounted display also includes a first human-eye camera; calibrating the current analog display network model, including:

[0025] The first captured image is obtained by capturing the second image to be displayed on the first display module using a first human eye-like camera.

[0026] The first captured image is post-processed to obtain the first processed image;

[0027] The second image to be displayed is input into the current simulation display network model to obtain the second predicted image;

[0028] Based on the difference between the second predicted image and the first processed image, the parameters of the current simulation display network model are adjusted.

[0029] Optionally, based on the difference between the second predicted image and the first processed image, the parameters of the current simulation display network model are adjusted, including:

[0030] Calculate the loss values ​​of the second predicted image and the first processed image, and determine whether the loss values ​​exceed the third threshold;

[0031] If the judgment result is yes, then adjust the parameters of the current simulation display network model, and then repeat the steps of the second image display mode until the judgment result is no;

[0032] If the result is negative, the calibration of the current simulation display network model ends.

[0033] Optionally, the method further includes:

[0034] The third threshold is adaptively updated based on the loss values ​​of the second predicted image and the first processed image from history.

[0035] Optionally, the first bionic eye camera is rotatably constrained to the head-mounted display, which also includes an eye-tracking module. Before obtaining the first captured image by capturing a second image to be displayed on the first display module using the first bionic eye camera, the method further includes:

[0036] The eye-tracking module collects the user's current eye data.

[0037] Determine the user's current gaze point based on the user's current eye data;

[0038] Rotate the first human-eye camera so that its shooting direction is facing the user's current gaze point.

[0039] Optionally, the head-mounted display may also include a support frame, to which the first human-eye camera is rotatably constrained;

[0040] The bracket is movable and constrained to the head-mounted display, and the bracket can move to a first position and a second position; in the radial direction of the first display module, the first position is far away from the optical axis of the first display module, and the second position is adjacent to the optical axis of the first display module;

[0041] Furthermore, when the head-mounted display is in the first image display mode, the support is in the first position;

[0042] With the head-mounted display in the second image display mode, the stand is in the second position.

[0043] Optionally, the loss values ​​of the second predicted image and the first processed image are obtained by weighted summation of the loss values ​​of the gaze region and the loss values ​​of the non-gaze region;

[0044] Among them, the first weight of the gaze region is greater than the second weight of the non-gaze region.

[0045] Optionally, the simulated display network model is obtained by training the initial simulated display network model using a training dataset. The training dataset includes sample images and their corresponding second-processed images. The second-processed images are obtained by displaying the sample images on the screen, acquiring images from the simulated human eye, and post-processing.

[0046] Optionally, the training process of the simulated network model is completed on the head-mounted display and before the head-mounted display is in the first image display mode;

[0047] Alternatively, the training process of the simulated display network model may occur before the simulated display network model is integrated into the head-mounted display.

[0048] On the other hand, a system for constructing a training dataset for simulating a network model is provided, including:

[0049] Base;

[0050] The second display module is located on the base;

[0051] The second human-eye-like camera is constrained to the base in a way that allows it to move.

[0052] The control module is used for:

[0053] Acquire a sample image and input the sample image into the second display module for display;

[0054] Obtain a preset simulated gaze direction, rotate the second human eye camera to the preset simulated gaze direction, and capture the image displayed on the second display module through the second human eye camera to obtain the second captured image;

[0055] The second captured image is post-processed to obtain the second processed image.

[0056] In another aspect, a head-mounted display is provided, wherein there are two sets of first display modules and first human-eye cameras in the head-mounted display, which are respectively used for the left and right eyes of the user, and each set of first display modules and first human-eye cameras independently executes the image display method of the head-mounted display described above.

[0057] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image display method for a head-mounted display described above.

[0058] In another aspect, a computing control unit is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image display method of the head-mounted display described above.

[0059] In summary, the embodiments of this application provide an image display method, construction system, and computing control unit for a head-mounted display. In this method, a first image to be displayed is corrected through a pre-correction network model, which improves the visual quality of the image displayed by the first display module and reduces defects such as geometric distortion, spherical aberration, and field curvature of the image seen by the user, thereby reducing the user's visual fatigue and enhancing the immersiveness of the VR experience.

[0060] Furthermore, in the first image display mode of the head-mounted display, a first predicted image is obtained by inputting the pre-calibrated image into the simulated display network model. The parameters of the pre-calibrated network model are then adjusted based on the difference between the first predicted image and the first image to be displayed. This allows for real-time online acquisition of training data (i.e., the first predicted image and the first image to be displayed) while the user is using the head-mounted display. The pre-calibrated network model is then trained based on this acquired training data, avoiding the problem of excessive workload caused by pre-collecting large amounts of training data. This simplifies the training process and improves the efficiency of pre-calibrated network model training. Moreover, the online continuous use of training data from various real-world application scenarios enriches the types of training data available for the pre-calibrated network model, ensuring its accuracy.

[0061] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the structure of a head-mounted display provided in an embodiment of this application;

[0063] Figure 2 This is a flowchart of an image display method for a head-mounted display provided in an embodiment of this application;

[0064] Figure 3 This is a flowchart of another image display method for a head-mounted display provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of another head-mounted display provided in an embodiment of this application;

[0066] Figure 5 This is a flowchart illustrating the calibration of a current analog display network model, as provided in an embodiment of this application.

[0067] Figure 6 This is a schematic diagram of the structure of a computing control unit provided in an embodiment of this application. Detailed Implementation

[0068] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0069] The following explains the terminology used in the embodiments of this application:

[0070] “Difference” includes one or more of the following: structural similarity index (SSIM) difference, color difference, distortion difference, and optical blur difference.

[0071] "Loss values ​​of the second predicted image and the first processed image": including one or more of the following: SSIM loss, color consistency loss, and geometric distortion constraint.

[0072] "First threshold": a fixed value stored in advance, or a value that can be dynamically updated.

[0073] "Second threshold": a fixed value stored in advance, or a value that can be dynamically updated.

[0074] "Third threshold": a fixed value stored in advance, or a value that can be dynamically updated.

[0075] "First position": The optical axis furthest from the first display module.

[0076] "Second position": The optical axis adjacent to the first display module.

[0077] While head-mounted displays offer numerous advantages, their manufacturing process requires striking a balance between image quality and device size and production costs. Current head-mounted displays are often constrained by size limitations, preventing the use of multi-lens elements to improve visual clarity and reduce aberrations. Consequently, the visual quality of existing head-mounted displays often falls short of optimal standards, and users may experience significant geometric distortion, spherical aberration, and field-of-view curvature defects. These aberrations and imperfections can lead to visual fatigue and diminish the immersive experience of VR.

[0078] Figure 1 This is a schematic diagram of the structure of a head-mounted display provided in an embodiment of this application, as shown below. Figure 1 As shown, the head-mounted display 100 may integrate a computing control unit 10, a pre-calibration network model 20, an analog display network model 30, and a first display module 40, which may include a display screen 41 and a projection lens 42.

[0079] The computing control unit 10 is a highly integrated chip. The projection lens 42 can be a pancake lens, and the lens in the projection lens 42 can be aspherical, spherical, or Fresnel, etc.

[0080] refer to Figure 1 The pre-calibration network model 20 is used to calibrate the image to be displayed, resulting in a pre-calibrated image. By calibrating the image to be displayed through the pre-calibration network model 20, aberrations can be corrected purely in software, allowing users to comfortably enjoy the VR experience without wearing glasses. This improves the visual quality of the image displayed on the head-mounted display 100, reduces defects such as geometric distortion, spherical aberration, and field curvature in the image seen by the user, thereby reducing visual fatigue and enhancing the immersiveness of the VR experience.

[0081] refer to Figure 1 The first display module 40 is used to display a pre-calibrated image so as to project the pre-calibrated image onto the human eye. After the pre-calibrated image is displayed on the display screen 41, it is projected onto the display screen 41 through the projection lens 42.

[0082] refer to Figure 1 The simulated display network model 30 is used to simulate the process of the first display module 40 displaying an image. Since blurring is introduced during the imaging process of the projection lens 42, the simulated display network model 30 can simulate the blurring process introduced by the projection lens 42 by simulating the image display process of the first display module 40. The pre-correction network model 20 can be trained through this simulated display network model 30.

[0083] In this embodiment, both the pre-calibration network model 20 and the simulation display network model 30 are neural network models, such as deep learning network models. These deep network models can be convolutional neural networks (CNN), Transformer networks, etc. Using deep learning network models can better fit the entire nonlinear imaging process.

[0084] The basic architecture of the pre-calibration network model 20 and the simulated display network model 30 can take various forms. For example, this architecture can include the encoder-decoder architecture in a convolutional neural network, a cascaded architecture, or a self-attention mechanism. Specifically, the main architecture of the pre-calibration network model 20 can be a multi-scale encoder-decoder cascaded network. The input to this network is the image to be displayed and a downsampled image of the image to be displayed. The convolutional modules in the network's encoder-decoder perform downsampling, and the deconvolutional modules perform upsampling. The final output is a pre-calibrated image of the same size as the input image to be displayed. The basic architecture of the simulated display network model 30 can be a Unet network architecture.

[0085] Figure 2 This is a flowchart of an image display method for a head-mounted display provided in an embodiment of this application. This method can be applied to... Figure 1 The computing control unit 10 shown is as follows: Figure 2 As shown, the method may include:

[0086] Step 201: Control the head-mounted display to be in the first image display mode.

[0087] refer to Figure 2 The first image display mode may include:

[0088] Step 2011: Obtain the first image to be displayed.

[0089] The computing control unit 10 can render the current virtual scene to obtain a first image to be displayed, which is called an ideal digital image.

[0090] Step 2012: Input the first image to be displayed into the pre-calibration network model to obtain the pre-calibrated image.

[0091] refer to Figure 1 After acquiring the first image to be displayed, the calculation control unit 10 can input the first image to be displayed into the pre-correction network model 20 to obtain the pre-correction image output by the pre-correction network model 20.

[0092] Step 2013: Input the pre-corrected image into the first display module for display.

[0093] The calculation control unit 10 can input the pre-corrected image to the first display module 40 for display.

[0094] refer to Figure 1 The calculation control unit 10 can input the pre-calibrated image into the display screen 41 to display the pre-calibrated image on the display screen 41, and then project the pre-calibrated image displayed on the display screen 41 through the projection lens 42.

[0095] Step 2014: Input the pre-corrected image into the simulation display network model to obtain the first predicted image.

[0096] refer to Figure 1 The calculation control unit 10 can simultaneously input the pre-corrected image into the simulation display network model 30 to obtain the first predicted image. That is, steps 2013 and 2014 can be executed simultaneously.

[0097] Step 2015: Adjust the parameters of the pre-correction network model based on the difference between the first predicted image and the first image to be displayed.

[0098] After obtaining the first predicted image, the calculation control unit 10 can adjust the parameters of the pre-correction network model 20 based on the difference between the first predicted image and the first image to be displayed.

[0099] refer to Figure 1 The head-mounted display 100 may also include a difference calculation module 50. The calculation control unit 10 can input the first predicted image and the first image to be displayed into the difference calculation module 50 to obtain the difference between the first predicted image and the first image to be displayed output by the difference calculation module 50, and then adjust the parameters in the pre-correction network model 20 based on the difference.

[0100] The difference can be the loss value between the first preset image and the first image to be displayed, and the difference can include one or more of the following: SSIM difference, color difference, distortion difference, and optical blur difference.

[0101] It should be noted that when the head-mounted display 100 is in the first image display mode and the pre-calibrated network model 20 is being trained online in real time, the computation control unit 10 can turn off the calibration of the analog display network model 30.

[0102] In summary, the embodiments of this application provide an image display method for a head-mounted display. In this method, a first image to be displayed is corrected by a pre-correction network model, which improves the visual quality of the image displayed by the first display module and reduces defects such as geometric distortion, spherical aberration, and field curvature of the image seen by the user, thereby reducing the user's visual fatigue and enhancing the immersiveness of the VR experience.

[0103] Furthermore, in the first image display mode of the head-mounted display, a first predicted image is obtained by inputting the pre-calibrated image into the simulated display network model. The parameters of the pre-calibrated network model are then adjusted based on the difference between the first predicted image and the first image to be displayed. This allows for real-time online acquisition of training data (i.e., the first predicted image and the first image to be displayed) while the user is using the head-mounted display. The pre-calibrated network model is then trained based on this acquired training data, avoiding the problem of excessive workload caused by pre-collecting large amounts of training data. This simplifies the training process and improves the efficiency of pre-calibrated network model training. Moreover, the online continuous use of training data from various real-world application scenarios enriches the types of training data available for the pre-calibrated network model, ensuring its accuracy.

[0104] In this embodiment, the analog display network model 30 is turned off in the first image display mode, and the pre-correction network model 20 is turned off in the second image display mode. Figure 3 This is a flowchart of another image display method for a head-mounted display provided in an embodiment of this application. This method can be applied to... Figure 1 or Figure 4 The computing control unit 10 in the head-mounted display 100 shown. Figure 3 As shown, the method may include:

[0105] Step 301: Control the head-mounted display to be in the first image display mode.

[0106] When the training of the simulated display network model 30 is completed on the head-mounted display, the computation control unit 10 can control the head-mounted display 100 to be in a first image display mode when a target condition is detected. The target condition may include: the accuracy of the simulated display network model 30 is greater than a threshold. That is, after the simulated display network model 30 is trained to a stable state, online training of the pre-calibrated network model 20 can be achieved.

[0107] like Figure 3 As shown, the first image display mode may include:

[0108] Step 3011: Obtain the first image to be displayed.

[0109] The computing control unit 10 can render the current virtual scene to obtain a first image to be displayed, which is called an ideal digital image.

[0110] Step 3012: Input the first image to be displayed into the pre-calibration network model to obtain the pre-calibrated image.

[0111] refer to Figure 1 After acquiring the first image to be displayed, the calculation control unit 10 can input the first image to be displayed into the pre-correction network model 20 to obtain the pre-correction image output by the pre-correction network model 20.

[0112] Step 3013: Input the pre-corrected image into the first display module for display.

[0113] refer to Figure 1 The calculation control unit 10 can input the pre-corrected image to the first display module 40 for display.

[0114] refer to Figure 1 The calculation control unit 10 can input the pre-calibrated image into the display screen 41 to display the pre-calibrated image on the display screen 41, and then project the pre-calibrated image displayed on the display screen 41 through the projection lens 42.

[0115] Step 3014: Input the pre-corrected image into the simulation display network model that has recently completed calibration to obtain the first predicted image.

[0116] refer to Figure 1The calculation control unit 10 can also simultaneously input the pre-corrected image into the most recently calibrated analog display network model 30 to obtain the first predicted image output by the most recently calibrated analog display network model 30. That is, steps 3013 and 3014 can be executed simultaneously.

[0117] Step 3015: Adjust the parameters of the pre-correction network model based on the difference between the first predicted image and the first image to be displayed.

[0118] After obtaining the first predicted image, the calculation control unit 10 can adjust the parameters of the pre-correction network model 20 based on the difference between the first predicted image and the first image to be displayed.

[0119] refer to Figure 1 The calculation control unit 10 can input the first predicted image and the first image to be displayed into the difference calculation module 50 to obtain the difference between the first predicted image and the first image to be displayed output by the difference calculation module 50, and then adjust the parameters in the pre-correction network model 20 based on the difference.

[0120] The difference can be the loss value between the first preset image and the first image to be displayed, and the difference can include one or more of the following: SSIM difference, color difference, distortion difference, and optical blur difference.

[0121] It should be noted that when the head-mounted display 100 is in the first image display mode and the pre-calibrated network model 20 is being trained online in real time, the computation control unit 10 can turn off the calibration of the analog display network model 30.

[0122] Step 302: Check whether the preset conditions are met.

[0123] The computation control unit 10 can trigger the calibration of the simulation display network model 30 based on preset conditions.

[0124] Optionally, the computation control unit 10 can detect whether preset conditions are met, wherein the preset conditions may include at least one of the following: the current number of times the simulated display network model 30 has been used reaches a first threshold; the difference between the first predicted image and the first image to be displayed is greater than a second threshold. The first threshold may be a fixed value pre-stored in the computation control unit 10, or the first threshold may be a dynamically updated value. The second threshold may be a fixed value pre-stored in the computation control unit 10, or the second threshold may be a dynamically updated value.

[0125] When the preset conditions are met, the calculation control unit 10 can determine that the simulation effect of the simulated display network model 30 is poor, and therefore can execute step 303. When the preset conditions are not met, the calculation control unit 10 can determine that the simulation effect of the simulated display network model 30 is good, and therefore can continue to execute step 3011.

[0126] Step 303: Switch the head-mounted display to the second image display mode.

[0127] When preset conditions are met, the calculation control unit 10 can switch the head-mounted display 100 to the second image display mode. At this time, the pre-calibration network model 20 can be set to the off state. (The reference...) Figure 3 The second image display mode may include:

[0128] Step 3031: Obtain the second image to be displayed.

[0129] The computing control unit 10 can render the current virtual scene to obtain a second image to be displayed, which can be called an ideal digital image.

[0130] Step 3032: Input the second image to be displayed into the first display module for display.

[0131] refer to Figure 4 The calculation control unit 10 can input the second image to be displayed to the first display module 40 for display.

[0132] refer to Figure 4 The computing control unit 10 can input the second image to be displayed into the display screen 41 to display the second image to be displayed on the display screen 41, and then project the second image to be displayed on the display screen 41 through the projection lens 42.

[0133] Step 3033: Calibrate the current simulation display network model.

[0134] After the computing control unit 10 inputs the second image to be displayed to the first display module 40 for display, it can also simultaneously calibrate the current analog display network model 30. That is, the process of displaying the second image to be displayed (including steps 3032 and 3033) and the process of calibrating the current analog display network model (step 3033) are executed simultaneously.

[0135] refer to Figure 4 The head-mounted display 100 also includes a first bionic eye camera 60 and an eye-tracking module 70. The first bionic eye camera 60 is rotatably mounted on the head-mounted display 100.

[0136] refer to Figure 5Calibrling the current analog display network model 30 may include:

[0137] Step A1: Collect the user's current eye data through the eye-tracking module.

[0138] The computing control unit 10 can collect the user's current eye data through the eye tracking module 70. The current eye data may include data such as the pupil, cornea, and the current rotation angle of the human eye.

[0139] refer to Figure 4 The eye-tracking module 70 can collect the user's current eye data and transmit the user's current eye data to the computing control unit 10.

[0140] Step A2: Determine the user's current gaze point based on the user's current eye data.

[0141] After the calculation control unit 10 collects the user's current eye data, it can determine the user's current gaze point based on the current eye data.

[0142] Step A3: Rotate the first human eye camera so that its shooting direction is facing the user's current gaze point.

[0143] refer to Figure 4 The head-mounted display 100 may also include a motor control module 80. After obtaining the user's current gaze point, the calculation control unit 10 can determine the pose control parameters of the first human eye camera 60 based on the user's current gaze point, and control the motor control module 80 to rotate the first human eye camera 60 so that the shooting direction of the first human eye camera 60 is towards the user's current gaze point.

[0144] refer to Figure 4 The calculation control unit 10 can input the pose control parameters into the motor control module 80, so that the motor control module 80 rotates the first humanoid eye camera 60 so that the shooting direction of the first humanoid eye camera 60 is facing the user's current gaze point based on the pose control parameters.

[0145] The motor control module 80 controls the first and second rotation components in the first humanoid eye camera 60 based on the pose control parameters, so that the first and second rotation components drive the simulated eyeball in the first humanoid eye camera 60 to perform pitch and horizontal rotation, realize the spatial rotation of the eyeball to simulate the user's current gaze point, and then control the eye module device in the first humanoid eye camera 60 to take pictures to simulate the image seen by the user at the current gaze point.

[0146] Step A4: Take a picture of the second image to be displayed on the first display module using the first human eye camera to obtain the first captured image.

[0147] After the first bionic eye camera 60 is oriented towards the user's current gaze point, referencing Figure 4 The calculation control unit 10 can capture the second image to be displayed on the first display module 40 by the first human eye camera 60 to obtain the first captured image.

[0148] Step A5: Post-process the first captured image to obtain the first processed image.

[0149] Post-processing of the first captured image is mainly used to correct defects caused by the first human-eye camera 60.

[0150] The defects of the first human-eye camera 60 may include distortion and color difference in the images captured by the first human-eye camera 60 due to its inherent limitations.

[0151] refer to Figure 4 The head-mounted display 100 may also include a post-processing module 90. After acquiring the first captured image, the computing control unit 10 can input the first captured image into the post-processing module 90. The post-processing module 90 can perform post-processing on the first captured image to obtain a first processed image and transmit the first processed image to the computing control unit 10.

[0152] In some embodiments of this application, the post-processing module 90 can perform image distortion correction, color difference correction, and interpolation on the first captured image, thereby realizing post-processing of the first captured image.

[0153] Step A6: Input the second image to be displayed into the current simulation display network model to obtain the second predicted image.

[0154] refer to Figure 4 After obtaining the first processed image, the calculation control unit 10 can input the second image to be displayed into the current analog display network model 30 to obtain the second predicted image. This second image to be displayed corresponds to the first captured image; that is, the second image to be displayed input into the current analog display network model 30 and the second image to be displayed that was displayed by the first display module 40 and then captured by the first human-eye camera are the same frame.

[0155] The calculation control unit 10 can execute step A6 while performing any of steps A1 to A5. Figure 5 Taking the example of the computing control unit 10 executing step A6 simultaneously with step A4, that is, while capturing the second image to be displayed on the first display module 40 through the first human-eye camera 60, the second image to be displayed is input into the current analog display network model 30.

[0156] Step A7: Calculate the loss values ​​of the second predicted image and the first processed image.

[0157] The computation control unit 10 can calculate the loss value of the second predicted image and the first processed image, wherein the loss value may include one or more of SSIM loss, color consistency loss and geometric distortion constraint.

[0158] refer to Figure 4 The head-mounted display 100 may also include a loss calculation module 91. The calculation control unit 10 can input the second predicted image and the first processed image into the loss calculation module 91 to obtain the loss value.

[0159] In this embodiment, the user's current eye data is further used to weight the gaze region during the calculation of the loss values ​​of the second predicted image and the first processed image. The loss values ​​of the second predicted image and the first processed image are obtained by weighted summation of the loss values ​​of the gaze region and the non-gaze region. That is, different weights are assigned to the gaze region and the non-gaze region; for example, a first weight is assigned to the gaze region and a second weight to the non-gaze region. The first weight is greater than the second weight, and the sum of the first and second weights is 1. Therefore, the loss value of the second predicted image and the first processed image = loss value of the gaze region × first weight + loss value of the non-gaze region × second weight. This allows the loss function to have a greater "penalty" or "contribution" in the gaze region, thereby guiding the model to prioritize optimization of the gaze region during training.

[0160] Understandably, the loss values ​​of the aforementioned first preset image and the first image to be displayed can also be combined with the differentiated weights of the gaze region and the non-gaze region to optimize the training of the pre-calibration network model 20. For example, the loss value of the first preset image and the first image to be displayed = loss value of the gaze region × third weight + loss value of the non-gaze region × fourth weight, where the third weight is greater than the fourth weight, and the sum of the third weight and the fourth weight is 1. The third weight and the first weight can be equal or unequal, and the fourth weight and the second weight can be equal or unequal.

[0161] Step A8: Determine whether the loss value exceeds the third threshold.

[0162] After obtaining the loss values ​​of the second predicted image and the first processed image, the calculation control unit 10 can determine whether the loss value exceeds a third threshold. The third threshold can be a fixed value pre-stored in the calculation control unit 10, or it can be a dynamically updated value.

[0163] If the judgment result is yes, the calculation control unit 10 can determine that the accuracy of the current analog display network model 30 is less than or equal to the accuracy threshold, that is, the current analog display network model 30 needs to be calibrated, so step A9 can be executed.

[0164] If the result of the judgment is negative when the second image display mode is executed for the first time, it can be determined that the accuracy of the current analog display network model 30 is greater than the accuracy threshold. Therefore, there is no need to calibrate the current analog display network model 30, and the head-mounted display 100 is switched to the first image display mode, that is, step 301 is executed again.

[0165] If the result is negative when executing the second image display mode for the first time, it can be determined that the accuracy of the current analog display network model 30 is greater than the accuracy threshold after adjusting the parameters of the current analog display network model 30 at least once. Therefore, the calibration of the current analog display network model 30 can be ended, and the head-mounted display 100 can be switched to the first image display mode, that is, step 301 is executed again. At this time, the pre-calibration network model 20 is re-enabled, and the calibration of the analog display network model 30 is turned off.

[0166] When the third threshold is a dynamically updated value, the calculation control unit 10 can adaptively update the third threshold based on the loss values ​​of the historical second predicted image and the first processed image. This adaptive update typically converges according to the calibration process; for example, the third threshold may be higher in the early stages of calibration, and then lowered as the calibration progresses, such as after 100 calibrations.

[0167] Step A9: Adjust the parameters of the current simulation display network model.

[0168] If the judgment result is yes, the calculation control unit 10 can adjust the parameters of the current analog display network model 30, and then repeat the steps of the second image display mode until the judgment result is no.

[0169] The computation control unit 10 adjusts the parameters of the current simulation display network model 30 based on the difference between the second predicted image and the first processed image by executing steps A7 to A9.

[0170] It should be noted that when the head-mounted display 100 is in the second image display mode and the current analog display network model 30 is being calibrated, the calculation control unit 10 can turn off the use of the pre-calibration network model 20, that is, it does not use the pre-calibration network model 20 to calibrate the second image to be displayed.

[0171] Under preset conditions, the computation control unit 10 can switch the head-mounted display 100 to the second image display mode. Upon first detecting that the loss value between the second predicted image and the first processed image exceeds a third threshold, it determines that the accuracy of the current simulated display network model 30 requires fine-tuning or incremental learning. Therefore, it can disable the use of the pre-calibrated network model 20 and adjust the parameters of the current simulated display network model 30. Then, it repeats the steps of the second image display mode until the judgment result is negative. Afterward, it can disable the calibration of the current simulated display network model 30 and switch the head-mounted display 100 to the first image display mode to restart online training of the pre-calibrated network model 20.

[0172] In this embodiment, the head-mounted display 100 may further include a support, to which the first human-eye camera 60 is rotatably constrained.

[0173] The support is movable and constrained to the head-mounted display 100, and the support can move to a first position and a second position. In the radial direction of the first display module 40, the first position is away from the optical axis of the first display module 40, and the second position is adjacent to the optical axis of the first display module 40.

[0174] When the head-mounted display 100 is in a first image display mode, the support is in a first position. When the head-mounted display 100 is in a second image display mode, the support is in a second position.

[0175] Since the first human-eye camera 60 does not need to capture the first image to be displayed displayed by the first display module 40 in the first image display mode, the bracket needs to be in the first position so that the first human-eye camera 60 is located away from the optical axis of the first display module 40, thereby reducing the obstruction of the user's field of view.

[0176] Since the first human eye camera 60 needs to capture the second image to be displayed displayed by the first display module 40 in the second image display mode, the bracket needs to be located in the second position so that the first human eye camera 60 is located near the optical axis of the first display module 40, reducing the difference between the position of the first human eye camera 60 and the human eye and the image acquisition direction. By being close to the optical axis, high-precision image acquisition is achieved, so that the first captured image is closer to the image seen by the human eye.

[0177] The computation control unit 10 determines the user's current gaze point based on the user's current eye data, and determines the pose control parameters of the first humanoid eye camera 60 based on the user's current gaze point. The pose control parameters are used to control the first humanoid eye camera 60 to rotate and / or translate (e.g., first control the support located at the first position to move and / or rotate to the second position, and then control the first humanoid eye camera 60 to rotate so that its acquisition direction faces the human eye's gaze point) to simulate the eye movement of the human eye when using a VR device, ensuring that the acquired first captured image is close to the image seen by the human eye, thereby ensuring the robustness of the trained current simulation display network.

[0178] It should be noted that since the shooting process of the first bionic eye camera 60 is not a differentiable backpropagation process, the loss value between the shooting result of the first bionic eye camera 60 and the second image to be displayed cannot be used to calibrate the pre-calibration network model 20. Therefore, it is necessary to introduce the simulated display network model 30 to help train the pre-calibration network model 20.

[0179] In some embodiments of this application, the simulated display network model 30 may be obtained by training an initial simulated display network model using a training dataset. The training dataset includes sample images and their corresponding second processed images. The second processed images are obtained by displaying the sample images on the screen, acquiring images at the simulated human eye, and post-processing.

[0180] The training process of the simulated display network model 30 is completed on the head-mounted display 100, and before the head-mounted display 100 is in the first image display mode. Alternatively, the training process of the simulated display network model 30 is completed before the simulated display network model 30 is integrated into the head-mounted display 100.

[0181] In some embodiments of this application, if the training process of the simulated display network model 30 is completed on the head-mounted display 100, and before the head-mounted display 100 is in the first image display mode, the computation control unit 10 can display the sample image through the first display module 40, and then have the first bionic eye camera 60 in the head-mounted display 100 acquire the second captured image, and perform post-processing on the second captured image to obtain the second processed image. The specific implementation process of displaying the sample image through the first display module 40, acquiring the second captured image through the first bionic eye camera 60, and performing post-processing on the second captured image to obtain the second processed image can be referred to steps A1 to A5 above.

[0182] If the training process of the current simulated display network model 30 is performed before the simulated display network model 30 is integrated into the head-mounted display 100, then the second processed image is the image obtained by the second human-eye camera in the construction system after the sample image is displayed by the second display module in the construction system, and then the second captured image is post-processed.

[0183] This application provides a system for constructing a training dataset for a simulated display network model 30. This system may include:

[0184] Base

[0185] The second display module is located on the base.

[0186] The control module is used for:

[0187] A sample image is acquired and input into the second display module for display. A preset simulated gaze direction is acquired, and the second human eye camera is rotated to the preset simulated gaze direction. The image displayed on the second display module is captured by the second human eye camera to obtain the second captured image. The second captured image is then post-processed to obtain the second processed image.

[0188] The control module can pre-store multiple simulated gaze directions, thereby obtaining multiple sample images and a second processed image corresponding to the multiple sample images.

[0189] Post-processing of the second captured image is mainly used to correct defects caused by the second bionic eye camera.

[0190] The drawbacks of the second bionic eye camera may include distortion and color difference in the images captured by the camera due to its inherent limitations.

[0191] The second bionic eye camera is movably constrained to the base. The second bionic eye camera can move to a third position. This third position is located radially on the optical axis of the second display module. The second bionic eye camera is in this third position during the display of a sample image on the second display module.

[0192] In this embodiment, sample images and simulated gaze directions can be randomly generated to form a small number of data pairs. After the parameters of the simulated display network model 30 are trained to a stable state, the online training mode can be started, that is, execution can begin from step 101 or step 301 in this embodiment.

[0193] During the online calibration of the simulated display network model 30, the eye-tracking module 70 is activated, and the simulated display network model 30 is continuously optimized through new scenes and eye-tracking information, ultimately converging the parameters of the simulated display network model 30 to a range that can be directly forward-inferred.

[0194] The comparative example optimizes the pre-calibrated network model 20 by using only offline collected datasets for training. The final effect of the pre-calibrated network model 20 in this comparative example is limited to the effect after training on the offline collected dataset, and further optimization is not possible. From the final metrics, its performance is lower than that of this embodiment, and the time cost of collecting a large amount of offline data for training is greatly increased.

[0195] For example, by acquiring 100 frames of images at a resolution of 3840×3552 and performing online optimization, the PSNR was improved by approximately 5.1% and the SSIM by approximately 4.8%.

[0196] In terms of final results, this embodiment surpasses the comparative embodiment after extensive real-time data collection and optimization. From a time cost perspective, this embodiment, using online training, incurs significantly less time than the offline training time required for collecting large amounts of data in the comparative embodiment.

[0197] The method provided in this embodiment solves the problem of excessive time cost in training the pre-calibration network model 20 in the current head-mounted display 100. At the same time, by directly constructing data pairs between the image to be displayed and the captured image in real time, it stably represents the image mapping relationship from the first display module 40 to the human eye.

[0198] The pre-calibration network model 20 and the simulated display network model 30 are trained online using the method provided in the embodiments of this application. There is no need to collect a large number of data pairs in advance. Therefore, for VR devices with a large field of view, there is no need to consider the blurring changes of the eyeball at different rotation angles, which would lead to the problem of excessive data collection. Furthermore, the pre-calibration network model 20 and the simulated display network model 30 can be optimized in real time when the user starts using the head-mounted display 100.

[0199] In summary, the embodiments of this application provide an image display method for a head-mounted display, a construction system, and a computing control unit. In this method, a first image to be displayed is corrected through a pre-correction network model, which improves the visual quality of the image displayed by the first display module and reduces defects such as geometric distortion, spherical aberration, and field curvature of the image seen by the user, thereby reducing the user's visual fatigue and enhancing the immersiveness of the VR experience.

[0200] Furthermore, in the first image display mode of the head-mounted display, a first predicted image is obtained by inputting the pre-calibrated image into the simulated display network model. The parameters of the pre-calibrated network model are then adjusted based on the difference between the first predicted image and the first image to be displayed. This allows for real-time online acquisition of training data (i.e., the first predicted image and the first image to be displayed) while the user is using the head-mounted display. The pre-calibrated network model is then trained based on this acquired training data, avoiding the problem of excessive workload caused by pre-collecting large amounts of training data. This simplifies the training process and improves the efficiency of pre-calibrated network model training. Moreover, the online continuous use of training data from various real-world application scenarios enriches the types of training data available for the pre-calibrated network model, ensuring its accuracy.

[0201] This application provides a head-mounted display, which has two sets of first display modules and first human-eye cameras, respectively used for the left and right eyes of the user. Each set of first display modules and first human-eye cameras independently executes the image display method of the head-mounted display described above.

[0202] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image display method for a head-mounted display described in the above embodiments.

[0203] Figure 6 This is a schematic diagram of the structure of a computing control unit provided in an embodiment of this application, as shown below. Figure 6 As shown, the computing control unit 10 may include a memory 11, a processor 12, and a computer program stored in the memory 11 and executable on the processor 12. When the processor 12 executes the computer program, it implements the image display method of the head-mounted display described in the above embodiments.

[0204] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0205] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0206] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0207] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0208] Furthermore, the terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this invention can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this invention, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly specified in the embodiments.

[0209] In this invention, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific implementation.

[0210] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for displaying images on a head-mounted display, characterized in that, The head-mounted display integrates a pre-calibration network model, a simulated display network model, a first display module, and a first human-eye-like camera. The method includes: The head-mounted display is controlled to be in a first image display mode, the first image display mode including: Get the first image to be displayed; The first image to be displayed is input into the pre-correction network model to obtain a pre-corrected image; The pre-corrected image is input into the first display module for display. The pre-corrected image is input into the simulated display network model to obtain the first predicted image; Based on the difference between the first predicted image and the first image to be displayed, the parameters of the pre-correction network model are adjusted; The calibration of the simulated display network model is triggered based on preset conditions; The pre-corrected image is input into the simulated display network model to obtain a first predicted image, including: The pre-corrected image is input into the most recently calibrated simulation display network model to obtain the first predicted image; The calibration of the simulated display network model triggered based on preset conditions includes: When the preset conditions are met, the head-mounted display is switched to a second image display mode, which includes: Obtain the second image to be displayed; The second image to be displayed is input into the first display module for display. The current simulation display network model is calibrated; The method further includes: After the current simulation display network model has been calibrated, the head-mounted display is switched to the first image display mode; The current simulation display network model is calibrated, including: The first captured image is obtained by capturing the second image to be displayed on the first display module using the first human eye-like camera. The first captured image is post-processed to obtain a first processed image; The second image to be displayed is input into the current simulation display network model to obtain the second predicted image; Based on the difference between the second predicted image and the first processed image, the parameters of the current simulation display network model are adjusted.

2. The method according to claim 1, characterized in that, The preset conditions include at least one of the following: the current number of times the simulated display network model is used reaches a first threshold; the difference between the first predicted image and the first image to be displayed is greater than a second threshold.

3. The method according to claim 1, characterized in that, The step of adjusting the parameters of the current simulated display network model based on the difference between the second predicted image and the first processed image includes: Calculate the loss value of the second predicted image and the first processed image, and determine whether the loss value exceeds a third threshold; If the judgment result is yes, then adjust the parameters of the current simulated display network model, and then repeat the steps of the second image display mode until the judgment result is no; If the result is negative, the calibration of the current simulated display network model ends.

4. The method according to claim 3, characterized in that, The method further includes: The third threshold is adaptively updated based on the loss values ​​of the second predicted image and the first processed image from history.

5. The method according to claim 1, characterized in that, The first bionic eye camera is rotatably constrained to the head-mounted display, which further includes an eye-tracking module. Before capturing a second image to be displayed on the first display module using the first bionic eye camera to obtain the first captured image, the method further includes: The eye-tracking module collects the user's current eye data. Based on the user's current eye data, determine the user's current gaze point; Rotate the first bionic eye camera so that its shooting direction is toward the user's current gaze point.

6. The method according to claim 5, characterized in that, The head-mounted display also includes a support frame, to which the first human eye camera is rotatably constrained; The bracket is movably constrained to the head-mounted display, and the bracket is movable to a first position and a second position; in the radial direction of the first display module, the first position is far from the optical axis of the first display module, and the second position is adjacent to the optical axis of the first display module; Furthermore, when the head-mounted display is in the first image display mode, the support is located in the first position; When the head-mounted display is in the second image display mode, the support is in the second position.

7. The method according to claim 6, characterized in that, The loss values ​​of the second predicted image and the first processed image are obtained by weighted summation of the loss values ​​of the gaze region and the loss values ​​of the non-gaze region; The first weight of the gaze region is greater than the second weight of the non-gaze region.

8. The method according to any one of claims 1 to 7, characterized in that, The simulated display network model is obtained by training the initial simulated display network model using a training dataset. The training dataset includes sample images and their corresponding second processed images. The second processed images are obtained by displaying the sample images on the screen, acquiring images from the simulated human eye, and post-processing.

9. The method according to claim 8, characterized in that, The training process of the simulated display network model is completed on the head-mounted display and before the head-mounted display is in the first image display mode; Alternatively, the training process of the simulated display network model may be performed before the simulated display network model is integrated into the head-mounted display.

10. A head-mounted display, characterized in that, The head-mounted display has two sets of first display modules and first bionic eye cameras, which are respectively used for the left and right eyes of the user. Each set of first display modules and first bionic eye cameras independently executes the image display method of the head-mounted display according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the image display method of the head-mounted display according to any one of claims 1-9.

12. A computing control unit, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the image display method for a head-mounted display according to any one of claims 1-9.

Images