Non-invasive, non-wearable method and system for refractive correction
By employing a non-invasive image preprocessing method and jointly training an image pre-correction model and a human eye imaging distortion simulation model, personalized refractive correction on a display screen is achieved, solving the problems of high risk and inconvenience in existing technologies and improving image quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-06-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing visual aberration correction technologies are high-risk, costly, subject to individual differences, and inconvenient. They also cannot update changes in optical path parameters in a timely manner and require specific head-mounted display devices.
Using a non-invasive method, this method utilizes artificial intelligence image generation and transformation algorithms. Through joint training of an image pre-correction model and a human eye imaging distortion simulation model, personalized refractive correction images are generated, enabling non-wearable refractive correction.
Refractive correction can be achieved directly on computer and mobile phone screens, avoiding invasive or wearable devices, improving image quality, and enhancing image contrast, brightness, and edge strength.
Smart Images
Figure CN118608406B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a technology in the field of image processing, specifically a non-invasive, non-wearable refractive correction method and system. Background Technology
[0002] Due to defects in the human eye's refractive system, there is a deviation between the wavefront of the actual image and the ideal wavefront. This deviation usually leads to a decrease in image quality and a negative impact on vision. Existing visual aberration correction technologies still have problems such as high risk, high cost, individual variability, and inconvenience. For example, they require specific head-mounted display devices and only adjust for the eye's refractive power, and cannot be updated in a timely manner when optical path parameters change. Summary of the Invention
[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a non-invasive, non-wearable refractive correction method and system. By utilizing current artificial intelligence image generation and transformation algorithm models, a position-related transformation is pre-applied to the image, thereby pre-counteracting the imaging process caused by imperfections in the refractive system, enabling myopic patients to ultimately see a clear image.
[0004] This invention is achieved through the following technical solution:
[0005] This invention relates to a non-invasive, non-wearable refractive correction method. In the offline stage, ray tracing processing based on defocus blur is performed according to the user's refractive parameters to generate a set of blurred and distorted images corresponding to the original image after optical path distortion. This set is used to pre-train a human eye imaging distortion simulation model. Then, an image pre-correction model is constructed to generate refractive corrected images from the original image, and it is jointly trained with the human eye imaging distortion simulation model after parameters are frozen. In the online stage, the jointly trained image pre-correction model is used to generate a refractive corrected image for the user's personalized display in real time based on the clear original image.
[0006] The real-time generation preferably involves training the image pre-correction model using a multi-model interpolation method in the offline stage to obtain the image pre-correction model for the user under different viewing distances and angles. In the online stage, the image pre-correction model for the current position is constructed using the previous image pre-correction models based on the user's current distance and angle, so as to ensure that the corresponding personalized refractive correction image can be generated at each position.
[0007] The joint training refers to: using the pre-trained human eye imaging distortion simulation model with frozen parameters, generating a reconstructed image based on the refractive corrected image output by the image pre-correction model, and calculating the MSELoss between the reconstructed image and the original image as the training objective function to train the image pre-correction model.
[0008] The user's refractive parameters are obtained by the user viewing the display screen at a fixed distance and angle and by using ophthalmic measurement equipment, including corneal curvature, axial length, refractive index, axial length, width, lens curvature, anteroposterior diameter, and refractive index parameters.
[0009] The fixed distance and angle mentioned above can be achieved by mechanical means, but not limited to, bracket fixers, robotic arm screen fixers, etc.
[0010] The ophthalmic measurement equipment described herein employs, but is not limited to, corneal topography, OCT / OCTA, etc.
[0011] The aforementioned ray tracing processing based on defocus blur is implemented using, but is not limited to, the techniques described in Real-Time Rendering 3rd.
[0012] The human eye imaging distortion simulation model is a deep neural network based on adaptive convolutional kernel weight allocation. This deep neural network uses a set of M convolutional kernels for each convolutional layer. At the position of each pixel in the output feature of the convolutional layer, each convolutional kernel is weighted to obtain a specific convolutional kernel corresponding to that pixel position. This convolutional kernel is used to calculate the result value at that pixel position. The weight map is a learnable feature tensor. The formula for calculating the specific convolutional kernel at pixel position (i,j) is as follows: Deep neural networks, on the other hand, reconstruct the sharp original image into a simulated blurred and distorted image.
[0013] The pre-training process specifically involves: inputting a clear original image into the human eye imaging distortion simulation model, using the corresponding blurred distortion image obtained from the clear original image through ray tracing based on defocus blur as the model training target, and using MSE Loss as the loss function to supervise the training of the model.
[0014] The image pre-correction model is a deep neural network with the same structure as the human eye imaging distortion simulation model, which reconstructs the input clear original image into a refractive corrected image.
[0015] The joint training specifically involves: using the original clear image as input to the image pre-correction model, inputting the generated refractive corrected image into the human eye imaging distortion simulation model with frozen parameters, reconstructing the output image of the human eye imaging distortion simulation model into a clear original image as the training target, and using MSE Loss as the loss function to supervise the training of the model.
[0016] The multi-model interpolation method refers to: extracting personalized refractive parameters of the user using multiple sets of different distances and angles, and jointly training the corresponding image pre-correction models, using each combination of distances and angles as points to form a grid. During the online phase, when the user is at a certain position (x, y), the network parameters F1, F2, F3, and F4 of the image pre-correction model corresponding to the four points (x0, y0), (x1, y0), (x0, y1), and (x1, y1) of the grid vertex at that position are taken and bilinearly interpolated to construct the network parameters F of the image pre-correction model corresponding to that position, which is used to generate the refractive corrected image. Specifically:
[0017] This invention relates to a system for implementing the above method, comprising: a refractive parameter acquisition unit, a ray tracing unit, a human eye imaging distortion simulation model, an image pre-correction model network model, and a multi-model interpolation unit, wherein: the refractive parameter acquisition unit acquires the user's refractive parameters; the ray tracing unit uses the refractive parameters to construct a simulated blurred image based on the original clear image; the original clear image and the simulated blurred image are used as data pairs to participate in the training of the human eye imaging distortion simulation model and the image pre-correction model network model; finally, the image is processed using the image pre-correction model reconstructed by the multi-model interpolation unit to obtain a refractive corrected image, so that the human eye can directly see a clear image. Technical effect
[0018] This invention uses deep learning to preprocess images to achieve non-invasive, non-wearable refractive correction. Compared to existing technologies, it can be directly applied to computer and mobile phone displays without requiring any additional invasive or wearable devices. Attached Figure Description
[0019] Figure 1 This is a flowchart of the present invention;
[0020] Figure 2 This is a schematic diagram of the workflow of the present invention;
[0021] Figure 3 A schematic diagram of a simulation model of human eye imaging distortion;
[0022] Figure 4 This is a schematic diagram of an image pre-correction model;
[0023] Figure 5 This is a schematic diagram illustrating the effect of an example. Detailed Implementation
[0024] like Figure 1 As shown, this embodiment relates to a non-invasive, non-wearable refractive correction method, including:
[0025] Step 1: Obtain the user's refractive parameters using existing technology.
[0026] Step 2: Establish a simulation model N() for human eye imaging distortion using ray tracing, specifically including:
[0027] 2.1 Data Pair Construction: Based on the refractive parameters obtained in step 1, a simulated blurred image corresponding to the clear original image is generated using existing technology.
[0028] 2.2 Human Eye Refractive System Network Simulation: The image data pairs obtained in step 2.1 are learned through a deep neural network to simulate the human eye imaging and distortion model N of a patient with refractive errors, i.e., minimizing: Where: N is a parameterized simulation model of human eye imaging distortion.
[0029] like Figure 3 As shown, the human eye imaging distortion simulation model is a deep neural network based on adaptive convolutional kernel weight allocation, including: an encoder and a decoder, wherein: the encoder encodes the clear original image into high-dimensional image features, and the decoder decodes the high-dimensional image features into a predicted simulated blurred distortion image.
[0030] The encoder includes five convolutional modules and a max pooling layer. Each convolutional module has a structure of convolutional layer-batch processing layer-ReLU activation function-convolutional layer-batch processing layer-ReLU activation function. The max pooling layer reduces the length and width of the feature vector, ultimately obtaining a 16×16×1024 feature vector.
[0031] The decoder includes four upconvolutional modules, four residual convolutional layers, and one convolutional layer with a 1×1 kernel. Each upconvolutional module is an upsampling layer-convolutional layer-batch processing layer-ReLU activation function structure. Each upsampling layer concatenates the vector of the corresponding layer in the encoder with the vector obtained by the upconvolutional layer at the channel number level. Then, the residual convolutional layer restores the channel number. After the above steps are performed four times, a 256×256×64 vector is obtained. This vector is passed through the convolutional layer with a 1×1 kernel to finally obtain a 256×256×3 simulated blurred and distorted image.
[0032] All convolutional layers of the human eye imaging distortion simulation model incorporate prior knowledge of visual distortion, such as using dynamic convolution (position-dependent) and adaptive convolution kernel weight allocation methods to simulate different degrees of defocus in different imaging regions; multi-scale network construction is incorporated to perform pixel-level learning from coarse to fine.
[0033] The human eye imaging distortion simulation model is trained by obtaining a large amount of paired training data through random images and the ray tracing simulation unit in step 1, to obtain a parameterized model N() for image perception distortion.
[0034] Step 3: Establish an image pre-correction model F() based on the image. Specifically, before the human eye imaging distortion simulation model in Step 2, an image pre-correction model is set up to sharpen the blurred image. This ensures that the original sharp image, after passing through the image pre-correction model and the trained human eye imaging distortion simulation model, can be reconstructed into a sharp image. Since the human eye imaging distortion simulation model and the image pre-correction model are inverse functions of each other, i.e., Y = N(X), F(Y) = X, then N(F(X)) = X. Since the N() function is a simulation model for human eye imaging distortion, after processing the image using the F() function, the user can see a sharp image without wearing any devices.
[0035] like Figure 4 As shown, the image pre-correction model includes a deep neural network, wherein the deep neural network reconstructs the input clear original image into a refractive corrected image.
[0036] The image pre-correction model described above uses the following loss function for training: Specifically, during the training of the parameters of F(), the parameters of the human eye imaging distortion network N() are frozen, and only the parameters of the anti-distortion network F() are optimized.
[0037] Step 4: Reconstruct the image pre-correction model using a multi-model interpolation method. This includes extracting the user's personalized refractive parameters at different locations and training the corresponding image pre-correction models at those locations, referring to steps 1-3, so that these locations form a grid. At this stage, when the user is at a certain location (x, y), the parameters F1, F2, F3, and F4 of the image pre-correction model corresponding to the four points (x0, y0), (x1, y0), (x0, y1), and (x1, y1) of the grid vertex at that location are bilinearly interpolated to construct the corresponding image pre-correction model F. The newly constructed image pre-correction model F is then used to generate the refractive corrected image. The specific formula is as follows:
[0038] This method involved a total of three experiments, which are listed below.
[0039] Experiment 1 involved simulation testing on 200 typical images. The images were simulated with blurring and distortion based on the patient's refractive parameters. The peak signal-to-noise ratio (PSNR) of the blurred image was compared to the real image. Simultaneously, an anti-distortion network was added, and the PSNR of the image after correction by the refractive system was measured. The average simulation results for the 200 images were statistically analyzed. The average PSNR for images without anti-distortion was 22.85 ± 1.68, while the average PSNR for images with anti-distortion was 32.66 ± 2.11. The experiment showed that the images could be largely corrected back to their true values in the simulation system, demonstrating the effectiveness of the proposed method.
[0040] Experiment 2 presents the following images: the ground truth image (1 column), the blurred image after passing through the refractive system (2 columns), the anti-distortion image (3 columns), and the restored image after anti-distortion and passing through the refractive system (4 columns). Figure 5 .
[0041] Compared to existing technologies, this invention requires no invasive or wearable devices; it only corrects the pattern on the display screen. This invention generates a personalized correction network that acts directly on the displayed image, avoiding the drawbacks of wearable and invasive correction methods. The corrected image can be further enhanced, requiring further adjustments in contrast, brightness, and edge strength. It can be used to improve the visual quality of images, making them closer to the original, clear image.
[0042] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.
Claims
1. A non-invasive, non-wearable refractive correction method, characterized in that, Based on the user's refractive parameters, ray tracing processing based on defocus blur is performed to generate a set of blurred and distorted images corresponding to the original image after optical path distortion. This set is used to pre-train the human eye imaging distortion simulation model. Then, an image pre-correction model is constructed to generate refractive corrected images from the original image, and it is jointly trained with the human eye imaging distortion simulation model after parameters are frozen. In the online phase, the jointly trained image pre-correction model generates a refractive corrected image for the user's personalized display in real time based on the clear original image. The real-time generation process involves training the image pre-correction model using a multi-model interpolation method in the offline stage to obtain image pre-correction models for different viewing positions. In the online stage, the image pre-correction model for the current position is constructed using the previous image pre-correction models to ensure that a corresponding personalized refractive correction image can be generated at each position. The human eye imaging distortion simulation model is a deep neural network based on adaptive convolutional kernel weight allocation. This deep neural network uses a set of M convolutional kernels for each convolutional layer. At the position of each pixel in the output feature of the convolutional layer, each convolutional kernel is weighted to obtain a specific convolutional kernel corresponding to that pixel position. This convolutional kernel is used to calculate the result value at that pixel position. The weight map is a learnable feature tensor. The formula for calculating the specific convolutional kernel at pixel position (i, j) is as follows: Deep neural networks reconstruct the input, clear, original image into a simulated blurred and distorted image. The multi-model interpolation method refers to: extracting personalized refractive parameters of the user using multiple sets of different distances and angles, and jointly training the corresponding image pre-correction models. Each combination of distances and angles is used as points to form a grid. During the online phase, when the user is at a certain location... At that time, take the four points of the grid vertex where the position is located. , Network parameters of the corresponding image pre-correction model , Perform bilinear interpolation to construct the network parameters of the image pre-correction model corresponding to this location. It is used to generate refractive-corrected images, specifically: .
2. The non-invasive, non-wearable refractive correction method according to claim 1, characterized in that, The joint training refers to: using the pre-trained human eye imaging distortion simulation model with frozen parameters, generating a reconstructed image based on the refractive corrected image output by the image pre-correction model, and calculating the MSE Loss between the reconstructed image and the original image as the training objective function to train the image pre-correction model.
3. The non-invasive, non-wearable refractive correction method according to claim 1, characterized in that, The user's refractive parameters are obtained by the user viewing the display screen at a fixed distance and angle and by using ophthalmic measurement equipment to obtain corneal geometric parameters, axial length parameters, and retinal parameters.
4. The non-invasive, non-wearable refractive correction method according to claim 1, characterized in that, The pre-training process specifically involves: inputting a clear original image into the human eye imaging distortion simulation model, using the corresponding blurred distortion image obtained from the clear original image through ray tracing based on defocus blur as the model training target, and using MSE Loss as the loss function to supervise the training of the model.
5. The non-invasive, non-wearable refractive correction method according to claim 1, characterized in that, The image pre-correction model is a deep neural network with the same structure as the human eye imaging distortion simulation model, which reconstructs the input clear original image into a refractive corrected image.
6. The non-invasive, non-wearable refractive correction method according to claim 1, characterized in that, The joint training specifically involves: using the original clear image as input to the image pre-correction model, inputting the generated refractive corrected image into the human eye imaging distortion simulation model with frozen parameters, reconstructing the output image of the human eye imaging distortion simulation model into a clear original image as the training target, and using MSE Loss as the loss function to supervise the training of the model.
7. A non-invasive, non-wearable refractive correction system implementing the method of any one of claims 1-6, characterized in that, include: The system comprises a refractive parameter acquisition unit, a ray tracing unit, a human eye imaging distortion simulation model, an image pre-correction model network model, and a multi-model interpolation unit. Specifically, the refractive parameter acquisition unit acquires the user's refractive parameters, the ray tracing unit uses the refractive parameters to construct a simulated blurred image based on the original clear image, and the original clear image and the simulated blurred image serve as data pairs to train the human eye imaging distortion simulation model and the image pre-correction model network model. Finally, the image pre-correction model, reconstructed by the multi-model interpolation unit, is used to process the image to obtain a refractive-corrected image, enabling the human eye to directly see a clear image.