Digital adaptive aberration correction method for liquid lens zoom system

Abstract

The invention provides a digital adaptive aberration correction method for a liquid lens zoom system, and the method is based on a digital adaptive principle, and has general applicability for a continuous zoom optical system module formed based on a liquid lens. Images under the continuous magnification are acquired by adjusting the driving voltage of the liquid lens in the optical imaging system module based on the liquid lens; estimating a point spread function representing aberration from the collected image by using a trained aberration estimation model, wherein the aberration estimation model is obtained by training a deep linear neural network by using a paired degraded image data set; carrying out aberration correction on the collected image by combining the estimated point spread function with a trained aberration correction model, wherein the aberration correction model is obtained by training a neural network image recovery model by using a paired degraded image data set; and outputting the high-resolution continuous magnification image after the adaptive aberration correction.

Description

I. TECHNICAL FIELD

[0001] The present application relates to a method for aberration correction based on digital adaptive technology, more particularly, the present application relates to a digital adaptive aberration correction method for liquid lens zoom system. II. BACKGROUND

[0002] Under the background of the rapid development of modern optical imaging technology, various optical imaging systems have been widely used in industrial detection, biological medicine, aerospace, intelligent manufacturing and other fields. Although the existing optical imaging system has made significant progress in resolution, imaging speed and multi-functional integration, there are still many limitations in realizing multi-scale continuous zoom imaging, mainly in the following two aspects:

[0003] On the one hand, the traditional optical imaging system generally adopts solid lens structure, and its zoom function usually relies on mechanical structure to drive multiple optical lenses to move along the optical axis, or to replace lens components with different focal lengths to realize zoom adjustment, so as to obtain the required field of view range and imaging magnification. Therefore, the imaging magnification has obvious discreteness, and it is difficult to realize continuous and smooth zoom observation, which cannot meet the dynamic acquisition demand of image details in multi-scale scenes. On the other hand, the optical system based on liquid lens for continuous zoom often faces complex and dynamic optical aberration problems under different focal lengths and spatial fields of view. The existing aberration correction methods mostly rely on optical design optimization based on physical modeling, or use traditional image processing methods for post-correction. However, these methods often have insufficient generalization ability, poor method adaptability and weak real-time performance when dealing with complex and dynamic aberrations, which makes it difficult to meet the demand of high-precision and high-efficiency imaging in practical applications. Therefore, it is urgent to provide a method for adaptive correction of complex aberrations during liquid lens zoom, so as to improve the image quality and system robustness of the imaging system under multi-zoom state, and expand the application ability of optical imaging system in complex environment. III. SUMMARY

[0004] The present application proposes a digital adaptive aberration correction method for liquid lens zoom system, hereinafter referred to as the aberration correction method. Digital adaptive here refers to the technology of actively measuring and correcting optical aberration through neural network.

[0005] The aberration correction method first collects images through the optical imaging system module based on liquid lens, then measures the aberration of the collected images through the aberration estimation model, and then performs adaptive aberration correction on the collected images guided by the obtained aberration measurement results through the aberration correction model, so as to realize real-time aberration correction of the optical imaging system module based on liquid lens in continuous zoom, and obtain high-resolution continuous magnification imaging results.

[0006] Specifically, the flow of the aberration correction method is as follows:

[0007] Adjusting the driving voltage of the liquid lens-based optical imaging system module to collect images under continuous magnifications;

[0008] Using the trained aberration estimation model to estimate the point spread function representing the aberration from the collected images; the aberration estimation model can actively measure the aberration from the images, and the aberration estimation model is obtained by training a deep linear neural network using a paired degraded image dataset;

[0009] Combining the estimated point spread function with the trained aberration correction model to correct the aberration of the collected images; the aberration correction model performs adaptive aberration correction under the guidance of the obtained aberration measurement results, and the aberration correction model is obtained by training an image restoration neural network using a paired degraded image dataset;

[0010] Outputting the high-resolution continuous magnification image after aberration correction.

[0011] The technical features of the design of the aberration correction method are as follows:

[0012] In the aberration correction method, the image collected by the liquid lens-based optical imaging system module is an image I DR degraded by optical aberration, and the ideal image before optical degradation is defined as I GT , and their relationship is as follows:

[0013]

[0014] Where PSF represents the point spread function of the liquid lens-based optical imaging system module, which is used to describe the imaging quality of the optical system, represents the convolution operation.

[0015] According to the above relationship, we first collect the image I DR degraded by optical aberration based on the liquid lens-based optical imaging system module, measure the aberration from I DR using the aberration estimation model to obtain the point spread function PSF of the liquid lens-based optical imaging system module, and then perform adaptive aberration correction under the guidance of the obtained aberration measurement results, i.e., the point spread function PSF, using the aberration correction model, thereby realizing adaptive aberration correction and obtaining the high-resolution imaging result, i.e., the ideal image I GT before optical degradation.

[0016] The specific implementation flow of the aberration correction method is as follows: Figure 1 ​

[0017] In the first step, a set of initial signals is inputted into the liquid lens in the liquid lens-based optical imaging system module, the optical power and magnification of the system are initialized, the image to be measured passes through the system and is focused on the image sensor to form an image, by adjusting the driving signals of the liquid lens, the optical power of the liquid lens changes, the overall optical power distribution of the system changes accordingly, and the magnification changes accordingly, the mathematical relationship between the structure parameters and relative positions of the components of the system and the mathematical relationship between the imaging magnification and the field of view change during zooming so that the object image always satisfies the conjugate condition, and the image signals of the image sensor array are collected in real time;

[0018] In the second step, the collected image is subjected to aberration measurement by the trained aberration estimation model to obtain the point spread function of the liquid lens-based optical imaging system module under the current voltage driving state.

[0019] In the third step, the collected image is subjected to adaptive aberration correction by the aberration correction model under the guidance of the measured point spread function of the liquid lens-based optical imaging system module.

[0020] In the fourth step, a high-resolution continuous magnification image after adaptive aberration correction based on the imaging result of the liquid lens-based optical imaging system module is outputted.

[0021] Optionally, the liquid lens-based optical imaging system module comprises an optical imaging system comprising a liquid lens and a circuit module for driving the liquid lens.

[0022] The aberration estimation model comprises a first input layer, a first feature extraction layer, a parameter generation layer, a decoding generation layer, a kernel generation layer, a normalization layer and a first output layer connected in sequence.

[0023] The first feature extraction layer comprises a first convolution layer, a first linear residual block, a second linear residual block and a third linear residual block connected in sequence, and the output end of the first convolution layer is further connected to the input end of the third linear residual block.

[0024] The first linear residual block comprises a second convolution layer, an activation layer and a third convolution layer connected in sequence, and the output end of the second convolution layer is further connected to the input end of the third convolution layer.

[0025] The parameter generation layer comprises a fourth linear residual block, a fifth linear residual block and a pooling layer connected in sequence.

[0026] The first linear residual block, the second linear residual block and the third linear residual block have the same structure.

[0027] The inner core synthesis layer is sequentially connected with a grouping convolutional layer after being combined by three parallel first, second and third fully connected layers.

[0028] The aberration correction model comprises a second input layer, an image block embedding, a transform encoder, a multi-layer perceptron and a second output layer connected in sequence, and a third input layer and a second feature extraction layer connected in sequence, and an output end of the second feature extraction layer is further connected to an input end of the transform encoder.

[0029] The first feature extraction layer and the second feature extraction layer have the same structure.

[0030] The transform encoder comprises a first normalization layer, a multi-head attention mechanism, a second normalization layer, a feedforward neural network and a third output layer connected in sequence.

[0031] An output end of the first normalization layer is further connected to an input end of the second normalization layer, and an output end of the second normalization layer is further connected to an input end of the third output layer.

[0032] The second input layer is used for inputting the collected degraded image.

[0033] The third input layer is used for inputting a point spread function image output by the aberration estimation model after the collected degraded image.

[0034] The image block embedding is used for dividing the collected degraded image into a plurality of image blocks according to a preset size, respectively flattening each image block into a one-dimensional vector, and mapping each one-dimensional vector to a fixed vector dimension through linear transformation to obtain a plurality of patch embeddings, simultaneously generating learnable class labels and position embeddings.

[0035] The transform encoder is used for capturing global dependency relationships based on the learnable class labels and position embeddings and the plurality of patch embeddings to obtain an output sequence.

[0036] The second output layer outputs a recovered image after aberration correction of the collected degraded image.

[0037] Optionally, before determining the deep linear network as the aberration estimation model, the method further comprises:

[0038] Obtaining a dataset of a collected original image and a paired point spread function as a dataset for training the aberration estimation;

[0039] Dividing the paired dataset into a first training set and a first validation set;

[0040] Inputting the first training set into the deep linear network to obtain an aberration estimation result of the first training set;

[0041] Based on the aberration estimation training results of the first training set and the paired point spread function, adjust the parameters of the improved deep linear network and return to the step "obtain the dataset of the acquired original images and their paired point spread functions as the paired image set" until the number of iterations reaches the first preset number of iterations, and obtain the aberration estimation model to be determined.

[0042] The undetermined aberration estimation model is validated using the first validation set;

[0043] If the verification fails, adjust the parameters of the undetermined aberration estimation model, set the number of iterations to zero, treat the undetermined aberration estimation model as a deep linear network, update the image set, and return to the step "obtain the dataset of the acquired original images and their paired point spread functions as the paired image set".

[0044] If the verification passes, the undetermined aberration estimation model is determined to be an aberration estimation model.

[0045] Optionally, after determining the aberration estimation model to be determined as an aberration estimation model, the following steps are also included:

[0046] Obtain a dataset of the acquired original images and their paired aberration-free images as a training dataset for aberration correction.

[0047] The dataset for training aberration correction is divided into a second training set and a second validation set;

[0048] The image recovery neural network is trained using the second training set to obtain the undetermined aberration correction model.

[0049] The undetermined aberration correction model is validated using the second validation set;

[0050] If the verification fails, adjust the parameters of the undetermined aberration correction model, use the undetermined aberration correction model as an image recovery neural network, update and return to the step "input the aberration correction dataset into the aberration correction model to obtain the training aberration correction image set";

[0051] If the verification is successful, the undetermined aberration correction model is determined to be an aberration correction model. IV. Description of the attached drawings

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Appendix Figure 1 This is a schematic diagram illustrating the implementation process of the method of the present invention;

[0054] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 2 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0055] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 3 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0056] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 4 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0057] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 5 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0058] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 6 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0059] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 7 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0060] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 8 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0061] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 9 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application;

[0062] Figure 1 is a schematic diagram of an aberration estimation model structure in the present application; Figure 10 Figure 1 is a schematic diagram of an aberration estimation model structure in the present application. V. DETAILED DESCRIPTION

[0063] The following detailed description of an embodiment of a digital adaptive aberration correction method for a liquid lens zoom system is provided to further describe the present application. It is necessary to point out that the following embodiment is only used to further describe the present application and cannot be understood as a limitation on the protection scope of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application according to the above description, which still falls within the protection scope of the present application.

[0064] The liquid lens-based optical imaging system module in the present embodiment is a continuously zoomable optical microscope based on liquid lens design and its driving. The objective lens of the microscope is composed of a plurality of liquid lenses and solid lenses, which is used for continuously zooming microscopic imaging of a sample, and a liquid lens driving module is used to provide driving signals for the liquid lenses in the objective lens. On the software part, the aberration estimation model can measure aberration from an image; and the aberration correction model performs adaptive aberration correction under the guidance of the obtained aberration measurement results.

[0065] In the working process, the working state of the microscopic system is always kept unchanged, and each element does not move mechanically, and the zooming function of the microscopic system is completely realized by regulating and controlling the liquid lenses.

[0066] AsFigure 1 The method comprises: adjusting a driving voltage of a liquid lens, collecting images at continuous magnifications, estimating a point spread function representing aberration from the collected images using a trained aberration estimation model, correcting aberration of the collected images using the estimated point spread function and a trained aberration correction model, and outputting high-resolution continuous magnification images after adaptive aberration correction.

[0067] As Figure 2 The aberration estimation model comprises: a first input layer, a first feature extraction layer, a parameter generation layer, a decoding generation layer, a kernel synthesis layer, a normalization layer and a first output layer connected in sequence.

[0068] As Figure 3 The first feature extraction layer comprises: a first convolutional layer, a first linear residual block, a second linear residual block and a third linear residual block connected in sequence. The output end of the first convolutional layer is also connected to the input end of the third linear residual block.

[0069] As Figure 4 The first linear residual block comprises: a second convolutional layer, an activation layer and a third convolutional layer connected in sequence. The output end of the second convolutional layer is also connected to the input end of the third convolutional layer.

[0070] As Figure 5 The parameter generation layer comprises: a fourth linear residual block, a fifth linear residual block and a pooling layer connected in sequence.

[0071] As Figure 6 The kernel synthesis layer comprises: a first full connection layer, a second full connection layer, a third full connection layer, and the output ends of the three full connection layers are connected to the input end of a grouped convolutional layer.

[0072] As Figure 7 The aberration correction model comprises: a second input layer, an image block embedding, a transform encoder, a multi-layer perceptron and a second output layer connected in sequence, and an additional feature extraction layer.

[0073] As Figure 8 The transform encoder comprises a first normalization layer, a multi-head attention mechanism, a second normalization layer, a feedforward neural network and a third output layer connected in sequence.

[0074] The output end of the first normalization layer is also connected to the input end of the second normalization layer. The output end of the second normalization layer is also connected to the input end of the third output layer.

[0075] The second input layer is used for inputting the collected images.

[0076] The third input layer is used for inputting the point spread function image obtained by the aberration estimation model.

[0077] The second feature extraction layer is configured to extract features from the input point spread function image and is connected to the transform encoder.

[0078] The image patch embedding is configured to divide the captured image into a plurality of image patches according to a preset size, flatten each image patch into a one-dimensional vector respectively, and map each one-dimensional vector to a fixed vector dimension through linear transformation to obtain a plurality of patch embeddings, and simultaneously generate learnable class labels and position embeddings.

[0079] The transform encoder is configured to capture global dependencies based on the learnable class labels and position embeddings and the plurality of patch embeddings to obtain an output image.

[0080] The second output layer outputs the aberration-corrected image.

[0081] As Figure 9 , the determining step of the aberration estimation model specifically includes:

[0082] An original picture set is obtained, and the original picture set includes captured images and paired point spread function images.

[0083] The original picture set is randomly divided into a first training set and a first test set according to a ratio of 7:3.

[0084] The first training set is used to train the deep linear neural network to obtain a to-be-determined aberration estimation model.

[0085] The first validation set is used to verify the to-be-determined aberration estimation model.

[0086] If the verification fails, the parameters of the to-be-determined aberration estimation model are adjusted, the to-be-determined neural network is used as the aberration estimation model, the degraded image set is updated, and the first step is returned.

[0087] If the verification passes, the to-be-determined aberration estimation model is determined as the aberration estimation model.

[0088] The used aberration estimation model is a deep linear neural network, and the overall structure is composed of an input layer, a feature extraction layer, a parameter generation layer, a decoding generation layer, a kernel synthesis layer, and a normalized output layer. The input layer accepts a low-resolution image with a size of (B, C, H, W); the feature extraction layer is composed of a 7x7 convolution (output channel number 64) and three residual blocks without BatchNorm, the activation function is LeakyReLU (0.1), and the shallow features are fused through residual connection; the parameter generation layer performs 3x3 convolution+LeakyReLU on the feature map twice, then compresses to 1x1 through global average pooling, and then maps to a vector with a length of 10 through a 1x1 convolution; the decoding generation layer includes three fully connected branches, which respectively map the 10-dimensional vector to small convolution kernels with sizes of 11x11, 7x7, and 3x3, and the channel structures are 3→16, 16→16, and 16→3 in turn; the kernel synthesis layer performs group convolution on the five groups of small kernels in order to generate a point spread function from a unit input; the normalization layer normalizes the point spread function of each sample and each channel according to the total sum of pixels, and finally the output layer outputs the obtained point spread function.

[0089] As Figure 10 , the determination step of the aberration correction model is:

[0090] The collected images and the point spread functions obtained by the aberration estimation model are combined to obtain a training aberration correction image set.

[0091] The training aberration correction image set is divided into a second training set and a second validation set.

[0092] The second training set is used to train the image restoration neural network model to obtain a to-be-determined aberration correction model.

[0093] The second validation set is used to verify the to-be-determined aberration correction model.

[0094] If the verification fails, the parameters of the to-be-determined aberration correction model are adjusted, the to-be-determined image restoration neural network is used as the aberration correction model, the degraded image set is updated, and the first step is returned.

[0095] If the verification passes, the to-be-determined aberration correction model is determined as the aberration correction model.

[0096] The training process of the aberration correction model is as follows:

[0097] The point spread function obtained by the estimation model is obtained, and the aberration-corrected image is a high-resolution microscopic image.

[0098] The collected images and the corresponding estimated point spread functions are randomly divided into a training set and a test set according to a ratio of 7:3.

[0099] The training set input image is input into the image restoration neural network for training to obtain an aberration correction model.

[0100] The image restoration neural network used by the aberration correction model is a Vision Transformer. The Vision Transformer (ViT) architecture is a neural network that applies the Transformer to computer vision tasks. The overall model is composed of an input layer, patch embedding, a transformer encoder, and an output layer. First, the input layer accepts an image of size (H, W, C), then the image is divided into fixed-size image patches, each patch of size P x P, these image patches are flattened into a one-dimensional vector, and mapped to a fixed vector dimension D through a linear transformation, forming patch embeddings, while generating a learnable class token and position embedding. The transformer encoder contains L layers, each layer is connected in turn by multi-head self-attention (MHSA), layer normalization, feed forward network (FFN), and residual, to capture global dependencies. The final output layer is connected to the input layer with a residual connection to obtain the aberration-corrected image.

[0101] By separating the aberration estimation module and the aberration correction module, the computational burden of a single model is reduced, and the corresponding models can be optimized for aberration estimation and aberration correction tasks, respectively, to improve overall processing efficiency. During the training of the aberration estimation and correction models, data augmentation techniques based on liquid lens focal length changes and distortion parameter simulation are used to increase the diversity of training data and the generalization ability of the model, while regularization techniques such as Dropout and Batch Normalization are applied to prevent overfitting and improve the robustness and stability of the system. By removing background light and irrelevant visual interference, the correction model focuses on aberration information in microscopic imaging, significantly improving correction accuracy. Through an automated aberration estimation-correction pipeline, manual adjustment and repeated calibration are reduced, improving overall imaging efficiency. Through the collaborative optimization of hardware continuous zoom and software digital adaptive methods, the system's adaptability to different magnification and environmental conditions is enhanced, improving the flexibility and robustness of imaging. Through modular design and separation of the estimation-correction steps, the models are optimized, reducing the demand for computing resources, and improving the scalability and flexibility of the overall system.

[0102] The aberration correction method is a digital self-adaptive method of prior aberration estimation and subsequent aberration correction, and the application can provide a more efficient, accurate and robust aberration correction solution, solves the key problems of manual correction complexity, model coupling degree and the like in the prior art, and significantly improves the practical application value of the optical imaging system module based on the liquid lens.

Claims

1. A digital adaptive aberration correction method for liquid lens zoom systems, characterized in that, This aberration correction method is based on the principle of digital adaptation and consists of an aberration estimation model and an aberration correction model. The aberration correction method is described as follows: By adjusting the driving voltage of the liquid lens in the liquid lens-based optical imaging system module, images at continuous magnification are acquired; a point spread function characterizing aberrations is estimated from the acquired images using a trained aberration estimation model, which is obtained by training a deep linear neural network using a paired degraded image dataset; The estimated point spread function is combined with a trained aberration correction model to correct aberrations in the acquired image. The aberration correction model is obtained by training a neural network image restoration model using a paired degraded image dataset. Finally, a high-resolution image after adaptive aberration correction is output.

2. The digital adaptive aberration correction method for a liquid lens zoom system according to claim 1, characterized in that, The aberration estimation model is obtained as follows: Obtain the acquired raw image set, which consists of the acquired images and their paired point spread function images; The image set used for training aberration estimation was randomly divided into a first training set and a first test set in a 7:3 ratio; The neural network image classification model is trained using the first training set to obtain the undetermined aberration estimation model; The undetermined aberration estimation model was validated using the first validation set. If the verification fails, adjust the parameters of the undetermined aberration estimation model, update the undetermined aberration estimation model to a deep linear neural network, update the degraded image set, and return to step one. If the verification passes, the undetermined aberration estimation model is determined to be an aberration estimation model.

3. The digital adaptive aberration correction method for a liquid lens zoom system according to claim 1, characterized in that, The aberration correction model is obtained as follows: The acquired images and the point spread function obtained by the aberration estimation model are combined to obtain the training aberration-corrected image set; The training aberration-corrected image set was randomly divided into a second training set and a second validation set in a 7:3 ratio. The neural network image classification model is trained using the second training set to obtain the undetermined aberration correction model; The undetermined aberration correction model was validated using the second validation set. If the verification fails, adjust the parameters of the undetermined aberration correction model, update the undetermined aberration correction model to an image restoration neural network, update the degraded image set, and return to step one; If the verification is successful, the undetermined aberration correction model is determined to be an aberration correction model.

4. The digital adaptive aberration correction method for a liquid lens zoom system according to claim 1, characterized in that, The optical imaging system in the liquid lens-based optical imaging system module can be a microscope, telescope, etc.

5. The digital adaptive aberration correction method for a liquid lens zoom system according to claim 1, characterized in that, Image acquisition is achieved through continuous zooming using an optical imaging system module based on liquid lenses. The number of liquid lenses used in the optical system should be greater than or equal to 1.

6. The digital adaptive aberration correction method for a liquid lens zoom system according to claim 1, characterized in that, Liquid lenses achieve changes in optical power by altering the curvature of their interface through external driving. The types of liquid lenses used in optical imaging system modules based on liquid lenses can be electrowetting-driven liquid lenses, mechanically driven elastic thin-film liquid lenses, dielectric force-driven liquid lenses, magnetically controlled liquid lenses, etc.

Images