General aberration degradation image restoration method based on lens database construction

Abstract

This invention discloses a general aberration-degraded image restoration method based on lens database construction, including a lens database construction method based on automatic lens design and a general aberration-degraded image restoration algorithm based on feature quantization. The lens database is constructed by an automatic lens design algorithm; the general aberration-degraded image restoration algorithm includes two stages: feature quantization representation pre-training and feature restoration matching, enabling the restoration of aberration-affected images from any input lens. This invention overcomes the limitation of traditional deep learning-based aberration-degraded image restoration algorithms, which can only restore images from a single specific lens. It achieves aberration-degraded image restoration for any lens using a single general model, significantly saving the time cost of data preparation and algorithm training in computational imaging systems. The lens database construction method proposed in this invention solves the problem of missing training data for the general model, greatly improving the generalization performance of the general aberration-degraded image restoration algorithm.

Description

Technical Field

[0001] This invention relates to the fields of optical design, computational imaging, and aberration-degraded image restoration, and particularly to a general aberration-degraded image restoration method based on a lens database. Background Technology

[0002] With the widespread application of mobile and portable devices such as assistive devices for the blind, drones, virtual reality glasses, intelligent robots, and mobile phones, the demand for optical imaging lenses with high image quality, large field of view, small size, and light weight is gradually increasing, which has become a new development trend in optical imaging lens design. However, in the design of optical imaging lenses, image quality and size are often difficult to balance. To obtain excellent image quality, commonly used optical imaging lenses usually have the disadvantages of complex structure, large size and weight, and are not conducive to integration and portability.

[0003] To address this challenge, computational imaging emerged, where optical imaging lenses are designed to tolerate uncorrected aberrations in order to achieve a lighter structure. Backend algorithms then recover the aberration-degraded image, allowing the lightweight imaging lens to achieve image quality approaching that of a complex lens. Current mainstream aberration-degraded image recovery algorithms use deep learning-based image recovery models. Before use, these models need to be trained on a pair of aberration-degraded images and high-resolution aberration-free images corresponding to the lens being used. The trained model is only used to recover aberration-degraded images under that specific lens. However, when the imaging lens structure is changed, the aberration distribution changes, rendering the trained model ineffective. This necessitates re-performing the steps of dataset preparation and model training for the new lens structure before the model can be applied to the new lens. This close link between the recovery model and the imaging lens structure significantly increases the time cost of designing and deploying lightweight computational imaging systems, limiting their updates and iterations. Therefore, the current field needs a general method for aberration-degraded image restoration, capable of restoring aberration-degraded images from any lightweight imaging lens by training only a single model. The challenges of this technique lie in: 1) Dataset preparation: Imaging lenses have diverse structures and varying aberration distributions, making it difficult to cover all possible aberration distributions with a single dataset; 2) Model design: Even if the training data can encompass diverse aberrations, the model needs strong generalization ability and the capacity to analyze and process different aberration degradations to achieve general aberration-degraded image restoration. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a general aberration-degraded image restoration method based on lens database construction.

[0005] This invention proposes an automatic imaging lens design method to construct a lens database. This method can efficiently search the solution space of different imaging lens structures, generating rich and multi-specification imaging lens samples. The constructed lens database covers a large number of aberrations with different distribution characteristics and severity, providing a reliable training dataset for a general aberration degraded image restoration model. This invention also proposes to simultaneously use degradation and high-resolution feature prior quantization to improve the generalization of the aberration degraded image restoration model, thereby designing a structure for a general aberration degraded image restoration model, enabling the restoration of aberration degraded images of any imaging lens using only a trained model.

[0006] This invention is achieved through the following technical solution:

[0007] A general method for restoring aberration-degraded images based on a lens database includes the following steps:

[0008] 1) Multiple lens structures are generated using an automatic lens design method, and multiple lens samples are obtained after optimizing the lens structures according to the evaluation function;

[0009] 2) The lens samples are selected based on their aberration distribution characteristics, and the selected lens samples form a lens database;

[0010] 3) Obtain an aberration-free image dataset containing multiple aberration-free images. Each aberration-free image is simulated using a lens sample from the lens database to obtain a corresponding aberration image, thus forming an aberration image dataset.

[0011] 4) Initialize the degraded feature prior codebook and the high-definition feature prior codebook, construct the aberration-degraded image reconstruction model and the high-definition image reconstruction model, and perform feature quantization representation pre-training respectively. Update the degraded feature prior codebook using the pre-training process of the aberration-degraded image reconstruction model, and update the high-definition feature prior codebook using the pre-training process of the high-definition image reconstruction model. After the pre-training is completed, the final degraded feature prior codebook and the high-definition feature prior codebook are obtained.

[0012] 5) Construct a general aberration restoration image model by combining the final degradation feature prior codebook and the high-definition feature prior codebook, and train the general aberration restoration image model using aberration-free image dataset and aberration image dataset;

[0013] 6) Use the trained general aberration restoration image model to restore the aberration-degraded image taken by the actual lens to be restored, and output the restored aberration-free image.

[0014] Preferably, in step 1), multiple lens structures are generated using an automatic lens design method, and multiple lens samples are obtained after optimization based on an evaluation function. Specifically:

[0015] 1.1) Randomly generate multiple lens structures using preset design specifications as the initial lens group P; the preset design specifications are the field of view, F-number, aperture position, and number of lens elements;

[0016] 1.2) The optimized lens group P is obtained by performing global optimization on the lens group P based on the evaluation function;

[0017] 1.3) Select 10%-40% of the lenses with higher evaluation function values ​​from the optimized lens group P* to obtain a high-quality lens group Q;

[0018] 1.4) The optimized high-quality lens group Q* is obtained by performing local optimization on the high-quality lens group Q according to the evaluation function;

[0019] 1.5) The optimized high-quality lens group Q* is mutated to obtain the mutated lens group M. The optimized high-quality lens group Q* and the mutated lens group M are merged as the update value of the lens group P and returned to step 1.2) Iterative optimization is performed until the number of iterations meets the preset maximum number of iterations. All lens structures in the current optimized high-quality lens group Q* are output as lens samples.

[0020] Preferably, the value of the evaluation function in step 1) is obtained by weighted summation of image quality evaluation index and physical constraint evaluation index. The image quality evaluation index is the average value of the RMS radius of the dot plot, the modulation transfer function or wave aberration, and the physical constraint evaluation index is the weighted summation of effective focal length, total system length, back working distance, lens center thickness, lens edge thickness and air center distance.

[0021] Preferably, the global optimization in step 1.2) adopts the selection simulated annealing algorithm, particle swarm optimization algorithm or ant colony optimization algorithm; the local optimization in step 1.4) adopts the damped least squares method, ADAM algorithm or quasi-Newton method.

[0022] Preferably, the variation in step 1.5) is to randomly change some parameters of the lens structure, such as curvature, glass thickness, air gap, material refractive index, and material Abbe number. The sum of the variable values ​​of the changed glass thickness and air gap needs to be greater than 1, and the sum of the variable values ​​of the glass thickness and air gap remains consistent before and after the change.

[0023] Preferably, the aberration distribution characteristics of the lens samples in step 2) are the average value of the RMS radius of the point array of each field of view of the lens samples and the absolute value difference between the point spread function arrays of each lens sample.

[0024] The method of filtering lens samples based on aberration distribution characteristics specifically involves: setting a range for the average RMS radius of the point spread function (PSF) array of each field of view of the lens samples; sampling lens samples within this range to ensure a uniform distribution of the average PSF radius of each field of view after sampling; and ensuring that the absolute difference between the PSF array of each sampled lens sample and the PSF arrays of all other sampled lens samples is greater than a set threshold, which is 0.25 × 10⁻⁶. -6 -1×10 -6 .

[0025] Preferably, in step 3), each aberration-free image is obtained by simulation using a lens sample from the lens database to obtain a corresponding aberration image. The specific process is as follows:

[0026] 3.1) Calculate the point spread function distribution matrix, distortion distribution matrix, and illuminance distribution matrix with respect to the field of view for each lens sample in the lens database;

[0027] 3.2) For each aberration-free image, randomly select the point spread function distribution matrix, distortion distribution matrix, and illumination distribution matrix as a function of the field of view of a lens sample;

[0028] 3.3) Perform block convolution between the aberration-free image and a randomly selected point spread function distribution matrix. Multiply the block convolution result by the illumination distribution matrix, and then perform distortion transformation through the distortion distribution matrix to output the corresponding aberration image.

[0029] Preferably, the aberration-degraded image reconstruction model and the high-definition image reconstruction model described in step 4) have the same structure, both including a feature extraction module, a feature quantization module, and an image reconstruction module;

[0030] Both the degraded feature prior codebook and the high-definition feature prior codebook are codebooks composed of multiple discrete vectors, and each discrete vector is a learnable parameter.

[0031] The steps are as follows: Pre-training is performed on feature quantization representations, the prior codebook of degraded features is updated using the pre-training process of the aberration-degraded image reconstruction model, and the prior codebook of high-definition features is updated using the pre-training process of the high-definition image reconstruction model.

[0032] For the aberration-degraded image reconstruction model, the aberration image dataset is input into the aberration-degraded image reconstruction model for reconstruction. The reconstruction specifically involves: the feature extraction module extracting the features of the input aberration image; the features of the aberration image being quantized by the feature quantization module using the degraded feature prior codebook; and the quantized features being input into the image reconstruction module to obtain the reconstructed image corresponding to the aberration image.

[0033] The image reconstruction loss function is calculated, which includes the pixel absolute value loss, perceptual loss and adversarial generation loss between the reconstructed image and the input aberration image, and the semantic loss is calculated on the quantized features output by the feature quantization module.

[0034] Backward gradient propagation is performed based on the image reconstruction loss function and semantic loss to update the parameters of the feature extraction module, image reconstruction module, and degradation feature prior codebook of the aberration-degraded image reconstruction model; after iterating the expected number of training iterations, the required degradation feature prior codebook is obtained.

[0035] Similarly, for the high-definition image reconstruction model, the aberration-free image dataset is input into the high-definition image reconstruction model for reconstruction. During the reconstruction process, the feature quantization module uses the high-definition feature prior codebook for quantization, and the reconstructed image corresponding to the aberration-free image is obtained. The image reconstruction loss function is calculated and the semantic loss is calculated for the quantized features during the reconstruction process. Backward gradient propagation is performed based on the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, the image reconstruction module and the high-definition feature prior codebook of the high-definition image reconstruction model. After iterating the expected number of training iterations, the required high-definition feature prior codebook is obtained.

[0036] Preferably, the general aberration restoration image model described in step 5) includes a degradation feature prior codebook, a high-resolution feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-resolution feature quantization and fusion module, and a high-resolution image reconstruction module; the parameters of the high-resolution image reconstruction module inherit the parameters of the image reconstruction module obtained by the feature quantization representation pre-training.

[0037] The specific steps for training the general aberration restoration image model using aberration-free image dataset and aberration image dataset are as follows:

[0038] Freeze the degradation feature prior codebook, high-definition feature prior codebook, and high-definition image reconstruction module in the general aberration restoration image model;

[0039] The aberration image dataset is input into the degradation feature extraction module to extract degradation features. The degradation features are then quantized and fused with the degradation feature prior codebook using the degradation feature quantization and fusion module. The fused degradation features are then enhanced by the degradation feature enhancement module and input into the high-definition feature quantization and fusion module. The high-definition features are quantized using the high-definition feature prior codebook and then concatenated and fused with the enhanced features before quantization. The fused high-definition features are then input into the high-definition image reconstruction module to output the restored high-definition image.

[0040] The image restoration loss function is calculated, which includes the absolute pixel value loss, perceptual loss, and adversarial generation loss between the restored high-resolution image and the input aberration image. The quantization loss is also calculated for the quantized features obtained in the high-resolution feature quantization and fusion module.

[0041] Based on the image restoration loss function and quantization loss, backpropagation is performed to update all modules in the general aberration restoration image model except for the frozen ones; after iterating the expected number of training iterations, the training ends.

[0042] Preferably, for the feature quantization module in the aberration-degraded image reconstruction model and the high-resolution image reconstruction model, the degradation feature quantization and fusion module in the general aberration recovery image model, and the high-resolution feature quantization and fusion module, the method for quantizing features using a priori codebooks is the same, specifically:

[0043] The feature to be quantized is matched with the feature prior codebook. The distance between the feature vector corresponding to each pixel position of the feature to be quantized and the codebook vectors in the feature prior codebook is calculated. The feature vector at the pixel position of the feature to be quantized is replaced with the codebook vector with the smallest distance. After replacing the feature vectors at all pixel positions, the feature quantization is completed, and the quantized feature is obtained.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1) This invention proposes a lens database construction method that can automatically search the solution space under preset design specifications and evaluation functions, outputting diverse, condition-satisfied, and optimized lens structures; the lens database obtained after screening these lens structures covers different aberration distribution characteristics. This lens database construction method solves the problem of missing training data for the aforementioned general model, and by proposing an automatic optical design method to generate a large number of lens samples with multiple specifications, it greatly improves the generalization performance of general aberration degradation image restoration algorithms.

[0046] 2) Because the aberration-degraded image restoration of this invention employs a priori pre-training-matching reconstruction and decomposes degradation features and content features, the general aberration-degraded image restoration model can extract high-definition content information from different types of aberration-degraded images. Therefore, it has strong generalization ability and can perform high-quality restoration of aberration-degraded images from any imaging lens. This invention overcomes the limitation of traditional deep learning-based aberration-degraded image restoration methods, which can only restore images from a single specific lens. It realizes the use of a general model to restore aberration-degraded images from any lens, greatly saving the time cost of data preparation and model training in computational imaging systems. Attached Figure Description

[0047] Figure 1 An overview diagram of a general aberration-degraded image restoration method based on a lens database;

[0048] Figure 2 A comparison chart of general aberration-degraded image restoration methods and traditional aberration-degraded image restoration methods;

[0049] Figure 3 Design an algorithm flowchart for an automatic imaging lens;

[0050] Figure 4 This is a distribution map of lens samples;

[0051] Figure 5 This is a schematic diagram of the structure of a general aberration degradation recovery model;

[0052] Figure 6 This is a schematic diagram of the feature quantization module;

[0053] Figure 7 This is a schematic diagram of feature quantization representation pre-training;

[0054] Figure 8 Flowchart for training a general aberration degradation recovery model;

[0055] Figure 9 The results are from a general aberration degradation recovery model test. Detailed Implementation

[0056] To make the technical solution and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0057] Figure 1 This is an overview of a general aberration-degraded image restoration method based on a lens database. The invention utilizes an automatic lens design algorithm to automatically search the solution space under preset design specifications and evaluation functions, outputting diverse, condition-satisfied, and optimized lens structures. These lens structures are selected to form a lens database, which covers different aberration distribution characteristics. The lens structures from the lens database and clear, aberration-free image data are input into an optical simulation model to obtain aberration-degraded image data containing diverse aberration distributions, forming data pairs with the clear, aberration-free image data. These data pairs are then used to train a general aberration-degraded image restoration model. This model uses pre-trained degradation feature priors and high-definition feature priors, enabling it to extract high-definition content information from different types of aberration-degraded images. It exhibits strong generalization ability, and the trained model can perform high-quality restoration of aberration-degraded images from any imaging lens.

[0058] This invention proposes a general aberration-degraded image restoration method based on a lens database, which includes the following steps:

[0059] 1) Automatic lens design: Input design specifications, set evaluation functions, generate possible lens structures under the corresponding specifications through automatic lens design algorithm, and output lens samples;

[0060] 2) Lens sample selection: Based on the aberration distribution characteristics of lens samples, a sampling strategy and sampling quantity are set to select samples and build a lens database;

[0061] 3) Construction of a general aberration-degraded image restoration dataset: Obtain an aberration-free image dataset, for clear aberration-free images I gt The aberration images I corresponding to the lenses in the lens database are simulated using a simulation algorithm. ab The image data pairs {I} are composed of... gt I ab All image data pairs constitute a universal aberration-degraded image restoration dataset;

[0062] 4) Model training and application: Feature quantization representation pre-training is used to obtain feature prior codebook. A general aberration restoration image model is constructed by combining the feature prior codebook. The general aberration restoration image model is trained and used to restore aberration-degraded images of any lens, outputting a high-quality restored image.

[0063] like Figure 3 As shown, the general aberration-degraded image restoration method based on lens database proposed in this invention, compared with traditional aberration-degraded image restoration methods, only needs to be trained once using the lens database to obtain a general aberration-degraded image restoration model, which can restore aberration-degraded images of lenses with various aberrations. This greatly saves the time cost of data preparation and algorithm training in the computational imaging system, and also broadens the application scope and convenience of the model.

[0064] The flowchart of the automatic lens design algorithm in step 1) is as follows: Figure 3 As shown, the steps are as follows:

[0065] 1.1) Randomly initialize the lens group P according to the input design specifications.

[0066] 1.2) Based on the input evaluation function, P is globally optimized to obtain P*.

[0067] 1.3) Based on the input evaluation function, select a subset of lenses with higher evaluation function values ​​from P* to obtain Q.

[0068] 1.4) Based on the input evaluation function, Q is locally optimized to obtain Q*.

[0069] 1.5) If the number of iterations reaches the expected number of training iterations, output Q*; otherwise, Q* mutates to M, Q* and M are merged into P and the process returns to step 1.2) to continue optimization; in this embodiment, the expected number of training iterations is set to 10.

[0070] The design specifications in step 1.1) include field of view, F-number, aperture position, and number of lens elements. The half field of view sampling range is 20 degrees to 40 degrees with a sampling interval of 2 degrees. The F-number sampling range is 2.0 to 5.0 with a sampling interval of 0.3. The aperture stop position sampling position is all possible positions of the aperture stop for a given number of lens elements. The number of lens elements is set to 1 to 6.

[0071] The random initialization in step 1.1) is to normalize the parameters based on the range of values ​​for lens curvature, glass thickness, air gap, material refractive index, and material Abbe number, and obtain the initial lens group by randomly generating a parameter vector group in the range of 0 to 1.

[0072] The input parameters for the evaluation function in steps 1.1)-1.4) are the radius of curvature, glass thickness, air gap, refractive index, and Abbe number of each lens in the lens group. The evaluation function value of each lens in the lens group is calculated using a ray tracing simulation algorithm. The lower the evaluation function value of a lens, the higher its quality. The evaluation function is obtained by weighted summation of image quality evaluation and physical constraint evaluation. Image quality evaluation can use traditional optical design evaluation indicators such as dot plot, modulation transfer function (MTF), and wave aberration. In this embodiment, the average value of the RMS radius of the dot plot of all sampled fields of view and wavelengths is used. Physical constraint evaluation includes lens curvature, lens thickness, air gap, material refractive index, material Abbe number, total system length, back working distance, and edge image height. The physical constraint evaluation is a weighted summation of the above parameters. The lens curvature constraint range is -0.1 to 0.1, the lens thickness constraint range is 4 mm to 15 mm, the air gap constraint range is 1 mm to 15 mm, the material refractive index constraint range is 1.51 to 1.95, and the material Abbe number constraint range is 18.9 to 81.6. Assuming the number of lenses is x, the total system length constraint range is no more than (x*15+25) mm, the back working distance constraint range is more than 15 mm, and the edge image height constraint range is 14.2 mm to 14.4 mm. Under these settings, the automatic lens design algorithm in this embodiment can output approximately 12,000 lens samples.

[0073] The global optimization in step 1.2) can be performed using heuristic global search algorithms such as simulated annealing, particle swarm optimization, and ant colony optimization; the local optimization in step 1.4) can be performed using local optimization algorithms such as damped least squares, ADAM algorithm, and quasi-Newton method. In this embodiment, the global optimization uses the simulated annealing algorithm, and the local optimization uses the ADAM algorithm.

[0074] In step 1.3), the lens group is sorted according to the evaluation function value, and 10%-40% of the lenses are selected from high to low. If there are two similar lenses, only the lens with the better evaluation function value is retained. In this embodiment, 20% of the lenses are selected from the lens group.

[0075] The lens sample aberration distribution characteristics in step 2) are as follows:

[0076] a) The average RMS radius of the dot plot of each field of view of the lens sample indicates the severity of the aberration of the lens sample;

[0077] b) The absolute difference between the spread function arrays of each lens sample point indicates the degree of difference between the lens samples;

[0078] The output of approximately 12,000 lens samples includes a point plot of each field of view, an average distribution plot of the RMS radius, and a distribution plot of each specification, as shown in the figure. Figure 4 As shown.

[0079] The sampling strategy used in step 2) requires that the sampled lens samples cover a uniformly distributed range of aberration severity, and that the difference between lens samples is greater than a set threshold. Specifically, in this embodiment, the average range of the RMS radius of the point spread function (PSF) of each field of view of the lens samples is set to 0–0.2. Within this range, 12,000 generated lens samples are sampled, ensuring that the average RMS radius of the PSF of each field of view of the sampled lens samples is uniformly distributed between 0 and 0.2. Simultaneously, for each sampled sample, the absolute difference between its point spread function array and the point spread function arrays of all sampled samples needs to be calculated. When the absolute difference is greater than a set threshold, the difference of the sampled lens sample is considered to meet the requirements, and the lens sample is accepted. In this embodiment, the threshold for the absolute difference is set to 0.35 × 10⁻⁶. -6 After this screening and sampling process, 1,000 lens samples were obtained, forming a lens database.

[0080] Based on the constructed lens database, the steps for constructing the general aberration-degraded image restoration dataset in step 3) are as follows:

[0081] 3.1) Calculate the point spread function, distortion, and illuminance distribution matrix of each lens in the lens database using an optical simulation model;

[0082] 3.2) For each aberration-free image I gt The point spread function, distortion, and illuminance distribution matrix of a lens in the lens database as calculated in 3.1) are randomly selected; the selection of an aberration-free image in advance does not affect the selection of an aberration-free image in the future.

[0083] 3.3) The aberration-free image I gt The image is then convolved in blocks with a randomly selected point spread function distribution matrix, multiplied by the illumination distribution matrix, and finally distorted using the distortion distribution matrix to output the corresponding aberration image I. ab .

[0084] Wherein, the optical simulation model described in 3.1) is a ray tracing simulation model or a wave optical simulation model; this embodiment uses a ray tracing simulation model; 3.2) the aberration-free image I gt The approach is to directly use an existing public dataset (Flickr2K in this example), where the clear, aberration-free data is used as the ground truth, denoted as I. gt The random selection is a uniform, equally probable random selection.

[0085] The general aberration restoration image model in step 4) includes a degradation feature prior codebook, a high-resolution feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-resolution feature quantization and fusion module, and a high-resolution image reconstruction module, such as... Figure 5 As shown. This invention proposes to enhance the generalization of a general aberration degradation recovery model using a feature prior quantization method. Both the degradation feature prior codebook and the high-resolution feature prior codebook are codebooks composed of multiple discrete vectors, whose vector values ​​are learnable parameters. These are obtained through feature quantization pre-training and are used to store degradation prior information of aberration data and high-resolution prior information of clear, aberration-free data. The degradation feature extraction module and the high-resolution image reconstruction module can be symmetric convolutional neural network structures, such as ResNet or DenseNet. This embodiment uses a ResNet structure to extract depth features from aberration-degraded images and reconstruct high-resolution images based on the fused high-resolution features, respectively. The degradation feature enhancement module is a Transformer structure; this embodiment uses a residual Swin-Transformer module to handle spatially uneven aberration degradation features.

[0086] This invention proposes a degradation / high-definition feature quantization and fusion module, the structure of which is as follows: Figure 6As shown, prior information from degraded / high-resolution images contained in the quantized features is used to jointly promote the recovery of aberration-degraded images. Specifically, feature matching is performed between the extracted / enhanced features and the prior codebook of degraded / high-resolution features. The distance between the feature vector corresponding to each pixel position and each vector in the codebook is calculated. The feature vector at that pixel position in the features is replaced with the codebook vector with the smallest distance. Feature quantization is completed after all feature vectors are replaced. The quantized features are then fused with the original features and input into the next module for processing. The fusion method can be addition, concatenation, affine transformation, cross-attention, etc. In this embodiment, concatenation fusion is used.

[0087] In the feature quantization representation pre-training stage, which obtains the prior codebooks of degraded and high-resolution features, the networks used are an aberration-degraded image reconstruction model and a high-resolution image reconstruction model, respectively. Both models include a feature extraction module, an image reconstruction module, and the aforementioned feature quantization module, which are structurally identical to the degraded feature extraction module, high-resolution image reconstruction module, and degraded / high-resolution feature quantization module in the general aberration restoration image model. The pre-training task for the feature quantization representation pre-training is self-supervised reconstruction of the input image, such as... Figure 7 As shown, the specific steps are as follows (taking the pre-training of the degenerate feature prior codebook as an example; the high-definition feature prior codebook is similar, just change the input data):

[0088] 4.1) Feature Quantization Representation Pre-training

[0089] 4.1.1) Randomly initialize the prior codebook for degenerate features; batch load the dataset, and pre-training uses only a single aberration image dataset I. ab ;

[0090] 4.1.2) The aberration image is input into the aberration degradation image reconstruction model for reconstruction. Specifically, the aberration image is input into the feature extraction module, the extracted features are quantized using the degradation feature prior codebook through the feature quantization module, and the quantized features are input into the image reconstruction module to obtain the reconstructed image I. recon ;

[0091] 4.1.3) Loss function calculation: Calculate I recon with I ab The pixel absolute value loss, perceptual loss, and adversarial generation loss are calculated, and the semantic loss is calculated on the quantized features output by the feature quantization module.

[0092] 4.1.4) Based on the loss calculated in 4.1.3), perform backpropagation to update the feature extraction module, image reconstruction module, and degradation feature prior codebook; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training; if it has not, return to step 4.1.1) and continue training. In this embodiment, the expected number of training iterations is set to 200,000.

[0093] For the aforementioned pre-training results, only the parameters of the degradation feature prior codebook need to be retained during the degradation feature prior codebook training phase; however, during the high-definition feature prior codebook training phase, the parameters of the entire high-definition image reconstruction model are retained. These retained parameters will be used for training the general aberration degradation recovery model, where the parameters of the image reconstruction module in the high-definition image reconstruction model are used to construct the high-definition image reconstruction module in the general aberration degradation recovery model. The aforementioned pre-trained parameters will be frozen during the training phase of the general aberration degradation recovery model.

[0094] Based on the pre-trained degradation feature prior codebook, high-definition feature prior codebook, and parameters of the high-definition image reconstruction model, a general aberration degradation image restoration model is trained. The training flowchart is as follows: Figure 8 As shown, the specific steps are as follows:

[0095] 4.2) Training of a general aberration restoration image model

[0096] 4.2.1) Load the pre-training parameters that are required to be retained as described above and freeze them (without performing gradient calculation and gradient propagation);

[0097] 4.2.2) Batch loading of datasets. The training data consists of aberration-degraded image pairs covering diverse aberration distribution characteristics of the lens database, including aberration images I. ab Aberration-free image I gt ;

[0098] 4.2.3) Convert the aberration image I ab The degraded features are extracted by the degradation feature extraction module, quantized using a priori codebook, and fused with the original degraded features. The fused features are then enhanced by the degradation feature enhancement module, and then input into the high-definition feature quantization and fusion module. Here, they are quantized using a priori codebook and fused with the original enhanced features. Finally, the fused high-definition features are input into the high-definition image reconstruction module, which outputs the restored high-definition image I. restore .

[0099] 4.2.4) Loss function calculation, calculate I restore with I gt The pixel absolute value loss, perceptual loss, and adversarial generation loss are calculated between the high-definition feature quantization and fusion modules, and the quantization loss is calculated for the quantized features generated in the high-definition feature quantization and fusion modules.

[0100] 4.2.5) Based on the loss function calculated in 4.2.4), perform backpropagation to update all modules except the frozen ones; determine whether the current number of training iterations has reached the expected number of training iterations. If it has, end the training; if it has not, return to step 4.2.2) and continue training. In this embodiment, the expected number of training iterations is set to 200,000.

[0101] The quantization loss mentioned in step 4.2.4) is calculated only during high-definition feature quantization. The calculation method is to convert I... gt Input the high-resolution image reconstruction model used for pre-training to obtain I gt The result of quantization using the high-definition feature prior codebook is used as the aberration image I. ab In the above recovery process, the true value of the high-definition feature quantization result is used, and the absolute pixel loss is calculated by combining the true value with the quantization features generated in the high-definition feature quantization and fusion module.

[0102] Finally, the trained general aberration degradation image restoration model can be used to restore aberration degradation images from any imaging lens. Specifically, an image is captured using any imaging lens and camera, which is an aberration degradation image. This image is then input into the trained general aberration degradation image restoration model. After training, the model can match and fuse the degradation information of the input aberration degradation image with a priori codebook of degradation features in the lens library to obtain quantified degradation information to guide the restoration process. Then, the content information of the aberration degradation image is matched and fused with a priori codebook of high-definition features to reconstruct a high-definition restored image corresponding to the content.

[0103] Figure 9 The general aberration restoration image trained in this embodiment shows the restoration results of aberration-degraded images from four different lenses: the structures of these four lenses are as follows. Figure 9 As shown in the first row; its aberration degradation image is as follows: Figure 9 As shown in the second row; Figure 9 The third row shows the restoration results of a general aberration-degraded image restoration model that does not use the feature quantization and fusion module proposed in this invention; Figure 9 The fourth line shows the restoration results of the complete aberration-degraded image restoration model proposed in this invention. These results demonstrate that the general aberration-degraded image restoration model trained on the lens database constructed in this invention can effectively handle aberration-degraded image restoration for lenses with various aberration-degraded characteristics. Furthermore, the degradation / high-definition feature quantization and fusion module proposed in this invention further enhances the model's generalization ability, improving the quality of the restored images under all test lenses.

[0104] It should be noted that the terms "high definition" and "clear" used in this invention are only used to distinguish them from aberration images and modules for processing aberration images, and are not intended to limit this invention.

[0105] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A general aberration-degraded image restoration method based on a lens database, characterized in that, The method includes the following steps: 1) Multiple lens structures are generated using an automatic lens design method, and multiple lens samples are obtained after optimizing the lens structures according to the evaluation function; 2) The lens samples are selected based on their aberration distribution characteristics, and the selected lens samples form a lens database; 3) Obtain an aberration-free image dataset containing multiple aberration-free images. Each aberration-free image is simulated using a lens sample from the lens database to obtain a corresponding aberration image, thus forming an aberration image dataset. 4) Initialize the degraded feature prior codebook and the high-definition feature prior codebook, construct the aberration-degraded image reconstruction model and the high-definition image reconstruction model, and perform feature quantization representation pre-training respectively. Update the degraded feature prior codebook using the pre-training process of the aberration-degraded image reconstruction model, and update the high-definition feature prior codebook using the pre-training process of the high-definition image reconstruction model. After the pre-training is completed, the final degraded feature prior codebook and the high-definition feature prior codebook are obtained. The aberration-degraded image reconstruction model and the high-definition image reconstruction model have the same structure, both including a feature extraction module, a feature quantization module, and an image reconstruction module; 5) Construct a general aberration restoration image model by combining the final degradation feature prior codebook and the high-definition feature prior codebook, and train the general aberration restoration image model using aberration-free image dataset and aberration image dataset; The general aberration restoration image model includes a degradation feature prior codebook, a high-resolution feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-resolution feature quantization and fusion module, and a high-resolution image reconstruction module; the parameters of the high-resolution image reconstruction module inherit the parameters of the image reconstruction module obtained by the feature quantization representation pre-training. The specific steps for training the general aberration restoration image model using aberration-free image dataset and aberration image dataset are as follows: Freeze the degradation feature prior codebook, high-definition feature prior codebook, and high-definition image reconstruction module in the general aberration restoration image model; The aberration image dataset is input into the degradation feature extraction module to extract degradation features. The degradation features are then quantized and fused with the degradation feature prior codebook using the degradation feature quantization and fusion module. The fused degradation features are then enhanced by the degradation feature enhancement module and input into the high-definition feature quantization and fusion module. The high-definition features are quantized using the high-definition feature prior codebook and then concatenated and fused with the enhanced features before quantization. The fused high-definition features are then input into the high-definition image reconstruction module to output the restored high-definition image. 6) Use the trained general aberration restoration image model to restore the aberration-degraded image taken by the actual lens to be restored, and output the restored aberration-free image.

2. The general aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that, Step 1) describes generating multiple lens structures using an automatic lens design method and optimizing these structures based on an evaluation function to obtain multiple lens samples. Specifically: 1.1) Randomly generate multiple lens structures using preset design specifications as the initial lens group P; the preset design specifications are the field of view, F-number, aperture position, and number of lens elements; 1.2) The optimized lens group P is obtained by performing global optimization on the lens group P based on the evaluation function; 1.3) Select 10%-40% of the lenses with higher evaluation function values ​​from the optimized lens group P* to obtain a high-quality lens group Q; 1.4) The optimized high-quality lens group Q* is obtained by performing local optimization on the high-quality lens group Q according to the evaluation function; 1.5) The optimized high-quality lens group Q* is mutated to obtain the mutated lens group M. The optimized high-quality lens group Q* and the mutated lens group M are merged as the update value of the lens group P and returned to step 1.2) Iterative optimization is performed until the number of iterations meets the preset maximum number of iterations. All lens structures in the current optimized high-quality lens group Q* are output as lens samples.

3. The general aberration-degraded image restoration method based on lens database construction according to claim 1 or 2, characterized in that, The value of the evaluation function in step 1) is obtained by weighted summation of image quality evaluation index and physical constraint evaluation index. The image quality evaluation index is the average value of the RMS radius of the dot plot, the modulation transfer function or wave aberration, and the physical constraint evaluation index is the weighted summation of effective focal length, total system length, back working distance, lens center thickness, lens edge thickness and air center distance.

4. The general aberration-degraded image restoration method based on lens database construction according to claim 2, characterized in that, The global optimization in step 1.2) adopts the selection simulated annealing algorithm, particle swarm optimization algorithm or ant colony optimization algorithm; the local optimization in step 1.4) adopts the damped least squares method, ADAM algorithm or quasi-Newton method.

5. The general aberration-degraded image restoration method based on lens database construction according to claim 2, characterized in that, The variation in step 1.5) is to randomly change some parameters of the lens structure, such as curvature, glass thickness, air gap, material refractive index, and material Abbe number. The sum of the variable values ​​of the changed glass thickness and air gap needs to be greater than 1, and the sum of the variable values ​​of the glass thickness and air gap remains consistent before and after the change.

6. The general aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that, The aberration distribution characteristics of the lens samples mentioned in step 2) are the average value of the RMS radius of the point array of each field of view of the lens samples and the absolute value difference between the point spread function arrays of each lens sample. The method of selecting lens samples based on the aberration distribution characteristics of lens samples specifically includes: A range is defined for the average RMS radius of the dot matrix plots for each field of view of the lens samples. Lens samples within this range are sampled to ensure a uniform distribution of the average RMS radius of the dot matrix plots for each field of view of the sampled lens samples. Simultaneously, the absolute difference between the dot matrix spread function array of each sampled lens sample and the dot matrix spread function arrays of all other sampled lens samples is greater than a set threshold, which is 0.25 × 10⁻⁶. -6 -1×10 -6 .

7. The general aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that, In step 3), each aberration-free image is simulated using a lens sample from the lens database to obtain a corresponding aberration image. The specific process is as follows: 3.1) Calculate the point spread function distribution matrix, distortion distribution matrix, and illuminance distribution matrix with respect to the field of view for each lens sample in the lens database; 3.2) For each aberration-free image, randomly select the point spread function distribution matrix, distortion distribution matrix, and illumination distribution matrix as a function of the field of view of a lens sample; 3.3) Perform block convolution between the aberration-free image and a randomly selected point spread function distribution matrix. Multiply the block convolution result by the illumination distribution matrix, and then perform distortion transformation through the distortion distribution matrix to output the corresponding aberration image.

8. The general aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that, Step 4) Both the degraded feature prior codebook and the high-definition feature prior codebook are codebooks composed of multiple discrete vectors, and each discrete vector is a learnable parameter. The steps are as follows: Pre-training is performed on feature quantization representations, the prior codebook of degraded features is updated using the pre-training process of the aberration-degraded image reconstruction model, and the prior codebook of high-definition features is updated using the pre-training process of the high-definition image reconstruction model. For the aberration-degraded image reconstruction model, the aberration image dataset is input into the aberration-degraded image reconstruction model for reconstruction. The reconstruction specifically involves: the feature extraction module extracting the features of the input aberration image; the features of the aberration image being quantized by the feature quantization module using the degraded feature prior codebook; and the quantized features being input into the image reconstruction module to obtain the reconstructed image corresponding to the aberration image. The image reconstruction loss function is calculated, which includes the pixel absolute value loss, perceptual loss and adversarial generation loss between the reconstructed image and the input aberration image, and the semantic loss is calculated on the quantized features output by the feature quantization module. Backward gradient propagation is performed based on the image reconstruction loss function and semantic loss to update the parameters of the feature extraction module, image reconstruction module, and degradation feature prior codebook of the aberration-degraded image reconstruction model; after iterating the expected number of training iterations, the required degradation feature prior codebook is obtained. Similarly, for the high-definition image reconstruction model, the aberration-free image dataset is input into the high-definition image reconstruction model for reconstruction. During the reconstruction process, the feature quantization module uses the high-definition feature prior codebook for quantization, and the reconstructed image corresponding to the aberration-free image is obtained. The image reconstruction loss function is calculated and the semantic loss is calculated for the quantized features during the reconstruction process. Backward gradient propagation is performed based on the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, the image reconstruction module and the high-definition feature prior codebook of the high-definition image reconstruction model. After iterating the expected number of training iterations, the required high-definition feature prior codebook is obtained.

9. The general aberration-degraded image restoration method based on lens database construction according to claim 8, characterized in that, Step 5) further includes training the general aberration restoration image model using the aberration-free image dataset and the aberration image dataset: The image restoration loss function is calculated, which includes the absolute pixel value loss, perceptual loss, and adversarial generation loss between the restored high-resolution image and the input aberration image. The quantization loss is also calculated for the quantized features obtained in the high-resolution feature quantization and fusion module. Based on the image restoration loss function and quantization loss, backpropagation is performed to update all modules in the general aberration restoration image model except for the frozen ones; after iterating the expected number of training iterations, the training ends.

10. The general aberration-degraded image restoration method based on lens database construction according to claim 9, characterized in that, The feature quantization module in the aberration-degraded image reconstruction model and the high-resolution image reconstruction model, the degradation feature quantization and fusion module in the general aberration restoration image model, and the high-resolution feature quantization and fusion module. The methods for quantizing features using feature prior codebooks are all the same, and the specific methods are as follows: The feature to be quantized is matched with the feature prior codebook. The distance between the feature vector corresponding to each pixel position of the feature to be quantized and the codebook vectors in the feature prior codebook is calculated. The feature vector at the pixel position of the feature to be quantized is replaced with the codebook vector with the smallest distance. After replacing the feature vectors at all pixel positions, the feature quantization is completed, and the quantized feature is obtained.

Images