General aberration degraded image restoration method constructed based on lens database
By constructing a lens database and using degraded and high-definition feature prior quantization, a general aberration degradation image recovery model is designed, which solves the problem that the model can only be used for a specific lens in the prior art, and realizes aberration image recovery for any lens. Reduces time costs and improves the generalization performance of the model.
Patent Information
- Application Number
- CN202410927760.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-11
AI Technical Summary
In the prior art, the aberration degradation image recovery model can only be used for a specific lens, and it needs to be retrained when reselling the lens, resulting in high time cost, and the data set is difficult to cover all possible aberration distributions, and the model generalization ability is insufficient.
By constructing a lens database, we automatically design multi-special lens samples, filter samples with different distribution characteristics, combine degradation and high-definition feature prior quantization, and design a general aberration degradation image recovery model to realize one model to restore aberration images of any lens.
It greatly reduces the data preparation and model training time of the computational imaging system, improves the generalization performance of the model, and can recover high-quality aberration degradation images on any imaging lens.
Smart Images

Figure CN118941468A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the fields of optical design, computational imaging and aberration-degraded image restoration, and in particular to a universal aberration-degraded image restoration method based on a lens database. Background Art
[0002] With the widespread application of mobile portable devices such as blind assistive devices, drones, virtual reality glasses, smart robots, and mobile phones, people's demand for optical imaging lenses with high image quality, large field of view, small size, and light weight has gradually increased, which has become a new development trend in the design of optical imaging lenses. However, when designing optical imaging lenses, it is often difficult to balance image quality and size. In order to obtain high-quality imaging quality, commonly used optical imaging lenses usually have the disadvantages of complex structure, large size and weight, which are not conducive to integration and portability.
[0003] In order to solve this problem, the concept of computational imaging came into being. That is, the optical imaging lens is designed to allow uncorrected aberrations in exchange for a lighter structure, and then the aberration-degraded image is restored through the back-end algorithm, so that the imaging quality of the lightweight imaging lens is close to that of the complex imaging lens. The current mainstream aberration-degraded image restoration algorithm uses an image restoration model based on deep learning technology. Before use, the model needs to be trained on the aberration-degraded image-high-definition aberration-free image data corresponding to the imaging lens used. The trained model is only used to restore the aberration-degraded image under this specific lens. However, when the imaging lens structure is changed, the above-mentioned trained model will become invalid due to the change in the aberration distribution of the imaging lens. At this time, it is necessary to re-execute the steps of data set preparation and model training for the new lens structure so that the model can be applied to the new imaging lens. The close connection between the restoration model and the imaging lens structure adds a lot of time cost to the design and deployment of lightweight computational imaging systems, limiting its update and iteration. Therefore, a universal aberration-degraded image restoration method is needed in this field, which can restore the aberration-degraded image of any lightweight imaging lens by training only one model. The difficulties of this technology are: 1) data set preparation. There are many types of imaging lenses, and the aberration degradation distribution is different. It is difficult to cover all possible aberration distributions with one data set; 2) model design. Even if the training data can cover a variety of aberrations, the model needs to have strong generalization ability and the ability to analyze and process different aberration degradations in order to achieve universal aberration degradation restoration. Summary of the invention
[0004] In order to solve the problems existing in the background technology, the present invention proposes a universal aberration-degraded image restoration method based on a lens database.
[0005] Among them, the present invention proposes an automatic imaging lens design method to construct a lens database. The automatic imaging lens design method can efficiently search the solution space of different structures of imaging lenses, generate rich, multi-specification imaging lens samples, and the constructed lens library covers a large number of aberration degradations with different distribution characteristics and different degrees of severity, providing a reliable training data set for a universal aberration-degraded image restoration model; the present invention also proposes to simultaneously use degradation and high-definition feature prior quantization to improve the generalization of the aberration-degraded restoration model, thereby designing a structure of a universal aberration-degraded image restoration model, which realizes the restoration of the aberration-degraded image of any imaging lens using only one trained model.
[0006] The present invention is achieved through the following technical solutions:
[0007] A general aberration-degraded image restoration method based on a lens database comprises the following steps:
[0008] 1) Generate multiple lens structures using an automatic lens design method, and obtain multiple lens samples after optimizing the lens structure according to an evaluation function;
[0009] 2) screening the lens samples based on the aberration distribution characteristics of the lens samples, and forming a lens database with the screened lens samples;
[0010] 3) obtaining an aberration-free image data set including a plurality of aberration-free images, wherein each aberration-free image is simulated through a lens sample in the lens database to obtain a corresponding aberration image, thereby forming an aberration image data set;
[0011] 4) Initializing the degradation feature prior codebook and the high-definition feature prior codebook, constructing the aberration-degraded image reconstruction model and the high-definition image reconstruction model, and performing feature quantization representation pre-training respectively, updating the degradation feature prior codebook by using the pre-training process of the aberration-degraded image reconstruction model, and updating the high-definition feature prior codebook by using the pre-training process of the high-definition image reconstruction model, and obtaining the final degradation feature prior codebook and high-definition feature prior codebook after the pre-training is completed;
[0012] 5) constructing a universal aberration restoration image model by combining the final degradation feature prior codebook and the high-definition feature prior codebook, and training the universal aberration restoration image model using the aberration-free image dataset and the aberration image dataset;
[0013] 6) Using the trained universal aberration restoration image model, the aberration-degraded image captured by the actual lens to be restored is restored, and a restored aberration-free image is output.
[0014] Preferably, the automatic lens design method is used in step 1) to generate multiple lens structures, and multiple lens samples are obtained after optimization according to the evaluation function, specifically:
[0015] 1.1) randomly generating multiple lens structures using preset design specifications as the initial lens group P; the preset design specifications are field angle, F number, aperture position and number of lens pieces;
[0016] 1.2) Performing global optimization on the lens group P according to the evaluation function to obtain the optimized lens group P*;
[0017] 1.3) Selecting 10%-40% of lenses with higher evaluation function values from the optimized lens group P* to obtain a high-quality lens group Q;
[0018] 1.4) Locally optimizing the high-quality lens group Q according to the evaluation function to obtain an optimized high-quality lens group Q*;
[0019] 1.5) The optimized high-quality lens group Q* is mutated to obtain a variant lens group M. The optimized high-quality lens group Q* is combined with the variant lens group M as the updated value of the lens group P and returns to step 1.2) to iterate the optimization until the number of iterations meets the preset maximum number of iterations, and all lens structures in the current optimized high-quality lens group Q* are output as lens samples.
[0020] Preferably, the value of the evaluation function in step 1) is obtained by weighted summation of an image quality evaluation index and a physical constraint evaluation index, wherein the image quality evaluation index is the average value of the RMS radius of the point diagram, the modulation transfer function or the wave aberration, and the physical constraint evaluation index is the weighted summation of the effective focal length, the total length of the system, the back working distance, the lens center thickness, the lens edge thickness and the air center distance.
[0021] Preferably, the global optimization in step 1.2) adopts a simulated annealing algorithm, a particle swarm algorithm or an ant colony algorithm; the local optimization in step 1.4) adopts a damped least squares method, an ADAM algorithm or a quasi-Newton method.
[0022] Preferably, the variation in step 1.5) is to randomly change some parameters of the curvature of the lens structure, glass thickness, air spacing, material refractive index and material Abbe number, and the sum of the variable numbers of the changed glass thickness and space spacing needs to be greater than 1, and the sum of the variable values of the glass thickness and space spacing remains consistent before and after the change.
[0023] Preferably, the aberration distribution characteristics of the lens sample in step 2) are the average value of the RMS radius of the point diagram of each field of view of the lens sample and the absolute value difference between the point spread function arrays of each lens sample;
[0024] The lens samples are screened based on the aberration distribution characteristics of the lens samples, specifically, the average value range of the RMS radius of the spot diagram of each field of view of the lens samples is set, and the lens samples within the range are sampled so that the average values of the RMS radius of the spot diagram of each field of view of the lens samples after sampling are uniformly distributed, and at the same time, the absolute value difference between the point spread function array of each sampled lens sample and the point spread function arrays of all other sampled lens samples is greater than a set threshold value, and the set threshold value is 0.25×10 -6 -1×10 -6 .
[0025] Preferably, each aberration-free image in step 3) is simulated by a lens sample in the lens database to obtain a corresponding aberration image, and the specific process is as follows:
[0026] 3.1) Calculate the point spread function distribution matrix, distortion distribution matrix and illumination distribution matrix with field of view for each lens sample in the lens database;
[0027] 3.2) For each aberration-free image, randomly select a lens sample’s point spread function distribution matrix, distortion distribution matrix, and illumination distribution matrix with field of view;
[0028] 3.3) Perform block convolution on the aberration-free image and the randomly selected point spread function distribution matrix, multiply the block convolution result by the illumination distribution matrix, and then perform distortion transformation through the distortion distribution matrix to output the corresponding aberration image.
[0029] Preferably, the aberration-degraded image reconstruction model and the high-definition image reconstruction model in step 4) have the same structure, both comprising a feature extraction module, a feature quantization module and an image reconstruction module;
[0030] The degraded feature prior codebook and the high-definition feature prior codebook are both codebooks composed of multiple discrete vectors, and the discrete vectors are all learnable parameters;
[0031] The feature quantization representation pre-training is performed respectively, and the degradation feature priori codebook is updated by using the pre-training process of the aberration-degraded image reconstruction model, and the high-definition feature priori codebook is updated by using the pre-training process of the high-definition image reconstruction model. The specific steps are:
[0032] For the aberration-degraded image reconstruction model, the aberration image data set is input into the aberration-degraded image reconstruction model for reconstruction, wherein the reconstruction is specifically as follows: a feature extraction module extracts features of the input aberration image, the features of the aberration image are quantized by a feature quantization module using a degradation feature priori codebook, and the quantized features are input into an image reconstruction module to obtain a reconstructed image corresponding to the aberration image;
[0033] Calculating an image reconstruction loss function, wherein the image reconstruction loss function includes a pixel absolute value loss, a perceptual loss, and an adversarial generation loss between the reconstructed image and the input disparity image, and calculating a semantic loss for the quantized features output by the feature quantization module;
[0034] Backward gradient propagation is performed according to the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, the image reconstruction module and the degradation feature prior codebook of the aberration-degraded image reconstruction model; after iterating the expected number of training iterations, the required degradation feature prior codebook is obtained;
[0035] Similarly, for the high-definition image reconstruction model, the aberration-free image data set is input into the high-definition image reconstruction model for reconstruction. During the reconstruction process, the feature quantization module uses the high-definition feature prior codebook for quantization to reconstruct the reconstructed image corresponding to the aberration-free image; the image reconstruction loss function is calculated and the semantic loss is calculated for the quantized features during the reconstruction process, and reverse gradient propagation is performed based on the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, image reconstruction module and high-definition feature prior codebook of the high-definition image reconstruction model; after iterating the expected number of training iterations, the required high-definition feature prior codebook is obtained.
[0036] Preferably, the universal aberration restoration image model described in step 5) includes a degradation feature prior codebook, a high-definition feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-definition feature quantization and fusion module, and a high-definition image reconstruction module; the parameters of the high-definition image reconstruction module inherit the parameters of the image reconstruction module obtained by the feature quantization representation pre-training;
[0037] The specific steps of using the aberration-free image dataset and the aberration image dataset to train the universal aberration restoration image model are as follows:
[0038] Freezing the degradation feature priori codebook, the high-definition feature priori codebook and the high-definition image reconstruction module in the universal aberration restoration image model;
[0039] The aberration image data set is input into the degradation feature extraction module to extract the degradation features. The degradation features are quantized by the degradation feature quantization and fusion module using the degradation feature prior codebook and fused with the degradation features before quantization. The fused degradation features are enhanced by the degradation feature enhancement module and then input into the high-definition feature quantization and fusion module. The high-definition features are quantized by the high-definition feature prior codebook and fused with the enhanced features before quantization. The high-definition features are obtained after fusion. The high-definition features are finally input into the high-definition image reconstruction module to output the restored high-definition image.
[0040] Calculating an image restoration loss function, the image restoration loss function including a pixel absolute value loss, a perceptual loss, and an adversarial generation loss between a restored high-definition image and an input aberration image, and calculating a quantization loss for the quantized features obtained in the high-definition feature quantization and fusion module;
[0041] According to the image restoration loss function and the quantization loss, reverse gradient propagation is performed to update all modules except the frozen modules in the general aberration restoration image model; after iterating the expected number of training iterations, the training is terminated.
[0042] Preferably, for the feature quantization module in the aberration-degraded image reconstruction model and the high-definition image reconstruction model features, the degradation feature quantization and fusion module in the general aberration restoration image model, and the high-definition feature quantization and fusion module, the method for quantizing the features using the feature prior codebook is the same, and the specific method is:
[0043] The feature to be quantized is matched with the feature prior codebook, and the distance between the feature vector corresponding to each pixel position of the feature to be quantized and each codebook vector in the feature prior codebook is calculated. The codebook vector with the minimum distance is used to replace the feature vector of the pixel position in the feature to be quantized. After replacing the feature vectors of all pixel positions, feature quantization is completed to obtain the quantized feature.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] 1) The present invention proposes a lens database construction method, which can automatically search the solution space under preset design specifications and evaluation functions, and output a variety of lens structures that meet the conditions and are optimized; the lens database obtained after screening the lens structure covers different aberration distribution characteristics. The lens database construction method of the present invention solves the problem of missing training data for the above-mentioned general model, and by proposing an automatic optical design method to generate a large number of lens samples with multiple specifications, the generalization performance of the general aberration-degraded image restoration algorithm is greatly improved.
[0046] 2) Since the aberration-degraded image restoration of the present invention adopts the process of prior pre-training-matching reconstruction, and decomposes the degradation features and content features, the universal aberration-degraded image restoration model can extract high-definition content information from different types of aberration-degraded images, and therefore has strong generalization ability, and can perform high-quality restoration of aberration-degraded images of any imaging lens. The present invention makes up for the limitation that the traditional aberration-degraded image restoration method based on deep learning can only restore the image of a single specific lens, and realizes the use of a universal model to restore the aberration-degraded image of any lens, which greatly saves the time cost of data preparation and model training of the computational imaging system. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 An overview of the general aberration-degraded image restoration method based on the lens database;
[0048] Figure 2 A comparison chart between the general aberration-degraded image restoration method and the traditional aberration-degraded image restoration method;
[0049] Figure 3 Design algorithm flow chart for automatic imaging lens;
[0050] Figure 4 is the lens sample distribution diagram;
[0051] Figure 5 It is a schematic diagram of the structure of the universal aberration degradation restoration model;
[0052] Figure 6 It is a schematic diagram of the feature quantization module;
[0053] Figure 7 Schematic diagram of pre-training for feature quantization representation;
[0054] Figure 8 Training flow chart for general aberration degradation restoration model;
[0055] Fig. 9 These are the test results of the general aberration degradation restoration model. DETAILED DESCRIPTION
[0056] In order to make the technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0057] Figure 1 The invention provides an overview of a universal aberration-degraded image restoration method based on a lens database, wherein the invention uses an automatic lens design algorithm to automatically search a solution space under preset design specifications and evaluation functions, and outputs a variety of lens structures that meet conditions and are optimized; the lens structures are screened to form a lens database, and the lens database covers different aberration distribution characteristics; the lens structures and clear aberration-free image data of the lens database are input into an optical simulation model to obtain aberration-degraded image data containing a variety of aberration distributions, and form a data pair with the clear aberration-free image data; the data pair is used to train a universal aberration-degraded image restoration model, and the universal aberration-degraded image restoration model uses pre-trained degradation feature priors and high-definition feature priors, can extract high-definition content information from different types of aberration-degraded images, has strong generalization ability, and the trained model can perform high-quality restoration of aberration-degraded images of any imaging lens.
[0058] The present invention proposes a universal aberration-degraded image restoration method based on a lens database, which comprises the following steps:
[0059] 1) Automatic lens design: Input design specifications, set evaluation functions, generate possible lens structures under corresponding specifications through automatic lens design algorithms, and output lens samples;
[0060] 2) Lens sample screening: Based on the aberration distribution characteristics of lens samples, set the sampling strategy and sampling quantity, screen samples and build a lens database;
[0061] 3) Construction of a general aberration-degraded image restoration dataset: Obtain a dataset of aberration-free images. gt , the aberration image I corresponding to the lens in the lens database is simulated by the simulation algorithm ab , forming the image data pair {I gt , I ab}, all image data pairs constitute a universal aberration-degraded image restoration dataset;
[0062] 4) Model training and application: Feature quantization representation pre-training is used to obtain a feature prior codebook, and a universal aberration restoration image model is constructed in combination with the feature prior codebook. The universal aberration restoration image model is trained, and the trained universal aberration restoration image model is used to restore the aberration-degraded image of any lens and output a high-quality restored image.
[0063] like Figure 3 As shown, compared with the traditional aberration-degraded image restoration method, the universal aberration-degraded image restoration method proposed in the present invention, which is constructed based on the lens database, only needs to be trained once using the lens database to obtain a universal aberration-degraded image restoration model, and can restore the aberration-degraded images of lenses with various aberration degraded lenses, which greatly saves the time cost of data preparation and algorithm training of the computational imaging system, and also broadens the application scope and convenience of the model.
[0064] The automatic lens design algorithm flow chart in step 1) is as follows: Figure 3 As shown, the steps are:
[0065] 1.1) Randomly initialize the lens group P according to the input design specifications.
[0066] 1.2) According to the input evaluation function, P is globally optimized to obtain P*.
[0067] 1.3) Based on the input evaluation function, select a portion of lenses with higher evaluation function values from P* to obtain Q.
[0068] 1.4) According to the input evaluation function, Q is locally optimized to obtain Q*.
[0069] 1.5) If the number of iterations reaches the expected number of training iterations, output Q*; otherwise, Q* mutates to obtain M, Q* and M are merged into P and return to step 1.2) to continue optimization; in this embodiment, the expected number of training iterations is set to 10.
[0070] The design specifications in step 1.1) include field of view angle, F number, aperture position and number of lens elements, wherein the half field of view angle sampling range is 20 degrees to 40 degrees, the sampling interval is 2 degrees, the F number sampling range is 2.0 to 5.0, the sampling interval is 0.3, the aperture stop position sampling position is all possible positions of the aperture stop under a given number of elements, and the number of lens elements range is set to 1 to 6 elements.
[0071] The random initialization in step 1.1) is to perform parameter normalization according to the value range of the lens curvature, glass thickness, air spacing, material refractive index and material Abbe number, and obtain the initial lens group by randomly generating a parameter vector group in the range of 0 to 1.
[0072] The input parameters of the evaluation function in steps 1.1)-1.4) are the radius of curvature, glass thickness, air spacing, material refractive index and material Abbe number of each lens in the lens group, and the evaluation function value of each lens in the lens group is calculated by the ray tracing simulation algorithm. The lower the evaluation function value of the lens, the higher the quality of the lens. The evaluation function is obtained by weighted summation of image quality evaluation and physical constraint evaluation. The image quality evaluation can adopt traditional optical design evaluation indicators such as spot diagram, modulation transfer function (MTF), wave aberration, etc. The present embodiment adopts the average value of the RMS radius of the spot diagram of all sampled fields of view and wavelengths. The physical constraint evaluation includes lens curvature, lens thickness, air spacing, material refractive index, material Abbe number, system total length, back working distance, and edge image height. The physical constraint evaluation is the weighted summation of the above parameters, wherein the lens curvature constraint range is -0.1 to 0.1, the lens thickness constraint range is 4 mm to 15 mm, the air spacing constraint range is 1 mm to 15 mm, the material refractive index constraint range is 1.51 to 1.95, and the material Abbe number constraint range is 18.9 to 81.6. Assuming that the number of lenses is x, the system total length constraint range is not more than (x*15+25) mm, the back working distance constraint range is above 15 mm, and the edge image height constraint range is 14.2 mm to 14.4 mm. Under these settings, the automatic lens design algorithm in this embodiment can output approximately 12,000 lens samples.
[0073] For the global optimization in step 1.2), a heuristic global search algorithm such as simulated annealing algorithm, particle swarm algorithm, ant colony algorithm, etc. can be selected; for the local optimization in step 1.4), a local optimization algorithm such as damped least squares method, ADAM algorithm, quasi-Newton method, etc. can be selected. In this embodiment, the global optimization adopts the simulated annealing algorithm, and the local optimization adopts the ADAM algorithm.
[0074] In the screening in step 1.3), the lens group is sorted according to the evaluation function value, and 10%-40% of the lenses are selected from high to low. If there are two similar lenses, only the lens with the better evaluation function value is retained; in this embodiment, 20% of the lenses are screened from the lens group.
[0075] The aberration distribution characteristics of the lens sample in step 2) are specifically:
[0076] a) The average value of the RMS radius of the spot diagram of each field of view of the lens sample, indicating the severity of the aberration of the lens sample;
[0077] b) The absolute value difference between the point spread function arrays of each lens sample, indicating the degree of difference between the lens samples;
[0078] The output of about 12,000 lens samples, the average distribution of the point diagram RMS radius of each field of view and the distribution of each specification are shown as follows: Figure 4 shown.
[0079] The sampling strategy used in step 2) requires that the sampled lens samples cover the evenly distributed lens sample aberration severity, and the difference between the lens samples is greater than the set threshold. Specifically, in this embodiment, the average value of the RMS radius of the point diagram of each field of view of the lens sample is set to 0 to 0.2. In this interval, the 12,000 lens samples generated are sampled so that the average value of the RMS radius of the point diagram of each field of view of the lens sample after sampling is evenly distributed between 0 and 0.2; at the same time, for each sampled sample, it is necessary to calculate the absolute value difference between its point spread function array and the point spread function array of all sampled samples. When the absolute value difference is greater than the set threshold, it is considered that the difference of the sampled lens sample meets the requirement and the lens sample is accepted; in this embodiment, the threshold of the absolute value difference is set to 0.35×10 -6 ,After this screening and sampling process, 1000 lens samples are obtained to form the lens ,database.
[0080] Based on the constructed lens database, the steps for constructing the universal aberration-degraded image restoration dataset described in step 3) are as follows:
[0081] 3.1) Calculate the point spread function, distortion, and illumination distribution matrix of each lens in the lens database through the optical simulation model;
[0082] 3.2) For each aberration-free image I gt , randomly select a lens in the lens database and calculate the point spread function, distortion, and illumination distribution matrix with field of view in 3.1); the selection of the first aberration-free image does not affect the selection of the subsequent aberration-free image;
[0083] 3.3) The aberration-free image I gt Perform block convolution with the randomly selected point spread function distribution matrix, multiply the result by the illumination distribution matrix, and finally perform distortion transformation through the distortion distribution matrix to output the corresponding aberration image I ab .
[0084] The optical simulation model described in 3.1) is a ray tracing simulation model or a wave optics simulation model. This embodiment uses a ray tracing simulation model. The aberration-free image I described in 3.2) gt The existing public data set (Flickr2K in this embodiment) is directly used, in which the clear and aberration-free data is taken as the true value, denoted as I gt ; The random selection is a uniform random selection with equal probability.
[0085] The general aberration restoration image model in step 4) includes a degradation feature prior codebook, a high-definition feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-definition feature quantization and fusion module, and a high-definition image reconstruction module. Figure 5 As shown. The present invention proposes to enhance the generalization of the general aberration degradation restoration model by using a feature prior quantization method, wherein the degradation feature prior codebook and the high-definition feature prior codebook structure are both codebooks composed of multiple discrete vectors, and their vector values are all learnable parameters, which are obtained through feature quantization characterization pre-training, and are used to store the degradation prior information of aberration data and the high-definition prior information of clear aberration-free data. The degradation feature extraction module and the high-definition image reconstruction module can be symmetrical convolutional neural network structures, such as ResNet, DenseNet, etc. This embodiment uses the ResNet structure, which is respectively used to extract the deep features of the aberration-degraded image and reconstruct the restored high-definition image based on the high-definition features obtained by fusion; the degradation feature enhancement module is a Transformer structure, and this embodiment uses the residual Swin-Transformer module to process the aberration degradation features with uneven spatial distribution.
[0086] The present invention proposes a degradation / high-definition feature quantization and fusion module, the structure of which is as follows: Figure 6As shown, the prior information of the degraded / high-definition image contained in the quantized features is used to jointly promote the restoration of the aberration-degraded image. Specifically, feature matching is performed on the extracted / enhanced features and the degraded / high-definition feature prior codebook, the distance between the feature vector corresponding to each pixel position and each vector in the codebook is calculated, and the feature vector of the pixel position in the feature is replaced with the codebook vector with the minimum distance. After all feature vectors are replaced, feature quantization is completed; the quantized features are then fused with the original features and then input into the next module for processing; the fusion method can be addition, splicing, affine transformation, cross attention, etc., and the splicing fusion method is adopted in this embodiment.
[0087] The feature quantization representation pre-training stage of obtaining the degradation feature prior codebook and the high-definition feature prior codebook uses aberration degradation image reconstruction model and high-definition image reconstruction model, respectively, and their structures include feature extraction module, image reconstruction module and feature quantization module, which are the same as the degradation feature extraction module, high-definition image reconstruction module and degradation / high-definition feature quantization module in the general aberration restoration image model. The feature quantization representation pre-training task is to self-supervise the reconstruction of the input image, such as Figure 7 As shown, the specific steps are as follows (taking the pre-training of the degraded feature prior codebook as an example, the high-definition feature prior codebook is the same, just replace the input data):
[0088] 4.1) Feature Quantization Representation Pre-training
[0089] 4.1.1) Randomly initialize the degradation feature prior codebook; batch load the dataset, and pre-train only use a single aberration image dataset I ab ;
[0090] 4.1.2) Input the aberration image into the aberration degraded image reconstruction model for reconstruction, specifically: input the aberration image into the feature extraction module, the extracted features are quantized by the feature quantization module using the degradation feature prior codebook, and the quantized features are input into the image reconstruction module to obtain the reconstructed image I recon ;
[0091] 4.1.3) Loss function calculation: Calculation I recon with I ab The pixel absolute value loss, perceptual loss and adversarial generation loss are calculated between them, and the semantic loss is calculated for the quantized features output by the feature quantization module;
[0092] 4.1.4) According to the loss calculated in 4.1.3), reverse gradient propagation is performed to update the feature extraction module, image reconstruction module and degradation feature prior codebook; it is determined whether the current number of training iterations reaches the expected number of training iterations. If so, the training is terminated; if not, the training is continued by returning to step 4.1.1); in this embodiment, the expected number of training iterations is set to 200,000.
[0093] For the above pre-training results, in the degradation feature prior codebook training stage, only the parameters of the degradation feature prior codebook need to be retained; and in the high-definition feature prior codebook training stage, the parameters of the entire high-definition image reconstruction model are retained; the above retained parameters will be used for the training of the general aberration degradation restoration model, wherein the parameters of the image reconstruction module in the high-definition image reconstruction model are used to construct the high-definition image reconstruction module in the general aberration degradation restoration model. The above pre-trained parameters will be frozen in the training stage of the general aberration degradation restoration model.
[0094] Based on the degradation feature prior codebook obtained by pre-training, the high-definition feature prior codebook and the parameters of the high-definition image reconstruction model, the general aberration-degraded image restoration model is trained. The training flow chart is as follows: Figure 8 As shown, the specific steps are:
[0095] 4.2) General aberration restoration image model training
[0096] 4.2.1) Load the pre-trained parameters required to be retained and freeze them (without gradient calculation and gradient propagation);
[0097] 4.2.2) Batch loading of data sets. The training data is aberration-degraded image data pairs covering various aberration distribution characteristics of the lens database, including aberration image I ab , aberration-free image I gt ;
[0098] 4.2.3) The aberration image I ab The degradation feature extraction module is input, and the extracted degradation feature is quantized and fused with the original degradation feature by the degradation feature quantization and fusion module using the degradation feature prior codebook; the fused degradation feature is enhanced by the degradation feature enhancement module, and then input into the high-definition feature quantization and fusion module, quantized by the high-definition feature prior codebook and fused with the original enhanced feature; the fused high-definition feature is finally input into the high-definition image reconstruction module, and the restored high-definition image I is output. restore .
[0099] 4.2.4) Loss function calculation, calculation I restore with I gt The pixel absolute value loss, perceptual loss and adversarial generation loss are calculated between them, and the quantization loss is calculated for the quantized features generated in the HD feature quantization and fusion modules;
[0100] 4.2.5) According to the loss function calculated in 4.2.4), reverse gradient propagation is performed to update all the above modules except the frozen ones; it is determined whether the current number of training iterations reaches the expected number of training iterations. If it reaches, the training is terminated; if not, the training is continued by returning to step 4.2.2); in this embodiment, the expected number of training iterations is set to 200,000;
[0101] The quantization loss described in step 4.2.4) is only calculated when the high-definition features are quantized. The calculation method is to convert I gt Input the high-definition image reconstruction model used in pre-training to obtain I gt The result of quantization by the high-definition feature prior codebook is used as the aberration image I ab The true value of the high-definition feature quantization result in the above restoration process is used to calculate the pixel absolute value loss together with the quantization features generated in the high-definition feature quantization and fusion module.
[0102] Finally, the trained universal aberration-degraded image restoration model can be used to restore the aberration-degraded image of any imaging lens. Specifically, an image is taken with any imaging lens and camera. The image is an aberration-degraded image. The image is input into the trained universal aberration-degraded image restoration model. After training, the model can match and fuse the degradation information of the input aberration-degraded image with the degradation feature prior codebook on the lens library to obtain quantized degradation information to guide the restoration process; then the content information of the aberration-degraded image is matched and fused with the high-definition feature prior codebook to reconstruct a high-definition restored image corresponding to the content.
[0103] Fig. 9 The results of restoring the aberration-degraded images of four different lenses using the universal aberration-restored image trained in this embodiment are as follows: Fig. 9 As shown in the first row; its aberration-degraded image is Fig. 9 As shown in the second row; Fig. 9 The third row is the restoration result of the general aberration-degraded image restoration model without using the feature quantization and fusion module proposed in the present invention; Fig. 9 The fourth row is the restoration result of the complete aberration-degraded image restoration model proposed by the present invention. The result shows that the universal aberration-degraded image restoration model trained by the lens database constructed by the present invention can effectively handle the aberration-degraded image restoration work of lenses with different aberration degradation characteristics; at the same time, the degradation / HD feature quantization and fusion module proposed by the present invention can further improve the generalization ability of the model and enhance the quality of the restored image under all test lenses.
[0104] It should be noted that the terms high definition, clarity, etc. mentioned in the present invention are only used to distinguish from aberration images and modules for processing aberration images, and are not used to limit the present invention.
[0105] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A general aberration-degraded image restoration method based on a lens database, characterized in that: The method comprises the following steps: 1) Generate multiple lens structures using an automatic lens design method, and obtain multiple lens samples after optimizing the lens structure according to an evaluation function; 2) screening the lens samples based on the aberration distribution characteristics of the lens samples, and forming a lens database with the screened lens samples; 3) obtaining an aberration-free image data set including a plurality of aberration-free images, wherein each aberration-free image is simulated through a lens sample in the lens database to obtain a corresponding aberration image, thereby forming an aberration image data set; 4) Initializing the degradation feature prior codebook and the high-definition feature prior codebook, constructing the aberration-degraded image reconstruction model and the high-definition image reconstruction model, and performing feature quantization representation pre-training respectively, updating the degradation feature prior codebook by using the pre-training process of the aberration-degraded image reconstruction model, and updating the high-definition feature prior codebook by using the pre-training process of the high-definition image reconstruction model, and obtaining the final degradation feature prior codebook and high-definition feature prior codebook after the pre-training is completed; 5) constructing a universal aberration restoration image model by combining the final degradation feature prior codebook and the high-definition feature prior codebook, and training the universal aberration restoration image model using the aberration-free image dataset and the aberration image dataset; 6) Using the trained universal aberration restoration image model, the aberration-degraded image taken by the actual lens to be restored is restored, and a restored aberration-free image is output.
2. The general aberration-degraded image restoration method based on the lens database according to claim 1, characterized in that: The automatic lens design method described in step 1) generates multiple lens structures, and optimizes the lens structure according to the evaluation function to obtain multiple lens samples, specifically: 1.1) randomly generating multiple lens structures using preset design specifications as the initial lens group P; the preset design specifications are field angle, F number, aperture position and number of lens pieces; 1.2) Performing global optimization on the lens group P according to the evaluation function to obtain the optimized lens group P*; 1.3) Selecting 10%-40% of lenses with higher evaluation function values from the optimized lens group P* to obtain a high-quality lens group Q; 1.4) Locally optimizing the high-quality lens group Q according to the evaluation function to obtain an optimized high-quality lens group Q*; 1.5) The optimized high-quality lens group Q* is mutated to obtain a variant lens group M. The optimized high-quality lens group Q* is combined with the variant lens group M as the updated value of the lens group P and returns to step 1.2) to iterate the optimization until the number of iterations meets the preset maximum number of iterations, and all lens structures in the current optimized high-quality lens group Q* are output as lens samples.
3. The general aberration-degraded image restoration method based on lens database construction according to claim 1 or 2, characterized in that: The value of the evaluation function in the step 1) is obtained by weighted summation of an image quality evaluation index and a physical constraint evaluation index, wherein the image quality evaluation index is the average value of the RMS radius of the point diagram, the modulation transfer function or the wave aberration, and the physical constraint evaluation index is the weighted summation of the effective focal length, the total length of the system, the back working distance, the lens center thickness, the lens edge thickness and the air center spacing.
4. The universal aberration-degraded image restoration method based on lens database construction according to claim 2, characterized in that: The global optimization in step 1.2) adopts the simulated annealing algorithm, particle swarm algorithm or ant colony algorithm; the local optimization in step 1.4) adopts the damped least squares method, ADAM algorithm or quasi-Newton method.
5. The universal aberration-degraded image restoration method based on lens database construction according to claim 2, characterized in that: The variation in step 1.5) is to randomly change some parameters of the lens structure, such as curvature, glass thickness, air spacing, material refractive index and material Abbe number. The sum of the variable numbers of the changed glass thickness and space spacing needs to be greater than 1, and the sum of the variable values of the glass thickness and space spacing remains consistent before and after the change.
6. The universal aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that: The aberration distribution characteristics of the lens sample in step 2) are the average value of the RMS radius of the point diagram of each field of view of the lens sample and the absolute value difference between the point spread function arrays of each lens sample; The method of screening the lens samples based on the aberration distribution characteristics of the lens samples is specifically as follows: The average value range of the RMS radius of the spot diagram of each field of view of the lens sample is set, and the lens samples within the range are sampled so that the average value of the RMS radius of the spot diagram of each field of view of the lens sample after sampling is evenly distributed. At the same time, the absolute value difference between the point spread function array of each sampled lens sample and the point spread function array of all other sampled lens samples is greater than the set threshold value, and the set threshold value is 0.25×10 -6 -1×10 -6 .
7. The universal aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that: Each aberration-free image in step 3) is simulated by a lens sample in the lens database to obtain a corresponding aberration image. The specific process is as follows: 3.1) Calculate the point spread function distribution matrix, distortion distribution matrix and illumination distribution matrix with field of view for each lens sample in the lens database; 3.2) For each aberration-free image, randomly select a lens sample’s point spread function distribution matrix, distortion distribution matrix, and illumination distribution matrix with field of view; 3.3) Perform block convolution on the aberration-free image and the randomly selected point spread function distribution matrix, multiply the block convolution result by the illumination distribution matrix, and then perform distortion transformation through the distortion distribution matrix to output the corresponding aberration image.
8. The universal aberration-degraded image restoration method based on lens database construction according to claim 1, characterized in that: The aberration-degraded image reconstruction model and the high-definition image reconstruction model in step 4) have the same structure, both including a feature extraction module, a feature quantization module and an image reconstruction module; The degraded feature prior codebook and the high-definition feature prior codebook are both codebooks composed of multiple discrete vectors, and the discrete vectors are all learnable parameters; The feature quantization representation pre-training is performed respectively, and the degradation feature priori codebook is updated by using the pre-training process of the aberration-degraded image reconstruction model, and the high-definition feature priori codebook is updated by using the pre-training process of the high-definition image reconstruction model. The specific steps are: For the aberration-degraded image reconstruction model, the aberration image data set is input into the aberration-degraded image reconstruction model for reconstruction, wherein the reconstruction is specifically as follows: a feature extraction module extracts features of the input aberration image, the features of the aberration image are quantized by a feature quantization module using a degradation feature priori codebook, and the quantized features are input into an image reconstruction module to obtain a reconstructed image corresponding to the aberration image; Calculating an image reconstruction loss function, wherein the image reconstruction loss function includes a pixel absolute value loss, a perceptual loss, and an adversarial generation loss between the reconstructed image and the input disparity image, and calculating a semantic loss for the quantized features output by the feature quantization module; Backward gradient propagation is performed according to the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, the image reconstruction module and the degradation feature prior codebook of the aberration-degraded image reconstruction model; after iterating the expected number of training iterations, the required degradation feature prior codebook is obtained; Similarly, for the high-definition image reconstruction model, the aberration-free image data set is input into the high-definition image reconstruction model for reconstruction. During the reconstruction process, the feature quantization module uses the high-definition feature prior codebook for quantization to reconstruct the reconstructed image corresponding to the aberration-free image; the image reconstruction loss function is calculated and the semantic loss is calculated for the quantized features during the reconstruction process, and reverse gradient propagation is performed based on the image reconstruction loss function and the semantic loss to update the parameters of the feature extraction module, image reconstruction module and high-definition feature prior codebook of the high-definition image reconstruction model; after iterating the expected number of training iterations, the required high-definition feature prior codebook is obtained.
9. The universal aberration-degraded image restoration method based on lens database construction according to claim 8, characterized in that: The universal aberration restoration image model described in step 5) includes a degradation feature prior codebook, a high-definition feature prior codebook, a degradation feature extraction module, a degradation feature quantization and fusion module, a degradation feature enhancement module, a high-definition feature quantization and fusion module, and a high-definition image reconstruction module; the parameters of the high-definition image reconstruction module inherit the parameters of the image reconstruction module obtained by the feature quantization representation pre-training; The specific steps of using the aberration-free image dataset and the aberration image dataset to train the universal aberration restoration image model are as follows: Freezing the degradation feature priori codebook, the high-definition feature priori codebook and the high-definition image reconstruction module in the universal aberration restoration image model; The aberration image data set is input into the degradation feature extraction module to extract the degradation features. The degradation features are quantized by the degradation feature quantization and fusion module using the degradation feature prior codebook and fused with the degradation features before quantization. The fused degradation features are enhanced by the degradation feature enhancement module and then input into the high-definition feature quantization and fusion module. The high-definition features are quantized by the high-definition feature prior codebook and fused with the enhanced features before quantization. The high-definition features are obtained after fusion. The high-definition features are finally input into the high-definition image reconstruction module to output the restored high-definition image. Calculating an image restoration loss function, the image restoration loss function including a pixel absolute value loss, a perceptual loss, and an adversarial generation loss between a restored high-definition image and an input aberration image, and calculating a quantization loss for the quantized features obtained in the high-definition feature quantization and fusion module; According to the image restoration loss function and the quantization loss, reverse gradient propagation is performed to update all modules except the frozen modules in the general aberration restoration image model; after iterating the expected number of training iterations, the training is terminated.
10. The universal aberration-degraded image restoration method based on lens database construction according to claim 9, characterized in that: For the feature quantization module in the aberration-degraded image reconstruction model and the high-definition image reconstruction model features, the degradation feature quantization and fusion module and the high-definition feature quantization and fusion module in the general aberration-restored image model, The method of quantizing the features using the feature prior codebook is the same, and the specific method is: The feature to be quantized is matched with the feature prior codebook, and the distance between the feature vector corresponding to each pixel position of the feature to be quantized and each codebook vector in the feature prior codebook is calculated. The codebook vector with the minimum distance is used to replace the feature vector of the pixel position in the feature to be quantized. After replacing the feature vectors of all pixel positions, feature quantization is completed to obtain the quantized feature.
Citation Information
Patent Citations
Single lens space change image restoration method based on Encoder-Decoder depth learning model
CN107833186A
Discrete feature driven blind person face image loss detail reconstruction method
CN115565225A
Panoramic annular lens design method, lens and panoramic image restoration method
CN116068758A
Unsupervised learning wavefront restoration method for overcoming background aberration
CN116520565A
Method and apparatus for electron microscope image reconstruction
WO2024023537A1
