Compact panoramic annulus optical system and imaging method

Abstract

The invention discloses a compact panoramic annular optical system and an imaging method, and the method comprises the steps: building a monochromatic aberration tolerant panoramic annular optical imaging system lens group, and enabling the image surface of the lens group to be located on an image sensor of the panoramic annular optical imaging system; constructing a mathematical relationship between an input image of the panoramic belt optical imaging system and a simulation image output by the panoramic belt optical imaging system based on deep learning; and performing spherical aberration, coma aberration, astigmatism and curvature-of-field correction on a blurred image output by an image sensor in the panoramic annulus optical imaging system by using the neural network model subjected to deep learning training to obtain a restored image. The image restoration network model used in the invention effectively reduces the burden of aberration correction of the relay lens group, so that the panoramic annulus optical system device realizes the same imaging effect by adopting a smaller number of lenses, and the compactness of the system structure is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of optical imaging, and particularly relates to a compact panoramic annular optical system and an imaging method. Background Art

[0002] The panoramic annular imaging technology is a panoramic imaging technology based on the principle of planar cylindrical projection, which can image the scenes around the optical axis by 360° on the annular area of the image sensor plane. Without the need for scanning moving parts, this technology can obtain 360° panoramic images in a staring manner, so it has important applications in fields such as security monitoring, driverless, machine vision, and medical endoscopy that require large field-of-view observation. The lens group involved in the panoramic annular imaging technology mainly consists of a block lens group for realizing the light turning of a large field of view and a relay lens group for realizing the transfer of the intermediate image plane. In order to correct the serious geometric aberration caused by the large-angle light turning, if the traditional optical design method is adopted, the relay lens group often needs to use multiple lenses, resulting in the current panoramic annular imaging system being large in size and bulky, and it is difficult to be applied in some scenarios of medical endoscopy. A panoramic imaging device and method disclosed in CN103293845 A, the relay lens group of this device consists of 10 lenses; a panoramic imaging lens disclosed in CN102495460 A, the relay lens group of this lens consists of 6 lenses; a long-focus panoramic annular imaging lens disclosed in CN103969800 A, the relay lens group of this lens consists of 8 lenses; a large field-of-view panoramic imaging system based on multiplexed reflectors disclosed in patent CN116801107 A, the relay lens group of this system consists of 7 lenses. The above panoramic imaging systems are all optically designed based on traditional design methods, and there are problems such as a large number of lenses and a large volume in the relay lens group.

[0003] The deep learning-based blurred image restoration network model has shown great potential in correcting the aberration of optical systems and has been successfully applied in some fields. Therefore, in order to meet the requirements of large field-of-view panoramic imaging and at the same time achieve the compactness and miniaturization of the panoramic annular optical system, it is necessary to design an optical system and an imaging method that combines the deep learning-based image restoration network model with the panoramic annular optical system. Summary of the Invention

[0004] The technical problem solved by the present invention is: overcoming the deficiencies of the prior art, a compact panoramic annular optical system and an imaging method are proposed, which combines the deep learning-based image restoration network model with the design of the panoramic annular optical system, and realizes the compactness and miniaturization of the panoramic annular optical system device through optical-digital joint design.

[0005] The technical solution of the present invention is: A compact panoramic annular optical system imaging method, including:

[0006] Establish a lens group for a panoramic annular optical imaging system that tolerates monochromatic aberrations, and the image plane of the lens group is located on the image sensor of the panoramic annular optical imaging system;

[0007] Based on deep learning, establish the mathematical relationship between the input image of the panoramic annular optical imaging system and the simulated image output by the panoramic annular optical imaging system;

[0008] Use the neural network model trained by deep learning to correct spherical aberration, coma, astigmatism, and field curvature of the blurred image output by the image sensor in the panoramic annular optical imaging system to obtain a restored image.

[0009] The establishment of the lens group for the panoramic annular optical imaging system that tolerates monochromatic aberrations includes:

[0010] Establish a light-turning lens group and simulate and optimize the optical path structure of the light-turning lens group; during the simulation process, constrain the distortion of the sampled field of view and relax the constraints on spherical aberration, coma, astigmatism, field curvature, and chromatic aberration;

[0011] Establish a relay lens group and simulate and optimize the optical path structure of the relay lens group; during the simulation process, constrain the distortion of the sampled field of view and relax the constraints on spherical aberration, coma, astigmatism, field curvature, and chromatic aberration;

[0012] According to the object-image relationship and the pupil matching relationship, optically combine the optimized light-turning lens group and relay lens group to obtain a complete lens group for the panoramic annular optical imaging system;

[0013] Simulate and optimize the optical path structure of the obtained lens group for the panoramic annular optical imaging system. During the simulation process, constrain the distortion of the sampled field of view and relax the constraints on spherical aberration, coma, astigmatism, and field curvature.

[0014] The establishment of the mathematical relationship between the input image of the panoramic annular optical imaging system and the simulated image output by the panoramic annular optical imaging system based on deep learning includes:

[0015] Divide the scene on the object plane of the panoramic annular optical imaging system into blocks, and use the divided scene as the input image of the panoramic annular optical imaging system;

[0016] Calculate the point spread function of the field of view corresponding to the divided scene on the object plane of the panoramic annular optical imaging system at each sampling wavelength, and then sum the wavelengths weighted to obtain a complex point spread function;

[0017] Convolve the input image with the complex point spread function of the corresponding field of view, and splice the convolution results to obtain the simulated image of the panoramic annular optical imaging system;

[0018] Taking the input image and the simulated image as the training data set, training a neural network model based on deep learning, and constructing the mathematical relationship between the input image and the simulated image.

[0019] The light-turning lens group has the function of turning the light with a half field of view greater than 90° into the light with a half field of view less than 90°; the light-turning lens group is composed of no less than 2 pieces of optical glass and at least has 2 kinds of glass materials with different refractive indexes.

[0020] The relay lens group has the function of relaying the image formed by the light-turning lens group to the image plane of the panoramic annular optical imaging system; the relay lens group is composed of no less than 2 pieces of optical glass and at least has 2 kinds of glass materials with different refractive indexes; the relay lens group has at least 1 positive focal length lens and 1 negative focal length lens.

[0021] Constraining the distortion of the sampling field of view, that is, constraining the distortion within 1.5%; relaxing the constraints on spherical aberration, coma, astigmatism, field curvature and chromatic aberration, that is, constraining the peak-to-valley value of the wave aberration within 1 wavelength.

[0022] In the simulation optimization of the light-turning lens group and the lens group of the panoramic annular optical imaging system, the constrained distortion is characterized in the f′-θ manner, that is

[0023]

[0024] where distortion represents distortion, y represents the actual height of the chief ray of the sampling field of view on the image plane after passing through the lens group of the panoramic annular optical imaging system, f′ represents the effective focal length of the lens group of the panoramic annular optical imaging system, and θ represents the field of view of the panoramic annular optical imaging system.

[0025] Taking the input image and the simulated image as the training data set, training a neural network model based on deep learning, and constructing the mathematical relationship between the input image and the simulated image includes: based on the deep neural network Res-UNet in deep learning, using the residual module to extract feature information from the training data set of the panoramic annular optical imaging system during the encoding process, and using the backpropagation gradient descent algorithm to minimize the loss function to obtain a neural network model with good image restoration effect, and thus constructing the mathematical relationship between the input image and the blurred image.

[0026] A system for implementing the above-mentioned imaging method of a compact panoramic annular optical system, including:

[0027] A light-turning lens group for turning the light with a large angle in the object space into the light with a small angle and imaging it on an intermediate image plane;

[0028] A relay lens group for imaging the intermediate image formed by the light-steering lens group again;

[0029] An image sensor for receiving the image formed by the relay lens group;

[0030] A data processing module that uses a deep learning-based image restoration network model to perform aberration correction on the blurred image output by the image sensor to obtain a clear restored image.

[0031] The field of view of the panoramic annular optical imaging system device is 360°×(70° - 110°), the entrance pupil diameter is 1.5 mm, the effective focal length is 9 mm, and the working wavelength range is 400 nm - 760 nm.

[0032] The advantages of the present invention compared with the prior art are as follows: The present invention proposes to design a panoramic annular optical imaging system using an optical-digital joint design method based on deep learning; compared with the traditional design method of panoramic annular optical imaging systems, the method proposed by the present invention reduces the burden of large-field optical aberration correction, simplifies the structure of the relay optical system, and realizes the compactness and miniaturization of the panoramic annular optical imaging system. Description of the Drawings

[0033] Figure 1 This is a computational panoramic annular optical imaging system of the present invention.

[0034] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0035] Figure 3 This is the geometric spot of the panoramic annular optical imaging system at the sampling field of view in the embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the input image, simulated image, and restored image of the panoramic annular optical imaging system in the embodiment of the present invention.

[0037] Figure 5 This is the image restoration network structure in the embodiment of the present invention.

[0038] Figure 6 This is the comparison system of the panoramic annular optical imaging system in the embodiment of the present invention. Detailed Embodiments

[0039] As Figure 1 shown, a compact panoramic annular optical imaging system of the present invention includes a light-steering lens group 1, a relay lens group 2, an image sensor 3, and a data processing module 4; the data processing module uses a deep learning-based image restoration network model to perform geometric aberration correction on the blurred image output by the image sensor of the panoramic annular optical imaging system to obtain a clear restored image.

[0040] In this embodiment, the light-turning lens group uses 3 glass lenses, and the material names are HK9L_CDGM, HZF3_CDGM, and HK9L_CDGM respectively; the relay lens group in this embodiment uses 5 glass lenses, and the material names are HLAK4L_NHG, HZF52N_NHG, NFK58_SCHOTT, NBK10_SCHOTT, and NLAF34_SCHOTT respectively.

[0041] The relay lens group in this embodiment uses 3 lenses with positive optical power and 2 lenses with negative optical power.

[0042] Preferably, the optical power distribution form of the relay lens group in this embodiment is "positive-negative-positive-positive-negative".

[0043] The field of view of the panoramic annular optical system in this embodiment is 360°×(70° to 110°), the entrance pupil diameter is 1.5 mm, the effective focal length is 9 mm, and the working wavelength range is 400 nm to 760 nm.

[0044] As Figure 2 shown, a method for imaging a compact panoramic annular optical system provided in this embodiment includes the following steps:

[0045] 1) Establish a panoramic annular optical imaging system lens group that tolerates monochromatic aberrations, including:

[0046] 1.1) Select a light-turning lens with similar technical indicators from the existing patent library as the initial structure of the light-turning lens group in this embodiment, and then simulate and optimize the optical path structure in simulation software; the constraint control conditions during the simulation process include the effective focal length of the lens group and the distortion of the sampled field of view, expressed as formula (2), and do not constrain spherical aberration, coma, astigmatism, field curvature, and chromatic aberration;

[0047]

[0048] 1.2) Select 5 lenses made of different glass materials as the initial structure of the relay lens group, and then simulate and optimize the optical path structure in simulation software; the constraint control conditions during the simulation process include the image height of the sampled field of view of the lens group and the optical power of the lens;

[0049] Preferably, the lateral magnification of the relay lens group is -1.

[0050] 1.3) According to the object-image relationship and the pupil matching relationship, optically combine the optimized light-turning lens group in step 1.1 with the optimized relay lens group in step 1.2 to obtain a complete panoramic annular optical imaging system lens group;

[0051] 1.4) Optimize the optical path structure of the panoramic annular optical imaging system lens group in step 1.3, and only constrain the distortion of the sampling field of view during the simulation process;

[0052] Preferably, it is allowed that the geometric spot diameter of the panoramic annular optical imaging system lens group at the sampling field of view is not greater than 600 μm, and the geometric spot distribution of the system after simulation optimization is as Figure 3 shown.

[0053] 2) Construct the mathematical relationship between the input image and the simulation image based on deep learning, including:

[0054] 2.1) Divide the annular scene on the object surface of the panoramic annular optical imaging system into blocks. Specifically, sample 10 fields of view radially and uniformly, and sample 36 fields of view tangentially and uniformly at equal angles, for a total of 360 sampled fields of view. The divided scene is used as the input image of the panoramic annular optical imaging system;

[0055] 2.2) Calculate the point spread function of the panoramic annular optical imaging system corresponding to the divided object surface fields of view at each sampling wavelength, store the point spread function results in a 128×128 grid, and then sum them weighted by wavelength. The sampling wavelengths are 486 nm, 587 nm, and 656 nm to obtain the chromatic point spread function;

[0056] 2.3) Convolve the divided input image with the chromatic point spread function of the corresponding field of view, and splice the convolution results to obtain a simulation image with blurred image quality;

[0057] 2.4) A total of 400 high-definition input images are selected. According to the above steps 2.1 to 2.3, a total of 400 simulation images are generated. The input images and the simulation images form 400 groups of image pairs as the training data set, and a neural network model is trained based on deep learning to construct the mathematical relationship between the input image and the simulation image; among them, the input image and the simulation image of the panoramic annular optical imaging system are composed of annular scenes corresponding to the designed annular fields of view, as Figure 4 shown in a and b, and the input image in the form of an annulus and the simulation image in the form of an annulus are used as the training data set.

[0058] Step 2.4) is specifically: Based on the deep neural network Res-UNet, use the residual module to extract feature information from the simulated imaging of the panoramic annular optical imaging system during the encoding process. Select the Adam optimizer and use the backpropagation gradient descent algorithm to minimize the loss function to obtain a network model with good image restoration effect. The specific network structure is as Figure 5 shown. The input of this network is the simulation image of the panoramic annular optical imaging system, and the output of the network is the restored clear image.

[0059] 3) Use the trained neural network model to correct the spherical aberration, coma, astigmatism and field curvature of the blurred simulated image output by the panoramic annular optical imaging system to restore the blurred image. Figure 4 As shown, Figure a is an input image in the test set, Figure b is a simulated image of Figure a output by the panoramic annular optical imaging system, and Figure c is a clear image of Figure b after restoration by the image restoration network model. PSNR is used to evaluate the root square error between the restored image and the corresponding pixel points of the input image. The larger the PSNR value, the better the quality of the restored image. SSIM is used to evaluate the similarity between the restored image and the input image. It can evaluate the differences in brightness, contrast and structure. The larger the SSIM value, the better. If the SSIM value is 1, it indicates that the restored image and the input image are exactly the same. The embodiment of the present invention uses the two indicators of PSNR and SSIM to evaluate the quality of the restored image. The evaluation results are listed in Table 1. Figure 4 It can be seen that the clarity of the image has been greatly improved, the details have been well restored, and there is no obvious color difference.

[0060] Table 1 Image quality evaluation results of simulation images before and after restoration by the restoration network model

[0061]

[0062] Furthermore, according to the technical indicators of the panoramic annular optical imaging system device in this embodiment, a comparison system with the same technical indicators was designed using a traditional optical design method. The results are as follows: Figure 6 As shown. The input image of this embodiment is used as the input of the comparison system. The quality of the image output by the comparison system is close to the clear image restored by the image restoration network model in this embodiment in terms of PSNR and SSIM. The length of the relay lens group of the comparison system is 105mm. In contrast, the length of the relay lens group of the panoramic annular optical imaging system of this embodiment is only 50mm, and the structure is more compact, which reflects the advancement of the design method and system device of the present invention.

[0063] In addition, those skilled in the art can understand that the present invention is not limited to the above embodiments, and more variations and modifications can be made according to the teachings of the present invention, and these variations and modifications all fall within the scope of protection claimed by the present invention.

Claims

1. A compact panoramic annular optical system imaging method, characterized in that: include: Establishing a monochromatic aberration-tolerant panoramic annular optical imaging system lens group, wherein the image plane of the lens group is located on an image sensor of the panoramic annular optical imaging system; Constructing the mathematical relationship between the input image of the panoramic annular optical imaging system and the simulated image output by the panoramic annular optical imaging system based on deep learning; The neural network model trained by deep learning is used to correct the spherical aberration, coma, astigmatism and field curvature of the blurred image output by the image sensor in the panoramic annular optical imaging system to obtain a restored image.

2. The method according to claim 1, characterized in that The panoramic annular optical imaging system lens group for establishing monochromatic aberration tolerance comprises: Establish a light steering lens group and simulate and optimize the optical path structure of the light steering lens group; constrain the distortion of the sampling field of view during the simulation process, and relax the constraints on spherical aberration, coma, astigmatism, field curvature and chromatic aberration; Establish a relay lens group and simulate and optimize the optical path structure of the relay lens group; constrain the distortion of the sampling field of view during the simulation process, and relax the constraints on spherical aberration, coma, astigmatism, field curvature and chromatic aberration; According to the object-image relationship and pupil matching relationship, the optimized light is directed to the lens group and the relay lens group for optical merging to obtain a complete panoramic annular optical imaging system lens group; The optical path structure of the obtained panoramic annular optical imaging system lens group is simulated and optimized. During the simulation process, the distortion of the sampling field of view is constrained, and the constraints on spherical aberration, coma, astigmatism and field curvature are relaxed.

3. The method according to claim 1, characterized in that The method of constructing a mathematical relationship between an input image of a panoramic annular optical imaging system and a simulated image output by the panoramic annular optical imaging system based on deep learning includes: Dividing the scene on the object plane of the panoramic annular optical imaging system into blocks, and using the divided scenes as input images of the panoramic annular optical imaging system; The point spread function of the field of view corresponding to the block scene on the object plane of the panoramic annular optical imaging system at each sampling wavelength is calculated, and then the wavelength is weighted and summed to obtain the complex color point spread function; The input image is convolved with the complex point spread function of the corresponding field of view, and the convolution results are spliced ​​to obtain a simulation image of the panoramic annular optical imaging system; The input image and the simulated image are used as training data sets, and a neural network model is trained based on deep learning to construct a mathematical relationship between the input image and the simulated image.

4. The method according to claim 2, characterized in that The light redirecting lens group has the function of redirecting light with a half field of view greater than 90° into light with a half field of view less than 90°; the light redirecting lens group is composed of no less than 2 pieces of optical glass and has at least 2 glass materials with different refractive indices.

5. The method according to claim 2, characterized in that The relay lens group has the function of relaying the image formed by the light redirected to the lens group to the image plane of the panoramic annular optical imaging system; the relay lens group is composed of no less than 2 pieces of optical glass and at least 2 glass materials with different refractive indices; the relay lens group has at least 1 positive focal power lens and 1 negative focal power lens.

6. The method according to claim 2, characterized in that The constraint on the distortion of the sampling field of view is to constrain the distortion within 1.5%; the relaxation of the constraints on spherical aberration, coma, astigmatism, field curvature and chromatic aberration is to constrain the peak and valley values ​​of wave aberration within 1 wavelength.

7. The method according to claim 6, characterized in that In the simulation optimization of the light redirection lens group and the panoramic annular optical imaging system lens group, the constrained distortion is characterized by the f′-θ method, that is, Wherein, distortion represents distortion, y represents the actual height of the sampling field of view main ray on the image plane after passing through the panoramic annular optical imaging system lens group, f′ represents the effective focal length of the panoramic annular optical imaging system lens group, and θ represents the field of view of the panoramic annular optical imaging system.

8. The method according to claim 3, characterized in that The method uses the input image and the simulated image as training data sets, trains a neural network model based on deep learning, and constructs a mathematical relationship between the input image and the simulated image, including: based on the deep neural network Res-UNet in deep learning, using a residual module in the encoding process to extract feature information from the training data set of the panoramic annular optical imaging system, using a back-propagation gradient descent algorithm to minimize the loss function, and obtaining a neural network model with good image restoration effect, thereby constructing a mathematical relationship between the input image and the blurred image.

9. A system for executing the compact panoramic annular optical system imaging method according to claim 1, characterized in that: include: The light-redirecting lens group is used to redirect the light of large angle on the object side into light of small angle and form an image on an intermediate image plane; The relay lens group is used to re-image the intermediate image formed by the light redirecting lens group; An image sensor, used for receiving an image formed by the relay lens group; The data processing module uses an image restoration network model based on deep learning to perform aberration correction on the blurred image output by the image sensor to obtain a clear restored image.

10. The system according to claim 9, characterized in that The panoramic annular optical imaging system device has a field of view of 360°×(70°~110°), an entrance pupil diameter of 1.5 mm, an effective focal length of 9 mm, and an operating band of 400 nm~760 nm.

Images