A liquid lens-based large zoom ratio high-resolution camera and image processing method

Abstract

The application provides a large-variation-ratio high-resolution camera based on a liquid lens and an image processing method. The zoom function is realized by controlling the liquid lens, and the dynamic aberration and image noise of the camera based on the liquid lens are processed by using the image processing method, so that the contrast of effective image information is improved. The camera comprises a zoom lens group, a diaphragm, a compensation lens group, an image sensor, a liquid lens driver, an image processing module and a mechanical structural part. By controlling the driving current of the liquid lens driver, the focal power of the liquid lens in the zoom lens group and the compensation lens group is changed, so that the focal length of the whole camera is changed, and the continuous zoom function is realized. In addition, the application provides an image processing method for the camera. The image processing method is based on physical priori, comprehensively considers the influence of factors such as a real environment, an image sensor, a machining tolerance and a liquid lens wavefront difference, establishes an imaging degradation model of the camera, and trains a neural network based on the degradation method, so that the dynamic aberration and image noise of the camera based on the liquid lens are processed, and the contrast of effective image information is improved.

Description

Technical Field

[0001] This invention relates to a high-resolution camera and an image processing method, and more specifically, to a high-zoom high-resolution camera and an image processing method based on a liquid lens. Background Technology

[0002] A large field of view and high resolution have always been key characteristics pursued by optical imaging systems, but a trade-off exists due to the limitation of the spatial bandwidth product. Zoom imaging technology, as a crucial imaging strategy, can adjust the field of view and resolution by switching the imaging focal length, thereby enabling global and local detection of the imaging system. Currently, most commercial zoom imaging lenses still use mechanical zoom, that is, adjusting the focal length through the mechanical movement of the lens assembly. However, this method is not only slow in response speed, but also results in a large imaging system size due to its reliance on mechanical drive components, severely limiting the application of zoom imaging technology.

[0003] In recent years, liquid lenses have attracted much attention due to their advantages such as rapid adjustability, high transmittance, high integration, and polarization insensitivity. Liquid lens technology offers a possible solution for the design of novel zoom imaging lenses. However, currently, liquid lenses are mainly used to adjust the focal plane of the imaging system, and their function is limited to focusing rather than zooming. Traditional mechanical zoom lenses typically have zoom ratios exceeding 3x, but lens designs based on liquid lenses with zoom ratios exceeding 3x are still relatively rare. Research on liquid lens-based zoom lenses for practical applications, especially zoom lenses with large zoom ranges, remains insufficient in terms of design methodologies, design constraint analysis, and reference design cases. Furthermore, zoom lens design inevitably requires the use of multiple liquid lenses, and the inherent aberrations and environmental sensitivity of liquid lenses may pose challenges to the actual performance of zoom lenses with large zoom ranges. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a high-zoom-ratio high-resolution camera based on a liquid lens and an image processing method. This method enables zoom functionality solely through control of the liquid lens and uses image processing methods to address dynamic aberrations and image noise in the liquid lens-based camera, thereby improving the contrast of effective image information. The camera structure is shown in the attached figure. Figure 1 As shown, it includes:

[0005] A zoom lens group, consisting of multiple solid and liquid lenses, is used to change the focal length of a camera, thereby altering the camera's field of view and magnification.

[0006] An aperture stop is used to limit the beam of light and the imaging range.

[0007] The compensating lens group, which consists of multiple solid lenses and liquid lenses, is used to compensate for image plane shift caused by changes in the focal length of the liquid lens in the zoom lens group.

[0008] An image sensor is used to capture images from a camera and transmit them to an image processing module for further processing.

[0009] A liquid lens driver for driving the liquid lenses contained in the zoom lens group and the compensation lens group.

[0010] The image processing module is used to receive image information from the image sensor, run the image processing algorithm, and perform image processing and reconstruction.

[0011] Mechanical structural components are used to fix the zoom lens group, aperture, compensation lens group, image sensor, and liquid lens driver.

[0012] The total effective focal length f of a zoom lens based on a liquid lens can be expressed as:

[0013] 1 / f = 1 / f1+1 / f2-d / f1f2 (1)

[0014] Where d is the distance between the image-side principal plane of the zoom lens group and the object-side principal plane of the compensation lens group, and f1 and f2 are the focal lengths of the zoom lens group and the compensation lens group, respectively, which are functions of the focal length of the internal liquid lens. By changing the focal length of the liquid lens, the total effective focal length of the high-resolution camera based on the liquid lens can be precisely adjusted.

[0015] The working optical paths of the camera in short-focal-length and long-focal-length modes are respectively shown in the attached figures. Figure 2 and attached Figure 3 As shown. During operation, the working distance of the high-zoom-ratio, high-resolution camera based on the liquid lens remains constant. Neither the zoom lens group nor the compensation lens group undergoes any mechanical structural movement, and the camera's continuous zoom function is achieved solely through liquid lens zoom.

[0016] This invention also provides an image processing method for the aforementioned camera, which aims to train a neural network model based on prior physical information, enabling it to effectively enhance feature information and suppress noise, thereby achieving back-end processing of images captured by a zoom camera based on a liquid lens. The workflow of the image processing method is shown in the attached figure. Figure 4 As shown, the description is as follows:

[0017] The first step is to simulate the camera's point spread function. Due to actual manufacturing tolerances and the wavefront stability of the liquid lens, there is a difference between the actual wavefront and the theoretical design value of a high-zoom, high-resolution camera based on a liquid lens. Although the wavefront of the liquid lens can theoretically be calibrated under different environments and attitudes, the practical operation is extremely difficult and lacks robustness. Therefore, by introducing tolerance and liquid lens wavefront perturbation factors, Monte Carlo analysis is performed to generate multiple optical models at different focal lengths. The point spread function is extracted from the optical models to simulate the camera's degradation process, taking into account the influence of liquid lens manufacturing tolerances and wavefront differences. The point spread function obtained from the optical models is used to represent the camera's blur kernel.

[0018] The second step is to establish an image degradation model. For image quality degradation caused by the camera, the point spread function convolution obtained in the first step is used for simulation; for image quality degradation caused by the transmission medium, random isotropic and anisotropic Gaussian kernel convolution is used for simulation; and for image noise caused by the image sensor, additive Gaussian noise is used for simulation. Based on this, the degradation model for a high-zoom, high-resolution camera based on a liquid lens is described as follows:

[0019] (2)

[0020] Where I(x, y) and O(x, y) represent the degraded image and the original image, respectively, ↓ represents the downsampling process, T(x,y) represents the degraded blur kernel of the transmission medium, PSF(x, y) represents the degraded blur kernel of the zoom lens, and N(x, y) represents noise.

[0021] The third step is to train the neural network. Based on the actual image degradation model, the high-resolution image dataset is degraded to generate a low-resolution image dataset. The low-resolution images are used as input to the neural network, and the resulting predicted images are compared with the original high-resolution images. The neural network is then trained by calculating a loss function.

[0022] The fourth step is quantitative testing and feedback optimization. Using the degradation model, the high-resolution test dataset is degraded to obtain a low-resolution image test dataset. The reconstructed images are then processed by the trained neural network, and quantitative evaluation is performed based on evaluation metrics. If the quantitative evaluation results do not meet the metric requirements, the model and neural network parameters are readjusted based on the evaluation results, and feedback optimization is performed. If the quantitative evaluation results meet the metric requirements, the trained network is used to process real images captured by the camera to obtain the processed high-resolution images.

[0023] This invention proposes a high-resolution camera with a large zoom ratio based on a liquid lens and its image processing method. This enables rapid continuous zoom imaging without mechanical movement, achieving a wide focal length range that balances a large field of view and high resolution. Furthermore, the use of a liquid lens for rapid zoom offers advantages such as fast adjustment speed, high integration, and low power consumption. In addition, this invention proposes an image processing method for the camera. This method, based on physical priors, comprehensively considers the influence of factors such as the real environment, image sensor, manufacturing tolerances, and wavefront aberration of the liquid lens to establish an imaging degradation model for the camera. A neural network is then trained based on this degradation model to process the dynamic aberrations and image noise of the liquid lens-based camera, thereby improving the contrast of effective image information. The combined control method of the image processing method and the liquid lens provides an economical and feasible solution for achieving large field-of-view, high-resolution observation with a camera.

[0024] Preferably, in order to achieve fast response and relatively large optical aperture, the light transmission aperture D of the liquid lens needs to satisfy D≥12mm, and the optional types of liquid lenses include electromagnetically driven elastic film liquid lenses.

[0025] Preferably, the zoom ratio M of the camera needs to satisfy M≥8, and the total number N of liquid lenses in the zoom lens group and the compensation lens group needs to satisfy N≥7.

[0026] Preferably, in the Monte Carlo analysis of the image processing method, the number p of generated optical models must satisfy p≥10, and each Monte Carlo analysis model needs to extract point spread functions at different working focal lengths, and the number q of its working focal lengths must satisfy q≥4. Attached Figure Description

[0027] Appendix Figure 1 This is a schematic diagram of the optical lens of the present invention.

[0028] Appendix Figure 2 This is a schematic diagram of the optical path in the short focal length state of the present invention.

[0029] Appendix Figure 3 This is a schematic diagram of the optical path in the long focal length mode of the present invention.

[0030] Appendix Figure 4 This is a flowchart illustrating the computational image enhancement method of the present invention.

[0031] Appendix Figure 5 (a)-(d) are optical path simulation diagrams for working focal lengths of 11mm, 22mm, 44mm and 88mm respectively in specific embodiments of the present invention.

[0032] Appendix Figure 6 (a)-(d) are modulation transfer function diagrams for working focal lengths of 11mm, 22mm, 44mm and 88mm in specific embodiments of the present invention, respectively.

[0033] Appendix Figure 7 (a) and (b) are the point spread function diagrams of the original optical model and the optical model after Monte Carlo analysis in a specific embodiment of the present invention, respectively.

[0034] The figure labels in the above figures are as follows:

[0035] (1) Zoom lens group, (2) Solid lens, (3) Liquid lens, (4) Mechanical structural component, (5) Aperture, (6) Compensation lens group, (7) Image sensor, (8) Liquid lens driver.

[0036] It should be understood that the above figures are only schematic and are not drawn to scale. Detailed Implementation

[0037] The following detailed description of an embodiment of a high-resolution camera with large zoom ratio based on a liquid lens and an image processing method proposed in this invention further illustrates the invention. It is important to note that the following embodiments are only for further illustrative purposes and should not be construed as limiting the scope of protection of this invention. Any non-essential improvements and adjustments made to this invention by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0038] First, an initial signal is input to the liquid lens in the zoom lens group and the compensation lens group to initialize the focal length of the zoom lens group and the compensation lens group. The beam of the object-side field of view passes through the zoom lens group, the aperture stop and the compensation lens group in sequence, and finally focuses on the image sensor.

[0039] Then, by adjusting the driving signal of the liquid lens, the optical power of the liquid lens is changed, and the optical power of the zoom lens group and the compensation lens group changes, causing the overall focal length of the camera to change and be adjusted to a suitable range, so that the camera can observe the target at a certain magnification.

[0040] This embodiment of a high-zoom-ratio high-resolution camera based on liquid lenses includes a zoom lens group, an aperture stop, a compensation lens group, and an image sensor. The zoom lens group, along the optical axis from object to image, consists of five meniscus solid lenses and five 12mm aperture elastic film liquid lenses. The five meniscus solid lenses are primarily used to deflect the light beam. These five liquid lenses are sequentially named the first liquid lens, second liquid lens, third liquid lens, fourth liquid lens, and fifth liquid lens. The aperture stop is located between the fourth and fifth liquid lenses. The compensation lens group, along the optical axis from object to image, consists of seven solid lenses, two 12mm aperture elastic film liquid lenses, and three solid lenses. The solid lenses are primarily used to correct aberrations. These two liquid lenses are sequentially named the sixth and seventh liquid lenses. The image sensor includes a 2 / 3-inch diagonal CMOS industrial camera. The liquid lens drive module has seven power supply channels to provide current drive signals to the liquid lenses.

[0041] This embodiment describes a high-zoom, high-resolution camera based on a liquid lens that operates in the visible light band, with a working focal length of 11mm to 88mm. When zooming from a short focal length to a long focal length, the camera's F-number changes from 3.91 to 8.03. The continuous zoom function of the camera is demonstrated below using four working focal lengths: 11mm, 22mm, 44mm, and 88mm.

[0042] When the camera's working focal length is 11mm, the liquid lens driver provides driving current to the seven elastic film liquid lenses of the zoom lens group and the compensation lens group, resulting in the following interface curvature radii: R1 of the first liquid lens is -76.56mm, R2 of the second liquid lens is -76.64mm, R3 of the third liquid lens is -76.77mm, R4 of the fourth liquid lens is -106.77mm, R5 of the fifth liquid lens is 54.66mm, R6 of the sixth liquid lens is 78.28mm, and R7 of the seventh liquid lens is 45.44mm, as shown in the attached diagram. Figure 5 As shown in (a). At this time, the object-side field of view angle a1 of the camera is 30°, and the modulation transfer function (MTF) graph is attached. Figure 6 As shown in (a). When the camera operates in the visible light band, the resolution is 235 lp / mm at MTF=0.1.

[0043] When the camera's working focal length is 22mm, the liquid lens driver is adjusted to apply a new driving current to the seven elastic film liquid lenses of the zoom lens group and the compensation lens group. This results in the following interface curvature radii: R1 of the first liquid lens is -143.85mm, R2 of the second liquid lens is 45.43mm, R3 of the third liquid lens is -877.33mm, R4 of the fourth liquid lens is -80.52mm, R5 of the fifth liquid lens is 89.02mm, R6 of the sixth liquid lens is 45.38mm, and R7 of the seventh liquid lens is 254.23mm, as shown in the attached diagram. Figure 5 As shown in (b). At this time, the object-side field of view angle a2 of the camera is 15.4°, and the modulation transfer function diagram is attached. Figure 6 As shown in (b), when the camera operates in the visible light band, the resolution is 253 lp / mm at MTF=0.1.

[0044] When the camera's working focal length is 44mm, the liquid lens driver is adjusted to apply a new driving current to the seven elastic film liquid lenses of the zoom lens group and the compensation lens group. This results in the following interface curvature radii: R1 of the first liquid lens is 45.41mm, R2 of the second liquid lens is 1396.32mm, R3 of the third liquid lens is 45.45mm, R4 of the fourth liquid lens is -236.32mm, R5 of the fifth liquid lens is -136.33mm, R6 of the sixth liquid lens is 46.02mm, and R7 of the seventh liquid lens is -76.86mm, as shown in the attached diagram. Figure 5 As shown in (c). At this time, the object-side field of view angle a3 of the camera is 7.8°, and the modulation transfer function diagram is attached. Figure 6 As shown in (c). When the camera operates in the visible light band, the resolution is 213 lp / mm at MTF=0.1.

[0045] When the camera's working focal length is 88mm, the liquid lens driver is adjusted to apply a new driving current to the seven elastic film liquid lenses of the zoom lens group and the compensation lens group. This results in the following interface curvature radii: R1 of the first liquid lens is 45.44mm, R2 of the second liquid lens is 45.42mm, R3 of the third liquid lens is 47.78mm, R4 of the fourth liquid lens is -76.77mm, R5 of the fifth liquid lens is -76.83mm, R6 of the sixth liquid lens is -76.82mm, and R7 of the seventh liquid lens is -76.84mm, as shown in the attached diagram. Figure 5As shown in (d). At this time, the object-side field of view angle a4 of the camera is 3.8°, and the modulation transfer function diagram is attached. Figure 6 As shown in (d), when the camera operates in the visible light band, the resolution is 145 lp / mm at MTF=0.1.

[0046] In one embodiment of the image processing method for the camera, the first step involves performing Monte Carlo tolerance analysis on the camera. The tolerance parameters are set as follows: the surface curvature radius tolerance, thickness tolerance, eccentricity tolerance, tilt tolerance, and surface irregularity are set to f / 3, 0.1 mm, 0.01 mm, 1', and 0.5, respectively; the element eccentricity tolerance and tilt tolerance are set to 0.02 mm and 1', respectively. The refractive index tolerance and Abbe number tolerance of the solid lens material are set to 0.0003 and 0.5%, respectively. The root mean square wavefront error of the liquid lens is set within 0.25 times the wavelength, and is simulated by introducing additional Zernike irregularities on the liquid film surface. After performing Monte Carlo tolerance analysis, 10 optical models affected by the tolerances were generated. For the four working focal lengths of 11mm, 22mm, 44mm, and 88mm in the visible light band, the point spread function was analyzed by dividing the field of view into 5×5 regions. The point spread functions of the original optical model and the optical model after Monte Carlo analysis are shown in the attached figures. Figure 7 As shown in (a) and (b).

[0047] The second step is to establish an image degradation model. For image quality degradation caused by the transmission medium, random isotropic and anisotropic Gaussian kernels are used to estimate the degradation process. The size of the Gaussian kernel is randomly selected within the range [5, 15] and is limited to an odd number, with its standard deviation randomly selected within the range [0, 1.4]. For the sampling noise of the imaging sensor, additive Gaussian noise is used for estimation, which follows a normal distribution with a standard deviation randomly selected within the range [1 / 75, 1 / 15]. Furthermore, a four-fold downsampling is used to represent the image resolution degradation process.

[0048] The third step is to train the neural network. The low-resolution image is used as input to the neural network, which processes it to generate a predicted image. This predicted image is then compared with the original high-resolution image, and the neural network is trained by calculating a loss function. The trained network is then used to process real images captured by the camera to obtain the processed high-resolution image. The SWinIR neural network, based on a Shifted Windows (SWin) attention mechanism, is used as the base network model.

[0049] During network training, a composite loss function was used, consisting of mean absolute error loss, adversarial loss, and perceptual loss. The PatchGAN model was used as the discriminator network to evaluate the adversarial loss and was jointly trained with the generator network to facilitate the recovery of high-frequency features. For the perceptual loss, the VGG19 network was used to extract image features, enabling the network to generate images with more realistic details and better visual effects. The Flickr2K and a portion of the OST dataset were used as the training set, containing a total of 6500 images. During training, the images were cropped into a series of 256×256×3 image patches. The learning rate and batch size were set to 1×10⁻⁶. -6 And 16.

[0050] The fourth step involves quantitatively evaluating the image reconstruction performance of the neural network using metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index (SSIM) of the test images. On the Set5 dataset, the average PSNR and average SSIM values ​​of the reconstructed images at different working focal lengths are higher than 27 dB and 0.74, respectively. By activating the camera and image processing module, capturing images of the real environment, and processing them using the trained network, a high-resolution image can be obtained after image processing.

Claims

1. A high zoom ratio, high-resolution camera based on a liquid lens, characterized in that, The camera includes a zoom lens group, an aperture stop, a compensation lens group, an image sensor, a liquid lens driver, an image processing module, and mechanical components. The zoom lens group comprises multiple solid and liquid lenses to change the camera's focal length, thus altering the field of view and magnification. The aperture stop limits the light beam and imaging range. The compensation lens group, also comprising multiple solid and liquid lenses, compensates for image plane shift caused by changes in the focal length of the liquid lenses. The image sensor acquires the image formed by the compensation lens group and transmits it to the image processing module for further processing. The liquid lens driver drives the zoom lens group and the liquid lens components within the compensation lens group. The image processing module receives image information from the image sensor and performs image processing and reconstruction. The mechanical components secure the zoom lens group, aperture stop, compensation lens group, image sensor, and liquid lens driver.

2. The high zoom ratio high-resolution camera based on a liquid lens according to claim 1, characterized in that, The camera's liquid lens driver inputs a driving current signal to the liquid lenses in the zoom lens group and the compensation lens group, adjusting the optical power of the liquid lenses to control the camera's focal length.

3. A high-zoom, high-resolution camera based on a liquid lens according to claim 1, characterized in that, The light-transmitting aperture D of the liquid lens is ≥12mm.

4. A high-zoom, high-resolution camera based on a liquid lens according to claim 1, characterized in that, The camera's zoom ratio M ≥ 8, and the total number of liquid lenses in the zoom lens group and compensation lens group N ≥ 7.

5. An image processing method for a high zoom ratio, high-resolution camera based on a liquid lens, characterized in that, The method consists of the following steps: First, by introducing tolerance and wavefront perturbation factors of the liquid lens, Monte Carlo tolerance analysis is performed on the camera to generate multiple optical models under different focal lengths. The point spread function is extracted from the optical model to simulate the degradation process of the camera, so as to take into account the influence of manufacturing tolerance and wavefront difference of the liquid lens. The second step involves establishing an image degradation model. For image quality degradation caused by the camera, the point spread function convolution obtained in the first step is used for simulation. For image quality degradation caused by the transmission medium, random isotropic and anisotropic Gaussian kernel convolution is used for simulation. For image noise caused by the image sensor, additive Gaussian noise is used for simulation. The third step uses the image degradation model established in the second step to degrade the high-resolution image dataset, obtaining a low-resolution image dataset. The low-resolution image is used as input to the neural network to generate a predicted image, which is compared with the original high-resolution image. The neural network is trained by calculating the loss function. The fourth step uses the degradation model to degrade the high-resolution test dataset, obtaining a low-resolution image test dataset. The reconstructed image is obtained by processing the trained neural network, and quantitative evaluation is performed based on evaluation metrics. If the quantitative evaluation results do not meet the requirements, the model and neural network parameters are readjusted based on the evaluation results for feedback optimization. If the quantitative evaluation results meet the requirements, the trained network is used to process real images captured by the camera to obtain the processed high-resolution image.

Images