Focus stack imaging method and system of liquid lens and electronic equipment

Abstract

The invention provides a focus stack imaging method and system of a liquid lens and electronic equipment, and belongs to the technical field of optical imaging detection, and the method comprises the steps: controlling a power supply module to provide a driving voltage for the liquid lens, so as to form an image on a CMOS array; acquiring images acquired by the CMOS array, and performing focus stack alignment on all the images based on the image when the driving voltage is the lowest to obtain an image sequence of a constant-magnification focus stack; performing focusing state evaluation on the image sequence, and performing full-focusing fusion on the image sequence based on a focusing state evaluation result to obtain a full-focusing image; and determining the object distance of the acquired image, generating a depth map based on the object distance and the full-focus image, and generating a target image based on the full-focus image and the depth map. According to the invention, focus stack alignment is carried out on all the acquired images based on the image at the lowest driving voltage, and the technical problem of low imaging precision in the existing focus stack imaging technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging detection technology, and in particular to a focal stacking imaging method, system and electronic equipment of a liquid lens. Background Art

[0002] In recent years, with the significant development of deep learning technology, focus stack imaging technology has been widely used in many fields such as depth of field extension and depth estimation. However, how to quickly and in real time collect focus stack image sequences with obvious out-of-focus and focus characteristics is still a challenge. In previous research work, optical zoom systems such as mechanical focus cameras, light field cameras or telecentric cameras were mostly used to collect focus stack image sequences. When using a mechanical focus camera to collect focus stacks, the imaging magnification and field of view of the captured focus stack image sequence will also change due to the change in the distance between the lens and the CMOS image sensor. This phenomenon is called lens breathing. This change in imaging magnification causes misalignment between pixels in different image frames in the stack. If the focus stack image sequence is not aligned, the fully focused image generated by the focus stack image will have obvious splicing misalignment and pixel loss, resulting in low image accuracy. Summary of the invention

[0003] In view of this, it is necessary to provide a focal stacking imaging method, system and electronic equipment of a liquid lens to solve the technical problem of low imaging accuracy in the existing focal stacking imaging technology.

[0004] In order to solve the above problems, on the one hand, the present invention provides a focal stack imaging method of a liquid lens, which is applied to a focal stack imaging device, wherein the device comprises: a power supply module, and a coaxial fixed-focus lens, a liquid lens and a CMOS array; the method comprises: Controlling the power supply module to provide a driving voltage for the liquid lens to form an image on the CMOS array; Acquire the images collected by the CMOS array, and perform focal stack alignment on all collected images based on the image when the driving voltage is the lowest, so as to obtain an image sequence of aberration-free constant magnification focal stack; Performing a focus state evaluation on the image sequence, and performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image; An object distance of the acquired image is determined, a depth map is generated based on the object distance and the all-in-focus image, and a target image is generated based on the all-in-focus image and the depth map.

[0005] In a possible implementation, controlling the power supply module to provide a driving voltage for the liquid lens includes: Determining a driving voltage required for the liquid lens based on the focal depth, target depth of field and image distance of the focal stack imaging device; Based on the driving voltage required by the liquid lens, the power supply module is controlled to provide the driving voltage to both ends of the liquid lens, so as to form an image of the target depth of field on the CMOS array.

[0006] In a possible implementation, all acquired images are focally stacked based on the image at the lowest driving voltage to obtain an image sequence of aberration-free constant magnification focal stack, including: Using the image captured by the CMOS array when the driving voltage is the lowest as a reference image, and magnifying the focal stack image of the CMOS array to an image having the same imaging magnification as the reference image based on a feature matching algorithm, so as to obtain a constant magnification focal stack image sequence without aberration; The focal stack includes images focused at different spatial depths and collected by the CMOS array.

[0007] In a possible implementation, the image of the focal stack of the CMOS array is magnified to an image having the same imaging magnification as the reference image based on a feature matching algorithm to obtain a constant magnification focal stack image sequence without aberration, including: Matching feature points of two adjacent frames of images in the focal stack of the CMOS array to determine a transformation relationship between the two adjacent frames of images; Based on the transformation relationship between each frame image in the focal stack of the CMOS array and all two frames of images between the reference image, each frame image is enlarged to an image with the same imaging magnification as the reference image to obtain a constant magnification focal stack image sequence without aberration.

[0008] In a possible implementation manner, performing focus state evaluation on the image sequence includes: The pixel clarity of each frame of the image in the image sequence is evaluated, and a focus state evaluation result is obtained based on the evaluation result of the pixel clarity.

[0009] In a possible implementation, the clarity of pixels of each frame of the image in the image sequence is evaluated, and a focus state evaluation result is obtained based on the evaluation result of the clarity of pixels, including: Convert each frame of the image in the image sequence into a grayscale image; Calculating the focus and clarity of the grayscale image within a window of a set size; Based on the focus and clarity of the grayscale image in a window of a set size, an evaluation result of the pixel clarity is obtained, and based on the evaluation result of the pixel clarity, a focus state evaluation result is obtained.

[0010] In a possible implementation, performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image includes: Based on the focus state evaluation result, the clearest imaging pixels of each frame image in the image sequence are spliced ​​into a fully focused image.

[0011] In a possible implementation manner, generating a depth map based on the object distance and the all-focus image includes: Extracting two-dimensional features of each frame of image, forming a focal body based on the two-dimensional features, and performing differentiation processing on the focal body based on the object distance to obtain a differential focal body; Extracting features from the differential focus volume, and predicting the probability of a pixel being in an optimal focus state based on the extracted features; Based on the predicted probability of the pixel being in the best focus state, probability regression is performed and combined with the focal length of each frame image to generate a depth map reflecting the image spatial depth information.

[0012] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the focal stacking imaging method of the liquid lens as described in any one of the above.

[0013] On the other hand, the present invention also provides a focal stacking imaging system of a liquid lens, comprising: a power supply module, and a coaxial fixed-focus lens, a liquid lens and a CMOS array, and the above-mentioned electronic device, wherein the electronic device is electrically connected to the power supply module and is communicatively connected to the CMOS array.

[0014] The beneficial effects of adopting the above implementation manner are as follows: the focal stack imaging method, system and electronic device of the liquid lens provided by the present invention obtain the image captured by the CMOS array, and perform focal stack alignment on all the captured images based on the image when the driving voltage is the lowest, so as to obtain an image sequence of aberration-free constant-magnification focal stack; when the driving voltage of the liquid lens is the lowest, the imaging system focuses on the farthest object and has the largest imaging magnification, therefore, the image captured when the driving voltage is the lowest is selected as the reference image, all the captured images are magnified to have the same imaging magnification as the reference image, and then an aberration-free constant-magnification focal stack image sequence is obtained, thereby reducing the more obvious splicing misalignment and pixel loss in the full-focus image generated by the focal stack image, thereby solving the technical problem of low imaging accuracy in the existing focal stack imaging technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flow chart of an embodiment of a focal stack imaging method of a liquid lens provided by the present invention; Figure 2 A focal stacking acquisition method based on a liquid lens provided by the present invention; Figure 3 The ZEMAX optical path simulation model provided by the present invention under the driving voltage of 80V and 55V; Figure 4 A schematic diagram of the all-focus image synthesis provided by the present invention; Figure 5 A schematic diagram of the depth of field of the optical imaging system provided by the present invention; Figure 6 A schematic diagram of a depth of field-image distance relationship curve of a focal stack acquisition camera provided by the present invention; Figure 7 A schematic diagram of a driving voltage-focal length relationship curve of a focal stack acquisition camera provided by the present invention; Figure 8 A schematic diagram of a driving voltage-object distance relationship curve of a focal stack acquisition camera provided by the present invention; Fig. 9 A schematic diagram of a depth estimation module based on the electrical addressing characteristics of the focal length of a liquid lens provided by the present invention; Fig.10 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0019] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0020] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0021] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] The present invention provides a focal stacking imaging method, system and electronic device of a liquid lens, which are respectively described below.

[0023] The present invention provides a focal stack imaging method of a liquid lens, which is applied to a focal stack imaging device, wherein the device comprises: a power supply module, and a coaxial fixed-focus lens, a liquid lens and a CMOS (complementary metal oxide semiconductor) array; Figure 1 As shown, the method includes: S101, controlling a power supply module to provide a driving voltage for the liquid lens to form an image on the CMOS array; S102, acquiring images acquired by the CMOS array, and performing focal stack alignment on all acquired images based on the image when the driving voltage is the lowest, to obtain an image sequence of aberration-free constant magnification focal stack; S103, performing a focus state evaluation on the image sequence, and performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image; S104, determining an object distance of the acquired image, generating a depth map based on the object distance and the all-focused image, and generating a target image based on the all-focused image and the depth map.

[0024] It is understandable that when a CMOS array, i.e., a CMOS image sensor array, uses a traditional optical zoom system to collect focal stacks, it is necessary to adjust the distance between the CMOS image sensor and the glass lens through mechanical displacement. This mechanical movement usually requires the use of micro-electromechanical structures such as servo motors, piezoelectric motors, and voice coil motors. This lens movement achieved by a micro-electromechanical structure usually has large sequential traversal and inertia, making it difficult to achieve jump interval sampling and dense sampling of the object space scene. In addition, the mechanical movement of the lens is difficult to ensure that it is completely parallel to the optical axis, which may cause lateral drift dislocation and rotational dislocation between the focal stack images. In addition, the mechanical movement of the glass lens will cause a more obvious change in the back focal length of the entire imaging system, thereby causing significant differences in the imaging magnifications of different images in the focal stack.

[0025] In some embodiments, controlling the power supply module to provide a driving voltage for the liquid lens includes: Determining a driving voltage required for the liquid lens based on the focal depth, target depth of field and image distance of the focal stack imaging device; Based on the driving voltage required by the liquid lens, the power supply module is controlled to provide the driving voltage to both ends of the liquid lens, so as to form an image of the target depth of field on the CMOS array.

[0026] It is understandable that for a traditional optical imaging system, the light emitted by an object point forms a circular diffuse light spot on the CMOS image sensor after being refracted by the lens. Only when the entire diffuse light spot is completely within a photosensitive pixel, the image of the object point is clear. Although some object points are not fully focused on the CMOS image sensor, the diffuse light spots formed by them after being refracted by the lens are still completely within a pixel, so their images are still clear. These object points can all be clearly imaged, and this distance is the depth of field.

[0027] The target depth of field can be determined based on the focal depth and image distance of the focal stack imaging device and the driving voltage required for the liquid lens. Conversely, the depth of field of the image formed on the CMOS array can also be determined based on the driving voltage.

[0028] In some embodiments, all captured images are focally stacked based on the image when the driving voltage is the lowest to obtain an image sequence of aberration-free constant magnification focal stack, including: Using the image captured by the CMOS array when the driving voltage is the lowest as a reference image, and magnifying the focal stack image of the CMOS array to an image having the same imaging magnification as the reference image based on a feature matching algorithm, so as to obtain a constant magnification focal stack image sequence without aberration; The focal stack includes images focused at different spatial depths and collected by the CMOS array.

[0029] It can be understood that the present invention proposes a focus stack alignment algorithm for the imaging magnification change problem. When the driving voltage of the liquid lens is the lowest, the imaging system focuses on the farthest object and has the largest imaging magnification. Therefore, the image collected when the driving voltage is the lowest is selected as the reference image, and the remaining frames in the focus stack are aligned with the reference image through the feature point matching algorithm and enlarged to have the same imaging magnification as the reference image, thereby obtaining a constant magnification focus stack image sequence without aberration.

[0030] In some embodiments, the image of the focal stack of the CMOS array is magnified based on a feature matching algorithm to an image having the same imaging magnification as the reference image to obtain a constant magnification focal stack image sequence without aberration, comprising: Matching feature points of two adjacent frames of images in the focal stack of the CMOS array to determine a transformation relationship between the two adjacent frames of images; Based on the transformation relationship between each frame image in the focal stack of the CMOS array and all two frames of images between the reference image, each frame image is enlarged to an image with the same imaging magnification as the reference image to obtain a constant magnification focal stack image sequence without aberration.

[0031] It is understandable that the present invention uses the SIFT operator for feature extraction, and uses the Flann and Knn algorithms for feature matching between images. Since the focusing and defocusing phenomena of different areas in the image in the focus stack image sequence collected by the liquid lens are very different, the two non-adjacent frames in the stack appear to be very different, and the glare and overexposure problems that may occur during the image acquisition process will further reduce the success rate of the feature point matching algorithm. Therefore, when performing feature point matching, feature point matching is first performed between two adjacent frames in the focus stack, and the transformation relationship between the two adjacent frames is determined based on the RANSAC registration algorithm. Two consecutive adjacent frames of focus stack images have relatively close focusing and defocusing phenomena, which ensures the success rate of the feature point matching algorithm. Finally, the transformation relationship between each frame image and the adjacent frames of the reference image is connected in series to obtain its transformation relationship with the reference image.

[0032] In some embodiments, performing focus status evaluation on the image sequence includes: The pixel clarity of each frame of the image in the image sequence is evaluated, and a focus state evaluation result is obtained based on the evaluation result of the pixel clarity.

[0033] In some embodiments, evaluating the clarity of pixels of each frame of the image in the image sequence, and obtaining a focus state evaluation result based on the evaluation result of the clarity of pixels, includes: Convert each frame of the image in the image sequence into a grayscale image; Calculating the focus and clarity of the grayscale image within a window of a set size; Based on the focus and clarity of the grayscale image in a window of a set size, an evaluation result of the pixel clarity is obtained, and based on the evaluation result of the pixel clarity, a focus state evaluation result is obtained.

[0034] It can be understood that the focal stack image collected by the liquid lens is converted into a grayscale image. For each pixel, the degree of focus and clarity can be evaluated by calculating the Laplace variance of the pixels in the 3×3 window where it is located. Since the defocused area in the image is similar to a low-pass filter, it causes severe attenuation of high-frequency information in the image. Therefore, the Laplacian operator is used to calculate the second-order derivative when solving the intensity change of the area in the 3×3 window.

[0035] In some embodiments, performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image includes: Based on the focus state evaluation result, the clearest imaging pixels of each frame image in the image sequence are spliced ​​into a fully focused image.

[0036] It can be understood that the 15 frames of focus stack images collected in the imaging test experiment are aligned using the focus stack alignment algorithm proposed above to obtain a focus stack image with a constant magnification. Since no alignment is performed, there are many stitching misalignment areas in the synthesized full-focus image, such as the metal rod on the left side of the image with obvious misalignment stitching, which shows that the algorithm proposed above can effectively solve the imaging magnification change caused by the change in the focal length of the liquid lens and generate a focus stack image with a constant magnification.

[0037] In some embodiments, generating a depth map based on the object distance and the all-focus image includes: Extracting two-dimensional features of each frame of image, forming a focal body based on the two-dimensional features, and performing differentiation processing on the focal body based on the object distance to obtain a differential focal body; Extracting features from the differential focus volume, and predicting the probability of a pixel being in an optimal focus state based on the extracted features; Based on the predicted probability of the pixel being in the best focus state, probability regression is performed and combined with the focal length of each frame image to generate a depth map reflecting the image spatial depth information.

[0038] In some embodiments, the liquid lens-based focal stack fast acquisition principle provided by the present invention is as follows: Figure 2 As shown in (a), the fixed-focus lens is used to increase the focusing ability of the imaging system on the light beam, thereby reducing the total mechanical length of the entire imaging system. It is a functional lens that compresses the optical path. Its position in the entire imaging system is fixed and remains unchanged. The imaging system only relies on the liquid lens to adjust the focal length. Therefore, except for the oscillation of the liquid-liquid interface in the liquid lens, the entire imaging system has no mechanical moving parts. The imaging targets A, B and C are located at different object distances in the object space. At this time, the three objects A, B and C are refracted by the fixed-focus lens and the liquid lens and are imaged at A', B' and C' respectively. At this time, image A' is located in front of the CMOS image sensor and is in a defocused state; image B' coincides with the CMOS image sensor and is in a focused state; image C' is located behind the CMOS image sensor and is in a defocused state. When the driving voltage increases, the focusing ability of the imaging system on the light beam is enhanced, and image C' moves toward the lens direction to coincide with the CMOS image sensor. When the driving voltage V decreases, the focusing ability of the imaging system on the light beam is weakened, and image A' moves toward the CMOS image sensor to coincide with it.

[0039] The prepared liquid lens has a zoom speed of milliseconds, so the imaging system can achieve a large range of focal length adjustment in a short time. Figure 2As shown in (b) in FIG. 2 , by greatly adjusting the driving voltage of the liquid lens, the focal plane of the imaging system can be quickly moved between objects A, B, and C that are far apart, thereby realizing jump interval sampling of the object space scene. This jump interval sampling mode can collect a focal stack image sequence with a large object distance range in a short time, and then synthesize a fully focused image with a large depth of field range. According to the imaging performance test results in FIG. 0 , the prepared liquid lens is still sensitive to a small change in the driving voltage of 0.1 V, so the dense sampling of the object space scene can be achieved by continuously and densely adjusting the driving voltage. As shown in (c) in FIG. 2 , dense sampling of object B can be achieved by continuously adjusting the driving voltage with a small amplitude. This dense sampling mode helps to synthesize fully focused images with richer texture details and higher resolution, and has a very broad application prospect in fields such as super-resolution imaging.

[0040] Changes in imaging magnification caused by changes in lens focal length: When using a traditional mechanical zoom system for focal stack acquisition, the mechanical displacement of the lens and the change in the distance between the lens and the CMOS image sensor will cause obvious differences in imaging magnification and field of view between the acquired image sequences. This phenomenon is called "lens breathing". Although the spatial relationship between the optical elements in the imaging system with a liquid lens as the core zoom element is fixed, the liquid-liquid interface in the liquid lens will still undergo slight displacement and deformation after the driving voltage changes. This deformation and displacement will still cause a slight change in the imaging magnification of the focal stack image sequence.

[0041] Based on the focal length data of the liquid lens obtained from experimental measurements, the optical path simulation model was established using the ZEMAX simulation software. For an imaging system consisting of two fixed-focus glass lenses and a liquid lens, when a driving voltage of 80 V is applied, the optical path simulation model parameter settings are shown in Table 1.

[0042] Table 1: Liquid lens ZEMAX simulation model parameters

[0043] When the driving voltage changes, only the thickness and curvature corresponding to plane 3, plane 4, plane 5 and plane 6 in Table 1 need to be changed. The optical path simulation models when the driving voltage is set to 80V and 55V are respectively Figure 3As shown in (a) and (b) in the figure. The distance from the object plane to the first fixed-focus glass lens is set to 1000mm, and the object height is 150mm. According to the simulation results of the ZEMAX optical path simulation model, when the driving voltage is 55V, the image height of the object on the CMOS image sensor is 4.542mm. When the driving voltage is 80V, the image height of the object on the CMOS image sensor is 3.688mm. The simulation results show that as the driving voltage changes, the image height of the object on the CMOS photosensitive chip changes, that is, the imaging magnifications of different images in the focal stack are inconsistent, and there are aberrations. And the smaller the driving voltage, the greater the imaging magnification.

[0044] There is a slight change in imaging magnification between the focal stack image sequences captured by the liquid lens in the imaging performance test. When the driving voltage is 59.0V, the pixel size of the "stop" pattern of the school bus body in the collected focal stack image is 66×66 pixels. When the driving voltage is 60.0V, the pixel size of the "stop" image of the school bus body in the collected focal stack image is 63×63. The change in pixel size indicates that the imaging magnification of the focal stack image sequence collected by the liquid lens is inconsistent, and the imaging magnification when the driving voltage is 59.0 V is slightly greater than the imaging magnification when the driving voltage is 60.0V, which is basically consistent with the analysis results of the optical path simulation model established based on ZEMAX. This imaging magnification change will cause the full focus image synthesized by the focal stack image sequence to produce splicing misalignment and pixel loss. It is necessary to align the focal stack through hardware or software methods to make the entire image sequence have a constant magnification.

[0045] Focus stack alignment algorithm: Some existing solutions, such as using six liquid lenses to jointly focus in an imaging system to achieve constant magnification imaging, but the increase in the number of liquid lenses will lead to a decrease in the imaging quality of the imaging system, and it is more complicated and time-consuming to adjust the focal length of multiple liquid lenses at the same time. Someone has proposed a liquid lens with a tunable liquid-liquid interface, which keeps the imaging magnification and back focal length of the imaging system constant by adjusting the position of the liquid-liquid interface, but its structure is relatively complex, the focusing speed is slow, and it is difficult to use in real-time application scenarios. Inspired by the tandem optical flow algorithm proposed by Suwajanakorn, the present invention proposes a focus stack alignment algorithm for the problem of imaging magnification change. According to the above simulation and experimental results, when the driving voltage of the liquid lens is the lowest, the imaging system focuses on the farthest object and has the largest imaging magnification. Therefore, the image collected when the driving voltage is the lowest is selected as the reference image, and the remaining frames in the focus stack are aligned with the reference image through the feature point matching algorithm shown in Table 2 and enlarged to have the same imaging magnification as the reference image, thereby obtaining a constant magnification focus stack image sequence without aberration.

[0046] Table 2: Focus stack alignment algorithm flow

[0047] The algorithm flow is shown in Table 2. The SIFT operator is used for feature extraction, and the Flann and Knn algorithms are used for feature matching between images. Since the focusing and defocusing phenomena of different areas in the focus stack image sequence collected by the liquid lens are very different, the two non-adjacent frames in the stack appear to be very different, and the glare and overexposure that may occur during the image acquisition process will further reduce the success rate of the feature point matching algorithm. It is impossible to establish sufficient pixel mapping relationships when the SIFT operator is used for feature point matching of the two frames of images collected when the driving voltage is 65V and 51V. Therefore, when performing feature point matching, feature point matching is first performed between the two adjacent frames in the focus stack, and the transformation relationship between the two adjacent frames is determined based on the RANSAC registration algorithm. The two consecutive adjacent frames of the focus stack image have relatively close focusing and defocusing phenomena, which ensures the success rate of the feature point matching algorithm. Finally, the transformation relationship between each frame image and the adjacent frames of the reference image is connected in series to obtain the transformation relationship between it and the reference image.

[0048] Depth of field extension and full focus image synthesis based on liquid lens: Depth of Field (DoF) refers to the distance before and after the focal plane that can be clearly imaged, and is an important performance parameter of the optical imaging system. The depth of field of a traditional imaging system is usually determined by parameters such as the focal length of the lens, the object distance, and the size of the photosensitive unit of the CMOS image sensor, and is relatively limited. By fusing the focus stack image to generate a fully focused image that is clearly imaged everywhere in the object space, the imaging system equipped with a liquid lens can have a larger depth of field range than the traditional imaging system. Figure 4 As shown in the figure, generating a fully focused image with a large depth of field through focus stacking is mainly divided into two steps: focus evaluation (FocusMeasure, FM) and image stitching. Focus evaluation refers to evaluating the clarity of each pixel in each frame of the image collected by the liquid lens, and establishing an index relationship (x, y, I, E) with the image number to which the pixel belongs, where (x, y) is the pixel with horizontal and vertical coordinates x and y in the image numbered I, and E is the focus evaluation result of the pixel. After establishing the index relationship (x, y, I, E), the fully focused image is generated by selecting the pixels with the clearest imaging in each frame of the image and stitching them together, as shown in the figure. Figure 4 As shown in (b) in .

[0049] 1. Focus on the evaluation function The focal stack image collected by the liquid lens is converted into a grayscale image. For each pixel, the Laplace variance of the pixel in the 3×3 window can be calculated to evaluate the focus and clarity. Since the defocused area in the image is similar to a low-pass filter, it causes severe attenuation of high-frequency information in the image. Therefore, the Laplacian operator is used to calculate the second-order derivative when solving the regional intensity change in the 3×3 window. E(x, y) is used to represent the regional intensity change, that is, the clarity.

[0050] (1) (2) The aligned focus stack images are named , where k = 1, …, N. Where N is the number of image frames in the focal stack. First, the image in the focal stack is converted into a grayscale image. Based on equations (1) and (2), the regional intensity change E(x, y) of all pixels (x, y) in the k-th frame image is calculated in turn and a four-tuple (x, y, I, E) is established to record the clarity-image number index relationship. The intensity change result E(x, y) of each pixel (x, y) is normalized and converted into a grayscale value. Among them, the driving voltage is 53V, 55V and 59V. According to the grayscale image, it can be seen that under the action of 53V, 55V and 59V, the imaging system focuses on the farthest tank, the fire truck in the middle distance and the nearest school bus respectively, which is consistent with the actual situation.

[0051] 2. All-focus image fusion method The stitching problem of all-focus images can be converted into a multi-label MRF optimization problem. The image composed of pixels can be regarded as a 4-connected grid, each pixel constitutes a node set ν, and the connection between adjacent pixels constitutes an edge set ε. The energy function is defined as: (3) In the formula is a weighting constant used to balance ν and ε. is the pixel defocus degree calculated by formula (2). Represents the change of image sequence number , the α-expansion algorithm can be used to iterate and calculate to minimize the energy function. The α-expansion algorithm tries to switch its label to the label of the adjacent node during the iteration process. If the energy function decreases, the label replacement is accepted, otherwise the iteration continues. After iteration, the energy function is minimized, and a good full-focus image can be synthesized by splicing according to the labels of each pixel node.

[0052] Using the focus stack alignment algorithm proposed above, the 15 frames of focus stack images collected in the imaging test experiment were aligned to obtain a focus stack image with a constant magnification. The fully focused image generated by the above image algorithm shows clear images of tanks, fire trucks and school buses, and the image depth of field is significantly improved. There are some blurred areas on the roof of the school bus. This is because the brightness of the lighting LED light source used during shooting is set too high, resulting in overexposure of the image in this area. The original focus stack image collected loses the details of this part of the image, resulting in partial blur. Due to the lack of alignment, there are many splicing misalignment areas in the synthesized fully focused image, such as the metal bar on the left side of the image has obvious misalignment splicing, which shows that the algorithm proposed above can effectively solve the imaging magnification change caused by the change in the focal length of the liquid lens and generate a focus stack image with a constant magnification.

[0053] 3. Extended depth of field based on liquid lenses In order to further quantitatively analyze the depth of field extension capability of the liquid lens on the imaging system, this section combines the prepared liquid lens with two fixed-focus glass lenses and a CMOS image sensor to make a focus stack acquisition camera. The fixed-focus glass lenses used in the focus stack acquisition camera are two H-K9L plano-convex glass lenses with a curvature radius of 25.75mm and a diameter of 12.7mm. The fixed-focus glass lenses mainly play two roles: one is to increase the focusing ability of the imaging system on the light beam, compress the optical path, and reduce the mechanical length of the imaging system; the other is to couple with the liquid lens to expand the image field so that the image field covers the CMOS image sensor as much as possible. The distance between the two fixed-focus glass lenses is set to 7mm to facilitate the insertion of the liquid lens, and the distance between the fixed-focus glass lens and the CMOS image sensor is 22.2mm. The size of the photosensitive unit of the selected CMOS image sensor is 2.2×2.2μm, the array size of the photosensitive unit is 2592×1944, the signal-to-noise ratio is 39dB, and the dynamic range is 85dB. Connectors are prepared by 3D printing to connect the components together.

[0054] Based on ZEMAX, an optical path simulation model of the focal stack acquisition camera is established. The prepared liquid lens can be equivalent to a double glued lens with a clear aperture of 5 mm. The two components of the equivalent double glued lens are 0.1% SDS deionized water solution and dimethyl silicone oil. Based on the Schott method, the refractive index of the two liquid materials is fitted and added to the ZEMAX glass material library. Then, the focal length and depth of field of the focal stack acquisition camera under different curvatures can be obtained through simulation analysis.

[0055] For traditional optical imaging systems, the light emitted by an object point forms a circular diffuse light spot on the CMOS image sensor after being refracted by the lens. Only when the entire diffuse light spot is completely within a photosensitive pixel, the image of the object point is clear. Figure 5 As shown in the figure, although object points A and C are not completely focused on the CMOS image sensor like object point B, the diffuse light spots formed by them after refraction by the lens are still completely within one pixel, so their images are still clear. All object points between object points A and C can be clearly imaged, and this distance is the depth of field. Assume that the current image distance is focused on object point B with an object distance of u, and the image distance is v. The size of a single pixel of the CMOS image sensor is δ, and D is the aperture size of the main lens. Then the front focal depth of the imaging system is and back focus depth It can be expressed as: (4) (5) Combining equations (4) and (5) with the thin lens imaging formula, we can obtain the depth of field of the imaging system: (6) When the liquid lens is not inserted, the focal length of the fixed-focus lens composed of two H-K9L fixed-focus glass lenses with a curvature radius of 25.75 mm and a spacing of 7 mm is 27.4435 mm. When the pixel size of the CMOS image sensor is 2.2 μm, the relationship curve between the depth of field and the image distance can be calculated by formula (6) as follows: Figure 6 As shown. The image distance is the distance between the equivalent optical center of the two fixed-focus lenses and the CMOS image sensor, which is 28.3mm. At this image distance, the depth of field of the focus stack acquisition camera is 23.2mm, and the depth of field range is relatively narrow.

[0056] The three imaging targets, tank, airplane and truck, were placed at 920mm, 740mm and 540mm in front of the focal stack acquisition camera, respectively. By adjusting the driving voltage, 12 focal stack images focused at different object distances were acquired. Among them, when the driving voltage was 61.2V, 62.4V and 63.0V, the imaging system focused on the tank, airplane and truck, respectively. Based on the focal stack alignment algorithm proposed above, an aberration-free image sequence with constant magnification was generated. In the fully focused image, objects within the object distance range of 540~920mm are clearly imaged, which shows that the depth of field of the system has been successfully extended to 380mm based on the liquid lens, which is at least an order of magnitude higher than the original depth of field range of 23.2mm.

[0057] And because the prepared liquid lens has an operating voltage range of 0~110V, it is possible to continue to capture focal stack images within a larger object distance range and synthesize a fully focused image with a depth of field range greater than 380mm by expanding the driving voltage adjustment range. In summary, the proposed focal stack acquisition method and fully focused image synthesis method based on liquid lenses can effectively achieve large depth of field imaging that exceeds traditional optical imaging systems.

[0058] Fast depth estimation method based on liquid lens: 1. Depth estimation principle based on liquid lens The depth information of an image, that is, the distance of an object from the lens in the image, is the key clue to restore the three-dimensional space scene of the object. In recent years, monocular depth estimation methods based on deep learning have developed rapidly, and can estimate depth directly from a single frame of an image, but this method relies heavily on image semantic information and requires a large amount of data sets for training. A focus stack is essentially a series of images focused at different spatial depths. In addition to the semantic information in each image in the stack, the focus and defocus state differences between different images in the stack can also be used for depth estimation. Methods for depth estimation based on focus stacks can be divided into depth estimation methods based on the degree of focus clarity (Depth from focus, DFF) and depth estimation methods based on the degree of defocus blur (Depth from defocus, DFD). The DFF technique assumes that each imaging target in the focus stack has a frame of its clearest image. This assumption is valid for densely sampled focus stack images. The use of DFF technology to quickly estimate the depth of object space scenes mainly faces the following problems: (1) rapid acquisition of focus stack image sequences; (2) locating the pixels with the best focus state in each frame image; (3) determining the focus distance of each frame image in the focus stack.

[0059] (1) Rapid acquisition of focal stack image sequences DFF technology places high demands on the dynamic focusing capability of the image sensor used to collect focal stacks. First, the image sensor should have a large focal length variation range to collect image sequences with a large object distance range. Second, the image sensor should have a high zoom speed to reduce the time required for image acquisition. Third, the focal length of the image sensor should be able to be continuously adjusted in small amplitudes to collect dense focal stacks. According to the optical properties of the prepared liquid lens, the effective focal length of the liquid lens can be between (-∞, -128.6mm) It can be continuously adjusted within the range of (48.6mm, +∞), with a large focal length variation range. Secondly, the liquid lens has a zoom speed of milliseconds, which can achieve image acquisition of about 10 to 30 frames per second. From the imaging performance test results of the liquid lens, it can be seen that the lens is still sensitive to small changes in the driving voltage, and dense sampling of the object space scene can be achieved. Therefore, the prepared liquid lens is sufficient to meet the requirements of DFF technology for image sensors. In addition to the above three points, the advantages of the liquid lens such as no mechanical moving parts also avoid the aberration caused by mechanical disturbance of the lens.

[0060] (2) Locate the pixel with the best focus state in each frame image Depth estimation through DFF technology also faces the problem of evaluating the focus degree of pixels, especially when estimating the depth of real spatial scenes, there will be a large number of textureless areas. The traditional Laplacian operator is difficult to accurately evaluate the focus degree of pixels in textureless areas. Secondly, in order to ensure that there is a frame of pixels with the clearest imaging for each coordinate pixel, a densely sampled focus stack is often used as input, which usually requires dozens of frames of images as input. The increase in the number of input images increases the image sequence acquisition cost on the one hand, and reduces the running speed of the model on the other hand. The network model based on differential focus volume (DFV) solves the above problems well. It uses the powerful feature extraction capability of 2D convolutional neural network (CNN) to evaluate the focus degree of pixels, and uses the focus stack image and the focus distance of each frame image to construct DFV. After successfully constructing DFV, it uses 3D CNN to predict the probability of pixels in each frame image reaching the best focus state, and combines it with the focus distance (object distance) when each frame image is acquired to complete depth estimation. This model currently has high depth estimation accuracy in multiple public datasets.

[0061] (3) Determination of the focus distance of each frame image in the focus stack How to obtain the focus distance of each frame of the focus stack image is the key to using DFF technology to perform absolute depth estimation on real-world scenes. Unlike performing depth estimation tasks on datasets, when using network models such as DFV to perform absolute depth estimation on real-space scenes, it is also necessary to obtain the focus distance of each frame image in the focus stack, that is, the object distance of the object corresponding to the clearest pixel in the image. In previous work, traditional optical zoom cameras were calibrated mainly through manual calibration and other methods to obtain the focus distance of the image. The focus distance determined by this method has large deviations and poor generalization. Liquid lenses based on the dielectric wetting effect have a very obvious advantage in the rapid calculation of focus distance.

[0062] 2. Focal electrical addressing characteristics of liquid lenses There is a one-to-one electrical addressing relationship between the focal length and driving voltage of the prepared liquid lens. This electrical addressing relationship enables the driving voltage of the liquid lens to be used to quickly calculate the focal length of the liquid lens and the focal length of the imaging system, thereby realizing the rapid calculation of spatial depth information such as depth of field and object distance. Taking the focal stack acquisition camera constructed above as an example, the optical path model is established based on ZEMAX to calculate the relationship between the focal length and driving voltage of its system. The obtained driving voltage-focal length relationship curve is shown in the figure below: Figure 7As shown. When the image of the object after refraction by the lens is located exactly at the CMOS image sensor, the image of the object is clearest. At this time, the image distance v is the distance from the equivalent optical center of the lens to the plane of the CMOS image sensor. From the lens imaging formula, it can be seen that there is a relationship between the focal length f of the lens, the object distance u and the image distance v. Figure 7 Substituting the simulation data in into equation (7) can calculate the focusing distance of the imaging system under different driving voltages, that is, the object distance between the clearest pixel point in the image and the object point, as Figure 8 As shown. Then, a one-to-one corresponding index relationship is established between the driving voltage and the focusing distance. This index relationship is calculated based on the objective physical law of the Lippmann-Young equation. Using it as a clue to estimate the depth information of the object space scene can ensure the estimation accuracy and generalization of the depth estimation method.

[0063] In summary, the unique focal length electrical addressing characteristics of the dielectric wetting liquid lens make it very suitable for combining with the DFV network model with high depth estimation accuracy at this stage to perform quantifiable absolute depth estimation of the object space scene.

[0064] 3. Depth Estimation Module Architecture and Implementation The network is built according to the DFV-DFF network architecture. The overall framework of the depth estimation module is as follows: Fig. 9 As shown in the figure, a driving voltage-object distance conversion module based on the focal length electrical addressing characteristics of the liquid lens is introduced on the basis of the network model. During training, the model input is the focus stack image in the public dataset FoD500 and DDFF-12. When conducting depth estimation experiments on real space scenes, the model input is the focus stack image sequence collected by the liquid lens. It is converted into an aberration-free focus stack image sequence with a constant magnification through the above-mentioned focus stack alignment and aberration correction algorithm. Each frame image in the stack is annotated with the driving voltage when it was collected. According to the Lippmann-Young equation and the lens imaging principle, the driving voltage when each frame image is collected is converted into the object distance (the focusing distance of the imaging system). The model first uses a 2D CNN model to extract the stack formed by the two-dimensional features of each frame image in the stack, namely the focus volume (FV). Next, the DFV can be constructed by calculating the first-order derivative of the two-dimensional feature stack. For the constructed DFV, the 3D CNN is then used to extract features from the DFV to predict the probability of whether the pixel is in the best focus state. Finally, a depth map reflecting the spatial depth information of the photo is formed through probabilistic regression and combined with the focus distance of each frame image.

[0065] The above model was implemented using PyTorch 1.10.2 and the Adam optimizer was used on an NVIDIA 3070 Laptop GPU with 8GB of video memory (learning rate lr = 0.0001, = 0.9, = 0.9) to optimize the model. The experiment uses a fixed random seed, a batch size of 10, and 500 training rounds. During training, 5 frames of images are randomly selected from the focal stack each time and cropped to a resolution of 224×224. Random horizontal flipping and random rotation are performed during training to avoid overfitting. During training, depth maps with four resolution sizes are formed by upsampling, so the loss function is defined as follows: (7) In the formula is the predicted depth of the pixel numbered j at the sampling level s, is the true depth of the pixel numbered j, M is the total number of pixels, To balance the weighting coefficients of different sampling levels, , , , The models are trained using the public datasets DoD500 and DDFF-12.

[0066] The FoD500 dataset contains 500 sets of samples. Each set of samples contains five focus stack images with known focus distances and one depth map representing the depth value of each pixel. The resolution of each frame is 256×256. Since the dataset was originally constructed for the DFD task, the focus distance range in each set of samples does not fully cover the true depth range, and the data out of range needs to be hidden during training and testing. The first 400 sets of samples are selected as the training set, and the last 100 sets of samples are selected as the test set.

[0067] The DDFF-12 dataset is a dataset collected from 12 different real-world scenes using a light field camera. Among them, the six spatial scenes of kitchen, office A, conference room, social corner, student laboratory and glass room each contain 100 sets of samples. For the other six spatial scenes: cafeteria, library, living room, office B and laboratory, each scene contains 20 sets of samples. Each set of samples contains 10 frames of focus stack images and corresponding field of view maps. The resolution of each frame image is 383×552. Samples in the kitchen, conference room, social corner and student laboratory scenes are selected as training sets, and samples in the glass room and office 1 scenes are selected as test sets.

[0068] The experiment uses the following evaluation indicators to evaluate the model estimation accuracy. The evaluation indicators are: Mean Square Error (MSE), Root Mean Square error (RMS), Absolute Relative Error (absRel) and estimation time. The trained model is evaluated based on the above evaluation indicators, and the results are shown in Table 3. It shows that the model trained by the network architecture proposed in the present invention has good depth estimation accuracy.

[0069] Table 3: Test results of the model on FoD500 and DDFF-12 datasets

[0070] 4. Absolute depth estimation experiment in real scenes The fast depth estimation method based on liquid lens proposed in the present invention is experimentally tested based on a preset experimental scene, in which three imaging targets of a tank, an airplane and a truck are placed at 920mm, 740mm and 540mm in front of the camera respectively. Two focus stack images with driving voltages of 61.2V and 63.0V are used as input. The model generated by training can better evaluate the focus probability of the pixels in each frame of the image. The output focus probability map shows that the model can better predict the optimal focus probability of the pixels in each frame of the image, and the object contour recognition is more accurate.

[0071] The warm and cold tones in the focus probability map reflect the probability of the pixel's focus state. The warmer the tones, the better the focus state. When the driving voltage is 61.2V, the imaging system focuses on the tank. At this time, the tones of the tank area in the focus probability map output by the model are significantly warmer than other areas. When the driving voltage is 63.0V, the imaging system focuses on the nearest truck, and the tones of the truck area are significantly warmer than other areas.

[0072] The depth image finally output by the model reflects the distance of objects in each area of ​​the image from the camera lens. The warm and cold tones of the image area in the depth map reflect the distance of the object from the lens. The warmer the tones, the greater the object distance, and the colder the tones, the smaller the object distance. The imaging area corresponding to the truck has a cold tone, the image area corresponding to the aircraft has a moderate tone, and the imaging area corresponding to the tank has a warm tone, which better reflects the spatial relative depth relationship of the three imaging targets. Although only two frames of focus stack images focusing on the tank target and the truck target were input, based on the powerful feature extraction capability of CNN, the DFV network model still accurately rated the focus clarity of each pixel through probabilistic regression and accurately estimated the object distance of the aircraft.

[0073] The driving voltage-object distance conversion module based on the focal length electrical addressing characteristics of the liquid lens converts the focus distance of the two frames of input images into the focus distance of the picture, and then estimates the object distance of the object corresponding to each pixel in the picture, realizing the reconstruction of the three-dimensional scene in the object space. The color bar on the right side of the depth map indicates the correspondence between the cold and warm tones in the depth map and the actual object distance. The actual object distance of the three imaging targets is compared with the object distance estimated by the model. The comparison data is shown in Table 4.

[0074] Table 4: Actual object distance of imaging target and estimated object distance estimated by algorithm

[0075] The experimental results show that by combining the focal length electrical addressing characteristics of the liquid lens with the DFV depth estimation network model, the object distance corresponding to each pixel in the focal stack image can be estimated with only two frames of focal stack images. The object distance of the rearmost background in the image is incorrectly estimated in the output depth map, which may be due to the interference of the overly large letters in the background of the image on the pixel clarity evaluation, which can be improved by increasing the number of focal stack image input frames. In summary, the proposed depth estimation method based on liquid lens has high depth estimation accuracy within a certain object distance range, with a maximum error rate of 4.1%, and an average error rate of 3.43% for the estimation results of the three imaging targets. And from the experimentally measured dynamic response time of the liquid lens and the model running time, it can be seen that the imaging system can complete the focal stack image acquisition within 200ms and complete the depth estimation task within about 50ms, realizing the fast depth estimation of the object space scene.

[0076] The present invention proposes a multiple-mode fast acquisition method for focal stacks based on the electrically controlled focusing performance of a liquid lens. For application scenarios such as depth of field extension and fast depth estimation, a jump interval sampling mode is proposed to quickly obtain a focal stack image sequence with a large object distance range. For application scenarios such as super-resolution imaging, a dense sampling mode is proposed to quickly obtain a focal stack image sequence with dense sampling of the object space. In view of the change in magnification of the focal stack image caused by the change in the back focal length during the zooming process of the liquid lens, a corresponding focal stack alignment and aberration correction algorithm is proposed. The experimental results show that the fully focused image synthesized by the aligned focal stack image sequence effectively avoids image splicing dislocation and has high imaging quality. Based on the excellent electrically controlled focusing performance of the liquid lens, the present invention proposes a corresponding depth of field extension and image fusion algorithm, which increases the depth of field of the imaging system by at least one order of magnitude. The present invention combines the Lippmann-Young equation with the differential focal volume network model to realize the object distance estimation of the imaging target in the object space scene, with a maximum error of 4.1% and an average error of 3.4%.

[0077] like Fig.10 As shown, the present invention also provides an electronic device 1000. The electronic device 1000 includes a processor 1001, a memory 1002 and a display 1003. Fig.10 Only some components of the electronic device 1000 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0078] In some embodiments, the memory 1002 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. In other embodiments, the memory 1002 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 1000.

[0079] Furthermore, the memory 1002 may include both an internal storage unit of the electronic device 1000 and an external storage device. The memory 1002 is used to store application software installed in the electronic device 1000 and various data.

[0080] In some embodiments, the processor 1001 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 1002, such as the focal stack imaging method of the liquid lens in the present invention.

[0081] In some embodiments, the display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1003 is used to display information on the electronic device 1000 and to display a visual user interface. The components 1001-1003 of the electronic device 1000 communicate with each other via a system bus.

[0082] In some embodiments of the present invention, when the processor 1001 executes the focal stack imaging program of the liquid lens in the memory 1002, the following steps may be implemented: Controlling the power supply module to provide a driving voltage for the liquid lens to form an image on the CMOS array; Acquire the images collected by the CMOS array, and perform focal stack alignment on all collected images based on the image when the driving voltage is the lowest, so as to obtain an image sequence of aberration-free constant magnification focal stack; Performing a focus state evaluation on the image sequence, and performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image; An object distance of the acquired image is determined, a depth map is generated based on the object distance and the all-in-focus image, and a target image is generated based on the all-in-focus image and the depth map.

[0083] It should be understood that, when the processor 1001 executes the focal stack imaging program of the liquid lens in the memory 1002 , in addition to the above functions, other functions may also be implemented, for details, see the description of the corresponding method embodiments above.

[0084] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1000 mentioned, and the electronic device 1000 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1000 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0085] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for focal stack imaging of a liquid lens provided by the above methods is implemented, and the method comprises: Controlling the power supply module to provide a driving voltage for the liquid lens to form an image on the CMOS array; Acquire the images collected by the CMOS array, and perform focal stack alignment on all collected images based on the image when the driving voltage is the lowest, so as to obtain an image sequence of aberration-free constant magnification focal stack; Performing a focus state evaluation on the image sequence, and performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image; An object distance of the acquired image is determined, a depth map is generated based on the object distance and the all-in-focus image, and a target image is generated based on the all-in-focus image and the depth map.

[0086] The present invention also provides a focal stacking imaging system of a liquid lens, comprising: a power supply module, and a coaxial fixed-focus lens, a liquid lens and a CMOS array, and the above-mentioned electronic device, wherein the electronic device is electrically connected to the power supply module and is communicatively connected to the CMOS array.

[0087] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0088] The focal stacking imaging method, system and electronic device of the liquid lens provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A focal stacking imaging method of a liquid lens, characterized in that: Applied in a focal stack imaging device, the device comprises: a power supply module, and a coaxial fixed focus lens, a liquid lens and a CMOS array; the method comprises: Controlling the power supply module to provide a driving voltage for the liquid lens to form an image on the CMOS array; Acquire the images collected by the CMOS array, and perform focal stack alignment on all collected images based on the image when the driving voltage is the lowest, so as to obtain an image sequence of aberration-free constant magnification focal stack; Performing a focus state evaluation on the image sequence, and performing full focus fusion on the image sequence based on the focus state evaluation result to obtain a full focus image; An object distance of the acquired image is determined, a depth map is generated based on the object distance and the all-in-focus image, and a target image is generated based on the all-in-focus image and the depth map.

2. The focal stacking imaging method of a liquid lens according to claim 1, characterized in that: The control power supply module provides a driving voltage for the liquid lens, including: Determining a driving voltage required for the liquid lens based on the focal depth, target depth of field and image distance of the focal stack imaging device; Based on the driving voltage required by the liquid lens, the power supply module is controlled to provide the driving voltage to both ends of the liquid lens, so as to form an image of the target depth of field on the CMOS array.

3. The focal stacking imaging method of a liquid lens according to claim 1, characterized in that: Based on the image at the lowest driving voltage, all the acquired images are focal stack aligned to obtain an image sequence of aberration-free constant magnification focal stack, including: Using the image captured by the CMOS array when the driving voltage is the lowest as a reference image, and magnifying the focal stack image of the CMOS array to an image having the same imaging magnification as the reference image based on a feature matching algorithm, so as to obtain a constant magnification focal stack image sequence without aberration; The focal stack includes images focused at different spatial depths and collected by the CMOS array.

4. The focal stacking imaging method of the liquid lens according to claim 3, characterized in that: The method magnifies the focal stack image of the CMOS array to an image having the same imaging magnification as the reference image based on a feature matching algorithm, so as to obtain a constant magnification focal stack image sequence without aberration, including: Matching feature points of two adjacent frames of images in the focal stack of the CMOS array to determine a transformation relationship between the two adjacent frames of images; Based on the transformation relationship between each frame image in the focal stack of the CMOS array and all two frames of images between the reference image, each frame image is enlarged to an image with the same imaging magnification as the reference image to obtain a constant magnification focal stack image sequence without aberration.

5. The focal stacking imaging method of the liquid lens according to claim 1, characterized in that: The focusing state of the image sequence is evaluated, comprising: The pixel clarity of each frame of the image in the image sequence is evaluated, and a focus state evaluation result is obtained based on the evaluation result of the pixel clarity.

6. The focal stacking imaging method of the liquid lens according to claim 5, characterized in that: The pixel clarity of each frame of the image in the image sequence is evaluated, and a focus state evaluation result is obtained based on the evaluation result of the pixel clarity, including: Convert each frame of the image in the image sequence into a grayscale image; Calculating the focus and clarity of the grayscale image within a window of a set size; Based on the focus and clarity of the grayscale image in a window of a set size, an evaluation result of the pixel clarity is obtained, and based on the evaluation result of the pixel clarity, a focus state evaluation result is obtained.

7. The focal stacking imaging method of a liquid lens according to claim 1, characterized in that: Based on the focus state evaluation result, the image sequence is fully focused and fused to obtain a fully focused image, including: Based on the focus state evaluation result, the clearest imaging pixels of each frame image in the image sequence are spliced ​​into a fully focused image.

8. The focal stacking imaging method of a liquid lens according to any one of claims 1 to 7, characterized in that: Generating a depth map based on the object distance and the all-focus image, including: Extracting two-dimensional features of each frame of image, forming a focal body based on the two-dimensional features, and performing differentiation processing on the focal body based on the object distance to obtain a differential focal body; Extracting features from the differential focus volume, and predicting the probability of a pixel point being in an optimal focus state based on the extracted features; Based on the predicted probability of the pixel being in the best focus state, probability regression is performed and combined with the focal length of each frame image to generate a depth map reflecting the image spatial depth information.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory, and is used to execute the program stored in the memory to implement the steps of the focal stacking imaging method of the liquid lens as described in any one of claims 1 to 8.

10. A focal stack imaging system of a liquid lens, characterized in that: include: A power supply module, a coaxial fixed-focus lens, a liquid lens and a CMOS array, and the electronic device according to claim 9, wherein the electronic device is electrically connected to the power supply module and is communicatively connected to the CMOS array.

Images