A full stokes three-dimensional polarization imaging device and method based on liquid lens scanning focus

Abstract

A kind of full stokes three-dimensional polarization imaging device and method based on liquid lens scanning focus, liquid lens changes optical power under the control of driving signal, and makes object side focal plane scan along depth direction;Polarization analyzer applies no less than four linear independent polarization analysis states to light beam;Image sensor outputs image data frame by frame.Synchronous controller unifiedly outputs driving signal to control focal plane scanning timing and polarization state switching timing, and generates and latches joint state identification containing focal plane number and polarization analysis state number at each frame exposure, ensure that the state information of each frame image is complete and traceable.Processing unit groups each frame image according to joint state identification, and reverses full stokes vector of corresponding focal plane according to each group of images, and then generates three-dimensional stokes data body and three-dimensional point cloud carrying complete polarization property.The present application realizes high-fidelity reconstruction of full stokes vector of different depth focal plane by synchronous control and joint state identification mechanism under unified time base.

Description

Technical Field

[0001] This invention relates to the fields of polarization imaging and computational imaging, and in particular to a full Stokes three-dimensional polarization imaging device and method based on liquid lens scanning. Background Technology

[0002] Current machine vision systems primarily acquire two-dimensional intensity or RGB information. Depth cameras add a geometric dimension, while conventional polarization cameras typically output two-dimensional polarization degree or orientation angle. Currently, there is a lack of a high-dimensional visual imaging method capable of simultaneously acquiring three-dimensional structure, full-focus sharpness, and complete S0, S1, S2, and S3 polarization information within a unified spatiotemporal coordinate system. For targets that are transparent, reflective, have weak texture, scatter, or possess similar material properties, intensity, color, or a single depth alone are often insufficient for stable differentiation. 3D reconstruction and robot vision, in particular, require the joint representation of geometric information and polarization attributes.

[0003] Stokes vector S = [S0, S1, S2, S3] T These represent the total light intensity, the two linearly polarized components, and the circularly polarized component, respectively. Existing time-division full Stokes cameras typically use two liquid crystal variable retarders and one fixed analyzer to form four or more analysis states, and then obtain the complete Stokes vector by inversion from the instrument matrix.

[0004] Liquid lenses adjust optical power by changing the curvature of the liquid-liquid interface or flexible film, eliminating the need to move solid lens groups along the optical axis. After acquiring images from multiple focal planes, full-focus images and depth maps can be obtained using sharpness evaluation, focal length-object distance calibration, and focus stacking, thus providing the foundation for developing compact 3D vision sensors.

[0005] However, to obtain the aforementioned high-dimensional visual data, liquid lens focusing and polarization time-division modulation cannot be simply cascaded. Focusing simultaneously alters the focal plane, magnification, distortion, vignetting, and point spread function, while a set of intensity maps used to invert Stokes vectors must correspond to the same focal plane. If the four polarization states fall at different focal lengths, or if frames are lost and then filled in using images from adjacent focal planes, both polarization and depth errors will occur simultaneously. Liquid lenses and LCVRs also suffer from response time, temperature drift, and hysteresis. Numbering them only according to the software reception sequence makes it impossible to confirm whether the two devices have stabilized at exposure time, and it is also impossible to reliably handle frame loss and buffer backlog. Therefore, focal length drive, polarization state, exposure trigger, and frame identification need to use the same time base and be calibrated separately for different focal lengths.

[0006] In summary, the existing technology mainly has the following problems: Existing variable-focus full Stokes systems typically control zoom and polarization measurement independently, lacking synchronous recording of focal plane and polarization states in image frames; When a liquid lens is unstable, it will lose focus or change magnification; when an LCVR is unstable, the actual analytical state deviates from the calibrated value. Ordinary focus stacks are designed for intensity or color images and cannot directly guarantee that the four Stokes components use the same set of focus weights; Many polarization 3D methods only use DoLP and AoLP, without providing the distribution of S0, S1, S2, and S3 with depth; When the focal length changes, the instrument matrix, spatial mapping, and point spread function will all change, so the same set of calibration parameters cannot always be used; When frames are dropped or the buffer is cleared, if only the receiving order is considered, it is easy to piece together images with different focal planes, different scanning directions, or different scanning cycles. During continuous focus scanning, the focal length continues to change within a single exposure. It is necessary to record the focal length range corresponding to that exposure, rather than simply assigning an ideal focal plane value.

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a full Stokes three-dimensional polarization imaging device and method based on liquid lens focusing.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A full Stokes three-dimensional polarization imaging device based on liquid lens scanning, comprising: A liquid lens is used to change the optical power under the control of a drive signal so that the object-side focal plane scans along a set trajectory in the depth direction; A polarization analyzer is disposed in the outgoing optical path of the liquid lens and is used to apply no less than four linearly independent polarization analysis states to the beam. An image sensor is used to receive the light signal emitted by the polarization analyzer and output image data frame by frame. A synchronization controller is connected to the liquid lens, the polarization analyzer and the image sensor respectively. It is used to output a unified drive signal to control the focal plane scanning timing and polarization state switching timing. It generates and latches a joint state identifier corresponding to each frame when each frame is exposed. The joint state identifier includes at least the focal plane number and the polarization analysis state number. The processing unit, connected to the image sensor and the synchronization controller, is used to receive the image data and the corresponding joint state identifier, group each frame of image according to the focal plane and polarization analysis state based on the joint state identifier, and invert the full Stokes vector of the corresponding focal plane based on each group of images.

[0010] A full Stokes three-dimensional polarization imaging method based on liquid lens scanning includes the following steps: S1. The synchronous controller controls the liquid lens to sequentially reach each preset focal plane and maintain stable optical power. Within the stable range of each focal plane, the polarization analyzer is controlled to sequentially switch to no less than four linearly independent polarization analysis states. After each polarization analysis state stabilizes, the image sensor is triggered to expose. At the same time, the joint state identifier corresponding to each frame image is latched. The joint state identifier includes at least the focal plane number, the polarization analysis state number, and the valid state flag. S2. The processing unit receives image frames and their joint status identifiers, performs status verification on each frame, discards frames with invalid valid status identifiers or conflicting status identifiers, and groups the frames that pass the verification according to the focal plane and the complete scan cycle. S3. For any focal plane, check whether the polarization analysis state images within the same complete scan cycle under that focal plane are complete and consistent. If they are complete, call the pre-stored calibration parameter set of that focal plane to perform geometric registration on each image, and invert the full Stokes vector of that focal plane from the registered intensity vector and the polarization instrument matrix of that focal plane. If any polarization analysis state is missing or inconsistent, the entire set of images will not participate in the reconstruction. S4. Combine the full Stokes vectors obtained from the inversion of each focal plane according to the depth direction to form a three-dimensional Stokes data volume of S0, S1, S2, and S3; calculate the sharpness evaluation function based on the S0 component of each focal plane and generate sharpness weights; apply the same set of weights to S0, S1, S2, and S3 respectively for weighted fusion to obtain a full Stokes image with full focus sharpness. S5. Based on the sharpness evaluation results of each focal plane and the pre-stored focal length-object distance mapping, estimate the coarse depth. Combine the polarization direction angle and polarization degree obtained by inversion to construct a joint optimization objective function, solve to obtain the refined depth map, and output a 3D point cloud containing spatial coordinates and complete Stokes polarization attributes, as well as a full-focus sharp full Stokes image.

[0011] The present invention has the following beneficial effects: This invention proposes a full Stokes 3D polarization imaging device and method based on liquid lens scanning. By jointly acquiring complete polarization information and 3D geometric information under unified spatiotemporal coordinates and the same measurement period, it can obtain full Stokes vectors at different depth focal planes and generate 3D Stokes data volumes, clear polarization maps at full focus, and 3D point clouds carrying complete polarization attributes. This effectively makes up for the shortcomings of existing machine vision systems in acquiring high-dimensional polarization and geometric information at the same time. It is especially suitable for 3D perception and recognition scenarios of transparent, reflective, weakly textured, scattering, and material properties of targets.

[0012] This invention employs direct focusing using a liquid lens, eliminating the need to move the solid lens assembly along the optical axis. This significantly shortens the focal plane switching time, facilitating the creation of a compact 3D vision sensor. More importantly, this invention centrally manages the liquid lens focal plane scanning timing, polarization analyzer state switching timing, and image sensor exposure triggering timing through a unified synchronous controller. During each frame exposure, a joint status identifier is generated and latched, containing the focal plane number, polarization state number, complete scan cycle number, focusing direction, and valid status flag. This ensures that each image frame carries complete timing and status information, fundamentally avoiding the error of reconstructing images with different focal planes, cycles, or focusing directions by grouping them together. Furthermore, this invention triggers exposure only after both the liquid lens and polarization analyzer have reached a preset effective stability window, ensuring that each exposure avoids the switching transients of the two devices, effectively eliminating defocusing deviations and polarization analysis state shifts introduced by device instability.

[0013] Furthermore, this invention addresses the system parameter changes caused by focusing of liquid lenses by employing a separate focal length calibration strategy. For each focal plane, corresponding geometric mapping parameters, point spread function, and polarization instrument matrix are stored separately. This effectively corrects for magnification, distortion, vignetting, and polarization crosstalk variations during focusing, ensuring the accuracy of Stokes inversion at different focal planes. In the image fusion stage, this invention uniformly calculates sharpness weights based on the S0 components of each focal plane and applies the same set of weights to the weighted fusion of all four Stokes components. This avoids the problem of losing the same physical measurement relationship after fusion due to independent focusing of each component. Regarding depth estimation, this invention jointly optimizes the coarse depth obtained from the focus stack with normal constraints based on polarization direction angle and polarization degree. This ensures that each point in the output 3D point cloud simultaneously carries spatial coordinates and complete polarization attributes such as S0, S1, S2, S3, polarization degree, polarization direction angle, and ellipticity angle, providing richer high-dimensional feature information for subsequent 3D vision tasks.

[0014] Furthermore, this invention employs a robust complete group verification and frame loss tolerance mechanism. For any focal plane, reconstruction is only performed when all polarization analysis state images within the same complete scanning cycle are complete and consistent. If any state is missing, the entire group is discarded, and cross-focal plane, cross-direction, or cross-cycle stitching is prohibited, ensuring the physical consistency of the reconstructed data. Simultaneously, when buffer backlog occurs, old data can be discarded while frame numbers and modulation phases continue to accumulate, effectively preventing the real-time preview latency from continuously increasing. This invention also allows for flexible configuration of acquisition parameters and working modes according to specific application requirements. For example, polarization modulation can be turned off to increase the frame rate when only depth or intensity information is needed, and focusing can be stopped when working on a fixed focal plane to increase the acquisition refresh rate of the entire Stokes image under that fixed focal plane. Therefore, it can adapt to various application scenarios, from microscopic 3D imaging, transparent and semi-transparent material detection, surface defect and stress detection, biological tissue layer imaging to industrial online inspection. Furthermore, combined with artificial intelligence algorithms, it can be further extended to cutting-edge fields such as high-dimensional visual feature learning, object recognition and segmentation, material perception, robot grasping and navigation, digital twins, and intelligent manufacturing.

[0015] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the optical path of a liquid lens focal plane scanning full Stokes three-dimensional polarization camera according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the combined timing of the focal plane state and polarization analysis state in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the joint state coding, complete group verification, and frame loss fault tolerance process in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the full focal length full Stokes data cube according to an embodiment of the present invention.

[0020] Figure 5 This is a flowchart of the full focal length and full Stokes 3D joint reconstruction process according to an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of the complete scanning cycle of multiple focal planes and the four-state combination of H / V / D / R of the same focal plane in an embodiment of the present invention.

[0022] Reference numerals: 1—Target under test; 2—Liquid lens; 3—Compensation imaging lens group; 4—First liquid crystal variable retarder; 5—Second liquid crystal variable retarder; 6—Fixed polarization analyzer; 7—Image sensor; 8—Unified synchronization controller. Detailed Implementation

[0023] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0024] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for fixing, coupling, or communication.

[0025] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] This invention aims to solve the technical problem that liquid lens focal plane scanning and polarization time-division modulation cannot jointly acquire complete polarization and three-dimensional geometric information under a unified spatiotemporal coordinate due to the separation of time sequence. It proposes a full Stokes three-dimensional polarization imaging scheme with a unified synchronous controller as the core and a joint state identifier as the link. By sharing the same time base for focal plane scanning, polarization switching and exposure triggering, exposure is only performed after the liquid lens and polarization analyzer have stabilized. Combined with independent calibration of focal length and complete group verification mechanism, high-fidelity reconstruction of full Stokes vectors of focal planes at different depths and output of three-dimensional point clouds with complete polarization attributes are achieved.

[0028] This invention provides a full Stokes three-dimensional polarization imaging device based on liquid lens scanning, such as... Figure 1As shown, the system includes: a liquid lens 2 in front of the target 1, used to change the optical power under the control of a driving signal so that the object-side focal plane scans along a set trajectory in the depth direction; a compensation imaging lens group 3, disposed in the output optical path of the liquid lens 2, used to perform imaging compensation on the beam; a polarization analyzer, disposed in the output optical path of the compensation imaging lens group 3, used to apply no less than four linearly independent polarization analysis states to the beam, specifically including a first liquid crystal variable retarder 4, a second liquid crystal variable retarder 5, and a fixed polarization analyzer 6; an image sensor 7, used to receive the light signal emitted by the polarization analyzer and output image data frame by frame; and a unified synchronization control. Device 8, connected to the liquid lens 2, the polarization analyzer, and the image sensor 7 respectively, is used to uniformly output driving signals to control the focal plane scanning timing and polarization state switching timing, and to generate and latch a joint state identifier corresponding to each frame when each frame is exposed. The joint state identifier includes at least a focal plane number and a polarization analysis state number. Processing unit, connected to the image sensor 7 and the unified synchronization controller 8, is used to receive the image data and the corresponding joint state identifier, group each frame of image according to the focal plane and polarization analysis state based on the joint state identifier, and invert the full Stokes vector of the corresponding focal plane based on each group of images.

[0029] In some embodiments, the synchronization controller is configured to: control the liquid lens 2 to maintain stable optical power at each focal plane, and control the polarization analyzer to sequentially switch to each polarization analysis state within the stable range, triggering the image sensor 7 to expose for each polarization analysis state. Figure 2 As shown, in a direct acquisition sequence of a focal plane, first switch to the current focal plane and wait for the liquid lens 2 to stabilize (FOCUS). VALID =1), then the polarization states Q1, Q2, Q3, and Q4 are stabilized and exposed to E1 to E4 in sequence, with all four polarization states completed on the same focal plane; the judgment rule is that only FOCUS is valid. VALID =1 and the current polarization state POL VALID Exposure is triggered only when the value is 1; the entire sequence is invalidated if any state is missing. Specifically, the synchronization controller latches the current focal plane number, polarization analysis state number, complete scan cycle number, focus direction, and valid state flag along the exposure trigger edge, and binds the latched information to the corresponding image frame. The valid state flag at least indicates the focal power stability state of the liquid lens 2 at the start of exposure, the polarization state stability state of the polarization analyzer, and whether the exposure process is complete.

[0030] In some embodiments, the synchronization controller is further configured to: determine whether the current focal length of the liquid lens 2 has entered a preset focal length tolerance range and the holding time exceeds a first threshold, and generate a valid focal length signal; determine whether the current analysis state of the polarization analyzer has entered a preset polarization state tolerance range and the holding time exceeds a second threshold, and generate a valid polarization signal; determine whether the image sensor 7 is in an allowable exposure state, and generate a camera ready signal; generate an exposure trigger signal only when the valid focal length signal, the valid polarization signal, and the camera ready signal are all valid, and latch the joint state flag corresponding to that moment; wherein, during the exposure, each valid signal is continuously sampled, and at the end of the exposure, it is confirmed that each signal remains valid within the exposure window, and the confirmation result is recorded in the valid state flag.

[0031] In some embodiments, the processing unit is configured with a focal length calibration parameter set, which is stored separately for each focal plane and includes at least: geometric mapping parameters under the focal plane, used to register images of each polarization analysis state to a unified spatial coordinate; polarization instrument matrix under the focal plane, used to invert the full Stokes vector of the image intensity corresponding to multiple polarization analysis states acquired by the focal plane; point spread function parameters under the focal plane, used for focus evaluation and image restoration; focal length-object distance mapping relationship under the focal plane, used to convert depth coordinates from the focal plane number; when reconstructing the full Stokes vector of any focal plane, the processing unit calls the calibration parameter set corresponding to the focal plane for inversion.

[0032] In some embodiments, such as Figure 6 As shown, the synchronization controller is configured to control the liquid lens 2 to sequentially reach each preset focal plane and remain stable within each complete scan cycle. At each focal plane, it waits for the focal length signal to be valid and confirms that the focal length is stable before sequentially acquiring images of the four polarization analysis states (H, V, D, R) within the stable range of each focal plane. Only after all four states are complete does it switch to the next focal plane. All polarization analysis state images of the same focal plane have the same complete scan cycle number, focal plane number, and scanning direction. For example... Figure 3 As shown, the processing unit is configured to perform a complete group verification for each focal plane. This verification includes: checking whether the focal plane contains all the image frames required for polarization analysis; checking whether the complete scan cycle number, focal plane number, and scanning direction of each frame are consistent; checking whether the valid status flags of each frame are all valid; and only when all the above verifications pass, the image group of that focal plane is used for full Stokes vector inversion. Figure 3 The mid-focal plane J2 is in a complete state and has the same period, allowing the reconstruction of S0, S1, S2, and S3; however, the focal plane J1 is incomplete due to the missing Q3, so the reconstruction of this group is rejected, and the use of image frames from adjacent focal planes or other periods for supplementation is prohibited.

[0033] In some embodiments, the polarization analyzer includes two liquid crystal variable retarders with different fast axis directions (i.e., the first liquid crystal variable retarder 4 and the second liquid crystal variable retarder 5) and a fixed linear polarizer (i.e., the fixed polarization analyzer 6). The synchronization controller switches the polarization analysis state by changing the voltage applied to the two liquid crystal variable retarders. Alternatively, the polarization analyzer employs a group of polarization modulation devices capable of generating a full-rank analysis matrix, whose analysis states have pairwise linearly independent normalized analysis vectors on the Poincaré sphere.

[0034] In some embodiments, such as Figure 4 As shown, the processing unit is further configured to: after inverting the full Stokes vector for each focal plane, combine them in the depth direction to form four three-dimensional Stokes data volumes: S0, S1, S2, and S3; each focal plane is inverted using a complete polarization state group, and multiple focal planes are combined in the depth direction to form four three-dimensional Stokes data cubes; for the S0 component of each focal plane, calculate the sharpness evaluation function and generate the sharpness weight of each pixel in each focal plane; apply the same set of sharpness weights to the four components S0, S1, S2, and S3 respectively for weighted fusion to obtain a full-focus sharp Stokes image; wherein, the sharpness weights are non-negative and normalized between each focal plane, and the weights corresponding to low signal-to-noise ratio or low confidence regions are reduced or set to zero.

[0035] In some embodiments, such as Figure 5 As shown, the processing unit is further configured to: estimate the coarse depth of each pixel in the scene using the sharpness evaluation results of the S0 components of each focal plane and the focal length-object distance mapping relationship; calculate the polarization direction angle using the inverted S1 and S2 components, and calculate the degree of polarization and ellipticity using S0, S1, S2, and S3; construct a joint optimization objective function including focus sharpness constraints, polarization normal constraints, and spatial smoothness constraints using the coarse depth as the initial value, and iteratively solve to obtain a refined depth map; associate the refined depth map with the full-focus sharp Stokes image and polarization attribute map, and output a three-dimensional point cloud of each three-dimensional point carrying spatial coordinates and S0, S1, S2, S3, degree of polarization, polarization direction angle, and ellipticity attributes.

[0036] In some embodiments, the synchronization controller is further configured to select an operating mode according to task requirements and stop modulation unrelated to the target output, including: stopping the state switching of the polarization analyzer and keeping it in a fixed transmittance state when only intensity or depth information is needed, and reducing the number of polarization analysis states to 1; stopping the focal plane scanning of the liquid lens 2 and fixing it to the target focal plane when only full Stokes information of the fixed focal plane is needed; stopping the generation of circular polarization analysis states when only linear polarization information is needed and not circular polarization information; and writing the current operating mode, the effective focal plane set, the effective polarization analysis state set, and the calibration parameter version into the frame metadata after the mode switch.

[0037] This invention also provides a full Stokes three-dimensional polarization imaging method based on liquid lens scanning, comprising the following steps: Step S1: The unified synchronization controller 8 controls the liquid lens 2 to sequentially reach each preset focal plane and maintain stable optical power. Within the stable range of each focal plane, the polarization analyzer (i.e., the first liquid crystal variable retarder 4, the second liquid crystal variable retarder 5, and the fixed polarization analyzer 6) is controlled to sequentially switch to no less than four linearly independent polarization analysis states. After each polarization analysis state stabilizes, the image sensor 7 is triggered to expose, and the joint state identifier corresponding to each frame image is latched. The joint state identifier includes at least the focal plane number, the polarization analysis state number, and the valid state flag.

[0038] In some embodiments, the generation of the valid state flag in step S1 includes: the synchronization controller continuously detects the focal length error and focal length change rate of the liquid lens 2 in real time; when the focal length error is lower than a preset focal length threshold and the focal length change rate is lower than a preset change rate threshold and the holding time is maintained for a first time, the liquid lens 2 is determined to be stable and the focal length valid signal is set to be valid; the synchronization controller continuously detects the deviation between the actual analysis state and the target analysis state of the polarization analyzer in real time; when the deviation is lower than a preset polarization state threshold and the holding time is maintained for a second time, the polarization analyzer is determined to be stable and the polarization valid signal is set to be valid; the synchronization controller detects the exposure allowable state of the image sensor 7 and sets the camera ready signal; the synchronization controller generates an exposure trigger signal only when the focal length valid signal, the polarization valid signal and the camera ready signal are all valid, and latches the current focal plane number, the polarization analysis state number and the complete scan cycle number at the exposure trigger edge; during the exposure, each valid signal is continuously sampled, and at the end of the exposure, it is confirmed that each signal remains valid throughout the entire exposure window, the confirmation result is recorded in the valid state flag, and the latched identification information is bound to the corresponding image frame.

[0039] Step S2: The processing unit receives image frames and their joint status identifiers, performs status verification on each frame, and determines whether each frame belongs to the same complete scan cycle, the same focal plane, and the same scanning direction based on the complete scan cycle number, focal plane number, polarization analysis status number, and valid status flag in the joint status identifier, and whether the valid status flag is valid. Frames with invalid valid status flags or conflicting status identifiers are discarded, and the frames that pass the verification are grouped according to focal plane and complete scan cycle.

[0040] Step S3: For any focal plane, check whether the polarization analysis state images within the same complete scan cycle under that focal plane are complete and consistent. If complete, call the pre-stored calibration parameter set of that focal plane to perform geometric registration on each image, and invert the full Stokes vector of that focal plane from the registered intensity vector and the polarization instrument matrix of that focal plane. If any polarization analysis state is missing or inconsistent, the entire set of images will not participate in the reconstruction.

[0041] In some embodiments, the calibration of the polarization instrument matrix in step S3 includes: generating multiple known polarization states in the object space using a polarization state generator under each preset focal plane, wherein the number of known polarization states is not less than the number of polarization analysis states and is uniformly distributed on the Poincaré sphere; acquiring the output intensity of the image sensor 7 for each known polarization state and each polarization analysis state to form a measurement matrix; solving the polarization instrument matrix of the focal plane in a pixel-by-pixel or partition-by-partition manner based on the Stokes vector matrix of the known polarization states and the measurement matrix; verifying using independent polarization states not involved in the solution, and when the error of the reconstructed Stokes vector exceeds a preset threshold, the instrument matrix of the corresponding pixel or region is marked as unqualified and interpolated or removed using adjacent valid values; the polarization instrument matrix of each focal plane, together with the temperature and scanning direction during measurement, is saved to constitute the calibration parameter set of the focal plane.

[0042] Step S4: Combine the full Stokes vectors obtained from the inversion of each focal plane according to the depth direction to form a three-dimensional Stokes data volume of S0, S1, S2, and S3; calculate the sharpness evaluation function based on the S0 component of each focal plane and generate sharpness weights; apply the same set of weights to S0, S1, S2, and S3 respectively for weighted fusion to obtain a full Stokes image with full focus sharpness.

[0043] In some embodiments, the generation of the full-focus sharp Stokes image in step S4 includes: calculating the sharpness evaluation value pixel by pixel for the S0 component of each focal plane to form the sharpness distribution of each pixel in each focal plane; spatially smoothing and normalizing the sharpness distribution so that the sum of the sharpness weights of each pixel in each focal plane is 1; and weighting and summing the normalized sharpness weights with the S0, S1, S2, and S3 components of each focal plane to obtain the full-focus sharp Stokes image; wherein, for low signal strength, saturation, or low confidence regions, the sharpness weight of the corresponding pixel is reduced or set to zero, and all four Stokes components at the same pixel position use the same weight value.

[0044] Step S5: Based on the sharpness evaluation results of each focal plane and the pre-stored focal length-object distance mapping, estimate the coarse depth, combine the polarization direction angle and polarization degree obtained by inversion to construct a joint optimization objective function, solve to obtain the refined depth map, and output a 3D point cloud containing spatial coordinates and complete Stokes polarization attributes, as well as a full-focus sharp full Stokes image.

[0045] In some embodiments, the construction of the joint optimization objective function in step S5 includes: constructing a focus sharpness constraint term based on the sharpness evaluation value of each pixel at each focal plane and the focal length-object distance mapping relationship; constructing a polarization normal constraint term based on the constraint of the polarization direction angle on the surface normal azimuth angle obtained by inversion, and the constraint of polarization degree and material refractive index on the surface normal tilt angle; constructing a spatial smoothing constraint term for the depth map; weighting and summing the above three terms to form a joint optimization objective function, and solving the depth distribution that minimizes the objective function through iterative optimization, which is used as the final refined depth map.

[0046] In some embodiments, the method further includes a working mode switching step: selecting the information output dimension according to task requirements, controlling the synchronization controller to stop modulation unrelated to the target output; when only intensity or depth information is needed, stopping the state switching of the polarization analyzer and maintaining it in a fixed transmittance state, reducing the number of effective polarization analysis states to 1, and increasing the effective image frame rate and depth map refresh rate; when only full Stokes information of a fixed focal plane is needed, stopping the focal plane scanning of the liquid lens 2 and fixing it at the target focal plane, reducing the number of effective focal planes to 1, and increasing the acquisition refresh rate of the full Stokes image under this fixed focal plane; when only linear polarization information is needed, stopping the generation of circular polarization analysis states, reducing the number of effective polarization analysis states to 3, and not outputting the S3 component; writing the switched working mode, effective focal plane set, effective polarization analysis state set, and calibration parameter version into the frame data, and the processing end adjusts the reconstruction process according to the working mode.

[0047] This invention relates to a full Stokes 3D polarization imaging device and method based on liquid lens scanning. By using a unified synchronous controller, the liquid lens focal plane scanning, polarization state switching, and camera exposure triggering share the same time base. Furthermore, each frame image embeds a joint state identifier, including focal plane number, polarization state, scanning period, and valid flag, enabling the system to sequentially acquire all polarization analysis states within the stable window of each focal plane. Based on this, an independent focal length calibration strategy is employed to correct magnification, distortion, and polarization response changes caused by focusing. Four Stokes components are fused with the same sharpness weight, and the refinement depth is obtained through joint optimization of focus stack and polarization normal. Finally, a high-dimensional data volume is output, with each 3D point carrying complete polarization attributes. This overcomes fundamental problems in traditional schemes, such as the separation of zoom and polarization modulation timing, untraceable states, and errors introduced by cross-focal plane stitching.

[0048] The following further describes the implementation methods and advantages of specific embodiments of the present invention.

[0049] System composition and optical path

[0050] The full Stokes 3D polarization imaging device based on liquid lens focusing includes a liquid lens 2, a compensation imaging lens group 3, a polarization analyzer (including a first liquid crystal variable retarder 4, a second liquid crystal variable retarder 5, and a fixed polarization analyzer 6), an image sensor 7, a synchronization controller 8, and a processing unit. It also includes status detection and calibration data storage to determine device stability and retrieve calibration parameters for the corresponding focal length. Figure 1 The optical path of a full-Stokes 3D polarization camera with a liquid lens focal plane scanning capability is demonstrated. The liquid lens 2 employs an electrowetting or film-type liquid lens with minimal polarization influence, its focal length altered by voltage or current. The main polarization analyzer uses two LCVRs (i.e., a first liquid crystal variable retarder 4 and a second liquid crystal variable retarder 5) with different fast axis directions and a fixed linear polarizer 6; alternatively, other modulators capable of generating a full-rank analysis matrix can be used. The image sensor 7 can be a CCD, CMOS, infrared array, or linear array sensor; the camera should ideally provide external triggering, hardware frame number, and timestamp. The synchronization controller 8 uniformly outputs the liquid lens drive, polarization state drive, and camera trigger, simultaneously generating the period number, focal plane number, polarization state number, and validity flag. The processing unit verifies the frame state and completes focal length registration, focal plane-by-focal plane Stokes inversion, focus stacking, depth estimation, and 3D output.

[0051] Calibration and Compensation of Polarization Effect in Liquid Lenses

[0052] The polarization effect of liquid lens 2 is determined using the focal length-dependent Mueller matrix M. L,JThe (λ,T,D) characterization includes polarization-dependent transmittance, equivalent phase retardation, depolarization, and interface reflection. During calibration, at least six known Stokes states are input on the object side, and the output is measured using a reference polarimeter after the liquid lens 2. M is solved for each focal plane, temperature, and up / down scanning direction. L,J The response is separated using a reference optical path without a liquid lens. During reconstruction, M can be... L,J Merged into A J It can be used for end-to-end calibration and also for robust pseudo-inverse compensation; interpolation is only performed on adjacent focal planes when verifying that the residual meets the standard.

[0053] Determining the steady state of a polarization analyzer (LCVR)

[0054] A beam splitting reference branch is used to determine the stable state of the LCVR (i.e., the first liquid crystal variable retarder 4 and the second liquid crystal variable retarder 5). The reference light passes sequentially through the fixed polarizer, LCVR, and analyzer, and the real-time phase delay δ is calculated by dual photodetectors; a temperature sensor is used to correct the voltage-delay lookup table. The preferred criterion is: |δ - δ target |≤1° (approximately 1.5nm at 550 nm), adjacent sampling variation ≤0.2°, temperature variation during exposure ≤0.1 ℃, and satisfied for 3 consecutive camera cycles. The final threshold is determined by independent validation states to ensure that the normalized Stokes reconstruction error does not exceed 1%.

[0055] Mechanism for generating and latching valid status flags

[0056] The synchronization controller uses a unified clock from either an FPGA or an MCU. FOCUS VALID POL VALID and CAMERA READY After synchronization and image stabilization by dual triggers, the system enters the state machine; an exposure trigger is issued only when all three parameters are simultaneously 1. During the exposure period, valid signals are continuously sampled based on the latch period number C, focal plane number J, polarization state Q, orientation D, and hardware frame count. At the end of the exposure, V is set only when all signals have been valid throughout the entire exposure window and the camera returns a confirmation of completion. n =1. The latch result and trigger number are written together into the FIFO and bound to the corresponding image frame.

[0057] Instrument Matrix A at Different Focal Lengths J Calibration scheme and verification method

[0058] A polarization state generator, consisting of a stable light source, a polarizer, and an achromatic quarter-wave plate, is used to generate at least 6, preferably 16, known Stokes states uniformly distributed on the Poincaré sphere in the object plane, and these states are calibrated using a reference polarimeter. Dark field, flat field, and calibration intensity are acquired for each focal plane J and each analytical state Q, forming a measurement matrix Y. J Press AJ = Y J ·S cal + Solve the instrument matrix pixel by pixel or partition by partition. Then verify using independent polarization states that were not involved in the fitting, preferably with normalized intensity residual ≤1%, DoP absolute error ≤0.02, and AoLP error ≤1°; mark or interpolate unqualified pixels, and save the calibration results for wavelength, temperature, and focusing direction respectively.

[0059] Focal length-polarization joint state coding

[0060] Let C be the period number of a complete focal plane scan, J be the focal plane number, Q be the polarization analysis state, D be the focusing direction, and V be the effective state. The state record of the nth frame is as follows:

[0061] T is a hardware timestamp. V records at least whether the focal length is stable, whether the polarization state is stable, and whether the exposure is valid. This information shares the same hardware counter with the device driver and exposure trigger, and is latched at the start of the exposure. The status can be written to camera metadata, frame headers, synchronization messages, or sensor reserved areas.

[0062] The processor cannot infer the polarization state solely from the received order mod(n,4). If a hardware frame number jump, state conflict, timestamp anomaly, or valid flag is zero, the frame will not participate in the reconstruction.

[0063] Stepped focal plane scanning and nested polarization period

[0064] The main scheme adopts a timing sequence of "first determining the focal plane, then scanning the polarization states". The liquid lens sequentially reaches f1, f2...f M Within each stable focal length range, the focal length remains constant, and exposures in states Q1, Q2, Q3, Q4, or more are completed sequentially before switching focal planes.

[0065] Figure 2 The joint timing of the focal plane state and the polarization analysis state is shown.

[0066] If each state occupies K camera cycles, the first few frames after the switch can be reserved as a settling time, and then one or more frames can be averaged from the settling interval. Polarization frames on the same focal plane must have the same C, J, and D.

[0067] Exposure after the liquid lens and polarization analyzer have stabilized.

[0068] The camera's effective exposure window satisfies:

[0069] The stability of a liquid lens can be determined by drive feedback, capacitance, current, pressure, position sensor, reference optical path, or rate of change of image sharpness; the stability of an LCVR can be determined by response model, temperature compensation, reference polarization detection, or delay feedback. Exposure is only permitted after the focal length error, rate of change of focal length, and analysis vector error are all below the threshold and maintained for a specified time.

[0070]

[0071] A pre-measured fixed waiting time can also be used, but actual state confirmation is more reliable. Regardless of the method used, the valid signal, focal plane number, and polarization state number should all be latched at exposure time.

[0072] Complete focal plane polarization group and frame loss tolerance

[0073] During a complete scan cycle C, the candidate image group for focal plane J is written as G. C,J ={I C,J,Q | Q∈Q set Only when all states are complete, and the period number, focal plane number, and scanning direction are consistent, and the valid flag is 1, is the Stokes vector for that focal plane calculated.

[0074] Figure 3 It demonstrates joint state coding, complete group check, and frame loss tolerance.

[0075] If a certain state is missing, the current focal plane data is discarded and cannot be filled in using the next focal plane, reverse scan, or frame from the next cycle. The modulation timing continues to run, and the processor selects the nearest complete focal plane group or complete data volume from the buffer to prevent the display latency from continuously increasing.

[0076] Full Stokes reconstruction of the focal plane

[0077] The vector I is composed of N effective intensity maps of focal plane J. J The corresponding instrument matrix is ​​denoted as A. J :

[0078] N=4 and A J When reversible, the inverse can be calculated directly; when N>4, a weighted pseudo-inverse, constrained least squares, or robust estimation can be used. The weights are set based on photon noise, readout noise, saturation, focal length confidence, and polarization state error.

[0079] Where H represents the horizontal linear polarization analysis state, V represents the vertical linear polarization analysis state, D represents the +45° linear polarization analysis state, and R represents the right-hand circular polarization analysis state. This specification stipulates that the right-hand circular polarization analysis state corresponds to +S3; when the actual system uses the opposite polarization definition, the sign of the reconstructed S3 is inverted. The HVDR instrument matrix of the ideal analyzer is:

[0080] The above equation represents the direct inversion relationship of the ideal HVDR four states. The actual setup uses the instrument matrix A obtained from calibration at each focal plane. J Perform inverse calculations, weighted pseudo-inverse calculations, or constrained least squares.

[0081] The supplementary scheme can employ the four analytical states of a Poincaré sphere inscribed in a regular tetrahedron. Four normalized three-dimensional analytical vectors u i Equilateral distribution in pairs improves the balance of the instrument matrix; the specific LCVR drive quantity is determined by calibration and is not limited to a set of fixed voltages. Its reconstruction still uses a unified instrument matrix format: .

[0082] Focal distance calibration

[0083] Each focal length state has a separate set of calibration parameters C. J ={f J Z J H J PSF J A J ,T J D J}. Among them, Z J It's a focal length-object distance relationship, H J It's a registration mapping, PSF J It is the point spread function, A J It is an instrument matrix, T J and D J Record the temperature and the direction of coking separately.

[0084] First use H during reconstruction J The images at each focal length are registered to the same coordinate system, and then the complete state set of that focal plane is inverted. Focal lengths that are not directly calibrated can only be obtained by interpolation from adjacent calibration values ​​within a specified range.

[0085] 3D Stokes data volume

[0086] After registration and Stokes inversion of each focal plane, four three-dimensional data volumes are obtained: S0(x,y,z), S1(x,y,z), S2(x,y,z), and S3(x,y,z).

[0087] Figure 4 It showcases the full focal length and full Stokes data cube.

[0088] The degree of polarization DoP, the degree of linear polarization DoLP, the polarization direction angle AoLP, the ellipticity χ, and the signed ellipticity e can also be calculated from the four Stokes components. The sign of S3 is consistent with the left-handed / right-handed convention used by the system; these derived quantities are not calculated when S0 is below the noise threshold.

[0089] .

[0090] Full-focus, clear Stokes image

[0091] During full-focus fusion, the sharpness weight w of each focal plane is first determined by S0 or depolarization intensity. J (x,y), and then apply the same set of weights to S0, S1, S2, and S3. If the four components are focused separately, the fused vector may no longer correspond to the same physical measurement.

[0092]

[0093] Weights can be obtained from gradient, Laplacian, wavelet, local contrast, calibrated PSF, or learned focus evaluation. Weights are taken as non-negative values ​​and normalized; low-signal, saturated, and low-confidence regions are not included in the fusion or have their weights reduced.

[0094] Depth estimation and polarization constraint

[0095] First, find the sharpest focal plane in the S0 focal stack and convert it into coarse depth using focal length-object distance calibration. Then, use AoLP to constrain the surface normal azimuth and use DoLP and material refractive index to estimate the normal tilt angle; S3 can be used to mark areas with strong circular polarization response or multiple scattering.

[0096] Figure 5 The process of full-focal-length, full-Stokes 3D joint reconstruction was demonstrated.

[0097] Each point in the output point cloud can simultaneously save its spatial coordinates, S0 to S3, DoP, AoLP, and elliptic deflection. The same set of data can also be output as a triangular mesh, volume data, depth slices, or full-focus Stokes plot.

[0098] Multi-focal plane complete scan cycle and four-state matching of the same focal plane

[0099] During a complete scan cycle, the liquid lens arrives at each discrete focal plane sequentially. For any focal plane J, the controller must wait for FOCUS. VALID =1 and keep the focal length unchanged, then complete the exposure of the four analysis states of H, V, D and R in sequence; only after the four state images of the focal plane are complete can the switch to the next focal plane be allowed.

[0100] .

[0101] The same acquisition rules are used for other focal planes.

[0102] Figure 6 This diagram illustrates a complete scanning cycle across multiple focal planes and the H / V / D / R four-state grouping of the same focal plane. It shows how each focal plane remains stable and completes H, V, D, and R state exposures before switching to the next focal plane.

[0103] If any focal plane is missing one of the states H, V, D, or R, or if the four frames do not belong to the same complete scan cycle, the same focal plane, or the same scanning direction, the entire focal plane group is invalidated, and it is prohibited to use adjacent focal plane images to complete it.

[0104] .

[0105] Frame rate, state rate and data volume refresh rate

[0106] Let the camera frame rate be F c The number of polarization states is N p The number of focal planes is N f After each focal plane switch, there is an additional wait time of K. f Each camera cycle, each polarization state occupies K... p One camera cycle. The refresh rate of a 3D Stokes data volume is calculated by the following formula.

[0107]

[0108] A set of acquisition parameters can be set to F c =200 fps (5 ms per frame), N p =4、N f =20、K f =2、K p =4. Each focal plane takes 90 ms, and the complete data volume takes 1.80 s, with a theoretical refresh rate of approximately 0.56 volumes / second. K f and K p Ultimately, the measured stable signal from the liquid lens and LCVR is determined; if a fast LCVR with a settling time of approximately 10 ms is used, K can be made... p =2, at which point the theoretical refresh rate is 1 unit / second.

[0109] Modulation shutdown on demand and effective output rate boost

[0110] The controller can select the operating mode based on the information dimensions required by the task and stop modulation unrelated to the target output. After stopping irrelevant modulation, the number of image frames required to generate a single effective output decreases, and the refresh rate of the effective image, polarization result, or 3D data volume increases accordingly. The camera hardware frame rate F is only set when the camera supports region-of-interest readout, shortened exposure, or high-speed readout modes.c Only then will it improve.

[0111] When only intensity maps, focus stacks, or depth information are needed, LCVR polarization state switching is stopped and maintained in a fixed high transmittance state. At this time, N p,act =1.

[0112] When only two-dimensional full Stokes information of the fixed focal plane is needed, stop the liquid lens focusing and fix it at the target focal length. At this time, N f,act =1 and no longer generate focal plane switching wait.

[0113] When only the linear polarization degree DoLP and orientation angle AoLP are needed, only the three analysis states H, V, and D can be acquired, and R-state modulation can be stopped. At this time, N p,act =3; This mode does not output S3.

[0114] When focusing on a specified depth range or a few focal planes, only the selected set of focal planes is scanned, with N... f,act Instead of the total focal plane number N f .

[0115]

[0116] After mode switching, the controller should write the working mode, effective focal plane set, effective polarization state set, and calibration version into the frame metadata; the processing end must not mark data lacking circular polarization state or focal plane dimension as complete full Stokes 3D data.

[0117] Example 1: Full Stokes 3D Imaging with Stepped Focal Surface Scanning

[0118] The driving force, actual focal power, and object depth of the liquid lens are calibrated, and M focal plane states f1 to f are selected. M And obtain the upscan and downscan calibration data respectively.

[0119] Input multiple known polarization states at each focal plane to calibrate the pixel-by-pixel or partitioned instrument matrix A. J Simultaneously calibrate the geometric mapping H J Point spread function (PSF) J And dark field / gain parameters.

[0120] The controller switches the liquid lens to f1; FOCUS received. VALID Then, within a range where the focal length remains constant, Q1 to Q4 are generated sequentially, and at each POL... VALID The window triggers an exposure.

[0121] Each frame latches (C, J, Q, D, V, T); after the four states of f1 are complete, it switches to f2, until f... M Completed, forming a full scanning cycle.

[0122] The processing module performs full group verification, geometric registration, and Stokes inversion on each focal plane to generate the S0 to S3 focal stack.

[0123] The sharpness weight and coarse depth are calculated uniformly using S0, and combined with polarization normal constraints to generate 3D point clouds and full-focus sharp Stokes images.

[0124] Example 2: Parameter Acquisition at 200 fps

[0125] The camera frame rate is set to 200 fps, with a single frame period of 5 ms. Two frames are reserved for focus stabilization after the liquid lens switches to the new focal plane; each polarization state occupies four frames, with the first three frames used for waiting or status confirmation, and the last frame in FOCUS. VALID With POL VALID Simultaneously, the exposure time is 1 hour. With 4 polarization states and 20 focal planes, a single focal plane occupies 18 frames, and the complete 3D Stokes data volume occupies 360 frames, with a theoretical refresh rate of approximately 0.56 volumes / second.

[0126] If the LCVR or liquid lens does not stabilize within the above window, K should be increased. p or K f Instead of maintaining a fixed frame rate and forcing exposure, 200 fps provides a timing resolution of 5 ms, but the actual measurement speed is still determined by the device settling time.

[0127] Example 3: Complete Cycle Acquisition of Multiple Focal Surfaces

[0128] Liquid lenses are arranged according to discrete focal planes f1 to f M Focusing is performed sequentially. Upon reaching each focal plane, the controller waits for FOCUS. VALID =1 and keep the focal length constant, and sequentially acquire H, V, D and R four-state images within this stable range; after all four states are complete, switch to the next focal plane.

[0129] The processing unit forms a four-state image group based on the complete scan cycle number and focal plane number, and calls the instrument matrix of that focal plane to reconstruct S0, S1, S2, and S3; if any state is missing, the entire focal plane group is invalidated. The Stokes vectors of all valid focal planes are combined into a three-dimensional data volume in depth order.

[0130] Example 4: Tetrahedral polarization state

[0131] The polarization analysis module uses four analytical states T1 to T4 corresponding to the four vertices of a Poincaré sphere tetrahedron. The four sets of driving variables from the two LCVRs are calibrated and optimized to make the actual analytical vectors approximate the tetrahedron vertices. Data is acquired at T1, T2, T3, and T4 within each focal plane, and inverted using the focal length-dependent instrument matrix.

[0132] Example 5: Microscopic Three-Dimensional Polarization Detection

[0133] The liquid lens is positioned at the rear end of the microscope objective or near the pupil plane in the relay optical path to minimize magnification changes during focusing. The system acquires full Stokes images layer by layer at different depths of the sample and outputs three-dimensional birefringence, polarization degree, orientation angle, ellipticity angle, or scattering characteristic distribution, which can be used for the detection of composite materials, thin films, biological tissues, and transparent samples.

[0134] Alternative implementation methods

[0135] Liquid lenses can be electrowetting, diaphragm, hydraulic, piezoelectric, or electromagnetically driven liquid lenses; if liquid crystal lenses are used, their polarization-dependent Mueller response should be additionally calibrated.

[0136] The main scheme uses a discrete stepped focal plane trajectory; if a triangular, sawtooth, or sinusoidal continuous trajectory is used, the polarization state should be collected by repeating the same trajectory, or the focal length difference corresponding to the four-state exposure in the same group should be lower than the calibration threshold.

[0137] The polarization state can be H / V / D / R, tetrahedral, any full-rank four-state, or N>4 over-quantitative state.

[0138] Condition monitoring can utilize electrical feedback, optical reference branches, temperature models, image sharpness, or combinations thereof.

[0139] 3D reconstruction can employ depth-from-focus, depth-from-defocus, PSF deconvolution, neural networks, polarization shape restoration, or a fusion of multiple methods.

[0140] The processing platform can be a CPU, GPU, FPGA, DSP, camera-in-camera processor, or edge computing device.

[0141] The output can be a real-time full-focus preview, 3D point cloud, mesh, volume data, focal plane slice, or raw joint encoded frame.

[0142] In summary, this invention proposes a full Stokes 3D polarization imaging device and method based on liquid lens scanning. Under unified spatiotemporal coordinates and the same measurement period, through depth coordination of liquid lens scanning and polarization time-division modulation, complete Stokes vectors at different focal planes are jointly acquired, and a 3D Stokes data volume, a clear polarization map at full focus, and a 3D point cloud carrying complete polarization attributes are generated. This provides high-dimensional visual information support for the 3D perception and intelligent recognition of transparent, reflective, weakly textured, and scattering targets.

[0143] Key innovative designs and technical points of this invention include: a unified synchronous controller generates the liquid lens focal length trajectory, polarization state sequence, exposure trigger, and joint state identifier, enabling focal plane scanning, polarization switching, and exposure trigger to share the same time base; the joint state identifier includes at least the focal plane number and polarization analysis state number, preferably also including the complete scan cycle number, focusing direction, and valid state flag, ensuring complete and traceable state information for each frame; the exposure window is located at the intersection of the liquid lens focal length valid window, the polarization analyzer valid window, and the camera's allowed window, ensuring that each exposure is within the device's stable range; each focal plane uses only the same complete scan cycle, the same focal plane, the same focusing direction, and a complete and valid image group for full Stokes reconstruction; different focal lengths use corresponding... The geometric mapping parameters, point spread function (PSF), and polarization instrument matrix are calibrated and reconstructed to correct for magnification, distortion, vignetting, and polarization crosstalk variations caused by focusing. During full-focus fusion, sharpness weights are uniformly calculated based on the S0 components of each focal plane, and the same set of weights is applied to all four Stokes components to maintain the physical consistency of Stokes in the full-focus results. Depth estimation employs joint optimization of focus sharpness and polarization normal constraints, ensuring that each point in the output 3D point cloud carries spatial coordinates and complete Stokes polarization attributes. Each focal plane undergoes exposure in at least four linearly independent analysis states while maintaining stable focus, switching to the next focal plane only after all four states are complete. When the buffer is backlogged, old data can be discarded, but the frame number and modulation phase continue to accumulate, preventing the real-time preview from slowing down.

[0144] Based on the above, compared with the prior art, the significant advantages of the present invention are reflected in the following aspects: Liquid lenses can be directly focused without moving solid lens groups along the optical axis, which can shorten the focal plane switching time and facilitate the realization of compact three-dimensional vision sensors.

[0145] Each frame of image synchronously records the focal plane number and polarization analysis status number, so the processing end will not mix images of different focal planes or different complete scan cycles into one group, fundamentally eliminating the risk of status confusion.

[0146] Exposure is triggered only when both the liquid lens and the liquid crystal variable delay device reach a preset effective stable state, which can effectively avoid the two devices being in the switching transient or unstable range, ensuring the true accuracy of the focal length and analysis state at the time of exposure.

[0147] To address system parameter variations caused by focusing of liquid lenses, the geometric mapping, point spread function, and polarization instrument matrix are stored independently at each focal length. This allows for precise correction of magnification, distortion, vignetting, and polarization crosstalk variations resulting from focusing, ensuring the accuracy of Stokes inversion at different focal planes.

[0148] The output data fully includes the distribution of S0, S1, S2, and S3 with depth, rather than just a two-dimensional image of polarization degree or polarization direction angle, which significantly enriches the dimensionality of the output information and provides more complete polarization features for subsequent high-dimensional vision tasks.

[0149] The four Stokes components share the same set of focus weights for full-focus fusion, which avoids the problem of losing the same physical measurement relationship after fusion due to independent focus selection of each component, and ensures the physical consistency of the full-focus Stokes results.

[0150] When the cache is piling up, you can choose to discard old data, but the frame number and modulation phase continue to accumulate, and the real-time preview display will not become slower and slower due to the continuous growth of the cache.

[0151] Each focal plane completes at least four linear independent analysis state groups within a stable window before switching to the next focal plane. This ensures that all images used for Stokes inversion of the same focal plane are acquired at the same stable focal length, avoiding polarization and depth errors introduced by images with different focal lengths participating in the same inversion.

[0152] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0153] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0154] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0155] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0156] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0157] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0159] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0161] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0162] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0163] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0164] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A full Stokes three-dimensional polarization imaging device based on liquid lens scanning, characterized in that, include: A liquid lens is used to change the optical power under the control of a drive signal so that the object-side focal plane scans along a set trajectory in the depth direction; A polarization analyzer is disposed in the outgoing optical path of the liquid lens and is used to apply no less than four linearly independent polarization analysis states to the beam. An image sensor is used to receive the light signal emitted by the polarization analyzer and output image data frame by frame. A synchronization controller is connected to the liquid lens, the polarization analyzer and the image sensor respectively. It is used to output a unified drive signal to control the focal plane scanning timing and polarization state switching timing. It generates and latches a joint state identifier corresponding to each frame when each frame is exposed. The joint state identifier includes at least the focal plane number and the polarization analysis state number. The processing unit, connected to the image sensor and the synchronization controller, is used to receive the image data and the corresponding joint state identifier, group each frame of image according to the focal plane and polarization analysis state based on the joint state identifier, and invert the full Stokes vector of the corresponding focal plane based on each group of images.

2. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The synchronization controller is configured as follows: At each focal plane, the liquid lens is controlled to maintain stable optical power, and within the stable range, the polarization analyzer is controlled to sequentially switch to each polarization analysis state, and the image sensor is triggered to expose for each polarization analysis state. The synchronous controller latches the current focal plane number, polarization analysis status number, complete scan cycle number, focus direction and valid status flag along the exposure trigger line, and binds the latched information to the corresponding image frame. The effective status flag at least indicates the focal length stabilization state of the liquid lens at the start of exposure, the polarization state stabilization state of the polarization analyzer, and whether the exposure process is complete.

3. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The synchronization controller is also configured to: Determine whether the current focal length of the liquid lens has entered the preset focal length tolerance range and whether the holding time exceeds the first threshold, and generate a valid focal length signal. Determine whether the current analysis state of the polarization analyzer has entered the preset polarization state tolerance range and the holding time exceeds the second threshold, and generate a valid polarization signal; Determine if the image sensor is in an exposure-allowed state and generate a camera-ready signal; An exposure trigger signal is generated only when the focal length valid signal, polarization valid signal, and camera ready signal are all valid at the same time, and the joint state identifier corresponding to that moment is latched. During the exposure period, each valid signal is continuously sampled. At the end of the exposure, it is confirmed that each signal remains valid within the exposure window, and the confirmation result is recorded in the valid status flag.

4. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The processing unit is configured with a focal length calibration parameter set, which is stored separately for each focal plane and includes at least: The geometric mapping parameters under the focal plane are used to register the images of each polarization analysis state to a unified spatial coordinate system; The polarization instrument matrix under this focal plane is used to invert the full Stokes vector of the image intensity corresponding to multiple polarization analysis states acquired from this focal plane; The point spread function parameter under this focal plane is used for focus evaluation and image restoration; The focal length-object distance mapping relationship under this focal plane is used to convert depth coordinates from the focal plane number; When reconstructing the full Stokes vector of any focal plane, the processing unit calls the calibration parameter set corresponding to that focal plane for inversion.

5. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The synchronization controller is configured to control the liquid lens to sequentially reach each preset focal plane and remain stable within each complete scan cycle, and to sequentially complete the image acquisition of all polarization analysis states within the stable range of each focal plane. All polarization analysis state images of the same focal plane have the same complete scan cycle number, focal plane number and scanning direction. The processing unit is configured to perform a complete group check on each focal plane, which includes: Check if the image frame containing all the required polarization analysis states is located under the focal plane; Check whether the complete scan cycle number, focal plane number, and scanning direction of each frame are consistent; Check whether all valid status flags in each frame are valid; The image set of the focal plane is used for full Stokes vector inversion only if all the above checks pass; if any check fails, the entire image set of the focal plane is not used for reconstruction, and the use of image frames from adjacent focal planes or other periods for supplementation is prohibited.

6. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The polarization analyzer includes two liquid crystal variable retarders with different fast axis directions and a fixed linear polarizer. The synchronization controller switches the polarization analysis state by changing the voltage applied to the two liquid crystal variable retarders. Alternatively, the polarization analyzer employs a group of polarization modulation devices capable of generating a full-rank analysis matrix, wherein the normalized analysis vectors of the analysis states on the Poincaré sphere are pairwise linearly independent.

7. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The processing unit is further configured to: After inverting each focal plane to obtain the full Stokes vector, they are combined according to the depth direction to form four three-dimensional Stokes data volumes: S0, S1, S2, and S3. For the S0 component of each focal plane, calculate the sharpness evaluation function and generate the sharpness weight of each pixel in each focal plane. The same set of sharpness weights are applied to the four components S0, S1, S2, and S3 respectively for weighted fusion to obtain a full-focus sharp Stokes image; The sharpness weights are non-negative and normalized across all focal planes, with the weights corresponding to low signal-to-noise ratio or low confidence regions being reduced or set to zero.

8. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The processing unit is further configured to: By using the sharpness evaluation results of the S0 components of each focal plane and the focal length-object distance mapping relationship, the coarse depth of each pixel in the scene is estimated. The polarization direction angle is calculated using the S1 and S2 components obtained from the inversion, and the degree of polarization and ellipticity are calculated using S0, S1, S2, and S3. Using the coarse depth as the initial value, a joint optimization objective function including focus sharpness constraint, polarization normal constraint and spatial smoothness constraint is constructed, and the refined depth map is obtained by iterative solution; The refined depth map is associated with a fully focused Stokes image and a polarization attribute map to output a 3D point cloud for each 3D point, carrying spatial coordinates and attributes such as S0, S1, S2, S3, degree of polarization, polarization direction angle, and ellipticity angle.

9. The full Stokes three-dimensional polarization imaging device based on liquid lens scanning as described in claim 1, characterized in that, The synchronization controller is also configured to select an operating mode according to task requirements and stop modulation unrelated to the target output, including: When only intensity or depth information is needed, stop the state switching of the polarization analyzer and keep it in a fixed transmittance state, and the number of polarization analysis states is reduced to 1. When only full Stokes information of the focal plane is needed, stop the focal plane scanning of the liquid lens and fix it on the target focal plane; When only linear polarization information is needed and circular polarization information is not needed, stop generating circular polarization analysis state; After mode switching, the synchronization controller writes the current working mode, effective focal plane set, effective polarization analysis state set, and calibration parameter version into the frame data.

10. A full Stokes three-dimensional polarization imaging method based on liquid lens scanning, characterized in that, Includes the following steps: S1. The synchronous controller controls the liquid lens to sequentially reach each preset focal plane and maintain stable optical power. Within the stable range of each focal plane, the polarization analyzer is controlled to sequentially switch to no less than four linearly independent polarization analysis states. After each polarization analysis state stabilizes, the image sensor is triggered to expose. At the same time, the joint state identifier corresponding to each frame image is latched. The joint state identifier includes at least the focal plane number, the polarization analysis state number, and the valid state flag. S2. The processing unit receives image frames and their joint status identifiers, performs status verification on each frame, discards frames with invalid valid status identifiers or conflicting status identifiers, and groups the frames that pass the verification according to the focal plane and the complete scan cycle. S3. For any focal plane, check whether the polarization analysis state images within the same complete scan cycle under that focal plane are complete and consistent. If they are complete, call the pre-stored calibration parameter set of that focal plane to perform geometric registration on each image, and invert the full Stokes vector of that focal plane from the registered intensity vector and the polarization instrument matrix of that focal plane. If any polarization analysis state is missing or inconsistent, the entire set of images will not participate in the reconstruction. S4. Combine the full Stokes vectors obtained from the inversion of each focal plane according to the depth direction to form a three-dimensional Stokes data volume of S0, S1, S2, and S3; calculate the sharpness evaluation function based on the S0 component of each focal plane and generate sharpness weights; apply the same set of weights to S0, S1, S2, and S3 respectively for weighted fusion to obtain a full Stokes image with full focus sharpness. S5. Based on the sharpness evaluation results of each focal plane and the pre-stored focal length-object distance mapping, estimate the coarse depth. Combine the polarization direction angle and polarization degree obtained by inversion to construct a joint optimization objective function, solve to obtain the refined depth map, and output a 3D point cloud containing spatial coordinates and complete Stokes polarization attributes, as well as a full-focus sharp full Stokes image.

Images