High-time-resolution Shack-Hartmann wavefront sensor based on event camera

Abstract

The invention relates to the technical field of Shack-Hartmann wavefront sensors, in particular to a high-time-resolution Shack-Hartmann wavefront sensor based on an event camera, which comprises a regularly arranged microlens array and an event camera based on a neuromorphic visual detection mechanism, the event camera is located on a focal plane of the microlens array; the micro-lens array divides a high-dynamic distortion light field incident on the micro-lens array and focuses the high-dynamic distortion light field on an event camera located on a micro-lens focal plane, and the event camera is used for receiving incident photons; the event camera converts the light intensity of the incident light field and outputs the light intensity as an event stream signal. In view of a unique response mechanism of the event camera, the method has high-time-resolution wavefront sensing capability; meanwhile, since the pixels in the event camera asynchronously and independently respond to the light intensity change of the incident light field, compared with a traditional camera based on the frame detection principle, the data volume is remarkably reduced, and the data transmission and processing pressure at the rear end is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of Shack-Hartmann wavefront sensors, in particular to a high time-resolution Shack-Hartmann wavefront sensor based on an event camera. Background Art

[0002] Traditional Shack-Hartmann wavefront sensors primarily consist of a regularly arranged microlens array and an array-type image sensor. Specific array-type image sensors are primarily photosensitive devices based on the frame detection principle, such as CCD cameras, CMOS cameras, and photodiode array detectors, as exemplified by the Chinese invention patent CN202111630802.7. However, current array-type image sensors based on the frame detection principle generally have a frame rate of kHz. Due to this hardware limitation, they cannot be used for transient sensing of faster-changing wavefronts, significantly restricting the further development of adaptive optics technology. Furthermore, due to the array output, they require high transmission bandwidth and consume high power.

[0003] To this end, a high time-resolution Shack-Hartmann wavefront sensor based on an event camera is proposed. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low temporal resolution of wavefront sensing in the existing traditional Shack-Hartmann wavefront sensor based on the frame detection principle, and at the same time break through the high data bandwidth limitation brought about by the use of high-resolution cameras to achieve high-precision wavefront perception.

[0005] The above technical problems are solved by the following technical solutions: The present invention proposes a high time-resolution Shack-Hartmann wavefront sensor based on an event camera, comprising a regularly arranged microlens array and an event camera based on a neuromorphic vision detection mechanism;

[0006] The event camera is located at the focal plane of the microlens array;

[0007] The microlens array divides the high-dynamic distortion light field incident thereon and focuses the light onto an event camera located at a focal plane of the microlens, wherein the event camera is used to receive incident photons;

[0008] The event camera converts the light intensity of the incident light field and outputs it as an event stream signal;

[0009] Restore the event stream signal into the incident high-dynamic distorted light field.

[0010] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention: the event stream signal includes:

[0011] The pixel location where the light intensity change occurs;

[0012] Markings indicating increases or decreases in light intensity;

[0013] The moment when a change in light intensity occurs.

[0014] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on the event camera of the present invention, restoring the event stream signal into the incident high dynamic distortion light field includes:

[0015] Perform denoising preprocessing on event stream signals;

[0016] The event stream signal that has been pre-processed by denoising is used to reconstruct the incident high-dynamic distortion light field through the wavefront restoration algorithm;

[0017] The denoising preprocessing can separate valid event stream signals from noisy event stream signals.

[0018] In a preferred embodiment of the high time-resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention, the wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction.

[0019] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on the event camera of the present invention: the indirect wavefront restoration algorithm includes:

[0020] Generative models based on event stream signals;

[0021] Reconstructing a frame signal from a time-unaligned historical event stream signal;

[0022] Reconstruct the incident high dynamic distortion light field.

[0023] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention: the reconstruction of the incident high dynamic distortion light field adopts a centroid algorithm to perform wavefront reconstruction.

[0024] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention: the wavefront reconstruction performed by the centroid algorithm includes:

[0025] Extracting the centroid of each microlens sub-aperture spot from the reconstructed frame signal;

[0026] Combined with the pre-calibrated reconstruction matrix, the calculated centroid coordinates are used to invert the wavefront information.

[0027] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on the event camera of the present invention: the reconstruction of the incident high dynamic distortion light field adopts a wavefront reconstruction method based on a deep neural network to perform wavefront reconstruction.

[0028] In a preferred embodiment of the high time resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention: the wavefront reconstruction method based on a deep neural network includes:

[0029] Using the constructed high-temporal-resolution wavefront spot array frame database, a deep neural network is pre-trained to establish a nonlinear mapping relationship between the spot array frame and the wavefront;

[0030] The reconstructed frame signal is used as the input of the network, and the output of the network corresponds to the predicted reconstructed wavefront.

[0031] In a preferred embodiment of the high time-resolution Shack-Hartmann wavefront sensor based on an event camera of the present invention, the restored wavefront is its corresponding orthogonal mode coefficient or sampling point-by-sampling phase.

[0032] The beneficial effects of the present invention are: the event camera's response to changes in light intensity can reach the microsecond level, which will be able to record instantaneous wavefront information more quickly, and is particularly suitable for transient wavefront perception needs in high-dynamic scenes; because the pixels in the event camera respond asynchronously and independently to changes in the light intensity of the incident light field, compared with traditional cameras based on the frame detection principle, the amount of data is significantly reduced, reducing the data transmission and processing pressure on the back-end; the event camera itself is invented based on bionic principles, and the new Shack-Hartmann wavefront sensor constructed by the event camera has the practical advantage of low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention. Obviously, the drawings described below only relate to some embodiments of the present invention, and are not intended to limit the present invention.

[0034] Figure 1 A schematic diagram of the overall structure of a high-time-resolution Shack-Hartmann wavefront sensor based on an event camera is shown.

[0035] Figure 2 A schematic diagram showing event stream signal generation of an event camera is shown.

[0036] Figure 3 The stages of evolution from incident wavefront to incident flow signal are shown.

[0037] Figure 4 The process of recovering high dynamic wavefront from the acquired noisy event stream signal is shown.

[0038] Figure 5 An indirect wavefront restoration method based on frame signal reconstruction is shown.

[0039] Figure 6The second indirect wavefront restoration method based on frame signal reconstruction is shown. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to specific embodiments and the accompanying drawings.

[0041] The terms used in the present invention are those commonly used in the art in view of the functions of the present invention, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the present invention. Therefore, the terms used in the specification should not be understood as simple names, but rather as the meanings of the terms and the overall description of the present invention.

[0042] Reference Figure 1 This embodiment provides a high time-resolution Shack-Hartmann wavefront sensor based on an event camera, comprising a regularly arranged microlens array and an event camera based on a neuromorphic vision detection mechanism; in this embodiment, the microlenses are convex lenses, which can be plano-convex or bi-convex lenses, and when arranged, they can be closely packed without spacing, or non-close packed with spacing, and can be rectangular or hexagonal, and the microlens sub-apertures can be circular, rectangular, or hexagonal.

[0043] The event camera is located at the focal plane of the microlens array;

[0044] The microlens array divides the high-dynamic distortion light field incident thereon and focuses the light onto an event camera located at a focal plane of the microlens, wherein the event camera is used to receive incident photons;

[0045] It should be noted that each pixel in the event camera responds asynchronously and independently to changes in the intensity of the incident light field. The delay to light intensity changes is on the order of microseconds, and the event signal output frequency can be as high as the MHz level.

[0046] High dynamic distortion light field means that when the light from a natural guide star or a laser guide star passes through the atmosphere and enters the optical system, the light field received by the optical system is constantly changing due to the time-varying nature of atmospheric turbulence, and the wavefront is disturbed by the atmospheric turbulence, resulting in distortion.

[0047] The event camera converts the light intensity of the incident light field and outputs it as an event stream signal;

[0048] The event stream signal includes:

[0049] The pixel position (x,y) where the light intensity change occurs;

[0050] Mark p indicating increase or decrease of light intensity;

[0051] The time t when the light intensity changes.

[0052] That is, an element of the event stream signal can be recorded as {(x,y),t,p};

[0053] like Figure 2 As shown, the mathematical principle of this process can be expressed as:

[0054] lgI(x k ,y k ,t j )-lgI(x k ,y k ,t j-1 )=p j C

[0055] Among them, (x k ,y k ) is the detector pixel coordinate; C is the set threshold; p j is the polarity of the jth event signal. When the light intensity increases, p j =1, when the light intensity decreases, p j =-1; t j and t j-1 are the moments when the event is generated; I(x,y,t) is the light intensity at the spatial position (x,y) and time t; lgI is the logarithmic light intensity;

[0056] As an optional embodiment: the process of the incident high dynamic distortion light field passing through the sensor is as follows: Figure 3 shown.

[0057] The process of restoring the event stream signal to the incident high dynamic distortion light field includes, for example Figure 4 As shown;

[0058] Perform denoising preprocessing on event stream signals;

[0059] The event stream signal that has been pre-processed by denoising is used to reconstruct the incident high-dynamic distortion light field through the wavefront restoration algorithm;

[0060] The denoising preprocessing can separate valid event stream signals from noisy event stream signals.

[0061] The noisy event stream signal collected by the event camera is subjected to a denoising preprocessing algorithm to obtain a denoised event stream signal, and the denoised event stream signal can be inverted using a wavefront restoration algorithm to obtain a restored wavefront.

[0062] As an optional embodiment: the wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction.

[0063] In this embodiment, the restored wavefront can be its corresponding orthogonal mode coefficient (such as Zernike mode) or the sampling point-by-sample phase;

[0064] The indirect wavefront restoration algorithm includes:

[0065] Generative model based on event stream signals (forward model from light intensity of light spot array to event stream signals);

[0066] Reconstruct the frame signal from the time-unaligned historical event stream signal (which is the inverse process of the forward model, that is, the inverse process from the event stream signal to the light intensity of the spot array). The frame signal corresponds to the light intensity distribution of the focal plane spot at a certain moment;

[0067] Reconstruct the incident high dynamic distortion light field.

[0068] After capturing wavefront information at a finer timescale using an event camera, an indirect wavefront reconstruction algorithm based on frame reconstruction can be used to restore this finer timescale. This is a limitation of traditional frame cameras and the key technological innovation of this invention in achieving high-dynamic wavefront sensing.

[0069] As an optional embodiment: the reconstruction of the incident high dynamic distortion light field adopts the centroid algorithm to perform wavefront reconstruction, referring to Figure 5 ;

[0070] The centroid algorithm for wavefront reconstruction includes:

[0071] Extracting the centroid of each microlens sub-aperture spot from the reconstructed frame signal;

[0072] Combined with the pre-calibrated reconstruction matrix, the calculated centroid coordinates are used to invert the wavefront information.

[0073] For the form in which the wavefront is expressed by its corresponding orthogonal mode coefficients, the reconstruction matrix is calibrated with the centroid of the spot of each orthogonal mode of unit amplitude on each sampling sub-aperture;

[0074] For the form in which the wavefront is expressed by the phase of each sampling point, the reconstruction matrix calibrates the relationship between the difference of the wavefront at the sampling point and the centroid. Its models include the Hudgin model, the Fried model, and the Southwell model.

[0075] As an optional embodiment: the reconstruction of the incident high dynamic distortion light field adopts a wavefront reconstruction method based on a deep neural network to perform wavefront reconstruction, referring to Figure 6 .

[0076] The wavefront reconstruction method based on deep neural network performs wavefront reconstruction, including:

[0077] Using the constructed high-temporal-resolution wavefront spot array frame database, a deep neural network is pre-trained to establish a nonlinear mapping relationship between the spot array frame and the wavefront;

[0078] The reconstructed frame signal is used as the input of the network, and the output of the network corresponds to the predicted reconstructed wavefront.

[0079] Because pixels in an event camera respond asynchronously and independently to changes in the intensity of the incident light field, compared to traditional frame-detection cameras, they significantly reduce data throughput and ease the burden on back-end data transmission and processing. When the logarithmic intensity change of a pixel does not exceed threshold C, it will not output a signal; only when the logarithmic intensity change of a pixel exceeds threshold C will it output a signal. Therefore, event cameras have pixel-level output. However, for array-type image sensors based on the frame-detection principle, all pixels will output a signal during any exposure time, regardless of the incident light intensity.

[0080] The event camera can respond to changes in light intensity at the microsecond level, which will enable it to record instantaneous wavefront information more quickly. It is particularly suitable for transient wavefront perception needs in high-dynamic scenes.

[0081] The event camera itself is invented based on bionic principles, and the new Shack-Hartmann wavefront sensor constructed by the event camera has the practical advantage of low power consumption.

[0082] Finally, it should be pointed out that the methods and devices described in detail above are merely embodiments, and those skilled in the art can modify these embodiments in different ways without departing from the scope of the present invention.

Claims

1. A high time-resolution Shack-Hartmann wavefront sensor based on an event camera, characterized by: Includes a regularly arranged microlens array and an event camera based on a neuromorphic vision detection mechanism; The event camera is located at the focal plane of the microlens array; The microlens array divides the high-dynamic distortion light field incident thereon and focuses the light onto an event camera located at a focal plane of the microlens, wherein the event camera is used to receive incident photons; The event camera converts the light intensity of the incident light field and outputs it as an event stream signal; Restore the event stream signal into the incident high-dynamic distorted light field.

2. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 1, characterized in that: The event stream signal includes: The pixel location where the light intensity change occurs; Markings indicating increases or decreases in light intensity; The moment when a change in light intensity occurs.

3. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 1 or 2, characterized in that: Restoring the event stream signal to the incident high dynamic distortion light field includes, Perform denoising preprocessing on event stream signals; The event stream signal that has been pre-processed by denoising is used to reconstruct the incident high-dynamic distortion light field through the wavefront restoration algorithm; The denoising preprocessing can separate valid event stream signals from noisy event stream signals.

4. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 3, characterized in that: The wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction.

5. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 4, characterized in that: The indirect wavefront restoration algorithm includes: Generative models based on event stream signals; Reconstructing a frame signal from a time-unaligned historical event stream signal; Reconstruct the incident high dynamic distortion light field.

6. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 5, characterized in that: The incident high-dynamic distortion light field is reconstructed by using a centroid algorithm to perform wavefront reconstruction.

7. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 6, characterized in that: The centroid algorithm for wavefront reconstruction includes: Extracting the centroid of each microlens sub-aperture spot from the reconstructed frame signal; Combined with the pre-calibrated reconstruction matrix, the calculated centroid coordinates are used to invert the wavefront information.

8. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 5, characterized in that: The reconstruction of the incident high-dynamic distortion light field adopts a wavefront reconstruction method based on a deep neural network to perform wavefront reconstruction.

9. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 8, characterized in that: The wavefront reconstruction method based on deep neural network performs wavefront reconstruction, including: Using the constructed high-temporal-resolution wavefront spot array frame database, a deep neural network is pre-trained to establish a nonlinear mapping relationship between the spot array frame and the wavefront; The reconstructed frame signal is used as the input of the network, and the output of the network corresponds to the predicted reconstructed wavefront.

10. The high time-resolution Shack-Hartmann wavefront sensor based on an event camera according to claim 6 or 8, characterized in that: The recovered wavefront is its corresponding orthogonal mode coefficient or sampling point phase.

Images