An event camera based high temporal resolution shack-hartmann wavefront sensor

Abstract

The present application relates to the technical field of Shack-Hartmann wavefront sensor, and particularly relates to a high-time-resolution Shack-Hartmann wavefront sensor based on an event camera, which comprises a regularly arranged microlens array and an event camera based on a neuromorphic visual detection mechanism; the event camera is located on the focal plane of the microlens array; the microlens array divides a high-dynamic-distortion light field incident thereon and focuses the light field on the event camera on the focal plane of the microlens, and the event camera is used for receiving the incident photons; the event camera converts and outputs the light intensity of the incident light field into an event stream signal. In view of the unique response mechanism of the event camera, the present application has a high-time-resolution wavefront sensing capability; meanwhile, since the pixels in the event camera respond to the light intensity change of the incident light field asynchronously and independently, compared with a traditional camera based on a frame detection principle, the data amount is significantly reduced, and the data transmission and processing pressure of the rear end is reduced.

Description

Technical Field

[0001] This invention relates to the field of Shakhartmann wavefront sensor technology, and in particular to a high time-resolution Shakhartmann wavefront sensor based on an event camera. Background Technology

[0002] Traditional Shaker-Hartmann wavefront sensors primarily consist of a regularly arranged array of microlenses and an array-type image sensor. Specific array-type image sensors are mainly photosensitive devices based on the frame detection principle, such as CCD cameras, CMOS cameras, and photodiode array detectors, as exemplified by the Chinese authorized invention patent CN202111630802.7. However, current array-type image sensors based on the frame detection principle generally have frame rates in the kHz range. This hardware limitation prevents them from being used for transient sensing of faster-changing wavefronts, significantly hindering the further development of adaptive optics technology. Furthermore, due to the array output, they require high transmission bandwidth and consume significant power.

[0003] To address this, a high temporal resolution Shakhartmann wavefront sensor based on an event camera is proposed. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low wavefront sensing time resolution in the existing traditional Shakhartmann wavefront sensors based on the frame detection principle, and at the same time to overcome the high data bandwidth limitation brought about by the use of high-resolution cameras to achieve high-precision wavefront sensing.

[0005] The above-mentioned technical problems are solved by the following technical solution: The present invention proposes a high temporal resolution Shakhartmann wavefront sensor based on an event camera, including a regularly arranged microlens array and an event camera based on a neuromorphic visual detection mechanism.

[0006] The event camera is located on the focal plane of the microlens array;

[0007] The microlens array segments the highly dynamic distorted light field incident upon it and focuses it onto an event camera located at the focal plane of the microlens, the event camera being used to receive incident photons;

[0008] The event camera converts the light intensity of the incident light field and outputs it as an event stream signal;

[0009] The event stream signal is restored into the incident high dynamic range distorted light field.

[0010] In a preferred embodiment of the high time-resolution Shakhartmann wavefront sensor based on an event camera according to the present invention: the event stream signal includes,

[0011] The pixel location where the light intensity changes;

[0012] A marker indicating an increase or decrease in light intensity;

[0013] The moment when the light intensity changes.

[0014] In a preferred embodiment of the high time-resolution Shakhartmann wavefront sensor based on an event camera described in this invention: restoring the event stream signal into an incident high dynamic distortion light field includes...

[0015] Denoising preprocessing is performed on the event stream signal;

[0016] The denoised event stream signal is reconstructed using a wavefront restoration algorithm to reconstruct the incident high dynamic distortion light field.

[0017] The denoising preprocessing can separate the valid event stream signal from the noisy event stream signal.

[0018] In a preferred embodiment of the high temporal resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction.

[0019] In a preferred embodiment of the high temporal resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the indirect wavefront reconstruction algorithm includes...

[0020] Generation model based on event stream signals;

[0021] Reconstructing frame signals from time-misaligned historical event stream signals;

[0022] Reconstruction of the incident high dynamic distortion light field.

[0023] In a preferred embodiment of the high time-resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the reconstruction of the incident high dynamic distortion light field is performed using a centroid algorithm for wavefront reconstruction.

[0024] In a preferred embodiment of the high temporal resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the centroid algorithm for wavefront reconstruction includes...

[0025] Extract the centroid of the light spot of each microlens sub-aperture from the reconstructed frame signal;

[0026] By combining the pre-calibrated reconstruction matrix, wavefront information is inverted using the calculated centroid coordinates.

[0027] In a preferred embodiment of the high time-resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the reconstruction of the incident high dynamic distortion light field is performed using a wavefront reconstruction method based on a deep neural network.

[0028] In a preferred embodiment of the high temporal resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the wavefront reconstruction method based on a deep neural network includes the following steps:

[0029] Using a constructed high temporal resolution wavefront spot array frame database, a deep neural network was pre-trained to establish a nonlinear mapping relationship between the spot array frames and the wavefront.

[0030] The reconstructed frame signal is used as the input to the network, and the output of the network corresponds to the predicted reconstructed wavefront.

[0031] In a preferred embodiment of the high time-resolution Shakhartmann wavefront sensor based on an event camera described in this invention: the recovered wavefront is its corresponding orthogonal mode coefficient or sample-by-sample phase.

[0032] The beneficial effects of this invention are as follows: the response of the event camera to changes in light intensity can reach the microsecond level, which will enable faster recording of instantaneous wavefront information, making it particularly suitable for transient wavefront sensing needs in high dynamic scenes; since the pixels in the event camera respond asynchronously and independently to changes in the light intensity of the incident light field, compared with traditional cameras based on the frame detection principle, the amount of data is significantly reduced, reducing the data transmission and processing pressure on the backend; the event camera itself is invented based on biomimetic principles, and the novel Shaker-Hartmann wavefront sensor constructed by the event camera has the practical advantage of low power consumption. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0034] Figure 1 A schematic diagram of the overall structure of a high time-resolution Shaker-Hartmann wavefront sensor based on an event camera is shown.

[0035] Figure 2 A schematic diagram of event stream signal generation from an event camera is shown.

[0036] Figure 3 The evolution from the incident wavefront to the event stream signal is shown at each stage.

[0037] Figure 4 The process of reconstructing a high dynamic range wavefront from a noisy event stream signal is shown.

[0038] Figure 5 An indirect wavefront reconstruction method based on frame signal reconstruction is shown.

[0039] Figure 6A second indirect wavefront reconstruction method based on frame signal reconstruction is shown. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0041] The terminology used in this invention is that which is currently widely used in the art in consideration of the function of the invention; however, these terms may vary according to the intent of those skilled in the art, precedent, or new technology in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the invention. Therefore, the terms used in this specification should not be construed as simple names, but rather based on their meanings and the overall description of the invention.

[0042] Reference Figure 1 This embodiment provides a high temporal resolution Shakhartmann wavefront sensor based on an event camera, including a regularly arranged microlens array and an event camera based on a neuromorphic visual detection mechanism. In this embodiment, the microlenses are convex lenses, which can be plano-convex or biconvex lenses. When arranged, they can be closely packed without spacing or non-closely packed with spacing. They can be arranged in rectangles or hexagons. The microlens sub-apertures can be circular, rectangular, or hexagonal.

[0043] The event camera is located on the focal plane of the microlens array;

[0044] The microlens array segments the highly dynamic distorted light field incident upon it and focuses it onto an event camera located at the focal plane of the microlens, the event camera being used to receive incident photons;

[0045] It should be noted that each pixel in the event camera responds asynchronously and independently to changes in the intensity of the incident light field, with a delay in response to these changes in the order of microseconds, and the event signal output frequency can reach the order of MHz.

[0046] High dynamic distortion light field refers to the light field received by an optical system when the light from a natural guide star or a laser guide star passes through the atmosphere and is incident on the optical system. Due to the time-varying nature of atmospheric turbulence, the light field is constantly changing and the wavefront is disturbed by atmospheric turbulence, resulting in distortion.

[0047] The event camera converts the light intensity of the incident light field and outputs it as an event stream signal;

[0048] The event stream signal includes,

[0049] The pixel position (x, y) where the light intensity changes;

[0050] The marker p indicates an increase or decrease in light intensity;

[0051] The moment t when the light intensity changes.

[0052] That is, an element of the event stream signal can be denoted as {(x,y),t,p};

[0053] like Figure 2 As shown, the mathematical principle of this process can be expressed as follows:

[0054] lgI(x k ,y k ,t j )-lgI(x k ,y k ,t j-1 ) = p j C

[0055] Among them, (x k ,y k ) represents the detector pixel coordinates; C represents the threshold value; p j Let p be the polarity of the signal for the j-th event. As the light intensity increases, j =1, when the light intensity decreases, p j =-1; t j and t j-1 , where represents the time when the event was generated; I(x,y,t) is the light intensity value at spatial location (x,y) and time t; lgI is the logarithmic light intensity;

[0056] As an alternative embodiment: the incident high dynamic range distorted light field passes through the sensor as follows Figure 3 As shown.

[0057] The process of restoring the event stream signal into the incident high dynamic range distorted light field includes, for example: Figure 4 As shown;

[0058] Denoising preprocessing is performed on the event stream signal;

[0059] The denoised event stream signal is reconstructed using a wavefront restoration algorithm to reconstruct the incident high dynamic distortion light field.

[0060] The denoising preprocessing can separate the valid event stream signal from the noisy event stream signal.

[0061] The noisy event stream signal acquired by the event camera is processed by a denoising preprocessing algorithm to obtain a denoised event stream signal. The denoised event stream signal can be inverted by a wavefront restoration algorithm to obtain the restored wavefront.

[0062] As an optional embodiment: the wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction.

[0063] In this embodiment, the recovered wavefront can be its corresponding orthogonal mode coefficients (such as the Zernike mode) or the phase at each sampling point.

[0064] The indirect wavefront reconstruction algorithm includes,

[0065] A generation model based on event stream signals (a forward model from light intensity of light spot array to event stream signals);

[0066] Reconstructing frame signals from historical event stream signals that are not time-aligned (which is the reverse process of the forward model, that is, the reverse process from event stream signals to light intensity of the light spot array), the frame signals correspond to the light intensity distribution of the focal plane light spot at a certain moment;

[0067] Reconstruction of the incident high dynamic distortion light field.

[0068] After acquiring wavefront information with a finer timescale through an event camera, the fine-scale wavefront information can be restored using an indirect wavefront reconstruction algorithm based on frame reconstruction. This is something that traditional frame cameras lack, and it is also the innovation of this invention in achieving high dynamic wavefront sensing from a technical perspective.

[0069] As an optional embodiment: the reconstruction of the incident high dynamic distortion light field uses the centroid algorithm for wavefront reconstruction, referring to... Figure 5 ;

[0070] The centroid algorithm for wavefront reconstruction includes,

[0071] Extract the centroid of the light spot of each microlens sub-aperture from the reconstructed frame signal;

[0072] By combining the pre-calibrated reconstruction matrix, wavefront information is inverted using the calculated centroid coordinates.

[0073] For a wavefront expressed by its corresponding orthogonal mode coefficients, the reconstruction matrix indicates the centroid of the spot on each sampling sub-aperture for each orthogonal mode of unit amplitude.

[0074] For wavefronts expressed as phases at each sampling point, the reconstruction matrix defines the relationship between the difference of the wavefront at each sampling point and the centroid. The models include the Hudgin model, the Fried model, and the Southwell model.

[0075] As an optional embodiment: the reconstruction of the incident high dynamic distortion light field is performed using a wavefront reconstruction method based on deep neural networks, referring to... Figure 6 .

[0076] The wavefront reconstruction method based on deep neural networks includes the following steps:

[0077] Using a constructed high temporal resolution wavefront spot array frame database, a deep neural network was pre-trained to establish a nonlinear mapping relationship between the spot array frames and the wavefront.

[0078] The reconstructed frame signal is used as the input to the network, and the output of the network corresponds to the predicted reconstructed wavefront.

[0079] Because the pixels in an event camera respond asynchronously and independently to changes in the intensity of the incident light field, it significantly reduces data volume and lowers the data transmission and processing pressure on the backend compared to traditional frame-detection-based cameras. When the logarithmic change in light intensity of a pixel does not exceed a threshold C, it will not output a signal; only when the logarithmic change in light intensity exceeds the threshold C will it output a signal. Therefore, event cameras provide pixel-level output. However, for array-type image sensors based on the frame-detection principle, all pixels output signals regardless of the intensity of the incident light during any given exposure time.

[0080] Event cameras can respond to changes in light intensity at the microsecond level, enabling them to record instantaneous wavefront information more quickly. They are particularly suitable for transient wavefront sensing needs in high dynamic scenes.

[0081] The event camera itself is based on the principle of biomimicry, and the novel Shaker-Hartmann wavefront sensor built from the event camera has the practical advantage of low power consumption.

[0082] Finally, it should be noted that the methods and devices described in detail above are merely embodiments, and those skilled in the art can modify these embodiments in different ways as long as they do not depart from the scope of the present invention.

Claims

1. A high time-resolution Shakhartmann wavefront sensor based on an event camera, characterized in that: This includes regularly arranged microlens arrays and event cameras based on neuromorphic visual detection mechanisms; The event camera is located on the focal plane of the microlens array; The microlens array segments the highly dynamic distorted light field incident upon it and focuses it onto an event camera located at the focal plane of the microlens, the event camera being used to receive incident photons; The event camera converts the light intensity of the incident light field and outputs it as an event stream signal; The event stream signal is restored into the incident high dynamic distortion light field; The event stream signal includes, The pixel location where the light intensity changes; A marker indicating an increase or decrease in light intensity; The moment when the light intensity changes; The process of restoring the event stream signal into the incident high dynamic range distorted light field includes... Denoising preprocessing is performed on the event stream signal; The denoised event stream signal is reconstructed using a wavefront restoration algorithm to reconstruct the incident high dynamic distortion light field. The denoising preprocessing can separate the valid event stream signal from the noisy event stream signal; The wavefront restoration algorithm is an indirect wavefront restoration algorithm based on frame reconstruction. The indirect wavefront reconstruction algorithm includes, Based on the event stream signal generation model, frame signals are reconstructed from time-misaligned historical event stream signals; Reconstruction of the incident high dynamic distortion light field.

2. The high time-resolution Shakhartmann wavefront sensor based on an event camera according to claim 1, characterized in that: The reconstruction of the incident high dynamic distortion light field uses the centroid algorithm for wavefront reconstruction.

3. The high time-resolution Shakhartmann wavefront sensor based on an event camera according to claim 2, characterized in that: The centroid algorithm for wavefront reconstruction includes, Extract the centroid of the light spot of each microlens sub-aperture from the reconstructed frame signal; By combining the pre-calibrated reconstruction matrix, wavefront information is inverted using the calculated centroid coordinates.

4. The high time-resolution Shakhartmann wavefront sensor based on an event camera according to claim 1, characterized in that: The reconstruction of the incident high dynamic distortion light field is performed using a wavefront reconstruction method based on deep neural networks.

5. The high time-resolution Shakhartmann wavefront sensor based on an event camera according to claim 4, characterized in that: The wavefront reconstruction method based on deep neural networks includes the following steps: Using a constructed high temporal resolution wavefront spot array frame database, a deep neural network was pre-trained to establish a nonlinear mapping relationship between the spot array frames and the wavefront. The reconstructed frame signal is used as the input to the network, and the output of the network corresponds to the predicted reconstructed wavefront.

6. The high time-resolution Shakhartmann wavefront sensor based on an event camera according to claim 2 or 4, characterized in that: The recovered wavefront is its corresponding orthogonal mode coefficient or sample-by-sample phase.

Images