High-speed microscopic imaging device and method

Abstract

The present disclosure relates to high-speed microscopic imaging apparatuses and methods. A high-speed microscopic imaging apparatus includes a coherent light source configured to illuminate a target with coherent light, a spatial light modulator configured to receive a diffraction pattern of the target illuminated by the coherent light and encode the diffraction pattern of the target according to an encoding pattern of the spatial light modulator to generate an encoded diffraction pattern, and an imaging sensor configured to receive multiple frames of the encoded diffraction pattern from the spatial light modulator within an exposure time of the imaging sensor to generate a single compressed encoded diffraction pattern of the target, wherein the spatial light modulator is configured to switch the encoding pattern multiple times within the exposure time of the imaging sensor to generate the multiple frames of the encoded diffraction pattern by encoding multiple frames of the diffraction pattern of the target at different times within the exposure time of the imaging sensor according to respective different encoding patterns.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of imaging technology, and more particularly, the present disclosure relates to a high-speed microscopic imaging apparatus and method. BACKGROUND

[0002] In microscopic imaging technology, it is often necessary to combine a microscope with a camera, thereby taking pictures of tiny targets that cannot be observed by the human eye. The size of the tiny targets is usually in the order of microns or even nanometers (e.g., as small as a few hundred microns or less), so it is usually necessary to use a microscopic objective lens with a very high magnification (e.g., as high as tens or even hundreds of times or more) to obtain clear imaging of the tiny targets. In high-speed imaging technology, it is often necessary to use a high-speed camera with a very high frame rate (e.g., as high as hundreds of frames per second or more) to obtain clear imaging of targets moving at high speed. SUMMARY

[0003] According to an aspect of the present disclosure, there is provided a high-speed microscopic imaging apparatus, comprising: a coherent light source configured to illuminate a target with coherent light; a spatial light modulator configured to receive a diffraction pattern of the target illuminated by the coherent light and encode the diffraction pattern of the target according to an encoding pattern of the spatial light modulator to generate an encoded diffraction pattern; and an imaging sensor configured to receive multiple frames of the encoded diffraction pattern from the spatial light modulator within an exposure time of the imaging sensor to generate a single compressed encoded diffraction pattern of the target, wherein the spatial light modulator is configured to switch the encoding pattern multiple times within the exposure time of the imaging sensor to encode multiple frames of the diffraction pattern of the target at different times within the exposure time of the imaging sensor according to respective different encoding patterns to generate the multiple frames of the encoded diffraction pattern.

[0004] In some embodiments, the diffraction pattern of the target is formed by light reflected by the target through the coherent light; or the diffraction pattern of the target is formed by light transmitted by the target through the coherent light.

[0005] In some embodiments, the spatial light modulator is positioned at a far field or a Fourier plane of the target.

[0006] In some embodiments, the spatial light modulator and the imaging sensor are positioned to be optically conjugated to each other.

[0007] In some embodiments, the high-speed microscopic imaging apparatus further comprises a first lens group disposed between the coherent light source and the target, the first lens group being configured to collimate the coherent light and / or adjust a beam diameter of the coherent light such that the coherent light fully illuminates the target.

[0008] In some embodiments, the high-speed microscopic imaging device further comprises a Fourier lens disposed between the target and the spatial light modulator, wherein the target is positioned on a front focal plane of the Fourier lens, and the Fourier lens is configured to Fourier transform a diffraction pattern of the target.

[0009] In some embodiments, the spatial light modulator is positioned on a back focal plane of the Fourier lens.

[0010] In some embodiments, the high-speed microscopic imaging device further comprises a second lens group disposed between a back focal plane of the Fourier lens and the spatial light modulator, wherein the second lens group is configured such that the diffraction pattern of the target is imaged within an encoding region of the spatial light modulator after being Fourier transformed by the Fourier lens.

[0011] In some embodiments, the high-speed microscopic imaging device further comprises a third lens group disposed between the spatial light modulator and the imaging sensor, wherein the third lens group is configured such that an encoded diffraction pattern generated by the spatial light modulator is imaged within a sensing region of the imaging sensor.

[0012] In some embodiments, pixels of the encoding pattern of the spatial light modulator are binary encoded to allow or not allow received light to reach the imaging sensor, and a pixel size is larger than a Bragg spacing of the diffraction pattern of the target.

[0013] In some embodiments, the binary encoding of the pixels of the encoding pattern of the spatial light modulator is performed in one of the following ways: a random encoding way; an encoding way that causes more information loss of low frequency parts of the diffraction pattern of the target than high frequency parts of the diffraction pattern of the target; an encoding way that causes more information loss of high intensity parts of the diffraction pattern of the target than low intensity parts of the diffraction pattern of the target.

[0014] In some embodiments, the spatial light modulator is a digital micromirror array comprising a plurality of micromirrors, a rotation angle of each micromirror being digitally configurable to reflect light received by the micromirror to be received or not to be received by the imaging sensor, different encoding patterns of the spatial light modulator being provided by the digital configuration; or the spatial light modulator comprises a plurality of mask plates and switching components for switching the plurality of mask plates, each mask plate of the plurality of mask plates comprising a plurality of regions arranged as an array and each configured to allow light to pass therethrough to be received or not to be received by the imaging sensor, different encoding patterns of the spatial light modulator being provided by different mask plates of the plurality of mask plates; or the spatial light modulator comprises a mask plate having a plurality of regions arranged as an array and a moving component for moving the mask plate such that the diffraction pattern of the target is received by different regions of the plurality of regions of the mask plate, each region of the plurality of regions of the mask plate comprising a plurality of sub-regions arranged as an array and each configured to allow light to pass therethrough to be received or not to be received by the imaging sensor, different encoding patterns of the spatial light modulator being provided by different regions of the plurality of regions of the mask plate.

[0015] In some embodiments, the high-speed microscopic imaging apparatus further comprises an aperture stop disposed between the target and the spatial light modulator and in close proximity to the target for defining an imaging field of view.

[0016] In some embodiments, the imaging sensor is in communication with a processor and configured to transmit the compressed encoded diffraction pattern to the processor, and wherein the processor is configured to decode the compressed encoded diffraction pattern according to the respective encoding patterns of the spatial light modulator and their temporal order within the exposure time of the imaging sensor to obtain the multi-frame diffraction patterns of the target at different times, and to perform phase retrieval on the multi-frame diffraction patterns to reconstruct multi-frame images of the target at different times.

[0017] In some embodiments, the processor is configured to decode the compressed encoded diffraction pattern to obtain the multi-frame diffraction patterns of the target at different times by a multi-frame reconstruction neural network model having a plurality of stages, each stage of the plurality of stages comprising: a projection module configured to map an input of the stage from a compressed domain to a signal domain to obtain a plurality of signal maps, the plurality of signal maps being identical in number to the multi-frame diffraction patterns; a feature extraction module configured to receive the plurality of signal maps from the projection module to extract features; and a feature integration module configured to integrate the extracted features to obtain an output of the stage.

[0018] In some embodiments, each stage of the plurality of stages further comprises a plurality of feature transfer layers s disposed between the feature extraction module and the feature integration module for transferring extracted features i , i being a positive integer greater than 2, wherein each feature transfer layer s i comprises a first reversible block and a second reversible block The output channels of the feature extraction module are divided into two parts to be coupled to the first reversible block 1 and the second reversible block of the first feature transfer layer s nearest to the feature extraction module respectively, and the following relationship is satisfied between adjacent feature transfer layers s i and s i+1 wherein and are the same activation function.

[0019] In some embodiments, the processor is configured to perform the phase retrieval on each of the plurality of frames of diffraction patterns by combining a phase retrieval process with a denoising process to reconstruct a frame of images of the target corresponding to the frame of diffraction patterns.

[0020] In some embodiments, the phase retrieval process is a hybrid input-output (HIO) process, and wherein the processor is configured to, in each iteration of the HIO process performed on each of the plurality of frames of diffraction patterns, input a real-space signal corrected in real space into a denoising neural network model configured to extract a plurality of sub-images from the received real-space signal corrected in real space, input the plurality of sub-images and a noise level map set for the frame of diffraction patterns into a multi-layer neural network, output a plurality of denoised sub-images from the multi-layer neural network, the plurality of denoised sub-images being identical in number and size to the plurality of sub-images, and integrate the plurality of denoised sub-images into a denoised real-space signal for the next iteration.

[0021] In some embodiments, extracting a plurality of sub-images from the received real-space signal corrected in real space comprises dividing an image of the real-space signal into a plurality of primary regions, each of the plurality of primary regions comprising a plurality of secondary regions identical in number to the plurality of sub-images, and extracting each of the plurality of sub-images as comprising a respective one of the secondary regions in each primary region.

[0022] In some embodiments, the high-speed microscopic imaging device further comprises the processor. ​

[0023] According to another aspect of the present disclosure, there is provided a high-speed microscopic imaging method, comprising: illuminating a target with coherent light; receiving and spatially modulating a diffraction pattern of the target illuminated by the coherent light, the spatial modulation comprising encoding the diffraction pattern of the target according to an encoding pattern to generate an encoded diffraction pattern; and compressively imaging a plurality of frames of the encoded diffraction pattern generated by the spatial modulation within a period of time to generate a single compressed encoded diffraction pattern of the target, wherein the spatial modulation comprises switching the encoding pattern a plurality of times within the period of time to encode a plurality of frames of the diffraction pattern of the target at different times within the period of time according to a respective different encoding pattern to generate the plurality of frames of the encoded diffraction pattern.

[0024] In some embodiments, the diffraction pattern of the target is formed by light reflected by the target through the coherent light; or the diffraction pattern of the target is formed by light transmitted by the target through the coherent light.

[0025] In some embodiments, the spatial modulation is performed at a far field or a Fourier plane of the target.

[0026] In some embodiments, the high-speed microscopic imaging method further comprises: decoding the compressed encoded diffraction pattern according to each encoding pattern used by the spatial modulation within the period of time and a time order thereof to obtain the plurality of frames of the diffraction pattern of the target at different times, and performing phase retrieval on the plurality of frames of the diffraction pattern to reconstruct a plurality of frames of images of the target at different times.

[0027] In some embodiments, decoding the compressed encoded diffraction pattern comprises decoding the compressed encoded diffraction pattern by a multi-frame reconstruction neural network model having a plurality of stages, each stage of the plurality of stages comprising: a projection module configured to map an input of the stage from a compressed domain to a signal domain to obtain a plurality of signal maps, the plurality of signal maps being identical in number to the plurality of frames of the diffraction pattern of the target; a feature extraction module configured to receive the plurality of signal maps from the projection module to extract features; and a feature integration module configured to integrate the extracted features to obtain an output of the stage.

[0028] In some embodiments, each stage of the plurality of stages further comprises: a plurality of feature transfer layers si, i being a positive integer greater than 2, disposed between the feature extraction module and the feature integration module for transferring the extracted features, wherein each feature transfer layer si comprises a first reversible block and a second reversible block The output channels of the feature extraction module are divided into two parts to be coupled to the first feature transfer layer s 1 a first reversible block and a second reversible block and the adjacent feature transfer layers s i and s i+1 satisfy the following relationship: wherein and are the same activation function.

[0029] In some embodiments, the phase retrieval on the multiple frames of diffraction patterns includes performing the phase retrieval on each frame of diffraction patterns in the multiple frames of diffraction patterns by a phase retrieval process combined with a denoising process to reconstruct a frame of images of the target corresponding to the frame of diffraction patterns.

[0030] In some embodiments, the phase retrieval process is a hybrid input-output (HIO) process, and wherein the phase retrieval on the multiple frames of diffraction patterns includes, in each iteration of the HIO process performed on each frame of diffraction patterns in the multiple frames of diffraction patterns, inputting a real-space corrected real-space signal to a denoising neural network model configured to: extract a plurality of sub-images from the received real-space corrected real-space signal; input the plurality of sub-images and a noise level map set for the frame of diffraction patterns to a multi-layer neural network; output a plurality of denoised sub-images from the multi-layer neural network, the plurality of denoised sub-images being identical in number and size to the plurality of sub-images; and integrate the plurality of denoised sub-images into a denoised real-space signal for next iteration.

[0031] In some embodiments, the extracting a plurality of sub-images from the received real-space corrected real-space signal includes: dividing an image of the real-space signal into a plurality of primary regions, each of the plurality of primary regions including a plurality of secondary regions, the plurality of secondary regions being identical in number to the plurality of sub-images; and extracting each sub-image in the plurality of sub-images as including a respective one of the secondary regions in the respective primary region.

[0032] Other features of the present disclosure, and their advantages, will become apparent from the following detailed description of exemplary embodiments of the present disclosure, with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0033] The foregoing and other features and advantages of the present disclosure will become apparent to those of ordinary skill in the art from the following detailed description of exemplary embodiments of the present disclosure, with reference to the accompanying drawings. The accompanying drawings form a part of the specification, and are included to further illustrate the principles of the present disclosure and to enable a person of ordinary skill in the art to make and use the present disclosure. In the drawings:

[0034] Figure 1 is a schematic block diagram illustrating a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0035] Figure 2 is a schematic diagram illustrating an encoding pattern used by a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0036] Figure 3 is a schematic diagram illustrating the principle of time-compressed coherent diffraction imaging implemented by a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0037] Figure 4 is a process schematic diagram illustrating the generation of a compressed encoding diffraction pattern using a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0038] Figure 5 is an apparatus diagram illustrating one example of a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0039] Figure 6 is an apparatus diagram illustrating another example of a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure;

[0040] Figure 7 is a process schematic diagram illustrating multi-frame reconstruction and phase retrieval of a compressed encoding diffraction pattern according to some embodiments of the present disclosure;

[0041] Figure 8 illustrates schematically the relationship between adjacent feature passing layers of a multi-frame reconstruction neural network model of Figure 7 ;

[0042] Figure 9 illustrates a schematic block diagram of a denoising neural network model of Figure 7 ;

[0043] Figure 10 illustrates a schematic process of extracting sub-images from real-space signals by a denoising neural network model of Figure 7 ;

[0044] Figure 11 illustrates results of photographing a target using a high-speed imaging microscopy apparatus according to some embodiments of the present disclosure and results of directly photographing the target using a high-frame-rate microscopy camera;

[0045] Figure 12 is a schematic flow diagram illustrating a high-speed imaging microscopy method according to some embodiments of the present disclosure.

[0046] Note that, in the following description of embodiments, the same drawing reference numerals are used in different drawings to denote the same or similar components. In some cases, like reference numerals and letters are used to denote like items, and once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0047] For ease of understanding, the positions, sizes, shapes, and ranges of the structures illustrated in the drawings, and the like, are sometimes schematically illustrated. Therefore, the present disclosure is not limited to the positions, sizes, shapes, and ranges disclosed in the drawings, and the like. DETAILED DESCRIPTION

[0048] Various exemplary embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. Note that the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in these embodiments are not limiting to the scope of the present disclosure unless otherwise specifically stated.

[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the scope of the disclosure, its application, or uses. That is, the structures and methods described herein are shown by way of illustration in the drawings and are merely exemplary of the structures and methods in the disclosure. However, those skilled in the art will recognize that the structures and methods described herein are merely exemplary of the structures and methods that can be employed, and that the scope of the disclosure is not limited to the structures and methods described herein. Furthermore, the drawings are not necessarily drawn to scale, with some features possibly being exaggerated to illustrate details that would otherwise be difficult to see.

[0050] Furthermore, techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, the techniques, methods, and apparatus should be considered part of the specification, to the extent that they are appropriate for use with the disclosure.

[0051] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative, and not as limiting. Thus, other examples of the exemplary embodiments can have different values.

[0052] In many important applications such as biomedical high-speed microscopic imaging, high-speed particle imaging, etc., it is necessary to image a micro target in high-speed motion. Since the higher the micro target is, the higher the stability requirement of the target for clear resolution of microscopic imaging, in high-speed microscopic imaging applications, the speed of the order of meters per second or centimeters per second can be considered as high speed for the micro target of the order of microns or nanometers. If the conventional microscopic imaging technology and high-speed imaging technology are combined to shoot the micro target in high-speed motion, both the expensive high-power microscope objective and the high-frame-rate camera are usually needed, and the micro target in high-speed motion needs to be strictly on the image plane of the microscope objective at all times, but such a system not only has high cost, but also any vibration can cause imaging blur.

[0053] To this end, the present disclosure proposes a high-speed microscopic imaging device and related method, which can restore a large segment of the motion of the target by a single exposure, and can achieve high temporal resolution and high spatial resolution with ordinary low-cost low-power lenses and low-frame-rate cameras. The high-speed microscopic imaging device and related method according to various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0054] Figure 1 A high-speed microscopic imaging device 100 according to some embodiments of the present disclosure is shown. It should be understood that there can be other components of the actual high-speed microscopic imaging device 100, and in order to avoid obscuring the points of the present disclosure, other components are not shown in the drawings and are not discussed herein.

[0055] The high-speed microscopic imaging device 100 can include a coherent light source 102 configured to irradiate a target 104 with coherent light. The coherent light source 102 includes but is not limited to, for example, a laser, etc. The coherent light source 102 can be configured to emit light of any suitable wavelength, including but not limited to, for example, near-infrared wavelengths, etc. The target 104 can be a micro target, can be a target in motion, especially in high-speed motion, and can be a micro target in motion, especially in high-speed motion. By irradiating the target with coherent light, the target can be imaged by coherent diffraction. The resolution of the coherent diffraction imaging technology is much higher than that of direct imaging, and is particularly suitable for imaging micro targets, which can convert small spatial domains to large frequency domains through Fourier acquisition. The smaller the target is, the larger the corresponding spatial frequency spectrum distribution range is, the lower the required sampling accuracy is, and the smaller the information loss is when performing time-domain compression. Moreover, the partial loss of spatial frequency will only cause a slight reduction in image resolution, without affecting the integrity of the image. Thus, the coherent diffraction imaging technology helps the high-speed microscopic imaging device 100 to achieve high spatial resolution.

[0056] In some embodiments, the diffraction pattern of the target 104 can be formed by light reflected by the target 104 under coherent light. In some embodiments, the diffraction pattern of the target 104 can be formed by light transmitted by the target 104 under coherent light. The high-speed microscopic imaging apparatus 100 can be configured to perform transmissive coherent diffraction imaging or reflective coherent diffraction imaging according to specific application scenarios (e.g., according to the properties (transmissivity, reflectivity, etc.) of the target 104), which will be described later in conjunction with Figure 5 and Figure 6 respectively describe a transmissive coherent diffraction imaging example and a reflective coherent diffraction imaging example of the high-speed microscopic imaging apparatus 100.

[0057] It should also be understood that when microscopic imaging is performed with the high-speed microscopic imaging apparatus 100 according to the present disclosure, the target 104 is not limited to be a subject with a small overall size, and can also be a small local area of a subject with a large overall size.

[0058] The high-speed microscopic imaging apparatus 100 can also include a spatial light modulator 106. The spatial light modulator 106 can be configured to receive the diffraction pattern of the target 104 illuminated by coherent light, and encode the diffraction pattern of the target 104 according to the encoding pattern of the spatial light modulator 106 to generate an encoded diffraction pattern.

[0059] The spatial light modulator 106 may, for example, perform spatial modulation at the pixel level on the diffraction pattern of the target 104. In some embodiments, the pixels of the encoding pattern of the spatial light modulator 106 are binary encoded to allow or not allow the received light to reach the imaging sensor 108 (to be described later) and the pixel size is larger than the Bragg spacing of the diffraction pattern of the target 104. For example, refer to Figure 2 “1” (white small block) can represent allowing the received light to reach the imaging sensor 108, and “0” (black small block) can represent not allowing the received light to reach the imaging sensor 108, so the encoding pattern of the spatial light modulator 106 can be represented by a binary matrix. Further reference can be made to Figure 4For example, if the diffraction pattern obtained by illuminating the target 104 with coherent light at time t1 is the lowermost one in the second column, and the encoding pattern of the spatial light modulator 106 is the lowermost one in the first column (the white region allows the received light to reach the imaging sensor 108, and the black region does not allow the received light to reach the imaging sensor 108), then the encoded diffraction pattern obtained by the spatial light modulator 106 encoding the diffraction pattern at time t1 is shown as the lowermost one in the third column. In other embodiments, the pixels of the encoding pattern of the spatial light modulator 106 can also be alternatively binary-encoded to allow one of two different preset percentages of the received light to reach the imaging sensor 108, for example, “0” (black block) can represent allowing 20% of the received light to reach the imaging sensor 108, and “1” (white block) can represent allowing 80% of the received light to reach the imaging sensor 108. In other embodiments, the pixels of the encoding pattern of the spatial light modulator 106 can also be multi-ary-encoded, for example, can be multi-ary-encoded to allow one of multiple different preset percentages of the received light to reach the imaging sensor 108.

[0060] In some embodiments, the encoding pattern of the spatial light modulator 106 can adopt a random encoding manner. In addition, when the field of view size is fixed, the low-frequency part of each frame of diffraction pattern can have higher similarity, and therefore in some embodiments, the encoding pattern of the spatial light modulator 106 can adopt an encoding manner that causes the low-frequency part of the diffraction pattern of the target 104 to have more information loss than the high-frequency part of the diffraction pattern of the target 104. In this way, both over-acquisition of the low-frequency part can be avoided, and the high-frequency part can have a higher signal-to-noise ratio than single-frame acquisition. In some embodiments, the encoding pattern of the spatial light modulator 106 can adopt an encoding manner that causes the high-intensity part of the diffraction pattern of the target 104 to have more information loss than the low-intensity part of the diffraction pattern of the target 104. For example, referring to FIG. 2, the encoding pattern of the spatial light modulator 106 can be set to have a higher probability of appearing a black block at a position corresponding to a brighter position in the diffraction pattern of the target 104. In fact, the encoding pattern can be set in any desired encoding manner. Figure 4 For example, if the diffraction pattern obtained by illuminating the target 104 with coherent light at time t1 is the lowermost one in the second column, and the encoding pattern of the spatial light modulator 106 is the lowermost one in the first column (the white region allows the received light to reach the imaging sensor 108, and the black region does not allow the received light to reach the imaging sensor 108), then the encoded diffraction pattern obtained by the spatial light modulator 106 encoding the diffraction pattern at time t1 is shown as the lowermost one in the third column. In other embodiments, the pixels of the encoding pattern of the spatial light modulator 106 can also be alternatively binary-encoded to allow one of two different preset percentages of the received light to reach the imaging sensor 108, for example, “0” (black block) can represent allowing 20% of the received light to reach the imaging sensor 108, and “1” (white block) can represent allowing 80% of the received light to reach the imaging sensor 108. In other embodiments, the pixels of the encoding pattern of the spatial light modulator 106 can also be multi-ary-encoded, for example, can be multi-ary-encoded to allow one of multiple different preset percentages of the received light to reach the imaging sensor 108.

[0061] The spatial light modulator 106 can have a variety of different implementations. In some embodiments, the spatial light modulator 106 can be a digital micromirror array (DMD), which can include a plurality of micromirrors, each of which can have a rotation angle that can be digitally configured. For example, if the encoding pattern is to be binary encoded as allowing the received light to reach the imaging sensor 108 or not allowing the received light to reach the imaging sensor 108, the rotation angle of each micromirror of the digital micromirror array can be digitally configured to reflect the light received by that micromirror to be received by the imaging sensor 108 or not to be received by the imaging sensor 108. Alternatively, the rotation angle of each micromirror of the digital micromirror array can also be digitally configured to cause a different pre-set percentage of the reflected light of the light received by that micromirror to be received by the imaging sensor 108. The digital micromirror array can provide different encoding patterns for the spatial light modulator 106 by digitally configuring the rotation angle of the individual micromirrors, each of which acts as a pixel of the encoding pattern.

[0062] In some embodiments, the spatial light modulator 106 can include a plurality of mask plates and switching components for switching the plurality of mask plates. For example, if the encoding pattern is to be binary encoded as allowing the received light to reach the imaging sensor 108 or not allowing the received light to reach the imaging sensor 108, each mask plate of the plurality of mask plates can include a plurality of regions arranged as an array and each configured to allow light to pass therethrough to be received by the imaging sensor 108 or not to pass therethrough to be received by the imaging sensor 108. Alternatively, each region can also be configured to allow a different pre-set percentage of the received light to reach the imaging sensor 108, which can be achieved, for example, by materials of different transmittance. Different mask plates of the plurality of mask plates can provide different encoding patterns for the spatial light modulator 106, each region of a mask plate acting as a pixel of the encoding pattern.

[0063] In some embodiments, the spatial light modulator 106 can include a mask plate having a plurality of regions arranged as an array and a moving component for moving the mask plate so that the diffraction pattern of the target 104 is received by different regions of the plurality of regions of the mask plate. For example, if the encoding pattern is to be binary encoded to allow the received light to reach the imaging sensor 108 or not to allow the received light to reach the imaging sensor 108, each region of the plurality of regions of the mask plate can include a plurality of sub-regions arranged as an array and each configured to allow light to pass therethrough to be received by the imaging sensor 108 or not to allow light to pass therethrough to be received by the imaging sensor 108. Alternatively, each sub-region can also be configured to allow a different preset percentage of the received light to reach the imaging sensor 108, which can be achieved, for example, by materials of different transmittance. Different regions of the plurality of regions of the mask plate can provide different encoding patterns of the spatial light modulator 106, each sub-region of a region acting as a pixel of the encoding pattern.

[0064] Referring back to Figure 1 , the high-speed microscopic imaging apparatus 100 can further include an imaging sensor 108. The imaging sensor 108 can be positioned, for example, in optical conjugation with the spatial light modulator 106 so as to image the encoded diffraction pattern of the spatial light modulator 106 onto the imaging sensor 108. The imaging sensor 108 can be configured to receive multiple frames of the encoded diffraction pattern from the spatial light modulator 106 within an exposure time of the imaging sensor 108 to generate a single compressed encoded diffraction pattern of the target 104. The spatial light modulator 106 can be configured to switch the encoding pattern multiple times within the exposure time of the imaging sensor 108 to generate the multiple frames of the encoded diffraction pattern by encoding multiple frames of the diffraction pattern of the target 104 at different times within the exposure time of the imaging sensor 108 according to respective different encoding patterns. That is, the spatial light modulator 106 can periodically switch different encoding patterns at a preset switching speed within the exposure time of the imaging sensor 108. Each frame of the multiple frames of the encoded diffraction pattern is encoded according to the encoding pattern of the spatial light modulator 106 at the time of the respective one of the multiple frames of the diffraction pattern.

[0065] For example, referring to Figure 4 , the target 104 can be in a state of motion within the exposure time of the imaging sensor 108, but the high-speed microscopic imaging apparatus 100 of the present disclosure does not obtain an image of the target 104 by direct imaging, but obtains a total of T frames of diffraction patterns of the target 104 at different times tl, t2,..., t T within the exposure time of the imaging sensor 108 by coherent diffraction imaging (as Figure 4(As shown in the second column). Meanwhile, the spatial light modulator 106 can switch different coding patterns multiple times at a preset switching speed during the exposure time of the imaging sensor 108, thereby using T different coding patterns (such as...). Figure 4 The first column shows the corresponding T-frame diffraction pattern (as shown in the first column). Figure 4 The second column shows) modulated into a T-frame encoded diffraction pattern (as shown in the second column). Figure 4 (As shown in the third column). The imaging sensor 108 integrates the received light within a single exposure time, thereby compressing the T-frame encoded diffraction pattern into a single compressed encoded diffraction pattern (as shown in the third column). Figure 4 (As shown in the fourth column). Therefore, the single exposure result of the imaging sensor 108 actually contains information about the T-frame diffraction pattern of the target 104, and through the multi-frame reconstruction and phase retrieval processing described later, the target 104 at t1, t2, ..., t3 frames can be recovered from the single exposure result of the imaging sensor 108. T The original image at the location. The diffraction pattern of the target 104 is encoded using T different coded patterns during the exposure time of the imaging sensor 108 by the spatial light modulator 106, making the frame rate of the high-speed microscopy imaging device 100 equivalent to T times the frame rate of the imaging sensor 108. This significantly reduces the frame rate requirement of the imaging sensor 108. The spatial light modulator 106 effectively improves the temporal resolution of the high-speed microscopy imaging device 100. The frame rate of the high-speed microscopy imaging device 100 can be easily adjusted by adjusting the switching speed of the coded patterns of the spatial light modulator 106.

[0066] Since the spatial light modulator 106 can switch different coding patterns multiple times within a single exposure time of the imaging sensor 108 to generate multiple frames of diffraction patterns corresponding to the target 104 at different times within the exposure time of the imaging sensor 108 and the coding pattern of the spatial light modulator 106 at the corresponding time, multi-frame compressed coding acquisition of the target's diffraction pattern can be achieved by controlling the exposure time of the imaging sensor 108 and the switching speed of the coding pattern of the spatial light modulator 106.

[0067] This disclosure refers to the high-speed microscopic imaging technique achieved by the coherent light source 102 and spatial light modulator 106 as time-compressed coherent diffraction imaging. It modulates the coherent diffraction imaging results in the spatial domain and then compresses and acquires the modulated results in the time domain. The following is in conjunction with... Figure 3 This will illustrate the principle of time-compressed coherent diffraction imaging of this disclosure.

[0068] like Figure 3As shown, a moving object O(x, t) is illuminated by coherent light U0(x), where x indicates the spatial coordinate and t indicates the time coordinate. It can be understood that if a stationary object is considered, the dimension t can be omitted. According to Fraunhofer diffraction, the time-varying dynamic light field U in the Fraunhofer region... d (x, t) can be represented by the following formula.

[0069]

[0070] Where P(x) indicates the imaging field of view FoV, The Fourier transform is given, where λ is the wavelength, z is the spatial distance of the light field propagation, and x′ is the spatial coordinate on the plane of the original object. Considering plane wave illumination, i.e., U0(x′) is a constant, then U... d The light field U of (x, t) after dynamic modulation M(x, t) (the coded pattern of the spatial light modulator) by the spatial light modulator. t (x, t) can be represented by the following formula.

[0071] U t (x, t) = U d (x, t)M(x, t) (2)

[0072] Therefore, the intensity of the compressed signal I(x) detected by the imaging sensor can be represented by the following formula.

[0073]

[0074] Where Δt is the exposure time (integration time) of the imaging sensor.

[0075] If we write equation (3) in discrete form and consider measurement noise, then

[0076]

[0077] Among them Y, U d (:,:,t) And they are I(x) and U respectively. d The discrete forms of M(x,t) and M(x,t) are given, where T represents the number of diffraction patterns (frequency domain frames), G represents measurement noise, ⊙ represents the Hadamard product, and W and H are the width and height of each frame. Note that since M(x,t) is designed to be binary encoded, i.e., composed of {0,1}, where "0" indicates that light is not allowed to be received by the imaging sensor and "1" indicates that light is allowed to be received by the imaging sensor, the detected compressed signal intensity can be modeled as:

[0078]

[0079] Let U' = vec(M(:, :, t)) d ( :, :, t ) = || U d ( :, :, t ) 2 and vectorize Y, U', G as d Equation (5) is then converted to the following equation.

[0080] y = Φu + g (6)

[0081] where denotes a sensing matrix, which is a concatenation of T diagonal matrices, i.e.

[0082] Φ = [Φ1,..., Φ T ] (7)

[0083] where Φ t = Diag(vec(M(:, :, t))) is a diagonal matrix whose diagonal elements are composed of vec(M(:, :, t)(a vectorized representation of M(:, :, t)).

[0084] The above procedure clearly demonstrates how to encode and compress collect multiple frames of spatial spectrum (diffraction pattern) of a moving target. It can be further understood from the above procedure that the compressed encoded diffraction pattern detected by the imaging sensor 108 is an integration of the encoded diffraction pattern over the time domain, which is obtained by modulating the diffraction pattern with the encoded pattern, as shown in Figure 4

[0085] According to Figure 3 , the spatial light modulator 106 can be positioned in the far field or Fraunhofer diffraction region of the target 104 so that the spatial light modulator 106 receives and encodes the diffraction pattern of the target 104. In some embodiments, the spatial light modulator 106 can also be positioned on the Fourier plane of the target 104.

[0086] Figure 5 A non-limiting example apparatus 200 of performing transmission coherent diffraction imaging according to the high-speed microscopic imaging apparatus 100 of the present disclosure is described. Like the high-speed microscopic imaging apparatus 100, the apparatus 200 includes a coherent light source 202 to illuminate a target 204 with coherent light, a spatial light modulator 206 to encode a diffraction pattern of the target 204, and an imaging sensor 208 to image a surface of the spatial light modulator 206. In some embodiments, the apparatus 200 can also include an aperture stop disposed between the target 204 and the spatial light modulator 206 and in close proximity to the target 204 to define an imaging field of view. For example, in Figure 5 ​​In the example of FIG. 2, an aperture stop 205 is placed immediately behind the target 204. The aperture stop can cause the spatial spectrum of the target containing the information of interest to be collected, thereby reducing the complexity of the diffraction pattern so that the sampling accuracy can be lower than the pixel size of the spatial light modulator. The aperture stop is not necessarily required and can be set as appropriate. In some embodiments, the imaging field of view can also be defined by adjusting the beam diameter (e.g., via one or more lenses).

[0087] In some embodiments, the apparatus 200 can further include a first lens group disposed between the coherent light source 202 and the target 204. The first lens group can include one or more lenses. The first lens group can be configured to collimate the coherent light emitted by the coherent light source 202 and / or adjust the beam diameter of the coherent light so that the coherent light can fully illuminate the target 204. In some embodiments, the first lens group can be configured to form an optical 4F system. For example, in the example of FIG. 2, the first lens group 212 includes lenses AL1 and AL2 that are configured to form an optical 4F system that expands the coherent light beam. Figure 5 In the example of FIG. 2, the first lens group 212 includes lenses AL1 and AL2 that are configured to form an optical 4F system that expands the coherent light beam.

[0088] In some embodiments, the apparatus 200 can further include a Fourier lens AL3 disposed between the target 204 and the spatial light modulator 206. The target 204 can be positioned on a front focal plane of the Fourier lens AL3, which can be configured to Fourier transform the diffraction pattern of the target. The Fourier lens AL3 can enable accurate diffraction results in the near field. In some embodiments, the spatial light modulator 206 can be positioned on a back focal plane (i.e., Fourier plane FP) of the Fourier lens AL3. In some embodiments, the apparatus 200 can further include a second lens group disposed between the back focal plane FP of the Fourier lens AL3 and the spatial light modulator 206. The second lens group can include one or more lenses. The second lens group can be configured to cause the Fourier transform spectrum of the diffraction pattern of the target 204 to be imaged within the encoding region of the spatial light modulator 206 after being Fourier transformed by the Fourier lens AL3. The encoding region of the spatial light modulator 206 is the region of the spatial light modulator 206 that has the encoding pattern. If the target is small, the Fourier transform spectrum of the diffraction pattern of the target at the FP can be too large, in which case the second lens group can be used to downscale the Fourier transform spectrum to be imaged within the encoding region of the spatial light modulator 206. If the target is large, the Fourier transform spectrum of the diffraction pattern of the target at the FP can be small, in which case the second lens group can be used to upscale the Fourier transform spectrum to be imaged within the encoding region of the spatial light modulator 206. In other words, the second lens group can be configured to cause the Fourier transform spectrum of the diffraction pattern of the target to be imaged within the encoding region of the spatial light modulator 206 appropriately, both to enable the spatial light modulator 206 to encode the entire spectrum and to make the spectrum as large as possible. In some embodiments, the second lens group can be configured to form an optical 4F system. For example, in the example of FIG. 2B, the second lens group 216 includes lenses AL4 and AL5, which can be configured to form an optical 4F system to match the Fourier transform spectrum of the diffraction pattern of the target to the spatial light modulator 206. Figure 5

[0089] In some embodiments, the apparatus 200 can further include a third lens group disposed between the spatial light modulator 206 and the imaging sensor 208. The third lens group can include one or more lenses. The third lens group can be configured to cause the encoded diffraction pattern generated by the spatial light modulator 206 to be imaged within the sensing region of the imaging sensor 208. In some embodiments, the third lens group can be configured to form an optical 4F system. For example, in the example of FIG. 2B, the third lens group 218 includes lenses AL6 and AL7, which can be configured to form an optical 4F system to match the encoded diffraction pattern generated by the spatial light modulator 206 to the imaging sensor 208. Figure 5 ​In the example of FIG. 2, the third lens group 218 includes a lens AL6 and an objective lens OL, which can be configured to form an optical 4F system to match the encoded diffraction pattern generated by the spatial light modulator 206 with the imaging sensor 208. Since the apparatus 200 utilizes coherent diffraction imaging, the objective lens OL does not need to have a high magnification, e.g., its magnification can be less than ten times, or less than five times, or can be four times.

[0090] Figure 6 A non-limiting example apparatus 300 that performs reflective coherent diffraction imaging is described in accordance with the high-speed microscopic imaging apparatus 100 of the present disclosure. Like the high-speed microscopic imaging apparatus 100, the apparatus 300 includes a coherent light source 302 that illuminates a target 304 with coherent light, a spatial light modulator 306 that encodes a diffraction pattern of the target 304, and an imaging sensor 308 that images the surface of the spatial light modulator 306. In the example of FIG. 3, the target 304 is a reflective target, e.g., a mirror or a metallic surface, and the spatial light modulator 306 is a reflective spatial light modulator, e.g., a digital micromirror device (DMD). The apparatus 300 can be configured to perform reflective coherent diffraction imaging in a similar manner as the apparatus 200 of FIG. 2, e.g., the apparatus 300 can include a first lens group 312, a second lens group 314, and a third lens group 318, which can be configured to form an optical 4F system to match the encoded diffraction pattern generated by the spatial light modulator 306 with the imaging sensor 308. Figure 5 In the example of FIG. 3, the aperture stop 305 is placed in front of the target 304 (i.e., on the side facing the spatial light modulator 306), the first lens group 312 includes a lens AL1, the target 304 and the spatial light modulator 306 are located on the front focal plane and the back focal plane of a Fourier lens AL2, respectively, and the third lens group 318 includes a lens AL3 and an objective lens OL between the spatial light modulator 306 and the imaging sensor 308.

[0091] It can be appreciated that the high-speed microscopic imaging apparatus in accordance with the present disclosure can have additional or alternative optical components other than those shown above to optimize the optical path design.

[0092] After the imaging sensor 108, 208, 308 generates the compressed encoded diffraction pattern with a single exposure, the compressed encoded diffraction pattern can be processed to recover the multi-frame motion image of the target 104, 204, 304 during the exposure. In some embodiments, the imaging sensor can be in communication with a processor and configured to transmit the compressed encoded diffraction pattern to the processor. The compressed encoded diffraction pattern, although containing rich temporal and spatial information, has a small data size, which facilitates saving storage space and transmission bandwidth in data storage and transmission. The processor can be configured to decode the compressed encoded diffraction pattern to obtain the multi-frame diffraction patterns of the target at different times according to the respective encoded patterns of the spatial light modulator and their temporal order within the exposure time of the imaging sensor, and to perform phase retrieval on the multi-frame diffraction patterns to reconstruct the multi-frame images of the target at different times. Such a processor can be included in the high-speed microscopic imaging apparatus or provided by any computing device located outside the apparatus. The image recovery method that can be performed by the processor in accordance with some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0093] Neural network models can be used to recover multi-frame diffraction patterns of a target at different times from compressed-coded diffraction patterns. In some embodiments, the processor can be configured to decode the compressed-coded diffraction pattern using a multi-frame reconstruction neural network model with multiple stages to obtain multi-frame diffraction patterns of the target at different times (e.g., Figure 4 (As shown in the second column). For example, as shown in the second column. Figure 7 As shown, a multi-frame reconstruction neural network model may include a first stage and a second stage, but this is merely exemplary and not limiting; it may also include more than two stages. Each stage of the multi-frame reconstruction neural network model may include a projection module, a feature extraction module, and a feature integration module. The projection module may be configured to map the input of this stage from the compressed domain to the signal domain to obtain multiple signal maps, the number of which is the same as the number of multi-frame diffraction patterns corresponding to the compressed-coded diffraction pattern (e.g., in conjunction with a reference). Figure 4 (The number is T). In some examples, the projection module can be configured to perform analytical solving. However, the results of analytical solving may be coarse rather than optimal, so the output of the projection module needs to be fed into the subsequent feature extraction and feature integration modules for further processing. In some examples, the feature extraction and feature integration modules may each include multiple 3D convolutional layers. As a non-limiting example, the feature extraction module may include four 3D convolutional layers with kernel sizes of 5×5×5, 3×3×3, 1×1×1, and 3×3×3, and the feature integration module may include four 3D convolutional layers with kernel sizes of 3×3×3, 3×3×3, 1×1×1, and 3×3×3, respectively.

[0094] Multi-frame reconstruction neural network models can, for example, model the reconstruction process as the following optimization problem.

[0095]

[0096] Where R(u) represents the regularization part, and τ represents the regularization parameter. This multi-frame reconstruction neural network model combines the advantages of optimization-based and learning-based algorithms, making it flexible for different application scenarios and capable of quickly obtaining high-quality reconstruction results. Since y is known (i.e., the compressed coded diffraction pattern of the imaging sensor) and Φ is also known (i.e., the various coded patterns set), solving the above optimization problem can efficiently restore the compressed image to the multi-frame image before compression. Therefore, at each stage, the model can perform the following two steps:

[0097] u (j) =v (j-1) +Φ T (ΦΦ T ) -1 (y-Φv (j-1) (9)

[0098] v (j) = Network(u (j) ) (10)

[0099] where v (j) represents the estimation of the desired signal at the jth stage (this step can correspond to the operation of the projection module mapping from the compressed domain to the signal domain to obtain a plurality of signal maps), the network model backbone of each stage can have an encoder-decoder structure.

[0100] In some embodiments, each stage of the multi-frame reconstruction neural network model can further comprise a plurality of feature transfer layers s i (i is a positive integer greater than 2) for transferring the extracted features, which are arranged between the feature extraction module and the feature integration module. i Each feature transfer layer s comprises a first reversible block and a second reversible block The output channels of the feature extraction module can be divided into two parts to be coupled to the first reversible block 1 and the second reversible block of the first feature transfer layer s i nearest to the feature extraction module respectively, and the following relationship is satisfied between adjacent feature transfer layers s i+1 and s Figure 8 :

[0101]

[0102]

[0103] wherein and are the same activation function, which can be a ReLU function for example. These reversible blocks can be used to save the use of GPU memory, especially during the training process of the neural network model, because the backward path can be directly used with the known and are calculated by:

[0104]

[0105]

[0106] Therefore, it can no longer be necessary to save the activation values of each activation layer.

[0107] It is appreciated that the above multi-frame reconstruction neural network model is merely exemplary and not limiting, and other suitable methods now known or later developed other than neural network models can be utilized to reconstruct the multi-frame diffraction patterns of the target at different times from the compressed encoded diffraction patterns.

[0108] Phase retrieval can be performed on the reconstructed multi-frame diffraction patterns using any suitable process to obtain the multi-frame original images of the target. Existing algorithms for phase retrieval include, for example, Gerchbrg-Saxton (G-S) algorithm, Hybrid Input-Output (HIO) algorithm, Error Reduction (ER), Young-Gu (YG) algorithm, and other iterative projection algorithms. For example, refer back to Figure 7 , Figure 7 A process of phase retrieval on the multi-frame diffraction patterns reconstructed by the multi-frame reconstruction neural network model using classical HIO process is shown. F i and f are the Fourier space signal and the real space signal in the i-th iteration, respectively, k and n represent the corresponding Fourier space coordinate and real space coordinate, respectively, is the amplitude of the frequency domain frame (diffraction pattern). The following process can be performed separately for each frame of diffraction pattern. F i [k] can be randomly generated Gaussian distribution, in the i-th iteration, F is converted to and inverse Fourier transform is performed on

[0109] to obtain the real space signal f i [n]. The real space correction can then be performed by the following formula.

[0110] The above real space correction can be used to check whether the coordinates of the update variable belong to the support set S, and β provides feedback to the algorithm for the part beyond S. The real space signal f i+1 [n] after real space correction can enter the i+1-th iteration, and is converted to F i+1 [k] by Fourier transform, and then the above operation is repeated. Through repeated iterations of Fourier transform and inverse Fourier transform, the result of phase retrieval gradually approaches the true image.

[0111] In order to better cope with various noise levels that may exist in actual measurement, the present disclosure also performs phase retrieval on each frame of diffraction pattern in the multi-frame diffraction pattern by combining phase retrieval process with denoising process to reconstruct a frame of image of the target corresponding to the frame of diffraction pattern in some embodiments. Figure 7As an example of HIO processing, a denoising module can be introduced in each iteration of HIO processing, which can be based on a neural network model. Specifically, in some embodiments, the processor can be configured to, in each iteration of HIO processing performed for each frame of diffraction patterns in the plurality of frames of diffraction patterns, input a real-space corrected real-space signal into a denoising neural network model for denoising and use the denoised real-space signal for the next iteration, the denoising neural network model can be configured to: extract a plurality of sub-images from the received real-space corrected real-space signal; input the plurality of sub-images and a noise level map set for the frame of diffraction patterns into a multi-layer neural network; output a plurality of denoised sub-images from the multi-layer neural network, the plurality of denoised sub-images being identical in number and size to the plurality of sub-images; and integrate the plurality of denoised sub-images into a denoised real-space signal. Figure 9 An example structure of a denoising neural network model is shown according to some embodiments of the present disclosure. Figure 9 The convolutional layers in can be 2D convolutional layers, which can employ a convolution kernel size of 3x3, for example. In some of the convolutional layers, a batch normalization layer BN and / or an activation function layer (e.g., a ReLU layer) can also be provided. In some embodiments, extracting a plurality of sub-images from the received real-space corrected real-space signal can include: dividing an image of the real-space signal into a plurality of primary regions, each of the plurality of primary regions including a plurality of secondary regions, the plurality of secondary regions being identical in number to the plurality of sub-images; and extracting each of the plurality of sub-images as including a respective one of the secondary regions in the respective primary region. For example, referring to Figure 10 Assuming the real-space corrected real-space signal f i+1 [n] is divided into 16 primary regions (indicated by solid lines), each of which includes A, B, C, D four secondary regions (indicated by dashed lines), then the real-space corrected real-space signal f i+1[n] Four sub-images can be extracted, where the first sub-image includes all of the secondary regions A, the second sub-image includes all of the secondary regions B, the third sub-image includes all of the secondary regions C, and the fourth sub-image includes all of the secondary regions D. This interleaved sub-image extraction method can reduce the resolution relative to the original image compared to a direct division method, thereby effectively reducing the network parameters and increasing the receptive field under the condition of maintaining the accuracy of the results, so that the efficiency of the denoising neural network model is improved. The noise level map can be set for the frame diffraction pattern, which can be in a Gaussian form, for example. For example, in the application process, if the diffraction pattern appears to have a high quality, a lower level of noise level map can be set, and if the diffraction pattern appears to have a low quality, a higher level of noise level map can be set. In the training process, various levels of noise level maps from low to high can be input into the model for training in order to enhance the adaptability of the model to various noise levels. After the sub-images are extracted and the noise level map is set, the sub-images and the noise level map can be concatenated together and input into the subsequent convolutional layer. Although the above is described by way of example of adding denoising processing in HIO processing, this is not limiting, and the denoising processing disclosed herein can be applied to any phase recovery processing.

[0112] Figure 11 Results of photographing a target using a high-speed imaging microscope device according to some embodiments of the present disclosure and results of directly photographing the target using a high-frame-rate microscope camera are shown. The target is the tiny word "westlake" (with a stroke width of only 20 microns) inscribed on a stainless steel plate that is moving at a high speed (at a speed of 3 meters per second). Part (a) shows the compressed coded diffraction pattern of the target, part (b) shows eight target images directly photographed one by one using a conventional high-magnification high-frame-rate camera, part (c) shows eight target images obtained by multi-frame reconstruction of the compressed coded diffraction pattern of (a) according to the present disclosure and phase recovery using classical HIO processing, and part (d) shows eight target images obtained by multi-frame reconstruction of the compressed coded diffraction pattern of (a) according to the present disclosure and phase recovery using HIO processing combined with a denoising neural network model. Comparing (b), (c), and (d), it can be seen that even without using expensive high-magnification lenses and high-frame-rate cameras, multi-frame motion images of the target can be restored by time-compressed coherent diffraction imaging according to the present disclosure combined with subsequent processing, and the recovery quality is further improved after combining the denoising processing of the present disclosure. In addition, from (a), (c), and (d), it can be seen that eight frames of motion images are recovered from a single exposure of compressed sampling, different parts of the target are observed entering and leaving the field of view over time, and it is shown that the techniques of the present disclosure can recover the complete motion process of the target from a single exposure.

[0113] The present disclosure also provides, in another aspect, a high-speed microscopic imaging method, which can include: illuminating a target with coherent light; receiving and spatially modulating a diffraction pattern of the target illuminated by the coherent light, the spatial modulation including encoding the diffraction pattern of the target according to an encoding pattern to generate an encoded diffraction pattern; and compressively imaging a plurality of frames of the encoded diffraction pattern generated by the spatial modulation over a period of time to generate a single compressed encoded diffraction pattern of the target, wherein the spatial modulation includes switching the encoding pattern a plurality of times over the period of time to encode a plurality of frames of the diffraction pattern of the target at different times over the period of time according to respective different encoding patterns to generate the plurality of frames of the encoded diffraction pattern.

[0114] For example, with reference to Figure 12 The method 400 includes: illuminating a target with coherent light at step S402, which can be a (high-speed) moving and / or a tiny target; receiving and spatially modulating a diffraction pattern of the target at step S404, the spatial modulation including encoding the diffraction pattern of the target according to an encoding pattern to generate an encoded diffraction pattern, wherein the encoding pattern is periodically switched to different encoding patterns; compressively imaging a plurality of frames of the encoded diffraction pattern generated by encoding the diffraction pattern of the target at different times over a period of time with different encoding patterns to generate a single compressed encoded diffraction pattern of the target at step S406. For a faster target, the frame rate of the imaging can be increased by increasing the number of encoding patterns used over the period of time (i.e., speeding up the switching of the encoding pattern) to accommodate the high-speed movement of the target. Various embodiments of the method 400 can be similar to the aforementioned apparatus embodiments, which are not repeated here.

[0115] The words "left," "right," "front," "back," "top," "bottom," "over," "under," "upper," "lower," and the like in the description and the claims, if any, are used for descriptive purposes and not necessarily for describing permanent relative positions. It is to be understood that the words so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in other orientations than those illustrated or otherwise described herein. For example, if the apparatus in the figures is turned over, features described as above other features would then be described as below the other features, unless stated otherwise. The apparatus can also be oriented in other ways (rotated 90 degrees or in other orientations), and the relative spatial relationships would be correspondingly interpreted.

[0116] In the description and claims, the terms "on", "attached" to, "connected" to, "coupled" to, "coupled with", or "in contact" with relate to an element in direct physical contact with the other element, or there can be one or more intermediate elements between them. In contrast, the terms "directly on", "directly attached" to, "directly connected" to, "directly coupled" to, "directly coupled with", or "directly in contact" with mean that there is no intermediate element between the elements. In the description and claims, an element arranged "adjacent" to another element can mean that the element has a portion that overlaps the adjacent element or a portion that is above or below the adjacent element.

[0117] As used herein, the word "exemplary" means "serving as an example, instance, or illustration," and not "preferred" or "advantageous over other implementations." Furthermore, the disclosure is not to be limited to any expressed or implied theory of operation by examples described herein.

[0118] As used herein, the word "substantially" means including any minor variations as a result of design, manufacturing, and / or other factors that can cause variations in actual implementation. The word "substantially" also allows for differences that are within normal manufacturing tolerances and / or other factors that can cause minor variations.

[0119] Additionally, the terms "first", "second", and other such numerical terms can be used herein, merely for purposes of reference and thus do not imply a sequence or order of one step over another, unless it is clearly indicated by the context.

[0120] It is also to be understood that the phraseology "comprising", "including", "containing", "consisting", "consisting essentially of", and the like, when used in the present specification, specifies the presence of stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0121] In the present disclosure, the term "providing" is used in a broad sense and encompasses all means of obtaining an object, and thus "providing an object" includes, but is not limited to, "purchasing", "preparing / manufacturing", "arranging / setting", "installing / fitting", and / or "ordering" the object, etc.

[0122] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0123] Those skilled in the art will realize that the boundaries between the above described operations merely illustrative. The multiple operations can be combined into a single operation, a single operation can be distributed in additional operations and operations can be executed at least partially overlapping in time. Moreover, alternative embodiments can include multiple instances of a particular operation and the order of operations can be altered in other various embodiments. However, other modifications, variations and alternatives are also possible. The aspects and elements of all such embodiments can be combined in any manner and / or combination with other aspects or elements of other embodiments, as would be understood by one of ordinary skill in the art to provide further embodiments. The description and drawings are accordingly to be regarded in illustrative rather than restrictive sense.

[0124] While certain specific embodiments of the present disclosure have been described in detail above, it should be appreciated that the foregoing examples are illustrative only and are not intended to limit the scope of the present disclosure. The various embodiments disclosed herein can be combined in any manner and / or combination with other aspects or elements of other embodiments, as would be understood by one of ordinary skill in the art to provide further embodiments. The scope of the present disclosure is defined by the appended claims.

Claims

1. A high-speed microscopic imaging apparatus, comprising: a coherent light source configured to illuminate a target with coherent light to perform coherent diffraction imaging of the target; a spatial light modulator configured to receive a diffraction pattern of the target illuminated by the coherent light and encode the diffraction pattern of the target according to an encoding pattern of the spatial light modulator to generate an encoded diffraction pattern; and an imaging sensor configured to receive multiple frames of the encoded diffraction pattern from the spatial light modulator within an exposure time of the imaging sensor to generate a single compressed encoded diffraction pattern of the target, wherein the spatial light modulator is configured to switch the encoding pattern multiple times within the exposure time of the imaging sensor to encode multiple frames of the diffraction pattern of the target at different times within the exposure time of the imaging sensor according to respective different encoding patterns to generate the multiple frames of the encoded diffraction pattern, and wherein the high-speed microscopic imaging apparatus further comprises a Fourier lens disposed between the target and the spatial light modulator, wherein the target is positioned on a front focal plane of the Fourier lens, the Fourier lens is configured to perform a Fourier transform of the diffraction pattern of the target to form a Fourier-transformed spectrum of the diffraction pattern of the target in a near field of the target, and the Fourier-transformed spectrum of the diffraction pattern of the target is received by the spatial light modulator to generate the encoded diffraction pattern.

2. The high-speed microscopic imaging apparatus of claim 1, wherein: the diffraction pattern of the target is formed by light reflected by the target through the coherent light; or the diffraction pattern of the target is formed by light transmitted by the target through the coherent light.

3. The high-speed microscopic imaging device of claim 1, wherein, the spatial light modulator is positioned in a far field or a Fourier plane of the target.

4. The high-speed microscopic imaging device of claim 1, wherein, the spatial light modulator and the imaging sensor are positioned to be optically conjugated to each other.

5. The high-speed microscopic imaging apparatus of claim 1, further comprising a first lens group disposed between the coherent light source and the target, the first lens group is configured to collimate the coherent light and / or adjust a beam diameter of the coherent light such that the coherent light fully illuminates the target.

6. The high-speed microscopic imaging device of claim 1, wherein, the spatial light modulator is positioned on a back focal plane of the Fourier lens.

7. The high-speed microscopic imaging apparatus of claim 1, further comprising a second lens group disposed between a back focal plane of the Fourier lens and the spatial light modulator, wherein the second lens group is configured to cause the diffraction pattern of the target to be imaged within an encoding region of the spatial light modulator after being Fourier-transformed by the Fourier lens.

8. The high-speed microscopic imaging device of claim 1, further comprising a third lens group disposed between the spatial light modulator and the imaging sensor, wherein, the third lens group is configured to cause the encoded diffraction pattern generated by the spatial light modulator to be imaged within a sensing region of the imaging sensor.

9. The high-speed microscopic imaging device of claim 1, wherein, pixels of the encoding pattern of the spatial light modulator are binary encoded to allow or not allow received light to reach the imaging sensor, and a pixel size is larger than a Bragg spacing of the diffraction pattern of the target.

10. The high-speed microscopic imaging device of claim 9, wherein, The binary encoding of the pixels of the encoding pattern of the spatial light modulator is performed in one of the following ways: a random encoding way; an encoding way that causes more information loss in low frequency parts of the diffraction pattern of the object than in high frequency parts of the diffraction pattern of the object; an encoding way that causes more information loss in high intensity parts of the diffraction pattern of the object than in low intensity parts of the diffraction pattern of the object.

11. The high-speed microscopic imaging apparatus of claim 1, wherein the spatial light modulator is a digital micromirror array comprising a plurality of micromirrors, a rotation angle of each micromirror being digitally configurable to reflect light received by the micromirror to be received or not to be received by the imaging sensor, the digital micromirror array providing different encoding patterns of the spatial light modulator by digital configuration; or the spatial light modulator comprises a plurality of mask plates and switching components for switching the plurality of mask plates, each mask plate of the plurality of mask plates comprising a plurality of regions arranged as an array and each configured to allow light to pass therethrough to be received by the imaging sensor or not to pass therethrough to be received by the imaging sensor, different mask plates of the plurality of mask plates providing different encoding patterns of the spatial light modulator; or the spatial light modulator comprises a mask plate having a plurality of regions arranged as an array, each region of the plurality of regions of the mask plate comprising a plurality of sub-regions arranged as an array and each configured to allow light to pass therethrough to be received by the imaging sensor or not to pass therethrough to be received by the imaging sensor, different regions of the plurality of regions of the mask plate providing different encoding patterns of the spatial light modulator, and a moving component for moving the mask plate so that the diffraction pattern of the object is received by different regions of the plurality of regions of the mask plate.

12. The high-speed microscopic imaging apparatus of claim 1, further comprising an aperture stop disposed between the object and the spatial light modulator and in close proximity to the object for defining an imaging field of view.

13. The high-speed microscopic imaging apparatus of claim 1, wherein the imaging sensor is in communication with a processor and configured to transmit the compressed encoded diffraction pattern to the processor, and wherein the processor is configured to decode the compressed encoded diffraction pattern according to the respective encoding patterns of the spatial light modulator and their temporal order within the exposure time of the imaging sensor to obtain the plurality of frames of diffraction patterns of the object at different times, and to perform phase retrieval on the plurality of frames of diffraction patterns to reconstruct a plurality of frames of images of the object at different times.

14. The high-speed microscopic imaging device of claim 13, wherein, the processor is configured to decode the compressed encoded diffraction pattern to obtain the plurality of frames of diffraction patterns of the object at different times by a multi-frame reconstruction neural network model having a plurality of stages, each stage of the plurality of stages comprising: a projection module configured to map an input of the stage from a compressed domain to a signal domain to obtain a plurality of signal maps, the plurality of signal maps being identical in number to the plurality of frames of diffraction patterns; a feature extraction module configured to receive the plurality of signal patterns from the projection module to extract features; a feature integration module configured to integrate the extracted features to obtain an output of the stage.

15. The high-speed microscopic imaging device of claim 14, wherein, Each of the plurality of stages further comprises a plurality of feature transfer layers disposed between the feature extraction module and the feature integration module for passing extracted features wherein i is a positive integer greater than 2. each feature transfer layer comprises a first reversible block and a second reversible block stacked with each other , an output channel of the feature extraction module is divided into two parts to be coupled to the first reversible block and the second reversible block of a first feature transfer layer adjacent to the feature extraction module , and a relationship as follows is satisfied between ， ， wherein and are the same activation function.

16. The high-speed microscopic imaging device of claim 13, wherein, the processor is configured to perform the phase retrieval for each of the plurality of frames of diffraction patterns by combining a phase retrieval process with a denoising process to reconstruct a frame of images of the object corresponding to the frame of diffraction patterns.

17. The high-speed microscopic imaging device of claim 16, wherein, the phase retrieval process is a hybrid input-output HIO process, and wherein the processor is configured to, in each iteration of the HIO process performed for each of the plurality of frames of diffraction patterns, input a real-space signal corrected in real space into a denoising neural network model configured to: extract a plurality of sub-images from the received real-space signal corrected in real space; input the plurality of sub-images and a noise level map set for the frame of diffraction patterns into a multi-layer neural network; output a plurality of denoised sub-images from the multi-layer neural network, the plurality of denoised sub-images being identical in number and size to the plurality of sub-images; and integrate the plurality of denoised sub-images into a denoised real-space signal.

18. The high-speed microscopic imaging device of claim 17, wherein, extracting a plurality of sub-images from the received real-space signal corrected in real space comprises: dividing an image of the real-space signal into a plurality of primary regions, each of the plurality of primary regions comprising a plurality of secondary regions, the plurality of secondary regions being identical in number to the plurality of sub-images; extracting each of the plurality of sub-images as comprising a respective one of the secondary regions.

19. The high-speed microscopic imaging device of any one of claims 13 to 18, wherein, the high-speed microscopic imaging apparatus further comprises the processor.

20. A high-speed microscopic imaging method, comprising: illuminating an object with coherent light to perform coherent diffraction imaging of the object; receiving a diffraction pattern of the object illuminated by the coherent light and performing spatial modulation of the diffraction pattern of the object, the spatial modulation comprising encoding the diffraction pattern of the object according to an encoding pattern to generate an encoded diffraction pattern; and performing compressed imaging on a plurality of frames of encoded diffraction patterns generated by the spatial modulation over a period of time to generate a single compressed encoded diffraction pattern of the object, wherein the spatial modulation comprises switching the encoding pattern a plurality of times over the period of time to encode a plurality of frames of diffraction patterns of the object at different times of the period of time according to respective different encoding patterns to generate the plurality of frames of encoded diffraction patterns, and wherein performing a Fourier transform on the diffraction pattern of the object to form a Fourier transform spectrum of the diffraction pattern of the object in a near field of the object is performed the spatial modulation to generate the encoded diffraction pattern.

21. The high-speed microscopic imaging method of claim 20, wherein: the diffraction pattern of the object is formed by light reflected by the object through the coherent light; or the diffraction pattern of the object is formed by light transmitted by the object through the coherent light.

22. The high-speed microscopic imaging method of claim 20, wherein, The spatial modulation is performed at a far field or Fourier plane of the object.

23. The method of claim 20, further comprising: decoding the compressed encoded diffraction pattern according to each encoded pattern used by the spatial modulation and its time order within the period of time to obtain the multiple frames of diffraction patterns of the object at different times, and performing phase retrieval on the multiple frames of diffraction patterns to reconstruct multiple frames of images of the object at different times.

24. The high-speed microscopic imaging method of claim 23, wherein, decoding the compressed encoded diffraction pattern comprises decoding the compressed encoded diffraction pattern by a multi-frame reconstruction neural network model having multiple stages, each stage of the multiple stages comprising: a projection module configured to map the input of the stage from a compressed domain to a signal domain to obtain a plurality of signal patterns, the plurality of signal patterns being identical in number to the multiple frames of diffraction patterns; a feature extraction module configured to receive the plurality of signal patterns from the projection module to extract features; a feature integration module configured to integrate the extracted features to obtain the output of the stage.

25. The high-speed microscopic imaging method of claim 24, wherein, Each of the plurality of stages further comprises a plurality of feature transfer layers disposed between the feature extraction module and the feature integration module for passing extracted features wherein i is a positive integer greater than 2, wherein each feature transfer layer comprises a first reversible block and a second reversible block stacked with each other , an output channel of the feature extraction module is divided into two parts to be coupled to the first reversible block and the second reversible block of a first feature transfer layer closest to the feature extraction module respectively , and a relationship as follows is satisfied between adjacent feature transfer layers ， ， wherein and are the same activation function.

26. The high-speed microscopic imaging method of claim 23, wherein, performing phase retrieval on the multiple frames of diffraction patterns comprises performing the phase retrieval on each frame of diffraction pattern in the multiple frames of diffraction patterns by a phase retrieval process combined with a denoising process to reconstruct a frame of image of the object corresponding to the frame of diffraction pattern.

27. The high-speed microscopic imaging method of claim 26, wherein, the phase retrieval process is a hybrid input-output (HIO) process, and wherein performing phase retrieval on the multiple frames of diffraction patterns comprises, in each iteration of the HIO process performed for each frame of diffraction pattern in the multiple frames of diffraction patterns, inputting a real-space signal corrected in real space from the received real-space signal corrected in real space into a denoising neural network model to be denoised and using the denoised real-space signal for the next iteration, the denoising neural network model being configured to: extract a plurality of sub-images from the received real-space signal corrected in real space; input the plurality of sub-images and a noise level map set for the frame of diffraction pattern into a multi-layer neural network; output a plurality of denoised sub-images from the multi-layer neural network, the plurality of denoised sub-images being identical in number and size to the plurality of sub-images; and integrate the plurality of denoised sub-images into a denoised real-space signal.

28. The high-speed microscopic imaging method of claim 27, wherein, extracting a plurality of sub-images from the received real-space signal corrected in real space comprises: dividing an image of the real-space signal into a plurality of primary regions, each of the plurality of primary regions comprising a plurality of secondary regions, the plurality of secondary regions being identical in number to the plurality of sub-images; extracting each of the plurality of sub-images as including a respective one of the secondary regions in each of the primary regions.

Images