Multicore Fiber Imaging

Abstract

To provide a multicore fiber imaging such as used in endoscopy.SOLUTION: A method for generating an image from light received by an imaging device via a multiplicity of waveguides includes: receiving a digital image containing a plurality of pixels, the digital image including a plurality of regions within it wherein each of said regions corresponds to a waveguide core and includes a plurality of pixels corresponding to a gap space between the waveguide cores; determining a first pixel subset in each region at least partially correlated with light received by a corresponding core in a first spatial arrangement, the first pixel subset including less pixels than all of the pixels in the regions; and generating a first image from the first pixel subset from the regions.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001]

[0001] The present invention relates to multicore fiber imaging, such as those used primarily in endoscopy. In a preferred embodiment, a method is described for processing images captured by such a system to obtain improved depth of field images or to extract 3D information about the images, without requiring the addition of additional optical components. [Background technology]

[0002]

[0002] Multicore optical fibers (MOFs) are widely used as micro-endoscopic probes for imaging inside the body. Such MOFs contain thousands of waveguide cores, each 2-3 μm in diameter, that relay light from inside the body to an imaging device that acts as an external detector. The imaging device generates an image of the distal end of the MOF. The generated image includes a number of pixels, each of which includes a number of regions corresponding to a waveguide core. Each core will be imaged across a number of pixels in the image. The digital image will also include pixels corresponding to the interstitial spaces between the waveguide cores. Figure 1 shows a portion of such an image. The continuous black image portion 2 in Figure 1 corresponds to the pixels that imaged the interstitial spaces between the waveguide cores. The light areas of various grey levels (e.g. 4) correspond to the individual waveguide cores, with illumination levels corresponding to the light received at the distal facet of each fiber.

[0003]

[0003] Developments in MOF technology have included processing the received data to improve image quality, including filtering techniques to address image artifacts produced by gap regions between fiber cores, which can manifest as distracting patterns such as visible grids or Moire effects. For example, U.S. Patent No. 5,751,340 discloses reducing the contrast of grid pattern components by suitable filtering, such as by applying a dilation process that interpolates from the brighter center of each fiber to nearby gap regions.

[0004]

[0004] Multicore optical fibers enable a wide range of micro-endoscopic imaging modes, including optical coherence tomography, reflectance and fluorescence (confocal, wide-field, and multiphoton).

[0005]

[0005] MOFs operate at short working distances, typically around 50-100 μm, when fitted with a microlens on the distal facet. Imaging is also possible in contact mode, where the bare fiber facet is in direct contact with the sample. In this case, the Nyquist-limited resolution is twice the inter-core distance, whereas in lensed systems the resolution scales by the inverse magnification.

[0006]

[0006] Both lensless and lens systems suffer from a rapid loss of resolution away from the focal plane. This is particularly challenging in micro-endoscopes, which often lack sensitive focus control at the distal end due to size limitations. To avoid out-of-focus haze, confocal sectioning can be employed, restricting the collected signal to the MOF facet, i.e., the distal surface of the MOF (lensless systems) or the slice near the focal plane (lens systems). However, this can make focusing even more difficult due to confocal rejection of signals outside the optical slice. In many instances, it is ideal for a micro-endoscope system to collect "all-focus" images over an extended depth of field, which appear sharp even if the object is not exactly located at the focal plane or MOF facet.

[0007]

[0007] In other types of imaging systems, the depth of field can be extended by restricting the collection aperture to increasingly paraxial rays. This reduces the size of the blur circle of out-of-focus objects, thereby sharpening the image over the depth range. However, because MOFs do not have an adjustable aperture, this mechanism does not allow for control of the depth of field.

[0008]

[0008] Previous attempts to extend the depth of field of endoscopes have relied on mechanical engineering to result in larger and more complex probes at the distal end.

[0009]

[0009] Other imaging techniques using single-core multimode fibers with extended depth of field require the use of coherent light. As a result, these techniques are very sensitive to fiber bending, making them difficult to apply to real-world situations. Furthermore, imaging techniques using multicore fibers that use coherent light with extended depth of field, which is insensitive to fiber bending, require very sophisticated image reconstruction algorithms that tend to fail with complex objects. Summary of the Invention [Problem to be solved by the invention]

[0010]

[0010] It is therefore an object of the present invention to provide a method for generating an image from light received through a MOF that at least partially addresses one or more of the above-mentioned drawbacks.

[0011]

[0011] The reference to any prior art in the specification is not an admission or suggestion that this prior art forms part of common general knowledge in any jurisdiction or that this prior art could be combined with other prior art by a person skilled in the art. [Means for solving the problem]

[0012] In a first aspect, the present disclosure provides a method of generating an image from light received by an imaging device through multiple waveguides, the method comprising: receiving a digital image including a plurality of pixels, the digital image including a plurality of regions therein each corresponding to a waveguide core and including a plurality of pixels, and further including pixels corresponding to interstitial spaces between the waveguide cores; determining a first subset of pixels in each region that at least partially correlates with light received at a corresponding core in the first spatial arrangement, the first subset of pixels including fewer than all pixels in the region; and generating a first image from a first subset of pixels of the region.

[0013]

[0013] Producing the first image preferably comprises: For each region, determining an average pixel value of said pixels of a first subset and assigning said average pixel value as the pixel value of at least one pixel in said first subset of pixels. The method preferably includes generating pixel values ​​for pixels that are not said at least one pixel. In an embodiment, an average pixel value is assigned to one pixel in said first subset of pixels. Most preferably, the average pixel value is assigned to a pixel at a predetermined location in the image that represents the center of the waveguide core. An average pixel value may also be assigned to all pixels of a group of pixels at or around said predetermined location that represents said waveguide core center.

[0014]

[0014] Generating the first image includes: generating pixel values ​​for pixels other than the at least one pixel (e.g., a central pixel or a group of centrally located pixels). Preferably, generating pixel values ​​for the at least one non-pixel includes: assigning pixel values ​​according to a pixel value distribution function centered on said at least one pixel; or assigning pixel values ​​by interpolating between said at least one pixel of adjacent regions.

[0015]

[0015] Thus, in one variant, the method includes determining an average pixel value of the pixels of the first pixel subset in the region, assigning the average value to a designated pixel (or designated group of pixels) located at a position representing the waveguide core centre of the region, and assigning values ​​to the remaining pixels by interpolating (e.g., by linear interpolation) between adjacent designated pixels (or respective designated groups of pixels) of the region.

[0016]

[0016] The first pixel subset may include all pixels within a predetermined radius from the center of the region. The radius may be substantially smaller than the radius of the waveguide core. Alternatively, the subset may be any other shape or arrangement of the region as desired.

[0017] The method further comprises: generating a second image from the received digital image; and This can include combining the second image with the first image to generate a final image.

[0018] The second image can be generated in a similar manner to the first image, where: defining a second subset of pixels within each region, the second subset of pixels including fewer than all of the pixels within the region and different from the first subset of pixels; and generating a second image from a second subset of pixels of the region.

[0019] In an embodiment, one of the first and second subsets may include a subset of all pixels in the region, excluding pixels that partially or completely correspond to the interstitial spaces between the waveguide cores (i.e., the fiber cladding and the air gaps), where the other of the first and second subsets represents a smaller area of ​​the region, preferably a significantly smaller area.

[0020]

[0020] Producing the second image includes: For each region, an average pixel value of the pixels of the second subset may be determined and the average pixel value may be assigned as the pixel value of at least one pixel of the second pixel subset. This may include generating pixel values ​​for the at least one non-pixel pixel in the second pixel subset.

[0021]

[0021] Generating pixel values ​​for the at least one non-pixel pixel in the second pixel subset further includes: assigning pixel values ​​according to a pixel value distribution function centered on said at least one pixel in each region; or assigning pixel values ​​by interpolating between pixel values ​​of a second subset of adjacent regions.

[0022]

[0022] Thus, in one variant, the method includes determining an average pixel value of the pixels of the second pixel subset in the region, assigning the average value to a designated pixel (or designated group of pixels) located at a position representing the waveguide core centre of the region, and assigning values ​​to the remaining pixels by interpolating (e.g., by linear interpolation) between adjacent designated pixels (or respective designated groups of pixels) of the region.

[0023]

[0023] The second pixel subset includes all pixels within a second predetermined radius from the center of the region. The radius may be significantly smaller than the radius of the waveguide core. Like the first pixel subset, the second pixel subset may include pixel subsets of any shape.

[0024] Combining the second image with the first image may include, for each region, using one of the first and second images to modulate, weight or modify the other of the first and second images. The modified images for all regions are then combined to generate a modified digital image across the various waveguides.

[0025]

[0025] Combining the second image with the first image includes scaling the brightness of one or both images as necessary, and subtracting the second image from the first image. Brightness scaling is preferably performed such that the total intensity of both images is equal. It will be appreciated that other suitable approaches to combining the second image with the first image may also be used.

[0026]

[0026] By appropriate selection of the regions, this approach has the effect of selectively removing light near the periphery of each core and limiting the numerical aperture of each waveguide by calculating the difference between the light level in this (first) image and the light level in the unfiltered (second) image.

[0027]

[0027] The first image preferably has a greater effective depth of field than the second image.

[0028]

[0028] Preferably, the generation of the first image is biased towards selecting light rays received at the waveguide within a first angular range, and the generation of the second image is biased towards selecting light rays received at the waveguide within a second angular range.

[0029]

[0029] Preferably, the second angle range is wider than the first angle range.

[0030]

[0030] In some embodiments of any of the methods defined above, the method further comprises: determining at least one other subset of pixels in each region that at least partially correlates with light received at a corresponding core in a corresponding other spatial location, the other subset of pixels including fewer than all pixels in the region; generating at least one corresponding further image from said received digital image; and and combining the first image with the at least one further image according to the weighting to generate a final image. The at least one further image may include the second image.

[0031]

[0031] The predetermined weightings may be determined by a calibration process.

[0032]

[0032] In another aspect, the present specification further provides a method for improving the apparent depth of field of an image captured through a multi-core optical fiber (MOF), the digital image comprising a plurality of pixels, the digital image including a plurality of regions therein each corresponding to a core of the MOF and including a plurality of pixels, and further including pixels corresponding to interstitial spaces between the waveguide cores, the method comprising: determining a first subset of pixels in each region that at least partially correlates with light received at a corresponding core in the first spatial arrangement, the first subset of pixels including fewer than all pixels in the region; determining, for each region, an average pixel value of said pixels of a first subset and assigning said average pixel value as a pixel value of at least one pixel of said first pixel subset; generating pixel values ​​for pixels other than said at least one pixel; discloses a method that includes generating a first image having an improved depth of field.

[0033]

[0033] Generating pixel values ​​for the at least one non-pixel pixel in the first pixel subset includes: assigning pixel values ​​according to a pixel value distribution function centred on said at least one pixel of each first region; or assigning pixel values ​​by interpolating between pixel values ​​of the first subset of adjacent regions.

[0034]

[0034] The method further comprises generating a second image from the received digital image; and and combining the second image with the first image to generate a final image having an improved depth of field. The second image is defining a second subset of pixels within each region, the second subset of pixels including fewer than all of the pixels within the region and different from the first subset of pixels; and and generating a second image from a second subset of pixels of the region.

[0035] In a preferred embodiment, the first image has a greater effective depth of field than the second image.

[0036]

[0036] The generation of the first image is preferably biased towards selecting light rays received at the waveguide within a first angular range, and the generation of the second image is biased towards selecting light rays received at the waveguide within a second angular range.

[0037]

[0037] The second angular range is preferably wider than the first angular range.

[0038] Systems (eg, imaging systems, and image processing systems) configured to carry out these methods also constitute further aspects of the present disclosure.

[0039] A further aspect of the present disclosure relates to light field imaging.

[0040]

[0040] Specifically, in a further aspect, the present disclosure provides a method for determining a light field approximation for use in image processing corresponding to a pair of images generated from light received by an imager via multiple waveguides, the method comprising: acquiring a pair of images, a first member image having a first depth of field and a second member image having a second depth of field, the first member image and the second member image having the same focus position; generating a difference image from a pair of images; calculating a light field approximation from the difference image.

[0041]

[0041] The process of generating the difference image may initially involve at least one intensity scaling of the images, which preferably involves dividing each pixel value by the average pixel value of the images, such that the overall intensity of the image pair is equal.

[0042]

[0042] The process of calculating the light field approximation may involve using an assumed angular distribution of light propagation relative to the average ray orientation.

[0043]

[0043] The hypothesized angular distribution may be a Gaussian distribution.

[0044]

[0044] The second member image may be obtained using the method of any one of the embodiments of the above aspects of the present disclosure.

[0045]

[0045] The first member image can be obtained using an embodiment of the first aspect of the present disclosure, where the first member image and the second member image use different first pixel subsets within each region.

[0046]

[0046] Preferably, the first member image is taken from the same digital image as the second member image and is generated from substantially all pixels in a region of the digital image which corresponds to the waveguide core.

[0047]

[0047] In a further aspect, there is provided a method of generating an image, comprising the steps of: acquiring a pair of images, a first member image having a first depth of field and a second member image having a second depth of field, the first member image and the second member image having the same focus position; determining a light field approximation using a method embodying the above aspects of the present disclosure; and processing the image according to the light field approximation to generate a final image.

[0048] Processing the image according to the light field approximation includes: reconstructing images having different focus positions; Reconstructing an image having a different viewpoint than the received image; Reconstructing the 3D perspective of the image; may include any one or more of the following:

[0049]

[0049] Processing of images in this or any of the other aspects of the invention disclosed in this specification may include assuming non-linear stereo disparity of the light source as one moves away from the light source.

[0050]

[0050] In a further embodiment, there is provided a method of generating one or more images from light received by an imaging device through multiple waveguides, the light being generated from a light field incident on said multiple waveguides, the method comprising: receiving a digital image including a plurality of regions each corresponding to a waveguide core and including a plurality of pixels; processing the image intensity pattern over each of said regions to determine a light field angle dimension measure of the region; applying the angular dimension measure to one or more pixels contained in each region to generate one or more sets of modified image data; generating one or more images using the one or more sets of modified image data.

[0051]

[0051] Processing the image intensity pattern over each of the regions may include analyzing each region with a simulated aperture involving a computational comparison of image intensity under a first computational aperture and image intensity under a second computational aperture for each region. Pixels of one of the first and second computational apertures may include a subset of pixels of the other of the first and second computational apertures, or the pixel sets of each computational aperture may differ by a particular light field angle dimension measure extracted from the processing step.

[0052]

[0052] Alternatively or additionally, processing the image intensity pattern across each of said regions may include a pattern matching algorithm that compares the image intensity pattern with a stored pattern, which may be generated for each of said various waveguides by a pattern calibration process.

[0053]

[0053] As noted above, in a further aspect, the present disclosure also provides an imaging system, comprising: a multi-core optical fiber (MOF) extending from a proximal end to a distal end; a light source for illuminating a scene at the distal end of the MOF; an imaging device positioned relative to the proximal end of the MOF to capture an image of the light propagated along the MOF; and a data processing system configured to receive images captured by the imaging device, the data processing system configured to execute instructions that cause the data processing system to perform a method embodying any of the aspects disclosed herein.

[0054]

[0054] Preferably the MOF comprises an endoscope.

[0055] This disclosure describes the use of various illumination geometries for the object or scene being imaged. In one or more embodiments, the methods and systems described utilize reflection, transmission, fluorescence, or a combination thereof.

[0056]

[0056] As noted above, in a further aspect, the present disclosure also provides an image processing system comprising at least one processing unit and at least one memory for storing instructions executed by the at least one processing unit, the instructions being executed to perform a method embodying any of the aspects disclosed herein.

[0057]

[0057] Unless the context otherwise requires, when used in this specification, the term "comprise" and variations of this term such as "comprising", "comprises" and "comprised" are not intended to exclude further additional items, components, integers or steps.

[0058] Further aspects of the invention and further embodiments of the aspects described in the preceding paragraphs will become apparent from the following description, given by way of example and referring to the accompanying drawings, in which: [Brief description of the drawings]

[0059] [Figure 1]

[0059] Figure 1 represents a portion of an original image acquired from a MOF by an imaging device, showing the MOF core and interstitial areas. [Diagram 2]

[0060] FIG. 2 is a schematic MOF imaging system used in capturing images that can be processed by embodiments of the present invention. [Diagram 3]

[0061] FIG. 3 is a flowchart of a first image processing method according to an embodiment of the present invention. [Figure 4]

[0062] 4 shows a series of images illustrating the output from two embodiments of the present invention compared to an original image series. In this example, the images show a portion of Group 5 of a USAF 1951 target. [Figure 5A]

[0063] FIG. 5A is a flowchart of a second image processing method according to an embodiment of the present invention. [Figure 5B] FIG. 5B is a flow chart of another embodiment of the method of FIG. 5A. [Figure 6]

[0064] Figure 6 shows an image series (labeled eDOF, for "extended depth of field" images) generated using an embodiment of the process of Figure 5A. Also shown is a series of plots comparing the image series of Figure 6 with the image series of Figure 4. [Figure 7]

[0065] Figure 7a shows a light beam impinging on the input facet of a fiber core with an orientation described by the incidence angle θ and azimuth angle φ. Figure 7b shows the simulated input core intensity distribution resulting from a plane wave oriented at an angle (θ,φ) (scale bar: 5 μm). Figure 7c plots the simulated normalized intensity within two simulated apertures, R=3px and R=1px, as well as the full aperture image (R=7px) specifically for the circled fiber core input in Figure 7b. Figure 7d shows the experimentally recorded output intensity from the output facet of a single core varying the plane wave input angle (θ,φ) using the apparatus of Figure 2. [Figure 7E] FIG. 7E shows the simulated angular PSF normalized to the aperture area. [Figure 8]

[0066] FIG. 8 shows a series of images of a portion of Groups 6 and 7 of the USAF 1951 target used in experimental testing of an exemplary implementation of the present invention. [Figure 9]

[0067] Figure 9a shows textile fibers imaged using a standard full aperture approach compared to an embodiment of the invention using both 10x and 20x objectives (right column), and Figure 9b shows the intensity profile along the line shown in the middle row of Figure 9a. [Figure 10]

[0068] FIG. 10 is a flow chart of a method for estimating the light field of an image using an embodiment of one aspect of the present invention. [Figure 11]

[0069] Figure 11 shows the first and second images of the image pair used in the method of Figure 10, and a visualization of the difference in intensity values ​​between these images. The last row shows a composite image generated from a light field approximation generated from the images of Figure 11. [Figure 12]

[0070] Figure 12a depicts depth maps of a series of images of a USAF target (Group 5) generated using an embodiment of the present invention, and Figure 12b is a plot of depth metrics as a function of known ground truth depth (distance from the MOF fiber facet to the USAF target) for the targets used to generate this embodiment. [Figure 13]

[0071] 13a and 13b show the results of computational refocusing of a USAF target image at various distances. [Figure 14]

[0072] FIG. 14 is a block diagram illustrating a computer processing system suitable for use as an image processing system for processing images captured by the system of FIG. [Figure 15]

[0073] FIG. 15 shows the application of light moment imaging (LMI) to MOF images processed according to an embodiment of the present invention. [Figure 16]

[0074] FIG. 16 shows the stereo disparity relationship for a moving viewpoint (increasing axial distance from the object) when applying the method of an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0060]

[0075] 2 is a schematic diagram of an exemplary optical arrangement of a MOF in an imaging system 8 arranged to image a prepared sample illustrating an embodiment of the present invention. The imaging system may be an endoscopic system. Preferably, the imaging system is coupled to or includes a data processing system 400 for processing image data from the imaging system 8.

[0061]

[0076] The proximal facet of MOF 10 (e.g., Fujikura FIGH-30-600S or FIGH-10-350S) is illuminated with incoherent light from LED 12 (e.g., Thorlabs M565L3 LED, 565 nm center wavelength). The total illumination at the distal end of the MOF is up to 10 μW in this example. The light from LED 12 is collimated by a collimating lens (CL) and passes through a mirror (M), a 200 mm lens (L), a polarizer (P1), a beam splitter (BS), and a 20x objective lens (OBJ). The illumination source 12 is linearly polarized to facilitate the elimination of light reflected from the proximal facet of the MOF. Both ends of MOF 10 and the sample 14 are mounted on independent 3-axis translation stages (xyz). It is preferred that there is no lens between the distal MOF facet and the sample, although some embodiments of the invention may use such a lens arrangement.

[0062]

[0077] Light reaching the distal end of the MOF illuminates the sample 14 before the reflected light is recoupled into the MOF 10. The back-reflected light couples into different modes depending on the angle of incidence on the distal fiber facet. The output intensity pattern at the proximal end in the multiple cores is imaged via a microscope objective (e.g., Olympus Plan Achromat 20x 0.4NA), a beam splitter (BS), a 200 mm tube lens (TL) and a second polarizer (P2). The polarization axes of P1 and P2 are orthogonal to filter out the reflected light at the proximal facet. The image is captured by a camera (CAM) (e.g., Thorlabs DCC3240M, a black and white camera with 10 ms integration time). In this example, the core and cladding refractive indices of the MOF are ncore=1.5 and nclad=1.446, respectively, resulting in an NA of 0.398 that closely matches the 20x, 0.4NA objective lens (OBJ).

[0063]

[0078] The inventors have realized that the property that light arriving at the distal (receiving) end of a multicore fiber from different directions propagates to the proximal end and is received at the imaging device with different spatial intensity patterns can be used to emphasize or de-emphasize light received from specific directions in the processed image. Thus, the invention arises from the realization that MOFs convey 3D information in the form of light field information (the spatial angular distribution of light rays incident on the distal end of the MOF), and that the angular dimension of the light field is modulated into the heretofore ignored in-core intensity patterns of the fiber bundle. As discussed further below, these intensity patterns arise due to angle-dependent inter-modal coupling, and the invention involves relating these patterns to the angular dimension of the light field.

[0064]

[0079] An important observation is that light incident on the fiber core at different angles will produce different intensity distributions at the output of the fiber core. Specifically, light rays that impinge straight on the fiber core (paraxial rays) tend to excite primarily the fundamental mode of the fiber, resulting in an output pattern in which most of the light is concentrated in the middle of the core. On the other hand, as the angle of incidence increases, the output light density at the output of the fiber core tends to move towards the periphery of the core. Furthermore, the inventors realized that by emphasizing light that arrives nearly parallel to the axis of the distal end of the fiber, images with a large depth of field can be produced.

[0065]

[0080] According to the present invention, these intensity patterns, which arise due to angle-dependent inter-modal coupling, are quantitatively related to the angular structure of the light field.

[0066]

[0081] An embodiment of the present invention generates an image using a "simulated aperture" applied to each core of an optical fiber. The simulated aperture is applied by selectively emphasizing a subset of pixels corresponding to one or more spatial regions of each core from the image to weight the image generation process. In one form, the simulated aperture is applied by selecting a pixel subset that includes only pixels within a certain radius of the center of each core. In some embodiments, the pixel subset corresponding to each core may not be centered at the center of each core. For the avoidance of doubt, the pixel subsets that make up the "simulated aperture" need not be a single spatially contiguous subset, but may be composed of sub-subsets. Furthermore, the pixel subsets may be of any suitable shape.

[0067]

[0082] The embodiments may be applied to multi-core optical fibers used in contact mode (lensless) or lens mode.

[0068]

[0083] FIG. 3 illustrates an exemplary method of the present invention that may be used to generate an image with a larger DOF than the original captured image 202, which applies a simulated aperture to the received image to remove contributions from rays arriving at large oblique angles. As noted above, paraxial rays contribute primarily to the intensity in the middle of the fiber core, while oblique rays tend to generate a pattern with the intensity pushed toward the periphery of the core. Thus, by reconstructing the image using only the central pixels of each core, it is possible to remove a small amount of the oblique rays, or at least tilt the image generation in a direction that includes a larger proportion of paraxial rays. This has been found to have the effect of increasing the DOF compared to using all pixels from each core, or downsampling the raw fiber facet image.

[0069]

[0084] Method 200 begins by receiving an original image from the MOF, for example using an apparatus such as that shown in Figure 2. Then, in step 206, a subset of pixels from regions in the image that correspond to the waveguide cores (i.e., portions of the image related to the interstitial space are ignored) are selected and used to generate an image in step 208. These regions can be conceptually thought of as simulated apertures that, when applied computationally, have the effect of being more or less selective to light received at different angles of arrival at the distal end of the MOF.

[0070]

[0085] In some embodiments, this may be a prerequisite for selecting a pixel subset to identify regions in the image that correspond to the waveguide cores. This may be done using automated image analysis techniques, or alternatively, it may be assumed that the waveguide cores are spatially arranged in a known pattern, for example a regular grid, and thus the location of the regions in the image is known. In some embodiments, the identification of the regions in the image that contain the cores may be determined by a process of taking a reference image using a mirror instead of the sample. This image may have a high contrast between the core and the cladding in the image, and may be used to more easily identify the location of the cores.

[0071]

[0086] As will be appreciated by those skilled in the art, embodiments of the present invention may employ other image processing techniques to improve image quality, for example, a sample-free background image may be acquired and then subtracted from each raw camera image before further processing.

[0072]

[0087] Next, in step 208, an image is generated based on the pixels within the simulated aperture. This can include averaging (210) the pixel values ​​across the simulated aperture and assigning this value to pixels at the centers of the cores. The method then includes generating pixel values ​​between the core centers (step 212). Step 212 can include assigning pixel values ​​centered at the center of each core by applying a pixel value distribution function, or interpolating between pixel values ​​of adjacent core centers.

[0073]

[0088] In a preferred form, after averaging the intensity within each simulated aperture, the average value of each region is assigned to a grid location in the image that represents the center of the core, and the image is resampled. In the resampled image, the value corresponding to the location of each region (i.e., core) on the grid corresponds to its location on the fiber facet. The image is resampled using a Gaussian intensity profile with a full width at half maximum (FWHM) equal to twice the grid spacing. The FWHM of the Gaussian can be adjusted to provide a balance between signal-to-noise ratio and resolution. The inventors have found that a FWHM of twice the grid sampling slightly low-pass filters the image, but improves image resolution by averaging out high spatial frequency noise from non-uniform core spacing. The peak value of the Gaussian distribution representing a given core is equal to the average intensity within a selected subset of pixels (i.e., simulated aperture) within the core region.

[0074]

[0089] FIG. 4 shows a series of images illustrating the output from two embodiments of the present invention compared to an original image series. In this example, the example images show a portion of Group 5 of a USAF 1951 target. Each column of images displays the corresponding image acquired at a given depth. Different columns represent different depths, progressing from left to right in 10 μm increments from 10 to 100 μm.

[0075]

[0090] The top row shows the original image series. As can be seen, the original image series has been processed to filter out pixels in the interstitial space between the fiber cores. This is done using a similar method to that described above. The original image series is constructed by integrating all of the signal in each core (assuming a core radius R=7 pixels) and then resampling the cores onto a grid. These are called "full aperture" images.

[0076]

[0091] The images in the second and third rows of Figure 4 are constructed using a simulated aperture smaller than the overall width of the fiber. The second row uses a "medium" aperture and averages the intensity of pixels within R=3 of the center of the core. The bottom row uses a "small" aperture and averages the intensity of pixels within R=1 pixel of the center of the core.

[0077]

[0092] As can be seen, the smaller simulated aperture size results in higher contrast at larger depths. For example, the third element grating (top of each image) is resolvable using the small simulated aperture at 70 μm, but not using the full aperture. The imaged grating cannot be resolved in the full aperture image at depths greater than 60 μm. In fact, the higher order modes provide less light to the central pixel, and the diffraction provided by the microscope objective also tends to mix light from the edge and center of the core in the camera image. As a result, the contrast increase between the full aperture image and the small aperture image is moderate in these examples.

[0078]

[0093] FIG. 5A shows a further embodiment of the present invention that may further improve the DOF of the generated image. In FIG. 5A, a first image 302 is generated using the method of FIG. 3 using a first simulated aperture. Also, a second image 304, e.g., an original image or an image generated using the method of FIG. 3, may be acquired using a second simulated aperture different from that used to generate the first image 302. The first image 302 is combined with the second image 304 to generate a final image 308. The combining step 306 is a simple deblurring step that further improves image quality. In this step, an image created using all pixels in the fiber core (or a simulated aperture that effectively provides an image that is blurred and has a shallower DOF than that of the first image 302) is subtracted from the first image 302, which was created using only the central pixels. In this way, out-of-focus light is subtracted from the first image 302.

[0079]

[0094] Figure 6 shows an image series (labeled eDOF for "extended depth of field" images) generated using an embodiment of the process of Figure 5A, and more specifically obtained by subtracting the image series of radius R = 3 pixels from the image series of radius R = 1 pixel of Figure 4 (scale bar: 100 μm).

[0080]

[0095] Returning momentarily to the basic principles behind the present invention, consider a MOF illuminated with a light beam (or plane wave) traveling towards its input facet as shown in Figure 2. The output light intensity profile within each core of the MOF depends on the orientation of the excitation plane wave.

[0081]

[0096] Figure 7a shows a ray impinging on the input facet of a fiber core with an orientation described by the incidence angle θ and the azimuth angle φ. These angles relate to the orientation of the ray and are not related to the shape of the core.

[0082]

[0097] Figure 7b shows the simulated input core intensity distribution resulting from a plane wave oriented at an angle (θ,φ) (scale bar: 5 μm). The intensity distribution is calculated by forming an incoherent superposition of the linearly polarized (LP) modes of the fiber. The amplitude of each LP mode in the superposition is given by the coupling efficiency of the plane wave oriented at an angle (θ,φ). The intensity distribution is then integrated over the emission spectrum of the light-emitting diode (LED) used in the experiment (Thorlabs M565L3 LED, central wavelength 565 nm, FWHM maximum 100 nm). The central image is the intensity pattern at the input resulting from a normally incident plane wave, i.e., a paraxial ray. The corner images are the intensity patterns close to θc=sin-1(NAc).

[0083]

[0098] The intensity pattern is assumed to be invariant from the input facet to the output facet due to the temporally incoherent nature of the illumination. That is, the intensity distribution simulated in Figure 7b is located at the input facet. Since the fields in the core are temporally incoherent, it is expected that the output intensity pattern will not change significantly from the input. This is verified experimentally by imaging the output of a fiber core subjected to plane wave excitation input at various angles. Figure 7d shows the experimentally recorded output intensity from the output facet of a single core varying the plane wave input angle (θ,φ) using the apparatus of Figure 2 (scale bar: 5 μm). The plane wave input was achieved by digitally raster scanning a small circular aperture by a digital micromirror device (DMD) placed conjugate to the back pupil plane of the microscope objective. The measured output intensity distribution in Figure 7d shows good qualitative agreement with the distribution generated by simulation (Figure 7b) for the input facet, supporting the view that the intensity profile is relatively unchanged from the input facet to the output facet. Given this observation, the intensity pattern at the core output can be used as a proxy for the angular filtering of the rays at the core input.

[0084]

[0099] The relationship between the input plane wave and the output intensity pattern in the core can be expressed by the matrix equation Ax=b, where column A is the intensity pattern created by a particular plane wave input orientation (i.e., the pattern in FIG. 7d rearranged into a vector), vector x is the fraction of the input intensity at a given plane wave orientation, and vector b is the recorded core output intensity pattern. That is, the output intensity pattern in the core is a linear combination of the intensity patterns created by the individual plane waves (i.e., rays). By solving for x, it is possible to isolate the contributions coupled into the input facet of the core at low angles, thus reducing the collection aperture and increasing the depth of field. However, the coupling matrix A will be different for each core of the fiber due to the non-uniform geometry.

[0085]

[0100] Therefore, to solve for the contribution (x) for each ray direction in each core, it is necessary to measure the angular coupling matrix A for each core and then perform the matrix inversion for each core separately, which can be achieved by careful calibration or simulation.

[0086]

[0101] The preferred embodiment of the present invention does not pursue calibration but uses an approach that does not require calibration. The preferred technique starts from the observation that light rays traveling perpendicular to the facet interface (center image in FIG. 7(b), θ=0) tend to excite the fundamental mode with an intensity that peaks at the center of the core. Oblique light rays (large θ) excite modes with an intensity that is more localized toward the core / cladding boundary. Thus, for the fundamental mode (I=0), maximum excitation efficiency occurs at θ=0. All the incident power is coupled into the fundamental mode at normal incidence, but as θ becomes large, it is distributed between the fundamental mode and increasingly higher order modes. This makes it possible to extract information about the higher order modes of the received light by weighting the image toward the boundary between the core and the edge of the fiber (core / cladding boundary).

[0087]

[0102] In a preferred embodiment, the invention concerns the coupling efficiency as a function of input angle and how it varies with sub-regions of the core output. Figure 7c plots the normalized total intensity in different sub-regions of the simulated input / output core image (Figure 7b) as a function of the incidence angle θ. The inset in Figure 7c shows the size of each simulated aperture relative to the full size of the core, superimposed on the circled intensity pattern in Figure 7b. In Figure 7, 1px = 238.5 nm, consistent with the experiments in Figures 4 and 6. The shaded background indicates the accepted angular range within the NA of the MOF.

[0088]

[0103] Specifically, Figure 7c plots the simulated normalized intensity within the full aperture image (R=7px) and two simulated apertures, R=3px and R=1px, for the circled fiber core input in Figure 7b. The curve labeled "eDOF" is the difference between the R=1px and R=3px curves normalized by the aperture area. All curves in Figure 7c are normalized to have a maximum value of 1. The area normalization is performed as shown in Figure 7E, which shows the simulated angular PSF normalized to the aperture area, where the total intensity within each selected subset of pixels imaging the core is scaled by the area (πR 2 ), i.e., the simulated aperture. The small aperture angle PSF (R=1px) has the largest magnitude since the average pixel value is highest in this subregion. In contrast, the full aperture angle PSF (R=7px) has the lowest average value since it contains many blurred pixels that lower the average value. The eDOF curve is calculated as Ismall-Imedium using the curve from this plot directly. This helps remove any remaining oblique rays from the R=1px image. An offset is added so that the curve is positive everywhere. This curve is labeled "eDOF" in Figure 7E.

[0089]

[0104] In Figures 7c and 7E, the intensity within the small "simulated aperture" at a radius of R=1 pixel (1 pixel=238.5 nm) from the core center has a sharper angular distribution than the "medium" R=3 and "full aperture" R=7 images. This is qualitatively evident from the example in Figure 4. As can be seen in the eDOF plots in Figures 7c and 7e, the angular PSF can be further narrowed by subtracting the R=3px curve from the R=1px curve after normalizing the total intensity of each curve by the respective aperture area shown in Figure 7E.

[0090]

[0105] The image formed by this subtraction process will have greater resolution compared to the small aperture image at the expense of lower signal and increased noise (noise from the small and medium aperture images is added). Note also that the eDOF PSF has a higher background level due to the offset added.

[0091]

[0106] In general, other linear combinations of simulated through-aperture images can be used to generate images with different properties, e.g., different depths of field, tradeoffs between signal-to-noise ratio (SNR) and angular PSF width. More generally, it is possible to selectively target the imaging of plane waves oriented at any given angle (θ,φ). FIG. 5B illustrates such a process. In this example, n images (302, 303, 304), each resulting from a process similar to FIG. 3 but with different simulated apertures, are combined in step 306B. That is, each of the n images is generated from a different pixel subset from each image core, so that each image correlates with light received at a selected spatial arrangement. The images used in such a process may result from average values ​​within any pixel subset of the core, not necessarily concentric with the core center, as in the previous example.

[0092]

[0107] The combining in step 306B can be performed according to given weightings to generate the final image 308B. The weightings used for combining can be predetermined by a calibration process or can be derived in an optimization process. For example, the linear combination used in a given situation may be derived by optimizing the combining of a set of images for any image metric, such as contrast or signal-to-noise ratio. This optimization can be performed in a greedy manner. Prior to combining, the n images can be normalized as shown in FIG. 7e, or appropriate image normalization factors required to scale the relative amplitudes of the images can be incorporated into the weightings applied during the combining of the images.

[0093]

[0108] As shown, the particular process of FIG. 5A is a special case of the process of FIG. 5B in which images generated with a centrally located circular simulated aperture are synthesized and a weighting of −1 is applied to images with low angular discrimination.

[0094]

[0109] As can be seen in Figure 6, the resulting eDOF image shows very high fidelity with the top two gratings being resolvable over the entire 100 μm depth of the contiguous grating. However, the additional contrast in the eDOF image comes at the cost of additional noise that impacts resolution.

[0095]

[0110] Figure 8 shows image series of a portion of Groups 6 and 7 of the USAF1951 target. Each column displays images acquired when the target was placed at a depth of 10-100 μm. The top row shows the original image series constructed by integrating all the signal within each core region (R=7 pixels) and resampling onto a grid. The second and third rows are the same as the top row, but integrating only over pixel subsets with radii of R=1 and 3 pixels, respectively, centered on each core. The bottom row are eDOF images acquired by subtracting the image series of radius R=3 pixels from the image series of radius R=1 pixel. These images are used in addition to the images in Figures 4 and 6 to produce the SNR and resolution curves in Figures 6b, 6c, and 6d (scale bars: 100 μm).

[0096]

[0111] To quantify the true gain in image quality as a function of depth, the modulation depth of the grid lines of group 5 elements 3-6 and group 6 elements 1-6 is extracted as shown in Figure 8 and normalized to the noise in the image. The noise (N) is calculated by taking the standard deviation of one group of pixels in the blank area of ​​the image, and the modulation depth (M) is the average intensity difference between the grid lines and the space between the grid lines. The MATLAB Findpeaks function is used to find the grid line locations with the condition that the prominence of each peak must be ≥N / 2. If fewer than three grid lines are resolvable by this criterion, the grid is said to be unresolvable and M is set to 0. This noise-normalized modulation is called the signal-to-noise ratio (SNR=M / N). It is conceptually similar to the modulation transfer function (MTF) of an imaging system normalized by the noise-equivalent modulation.

[0097]

[0112] Plot 6b shows the SNR as a function of the grating spatial frequency at depths of 10, 50 and 100 μm for the image series of Figures 4 and 6. The SNR is the modulation depth of the grating with a given spatial frequency normalized by the noise in the image. The dotted and dashed curves at the same focus positions show the original full aperture image and the final image (corresponding eDOF series of Figure 6), respectively. At a depth of 10 μm, the SNR of the full aperture image exceeds that of the eDOF image up to 81 lp / mm before it becomes unresolvable. Despite the lower SNR at the lower spatial frequencies, the eDOF image has excellent resolution at 91 lp / mm. At larger depths, the eDOF processing becomes highly advantageous across all spatial frequencies. At 50 μm and 100 μm, the SNR of the eDOF image is equal to or exceeds that of the full aperture image at all spatial frequencies. At 50 μm, only 40 lp / mm can be resolved using the full aperture, but all gratings up to 72 lp / mm can be resolved in the eDOF image. At 100 μm, no gratings can be resolved using the full aperture, but both 40 and 45 lp / mm gratings are resolved in the eDOF image.

[0098]

[0113] These data can also be plotted as a function of depth for each spatial frequency as shown in Figure 6c. In the full aperture image, a spatial frequency of 81 lp / mm becomes unresolvable before 30 μm, while in the eDOF image it is still resolvable beyond 40 μm. The same trend is seen at 57 lp / mm, which is resolvable up to 30 μm and 70 μm depths in the full aperture and eDOF images, respectively. At this spatial frequency the depth of field is 2.3 times greater. The largest grating period imaged (40 lp / mm) is resolvable at all possible depths in the eDOF image, but only up to 60 μm in the full aperture image. Similarly, the depth of field of the highest frequency grating (81 lp / mm) is improved by a factor of two from 20 μm in the full aperture compared to 40 μm in the eDOF.

[0099]

[0114] The gain in resolution for distant objects becomes even more evident when plotting the minimum resolvable grating pitch as a function of object depth as shown in FIG. 6d. The slope of this curve indicates the effective aperture size since the radius of the blur circle predicted by geometric optics grows as Rblur=depth×tanθNA. The line corresponding to the predicted resolution of the full aperture of the MOF (NA=0.40, dashed line in FIG. 3(d)) has a slope approximately equal to the full aperture data set (dashed line in FIG. 6d). The best fit line of the eDOF data (second dashed line in FIG. 3(d)) has a slope corresponding to NA=0.15. Thus, the employed eDOF processing effectively reduces the aperture by a factor of 0.40 / 0.15=2.67 for resolution purposes.

[0100]

[0115] Next, an embodiment of the present invention was tested on a 3D object, namely the textile fibers of a protective lens pouch. Figure 9 shows "full aperture" and eDOF images of the textile fibers as seen through the MOF. In traditional "full aperture" imaging, only the textile fibers in contact with the MOF facet are imaged with high fidelity. The remaining fibers further away from the facet appear blurred, contributing to a diffuse background and resulting in lower contrast and resolution. Figure 9a shows the textile fibers imaged using a standard full aperture approach (left column) using both 10x and 20x objectives, compared to the eDOF technique according to the present invention (right column). The 10x objective provides a larger field of view than the 20x at the expense of poorer spatial resolution in the core and therefore less accurate aperture filtering. However, the employed eDOF technique provides significantly improved contrast even in the core region, which only measures up to 7 pixels in diameter. A movie was created of the textile fibers moving in the vicinity of the distal MOF facet, imaged through the MOF using the 20x objective. The bottom two rows show still images from this video 1 acquired using a 20x objective. (Scale bar: 100 μm.) The center line (frame 69) indicates the location and orientation of the intensity profile shown in Fig. 9b, respectively. Fig. 9b shows the intensity profile along the line shown in the middle row of Fig. 9a. The solid curve is the intensity profile of the eDOF image and the dotted curve is the intensity profile of the full aperture image. The dots in Fig. 9a indicate the location of three fabric fibers that are not resolvable in the full aperture image.

[0101]

[0116] Many of the fabric fibers that are not in contact with the MOF facets are still within the depth of field of the eDOF image and can still be resolved. In particular, it is the middle three fibers in the line profile that result in three separate peaks in the eDOF curve (also shown as dots), but are not visible in the full aperture curve. This shows that embodiments of this method not only improve contrast, but also fundamentally improve the resolution limit at large depths of the 3D structure.

[0102]

[0117] The preferred embodiment preferentially images light coupled into the core at a chosen angle by selecting a pixel subset from the image region that includes each fiber core to reconstruct the image. The embodiment preferentially selects more paraxial rays by selecting the central pixel. As with standard imaging devices, this reduction in collection angle is accompanied by a corresponding increase in depth of field and noise level. For high spatial frequencies at large depths that are completely suppressed in low-noise full aperture images, the high resolution of the eDOF image outweighs the additional noise, resulting in superior images in the preferred embodiment. In particular, the preferred embodiment provides double the depth of field at most spatial frequencies and potentially higher SNR for higher spatial frequencies for distant objects. It should be noted that the present embodiment is fundamentally different from image sharpening techniques such as unsharp masking, which can only rescale spatial frequencies and cannot preferentially filter light based on its input angle.

[0103]

[0118] Additionally, more advanced approaches to synthesizing images using different simulated apertures, such as high-low processing used in structured illumination, can be employed to further improve depth of field and contrast in some embodiments beyond the exemplary embodiments detailed herein.

[0104]

[0119] Embodiments of this aspect of the invention provide advantages for MOF imaging, particularly lensless microendoscopic probes, which allow for non-contact imaging without lenses or bulky scanning devices at the distal facet. This means that the MOF probe can be kept slim to reach narrow stenoses in the body. Eliminating the need for a lens assembly at the distal end also reduces the manufacturing costs of the microendoscopic probe. Even when a distal facet lens is required (e.g., to increase magnification), embodiments of the invention are applicable. In lensed MOF microendoscopic systems, the extended depth of field can occur on either side of the focal plane, not just in front of the MOF facet. Furthermore, embodiments of the technology are completely incoherent, and therefore can be used with wide-field fluorescence imaging. The incoherent nature of the technology makes it less sensitive to fiber bending, thereby eliminating the need for transfer matrix corrections after each fiber perturbation.

[0105]

[0120] As discussed elsewhere herein, images generated using the methods described herein can be advantageously employed in light field imaging or plenoptic imaging applications. In other words, applications of the present invention use MOFs as light field sensors. While images relayed by MOFs are inherently 2D, the present invention provides the observation that slim MOF-based imaging devices can record at least some aspects of the 3D structure of a sample. This is important in real-world clinical settings where samples are complex undulating structures rather than thin, flat tissue sections mounted on microscope slides.

[0106]

[0121] We show that the MOF transmits 3D image information via the modal structure within each core, and utilize this information to estimate the average light distribution of light rays impinging on each core within the MOF. This ray angle information, together with the raw transmitted image, describes what is known as the light field. Given the light field of the scene, the 3D viewpoint can be reconstructed and object depths can be calculated, and the scene can be partially refocused after acquisition.

[0107]

[0122] The light field dataset contains a full (θ,φ) parameterization of the incident light ray orientation, allowing 3D viewpoint movement, depth mapping and refocusing. Conventional light field imaging generally requires the collection of both light intensity and directional (angle) data over the entire image. In practice, this typically requires taking multiple images of the scene at multiple fields of view or focal lengths, for example using a microlens array in the imager to capture the images simultaneously. Or alternatively, the light field estimate can be obtained by taking two images at different focal positions or by measuring phase shift information.

[0108]

[0123] As can be seen, the inventors have surprisingly determined that capturing images at different focus positions, or additionally measuring phase shift information, is not necessary to realize at least some of the advantages of light field photography, and in one form the present invention provides a method by which a single image can be used to estimate the light field of a scene captured in that image.

[0109]

[0124] Using the above simulated aperture, the inventors have determined that multiple images with different effective depths of field (but the same focal position) can be created from a single captured image. These images can then be used to estimate the light field of the single captured image. Since only the average direction of ray propagation within the light field can be determined, it is necessary to apply assumptions of the angular distribution of ray propagation at each point. Despite these constraints, the resulting estimated light field will behave in the same way as any other light field image, i.e. Producing images at different focal lengths; Generating images from different perspectives; Generating a stereoscopic image by combining two images with spatially separated viewpoints; Measuring the distance to an object in an image It can be used in processes such as

[0110]

[0125] The inventors have further realised that these techniques can also be applied mutatis mutandis to multiple images of a scene captured at the same focus position and different depths of field, regardless of how the images are created (i.e., the two images do not have to be generated from a single image using the simulated aperture technique described herein, but can be captured separately in a more traditional manner using optical systems to achieve different depths of field). Note that "focus position" includes the concept of "focal length" when applied to optical systems having lenses.

[0111]

[0126] FIG. 10 illustrates a method for determining a light field approximation according to an embodiment of the present disclosure.

[0112]

[0127] Method 100 begins at step 1002 by acquiring a pair of images of a scene, each having a different depth of field and the same focus position. In a preferred form, the images may be derived using a method according to an aspect of the invention, such as that described in connection with Figure 3. A difference image is then generated from the pair of images at step 1004. This difference image is used to calculate a light field approximation at step 1006.

[0113]

[0128] As known to those skilled in the art, raw MOF images are often downsampled to remove pixilation artifacts due to discrete sampling of the core. This process assumes that the core itself does not contain any useful information. However, as discussed above in connection with the image generation aspect of the present invention, the core of a MOF is large enough to support a dozen or so modes in the visible spectrum. Incoherent superposition of such modes is readily observed at the output facet of the MOF. As the angle of incidence of the input light increases, higher order modes are preferentially excited. As a result, as the input incidence angle increases, the sub-core output is transformed from a central Gaussian spot (fundamental mode) into an expanding ring. In other words, light incident at an oblique input angle tends to give rise to output light that is localized at the core periphery. Conversely, light incident at a small angle preferentially remains at the core center (see FIG. 7d). The depth of field (DOF) of the image can be improved by generating the image using the above method. This does not teach the full orientation of the light rays required for 3D light field imaging. To extract this information, the technique of "light field moment imaging" (LMI) is employed, which relates the axial intensity derivative of the image to the average (first moment) ray orientation at each pixel. LMI is described in Orth. A. and Crozier. KB, 2013, Light field moment imaging. Optics letters, 38(15), pp. 2666-2668, the contents of which are incorporated herein for all purposes.

[0114]

[0129] The LMI described therein requires as input a pair of images with slightly different focus positions. However, as mentioned above, bare MOF imaging probes do not have delicate focus control devices. Instead, embodiments of the present invention use images with different depths of field, for example using simulated apertures as described herein. The inventors have noticed that images with small simulated aperture sizes with large depths of field resemble focused images of objects located far from the fiber facet. Similarly, large defocused images with small depths of field created by large simulated aperture sizes are intuitively similar to defocused images of objects located far from the fiber facet. Images of large simulated apertures can include full aperture images. Hereafter, this approximation is referred to as the "aperture-focus approximation". The LMI algorithm can then be used to extract the angular light field moments and construct a light field estimate by the following equation:

[0115]

number

[0116]

[0130] where the two images I1 and I2 forming the image pair are small and large simulated aperture images, respectively, and M x and M y are the average tilt angles of the light rays from the z-axis to the x-axis and y-axis, respectively (light field moments). Here Δz is not well-defined because the two images are used at different effective apertures rather than different focal positions. As a result, Δz is set to an unknown magnification that is adjusted later. The value of the constant Δz does not affect the resulting image, but simply sets the absolute scale of the resulting parallax.

[0117]

[0131] An experimental realization of this approach is shown in Figure 11. First, images of the large and small simulated aperture target (I1 and I2, respectively) are obtained. I2 is acquired as described above. However, in the large aperture image I1 (top left of Figure 11), the entire core region R (=5px) is used to generate the image in the same way as the full aperture image in Figure 4 is created. The small aperture image I2 (top center of Figure 11) is generated from a small central pixel subset of the region (R=1px).

[0118]

[0132] Because a lensless MOF is used, the entire scene appears more in focus in I2 than in I1 due to the limited aperture, emulating the defocusing process typically associated with LMI. The subtle difference ΔI between these two images is directly visualized in Figure 11 (top right).

[0119]

[0133] Using I1 and I2, ∇U = [M x ,M y ], which is related to the light field moments via the scalar potential U in Fourier space, x and M y Then, the Gaussian light field estimation L can be solved by x and M y It is constructed using the following:

[0120]

number

[0121]

[0134] where u and v are the tilt angles from the z axis to the x and y axes, respectively. The parameter σ is empirically set to tan θ′, and the light field moments are max{M x 2 +M y 2}=σ 2 This ensures that the average light field moment is inside the collection aperture.

[0122]

[0135] The Gaussian form L in (u,v) space is a manifestation of the fact that if a light field is densely sampled in the spatial dimension (x,y), it will necessarily be low-pass filtered in the angular (u,v) dimension due to the diffraction limit. In the most extreme case, this can result in a light field where the angular dimension contains a single broad spot that effectively reports the slope of the light ray (i.e., the wavefront) at each spatial location, similar to a Shack-Hartmann wavefront sensor.

[0123]

[0136] Once the light field L has been estimated according to an embodiment of the present invention, further image processing can be performed as required.

[0124]

[0137] In one example, the virtual viewpoint of a 3D scene can be varied by choosing a 2D slice (fixed angle (u,v) coordinates) of the 4D light field L. For example, the image of the scene as seen from horizontally opposed viewpoints is shown in Figure 11 (bottom left, bottom center). L = L(x,y,u0,0) and I R =L(x,y,-u0,0). These images form a stereo pair that can then be combined into a red-cyan stereo anaglyph as shown in Figure 11 (bottom right). This image can be viewed using red-cyan glasses to create a 3D effect. It is also possible to construct parallax scanning animations, where the scene is viewed from a moving virtual viewpoint. This method is particularly useful for conveying 3D information via motion parallax.

[0125]

[0138] Parallax is the result of depth changes (depth = distance from the fiber facet to the object) in a 3D scene. Assuming a light field L that contains parallax information for all angular directions, a depth map can be computed. This can be done using the method described in Adelson EH and Wang, JY, 1992. Single lens stereo with a plenoptic camera. IEEE transactions on pattern analysis and machine intelligence, 14(2), pp. 99-106.

[0126]

number

[0127] where d is the distance from the fiber facet to the object at position (x,y).

[0128]

[0139] In the following, d is called the "depth metric" due to the aperture-focus approximation. x and L y are the (discrete) partial derivatives of L in the x and y directions, respectively (similarly, L u and L v are in the u and v directions). The summation is performed over an image patch P centered at (x,y) and moving over all (u,v) coordinates. The size of the image patch can be adjusted depending on the desired smoothness of the result. A typical size is 9x9 pixels or larger. The resulting depth map for a sequence of images of a USAF target (group 5) transilluminated with white light is shown in Figure 12a. Figure 12b is a plot of the depth metric as a function of the known ground truth depth (distance from the MOF fiber facet to the USAF target) relative to the target. The depth metric values ​​are the average over all pixels at each ground truth depth. The error bars indicate one standard deviation of the measured depth metric for each ground truth depth.

[0129]

[0140] The data set in Fig. 12a is processed simultaneously in its entirety so that the constant Δz is the same for each image. As expected, the toned depth metric d shows that the USAF target moves farther from the fiber facet as the ground truth (applied via a manual micrometer step) increases from 0 to 90 μm. When a vertical tilt is applied to the USAF target, the depth change is clearly visible in the toned image where the top is closer to the fiber facet than the bottom. As a result of the aperture / focus approximation, the relationship with the depth metric is slightly non-linear, especially at short distances, as shown in Fig. 12(b). Nevertheless, useful 3D information such as the relative depth level of the object in the scene can still be obtained. If necessary, the depth value can be rescaled to its true value using a calibration lookup table from measurements of the specimen at known distances.

[0130]

[0141] Another well-known application of light field imaging is synthetic refocusing. The data contained in the light field allows for rearrangement of the spatial-angular structure of light in order to digitally change the focus of the image after capture. This is often easily realized by first imaging the 3D scene at all viewpoints in (u,v) space. To create a synthetically refocused image at a given depth, it is first necessary to correct for the parallax that may be generated for objects at each viewpoint at said depth. This means a translational shift of the image in (x,y) space proportional to the (u,v) vector describing the viewpoint coordinates. Once this parallax is taken into account, the shifted images are summed together to create the synthetically refocused image (this is sometimes called the "shift-and-add" method). Even with the aperture-focus approximation, synthetic refocusing is possible using light field estimates obtained from MOF images using embodiments of the present invention.

[0131]

[0142] Figure 13a shows USAF target images at 150 μm refocused at various distances increasing from left to right. The light field estimate L is then computationally refocused to successively larger distances from the fiber facet (image order is close → far from left to right). The image, which is initially out of focus (leftmost panel (13a)), becomes hyperfocal and blurred (rightmost panel (13a)) after converging to focus (middle panel (13a)). Synthetic refocusing was then tested for a range of distances from the fiber facet to the target. Figure 13b shows original images of the USAF target at distances from 0 to 250 μm in the top row (scale bar: 100 μm). The bottom row shows the sharpest synthetically refocused image judged by eye for each distance. Note that refocusing here is performed after image capture, so the target is not moved to achieve this effect. This result is purely computational. Also, the difference between the leftmost image in Fig. 13a and the "original" 150 μm image in Fig. 13b is due to the Gaussian light field model used to construct the light field estimate. Fig. 13a results from this light field estimate, whereas the "original" image in Fig. 13b is acquired before or without the light field estimate, since there is no need for light field data. A clear sharpening can be seen in the 50 μm and 100 μm distance images once synthetic refocusing has been performed. One might argue that these can be obtained by appropriate deconvolution or unsharp masking. However, the latter cannot be said for the 150-250 μm refocused images, where the grid lines, which are completely blurred in the original image, are resolved in the optimally refocused image.

[0132]

[0143] As mentioned above, there are various approaches to 3D visualization of objects using the images provided by the present invention. For example, the 3D structure of a scene can be directly observed by stereoscopic images such as stereophotography (e.g., using red-cyan stereo glasses or VR goggle devices) and stereo anaglyphs or viewpoint-shifting (parallax) animation. Alternatively, depth mapping techniques can be applied, for example using depth maps constructed by maximum intensity projection of deconvolved light field focal stacks.

[0133]

[0144] As can be seen, the image processing methods described herein enable the use of MOFs as light-field imaging elements. The use of MOFs for light-field imaging allows the use of significantly slimmer endoscopes than existing rigid stereoscopic endoscopes that rely on a pair of separate optical imaging paths to record stereo data.

[0134]

[0145] Even more advantageously, preferred forms of the technology disclosed herein do not require hardware modifications to MOF-based systems, as all data required for light field estimation is contained within the individual cores.

[0135]

[0146] Studies involving imaging scattered animal tissues using the present invention on cellular structures (specifically, 5 mm slices of mouse brain stained with proflavine imaged via a fiber bundle) have shown very good quantitative agreement between the proflavine depth distribution measured by the light field approach of the present invention and that obtained using a tabletop confocal microscope.

[0136]

[0147] Figure 14 is a block diagram illustrating an exemplary computer processing system 400 suitable for use / configuration as an image processing system for processing images captured by the camera CAM of the system of Figure 2, according to any of the various aspects and embodiments described herein. The image processing system may be a separate computer system (possibly located remotely from the imaging components) or may form part of the control system of the MOF imaging system.

[0137]

[0148] Computer processing system 400 comprises a processing unit 402. Processing unit 402 may comprise a single computer processing device (e.g., a central processing unit, a graphics processing unit, or other computing device) or may comprise multiple computer processing devices. In some examples, processing is performed solely by processing unit 402, while in other examples, processing may also alternatively be performed by a remote processing device accessible and usable by computer processing system 400 (either shared or dedicated).

[0138]

[0149] Via communication bus 404, processing unit 402 is in data communication with one or more machine-readable storage (memory) devices that store instructions and / or data for controlling the operation of computer processing system 400. In this example, computer processing system 400 includes system memory 406 (e.g., BIOS or flash memory), volatile memory 408 (e.g., random access memory such as one or more DRAM modules), and non-volatile / non-transitory memory 410 (e.g., one or more hard disks or solid state drives).

[0139]

[0150] Computer processing system 400 also includes one or more interfaces, generally indicated at 412, through which computer processing system 400 interfaces with various components, other devices and / or networks. The other components / devices may be physically integrated with computer processing system 400 or may be physically separate. If such devices are physically separate, their connections to computer processing system 400 may be via wired or wireless hardware and communication protocols, and may be direct or indirect (e.g., networked) connections.

[0140]

[0151] Wired connections to other devices / networks may be via any standard or proprietary hardware and connection protocol. For example, computer processing system 400 may be configured to connect to other devices / communications networks via one or more of the following: USB, FireWire, eSATA, Thunderbolt, Ethernet, parallel, serial, HDMI, DVI, VGA, AudioPort. Other wired connections are possible.

[0141]

[0152] The wireless connection with other devices / networks may similarly be by any standard or proprietary hardware and communication protocol. For example, the computer processing system 400 may be configured to wirelessly connect with other devices / communication networks using one or more of the following: infrared, Bluetooth (including early versions of Bluetooth, Bluetooth 4.0 / 4.1 / 4.2 (also known as Bluetooth Low Energy) and future versions of Bluetooth), WiFi, Near Field Communication (NFC), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Long Term Evolution (LTE), Wideband Code Division Multiple Access (W-CDMA), and Code Division Multiple Access (CDMA). Other wireless connections are possible.

[0142]

[0153] Generally speaking, the devices to which computer processing system 400 connects, whether by wired or wireless means, enable data to be input / received by computer processing system 400 for processing by processing unit 402, and data to be output by computer processing system 400. Exemplary devices are described below, however, it will be understood that not all computer processing systems will include all of the devices described, and that devices in addition to and alternative to those described may also be used.

[0143]

[0154] For example, the computer processing system 400 may include or be connected to one or more input devices through which information / data is entered into (received by) the computer processing system 400. Such input devices may include physical buttons, alphanumeric input devices (e.g., keyboards), pointing devices (e.g., mice, trackpads, etc.), touch screens, touch screen displays, microphones, accelerometers, proximity sensors, GPS devices, etc. The computer processing system 400 may also include or be connected to one or more output devices 414 that are controlled by the computer processing system 400 to output information. Such output devices may include indicators (e.g., LEDs, LCDs, or other lights), displays (e.g., LCD displays, LED displays, plasma displays, touch screen displays), audio output devices such as speakers, vibration modules, and other output devices. Computer processing system 400 may include or be connected to devices capable of both input and output, such as memory devices (hard drives, solid state drives, disk drives, compact flash cards, SD cards, etc.) from which computer processing system 400 can read data and / or write data, and a touchscreen display capable of both displaying data (output) and receiving touch signals (input).

[0144]

[0155] The computer processing system 400 may also be connected to a communications network (e.g., the Internet, a local area network, a wide area network, a personal hotspot, etc.) to exchange data with networked devices, which may be other computer processing systems.

[0145]

[0156] The architecture shown in FIG. 14 can be implemented in a variety of computer processing systems, such as laptop computers, netbook computers, tablet computers, smartphones, desktop computers, and server computers. It will also be understood that FIG. 14 does not show all functional or physical components of a computer processing system. For example, a power source or power interface is not shown, but computer processing system 400 may hold and / or be connected to a power source (e.g., a battery). It will further be understood that a particular type of computer processing system will determine the appropriate hardware and architecture, and that alternative computer processing systems may have additional, alternative, or fewer components than shown, may combine two or more components, and / or may have a different configuration or arrangement of components.

[0146]

[0157] The operations of the computer processing system 400 may also be effected by one or more computer program modules that configure the computer processing system 400 to receive, process, and output data.

[0147]

[0158] As used herein, the term "module" refers to computer program instructions and other logic for providing a particular function. Modules may be implemented in hardware, firmware, and / or software. Modules are typically stored in storage device 408, loaded into memory 406, and executed by processor 402.

[0148]

[0159] A module may include one or more processes and / or only a portion of a process may be provided. The embodiments described herein may include additional and / or different modules than those described herein. Also, functionality attributed to a module may be performed by another or different module in other embodiments. Moreover, this description may at times omit the term "module" for convenience and clarity.

[0149]

[0160] The type of computer processing system 400 used may vary depending on the embodiment and the processing power used by the entity. For example, a server system may include multiple blade servers working together to provide the functionality described herein.

[0150]

[0161] As can be seen, the inventive approach is camera frame rate limited, does not require calibration, is robust to moderate fiber bending, and is suitable for potential clinical applications.

[0151]

[0162] Other incoherent imaging modalities such as bright field imaging can also be modified to this approach and used with fiber bundles employing digital lenses.

[0152]

[0163] As discussed above, embodiments of the present invention are concerned with the relationship between the in-core intensity pattern and the angular dimension of the light field incident on the distal end of the fiber bundle. The analysis contained in Appendix A provides a quantification of this relationship.

[0153]

[0164] The essence of this relationship is the fact that the regular LMI equation (equation 2 above) is modified for application to image pairs with the same focal position but different collection apertures. This arises because the centroid shift of a point source (stereo parallax, or lateral shift) is not linear in z as it would be for a standard light field.

[0154]

[0165] While the above disclosure pertains to embodiments of the invention that generate or modify images using a "simulated aperture" technique applied to fiber cores, it will be appreciated that other methods of processing or analyzing the image intensity pattern across each core can be used to extract light field angle information for that core. For example, a pattern matching algorithm can be applied that compares the image intensity pattern to a stored pattern generated for the MOF by a pattern calibration process. The calibration process involves acquiring reference images for a point source at each of a number of angles. These reference images are then used to generate a stored pattern for each core that can be compared to the received image using standard computational pattern matching algorithms.

[0155]

[0166] It will be understood that the invention disclosed and defined in this specification extends to all alternative combinations of two or more individual features described or apparent from the text or drawings, all of whose different combinations constitute various alternative aspects of the invention.

[0156] Appendix A Operating principle Consider a point source imaged using an optical fiber bundle, at a distance z from the fiber facet. A ray makes an angle θ with the fiber facet at a position x,y from the centerline of the fiber bundle, and θ x and θ y are the inclination angles of the ray from the yz and xz planes, respectively. To demonstrate this, a raw output image of a fluorescent bead at axial distance z = 26 μm received at the proximal end of the fiber bundle is shown in Figure 15a (scale bar 5 μm). The radially symmetric pattern of the fiber modes is visible in the mode coupling efficiency versus input ray angle θ. The fiber bundle used in this work has an outer diameter of 750 μm and contains up to 30,000 substantially circular cores with an average center-to-center spacing of 3.2 μm, an average core radius a = 1 μm, and a numerical aperture (NA) of 0.39. On average, each core in this fiber bundle has an optical fiber distribution of approximately (2πaNA / λ) at λ = 550 nm. (24) 2 / 2≒10 modes are supported. As discussed elsewhere herein, post-processing of image data allows for digital manipulation of the numerical aperture (NA) of the fiber. This relies on the fact that higher order modes, which are preferentially excited at larger angles of incidence, carry more energy near the core / cladding boundary than lower order modes. As the ray angle increases, the light is effectively steered toward the edge of each core. A synthetically limited NA is achieved by the digital aperture filtering approach of an embodiment of the invention (which selectively removes light near the periphery of each core). This is illustrated in FIG. 15, particularly in FIGS. 15c and 15d. The full orientation of the input light cannot be ascertained from this observation alone due to the azimuthal degeneracy of the modes in the core. To address this, LMI is applied. In LMI, the continuity equation describing the conservation of energy between the two image planes is used to calculate the (light field moment vector

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

Claims

1. 1. A method of operating an image processing system for generating one or more images from light received by an imager through multiple waveguides, the light being generated from a light field incident on the multiple waveguides, the image processing system comprising: receiving a digital image including a plurality of regions each corresponding to a waveguide core and including a plurality of pixels; processing the image intensity pattern over each of said regions to determine a light field angle dimension for that region; applying the angular dimension to one or more pixels contained in each region to generate one or more sets of modified image data; generating one or more images using the one or more sets of modified image data.

2. 2. The method of claim 1, wherein processing the image intensity pattern across each of the regions includes analyzing each region with a simulated aperture technique, the simulated aperture technique involving a computational comparison of image intensity under a first computational aperture and image intensity under a second computational aperture for each region.

3. The method of claim 2 , wherein pixels in one of the first and second computational apertures comprise a subset of pixels in the other of the first and second computational apertures.

4. The method of claim 2 , wherein the set of pixels in each calculation aperture is different, and the set of pixels is selected according to the particular light field angle dimension extracted from the processing step.

5. a multi-core optical fiber (MOF) extending from a proximal end to a distal end; a light source for illuminating a scene at the distal end of the MOF; and an imaging device positioned relative to the proximal end of the MOF to capture an image of light propagated along the MOF; and 5. An imaging system comprising: a data processing system configured to receive an image captured by the imaging device, the data processing system configured to execute instructions that cause the data processing system to perform the method of any one of claims 1 to 4.

6. a multi-core optical fiber (MOF) extending from a proximal end to a distal end, the MOF providing multiple waveguides; a light source for illuminating a scene at the distal end of the MOF to generate a light field incident on the various waveguides; an imaging device positioned relative to the proximal end of the MOF to capture a digital image of light propagated along the MOF, the digital image including a plurality of regions, each of the regions corresponding to a waveguide core of one of the plurality of waveguides and including a plurality of pixels; and a data processing system, The data processing system comprises: receiving the digital image captured by the imaging device and processing the image intensity pattern across each of the regions to determine a light field angular dimension for that region; applying the angular dimension to one or more pixels in each region to generate one or more sets of modified image data; An imaging system configured to generate one or more images using the one or more sets of modified image data.

7. The imaging system of claim 5 or claim 6, wherein the MOF comprises an endoscope.

8. 5. An image processing system comprising at least one processing unit and at least one memory for storing instructions executed by said at least one processing unit, said instructions being configured, when executed, to perform the method according to any one of claims 1 to 4.

Images