Imaging process, endoscope and computer program product
Multimode optical fibers with a square section, leveraging a "quadruple memory effect," enable calibration-free, flexible, and minimally invasive endoscopic imaging, addressing the limitations of existing technologies.
Patent Information
- Application Number
- FR2021007170
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-01
- Publication Date
- 2025-05-23
- Estimated Expiration
- 2041-07-01
AI Technical Summary
Existing endoscopes with single-mode fiber bundles are bulky and difficult to use in minimally invasive procedures, while multimode fiber endoscopes require prior calibration and are limited by fiber shape changes, making them non-user-friendly and inflexible.
The use of multimode optical fibers with a square section, which exhibit a "quadruple memory effect" allowing for image reconstruction without prior calibration, even when the fiber is curved, enabling flexible and minimally invasive imaging.
This approach eliminates the need for calibration, reduces the diameter of the imaging probe, and allows for flexible fiber configurations, resulting in a more user-friendly and minimally invasive endoscopic imaging system.
Smart Images
Figure 00000021_0000 
Figure 00000021_0001 
Figure 00000021_0002
Abstract
Description
Title of the invention: Imaging method, endoscope and computer program product
[0001] The present invention relates to the technical field of imaging, and relates more particularly to an imaging method and to an endoscope implementing the imaging method.
[0002] Currently, most endoscopes use a bundle of single-mode fibers to produce an image of an object to be imaged placed at the output of the fiber bundle: fiber by fiber of the fiber bundle is lit by regularly scanning the fiber bundle, then the light received from the object to be imaged is collected for each fiber to deduce an image of the object to be imaged pixel by pixel.
[0003] The imaging results obtained by these existing endoscopes are good. However, these state-of-the-art endoscopes have drawbacks.
[0004] Essentially, the single-mode fiber bundle has a large diameter, which makes prior art endoscopes difficult to use in procedures where the imaging probe of the endoscope, consisting of the single-mode fiber bundle, must be minimally invasive.
[0005] Optical imaging in multimode optical fibers is a field of research that has been developing for about ten years. In particular, endoscopic optical imaging through multimode optical fibers makes it possible to produce endoscopic imaging systems with very small dimensions compared to the conventional approach based on bundles of single-mode optical fibers. Standard multimode optical fibers (with circular cross-section) are generally used, without an optical component on the sample side (terminology "lensless endoscope") and prior calibration is necessary to produce an image, due to the fact that a multimode optical fiber is a complex propagation medium. US patent application US2015 / 0015879A1 and US patents US5956447 and US10254534B2 disclose such examples of multimode fiber endoscopes.These endoscopes can be used in procedures where the endoscope's imaging probe is minimally invasive.
[0006] However, in multimode fiber, due to the use of a single medium to propagate several modes, the information is mixed / scrambled during propagation, compensation for this interference can only be done by means of prior calibration, making the use of such multimode fiber endoscopes long and not very user-friendly.
[0007] Furthermore, the calibration depends on the shape of the multimode optical fiber. If the shape If the multimode optical fiber changes after calibration, the calibration must be repeated, even for very small shape changes. To overcome the effects of the shape change of the multimode optical fiber, rigid multimode optical fibers are used, which greatly limits the areas of application and the practical use of the endoscope.
[0008] The possibility of imaging in flexible fibers has only been demonstrated with a particular type of optical fibers, called multi-core fibers, for which the calibration phase remains approximately valid even after a change in fiber conformation.
[0009] However, these fibers are similar in principle to bundles of single-mode fibers, and therefore have the same disadvantage in terms of size, namely having a section typically 20 to 30 times greater than the multimode optical fibers of the invention.
[0010] There is therefore a need for a minimally invasive endoscope, without a prior calibration step, which can allow the use of a flexible optical fiber, whatever its configuration (straight or curved) for greater comfort of use of the endoscope, which the invention proposes to resolve with the use of a multimode optical fiber with a square section.
[0011] In these multimode optical fibers with a square section, there is a translation memory effect that makes it possible to dispense with the calibration phase. Indeed, it has been demonstrated that a memory effect present in the case of scattering media makes it possible to image an object for imaging without prior calibration of the scattering medium. Note that this assumes a memory effect in all directions. In multimode optical fibers with a circular section, a rotation memory effect (therefore in one direction only) has been described, which is insufficient to produce an image.
[0012] The method of the invention for forming an image of the object to be imaged is based on two steps, 1) a measurement step, and 2) a step of reconstructing the object. Its strength lies in the fact that there are no calibration steps, and that it applies even when the fiber is curved.
[0013] The inventors have shown that the optical field at the output of the square-section multimode optical fiber is decomposed into the sum of four optical fields, all of which translate when the input lighting pattern is translated. This "quadruple memory effect" is a generalization of the memory effect well known in the community, which corresponds to the translation of a single output field when the input field translates.
[0014] The core of the invention is based on the existence of the "quadruple" memory effect in square-section multimode optical fibers, for reconstruction without calibration. Moreover, this effect has been observed even when the square-section multimode optical fiber is curved, which makes the method of the invention applicable to flexible fibers (provided that the fiber does not move during the measurement).
[0015] The present invention therefore relates to a method for imaging an object to be imaged, characterized in that it comprises:
[0016] - generating, by a lighting device, a series of lighting patterns;
[0017] - for each lighting pattern, carry out a stimulation phase in which translations of the lighting pattern are carried out at the input of a square-section multimode optical fiber by means of at least one translation device of an optical system arranged between the lighting device and the input of the square-section multimode optical fiber, the translations being carried out in a plane parallel to the input plane of the square-section multimode optical fiber;
[0018] - for each stimulation phase, carry out an acquisition phase in which the signal generated by the illumination pattern on the object to be imaged placed at the output of the square-section multimode optical fiber is measured by a signal measuring device; and
[0019] - reconstructed by a calculation device, using the signal measured for each acquisition phases, the object to be imaged.
[0020] The invention is based on the use of multimode optical fibers with a square cross-section. Unlike the fibers used conventionally, which have a circular cross-section, the inventors have demonstrated theoretically and experimentally that there exists in these multimode optical fibers with a square cross-section a translation invariance property (called in the literature translation memory effect: the pattern at the output of the medium (here the multimode optical fiber with a square cross-section) translates when the lighting pattern is translated), an effect which does not exist in multimode optical fibers with a circular cross-section.Exploiting this memory effect in square-section multimode optical fibers eliminates the need for the calibration phase (totally or partially), and in particular allows images to be produced in flexible fibers (whose shape can be modified during use), two limitations of current methods based on standard circular-section fibers, which require calibration and which only work for a fixed shape of the fiber. Square-section multimode optical fibers are an existing commercial product, used for controlling the shape of an illumination pattern. The invention relates to the completely new use of these fibers for endoscopic optical imaging.The square-section multimode optical fiber allows the fiber diameter to be greatly reduced compared to current endoscopes with single-mode fiber bundles, allowing for a cross-section of the square-section multimode optical fiber typically 20 to 30 times smaller than the single-mode fiber bundles of the state of the art, and therefore helps to reduce the intrusive nature of the endoscope to obtain a miniature device.
[0021] No optics are present at the output of the multimode optical fiber with section square, between the output of the square-section multimode optical fiber and the object to be imaged, which makes the fiber output structure extremely simple and robust.
[0022] The number of lighting patterns in the series must be greater than or equal to one. However, the greater the number of lighting patterns in the series, the better the reconstruction of the object to be imaged.
[0023] The lighting pattern is preferably a speckle pattern but may also be a focused spot, without departing from the scope of the present invention.
[0024] The optical system may comprise one or more optical components such as lenses, mirrors, gratings, prisms, diffusers, splitters, diaphragms, delay plates, polarizers, filters, beam splitters, beam expanders, concentrators, micro-mirror arrays, etc.
[0025] According to one embodiment, the at least one translation device is one of a liquid crystal-based spatial light modulator, a spatial light modulator of the micro-mirror matrix type (or DMD, acronym for Digital Micro-mirror Device), an optical diffuser associated with a translation stage, an optical diffuser associated with galvanometric mirrors. The optical diffusers may in particular be a diffusing glass.
[0026] According to one embodiment, the signal measuring device is an optical sensor for measuring the optical signal reflected by the object to be imaged associated with a detection optical fiber placed between the object to be imaged and the optical sensor, the detection optical fiber being one of the square-section multimode optical fiber and an auxiliary optical fiber. A beam splitter is present in the optical system for transferring the optical signal generated by the illumination pattern on the object to be imaged in the square-section multimode optical fiber to the optical sensor.
[0027] According to one embodiment, the signal measuring device is a fiber optic hydrophone measuring the photoacoustic signal of the object to be imaged. The optical fiber of the hydrophone is then connected to a photoacoustic detection device by its end opposite that facing the object to be imaged.
[0028] According to one embodiment, the object to be imaged is fluorescent and the signal measuring device is an optical sensor for measuring the fluorescence signal emitted by the object to be imaged associated with a detection optical fiber placed between the object to be imaged and the optical sensor, the detection optical fiber being one of the square-section multimode optical fiber and an auxiliary optical fiber. A dichroic filter is then present in the optical system to filter the optical signal generated by the illumination pattern on the object to be imaged in the square-section multimode optical fiber or the auxiliary optical fiber, which filtered signal is then sent by the optical system to the optical sensor.
[0029] According to one embodiment, in each stimulation phase, the translations are carried out in all or part of the input plane of the square-section multimode optical fiber.
[0030] According to one embodiment, the lighting device is a coherent light source. The lighting device must in fact lead to a speckle pattern at the output of the square-section multimode optical fiber. The lighting device will preferably be monochromatic, and more preferably a laser.
[0031] According to one embodiment, the computing device is selected from at least one of a computer, a microprocessor, a digital signal processor, DSP, a processor, a microcontroller, a programmable gate array, FPGA, an application-specific component, ASIC, and includes memory.
[0032] According to one embodiment, the reconstruction of the object to be imaged by the calculation device comprises a first step comprising, for each acquisition phase, the calculation of the spatial autocorrelation of the measured signal then the calculation of the averaged autocorrelation for all the lighting patterns, and a second step of deducing the object to be imaged from the averaged autocorrelation for all the lighting patterns obtained in the first step.
[0033] According to one embodiment, the object to be imaged is deduced from the averaged autocorrelation for all the lighting patterns by solving the following equation:
[0034] [Math.l] Ôestimated ( r ) = argminx || ,ure ( A r(-, ) - JJ di A ry„, A r( JcX( A rfJd2 A rout ||
[0035] where A rin = ( A xin, Ay} defines a scanning position in the input face of the square-section multimode optical fiber with xin and 3 the coordinates in the input face of the square-section multimode optical fiber, A roli! represents the coordinates of the output face of the square-section multimode optical fiber, q1 is the average autocorrelation of the N illumination patterns, ^Y is the autocorrelation of a test object X, p5 is the averaged autocorrelation for all illumination patterns and ^measure Ôestimated (r) is the estimate obtained from the object (O) to be imaged.
[0036] According to one embodiment, the object to be imaged is deduced from the averaged autocorrelation for all illumination patterns by a neural network previously trained on a defined class of objects, the neural network providing an estimation of the object to be imaged from the averaged autocorrelation, by comparison of the averaged autocorrelation known by the neural network on the objects of the defined class to the averaged autocorrelation calculated for the object to be imaged.
[0037] A U-NET type neural network can for example be used as the neural network. The method used is for example described in the scientific publication “Compensating for visibility artifacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties”, Guillaume Godefroy, Bastien Arnal, Emmanuel Bossy, Photoacoustics, 21:100218, 2021. 4.
[0038] The training dataset for the neural network with this method consists of a series of known test objects and their average autocorrelation function (also known). The objects in the series must belong to a defined object class (e.g., a series of handwritten characters, cell populations, blood vessels), and the network will reconstruct an unknown object provided that it is part of the family from which the network was trained. An example of a widely used known object class is the MNIST database of handwritten digits (https: / / en.wikipedia.org / wiki / MNIST_database).
[0039] According to one embodiment, the square-section multimode optical fiber is flexible. The invention thus allows imaging with a flexible fiber, with the possibility that the shape of the square-section multimode optical fiber changes between two measurements, provided that it does not change during a given measurement, allowing more flexibility for the imaging method and therefore greater ease of use.
[0040] According to one embodiment, the square-section multimode optical fiber comprises a sheath covering a square-section core, the core having dimensions of between 10 pm*10 pm and 1 mm*1 mm, preferably between 50 pm*50 pm and 200 pm*200 pm, more preferably between 100 pm*100 pm and 150 pm*150 pm. The miniaturization objective of the invention is thus achieved with fiber diameters 20 to 30 times smaller than the single-mode fiber bundles of the prior art.
[0041] The invention also relates to an endoscope for imaging an object to be imaged, characterized in that it comprises a lighting device, an optical system, at least one multimode optical fiber with a square section, the optical system optically coupling the lighting device to the input of the at least one multimode optical fiber with a square section, the object to be imaged being configured to be placed at the output of the at least one multimode optical fiber with a square section, a device for measuring the signal generated by the object to be imaged and a calculation device for implementing the method as described above.
[0042] Given the diameter of the square-section multimode optical fiber, the endoscope according to the invention is thus minimally invasive and does not require any prior calibration phase, greatly reducing the imaging time.
[0043] According to one embodiment, the optical system comprises a device for translation for translating a lighting pattern generated by the lighting device over all or part of the input plane of the at least one square-section multimode optical fiber, the translation device being configured to translate the lighting pattern in a plane parallel to the input plane of the at least one square-section multimode optical fiber, the translation device being one of a liquid crystal-based spatial light modulator, a micromirror matrix (DMD) type spatial light modulator, an optical diffuser associated with a translation stage, an optical diffuser associated with galvanometric mirrors. The optical diffusers may in particular be a diffusing glass.
[0044] According to one embodiment, the signal measuring device is constituted by at least one of:
[0045] - at least one multimode optical fiber with square section associated with a sensor optical input of the square-section multimode optical fiber to capture an optical signal reflected by the object to be imaged, the optical system then comprising a beam splitter to send the signal generated by the object through the at least one square-section multimode optical fiber to the optical sensor,
[0046] - at least one auxiliary optical fiber associated with an optical sensor on the input side the endoscope's square-section multimode optical fiber to capture an optical signal reflected by the object to be imaged,
[0047] - a fiber optic hydrophone for capturing the photoacoustic signal from the object to imager, associated with a photoacoustic detection device.
[0048] The at least one multimode optical fiber with a square section will preferably be attached, if necessary, to the auxiliary optical fiber or to the hydrophone optical fiber for the purpose of reducing bulk to make the intrusive character minimal for the end of the endoscope.
[0049] According to one embodiment, when the object to be imaged is fluorescent, the signal measuring device is constituted by at least one of:
[0050] - at least one multimode optical fiber with square section associated with a sensor optical input of the square-section multimode optical fiber to capture a fluorescence signal from the object to be imaged, the optical system then comprising a dichroic filter to send the signal generated by the object through the at least one square-section multimode optical fiber to the optical sensor,
[0051] - at least one auxiliary optical fiber associated with a sensor on the input side of the fiber square-section multimode optics of the endoscope for capturing a fluorescence signal from the object to be imaged, the optical system then comprising a dichroic filter for sending the signal generated by the object through the at least one auxiliary optical fiber to the optical sensor.
[0052] According to one embodiment, the computing device is selected from at least one of a computer, a microprocessor, a digital signal processor, DSP, a processor, a microcontroller, a programmable gate array, FPGA, an application-specific component, ASIC, and includes memory. The computing device may be part of the endoscope according to the invention or be remote and connected to the endoscope.
[0053] According to one embodiment, the at least one multimode fiber with a square section is flexible. It is thus possible to have a flexible endoscope.
[0054] According to one embodiment, the at least one multimode fiber with a square section comprises a sheath covering a core with a square section, the core having dimensions of between 10 pm*10 pm and 1 mm*1 mm, preferably of between 50 pm*50 pm and 200 pm*200 pm, more preferably of between 100 pm*100 pm and 150 pm*150 pm.
[0055] The invention also relates to a computer program product, characterized in that it comprises instructions which, when loaded and executed on a computing device of an endoscope as described above, make it possible to implement the method as described above.
[0056] To better illustrate the object of the present invention, particular embodiments thereof will now be described, in conjunction with the appended drawings.
[0057] In these drawings:
[0058] [Fig.l] represents an endoscope according to a first embodiment of the invention;
[0059] [Fig.2] represents an endoscope according to a second embodiment of the invention;
[0060] [Fig.3] represents an endoscope according to a third embodiment of the invention;
[0061] [Fig.4] schematically represents a multimode fiber with square section according to the invention; and
[0062] [Fig.5] schematically represents the steps of the imaging method according to the invention.
[0063] If we refer to [Fig.l], we can see that an endoscope 1 has been shown according to a first embodiment of the invention.
[0064] The endoscope 1 comprises a lighting device 2, generating coherent light, preferably monochromatic, preferably a laser.
[0065] Opposite the lighting device 2 in the endoscope 1 is an optical system 3 which sends the light generated by the lighting device 2 into a square-section multimode optical fiber 4. The output of the square-section multimode optical fiber 4 opens directly (without an optical element) onto an object to be imaged O. A measuring device 5 is connected to the optical system 3 and to a calculation device 6.
[0066] The optical system 3 comprises a translation device 3a for translating a lighting pattern generated by the lighting device 2 on all or part of the input plane of the at least one square-section multimode optical fiber 4. The translation device 3a may be one of a liquid crystal-based spatial light modulator, a micro-mirror matrix type spatial light modulator, an optical diffuser associated with a translation stage, an optical diffuser associated with galvanometric mirrors.
[0067] The optical system 3 also comprises a beam splitter 3b, making it possible to redirect the optical beam generated by the illumination of the object to be imaged O in the square-section multimode optical fiber 4 towards the measuring device 5.
[0068] In this first embodiment, the lighting device 2 generates a lighting pattern directed towards the optical system 3, inside which the translation device 3a translates the lighting pattern onto the input face of the square-section multimode optical fiber 4, which conducts this lighting pattern towards the object to be imaged O, which returns in the square-section multimode optical fiber 4 a reflected optical signal, transmitted via the beam splitter 3b towards the measuring device 5 which, associated with the calculation device 6, makes it possible to reconstruct the object to be imaged O with the method described below in more detail.
[0069] With this first embodiment, it is possible to capture, via the square-section multimode optical fiber 4, either the reflected optical signal or a fluorescence generated by the object to be imaged O, the measuring device being an optical sensor in both cases.
[0070] Referring to [Fig. 2], it can be seen that an endoscope 1 is shown according to a second embodiment of the invention.
[0071] As in the first embodiment, the endoscope 10 comprises an illumination device 2, generating coherent light, preferably monochromatic light, preferably a laser.
[0072] Opposite the lighting device 2 in the endoscope 10 is an optical system 3 which sends the light generated by the lighting device 2 into a square-section multimode optical fiber 4. The output of the square-section multimode optical fiber 4 opens directly (without an optical element) onto an object to be imaged O. A measuring device 5 is connected to the optical system 3 and to a calculation device 6.
[0073] An auxiliary optical fiber 7 captures the reflected optical signal generated by the object to be imaged O, and returns it to the beam splitter 3b of the optical system 3.
[0074] In this second embodiment, the lighting device 2 generates a lighting pattern directed towards the optical system 3, inside which the translation device 3a translates the lighting pattern onto the input face of the square-section multimode optical fiber 4, which conducts this lighting pattern towards the object to be imaged O, which returns a reflected optical signal into the auxiliary optical fiber 7, transmitted via the beam splitter 3b to the measuring device 5 which, associated with the calculation device 6, makes it possible to reconstruct the object to be imaged O with the method described below in more detail.
[0075] With this second embodiment, it is possible to capture by the auxiliary optical fiber 7 either the reflected light, in which case the measuring device 5 is an optical sensor of reflected light (with the same wavelength), or a fluorescence generated by the object to be imaged O, in which case the measuring device is an optical fluorescence sensor.
[0076] If we refer to [Fig. 3], we can see that an endoscope 1 has been shown according to a third embodiment of the invention.
[0077] As for the first embodiment, the endoscope 20 comprises a lighting device 2, generating coherent light, preferably monochromatic, preferably a laser.
[0078] Opposite the lighting device 2 in the endoscope 20 is an optical system 3 which sends the light generated by the lighting device 2 into a square-section multimode optical fiber 4. The output of the square-section multimode optical fiber 4 opens directly (without an optical element) onto an object to be imaged O. A hydrophone 9 with an optical fiber 8 is connected to the optical system 3 and to a computing device 6.
[0079] The optical fiber 8 of the hydrophone 9 picks up a photoacoustic signal generated by the object to be imaged O, and returns it to the hydrophone 9, which hydrophone 9 sends a signal to the computing device 6.
[0080] In this third embodiment, the lighting device 2 generates a lighting pattern directed towards the optical system 3, inside which the translation device 3a translates the lighting pattern onto the input face of the square-section multimode optical fiber 4, which conducts this lighting pattern towards the object to be imaged O, which returns in the optical fiber 8 a photoacoustic signal, transmitted via the hydrophone 9 towards the calculation device 6 to reconstruct the object to be imaged O with the method described below in more detail.
[0081] With this third embodiment, it is possible to capture a photoacoustic signal generated by the object to be imaged O.
[0082] In all three embodiments, the computing device 6 is at least one of a computer, a microprocessor, a digital signal processor, DSP, a processor, a microcontroller, a programmable gate array, FPGA, an application-specific component, ASIC, and includes memory.
[0083] [Fig.4] schematically illustrates the square-section multimode optical fiber 4, comprising a cylindrical sheath 4a covering a square-section core 4b.
[0084] Typically, the square-section multimode optical fiber 4 is flexible, the core having dimensions between 10 pm*10 pm and 1 mm*1 mm, preferably between 50 pm*50 pm and 200 pm*200 pm, more preferably between 100 pm*100 pm and 150 pm*150 pm.
[0085] It can be seen that a speckle pattern E is placed at the input of the square-section multimode optical fiber 4 and translated over all or part of the input of the square-section multimode optical fiber 4, generating at the output of the square-section multimode optical fiber 4 an illumination Eo on the object to be imaged.
[0086] The method according to the invention will now be described in more detail in connection with [Fig.5].
[0087] In a first step represented in A in [Fig.5], for N measurements, N a natural integer greater than or equal to 1, an illumination pattern is scanned on the input face of the square-section multimode optical fiber.
[0088] We consider a given electromagnetic field corresponding to a lighting pattern, numbered p'n(vxj , k = 1.. .N, corresponding to an intensity pattern: y^ifv ? irj
[0089] [Math.2] Tin / , ! T^in( \ S h (¾. yin) = 1¾K yj|
[0090] xin and yifl are the coordinates in the input face of the multimode optical fiber at square section. The measurement procedure consists of performing a two-dimensional scan of this field in the plane of the square-section multimode optical fiber, i.e. illuminating
[0091] the input face of the square-section multimode optical fiber with patterns of the form:
[0092] [Math.3] El^(xin, y. , &xin, A y. ) -^l(xjn- &xin, y. - A y. )
[0093] The set of all values of A xin and A y defines a scan, for a given pattern rv Y In practice, the extent of the scan is rectangular, with
[0094] [Math.4] 6 [ A<", A^]
[0095] and
[0096] [Math.5] A y E [ A y™1. A y"™ 1
[0097] and can correspond to:
[0098] - scan the pattern across the entire input surface of the multimode optical fiber square section;
[0099] - scan the pattern only on a limited part of the fiber input surface square section multimode optics.
[0100] This procedure is repeated for a number N of different patterns, different in the sense that the same patterns are never found in two scanning procedures (either because the basic patterns are different, or because the same pattern is scanned over different areas).
[0101] In practice, there are several optical devices for generating patterns and scanning them:
[0102] - use of a spatial light modulator device (SLM, acronym for Spatial Light Modulator), which allows both to choose a pattern defined on the SLM pixels, then to translate it in the input plane of the square-section multimode optical fiber. It is possible to use either a liquid crystal-based SLM (LCOS SLM), allowing to modulate the phase of a light wave on each pixel, or a micro-mirror matrix (DMD) type SLM allowing to modulate the amplitude on each pixel;
[0103] - use of an optical diffuser to generate a pattern of random appearance, speckle type. A modification of the position of the diffuser relative to the illumination beam then makes it possible to generate several patterns k = [1, ..., N]. This pattern can then be scanned in the plane of the square-section multimode optical fiber by translating the illumination beam and diffuser assembly, either using translation stage(s) or using galvanometric mirrors.
[0104] In a second step, the signal generated by the object to be imaged at the output of the square-section multimode optical fiber is measured.
[0105] We note ( xv A x Av ) the optical intensity at the output of the fiber cor-K \ • out' -■ Ml[' ■ m' J in ) corresponding to the field rv Av av \ in input, for a position of sweep defined by A = (A xin, A y
[0106] We note Sk ( A Xin, A y ) the signal measured for each position A rf7j = ( A xin, A yy We make the assumption for the subsequent reconstruction that this signal measured in the presence of an object O ( xout, y°ut} is of the form:
[0107] [Math.6] &xin' h (xaufyollf ^xin' yin}^ {Xout' yota}^XouAyout
[0108] This hypothesis corresponds to many situations, including the following situations:
[0109] - the signal corresponds to the light reflected by a reflective sample;
[0110] - the signal corresponds to the fluorescence intensity emitted by a fluorescent object; or
[0111] - the signal corresponds to a photoacoustic signal.
[0112] We therefore obtain at the end of the measurements a set of N tables of values noted Sk ( A xin, A y J , k =1 .. .N, values measured for each position of the scan, which contain information on the object to be imaged to be reconstructed.
[0113] In a third and final step, the object to be imaged is reconstructed from the previous measurements.
[0114] In all that follows, the spatial autocorrelation operator of a function f is defined by:
[0115] [Math.7] ïf 0 f] (r) = n(r).rtr+r)d2ArF
[0116] For each measurement Sk (A xin, A y J , we calculate the spatial autocorrelation:
[0117] [Math.8] cl = s* ® Sk
[0118] in order to finally calculate the averaged autocorrelation for all the illumination patterns (represented in B in [Fig.5]):
[0119] [Math.9] A V. ) = < &xi„, AV. ) > mesuu \ in' . ln / A. y m' w
[0120] At this stage, the procedure is identical to that described in the scientific publication “Non-invasive imaging through opaque scattering layers”, Jacopo Bertolotti, Elbert G Van Putten, Christian Blum, Ad Lagendijk, Willem L Vos, and Allard P Mosk, Nature, 491(7423):232-234, 2012. It can be shown that the average autocorrelation of the signal is written as a function of the average autocorrelation of the output intensity patterns:
[0121] [Math. 10] cLorie (A) = JJ AA you^C°( A r„ jd2 A
[0122] where q1 _ < > is the average autocorrelation of the N lighting patterns and C°(r) = < O(r) O(r) > the autocorrelation of the object.
[0123] It is the equation Math. 10 which is the basis of the possibility of reconstructing the object O(r) from the measurements (step C in [Fig.5]), the general method that we propose corresponding in fact to finding an object O(r) which verifies the equation Math. 10. To do this, we describe here at least two possible methods, without being exhaustive.
[0124] The first method is the so-called inverse problem solving method.
[0125] For a multimode optical fiber with a square section, it can in fact be shown that the average autocorrelation of the lighting patterns can be written explicitly in the form:
[0126] [Math. 11] Ar^; Ar;„) = ^| E e =[+, t, + H Ar^' Ar^ H )P
[0127] With
[0128] [Math. 12] ~ g
[0129] Ji being the first order Bessel function and
[0130] [Math. 13] A rt = + A r,:„
[0131] [Math. 14] A r * = ( - A xin, A y. J
[0132] [Math. 15] A r ■„ = - A r;„
[0133] [Math. 16] A (A Xjn, - A y
[0134] The four Arine vectors in the four directions of the memory effect observed in the case of the square fiber. We can then estimate the object by solving the following inverse problem.
[0135] [Math. 17] estimated — argmin^ || Cmes.ire ( Ar^j Jf C (Arzn, Ar(ra()c "(Ar^Jd^Ar^ ||
[0136] where — < x X > is the autocorrelation of a test object X. The equation Math. 17 means that the estimate of the object Ôestimated provided by the resolution of the problem is the object X which minimizes the deviation between the measurement result cs f A ri ct 'cv measurement V 1 in ' model
[0137] [Math. 18] theory ( A ) = fd( AAF„ JC*( A Vout)^ AF(ïaf
[0138] There are many classical methods to solve this type of inverse problem, for example the Adam optimization method, reference Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization" arXiv preprint arXiv:1412.6980 (2014).
[0139] The second method is the neural network method.
[0140] If the object to be reconstructed belongs to a well-defined class of objects, then we can use a trained neural network to go from an "image" defined by ( AF ) to the object O(r). Unlike the previous method, this method does not require knowing an expression of AFAF ()' ma's On the other hand, it is limited to a well-defined class of objects on which we can
[0141] train a neural network.
[0142] The method was implemented with a U-NET type network: for the phase
[0143] training, the network is given pairs f CS ( A r ) ' O(r)} for which both \ r and O(r) are known. The neural network is then able to provide an estimate of an object from ç A r ) The structure of the U-NET network used and the method is for example described in the scientific publication “Compensating for visibility artifacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties”, Guillaume Godefroy, Bastien Amal, Emmanuel Bossy, Photoacoustics, 21:100218, 2021. 4.
Claims
Claims
1. Method for imaging an object to be imaged (0), characterized in that it comprises: - generating, by a lighting device (2), a series of lighting patterns; - for each lighting pattern, carrying out a stimulation phase in which translations of the lighting pattern are carried out at the input of a square-section multimode optical fiber (4) by means of at least one translation device (3a) of an optical system (3) arranged between the lighting device (2) and the input of the square-section multimode optical fiber (4), the translations being carried out in a plane parallel to the input plane of the square-section multimode optical fiber (4); - for each stimulation phase, carrying out an acquisition phase in which the signal generated by the lighting pattern on the object to be imaged (0) placed at the output of the square-section multimode optical fiber (4) is measured by a signal measuring device (5; 9);and - reconstruct by a calculation device (6), using the signal measured for each of the acquisition phases, the object to be imaged (0).;
2. Imaging method according to claim 1, characterized in that the at least one translation device (3a) is one of a liquid crystal-based spatial light modulator, a micro-mirror matrix type spatial light modulator, an optical diffuser associated with a translation stage, an optical diffuser associated with galvanometric mirrors.
3. Imaging method according to claim 1 or claim 2, characterized in that the signal measuring device (5) is an optical sensor for measuring the optical signal reflected by the object to be imaged associated with a detection optical fiber placed between the object to be imaged and the optical sensor, the detection optical fiber being one of the square-section multimode optical fiber (4) and an auxiliary optical fiber (7).
4. Imaging method according to claim 1 or claim 2, characterized in that the signal measuring device is a hydrophone (9) with optical fiber (8) measuring the photoacoustic signal of the object to be imaged (0).
5. Imaging method according to claim 1 or claim 2, characterized in that the object to be imaged (0) is fluorescent and that the signal measuring device (5) is an optical sensor for measuring the fluorescence signal emitted by the object to be imaged associated with a detection optical fiber placed between the object to be imaged and the optical sensor, the detection optical fiber being one of the square-section multimode optical fiber (4) and an auxiliary optical fiber (7).
6. Imaging method according to any one of claims 1 to 5, characterized in that, in each stimulation phase, the translations are carried out in all or part of the input plane of the square-section multimode optical fiber (4).
7. Imaging method according to any one of claims 1 to 6, characterized in that the lighting device (2) is a coherent light source.
8. Imaging method according to any one of claims 1 to 7, characterized in that the computing device (6) is selected from at least one of a computer, a microprocessor, a digital signal processor, DSP, a processor, a microcontroller, a programmable gate array, FPGA, an application-specific component, ASIC, and comprises memory.
9. Imaging method according to any one of claims 1 to 8, characterized in that the reconstruction of the object to be imaged (0) by the calculation device (6) comprises a first step comprising, for each acquisition phase, the calculation of the spatial autocorrelation of the measured signal then the calculation of the averaged autocorrelation for all the lighting patterns, and a second step of deducing the object to be imaged (0) from the averaged autocorrelation for all the lighting patterns obtained in the first step.
10. Imaging method according to claim 9, characterized in that the object to be imaged (0) is deduced from the averaged autocorrelation for all the illumination patterns by solving the following equation: <Âinw(r) =argminx || Clesur,,( Ar(jI) - ffCl( Ar,,,, Arout )Cx( A rmt! ) d2 A rout || where A rin = ( A y ) defines a scanning position in the input face of the square-section multimode optical fiber with xin and y in the coordinates in the input face of the square-section multimode optical fiber, A rOHÎ represents the coordinates of the output face of the square-section multimode optical fiber, q1 is the averaged autocorrelation of the N illumination patterns, ^ is the autocorrelation of a test object X, is the averaged autocorrelation for all the patterns lighting and Ôestimated ( r ) is the estimate obtained from the object to be imaged (0).
11. Imaging method according to claim 9, characterized in that the object to be imaged (0) is deduced from the averaged autocorrelation for all the lighting patterns by a neural network previously trained on a defined class of objects, the neural network providing an estimate of the object to be imaged from the averaged autocorrelation, by comparing the averaged autocorrelation known by the neural network on the objects of the defined class with the averaged autocorrelation calculated for the object to be imaged.
12. Imaging method according to any one of claims 1 to 11, characterized in that the square-section multimode optical fiber (4) is flexible.
13. Imaging method according to any one of claims 1 to 12, characterized in that the square-section multimode optical fiber (4) comprises a sheath (4a) covering a square-section core (4b), the core (4b) having dimensions of between 10 pm*10 pm and 1 mm*1 mm, preferably of between 50 pm*50 pm and 200 pm*200 pm, more preferably of between 100 pm*100 pm and 150 pm*150 pm.
14. Endoscope (1; 10; 20) for imaging an object to be imaged (O), characterized in that it comprises a lighting device (2), an optical system (3), at least one multimode optical fiber with a square section (4), the optical system (3) optically coupling the lighting device (2) to the input of the at least one multimode optical fiber with a square section (4), the optical system (3) comprising a translation device (3a) for translating a lighting pattern generated by the lighting device (2) over all or part of the input plane of the at least one multimode optical fiber with a square section (4), the translation device (3a) being configured to translate the lighting pattern in a plane parallel to the input plane of the at least one multimode optical fiber with a square section (4), the object to be imaged (O) being configured to be placed at the output of the at least one multimode optical fiber with a square section (4), a translation device (3a) for translating the ... signal measurement (5,9) generated by the object to be imaged (O) and a computing device (6) for implementing the method according to any one of claims 1 to 13.,
15. Endoscope (1; 10; 20) according to claim 14, characterized in that the translation device (3a) being one of a modulator liquid crystal-based spatial light modulator, a micro-mirror matrix type spatial light modulator, an optical diffuser associated with a translation stage, an optical diffuser associated with galvanometric mirrors.
16. Endoscope (1; 10) according to claim 14 or claim 15, characterized in that the signal measurement device (5) consists of at least one of the following: - the at least one square-section multimode optical fiber (4) associated with an optical sensor at the input of the square-section multimode optical fiber (4) for capturing an optical signal reflected by the object to be imaged (0), - at least one auxiliary optical fiber (7) associated with an optical sensor on the input side of the square-section multimode optical fiber (4) of the endoscope (1; 10) for capturing an optical signal reflected by the object to be imaged (0), - an optical fiber hydrophone (9) (8) for capturing the photoacoustic signal of the object to be imaged (O).
17. Endoscope (1; 10) according to claim 14 or claim 15, characterized in that the object to be imaged (O) is fluorescent and that the signal measuring device (5) is constituted by at least one of: - at least one multimode optical fiber with square section (4) associated with an optical sensor at the input of the multimode optical fiber with square section (4) for capturing a fluorescence signal from the object to be imaged (O), - at least one auxiliary optical fiber (7) associated with a sensor on the input side of the multimode optical fiber with square section (4) of the endoscope (1; 10) for capturing a fluorescence signal from the object to be imaged (O).
18. Endoscope (1; 10; 20) according to any one of claims 14 to 17, characterized in that the computing device is selected from at least one of a computer, a microprocessor, a digital signal processor, DSP, a processor, a microcontroller, a programmable gate array, FPGA, an application-specific component, ASIC, and comprises memory.
19. Endoscope (1; 10; 20) according to any one of claims 14 to 18, characterized in that the at least one multimode fiber with square section (4) is flexible.
20. Endoscope (1; 10; 20) according to any one of claims 14 to 19, characterized in that the at least one multimode fiber with square section (4) comprises a sheath (4a) covering a core (4b) with square section, the core having dimensions between 10 pm*10 pm and 1 mm*l mm, preferably between 50 pm*50 pm and 200 pm*200 pm, more preferably between 100 pm*100 pm and 150 pm*150 pm.
21. Computer program product, characterized in that it comprises instructions which, when loaded and executed on a computing device of an endoscope according to any one of claims 14 to 20, make it possible to implement the method according to any one of claims 1 to 13.