Method and apparatus for orbit determination and imaging - Patents.com

Abstract

The present invention provides a method for determining a trajectory (T) of a biological tissue (O), wherein subsequent frames (F1, Fk, F1, Fn) representing images of the biological tissue are acquired by at least a first imaging device (FET), and at least a first segment of the trajectory (T) of the biological tissue is determined using the relative displacements of at least a first subset of the subsequent frames, recalculated into coordinates corresponding to the time of frame acquisition, to form a vector Tm with elements tm, where the element tm is at least one coordinate of the biological tissue, and the element tm of the vector Tm is a function of the relative displacement p of at least two of the subsequent frames. m,k The invention further relates to an imaging apparatus and a computer program product for use with the method according to the invention.The invention further relates to an imaging system.

Description

[Technical field]

[0001] The present invention relates to a method for determining a trajectory of biological tissue, an apparatus therefor, a computer program product therefor, and a biological tissue imaging system using said apparatus. [Background technology]

[0002] Imaging of biological tissue is prone to errors and distortions due to uncontrolled movement of the tissue during image acquisition. Acquiring a detailed image takes a long enough time that the biological tissue, e.g., the eye, moves much further than the desired spatial resolution. This problem is known in the art and is partially solved by either real-time tracking of the imaged tissue or by determining tissue trajectories and compensation in post-image processing.

[0003] US9867538 discloses a robust eye tracking method and an ophthalmic device therefor. The ophthalmic device includes an adaptive optics scanning laser ophthalmoscopy (AOSLO) device, a wide-field scanning light ophthalmoscopy (WFSLO) device, and a tracking processor that controls a tracking mirror based on position information calculated from the AOSLO and WFSLO images. In another embodiment, the tracking processor controls the tracking mirror based on position information calculated only from the WFSLO image. Depending on the amount of movement of a target image relative to a reference image and its detection accuracy, the tracking mirror of either the AOSLO device or the WFSLO device can be selectively controlled according to the difference in position of the target image relative to the reference image.

[0004] US Patent Application No. 20150077706A1 discloses real-time optical and digital image stabilization for adaptive optics scanning ophthalmoscope. A real-time ophthalmoscope system for imaging the eye includes a wide-field scanning laser ophthalmoscope (SLO) configured for wide-field imaging of the eye. The system also includes a small-field SLO configured for high-resolution imaging of the eye. A 2D tracking mirror electrically coupled to a computer and a 2D steering mirror electrically coupled to the computer are both disposed in an optical path between the small-field SLO and the eye. The system is configured to acquire a plurality of small field images at a plurality of positions on the eye as defined by the 2D steering mirror, each small field image being stabilized against eye movement by the 2D tracking mirror. A method for imaging the eye is also described.

[0005] US Patent Application No. 2016066781 discloses a method for extracting gaze velocity information from video footage having a plurality of frames, the method including detecting at least a portion of an eye in at least two frames of the video footage, applying an optical flow algorithm to the at least two frames of the video footage to extract pixel velocity information, and determining a statistical measure from the pixel velocity information in the at least a portion of the detected eye. The application further discloses a method for extracting head image trajectory information from video footage having a plurality of frames, the method including detecting at least a portion of a face region of a head image in at least two frames of the video footage, determining a rate of movement of at least a portion of the face region between the at least two frames, and determining a transformation map from the rate of movement. This is suitable for use with video footage of RGB images (1920x1080 pixels) at 25 frames / second. The use of camera rates of 30Hz and 25Hz is disclosed therein. .oneGenerally, high-resolution imaging devices conventionally used for ocular imaging are too slow to be used for reliable determination of biological tissue trajectories with respect to image sequences, and motion compensation techniques are not fully effective. Another drawback of known image-based approaches is their dependency on a selected reference frame, which itself may be distorted by motion artifacts. Algorithms have been proposed to correct these artifacts in the reference frame, but such approaches are not without limitations. For example, the algorithms may lack ground truth reconstruction validation.

[0006] Known methods for determining trajectories use reference frames and are therefore susceptible to random errors caused by unexpected movements outside the expected range of speeds and amplitudes, or require over-scaled hardware to capture frames fast enough and large enough.

[0007] It is an object of the present invention to provide a method and system for accurately determining the trajectory of biological tissue undergoing voluntary and involuntary movements, which allows analysis of the movements and compensation in the post-processing of images recorded with a high-resolution imaging device.

[0008] The present invention relates to a method for determining a trajectory of a tissue, in which subsequent frames representing images of the tissue are acquired by at least a first imaging device, and at least a first segment of the trajectory of the tissue is determined using the relative displacements of at least a first subset of the subsequent frames, recalculated to coordinates corresponding to the time of frame acquisition, to form a vector Tm with elements tm, where tm is at least one coordinate of the tissue. An element tm of the vector Tm is a function of the relative displacement p of at least two preceding frames. m,a ,p m,b The relative displacement p is determined using m,a ,p m,bis determined using a predetermined criterion. The frame acquisition rate of the first imaging device is at least 500 FPS. The frame size acquired from the first imaging device is at least 1000 pixels. The field of view of the first imaging device is at least 1° in the X and Y planes. That combination of parameters of the first imaging device has been proven feasible and sufficient to provide suitable frames for successful trajectory reconstruction of biological tissue. By using multiple previous frames, the error can be reduced, for example by averaging. Experiments have shown advantageous results with a field of view range of 1° to 4°. The upper limit facilitated providing sufficient sensitivity to fast tissue motion.

[0009] In an advantageous aspect, at least a second subset of the frames is used to calculate alignment frames corresponding to both the first and second segments, and the distance between the alignment frames is used to minimize accumulated errors due to recalculation of relative displacements between elements of the trajectory in the first and second segments.

[0010] Advantageously, the sequence frames are acquired by a first imaging device. Preferably, the trajectory is analyzed to detect loops and to identify the sequence frames as frames that are included in at least two segments of the trajectory.

[0011] Alternatively, the sequence frames are acquired by a second imaging device.

[0012] The invention further relates to a device for determining a trajectory of movement of biological tissue, comprising a first imaging device and a processing unit, the first imaging device adapted to acquire subsequent frames with a frame acquisition rate of at least 500 FPS, a frame size of at least 1000 pixels and a field of view of at least 1° in the X-plane and in the Y-plane, the processing unit adapted to execute the method according to the invention.

[0013] The invention further relates to a computer program product for determining a trajectory of biological tissue, executed on a processing unit supplied with subsequent frames comprising at least a subset of the frames acquired with a frame acquisition rate of at least 500 FPS, a frame size of at least 1000 pixels and a field of view of at least 1° in both the X-plane and the Y-plane.

[0014] The invention further relates to an imaging system including a second imaging device for biological tissue, characterized in that it further includes a device for determining a trajectory of biological tissue according to the invention and is configured to reposition images acquired by said second imaging device in order to compensate for the movement of said biological tissue, thus obtaining a synergistic effect between motion compensation and trajectory determination assistance.

[0015] Advantageously, the processing unit of the device for determining the trajectory of biological tissue is adapted to receive a frame from said second imaging device and use it as the alignment frame.

[0016] The present invention is explained in detail with reference to the following figures.

[0017] FIG. 1 shows a schematic representation of frames and regions along a trajectory.

[0018] FIG. 2a presents a flow chart of one embodiment of the method according to the invention.

[0019] FIG. 2b shows the displacement between the frame and the previous frame.

[0020] FIG. 3 shows a flow chart of another embodiment of the method according to the invention.

[0021] FIG. 4a shows a block diagram of an imaging device applicable for use in the system according to the invention and the method according to the invention.

[0022] FIG. 4b shows a Lissajous scan pattern, and FIG. 4c shows a rectangular frame reconstructed from the scan points.

[0023] Figure 4d shows an exemplary frame acquired by the first imaging device. Figures 5a-d show various loops of the trajectory. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Fig. 1 shows a schematic representation of an eye and a frame acquired with the imaging device FET. Subsequent frames are at different locations due to eye movements. The trajectory of the frame in the eye coordinate system is exactly complementary to the trajectory of the eye in the imaging device FET coordinate system. Thus, determining the frame is the same operation as determining the trajectory of the eye anatomy in the imaging device coordinate system, these two trajectories differ only in sign.

[0025] A flow chart of an example of a method according to the invention is shown in Fig. 2a. The method for determining the trajectory T of a biological tissue O is applied in this example to the eye. As the eyeball has the fastest movement in the body, this trajectory determination method can also be applied to the trajectory determination of other organs. In the settings described below, the eyeball can move at 1000° / s, which corresponds to 300 mm / s for an average eyeball.

[0026] Ocular frames are acquired with a first imaging device FET with a frame acquisition rate of 1240 FPS. The field of view was set to 3.37° × 3.24°. The imaging beam diameter at the cornea was 1.96 mm. The theoretical axial beam size on the retina in this case was 8.5 μm. The confocality of the system was intentionally sacrificed to obtain sensitivity using a 100 μm pinhole. The diameter of the Airy disk at the detection plane was 36.4 μm. The confocality, described by the time-diffraction-limited spot size, was equal to 2.75 and the frame size was 4432 pixels (140 × 140 pixels).

[0027] Further studies have shown that a frame rate as low as 500 FPS and a frame size as low as 1000px can provide reliable trajectories. Naturally, higher frame rates and larger frames allow for more accurate trajectory determination.

[0028] The first imaging device provided subsequent frames including F1, Fa, Fb, and Fm. The frames represent images of a small portion of the eye. The relationship between frame rate and frame size contributes to the result that every frame is part of a subset of subsequent frames.

[0029] According to this method, the position of a frame within said subset of frames relative to the position of at least a first region is determined, which is similar to the conventional method with the difference that no reference frame is selected.

[0030] The position is then recalculated to a set of coordinates (points) forming at least a first segment of the trajectory T corresponding to the acquisition time. The set of coordinates forms an element tm of a vector Tm, where element tm is a point of the trajectory expressed as a set of two coordinates:

[0031]

number

[0032] In this example, the trajectory reconstruction algorithm works in two stages. In the first stage, the retinal trajectory is estimated by the so-called N-back algorithm, where each new frame is aligned with respect to the n previous frames, as shown in Figures 2b and 2c. In the second stage, the accumulated errors of the trajectory reconstructed in the first stage are corrected.

[0033] After starting the process 200, a first frame f1 is acquired in an acquisition step 200. This involves scanning and recalculating the pattern to obtain a rectangular frame, as will be explained in more detail below.

[0034] Further frames are repeatedly acquired in acquisition step 211 .

[0035] New trajectory points of the retina were estimated by estimating 250 the displacements between the incoming frame in acquisition step 211 and a subset of previously acquired frames. These displacements are added to the trajectory points detected in the subsequent frames. In the simplest case, only the displacement from the previous frame is considered, which can be shown as follows:

[0036]

number

[0037] The accuracy and noise immunity of the simple trajectory construction algorithm can be improved in the N-back version of the algorithm by averaging the displacement of the new frame from the N previously acquired frames:

[0038]

number

[0039] where n is the index of the frame calculated relative to the newly acquired frame (i.e., n=1 represents the frame immediately preceding the new frame). Using trial-and-error experiments, a set of frame indices B∈{1, 2, 4, 8, 16, 32, 64, 128, 256, 512} was chosen. In this way, the method does not only compare consecutive frames, but also tries to align frames acquired approximately 0.5 seconds ago. For low SNR or displacements exceeding the size of the frames, the criterion D(f a ,f b ) cannot be calculated, the corresponding index n is removed from the set B, where N represents the total number of frames in the set B. After calculation, the frame fm is added 202 to the subset of frames used to determine the next trajectory point. Averaging can reduce errors. Those skilled in the art may find different configurations of the frame index B that maximize the efficiency of the method in terms of calculation speed and required accuracy.

[0040] The sequence of frames above is t m Propagating to p a,b Due to the recursive nature of the method, the orbit estimation error can be modeled as a random walk process, which results in an ideally tuned system having a non-stationary zero-mean error, with the variance of the error increasing linearly with time, i.e., the errors tend to accumulate.

[0041] To correct this non-stationary error, an additional approach can be applied. It makes use of the fact that the eye returns to the same position from time to time, and the accumulated error according to Eqs. 2-3 can be corrected by a new displacement calculation. The technique introduces an array (or key) frame KF, which can be intuitively understood as a subset of all frames in the acquired data set with a computable translation in the frames and in nearby spatial locations on the retina, independent of the time interval. In practice, the array frame can be detected 202 by detection of loops or crossings of the trajectory T in real time or in post-processing. The array frame allows to minimize the accumulated error by correcting the trajectory 251. Examples of loops with crossings are shown in Figs. 5a and 5c, and examples of loops without crossings are shown in Figs. 5b and 5d. If two non-adjacent segments of the trajectory T can be observed in the same frame, this frame can be used as the array frame KF.

[0042] Formally, these corrections to the KFs can be done using the multidimensional scaling (MDS) mathematical framework disclosed in I. Borg and P. Groenen "Modern multidimensional scaling: Theory and applications" Journal of Educational Measurement 40.3 (2003): 277-280, where the distances between the KFs are used to correct all trajectory points from the first stage. In other words, the matrix P KF is the total translation distance |p a,b KF It is composed of | trajectory T KF is found by minimizing the stress function σ with respect to the KF position (see below).

[0043]

number

[0044] Array frame trajectory T KF Although it has an inherently low sampling rate, it theoretically has zero non-stationary error. The final retinal trajectory T FET is the sequence frame trajectory T KF It is estimated by casting an N-back trajectory T on top and using interpolation of trajectory points between sequence frames.

[0045] Thus, once the alignment frame is detected in step 202, the trajectory is corrected in step 251.

[0046] After tracking 203 is completed, the corrected trajectory is output in step 204 and the algorithm ends 205.

[0047] The method is easily implemented in a computer program product running on a processing unit to which the frames are supplied, either in real time or in offline processing, the former having the advantage of recalculating the entire trajectory backwards in step 252 after the tracking 203 has been terminated.

[0048] Alternatively, the sequence frames can be obtained from a second imaging device providing a slower frame rate and higher resolution. Such a device can provide sequence frames corresponding to the entire orbit or at least a larger portion thereof. It is particularly advantageous if the method of determining the orbit together with the first imaging device FET is integrated with other imaging systems such as WFSLO.

[0049] A block diagram of an example of such an embodiment of the present invention is shown in Figure 3. After start 300, a frame from a first imaging modality is acquired 301, and then WFSLO is used in a loop to acquire images of the biological tissue in a strip-by-strip manner. Between strips of WFSLO, subsequent frames are acquired using the first imaging device in an acquisition step 310.

[0050] Displacement p a,b is calculated in the same way as in the previous example, and frames already in use are added to the pool of subsequent frames in step 301.

[0051] The difference is that in step 302, it is determined that the weather part of the image already acquired in WFSLO can be used as the alignment frame, and if so, the trajectory is corrected in step 351. After the tracking 303 is finished, the entire trajectory is optionally recalculated 352 and output 354. Then the process ends 305.

[0052] It should be emphasized that determining the trajectory without minimizing the accumulated error can still be useful in some applications, especially where the velocities of human tissue are important: one possible example is online image stabilization to prevent motion artifacts in the acquired images, which would cause image artifacts that cannot be corrected by post-processing using optical angiography.

[0053] A block diagram of a device FET applicable for use in the method according to the invention is shown in Fig. 4a. LD stands for laser diode; CL stands for collimating lens; BS stands for pellicle beam splitter; L1-L9 stands for achromatic doublet; MEMS 2D stands for 2-axis resonant MEMS scanning mirror; FET PS stands for positioning galvanometric mirror; D stands for tuned iris aperture; PH stands for pinhole; APD stands for avalanche photodiode; HIL stands for halogen illumination lamp; BST stands for Badal stage; AES stands for artificial eye scanner set; AEO stands for artificial eye target; AET stands for artificial eye test target. Stars and circles indicate conjugate planes. Inset A: Targets used for in vivo experiments. FT1 had a diameter of 1.5° on the retina and was used for static fixation experiments. FT2 was used for experiments with saccades; it had a diameter of 0.6° and the center-to-center distance ranged from 1 to 8°. Inset B: An artificial eye that can be mounted on the system for testing.

[0054] The illumination source in this scheme is a 785 nm laser diode LD coupled with a single mode fiber and collimated to a beam diameter of 0.7 mm by an aspheric lens CL. A pellicle beam splitter BS reflects and directs the beam onto a scanning mirror 2D MEMS with an active aperture of 1 mm, based on micro-electromechanical systems (MEMS) technology. After reflecting off the scanning mirror, the beam passes through a telescope system with lenses L4 and L3.

[0055] As shown in Fig. 4a, the MEMS 2D scanning mirror was set to its maximum frequency, resulting in a Lissajous scanning pattern. Therefore, to obtain a frame in the first step, the raw image acquired by the detector APD is resampled to obtain the frame shown in Fig. 4 b) to form a uniformly sampled rectangular image. This step is done by dense vector multiplication f = M f L This was done with a sparse matrix by f L =[f1 L ,f2 L ,...,fK L ] represents the intensity data vector directly acquired by the APD. i L is a single APD reading. The resulting frame f is of size N 2 The resampling matrix M is a rectangular image of size N 2 M is a s x K sparse matrix constructed from the Lissajous coordinates such that each row of M has a non-zero element at the index corresponding to the s-closest distance between the Lissajous coordinate and a coordinate of the derived frame f.

[0056] Those skilled in the art will be able to provide numerous alternative configurations of FET imaging devices once they have learned the teachings herein that must provide a frame acquisition rate of at least 500 FPS and a frame size N×N of at least 1000 pixels, as well as a field of view of at least 1 degree.

[0057] The FET device was tested in the system including tracking performance using a simplified artificial eye composed of an XY galvanometric scanner set GVSC, an imaging lens AEO, and a test target AF in a setup as shown in Figure 4a. The imaging lens was selected to be the same as L1 and arranged in a 4f system to maintain the same scan angle. The scanner was placed at the eye pupil plane and a voltage vs. optical angle calibration was performed using a USAF 1951 test target. For tracking verification, a printed fundus image was used as the test target. This image was prepared in such a way that images similar to those obtained from a preliminary study performed in vivo with a human eye could be obtained.

[0058] Figure 4c shows the reconstructed frames after a Lissajous scan. Figure 4d shows frames acquired with a field of view of 3.27° × 3.24°. Frames 1-2 show part of the optic nerve, frames 3-4 show the fovea, and frames 5-15 show the retinal vasculature of different sizes. Other experiments have shown that tracking works with fields of view as low as 1 degree in both planes.

[0059] The addition of the device according to the invention to a scanning laser ophthalmoscope known in the art allows simultaneous imaging and trajectory tracking of the living human eye. Knowledge of the eye trajectory allows compensation of said motion in the resulting images, which can then be stabilized. This can be done in real time or by post-processing.

[0060] The above embodiments are illustrative rather than limiting of the invention; those skilled in the art can design many alternative embodiments.

[0061] One of the claimed inventions is also a computer program product for running on a processing unit, and therefore includes at least code portions for performing the steps of the method according to the invention. A computer program may consist of or include subroutines, functions, procedures, classes, object methods, object implementations, executable applications, applets, servlets, source code, object code, shared libraries / dynamic load libraries and / or other sequences of instructions designed to be executed on a programmable device. The computer program may be provided on a data carrier representing the computer program, such as a CD-type optical disk, a DVD-type optical disk, a hard disk or a diskette, stored together with data loadable into the memory of a computer system. The data carrier may therefore be a tangible data carrier. The data carrier may be a data connection, such as a telephone cable or a network cable. The data carrier may also be an intangible data carrier, such as a wireless connection.

[0062] In the claims, reference signs placed in parentheses shall not be construed as limiting the scope of the claims. The verb "comprise" and its conjugations do not exclude the presence of elements or steps other than those stated in the claims. The use of the antecedent "a" or "an" of an element does not exclude the presence of a plurality of such elements. The use of "first", "second", and also numbers in the names of elements does not introduce a particular order, but merely serves as a way of distinguishing. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage.

[0063] A particular advantage of the method according to the invention is that no reference frame is used, but instead the trajectory for multiple frames is observed. This approach allows a better and more reliable analysis of eye trajectory and velocity. Eye movement analysis can also be applied to the diagnosis of several neurodegenerative diseases, including multiple sclerosis, Parkinson's disease, Alzheimer's disease, ataxia, stroke, paralysis, and others. It also helps in the diagnosis of diabetes and hypertension, etc.

[0064] Knowledge of the exact position and prediction of the movement of biological tissues is important, especially in surgery using manipulators and robotics. Thus, although there has been a strong emphasis on the eye in the above description, the invention is also applicable to determining the trajectories of many human organs, including muscles, especially the heart, bladder, liver, kidneys, ovaries, pancreas, thymus, trachea, etc.

[0065] The specific patterns of tissue movement can also be applied to biometric measurements.

Claims

1. A method for determining a trajectory (T) of a biological tissue (O), comprising the steps of: acquiring (210, 211, 310, 311) subsequent frames (fa, fb, fm) representing images of the biological tissue by at least a first imaging device (FET); and determining at least a first segment of the trajectory (T) of the biological tissue using the relative displacement of at least a first subset of the subsequent frames, recalculated to coordinates corresponding to the time of frame acquisition to form a vector Tm with elements tm, where element tm is at least one coordinate of the biological tissue; Here, the element t of the vector T is the relative displacement p of at least two previous frames (fa, fb). ｍ,ａ , p ｍ,ｂ and the relative displacement p ｍ,ａ , p ｍ,ｂ is determined using a predetermined criterion (250, 350), Where: the frame acquisition rate of the first imaging device (FET) is at least 500 FPS; a frame size acquired from the first imaging device (FET) is at least 1000 pixels; A method, wherein the field of view of the first imaging device is at least 1° in the X and Y planes.

2. 2. The method of claim 1, wherein at least a second subset of frames is used to calculate registration frames (KF) corresponding to both the first and second segments, and the distance between the registration frames is used to minimize (251, 351) the accumulated error due to recalculation of the relative displacement between the elements of the trajectory in the first and second segments.

3. 3. The method of claim 2, wherein the registration frame (KF) is acquired with a first imaging device.

4. 4. The method of claim 3, wherein the trajectory (T) is analyzed to detect (202) loops in the trajectory where at least two non-adjacent segments are close to each other and to identify alignment frames (KF) contained in said at least two non-adjacent segments.

5. 3. The method according to claim 2, characterized in that the registration frame (KF) is obtained with a second imaging device.

6. A device for determining a trajectory (T) of movement of a biological tissue (O), comprising a first imaging device (FET) and a processing unit, said first imaging device (FET) adapted to acquire subsequent frames (Fm, Fa, Fb, Fn), having a frame acquisition rate of at least 500 FPS, providing a frame size of at least 1000 pixels, and having a field of view in the X and Y planes of at least 1°, Apparatus, wherein the processing unit is adapted to perform a method according to any of claims 1 to 5.

7. 6. A computer program for determining the trajectory of biological tissue, characterized in that when it is executed on a processing unit supplied with subsequent frames comprising at least a subset of frames having a field of view in which the frames were acquired at a frame acquisition rate of at least 500 FPS and a frame size of at least 1000 pixels and with a field of view of at least 1° in the X and Y planes, it causes the processing unit to carry out the method according to any one of claims 1 to 5.

8. 7. An imaging system including a second imaging device for biological tissue, further comprising a device for determining a trajectory of biological tissue as described in claim 6 and configured to reposition images acquired by the second imaging device to compensate for movement of the biological tissue.

9. 9. The imaging system according to claim 8, characterized in that the processing unit of the device for determining the trajectory of biological tissue is adapted to receive a frame from the second imaging device and use it as a registration frame (KF).

Images