Spatial merging of imaging data from multiple modalities

Abstract

A system for spatially combining images from multiple modalities includes a controller having at least one processor and at least one non-temporary tangible memory. The controller is adapted to receive first and second imaging datasets of a target region from the first and second modalities. The controller is adapted to extract a first feature set from the first imaging dataset via a first neural network. A second feature set is extracted from the second imaging dataset via a second neural network. Feature pairs are generated by matching the respective data points in the first and second feature sets. The controller is adapted to determine coordinate transformations between feature pairs and to generate at least one spatially combined image of the target region based in part on the coordinate transformations.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 582,917 filed on September 15, 2023, which is hereby incorporated by reference in its entirety into the present specification.

Background Art

[0002] The present disclosure generally relates to blending imaging data from multiple modalities. More specifically, the present disclosure relates to spatially combining imaging data from multiple modalities. Various imaging modalities are commonly used worldwide to image various parts of the human body. Each of these imaging modalities provides a different set of information. To maximize the available information, it is desirable to synthesize the respective information provided by various imaging modalities. However, it is not a simple task for a user to blend captures created in multiple domains.

Summary of the Invention

Means for Solving the Problems

[0003] Disclosed herein is a system for spatially combining imaging data from multiple modalities. The system includes a controller having at least one processor and at least one non-temporary tangible memory on which instructions are stored. The controller is adapted to receive a first imaging dataset of a target site from a first modality and a second imaging dataset of the target site from a second modality. The controller is adapted to extract a first feature set extracted from the first imaging dataset via a first neural network. A second feature set is extracted from the second imaging dataset via a second neural network. Feature pairs are generated by matching each data point in the first feature set with each data point in the second feature set. The controller is adapted to determine coordinate transformations between feature pairs and to generate at least one spatially combined image of the target site, partially based on the coordinate transformations.

[0004] The target site may be the eye. In some embodiments, the first feature set is a limited set that does not represent information captured by the first modality, and the second feature is a representation of the eye region captured by the second modality. The controller may be configured to select at least one region of interest within the first imaging dataset, and the first feature set is extracted from at least one region of interest. The first imaging dataset may include data acquired by scanning multiple light source wavelengths.

[0005] In one embodiment, the first modality is multispectral imaging and the second modality is optical coherence tomography ("OCT"). Here, the first imaging dataset is captured by an OCT device and the target site is the eye. The spatially coupled images extend the peripheral portion of the first imaging dataset to enable visualization of one or more structures posterior to the iris of the eye. In another embodiment, the first modality is fluorescence angiography and the second modality is optical coherence tomography. The first neural network may be a multilayer perceptron. The second neural network may be a convolutional neural network.

[0006] In some embodiments, the first feature set includes each feature point indicating a degenerative disease. In some embodiments, the first imaging dataset includes multiple scans. The controller may be configured to identify and isolate abnormal regions as being within the range of one of the multiple scans or between two of the multiple scans. The controller is configured to add at least one annotation, which indicates an abnormal region, to at least one spatially combined image. In some embodiments, the controller is adapted to selectively transfer each annotation created on the first imaging dataset to a second imaging dataset, and from the second imaging dataset to the first imaging dataset.

[0007] The features and advantages of this disclosure, as well as other features and advantages, will be readily apparent from the following detailed description of the best mode for implementing this disclosure, in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram of a system for spatially combining imaging data from multiple modalities, which includes a controller. [Figure 2] Figure 1 is a schematic flowchart of the methods that can be performed by the controller. [Figure 3] Figure 1 is a schematic diagram of an exemplary architecture that can be used in the system. [Figure 4] This is a schematic diagram of an exemplary imaging dataset obtained from multispectral imaging. [Figure 5] This is a schematic diagram of an exemplary image obtained by optical coherence tomography. [Figure 6] This is a schematic diagram of an exemplary spatially combined image generated by the system shown in Figure 1. [Modes for carrying out the invention]

[0009] Representative embodiments of this disclosure are shown in the drawings as non-limiting examples and are described in further detail below. However, it should be understood that novel embodiments of this disclosure are not limited to the specific forms shown in the drawings listed above. Rather, this disclosure includes, for example, modifications, equivalents, combinations, secondary combinations, rearrangements, groupings and alternative forms that fall within the scope of this disclosure, such as those covered in the accompanying claims.

[0010] Referring to the drawings, similar reference numbers refer to similar components. Figure 1 schematically shows a system 10 having at least one processor P and a controller C having at least one memory M (or tangible non-temporary computer-readable storage medium) on which instructions for executing method 100 for spatially combining imaging data of a target site 12 are recorded. Method 100 is shown in Figure 2 and will be described below with reference to Figure 2. The imaging data is acquired from multiple modalities 14. An exemplary architecture 200 available to system 10 is shown in Figure 3 and will be described below.

[0011] In the embodiments shown, the target region 12 is the eye E. Eye regions captured by different imaging modalities provide different spatial information and other characteristics depending on the resolution, depth, type of light source used, light source wavelength, and other characteristics of the imaging modality. System 10 enhances the visualization of the target region 12 by coherently combining imaging data acquired from multiple modalities 14, including a first modality 16 and a second modality 18. In some embodiments, a third modality 20 may be used to acquire a third imaging dataset. The coherent spatial combination includes both tissue distribution information and depth information. System 10 may learn and improve over time, and the data may be shared in the field.

[0012] Multiple modalities 14 may include multispectral imaging. Multispectral imaging uses a device with multiple monochromatic light sources, such as LEDs, to illuminate the eye 12 with wavelengths in the range of approximately 550 nm to 950 nm. The reflected light is collected by an image sensor (not shown). Frames with different spectral reflectances are obtained by scanning the source wavelength. An exemplary imaging dataset 310 from multispectral imaging is shown in Figure 4 and described below.

[0013] Multiple modalities 14 may include optical coherence tomography (hereinafter "OCT") using an OCT device. The OCT device has an array of laser beams 22 that can cover the range or width of eye E. In one example, the OCT device is a high-resolution swept source OCT imaging device. The OCT device may use time-domain, frequency-domain, or other appropriate spectral coding, and may use single-point, parallel, or other types of scan patterns. The OCT device can take many different forms and may include multiple and / or alternative components. An exemplary raw image 410 from optical coherence tomography is shown in Figure 5 and described below.

[0014] Multiple modalities 14 may include fluorescence angiography, which involves the use of a fluorescent dye and a dedicated camera. For example, sodium fluorescein can be introduced into the patient's systemic circulation, and the eye 12 can be illuminated with light at a specific wavelength, e.g., 490 nanometers. The angiographic image is obtained by capturing the fluorescent green light emitted by the dye. It will be understood that multiple modalities 14 may include ultrasound biomicroscopy, fundus imaging, and other modalities available to those skilled in the art.

[0015] Referring to Figure 1, controller C may selectively execute several modules 30, such as a feature pairing module 222, a coordinate transformation module 232, and a coupling module 234. These modules 30 may be integrated into controller C or accessible from controller C. They may also be part of a remote server or cloud unit accessible from controller C via a network 40.

[0016] Referring here to Figure 3, an exemplary architecture 200 available to system 10 is shown. System 10 may be run in three phases, as shown in Figure 3: a first phase 210 of feature selection, a second phase 220 of feature pairing, and a third phase 230 of spatial joining. As described below, controller C is adapted to receive a first imaging dataset of the target region 12 from a first modality 16 and a second imaging dataset of the target region 12 from a second modality 18. In the first phase 210, the first imaging dataset 212 is sent to the first neural network N1. Similarly, the second imaging dataset 216 is sent to the second neural network N2. In the first phase 210, the first neural network N1 and the second neural network N2 extract a first feature set F1 and a second feature set F2, respectively, through the first neural network N1 and the second neural network N2.

[0017] In the second phase 220 shown in Figure 3, the feature pairing module 222 may be executed to pair or match each feature point in the first feature set F1 with the corresponding feature point in the second feature set F2. The result of the feature pairing module 222 is a pair of features, for example, for every unique feature identified on the first imaging dataset, there exists a corresponding feature in the second imaging dataset.

[0018] In the third phase 230 shown in Figure 3, the coordinate transformation module 232 may be executed to determine the coordinate transformation between feature pairs. The controller C may be adapted to generate at least one spatially joined image of the target region 12 via the execution of the joining module 234. The spatially joined image joins the first imaging dataset and the second imaging dataset through the coordinate transformation. The spatially joined image of the target region 12 may be displayed on the display 44, as shown in Figure 1.

[0019] By blending datasets acquired from different systems in a coherent manner, the system supports diagnosis at both the preoperative and intraoperative stages. System 10 allows annotations created on one image to be spatially combined and then transferred to another image, assisting surgeons in marking various features interchangeably.

[0020] Various components of system 10 may be configured to communicate via the network 40 shown in Figure 1. Network 40 may be a bidirectional bus implemented in various ways, for example, a serial communication bus in the form of a local area network. The local area network may include, but is not limited to, a Controller Area Network (CAN), a Controller Area Network with Flexible Data Rate (CAN-FD), Ethernet, Wi-Fi, Bluetooth®, and other data connection forms. Other types of connections may be used.

[0021] Controller C may be configured to receive and transmit data via a user-operable input device 42. The input device 42 may be installed on a laptop, tablet, desktop, or other electronic device. The input device 42 may include a touch screen interface, a keyboard, a joystick, a mouse, a foot switch, and other devices. The input device 42 may be a mobile application on a smartphone. Circuits and components of mobile applications ("apps") available to those skilled in the art may be employed. The input device 42 may include an on-board processor and on-board memory.

[0022] Referring now to Figure 2, a flowchart of a method 100 is shown. The method 100 may be embodied as computer readable code or instructions stored in the controller C of Figure 1 and partially executable by the controller C. The method 100 need not be applied in the specific order described herein and may be performed dynamically. Further, it should be understood that some steps may be omitted. As used herein, the terms "dynamic" and "dynamically" describe a step or process that is performed in real time during execution of a routine or during repeated execution of a routine, and is characterized by monitoring or determining the state of a parameter and updating the state of the parameter on a regular or periodic basis.

[0023] Beginning at block 102, the method 100 includes receiving a first imaging data set of the target site 12 from a first modality 16, and receiving a second imaging data set of the target site 12 from a second modality 18. In one embodiment, the first modality 16 is multispectral imaging, and the second modality 18 is optical coherence tomography (hereinafter "OCT"). In another embodiment, the first modality 16 is fluorescence angiography, and the second modality 18 is OCT.

[0024] Referring now to Figure 4, an exemplary imaging dataset 310 from multispectral imaging is shown. The imaging dataset 310 includes a plurality of scans 312 of the eye E. In other words, the imaging dataset 310 includes a plurality of two-dimensional images stacked together, extending across a horizontal axis 320 and a vertical axis 322. The imaging dataset 310 acquired from multispectral imaging covers a wavelength range 324, which may cover 20 to 40 spectral bands. Accordingly, scan 314 may cover shorter wavelengths (e.g., 550 nm), and scan 316 may cover longer wavelengths (e.g., 850 nm). Since the peak of each spectrum can be spatially mapped, each dataset includes a set of images. Thus, each scan or two-dimensional image has both spatial information and spectral information, i.e., each spatial trajectory has an associated spectrum when viewed across the available wavelengths.

[0025] Referring to Figure 5, an exemplary original image 410 acquired from optical coherence tomography is shown. The original image 410 shows the pupil 414, iris 416, retina 424, and lens 420 of the eye E. OCT imaging does not capture the peripheral portion 422 of the lens 420 located behind the iris 416. This is because the illumination laser used in OCT imaging cannot penetrate the iris 416. However, OCT imaging techniques provide a non-contact scanning method that offers high resolution and is convenient in terms of patient compliance and comfort in routine clinical settings. For example, OCT imaging is performed with the patient in a sitting position, and the time required is relatively short.

[0026] Proceeding to block 104, method 100 includes extracting a first feature set F1 from a first imaging dataset via a first neural network N1. In some embodiments, a controller C is adapted to select at least one region of interest, such as a first region of interest 326, within the first imaging dataset 310. The first feature set F1 is then extracted from within the region of interest 326 via the first neural network N1. In one example, the region of interest 326 contains feature points indicating a degenerative disease such as a tumor. The region of interest may be captured in the form of a cuboid 328 covering each of a plurality of scans 312.

[0027] In some embodiments, the first neural network N1 is a multilayer perceptron, which is a feedforward artificial neural network that generates a set of outputs from a set of inputs. As will be understood by those skilled in the art, a multilayer perceptron is characterized by several layers of input nodes connected as a directed graph between an input layer and an output layer.

[0028] Moving on to block 106, controller C is adapted to extract a second feature set from a second imaging dataset via a second neural network N2. In some embodiments, the second neural network N2 incorporates a convolutional neural network (CNN) based or other deep learning techniques available to those skilled in the art. A convolutional neural network is a special type of artificial neural network that uses convolution instead of general matrix multiplication in at least one of its layers. The second neural network N2 may be specifically designed to process pixel data for image recognition and processing. It should be understood that the first neural network N1 and the second neural network N2 may include any other type of neural network available to those skilled in the art.

[0029] In some embodiments, the user can select or pre-program various parameters for feature extraction, including a desired number of feature points per frame, a maximum number of feature points per frame, and a minimum spacing between feature points. In addition, the controller C may be programmed to adaptively calculate the optimal quality level for each feature within the region of interest. The quality level is a parameter that characterizes the minimum acceptable quality of an image feature point. In some embodiments, the quality level is a normalized number between 0 and 1. Feature points with a score or quality measured below the quality level may be excluded.

[0030] Moving on to block 108, method 100 includes generating feature pairs by matching each data point in a first feature set F1 with each data point in a second feature set F2. For example, the first feature set F1 may contain 80 feature points, and the second feature set may contain 100 features. Controller C may successfully identify matches with a portion of the first feature set F1, such as 60 of the 80 features in the first feature set F1. Feature matching may be based on common features such as edges, points, corners, and landmarks. Feature matching may also be based on regions with relatively high contrast and other factors.

[0031] Moving on to block 110, controller C determines the coordinate transformation between each feature pair and adapts it to transform the coordinates from the reference frame of the first imaging dataset to the reference frame of the second imaging dataset. The transformation may be based on a deterministic system using one or more predetermined equations or relations. The transformation may be a learning-based method using a neural network. Since the transformation involves redundant mapping, the data may be filtered for more accurate results.

[0032] Proceeding to block 112, method 100 includes generating at least one spatially joined image of a target site 12 based in part on a coordinate transformation. Figure 5 shows an exemplary OCT image of eye E. Figure 6 is a schematic diagram of an exemplary spatially joined image 510. The spatially joined image 510 shows the pupil 514, iris 516, retina 524, and lens 520 of eye E. System 10 enables reconstruction of a complete image of the lens 520 in Figure 6, including peripheral portions 422 that are not present in the original image 410 in Figure 5.

[0033] Controller C may be configured to identify and isolate an anomaly region 330 as being within the range of one of a plurality of scans 312 (of the first or second imaging modality) or between two of a plurality of scans 312. Controller C may be configured to display, annotate, or draw an overlay dashed line or sphere (see Figure 6) at the location of the anomaly region 330 to improve visualization. The anomaly region 530 may be indicated by pixels on the screen that have a different brightness or color compared to the surrounding area. Diagnosis and treatment are aided by combining the imaging dataset as described above and further highlighting or annotating the areas where the anomaly region 530 is found.

[0034] In some embodiments, the controller C is adapted to selectively transfer annotations created in the first imaging dataset to the second imaging dataset, and annotations created in the second imaging dataset to the first imaging dataset. As described above, the imaging dataset from multispectral imaging can cover 20–40 spectral bands. Longer wavelengths (e.g., 670 nm) allow for visualization of deeper layers of the retina, retinal pigment epithelium, and choroid. Information from different layers of the eye with multiple light source wavelengths enables the detection of melanin, the pigment in the retinal pigment epithelium. Regions where melanin is detected in the imaging dataset (acquired from multispectral imaging) can be marked within the same region and transferred to a spatially combined image 510. In addition, longer wavelengths allow for mapping the retinal vascular system by detecting blood oxygenation. Regions where oxygenation is detected in the imaging dataset (acquired from multispectral imaging) can be marked within the same region and transferred to a spatially combined image 510.

[0035] In summary, system 10 generates at least one spatially combined image of the target site 12, partially based on coordinate transformations between feature pairs from multiple modalities. Feature pairs are generated by matching each data point in a first feature set F1 with each data point in a second feature set F2. System 10 has several technical advantages. Firstly, the results of the spatial combining module 244 provide blending of multiple images, which will enhance the ophthalmologist's visualization. In addition, the results of the feature pairing module 222 can be used to select one feature in the first imaging modality and automatically predict the corresponding feature in the second imaging modality.

[0036] Controller C in Figure 1 includes computer-readable media (also called processor-readable media), which include non-temporary (e.g., tangible) media involved in providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including, but are not limited to, non-volatile and volatile media. Examples of non-volatile media include optical or magnetic disks and other persistent memory. Examples of volatile media include dynamic random-access memory (DRAM), which can constitute main memory. Such instructions can be transmitted by one or more transmission media, such as coaxial cables, copper wires, and optical fibers, including wiring that includes a system bus coupled to the computer's processor. Some forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, DVDs, other optical media, physical media, RAM, PROMs, EPROMs, flash EEPROMs, other memory chips or cartridges, or other media that can be read by a computer.

[0037] The lookup tables, databases, data repositories, or other data stores described herein may include various mechanisms for storing, accessing, and retrieving various types of data, including hierarchical databases, sets of files in a file storage system, application databases in ownership-protected formats, and relational database energy management systems (RDBMS). Each such data store may be contained within a computing device using a computer operating system such as one of the aforementioned, and may be accessed over a network by one or more of a variety of methods. File systems may be accessible from computer operating systems and may contain files stored in various formats. RDBMS may use structured query language (SQL) in addition to languages ​​for creating, saving, editing, and executing stored procedures such as the PL / SQL language described above.

[0038] The illustrated flowcharts illustrate the architecture, functionality, and operation of possible embodiments of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part of code, which includes one or more executable instructions for performing a specified logical function. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a system based on purpose-specific hardware or a combination of purpose-specific hardware and computer instructions that performs a specified function or action. These computer program instructions may also be stored in computer-readable media, which can instruct a controller or other programmable data processing device to function in a specific way to produce a product, and these include instructions that perform a specified function / action in a block in the flowchart and / or block diagram.

[0039] In this specification, including the appended claims, numerical values ​​of order (e.g., quantity or state) should be understood to be modified by “approximately,” regardless of whether the term “approximately” actually precedes the numerical value in each case. “Approximately” indicates that the stated numerical value allows for some degree of slight inaccuracy (somewhat close to the exact value, roughly the value, or reasonably close to it, almost). Where the inaccuracy provided by “approximately” is not understood in this ordinary sense in the art, “approximately” as used herein indicates at least the variation that may arise from the ordinary measurement and use of such orders. In addition, the disclosure of ranges includes disclosures of each value within the full range and further subdivided ranges. In this specification, each value within a range and the endpoints of the range are disclosed as separate embodiments.

[0040] While the detailed description and drawings or figures are supplementary and illustrative to this disclosure, the scope of this disclosure is defined solely by the claims. Although several best modes and other embodiments for carrying out the claimed disclosure have been described in detail, there are various alternative designs and embodiments for carrying out the disclosure as defined in the appended claims. Furthermore, the features of the embodiments shown in the drawings or the various embodiments referred to herein should not necessarily be understood as independent embodiments of each other. Rather, each of the features described in one example of an embodiment may be combined with one or more other desirable features from other embodiments, resulting in other embodiments that are not described in words or not described by reference to the drawings. Thus, such other embodiments are included within the framework of the appended claims.

Claims

1. It is a system, A controller comprising at least one processor and at least one non-temporary tangible memory on which instructions for a method of spatially combining imaging data are recorded, wherein the execution of the instructions by the processor results in the controller The system receives a first imaging dataset of the target area from the first modality, and a second imaging dataset of the target area from the second modality. A first set of features is extracted from the first imaging dataset via the first neural network. A second set of features is extracted from the second imaging dataset via a second neural network. A feature pair is generated by matching each data point in the first feature set with each data point in the second feature set. Determine the coordinate transformation between the aforementioned feature pairs. Based on the first imaging dataset, the second imaging dataset, and the coordinate transformation, at least one spatially combined image of the target region is generated. A system including the controller.

2. The system according to claim 1, wherein the target site is the eye.

3. The first set of features is a limited set that does not represent the information captured by the first modality. The second feature is a representation of the eye region captured by the second modality. The system according to claim 1.

4. The system according to claim 1, wherein the controller is configured to select at least one region of interest in the first imaging dataset, and the first feature set is extracted from the at least one region of interest.

5. The system according to claim 1, wherein the first imaging dataset includes data acquired by scanning multiple light source wavelengths.

6. The system according to claim 1, wherein the first modality is multispectral imaging and the second modality is optical coherence tomography ("OCT").

7. The first imaging dataset is captured by an OCT device, and the target area is the eye. The at least one spatially combined image extends the peripheral portion of the first imaging dataset to enable visualization of one or more structures behind the iris. The system according to claim 6.

8. The system according to claim 6, wherein the first neural network is a multilayer perceptron.

9. The system according to claim 8, wherein the second neural network is a convolutional neural network.

10. The system according to claim 1, wherein the first modality is fluorescence angiography and the second modality is photocoherence tomography.

11. The system according to claim 1, wherein the first set of features includes each feature point indicating a degenerative disease.

12. The first imaging dataset includes multiple scans, The controller is configured to identify and isolate abnormal regions as being within the range of one of the plurality of scans, or between two of the plurality of scans. The controller is configured to add at least one annotation, which indicates the abnormal region, to the at least one spatially combined image. The system according to claim 1.

13. The system according to claim 1, wherein the controller is adapted to selectively transfer each annotation created in the first imaging dataset to the second imaging dataset, and from the second imaging dataset to the first imaging dataset.

14. It is a system, A controller comprising at least one processor and at least one non-temporary tangible memory on which instructions for a method of spatially combining imaging data are recorded, wherein the execution of the instructions by the processor results in the controller A first imaging dataset of the eye is received from a first modality, and a second imaging dataset of the eye is received from a second modality, wherein the first modality is multispectral imaging, and the second modality is optical coherence tomography. Select at least one region of interest within the first imaging dataset, A first set of features is extracted from the at least one region of interest via the first neural network. A second set of features is extracted from the second imaging dataset via a second neural network. A feature pair is generated by matching each data point in the first feature set with each data point in the second feature set. Determine the coordinate transformation between the feature pairs, and generate at least one spatially combined image of the eye based in part on the coordinate transformation. A system including the controller.

15. The first set of features is a limited set that does not represent the information captured by the first modality. The second feature is a representation of the eye region captured by the second modality. The system according to claim 14.

16. The system according to claim 14, wherein the first neural network is a multilayer perceptron and the second neural network is a convolutional neural network.

17. The first imaging dataset includes multiple scans, The controller is configured to identify and isolate abnormal regions as being within the range of one of the plurality of scans, or between two of the plurality of scans. The controller is configured to add at least one annotation, which indicates the abnormal region, to the at least one spatially combined image. The system according to claim 14.

18. The system according to claim 14, wherein the controller is adapted to selectively transfer each annotation created in the first imaging dataset to the second imaging dataset, and from the second imaging dataset to the first imaging dataset.

Images