Ophthalmic image registration using deep learning based interpretable artificial intelligence
By using interpretable deep learning AI algorithms in ophthalmic image registration, detecting point features and performing transform model estimation, the problems of low efficiency and poor interpretability in the prior art are solved, and fast and accurate image registration is achieved and improved The safety of surgical procedures.
Patent Information
- Application Number
- CN202380034368.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-18
- Filing Date
- 2023-04-10
- Publication Date
- 2024-11-26
AI Technical Summary
Existing image registration technologies are inefficient and easy to converge to local optimal values in applications with limited resources and time-critical applications. Deep learning methods lack interpretability, making it difficult to obtain ground truth data, and skipping feature extraction and matching steps lead to It is difficult to understand.
Using an AI algorithm based on interpretable deep learning, deep neural networks are used to detect the point features of the eyes, perform point matching and transformation model estimation, image registration is realized through architectures such as U-shaped networks, and confidence scores are provided to judge the success of registration.
Improves the speed and accuracy of image registration, provides an interpretable registration process, ensures accurate alignment of images between different views, and enhances the safety and effectiveness of surgical procedures.
Smart Images

Figure CN119032379A_ABST
Abstract
Description
Background Art
[0001] Image registration is generally the process of transforming different image data sets into a coordinate system with matching imaging content (or features). Image registration can be used when analyzing images acquired from different viewpoints (or perspectives), multiple images acquired at different times, multiple images acquired using different sensors / modalities, or a combination thereof. Establishing image correspondences through image registration is essential for many clinical tasks, including but not limited to ophthalmic microsurgery (e.g., vitreoretinal surgery (such as retinotomy, retinal resection, autologous retinal transplantation, etc.), anterior segment surgery (such as cataract surgery, minimally invasive glaucoma surgery (MIGS)), etc.), diagnostic monitoring and evaluation (e.g., disease diagnosis and / or stage assessment), treatment, etc. Summary of the invention
[0002] In certain embodiments, an ophthalmic system for performing ophthalmic image registration is provided. The ophthalmic system includes one or more ophthalmic imaging devices, a memory including executable instructions, and a processor in data communication with the memory. The one or more ophthalmic imaging devices are configured to generate multiple images of a user's eyes. Each of the multiple images includes a different view of the eye. The processor is configured to execute the executable instructions to determine one or more segmented regions of the eye in each of the multiple images based on an evaluation of the image using one or more neural networks. The processor is also configured to execute the executable instructions to determine a set of point features of the eye in one or more segmented regions of the eye based on an evaluation of the image using at least one of the one or more neural networks in each of the multiple images. The processor is further configured to execute the executable instructions to generate a set of transformation information for transforming at least one of the multiple images based on performing one or more image processing operations on the set of point features in each of the multiple images. The processor is further configured to execute the executable instructions to transform at least one of the multiple images based on the set of transformation information. The processor is configured to transform at least one of the multiple images so that the multiple images with different views are in the same coordinate system by at least one of the following operations: scaling at least one of the multiple images, translating at least one of the multiple images, or rotating at least one of the multiple images.
[0003] In certain embodiments, a computer-implemented method for performing ophthalmic image registration is provided. The computer-implemented method includes obtaining multiple images of a user's eye. Each of the multiple images includes a different view of the eye. The computer-implemented method also includes determining, within each of the multiple images, one or more segmented regions of the eye within the image based on an evaluation of the image using one or more neural networks. The computer-implemented method also includes determining, within each of the multiple images, a set of point features of the eye within the one or more segmented regions of the eye based on an evaluation of the image using at least one of the one or more neural networks. The computer-implemented method further includes generating a set of transformation information for transforming at least one of the multiple images based on performing one or more image processing operations on the set of point features within each of the multiple images. The computer-implemented method further includes transforming at least one of the multiple images based on the set of transformation information. Transforming at least one of the multiple images includes at least one of the following operations so that the multiple images with different views are in the same coordinate system: scaling at least one of the multiple images, translating at least one of the multiple images, or rotating at least one of the multiple images. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] In order to be able to understand the above-mentioned features of the present disclosure in detail, the present disclosure briefly summarized above can be described in more detail with reference to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show exemplary embodiments and are therefore not to be considered as limiting the scope thereof, and other equally effective embodiments may be allowed.
[0005] Figure 1A An example system for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0006] Figure 1B Another example system for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0007] Figure 1C Another example system for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0008] Figure 2 An example workflow for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0009] Figure 3 Another example workflow for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0010] Figure 4 An example architecture of a neural network for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0011] Figure 5 is a flow chart of an example method for performing ophthalmic image registration in accordance with certain embodiments.
[0012] Figure 6 is a flow chart of an example method for performing ophthalmic image registration in accordance with certain embodiments.
[0013] Figure 7 An example workflow for the first stage of an ophthalmic image registration procedure according to certain embodiments is presented.
[0014] Figure 8 Another example workflow for the first stage of an ophthalmic image registration procedure according to certain embodiments is presented.
[0015] Fig. 9 An example workflow for the second stage of an ophthalmic image registration procedure according to certain embodiments is presented.
[0016] Fig.10 Another example workflow for the second stage of the ophthalmic image registration procedure according to certain embodiments is presented.
[0017] Fig.11A An example set of input images for ophthalmic image registration according to some embodiments is shown.
[0018] Fig. 11B Demonstrated in use Fig.11A An example of a transformed image after performing ophthalmic image registration on the set of input images shown in .
[0019] Fig. 11C Demonstrated Fig. 11B Example image overlay of the images shown in .
[0020] Fig.12 An example computing system for performing ophthalmic image registration in accordance with certain embodiments is presented.
[0021] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION
[0022] Conventional image registration typically involves iteratively performing multiple tasks, including, for example, feature extraction, feature matching, transformation model estimation, and image resampling and transformation, in order to find the best alignment between images. One problem with conventional image registration is that iteratively performing these tasks may be resource inefficient (e.g., a time-consuming / computation-intensive process), making it unsuitable for many time-critical and / or resource-limited applications. Another problem with conventional image registration may be that it may lead to inaccurate results because the process typically converges to a local optimum (e.g., the best solution within a set of neighboring candidate solutions) rather than a global optimum (e.g., the best solution among all possible solutions).
[0023] In some cases, deep learning methods can be applied to image registration to improve the speed and accuracy of the registration process. In a reference example, a one-step transformation estimation technique based on deep learning can receive a pair of images to be registered as input, and then output a transformation matrix. However, one challenge of these deep learning-based registration methods is that it may be difficult to obtain and generate ground truth data. For example, in the field of ophthalmology, such ground truth data may include reference standards for the measurements and / or positions of various eye tissues / structures. In addition, due to the skipping of the steps of feature extraction and feature matching, these deep learning methods are generally uninterpretable. That is, due to the complexity of deep neural networks, the content learned in the hidden layers of deep neural networks is generally unknown. This lack of interpretability generally makes it challenging to understand the behavior and prediction of deep learning neural networks. Therefore, it may be desirable to provide an improved deep learning-based method for performing ophthalmic image registration.
[0024] The embodiments described herein provide systems and techniques for performing ophthalmic image registration using an AI algorithm based on interpretable deep learning. The exemplary ophthalmic image registration techniques described herein can use deep neural networks to detect point features (also referred to as key points) in one or more regions of the eye. These point features can correspond to unique features of the eye that may always exist between different images of the eye. Together, these point features can be used as fingerprints of the eye. After detecting the point features, the image registration technique can perform point matching and transformation model estimation. In this way, the embodiments described herein can use interpretable image registration techniques (e.g., based on point features) to align images while achieving the speed and accuracy associated with deep learning-based methods (e.g., one-step transformation estimation based on deep learning).
[0025] Additionally or alternatively, in some embodiments, the ophthalmic image registration techniques described herein may determine whether images cannot be registered. For example, the ophthalmic image registration technique may receive as input images to be registered, determine a set of features within each image, determine a confidence score indicating the likelihood of successful image registration based on an analysis of a set of features from each image, and provide (e.g., display) an indication of the confidence score. If the confidence score is less than a threshold, the ophthalmic image registration technique may determine that the set of images does not meet the conditions for registration (e.g., there is a lack of correspondence between image features). On the other hand, if the confidence score is greater than or equal to the threshold, the ophthalmic image registration technique may determine that the set of images meets the conditions for registration (e.g., there is a correspondence between image features).
[0026] Although many of the following embodiments use a "U-NET" as a reference example of a deep learning artificial neural network that can be used by the ophthalmic image registration techniques described herein, the embodiments are not limited to the "U-NET" architecture and may include other types of deep learning artificial neural networks, such as encoder-decoder networks or autoencoders. As used herein, hyphenated reference numerals refer to specific instances of elements, while unhyphenated reference numerals refer to collective elements. Thus, for example, device '12-1' refers to an instance of a device class, which device class may be collectively referred to as device '12', and any one of the device classes may be generally referred to as device '12'.
[0027] Figure 1A A system 100A for performing ophthalmic image registration according to certain embodiments is shown. The system 100A includes an imaging device 110, a computing system 150, and an imaging device 120. The imaging devices 110, 120 represent various imaging devices. More specifically, the imaging device 110 represents a preoperative (pre-operative) imaging device used in a clinic to generate an image of a patient's eye prior to surgery. Examples of the imaging device 110 may be an optical coherence tomography (OCT) device, an optical biometer, a digital keratometer, a rotational (e.g., Scheimpflug) camera, an MRI device, a digital microscope (e.g., a three-dimensional stereo digital microscope), a digital camera, a digital fundus camera, and other imaging devices now known or later developed.
[0028] Imaging device 120 represents an intraoperative (intraoperative) device used in an operating room during a surgical procedure to generate images of a patient's eye. Examples of imaging device 110 are digital microscopes (e.g., three-dimensional stereo digital microscopes), digital cameras, digital fundus cameras, and other imaging devices now known or later developed. Computing system 150 represents various computing systems (or devices), including, for example, laptop computers, mobile computers (e.g., tablet computers or smart phones), server computers, desktop computers, imaging systems, systems embedded in medical devices, and the like.
[0029] In certain embodiments, the system 100A performs image registration of the plurality of images 142, 144 captured of the patient's eye, and presents the output of the image registration (e.g., output image 170A) to a user (e.g., a surgeon) during surgery. The images 142, 144 may include different views of the eye. The different views within the images 142, 144 may be due in part to the images 142, 144 being captured at different times, at different viewpoints, using different modalities, etc. Here, for example, the imaging device 110 may capture the image 142 of the patient's 102 eye during a preoperative phase, and the imaging device 120 may capture the image 144 of the patient's 102 eye during an intraoperative phase. In another example, the images 142, 144 may be captured during gyration of the eye when the patient 102 changes from a sitting position to a supine position during surgery.
[0030] In some embodiments, image 142 includes one or more annotations (or marks) made by a user (e.g., a surgeon or surgical staff member). For example, during the preoperative stage, the user can assess the clinical characteristics of the eye based on image 142 and annotate image 142 with information associated with the clinical characteristics.
[0031] The computing system 150 includes a registration component 152 that is configured to implement one or more techniques described herein to perform ophthalmic image registration. The registration component 152 may include a software component, a hardware component, or a combination thereof. In some embodiments, the registration component 152 obtains the images 142, 144, performs image registration on the images 142, 144, and generates an output image 170A based on the image registration. In some embodiments, the output image 170A is an image superposition of the aligned (or transformed) image 142 and image 144 (or an image superposition of the aligned (or transformed) image 144 and image 142). For example, the registration component 152 may generate the output image 170A in part by transforming the image 142 into the coordinate system of the image 144 or by transforming the image 144 into the coordinate system of the image 142.
[0032] As described in more detail below, the registration component 152 can use an explainable AI algorithm to perform ophthalmic image registration. In some embodiments, the AI algorithm includes a first stage and a second stage. In the first stage, (multiple) deep learning neural networks (also referred to herein as deep learning artificial neural networks, artificial neural networks, or neural networks) perform segmentation operations on each image 142, 144. For example, (multiple) deep learning neural networks can transform each image 142, 144 into a different score map / segmentation map, where the values in each score map / segmentation map indicate a specific area (or segment) of the eye (e.g., skin area, sclera, iris, pupil).
[0033] The second stage performs point feature detection (i.e., key point detection) in the eye for point matching. For example, the second stage may detect point features (corresponding to unique features) in a score map / segmentation map associated with images 142, 144. In addition, in the second stage, one or more computer vision algorithms (or techniques) are used to match the point features detected in the score maps of images 142 and 144. As a reference example, a root scale-invariant feature transform (SIFT) computer vision algorithm may be used to detect point features in the score maps of images 142, 144 and match them. In addition, a transformation matrix is generated (or calculated) based on the matching points. The transformation matrix may be used to transform (e.g., scale, translate, rotate, etc.) image 142 and / or image 144 into the same coordinate system so that the output image 170A includes the alignment of images 142 and 144.
[0034] Additionally, in some embodiments, a set of features of image 142 and a set of features of image 144 may be extracted from one or more layers of a (multiple) deep learning neural network in a first stage. In a second stage, a classification component (also referred to as a classifier) may compare the set of features of image 142 and the set of features of image 144 to determine whether images 142, 144 have a sufficient amount of features (e.g., above a threshold) that can be used for registration. The classification component may generate a confidence score indicating the likelihood of successful image registration based on the comparison. The classification component may provide (e.g., display) an indication of the confidence score to a user (e.g., a surgeon).
[0035] like Figure 1A As shown, output image 170A is provided to imaging device 120, which can present output image 170A for display to a user (e.g., a surgeon). When the user views the eye via imaging device 120, the display of output image 170A can provide the user with preoperative information about the eye, thereby improving the safety and / or effectiveness of the surgical procedure. For example, output image 170A can be used for eye tracking during surgery, lens alignment during surgery, image guidance, robot-assisted surgery, etc.
[0036] It should be noted that Figure 1A Reference examples of systems for performing ophthalmic image registration are shown, and in other embodiments, the system may have different configurations. Figure 1A Computing system 150 is depicted as being separate from imaging device 120, but in some embodiments, computing system 150 may be part of imaging device 120. Further, although Figure 1A The present invention demonstrates performing ophthalmic image registration for preoperative images and intraoperative images, but in other embodiments, the techniques described herein can be used to perform ophthalmic image registration in other scenarios / applications. By way of example and not limitation, according to some embodiments, Figure 1B An example system 100B for performing ophthalmic image registration is presented, and Figure 1C Another example system 100C for performing ophthalmic image registration is presented.
[0037] Figure 1B In some embodiments, the system 100B includes a computing system 150 and an imaging device 130. Figure 1B The system 100B in can perform ophthalmic image registration for accurate and robust three-dimensional (3D) eye tracking. For example, the imaging device 130 may include a 3D sensor configured to capture images 162-1 and 162-2 of the patient's 102 eye. Assuming that the 3D sensor is a stereo camera having two imaging sensors (or cameras or imagers) separated by a baseline, the image 162-1 can be captured using a first imaging sensor of the stereo camera, and the image 162-2 can be captured using a second imaging sensor of the stereo camera. The computing system 150 can perform ophthalmic image registration of the images 162 1-2 (via the registration component 152) to generate an output image 170B, and the output image 170B can be provided to the imaging device 130 for display (or presentation) to a user. The output image 170B can be used to diagnose various eye diseases, such as glaucoma, diabetic retinopathy, age-related macular degeneration, etc.
[0038] In certain embodiments, Figure 1B The system 100B in the embodiment may perform ophthalmic image registration to track certain eye tissues / structures (e.g., retinal vessels, scleral and limbal vessels, etc.) in real time. For example, the imaging device 130 may capture a plurality of images 162, wherein each image 162 corresponds to a different frame at a different time. For example, image 162-1 may correspond to t 0 The first frame at t, image 162-2 may correspond to t 1, the second frame at , etc. In these embodiments, computing system 150 (via registration component 152 ) may generate multiple output images 170B over a period of time based on the registration performed on captured images 162 for real-time tracking.
[0039] Figure 1C The system 100C in includes a computing system 150 and an imaging device 140. In some embodiments, the system 100C can perform ophthalmic image registration to diagnose, evaluate and / or monitor a patient's eye. Here, for example, the imaging device 140 can capture multiple images 172 1-N of the patient's 102 eye. Each image 172 can be an image of a different portion of a specific eye tissue / structure (e.g., fundus). The computing system 150 (via the registration component 152) can perform ophthalmic image registration of the images 172 1-N to integrate the images 172 1-N into a single image 170C. The computing system 150 can provide the image 170C to the display 174 and / or the imaging device 140 for presentation to a user.
[0040] It should be noted that although Figure 1B Computing system 150 is depicted as being separate from imaging device 130, but in some embodiments, computing system 150 may be part of imaging device 130. Similarly, although Figure 1C Computing system 150 is depicted as being separate from imaging device 140 , but in some embodiments computing system 150 may be part of imaging device 140 .
[0041] Figure 2 An example workflow 200 for performing ophthalmic image registration according to some embodiments is shown. The workflow 200 can be performed by the registration component 152. Here, the registration component 152 includes a deep learning tool 210, a post-processing tool 220, and a transformation tool 230, each of which can include hardware components, software components, or a combination thereof.
[0042] The deep learning tool 210 is typically configured to use one or more neural networks 212 (also referred to as artificial neural networks, deep neural networks, etc.) to perform segmentation operations on input images A and B and detect one or more point features within the input images A and B. The input images A and B can represent any set of input images to be registered (e.g., images 142, images 162, images 172, etc.). The (multiple) neural networks 212 are typically (multiple) deep learning networks that can be trained to perform various computer vision tasks, including, for example, image classification, image classification and localization, object detection, semantic segmentation, instance segmentation, etc.
[0043] The neural network(s) 212 may include a convolutional neural network. In some embodiments, the neural network(s) 212 are U-type networks. Figure 4 , Figure 4 An exemplary U-shaped network architecture 400 is shown in accordance with certain embodiments. The U-shaped network architecture 400 includes: (i) a plurality of multi-channel feature maps 412, each feature map having a plurality of channels (N) and a specific xy size; and (ii) a plurality of replicated multi-channel feature maps 414. The U-shaped network architecture 400 includes a contraction path 410 (left side) and an expansion path 420 (right side). The contraction path 410 may generally follow the architecture of a convolutional network. For example, each step of the contraction path 410 may include repeated application of one or more (e.g., two) 3×3 convolutions (e.g., unpadded convolutions), followed by rectified linear units (ReLUs) and 2×2 max pooling operations, where a stride of 2 is used for downsampling. At each downsampling step, the number of channels in each feature map 412 is doubled relative to the previous step.
[0044] Each step in the expansion path 420 includes upsampling the feature map 412, followed by a 2×2 convolution (e.g., upconvolution) that halves the number of feature channels, concatenating with the corresponding cropped feature map 412 from the contraction path 410, and one or more (e.g., two) 3×3 convolutions, each followed by a ReLU. In the last layer, a 1×1 convolution is used to map the feature vector to the desired number of classes.
[0045] Return to reference Figure 2 , the deep learning tool 210 is configured to generate (multiple) masks 214 (also referred to as segmentation results) and a set of point features 216 for each image A and image B using (multiple) neural networks 212. (Multiple) masks 214 indicate different regions of the eye within the respective images. For example, (multiple) masks 214 may indicate skin regions, sclera, iris, pupil, etc. Point features 216 generally include a set of features within the respective image that can be used to uniquely identify the image. In some examples, the unique features may correspond to different blood vessels of the eye visible within the respective images.
[0046] In some embodiments, the deep learning tool 210 can generate (multiple) masks 214 and point features 216 for each image A and image B using a single neural network 212. In these embodiments, the neural network 212 can be trained on a data set including multiple images, a set of point features within each image, and the boundaries of key areas (e.g., sclera, iris, pupil, etc.) within each image. The user can use machine learning tools or a combination thereof to annotate the point features and area boundaries within the set of training images. The single neural network 212 can be trained using various algorithms (including, for example, stochastic gradient descent).
[0047] In some embodiments, the deep learning tool 210 may use multiple neural networks 212 to generate (multiple) masks 214 and point features 216. For example, the deep learning tool 210 may generate (multiple) masks 214 using the first neural network 212 and generate point features 216 using the second neural network 22 for each image A and image B. In these embodiments, the first neural network may be trained on a dataset including multiple images and labeled boundaries of key areas within each image. The second neural network may be trained on a dataset including labeled boundaries of key areas within each image and a set of point features within the boundaries of the key areas within each image. Each neural network 212 may be trained using various algorithms, including, for example, stochastic gradient descent.
[0048] exist Figure 2 In the embodiment, the post-processing tool 220 uses one or more computer vision algorithms 222 (or image processing operations) (or a combination thereof) to evaluate the (multiple) masks 214 and point features 216 for images A and B to generate a set of transformation information 224. The computer vision algorithm 222 may include, but is not limited to, SIFT, random sample consensus (RANSAC), thresholding, non-maximum suppression (NMS), etc. The transformation information 224 may include scaling information, translation information, and / or rotation information for transforming at least one of the images A and B so that the point features 216 in each image A and B are aligned (or in the same coordinate system). In some embodiments, the transformation information 224 includes a homography matrix.
[0049] Transformation tool 230 receives transformation information 224 and applies transformation information 224 to at least one of images A and B to generate output image 170. As described above, in some embodiments, output image 170 may include a superposition of images A and B aligned using transformation information 224.
[0050] Figure 3 An example workflow 300 for performing ophthalmic image registration according to some embodiments is shown. The workflow 300 can be performed by the registration component 152. The workflow 300 is similar to the workflow 200, except that in the workflow 300, the post-processing tool 220 also includes a classification tool 330, which can include a software component, a hardware component, or a combination thereof.
[0051] The classification tool 330 is generally configured to determine whether a set of input images (e.g., images A and B) are eligible for registration based on an evaluation of a set of features from each input image. The classification tool 330 can generate a set of confidence information based on the evaluation, which indicates the likelihood that the transformation information 224 will result in successful alignment of the set of images. Here, for example, the post-processing tool 220 (via the classification tool 330) receives a set of image A features 310 and a set of image B features 312 from the deep learning tool 210. In some embodiments, the image A features 310 and the image B features 312 can be obtained from an intermediate layer of the neural network 212 (e.g., Figure 4 The feature map 412) in step i of the U-shaped network architecture 400 shown in FIG.
[0052] The post-processing tool 220 (via the classification tool 330) may compare the image A feature 310 with the image B feature 312 and generate a set of confidence information 340 based on the comparison. For example, the post-processing tool 220 may perform a cross-correlation of the image A feature 310 with the image B feature 312 and generate confidence information 340 including an output (or result) of the cross-correlation. In some embodiments, when the confidence information 340 indicates that the amount of correlation between the image A feature 310 and the image B feature 312 is greater than or equal to a threshold, the transformation tool 230 may proceed to transform the images A, B using the transformation information 224.
[0053] On the other hand, in some embodiments, when the confidence information 340 indicates that the amount of correlation between the image A feature 310 and the image B feature 312 is below a threshold, the transformation tool 230 can avoid using the transformation information 224 to transform the images A, B. In these embodiments, the registration component 152 can generate feedback with an indication that the images A, B do not meet the registration conditions. In some embodiments, this feedback information can be used to update the training of the neural network(s) 212.
[0054] Figure 5 is a flow chart of a method 500 for performing ophthalmic image registration according to some embodiments. The method 500 can be performed by a registration component (eg, the registration component 152).
[0055] The method 500 enters at block 502 where a registration component obtains a set of images of an eye of a patient (e.g., patient 102). Each image in the set of images can be captured at a different time, at a different angle, at a different location, using a different sensor / modality, etc.
[0056] At block 504, the registration component determines, for each image, one or more different regions of the eye within the image (e.g., mask(s) 214). For example, the registration component may use one or more neural networks (e.g., neural network 212) to perform a segmentation operation on each image to identify skin regions, sclera, iris, pupil, etc.
[0057] At block 506, the registration component determines, for each image, one or more point features of the eye within the image (e.g., point features 216). The one or more point features may correspond to one or more unique landmarks or points of the eye. The registration component may generate a set of point features based on evaluating each image using one or more neural networks (e.g., neural network 212).
[0058] In some embodiments, a single neural network 212 may be used to perform the operations in blocks 504 and 506. For example, Figure 7 A first stage 700 of ophthalmic image registration is shown using a single neural network 212 to identify masks 214 and point features 216 within each input image A and B, in accordance with certain embodiments. In the first stage 700, the neural network 212 may output, for each image, a mask 214-1 indicating skin regions, a mask 214-2 indicating sclera, and a mask 214-3 indicating pupils. The neural network 212 may also output, for each image, a set of point features within the image (e.g., point features 216A for image A and point features 216B for image B).
[0059] In some embodiments, multiple neural networks 212 may be used to perform the operations in blocks 504 and 506. For example, Figure 8 A first stage 800 of ophthalmic image registration using multiple neural networks 212-1 and 212-2 is shown in accordance with certain embodiments. In the first stage 800, the neural network 212-1 may segment different regions of each input image A and B. For example, the neural network 212-1 may identify within each input image a mask 214-1 indicating a skin region, a mask 214-2 indicating a sclera, and a mask 214-3 indicating a pupil. The neural network 212-2 receives the mask (or segment) information output from the neural network 212-1 and generates, for each image, a set of point features within the image (e.g., point features 216A for image A and point features 216B for image B).
[0060] Return to reference Figure 5At block 508, the registration component generates a set of transformation information (e.g., transformation information 224) based at least in part on the evaluation of the point features and the eye region within the set of images. In some embodiments, the registration component can generate the set of transformation information by evaluating the mask and the point features using one or more computer vision algorithms (e.g., computer vision algorithm 222). For example, Fig. 9 A second stage 900 of ophthalmic image registration is shown in which transformation information is generated using one or more computer vision algorithms in accordance with certain embodiments. As shown, during a post-processing stage of the second stage 900, point features 216 of input images A and B are matched using at least one computer vision algorithm (e.g., RANSAC, SIFT, NMS, etc.) and a set of transformation information 224 is generated.
[0061] At block 510, the registration component transforms at least one image in the set of images based on the set of transformation information. For example, the transformation information may include scaling information, translation information, and / or rotation information for transforming at least one image in the set of images so that point features in each image are aligned (or in the same coordinate system). For example, Fig. 11B Shows that in the given Fig.11A 1104B may be output from the registration component with the reference image 1102 and the surgical image 1104A shown in FIG. 1 as inputs to the registration component.
[0062] In some embodiments, the registration component can generate an image overlay having a set of transformed images (e.g., in the same coordinate system). Fig. 11C An image overlay 1106 is shown that may be generated by the registration component based on the transformed surgical image 1104B and the reference image 1102 .
[0063] Figure 6 is a flow chart of another method 600 for performing ophthalmic image registration according to some embodiments. The method 600 can be performed by a registration component (eg, the registration component 152).
[0064] The method 600 enters at block 602 where a registration component obtains a set of images of an eye of a patient (e.g., patient 102). Each image in the set of images can be captured at a different time, at a different angle, at a different location, using a different sensor / modality, etc.
[0065] At block 604, the registration component determines, for each image, one or more different regions of the eye within the image (e.g., mask(s) 214). For example, the registration component may use one or more neural networks (e.g., neural network 212) to perform a segmentation operation on each image to identify skin regions, sclera, iris, pupil, etc.
[0066] At block 606, the registration component determines, for each image, one or more point features of the eye within the image (e.g., point features 216). The one or more point features may correspond to one or more unique landmarks or points of the eye. The registration component may generate a set of point features based on evaluating each image using one or more neural networks (e.g., neural network 212).
[0067] At block 608, the registration component extracts a set of image features from the image for each image. In some embodiments, the set of image features can be extracted from an intermediate layer of one or more neural networks. For example, assuming that the U-type network architecture 400 is used as the neural network, the intermediate layer can correspond to the feature map 412 in step i of the U-type network architecture 400.
[0068] In some embodiments, a single neural network 212 may be used to perform the operations in blocks 604, 606, and 608. For example, Figure 7 A first stage 700 of ophthalmic image registration using a single neural network 212 is shown in accordance with certain embodiments. Here, the single neural network 212 receives input images A and B and outputs a mask 214 and point features 216 for each input image A, image A features 310, and image B features 312.
[0069] In some embodiments, multiple neural networks 212 may be used to perform the operations in blocks 604, 606, and 608. For example, Figure 8 A first stage 800 of ophthalmic image registration using multiple neural networks 212-1 and 212-2 is shown in accordance with certain embodiments. In the first stage 800, the neural network 212-1 can segment different regions of each input image A and B. The neural network 212-2 receives the mask (or segment) information output from the neural network 212-1 and generates a set of point features within the image (e.g., point features 216A for image A and point features 216B for image B), image A features 310, and image B features 312 for each image.
[0070] Return to reference Figure 6 At block 610, the registration component determines an amount of similarity between the multiple sets of image features. For example, the registration component can perform a cross-correlation (or another type of comparison) of the multiple sets of image features to determine an amount of correlation (or an amount of similarity) between the multiple sets of image features. Fig. 9In the second stage 900 shown in FIG. 1 , during a post-processing stage, the classification tool 330 may determine an amount of correlation between the image A features 310 and the image B features 312 and generate confidence information 340 including an indication of an amount of similarity between the multiple sets of image features.
[0071] At block 612, the registration component determines whether the amount of similarity satisfies a predetermined condition. The predetermined condition may include the amount of similarity being greater than a threshold. If the amount of similarity does not satisfy the predetermined condition, the registration component determines that the set of images does not meet the conditions for registration (block 614).
[0072] On the other hand, if the amount of similarity does satisfy the predetermined condition, the registration component generates a set of transformation information (e.g., using one or more computer vision algorithms 222) based at least in part on the evaluation of the point feature(s) and the eye region within the set of images (block 616). Fig. 9 In the second stage 900 shown in FIG. 1 , the registration component may output network predicted transformation information 910, the transformation information including the confidence information 340 and an indication of the transformation information 224. At block 618, the registration component transforms at least one image in the set of images based on the set of transformation information.
[0073] In some embodiments, in addition to determining the amount of similarity between multiple sets of image features, the registration component can also use information obtained from one or more computer vision algorithms to determine whether the input images are registerable. Fig.10 An example second stage 1000 of ophthalmic image registration is shown in accordance with certain embodiments. During a post-processing stage of the second stage 1000, point features of input images A and B are matched using a first one or more computer vision algorithms 222 (e.g., RANSAC / SIFT) to generate transformation information 224. Additionally, the input images A and B may be evaluated using a second one or more computer vision algorithms 222 to generate transformation information 1002. In certain embodiments, the second one or more computer vision algorithms 222 may include a thresholding algorithm. The registration component may perform thresholding on the input images A and B to estimate the value of each image (at Fig.10 1004A and 1004B). The transformation information 1002 may be generated based on a comparison of the limbus center and radius information from input image A with the limbus center and radius information from input image B. For example, the transformation information 1002 may include scaling information and / or translation information for transforming at least one image in the set of images.
[0074] continue Fig.10, the classification tool 330 receives the image A feature 310, the image B feature 312, the transformation information 1002, and the transformation information 224, and generates confidence information 340 based on the evaluation of the input information. The confidence information 340 may indicate whether the input images A and B meet the conditions for registration. For example, the confidence information 340 may include an indication of the amount of correlation between the features of the input images. In some embodiments, the confidence information 340 may indicate that the input images do not meet the conditions for registration when it is determined that at least one of the following: (i) the transformation information 1002 is inconsistent with the transformation information 224, or (ii) the amount of correlation between the image A feature 310 and the image B feature 312 is less than a threshold. For example, when the scaling information / translation information in the transformation information 1002 does not match the scaling information / translation information in the transformation information 224, the transformation information 1002 may be inconsistent with the transformation information 224.
[0075] Fig.12 An example computing system 1200 configured to perform ophthalmic image registration according to certain embodiments is shown. As shown, the computing system 1200 includes, but is not limited to, a processing unit 1205, a network interface 1215, a memory 1220, and a storage device 1260, each of which is connected to a bus 1217. The computing system 1200 may also include an I / O device interface 1210 that connects an I / O device 1212 (e.g., a keyboard, a display, and a mouse device) to the computing system 1200. The computing system 1200 is typically under the control of an operating system (not shown). Examples of operating systems include UNIX operating systems, multiple versions of Microsoft Windows operating systems, and distributions of Linux operating systems. (UNIX is a registered trademark of the International Open Standards Organization in the United States and other countries. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.) More generally, any operating system that supports the functionality disclosed herein may be used.
[0076] The processing unit 1205 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The processing unit 1205 retrieves and executes programming instructions stored in the memory 1220 and stored in the storage device 1260. The bus 1217 is used to transmit programming instructions and application data between the processing unit 1205, the I / O device interface 1210, the storage device 1260, the network interface 1215 and the memory 1220. Note that the processing unit 1205 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, a single GPU, multiple GPUs, a single GPU with multiple processing cores, or any combination thereof. The memory 1220 is generally included to represent random access memory. The storage device 1260 may be a disk drive or a flash memory device. Although shown as a single unit, the storage device 1260 may be a combination of fixed and / or removable storage devices, such as a fixed disk drive, a removable memory card, an optical storage device, a network attached storage device (NAS) or a storage area network (SAN). Illustratively, the memory 1220 includes the registration component 152, which is discussed in more detail above. Further, the storage device 1260 includes the mask 214, the point features 216, the transformation information 224, and the confidence information 340, which are also discussed in more detail above. The storage device 1260 also includes input images 1262, which can include any set of input images obtained for the ophthalmic image registration procedure.
[0077] In summary, embodiments of the present disclosure enable the use of interpretable deep learning-based AI to perform ophthalmic image registration. The systems and / or methods described herein are particularly advantageous for providing additional data points and information to improve diagnostic accuracy and / or effectiveness of surgical procedures.
[0078] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. For example, "at least one of a, b, or c" is intended to cover any combination of a, b, c, ab, ac, bc, and abc, as well as multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other order of a, b, and c).
[0079] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be given the full scope consistent with the language of the claims.
[0080] In the claims, unless otherwise specified, reference to a singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly recited in the claims, the content disclosed herein is not intended to be committed to the public. According to 35 U.S.C. §112 (f), the elements of any claim will not be interpreted unless the phrase "device for..." is used to expressly recite these elements, or in the case of a method claim, the phrase "step for..." is used to recite these elements. The word "exemplary" is used herein to mean "used as an example, instance, or illustration". Any aspect described as "exemplary" herein is not necessarily interpreted as preferred or advantageous over other aspects.
Claims
1. An ophthalmic system for performing ophthalmic image registration, include: one or more ophthalmic imaging devices configured to generate a plurality of images of an eye of a user, each of the plurality of images comprising a different view of the eye; a memory including executable instructions; as well as a processor in data communication with the memory and configured to execute the executable instructions to perform the following operations: determining, within each image of the plurality of images, one or more segmented regions of the eye within the image based on evaluating the image using one or more neural networks; determining, within each of the plurality of images, a set of point features of the eye within one or more segmented regions of the eye based on evaluation of the image using at least one of the one or more neural networks; generating a set of transformation information for transforming at least one of the plurality of images based on performing one or more image processing operations on the set of point features within each of the plurality of images; as well as Transforming at least one of the multiple images based on the set of transformation information, wherein the processor is configured to transform the at least one of the multiple images to include: scaling the at least one of the multiple images, translating the at least one of the multiple images, or rotating the at least one of the multiple images so that the multiple images with the different views are in the same coordinate system.
2. The ophthalmic system according to claim 1, in, The processor is further configured to execute the executable instructions to extract a set of features associated with each of the plurality of images from at least one of the one or more neural networks.
3. The ophthalmic system according to claim 2, in, The set of features associated with each of the plurality of images is extracted from an intermediate layer of the at least one of the one or more neural networks.
4. The ophthalmic system according to claim 3, in, The processor is further configured to execute the executable instructions to perform the following operations: determining an amount of similarity between the plurality of images based on comparing the set of features; and An indication of the amount of the similarity is provided.
5. The ophthalmic system according to claim 4, in, Upon determining that the amount of similarity satisfies a predetermined condition, the at least one image among the plurality of images is transformed.
6. The ophthalmic system of claim 1, in, The one or more neural networks is a single neural network.
7. The ophthalmic system of claim 1, in, The one or more neural networks include: (i) a first neural network configured to provide an indication of the one or more segmented regions of the eye; and (ii) a second neural network configured to provide an indication of the set of point features.
8. The ophthalmic system of claim 1, in, The processor is further configured to execute the executable instructions to generate an image overlay comprising at least one transformed image of the plurality of images and one or more untransformed images of the plurality of images.
Citation Information
Cited By
Ophthalmologic video image processing method, processing device and processing equipment
CN121545690A
Methods, devices, and equipment for processing ophthalmic video images
CN121545690B
