Training based on invisible spectral light image and use of machine learning model

Abstract

An illustrative system may access a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images including a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second sequence of images captured by the imaging device during the medical procedure, the second sequence of images comprising a second image based on illumination of the scene using the invisible spectral light; and providing the first image sequence and the second image sequence to a machine learning module.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 352,813, filed on June 16, 2022, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Light-based image data captured during a medical procedure has many uses during and after such procedure. For example, medical image data from an endoscope can be displayed during a medical procedure to assist medical personnel in performing the procedure. As another example, medical image data captured during a medical procedure can be used as control signals for a computer-assisted medical system. As another example, medical image data captured during a medical procedure can also be used after the medical procedure for postoperative evaluation, diagnosis, guidance, etc.

[0004] Various illumination and image sensing techniques have been used to capture images of medical procedures. Visible spectrum light emitting sources and image sensors have been used to capture color (white light) images of medical procedures. Non-visible spectrum image sensors, sometimes paired with non-visible spectrum light emitting sources, have been used to capture non-visible spectrum images of medical procedures. Summary of the invention

[0005] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements / elements of all aspects, nor to outline the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed embodiments presented below.

[0006] An illustrative system includes: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence including a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second image sequence captured by the imaging device during the medical procedure, the second image sequence including a second image, the second image being based on illumination of the scene using non-visible spectrum light; and processing the first image sequence and the second image sequence using a machine learning module.

[0007] Another illustrative system includes: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence comprising a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence comprising a second image; and performing an operation on the first image sequence based on an output of the machine learning model.

[0008] Another illustrative system includes: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence comprising a first image, the first image being based on illumination of a scene associated with the medical procedure by visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence comprising a second image; and generating a prediction based on an output of the machine learning model.

[0009] An illustrative method includes accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second image sequence captured by the imaging device during the medical procedure, the second image sequence comprising a second image based on illumination of the scene using non-visible spectrum light; and processing the first image sequence and the second image sequence using a machine learning module.

[0010] Another illustrative method includes: accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence including a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; and providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and performing an operation on the first image sequence based on an output of the machine learning model.

[0011] Another illustrative method includes: accessing a first image sequence captured by an imaging device during a medical procedure, the first image sequence including a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and generating a prediction based on an output of the machine learning model.

[0012] An illustrative non-transitory computer-readable medium may store instructions that, when executed, direct a processor of a computing device to: access a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images including a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; access a second sequence of images captured by the imaging device during the medical procedure, the second sequence of images including a second image based on illumination of the scene using non-visible spectrum light; and process the first sequence of images and the second sequence of images using a machine learning module.

[0013] Another illustrative non-transitory computer-readable medium may store instructions that, when executed, direct a processor of a computing device to: access a first image sequence captured by an imaging device during a medical procedure, the first image sequence comprising a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; and provide the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence comprising a second image; and perform an operation on the first image sequence based on an output of the machine learning model.

[0014] Another illustrative non-transitory computer-readable medium may store instructions that, when executed, direct a processor of a computing device to: access a first image sequence captured by an imaging device during a medical procedure, the first image sequence comprising a first image, the first image being based on illumination of a scene associated with the medical procedure using visible spectrum light; provide the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence comprising a second image; and generate a prediction based on an output of the machine learning model BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0016] Figure 1 A system for capturing an image of a scene associated with a medical procedure is shown.

[0017] Figure 2 A machine learning module is shown.

[0018] Figure 3 The machine learning data flow is shown.

[0019] Figure 4 An embodiment of generating labels for a first image sequence based on a second image sequence is shown.

[0020] Figure 5An embodiment using image blending is shown.

[0021] Figure 6 An example of the output of a trained machine learning model is shown.

[0022] Figure 7 A computer-assisted medical system is shown.

[0023] Figure 8 An illustrative computing device is shown. DETAILED DESCRIPTION

[0024] Techniques for implementing machine learning about visible spectrum images using invisible spectrum images are described herein. Given a first sequence of images of a scene associated with a medical procedure (e.g., visible spectrum images) and a second sequence of scene images (e.g., invisible spectrum images), both the first sequence and the second sequence can be used directly or indirectly to train a machine learning model that can produce outputs that would not be possible when only one or the other is used for training. As described herein, the output of the machine learning module can be used to perform various operations on the first image sequence and / or a computer-assisted medical system, which can advantageously provide the various benefits described herein.

[0025] As used herein, "visible spectrum images" and "visible spectrum videos" refer to images and videos whose pixel values ​​represent the sensed intensity of visible spectrum light. "Invisible spectrum images" and "invisible spectrum videos" refer to images and videos whose pixel values ​​represent the sensed intensity of invisible spectrum light. For the sake of brevity, this article will use "image" to refer to both images and videos. Illustrative invisible spectrum images include fluorescence images, hyperspectral images, and other types of images that do not rely entirely on visible spectrum illumination. For example, a fluorescence image is an image of fluorescent light emitted from a substance when the substance is illuminated by an invisible spectrum light source. Infrared images are another type of invisible spectrum image. Infrared images are images captured by a sensor capable of sensing infrared band light. For example, infrared light can include light emitted by an illuminated fluorophore.

[0026] As used herein, "label" means any type of data that indicates an object or other feature represented in an image, including but not limited to graphical or text-based annotations, identifiers, highlights, enhancements, and overlays. Labels applied to images may be embedded in the image file as metadata or stored in a separate data structure linked to the image file. Labels may be presented to a user, for example, as an enhancement to the image, or may be used for other purposes that do not necessarily involve presentation, such as the training of a machine learning model.

[0027] As used herein, a "medical procedure" may refer to any procedure performed on a patient using manual and / or instrumental techniques to investigate, diagnose, or treat a patient's physical condition. In addition, a medical procedure may refer to any non-clinical procedure, e.g., a procedure not performed on a live patient, such as a calibration or testing procedure, a training procedure, and an experimental or research procedure.

[0028] Figure 1 A system 100 is shown for capturing an image of a scene 102 associated with a medical procedure. The scene 102 may include a surgical field associated with a subject / body (e.g., a body of a living animal, a human or animal cadaver, a portion of a human or animal anatomy, tissue removed from a human or animal anatomy, a non-tissue artifact, a physical training model, etc.) on or in which a medical procedure is being performed. For example, the scene 102 may include various types of tissue (e.g., tissue 104), organs (e.g., organ 106), and / or non-tissue objects (e.g., object 108), such as instruments, objects held or manipulated by instruments, etc.

[0029] One or more light sources 110 can illuminate scene 102. As described above, light sources 110 can include any combination of white light sources, narrow-band light sources (whether in the visible spectrum or not, such as ultraviolet lamps), lasers, infrared light emitting diodes (LEDs), etc. If fluorescence is to be captured, the type of light source may depend on the fluorescent agent or protein used during the medical procedure. In some embodiments, the light source can provide light in the visible spectrum, but the fluorescence it induces may not be in the visible spectrum.

[0030] Further with respect to fluorescence, in some embodiments of the system 100, the light source (i.e., the excitation light source) used for fluorescence illumination can have any wavelength outside the visible spectrum. For example, a fluorescent light source, such as indocyanine green (ICG), can produce light with a wavelength in the infrared radiation region (e.g., about 700nm to 1mm), such as the near infrared ("NIR") radiation region (e.g., about 700nm-950nm), the short wave infrared ("SWIR") radiation region (e.g., about 1400nm-3000nm), or the long wave infrared ("LWIR") radiation region (e.g., about 8000nm-1500nm). Additionally or alternatively, the fluorescent light source can output light with a wavelength of about 350nm or shorter (e.g., ultraviolet radiation). In some embodiments, the fluorescent light source can be specifically configured for optical coherence tomography imaging.

[0031] The system 100 also includes an imaging device 112. The imaging device 112 receives light reflected, emitted, and / or fluoresced from a subject of a medical procedure and converts the received light into image data. The imaging device 112 senses light from the scene 102 and outputs a first image sequence 114 and a second image sequence 116 of the scene. The first image sequence 114 can be a sequence of visible spectrum images 118 of light sensed in the visible spectrum. The second image sequence 116 can be a sequence of invisible spectrum images 120 of light sensed in the invisible spectrum. For example, the second image sequence can be based on illumination of the scene using invisible spectrum light and / or illumination of the scene associated with different medical procedures. Alternatively, the second image sequence can include visible light images with labels generated based on invisible light images.

[0032] Figure 1 The image sequence shown in can be in the form of individual images, encoded video streams, etc. Figure 1 As shown, since the image sequences come from different spectra (or partially non-overlapping spectra), the contents of each image sequence may be different; some features of a site may be represented by one sequence but not by another sequence.

[0033] The imaging device 112 may have a first image capture device 122 and a second image capture device 124. Either image capture device may be any type of device capable of converting photons into electrical signals, such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, a photomultiplier, etc. Regardless of the type of image capture device used, the imaging device 112 may be configured to sense light within and outside the visible spectrum, as described above. In some embodiments, the first image capture device 122 senses light in the visible spectrum, while the second image capture device 124 senses light in the invisible spectrum. The first image capture device 122 and the second image capture device 124 may be separate sensors within a single camera, or they may be separate sensors in separate individual cameras. In some embodiments, the imaging device 112 may include only one image capture device (e.g., one sensor), and the image capture device may be capable of sensing in the visible spectrum and in one or more invisible spectrums at the same time. For example, some image sensors may be capable of sensing in the visible spectrum and in the infrared spectrum at the same time. In other embodiments, the imaging device 112 may be a stereo camera and may have two cameras, each capable of sensing in the visible spectrum and in the invisible spectrum. In some embodiments, the imaging device 112 and the light source 110 may be part of (or optically connected to) an endoscope.

[0034] In one embodiment, the first image capture device 122 may continuously capture the first image sequence 114 as video data of the medical procedure, and the second image capture device 124 may intermittently capture images of the second image sequence 116. For example, the first image capture device 122 may capture a video frame (first image 118) every 1 / 60 second, and the second image capture device 124 may capture the second image 120 once per second. This is more fully described in co-pending U.S. Provisional Patent Application No. __, entitled “Non-visible-spectrum Light Image-based Operations for Visible-spectrum Images,” filed on the same date as this application, and incorporated herein by reference in its entirety.

[0035] like Figure 1 As shown, image processing system 126 may be configured to access (eg, receive) first image sequence 114 and second image sequence 116 to perform various operations on the image sequences, as described below.

[0036] The image processing system 126 may be implemented by one or more computing devices and / or computer resources (e.g., processors, memory devices, storage devices, etc.), which may serve a particular implementation. As shown, the image processing system 126 may include, but is not limited to, a memory 128 and a processor 130 selectively and communicatively coupled to each other. The memory 128 and the processor 130 may each include or be implemented by computer hardware configured to store and / or process computer software. Figure 1 Various other components of computer hardware and / or software not explicitly shown in the drawings may also be included in image processing system 126. In some examples, memory 128 and processor 130 may be distributed among multiple devices and / or multiple locations, as may serve a particular implementation.

[0037] The memory 128 may store and / or otherwise maintain executable data used by the processor 130 to perform any of the functions described herein. For example, the memory 128 may store instructions 132 that may be executed by the processor 130. The memory 128 may be implemented by one or more memories or storage devices, including any memory or storage device described herein, which is configured to store data in a transient or non-transient manner. The instructions 132 may be executed by the processor 130 to cause the image processing system 126 to perform any of the functions described herein. The instructions 132 may be implemented by any suitable application, software, code, and / or other executable data instance. In addition, the memory 128 may also maintain any other data that is accessed, managed, used, and / or transmitted by the processor 130 in a particular embodiment.

[0038] The processor 130 may be implemented by one or more computer processing devices, including a general-purpose processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, etc.), a special-purpose processor (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), a digital signal processor, etc. Using the processor 130 (e.g., when the processor 130 is directed to perform operations represented by instructions 132 stored in the memory 128), the image processing system 126 may perform various operations described herein.

[0039] Various embodiments of the image processing system 126 and how the image processing system 126 can be configured to implement machine learning techniques will now be described with reference to the accompanying drawings. The various modules described herein can be included in the image processing system 126 and can be implemented by any suitable combination of hardware and / or software. Therefore, these modules represent various functions that can be performed by the image processing system 126 alone or in combination with any other functions performed by the image processing system 126 and / or its components as described herein.

[0040] Figure 2 A machine learning module 150 is shown that can be implemented by the image processing system 126. The machine learning module 150 receives the first image sequence 114, the second image sequence 116, or both. In some embodiments, the machine learning module 150 is a supervised machine learning algorithm that generates and trains a machine learning model based on either or both of the image sequences. In some embodiments, more than two image sequences can be input into the machine learning module 150, in which case one or more of the image sequences can be used to train the machine learning module 150.

[0041] In some embodiments, the machine learning module 150 may be implemented by one or more of a regression algorithm, a decision tree algorithm, a random forest algorithm, a logistic regression algorithm, a support vector machine algorithm, a naive Bayes classifier algorithm, a linear regression algorithm, a neural network algorithm, etc. In an embodiment, when the machine learning module 150 is a machine learning algorithm, the output 152 of the machine learning module 150 is a trained machine learning model.

[0042] In other embodiments, the machine learning module 150 is a machine learning model that has been trained based on one or more image sequences. In these embodiments, the output 152 of the machine learning module is a prediction based on the working data, which can be data such as a first image sequence (e.g., visible spectrum images), a second image sequence (e.g., non-visible spectrum images), or both. As described below, the predictions (output 152) (model) of the machine learning module 150 can be a predicted image (e.g., a synthetic image), a feature predicted in an image of the working data (e.g., a type of tissue or object), a predicted category of the working data image (e.g., a predicted stage of a medical procedure), a predicted segmentation, and other contents discussed below.

[0043] Figure 3 A machine learning data flow is shown. Training data 170 may include a first training image sequence 172 and a second training image sequence 174. Although training data 170 includes two image sequences of different types of images (e.g., visible spectrum images and non-visible spectrum images, respectively), in different embodiments, training data 170 passed to machine learning algorithm 176 may include one of the image sequences (e.g., modified or labeled according to the other image sequence), two of the image sequences (e.g., one modified or labeled according to the other), a third image sequence (not shown) of images derived from the two image sequences (e.g., a sequence of hybrid images), or an image sequence including a sequence based on non-visible light images imaged at a different wavelength than the second image sequence, an image sequence including hyperspectral images across a wide wavelength range, etc. Variants of training data 170 are discussed further below.

[0044] The machine learning algorithm 176 receives the training data 170 and generates a machine learning model 178. The type of model generated by the machine learning algorithm 176 will depend on which machine learning algorithm is used in any given implementation.

[0045] The trained machine learning model 178 is used by inputting the working data 180 into the machine learning model 178, which in turn generates and outputs a prediction 182. As with the training data 170, the working data 180 may include a first working image sequence 184 and a second working image sequence 186. Although the working data 180 may include two image sequences of different types of images (e.g., visible spectrum images and non-visible spectrum images, respectively), as with the training data 170, in different embodiments, the working data 180 passed to the machine learning model 178 may be one of the working data image sequences (e.g., modified or labeled according to the other image sequence), two working image sequences (e.g., one may be modified or labeled according to the other), a third working image sequence (not shown) of images derived from the two working image sequences (e.g., a sequence of hybrid images), etc. In any case, the machine learning model 178 generates a prediction 182 based on the working data 180.

[0046] In some examples, the number of sequences in working data 180 may be the same as the number of sequences in training data 170. In some alternative examples, the number of sequences in working data 180 may be different from the number of sequences in training data 170. For example, training data 170 may include visible light images and non-visible light images for labeling visible light images. Thus, machine learning model 178 may be trained using labeled visible light images to generate predictions based on working data 180 that includes, for example, only visible light images.

[0047] Predictions 182 may be labels indicating features in an image of working data 180 (e.g., tissue type, anatomical features, detected or identified objects, etc.), image segmentation in working data 180, images synthesized from images in working data 180 (synthetic images), features extracted from images in working data 180, predicted categories of images in working data 180 (or features thereof), the geometry of a scene represented by an image in working data 180, and other content discussed later.

[0048] In some embodiments, prediction 182 is a useful product in itself, without further computation being required therefrom. For example, prediction 182 may be used for post-procedural evaluation (e.g., prediction of the stage of a medical procedure), human guidance, medical diagnosis, etc. In some embodiments, prediction 182 about working data 180 is provided to computer-assisted medical system 188 (see below). Figure 7), the computer-assisted medical system 188 can use the prediction 182 in various ways. For example, the prediction 182 can be an image displayed by the computer-assisted medical system 188. The prediction 182 can be used to control the movement of various components of the computer-assisted medical system or various components connected thereto (e.g., by controlling a manipulator arm of the computer-assisted medical system, preventing instruments from moving near predicted anatomical features, etc.). The prediction 182 can be used to inform the content of a user interface of the computer-assisted medical system 188, for example, when to display markers and / or graphics of subsurface (or intratissue) anatomical structures or labels of anatomical features. The prediction 182 can be used to control the imaging mode of the computer-assisted medical system 188, trigger video capture, etc.

[0049] Figure 4 An embodiment is shown in which labels are generated for the first image sequence 114 based on the second image sequence 116 (eg, based on features in the second image sequence 116). As described above, the first image sequence 114 and the second image sequence 116 may be visible spectrum images and non-visible spectrum images, respectively.

[0050] exist Figure 4 In the illustrated embodiment, the second image sequence 116 is passed to an image processing module 200. The image processing module 200 may be encoded with one or more image processing algorithms to perform image analysis on the images in the second image sequence 116. The image processing module 200 may perform image processing operations such as feature detection and recognition, feature enhancement, and the like. Features 202 may be detected and recognized based on known properties of pixels of a particular type of non-visible spectrum imaging technique (e.g., fluorescence imaging) used for the second image sequence 116. For example, pixels having a color or intensity value within a color range or intensity range may correspond to one type of organ (or tissue or object), while pixels having a color or intensity value within another color range or intensity range may correspond to another type of organ, tissue, object, and the like. Areas (patches) of consecutive pixels of the same type may be labeled separately, pixels may be labeled according to their type, blocks (or other shapes) containing a threshold ratio of pixels of the same type may be labeled, and the like. In some embodiments, labels may be associated with each second image itself. For example, a second image determined to contain one or more types of tissue, organ, or object may be labeled accordingly. The second image having pixel values ​​indicative of the presence of cancerous tissue may be labeled accordingly.Because the image processing module 200 receives a sequence of images, in some examples, the image analysis performed by the image processing module 200 may include an inter-image analysis.

[0051] When the image processing module 200 completes processing of the second image, the image processing module 200 outputs a sequence of labeled second images 204, which, in one embodiment, is provided to the image labeling module 206. The image labeling module 206 may also receive the first image sequence 116. The image labeling module 206 may ensure that a given labeled second image 204 is associated with an image in the first image sequence 116 (a labeled second image 204 may be associated with and provided with a label for one or more first images, but for the sake of brevity, only one first image will be mentioned). This may involve steps such as comparing image timestamps to match the labeled second image 204 with the first image. In some embodiments, when the labeled second image 204 has been paired with the first image, a geometric transformation (e.g., affine, scaling) may be performed on one or both images to geometrically align the labeled second image 204 with its corresponding first image (i.e., any two corresponding pixels in the respective first and second images represent the same point of the scene). Note that in some embodiments the temporal pairing and transformation operations may be omitted; the pairing may be implicit (i.e., the image stream to the image labeling module 206 may inherently match the temporally related images) and geometric misalignment may not exist or may not affect the labeling of the first image.

[0052] Regardless of whether any temporal pairing or transformation operations are performed, the image labeling module 206 labels the first image according to the labels of the labeled second image 204. In the case where the labeled second image 204 is geometrically aligned with the first image (i.e., the first image and the second image represent the same scene on a pixel-by-pixel basis), the feature labels of the labeled second image 204 can be directly transferred to the first image. In some embodiments, the image labeling module 206 can perform object / feature detection, segmentation, etc., and then attempt to match features in the first image with features in the labeled second image 204, such as based on the shape, intensity, location, etc. of the features. When features in the labeled second image 204 match features in the first image, the features in the first image are labeled according to the matching features in the labeled second image 204. As described above, the labels of the second image can be associated with the image, but not with any particular feature of the image, in which case the first image itself is labeled accordingly.

[0053] As time goes by, the above labeling process is repeated for subsequent first images and second images, thereby forming a labeled first image sequence 208. Then, the images in the first image sequence 116 are provided to the machine learning module 150, that is, the labeled first image sequence 208 is provided to the machine learning algorithm to train the machine learning model, or provided to the trained machine learning model, which calculates the prediction of the corresponding labeled first image sequence 208.

[0054] In some embodiments, the image labeling module 206 and the labeling of the first image are omitted. Instead, the second image sequence is labeled by the image processing module 200 as described above. Figure 4 As indicated by the dashed arrow in , the labeled second image 204 and the first image sequence 116 are passed to the machine learning module 150. Assuming that the machine learning module 150 processes the first image and the second image pair simultaneously, the first image and the second image provide a combined signal of the associated visible spectrum image data and the labeled invisible spectrum image data for machine learning training or prediction (as the case may be). If the machine learning module 150 is a model trained using the labeled second image and the first image, it can output a prediction about the first image. Such a prediction can be a predicted label for a feature in the first image and / or the second image, an enhanced first image, a segmentation of the first image, a category of the first image, etc.

[0055] In some examples, the image processing module 200 and the image labeling module 206 can be implemented (e.g., as submodules) by the machine learning module 150. Therefore, in some examples, the machine learning module 150 can be configured to perform the operations performed by the image processing module 200 and the image labeling module 206 as described herein.

[0056] Figure 5 An embodiment using image blending is shown. In this embodiment, second images and time-corresponding first images are passed to the image blending module 230. Each second image received by the image blending module 230 is paired with one or more first images (for simplicity, the first images will be represented in the singular) also received by the image blending module 230. The image blending module 230 may perform a geometric transformation to geometrically align the first image and the second image (e.g., so that the images represent the same view of the scene (the corresponding pixels represent the same point of the scene). The image blending module 230 may create a composite image based on the image data from the first image and the second image. For example, the image blending module 230 may perform feature detection, segmentation, etc. on the two images, and may create a composite image 232 by combining the features from the two images. If it is determined that two features in the respective images match, one may be selected to be included in the composite image 232 by matching the two features (e.g., based on orientation, shape, intensity, etc.). If a feature (e.g., an object or a piece of tissue) is found in one image but not in the other image, the feature may be included in the composite image. In some embodiments, one of the images may be used as an initial version of the composite image 232, and the initial version is modified according to the content in the other image. In one embodiment, both images are segmented, and the composite image 232 is formed by the union of the segments of the two images.

[0057] The composite image 232 constructed from the corresponding first and second image pairs is passed to the machine learning module 150, which trains the model (if the machine learning module is a training algorithm) or generates predictions about the composite image 232. The predictions can be any of the types of predictions discussed above. In some embodiments, Figure 5 As indicated by the dashed arrows in , either or both of the first image sequence 114 and the second image sequence 116 are also passed to the machine learning module 150, thereby providing additional training or prediction data. In addition, feature detection and labeling can be performed on any image sequence provided to the machine learning module 150.

[0058] Figure 6 An example of the output of the trained machine learning model 178 is shown. The pre-processing module 250 can generate any variation of the above image sequence based on the working data 180. For example, the pre-processing module 250 can output various combinations of labeled second images, composite images, labeled first images, segmented images, etc. The machine learning model 178 then outputs one or more predictions. For example, the prediction can be a composite image 252, which can include image data from the first image and the second image. For example, the composite image 252 can be a union of features from the first image and the second image. The composite image 252 can be an enhanced first (or second) image, for example, the pixel values ​​are changed to highlight the features, to form a clearer or more uniform feature, etc. Another possible output can be a segmented image 254, where the segments are identified by separate bit masks, by enhancing the pixels on the boundaries of the segments, coloring the pixels therein according to the identified segment type, etc. Another possible output is a labeled image 256. The labeled image 256 can have predicted labels 258 for each feature (which can itself be a prediction), a set of identifiers for each feature, etc. Another possible output is a classified image 260. The machine learning model 178 outputs one or more predicted class identifiers 262 for each image, if any. For example, the machine learning model may predict the stage of a medical procedure, the class of any feature detected in the image (e.g., the type of object present, the type of tissue or organ present, etc.), or other. The machine learning model 178 may predict the geometry of the depicted scene, e.g., the predicted depth of a feature or a particular pixel, the predicted distance between features, the reconstructed three-dimensional geometry of the scene, etc. Any combination of the above predictions may be output.

[0059] As described above, machine learning predictions informed by invisible spectral image data can serve as useful outputs in their own right. For example, predictions can be used for teaching, postoperative assessment, identifying critical stages of a procedure, estimating anatomical dimensions, etc. During a medical procedure, notifications can be provided to operating room personnel. For example, notifications can be rendered as sounds or graphics, such as notifying personnel of critical stages of a medical procedure, the presence or proximity of sensitive tissues or organs, recommended actions, etc.

[0060] Also as described above, machine learning predictions can also be used as input to other systems or software, including computer-assisted medical systems 188. For example, the predictions can be provided to an augmented reality system (e.g., a virtual reality (VR) system or an augmented reality (AR) system), which can be implemented alone or by computer-assisted medical systems 188. VR systems can simulate three-dimensional scenes. Features such as objects can be enhanced. Subsurface or intra-tissue features can be displayed, which may be conditional, such as when the viewpoint is within a threshold distance of the feature. Features can be graphically labeled in a user interface. Predictions can be used for scene reconstruction, etc. AR systems can display such graphics during medical procedures to visualize the procedure in real time. The AR system or VR system can specify a three-dimensional zone from which instruments or objects can be excluded. The predicted synthetic image can provide image data displayed by the AR system or VR system (e.g., texture, color, or intensity to be mapped to scene geometry or surface).

[0061] As described above, predictions can be provided to the computer-assisted medical system 188 to improve its function during the medical procedure. For illustration, the prediction can be used to control the manipulation of one or more instruments by the computer-assisted medical system 188. For example, the prediction can provide information for instrument tracking, positioning or identification. In addition or alternatively, the prediction feature can form the basis of exclusion zones; the computer-assisted medical system 188 can automatically prohibit the instrument from contacting certain types of tissue or anatomical structures, or prohibit the instrument from moving to the area around the prediction feature. In addition or alternatively, the computer-assisted medical system 188 can display the prediction graphics as described above. For example, the predicted segmentation or label can be displayed. Various other operations can be based on the prediction (e.g., the presence of a specific type of anatomical structure), such as triggering video recording, user interface changes, rendering notifications, displaying information about the predicted surgical stage, control of the light source 110 or imaging device 112, and / or other peripheral events when a specific anatomical structure is identified based on the output of the machine learning model. In some embodiments, the tissue / structure model can be constructed from a local view (e.g., a blurred view of the biliary tree during dissection) to help the surgeon better track the anatomical structure.

[0062] As described above, in some examples, imaging device 112 and / or image processing system 126 may be associated with a computer-assisted medical system for performing medical procedures on a body (whether alive or not). Figure 7 An example of a computer-assisted medical system 700 that can be used to perform various types of medical procedures, including surgical and / or non-medical procedures, is shown. The imaging device 112 and the image processing system 126 can be part of or in addition to the computer-assisted medical system 188.

[0063] As shown, the computer-assisted medical system 188 may include a manipulator assembly 702 ( Figure 7 ), a manipulator cart 704, and an auxiliary device 706, all of which are communicatively coupled to each other. The medical team can use the computer-assisted medical system 188 to perform computer-assisted medical procedures or other similar operations on the body of the patient 708 or any other subject that can serve a specific embodiment. As shown, the medical team can include a first user 710-1 (e.g., a surgeon for a medical procedure), a second user 710-2 (e.g., a patient-side assistant), a third user 710-3 (e.g., another assistant, a nurse, an intern, etc.), and a fourth user 710-4 (e.g., an anesthesiologist for a medical procedure), all of which can be collectively referred to as users 710, and each user can control the computer-assisted medical system 188, interact with the computer-assisted medical system 188, or otherwise become a user of the computer-assisted medical system 188. During the medical procedure, there may be more, fewer, or alternative users, which can serve a specific embodiment. For example, the team composition for different medical procedures or non-medical procedures can be different and include users with different roles.

[0064] Although Figure 7 A minimally invasive medical procedure (e.g., a minimally invasive medical procedure) is shown being performed, but it should be understood that the computer-assisted medical system 188 can be similarly used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations can also be performed.

[0065] like Figure 7As shown, manipulator assembly 702 can include one or more manipulator arms 712 (e.g., manipulator arms 712-1 to 712-4) to which one or more instruments can be coupled. These instruments can be used for computer-assisted medical procedures on patient 708 (e.g., in the surgical example, by being at least partially inserted into patient 708 and manipulated within patient 708). Although manipulator assembly 702 is depicted and described herein as including four manipulator arms 712, manipulator assembly 702 can include a single manipulator arm 712 or any other number of manipulator arms. Although Figure 7 The example of the manipulator arm 712 is shown as a robotic manipulator arm, but it should be understood that in some examples, one or more instruments can be partially or fully manually controlled, such as by being held and manually controlled by a person. For example, these partially or fully manually controlled instruments can be coupled to Figure 7 The computer-assisted instrumentation of the manipulator arm 712 is shown for use in conjunction with, or as an alternative to, it.

[0066] During a medical procedure, the user control device 704 can be configured to facilitate remote operation control of the manipulator arm 712 and instruments attached to the manipulator arm 712 by the user 710-1. To this end, the user control device 704 can provide the user 710-1 with an image of the operating area associated with the patient 708 captured by the imaging device. To facilitate control of the instruments, the user control device 704 can include a set of master controls. These master controls can be manipulated by the user 710-1 to control the movement of the manipulator arm 712 or any instrument coupled to the manipulator arm 712.

[0067] The auxiliary device 706 may include one or more computing devices configured to perform auxiliary functions to support the medical procedure, such as providing insufflation, electrocautery energy, irradiation or other energy for the coordinating components of the imaging device, image processing, or computer-assisted medical system 188. In some examples, the auxiliary device 706 may be configured with a display monitor 714 configured to display one or more user interfaces or graphical or textual information to support the medical procedure. In some cases, the display monitor 714 may be implemented by a touch screen display and provide user input functions. The enhanced content provided by the zone-based augmentation system may be similar to or different from the content associated with the display monitor 714 or one or more display devices (not shown) in the operating area.

[0068] Manipulator assembly 702, user control device 704, and auxiliary device 706 may be communicatively coupled to one another in any suitable manner. Figure 7As shown, the manipulator assembly 702, the user control device 704, and the auxiliary device 706 can be communicatively coupled via a control line 716, which can represent any wired or wireless communication link that can serve a particular embodiment. To this end, the manipulator assembly 702, the user control device 704, and the auxiliary device 706 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0069] In some embodiments, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. In general, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0070] Computer-readable media (also referred to as processor-readable media) include any non-transient media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take a variety of forms, including, but not limited to, non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memories. Volatile media can include, for example, dynamic random access memory ("DRAM"), which typically constitutes main memory. Common forms of computer-readable media include, for example, disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital video disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM, any other memory chip or cartridge memory, or any other tangible medium from which a computer can read.

[0071] Figure 8 An exemplary computing device 800 is shown, which may be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 800 .

[0072] like Figure 8 As shown, computing device 800 may include a communication interface 802, a processor 804, a storage device 806, and an input / output ("I / O") module 808 that are communicatively coupled to one another via a communication infrastructure 810. Figure 8 An illustrative computing device 800 is shown in FIG. Figure 8 The components shown in are not intended to be limiting. Additional or alternative components may be used in other embodiments. The computing device 800 may be a virtual machine or may include virtualized components. Figure 8 Components of computing device 800 are shown in .

[0073] The communication interface 802 may be configured to communicate with one or more computing devices. Examples of the communication interface 802 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (such as a wireless network interface card, etc.), a modem, an audio / video connection, and any other suitable interface.

[0074] The processor 804 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 804 may perform operations by executing computer-executable instructions 812 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 806.

[0075] The storage device 806 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, the storage device 806 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in the storage device 806. For example, data representing computer executable instructions 812 configured to direct the processor 804 to perform any of the operations described herein may be stored within the storage device 806. In some examples, the data may be housed in one or more databases within the storage device 806.

[0076] The I / O module 808 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 808 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 808 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0077] I / O module 808 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, I / O module 808 is configured to provide graphics data to the display to be presented to the user. Graphics data may represent one or more graphical user interfaces and / or may serve any other graphics content of a particular embodiment.

[0078] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, certain features of one embodiment described herein may be combined or replaced with features of another embodiment described herein. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory for storing instructions; and One or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second sequence of images captured by the imaging device during the medical procedure, the second sequence of images comprising a second image based on illumination of the scene using non-visible spectrum light; and The first image sequence and the second image sequence are processed using a machine learning module. The system of claim 1 , wherein the second image is based on sensing of infrared light.

3. The system of claim 2, wherein the infrared light comprises light emitted by an illuminated fluorophore.

4. The system of claim 1, wherein the machine learning module comprises a machine learning algorithm, and wherein the processing comprises training a machine learning model based on the first image sequence and the second image sequence by the machine learning algorithm.

5. The system of claim 1, wherein the machine learning module comprises a trained machine learning model, and wherein the processing comprises generating predictions based on the first image sequence and the second image sequence by the trained machine learning model.

6. The system of claim 5, wherein the prediction comprises one or more of: a predicted image, a predicted label indicating a feature in one or more of the first image sequence or the second image sequence, an image segmentation, a predicted stage of a medical procedure, or a predicted geometry corresponding to the scene.

7. The system of claim 5, the process further comprising providing the prediction to a computer-assisted medical system that performs an operation based on the prediction.

8. The system of claim 1, wherein the processing comprises generating a label for the first image sequence based on the second image sequence.

9. The system of claim 8, wherein the processing further comprises training a machine learning model based on the first image sequence and the labels.

10. A system comprising: a memory for storing instructions; and One or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Based on the output of the machine learning model, an operation is performed on the first image sequence.

11. The system of claim 10, the process further comprising generating a prediction for use with a computer-assisted medical system based on the output of the machine learning model.

12. The system of claim 11, the process further comprising displaying a user interface via a display of the computer-assisted medical system based on the prediction.

13. The system of claim 11, the process further comprising controlling movement of a component of the computer-assisted medical system based on the prediction.

14. The system of claim 10, wherein the output comprises a modified version of an image in the first image, and wherein the operation comprises displaying the modified version of the image.

15. The system of claim 14, wherein the modified version of the image comprises a segmentation of the image.

16. The system of claim 10, wherein the operations include one or more of: segmenting an image in the first image, labeling the image, classifying the image, reconstructing geometry or metrics of the scene, or identifying features depicted in the image.

17. The system of claim 16, wherein the output comprises a label associated with the image, and wherein the label comprises an indication of at least one of a type of tissue, an identification of an organ, or an indication of a type of object.

18. The system of claim 10, wherein the operations include one or more of marking anatomical features in an image in the first image or enhancing anatomical features in the image.

19. The system of claim 10, wherein the operation comprises determining a category associated with the first image.

20. The system of claim 19, wherein the category comprises a stage of the medical procedure.

21. The system of claim 10, wherein the process further comprises: A third image sequence is provided to the trained machine learning model, the third image sequence comprising a third image based on illumination of the scene using non-visible spectrum light, wherein an output of the trained learning model is also based on the third image sequence.

22. The system of claim 10, wherein the second image is based on illumination of the scene using non-visible spectrum light.

23. The system of claim 10, wherein the second image is based on illumination of a scene associated with a different medical procedure using non-visible spectrum light.

24. The system of claim 10, wherein the second image comprises a visible light image having a label generated based on the invisible light image.

25. A system comprising: a memory for storing instructions; and One or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Generate predictions based on the output of the machine learning model.

26. The system of claim 25, wherein the prediction is provided to a computer-assisted medical system.

27. The system of claim 26, wherein the computer-assisted medical system performs an operation based on the prediction.

28. The system of claim 25, wherein the process further comprises performing an operation on a computer-assisted medical system based on the prediction.

29. A system according to claim 28, wherein performing the operation includes one or more of the following: displaying a graphical user interface through a display of the computer-assisted medical system, displaying an image included in the first image through the display of the computer-assisted medical system, or controlling the movement of components of the computer-assisted medical system.

30. The system of claim 25, wherein the prediction is provided to an augmented reality system configured to generate.

31. The system of claim 25, wherein the prediction comprises one or more of: an anatomical label, a segmentation of the first image, a marker of anatomical features within the tissue, or an enhanced first image.

32. The system of claim 25, wherein the trained machine learning model comprises a regression model, a decision tree model, a random forest model, a logistic regression model, a support vector machine model, a naive Bayes classifier model, a linear regression model, or a neural network model.

33. The system of claim 25, further comprising modifying or marking an image included in the first image based on the prediction.

34. The system of claim 33, wherein the modified or marked image is provided to a computer-assisted medical system.

35. The system of claim 25, wherein the second image is based on illumination of the scene using non-visible spectrum light.

36. The system of claim 25, wherein the second image is based on illumination of a scene associated with a different medical procedure using non-visible spectrum light.

37. The system of claim 25, wherein the second image comprises a visible light image having a label generated based on the invisible light image.

38. A method performed by one or more computing devices, the method comprising: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second sequence of images captured by the imaging device during the medical procedure, the second sequence of images comprising a second image based on illumination of the scene using non-visible spectrum light; and The first image sequence and the second image sequence are processed using a machine learning module.

39. A method performed by one or more computing devices, the method comprising: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Based on the output of the machine learning model, an operation is performed on the first image sequence.

40. A method performed by one or more computing devices, the method comprising: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Generate predictions based on the output of the machine learning model.

41. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; accessing a second sequence of images captured by the imaging device during the medical procedure, the second sequence of images comprising a second image based on illumination of the scene using non-visible spectrum light; and The first image sequence and the second image sequence are processed using a machine learning module.

42. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Based on the output of the machine learning model, an operation is performed on the first image sequence.

43. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to: accessing a first sequence of images captured by an imaging device during a medical procedure, the first sequence of images comprising a first image based on illumination of a scene associated with the medical procedure using visible spectrum light; providing the first image to a trained machine learning model, wherein the trained machine learning model has been trained using a second image sequence including a second image; and Generate predictions based on the output of the machine learning model.

Images