Invisible spectral light image-based operations for visible spectral images

Abstract

An illustrative system may access a first sequence of images including a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second sequence of images comprising a second image, the second image based on illumination of the scene using the invisible spectral light; detecting an object in the second image sequence; and performing an operation on the first image sequence based on the detected object.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 352,839, filed on June 16, 2022, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Light-based image data captured during a medical procedure has many uses during and after such procedure. For example, medical image data from an endoscope can be displayed during a medical procedure to assist medical personnel in performing the procedure. As another example, medical image data captured during a medical procedure can be used as control signals for a computer-assisted medical system. As another example, medical image data captured during a medical procedure can also be used after the medical procedure for postoperative evaluation, diagnosis, guidance, etc.

[0004] Various illumination and image sensing techniques have been used to capture images of medical procedures. Visible spectrum light emitting sources and image sensors have been used to capture color (white light) images of medical procedures. Non-visible spectrum image sensors, sometimes paired with non-visible spectrum light emitting sources, have been used to capture non-visible spectrum images of medical procedures. Summary of the invention

[0005] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements / elements of all aspects, nor to outline the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed embodiments presented below.

[0006] An illustrative system includes: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first image sequence including a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second image sequence including a second image based on illumination of the scene using non-visible spectrum light; detecting an object in the second image sequence; and performing an operation on the first image sequence based on the detected object.

[0007] An illustrative method performed by one or more computing devices may include: accessing a first image sequence including a first image, the first image being based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second image sequence including a second image, the second image being based on illumination of the scene using non-visible spectrum light; detecting an object in the second image sequence; and performing an operation on the first image sequence based on the detected object.

[0008] An illustrative non-transitory computer-readable medium may store instructions that, when executed, direct a processor of a computing device to perform a process comprising: accessing a first image sequence comprising a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second image sequence comprising a second image based on illumination of a surgical scene using non-visible spectrum light; detecting an object in the second image sequence; and performing an operation on the first image sequence based on the detected object. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0010] Figure 1 A system for capturing an image of a scene associated with a medical procedure is shown.

[0011] Figure 2 An image processing system is shown that generates an output that can be used to perform an operation on a first sequence of images.

[0012] Figure 3 Embodiments are shown in which more than two image sequences are captured and used.

[0013] Figure 4 A technique for capturing a scene in a first image sequence and a second image sequence is shown.

[0014] Figure 5 Techniques for generating a composite second image are shown.

[0015] Figure 6 Embodiments are shown for labeling a first sequence of images based on detecting and / or identifying objects in a second sequence of images.

[0016] Figure 7 An embodiment for segmenting a first sequence of images is shown.

[0017] Figure 8 An embodiment for modifying a first sequence of images is shown.

[0018] Fig. 9 Embodiments for generating and using three-dimensional (3D) models are shown.

[0019] Fig.10 A computer-assisted medical system is shown.

[0020] Fig.11 An illustrative computing device is shown. DETAILED DESCRIPTION

[0021] Imaging of scenes associated with medical procedures has many applications. For example, the display of an image of an obscured scene associated with a medical procedure can allow a surgeon performing the medical procedure to see and evaluate the obscured scene, or can allow the surgeon to see displayed information that is not available from direct observation. Different types of images have been used to capture and depict scenes associated with medical procedures. For example, visible light images have been used, and invisible light images have also been used. Previously, these different types of images were used independently. For example, during a surgical procedure, different image types can be displayed alternately, one then the other. It may not be possible to utilize information from two or more different types of images at the same time. For example, although two types of images of a scene may be available, only the information in one of the images may be used or displayed at any given time. Information that can be used to improve or enhance the other image may not be used.

[0022] Techniques for using non-visible spectrum images to implement operations performed on visible spectrum images are described herein. Given a first sequence of scene images (e.g., visible spectrum images) and a second sequence of scene images (e.g., non-visible spectrum images), object detection and / or recognition can be performed on the second sequence, and the output of the object detection and / or recognition can be used to perform operations on the first image sequence. For example, based on the object detection performed on the second sequence, a label can be applied to the first image sequence, which can assist a surgeon or other user in performing a surgical procedure while viewing the first image sequence. This article describes in detail this advantage and other advantages and benefits of using non-visible spectrum images to implement operations performed on visible spectrum images.

[0023] As used herein, "visible spectrum images" and "visible spectrum videos" refer to images and videos whose pixel values ​​represent the sensed intensity of visible spectrum light. "Invisible spectrum images" and "invisible spectrum videos" refer to images and videos whose pixel values ​​represent the sensed intensity of invisible spectrum light. For the sake of brevity, this article will use "image" to refer to both images and videos. Illustrative invisible spectrum images include fluorescence images, hyperspectral images, and other types of images that do not rely entirely on visible spectrum illumination. For example, a fluorescence image is an image of fluorescent light emitted from a substance when the substance is illuminated by an invisible spectrum light source. Infrared images are another type of invisible spectrum image. Infrared images are images captured by a sensor capable of sensing infrared band light. For example, infrared light can include light emitted by an illuminated fluorophore.

[0024] As used herein, "label" means any type of data that indicates an object or other feature represented in an image, including but not limited to graphical or text-based annotations, identifiers, highlights, enhancements, and overlays. Labels applied to images may be embedded in the image file as metadata or stored in a separate data structure linked to the image file. Labels may be presented to a user, for example, as an enhancement to the image, or may be used for other purposes that do not necessarily involve presentation, such as the training of a machine learning model.

[0025] As used herein, a "medical procedure" may refer to any procedure performed on a patient using manual and / or instrumental techniques to investigate, diagnose, or treat a patient's physical condition. In addition, a medical procedure may refer to any non-clinical procedure, e.g., a procedure not performed on a live patient, such as a calibration or testing procedure, a training procedure, and an experimental or research procedure.

[0026] Figure 1 A system 100 is shown implementing a method for capturing an image of a scene 102 associated with a medical procedure. The scene 102 may include a surgical field associated with a subject / body (e.g., a body of a living animal, a human or animal cadaver, a portion of a human or animal anatomy, tissue removed from a human or animal anatomy, a non-tissue artifact, a physical training model, etc.) on or in which a medical procedure is being performed. For example, the scene 102 may include various types of tissue (e.g., tissue 104), organs (e.g., organ 106), and / or non-tissue objects (e.g., object 108), such as instruments, objects held or manipulated by instruments, etc.

[0027] One or more light sources 110 can illuminate scene 102. As described above, light sources 110 can include any combination of white light sources, narrow-band light sources (whether in the visible spectrum or not, such as ultraviolet lamps), lasers, infrared light emitting diodes (LEDs), etc. If fluorescence is to be captured, the type of light source may depend on the fluorescent agent or protein used during the medical procedure. In some embodiments, the light source can provide light in the visible spectrum, but the fluorescence it induces may not be in the visible spectrum.

[0028] Further with respect to fluorescence, in some embodiments of system 100, the light source (i.e., the excitation light source) used for fluorescence illumination can have any wavelength outside the visible spectrum. For example, a fluorescent light source, such as indocyanine green (ICG), can generate light with a wavelength in the infrared radiation region (e.g., about 700nm to 1mm), such as the near infrared ("NIR") radiation region (e.g., about 700nm-950nm), the short wave infrared ("SWIR") radiation region (e.g., about 1400nm-3000nm), or the long wave infrared ("LWIR") radiation region (e.g., about 8000nm-1500nm). In some examples, the fluorescent light source can generate light with a wavelength of about 1000nm or greater (e.g., SWIR and LWIR). Additionally or alternatively, the fluorescent light source can output light with a wavelength of about 350nm or shorter (e.g., ultraviolet radiation). In some embodiments, the fluorescent light source can be specifically configured for optical coherence tomography imaging.

[0029] System 100 also includes imaging device 112. Imaging device 112 receives light reflected, emitted, and / or fluoresced from a subject of a medical procedure and converts the received light into image data. Imaging device 112 senses light from scene 102 and outputs a first image sequence 114 and a second image sequence 116 of scene 102. First image sequence 114 may be a sequence of visible spectrum images 118 of light sensed in the visible spectrum. Second image sequence 116 may be a sequence of non-visible spectrum images 120 of light sensed in the non-visible spectrum. The image sequence may be in the form of a single image, an encoded video stream, etc. As shown in FIG. Figure 1 As shown, since the image sequences are from different spectra (or partially non-overlapping spectra), the contents of the individual image sequences may be different; some features of the scene may be represented by one sequence but not by another.

[0030] The imaging device 112 may have a first image capture device 122 and a second image capture device 124. Either image capture device may be any type of device capable of converting photons into electrical signals, such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, a photomultiplier, etc. Regardless of the type of image capture device used, the imaging device 112 may be configured to sense light within and outside the visible spectrum, as described above. In some embodiments, the first image capture device 122 senses light in the visible spectrum, while the second image capture device 124 senses light in the invisible spectrum. The first image capture device 122 and the second image capture device 124 may be separate sensors within a single camera, or they may be separate sensors in separate individual cameras. In some embodiments, the imaging device 112 may include only one image capture device (e.g., one sensor), and the image capture device may be capable of sensing in the visible spectrum and in one or more invisible spectrums at the same time. For example, some image sensors may be capable of sensing in the visible spectrum and in the infrared spectrum at the same time. In other embodiments, the imaging device 112 may be a stereo camera and may have two cameras, each capable of sensing in the visible spectrum and in the invisible spectrum. In some embodiments, the imaging device 112 and the light source 110 may be part of (or optically connected to) an endoscope.

[0031] In one embodiment, the first image capture device 122 may continuously capture the first image sequence 114 as video data of the medical procedure, and the second image capture device 124 may intermittently capture images of the second image sequence 116. For example, the first image capture device 122 may capture a video frame (the first image 118) every 1 / 60 second, and the second image capture device 124 may capture the second image 120 once per second.

[0032] like Figure 1 As shown, the image processing system 126 may be configured to access (eg, receive) the first image sequence 114 and the second image sequence 116 to perform various operations on the first image sequence 114 , as described below.

[0033] The image processing system 126 may be implemented by one or more computing devices and / or computer resources (e.g., processors, memory devices, storage devices, etc.), which may serve a particular implementation. As shown, the image processing system 126 may include, but is not limited to, a memory 128 and a processor 130 selectively and communicatively coupled to each other. The memory 128 and the processor 130 may each include or be implemented by computer hardware configured to store and / or process computer software. Figure 1Various other components of computer hardware and / or software not explicitly shown in the drawings may also be included in image processing system 126. In some examples, memory 128 and processor 130 may be distributed among multiple devices and / or multiple locations, as may serve a particular implementation.

[0034] The memory 128 may store and / or otherwise maintain executable data used by the processor 130 to perform any of the functions described herein. For example, the memory 128 may store instructions 132 that may be executed by the processor 130. The memory 128 may be implemented by one or more memories or storage devices, including any memory or storage device described herein, which is configured to store data in a transient or non-transient manner. The instructions 132 may be executed by the processor 130 to cause the image processing system 126 to perform any of the functions described herein. The instructions 132 may be implemented by any suitable application, software, code, and / or other executable data instance. In addition, the memory 128 may also maintain any other data that is accessed, managed, used, and / or transmitted by the processor 130 in a particular embodiment.

[0035] The processor 130 may be implemented by one or more computer processing devices, including a general-purpose processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, etc.), a special-purpose processor (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), a digital signal processor, etc. Using the processor 130 (e.g., when the processor 130 is directed to perform operations represented by instructions 132 stored in the memory 128), the image processing system 126 may perform various operations described herein.

[0036] Various embodiments of the image processing system 126 will now be described with reference to the accompanying drawings, and how the image processing system 126 may be configured to implement the modules described below. The various modules described herein may be included in the image processing system 126 and may be implemented by any suitable combination of hardware and / or software. Thus, these modules represent various functions that may be performed by the image processing system 126 alone or in combination with any other functions described herein that are performed by the image processing system 126 and / or its components.

[0037] Figure 2Image processing system 126 is shown generating output 152, which may be used to perform operation 154 on first sequence of images 114. Image processing system 126 receives second sequence of images 116. Image processing system 126 may be configured to perform various image processing algorithms on second sequence of images 116. For example, image processing system 126 may implement any known object detection and / or recognition algorithm to detect and / or recognize objects in second sequence of images 116.

[0038] In some embodiments, the output 152 may include an indication of objects detected or identified in the second sequence of images 116. For example, the output 152 may be an area or blob corresponding to a continuous area of ​​pixels having values ​​within a given color range, a detected perimeter (e.g., a set of detected edges), an object detected based on inter-frame motion detection, a segment derived from a segmentation algorithm implemented by the image processing system 126 (e.g., foreground background segmentation), an area isolated by a time-varying signal (e.g., "pulsing" pixel values), etc. In some embodiments, the output 152 may include an indication of objects identified in the second sequence of images 116. For example, the output 152 may be a label or identifier of the type or category of the identified object, and may include information linking the label to its respective graphical portion in the corresponding first image (i.e., the label may be linked to a location in the first image). In some embodiments, the label or identifier may be associated with individual images, but not with a specific object therein.

[0039] Outputs 152 from the image processing system 126 may be provided to a module 156 that performs operations 154 on the first image sequence 114 based on the outputs 152. The operations 154 may include, for example, marking one or more of the first images 118, enhancing one or more of the first images 118 (e.g., by brightening, highlighting, adding text thereto, etc.), marking one or more objects in the first images 118 that correspond to a marked object identified in the second image sequence 116, and so on. The operations 154 may include, for example, segmenting the one or more first images 118 based on the outputs 152. The outputs 152 may be segments that may be directly applied or mapped to the first images 118, or they may be detected objects that provide additional input for finding segments in the first images 118 using an enhanced segmentation algorithm. In other embodiments, the outputs 152 are used to modify or transform the image content of one or more of the first images 118. For example, a region in the first image 118 may be enhanced (e.g., brightened) based on the corresponding outputs 152. Objects detected and / or identified in the second image sequence 116 may be superimposed or blended into the first image 118. A graphical text label may be added to the first image 118. These operations 154 and others are described below.

[0040] Figure 3 An embodiment is shown in which more than two image sequences are captured and used to perform various operations described herein. For example, the first image sequence 114 can be a visible spectrum image, the second image sequence 116 can be a fluorescence image, and the third image sequence 116A can correspond to another imaging modality, such as hyperspectral imaging. In such an embodiment, the second image sequence 116 and / or the third image sequence 116A can be captured in the background (e.g., while the first image sequence 114 is captured) or by using a sampling technique described later. The sampling of the second image sequence 116 and the third image sequence 116A can be at different rates or at the same rate but offset. In addition, the second image sequence 116 and the third image sequence 116A can be processed together or independently to perform an operation on the first image sequence 114. For example, when processed independently, the second image sequence 116 can be processed to perform a first operation 154 (e.g., detect a first type of anatomical feature), and the third image sequence 116A can be processed to perform a second operation 154A (e.g., detect a second type of anatomical feature). When processed together, the second image sequence 116 and the third image sequence 116A may both contribute to the output 152 provided to the first operation 154, for example, to improve object detection.

[0041] Figure 4 Techniques for capturing scene 102 in a first image sequence 114 and a second image sequence 116 are shown. As described above, it is not uncommon to use two imaging techniques or modes during a surgical procedure. Some imaging techniques may be difficult to use simultaneously. For example, if one imaging mode requires a particular type of illumination light, that type of light may be interfered with (or interfere with) another imaging mode. In some imaging systems, using two imaging modes simultaneously may present problems such as heat buildup due to increased illumination energy, increased data, and the like. In addition, in some imaging systems, there may be other factors that may limit the ability to capture images or videos simultaneously in two different modes. For example, some medical imaging systems (e.g., endoscopes) may have two capture modes and two corresponding display modes, and these modes may operate in a mutually exclusive manner such that image or video data of a mode can only be captured when the mode is actively displayed. In addition, some imaging modes may require a longer sensing phase (e.g., 1 second) and may not be able to sense new images at a faster rate (e.g., 1 per second).

[0042] Figure 4Techniques are shown for capturing a first image sequence 114 and a second image sequence 116 in a manner that can avoid some of the challenges associated with capturing simultaneously in two different types of imaging modes. As described above, a first image capture device 122 captures a first image sequence 114. The first image capture device 122 can perform a first operation 170 to capture a first image 118 once every N milliseconds. N can be small enough that the first image sequence 114 is rendered as a video. For example, the first image capture device 122 can capture the first image 118 once every 16.7 milliseconds (i.e., every 1 / 60 second). The high frame / image capture rate of the first image capture device 122 can be sufficient to render the first image sequence 114 as a video on a display, i.e., a real-time view of the scene 102 can be displayed. To illustrate, in Figure 4 In the example shown, the first image capture device 122 captures the first image 118 every 100 milliseconds.

[0043] The second image capture device 124 performs a second operation 172 of capturing a second image 120 every M milliseconds, where M is less than N. In some embodiments, N is significantly greater than M. For example, M may be twice N. That is, the second image capture device 124 may capture images at half the rate of the first image capture device 122. In one embodiment, N may be 16.7 milliseconds, and M may be 1000 milliseconds, i.e., one second image 120 is captured for every 60 first images 118 captured. Figure 4 In the example shown, M is 500 milliseconds, so the first image capture device 122 captures once every 100 milliseconds, and the second image capture device 124 captures once every 500 milliseconds; one second image 120 is captured for every five first images 118. The intermittent capture of the second image capture device 124 may be accompanied by coincident intermittent flashes of the illumination source. For example, if the second image capture device 124 is to capture fluorescence, the corresponding light source may be flashed for a sufficient duration to allow the second image capture device 124 to capture a single second image 120. By this technique, any effects of the fluorescent light source may be reduced, such as heat generation, interference with the first image capture device 122, etc. In some examples, the second image may be captured intermittently during the capture of the first image, so that some first images are captured during a period of time when no second image is captured.

[0044] In some embodiments, a single image sensor is used and only a single image (out of all captured image sequences) is captured at any given time. The rate at which the second image sequence 116 is captured may be selected based on a rate that is imperceptible to the human eye (i.e., human vision). For example, in such an embodiment, the first image sequence 114 may lose 1 frame every 1 / 60 second as the sensor is used to capture images in the second sequence 116. This approach may be imperceptible to humans when displayed at 60 frames per second. The sampling rates discussed above are examples; sampling rates that may be imperceptible may depend on the display rate as well as other factors such as lighting conditions, scene characteristics, whether the image capture device is moving relative to the scene, etc.

[0045] Figure 5 A technique for generating a synthesized second image 120A is shown. As described above, the first image sequence 114 may have a higher frame rate than the second image sequence 116. Therefore, there may be first images 118 that do not have corresponding (in time) second images 120. It may be helpful to "fill in" gaps in the second image sequence 116 that do not have second images 120 that correspond (in time) to the first images 118. As described below, some operations 154 with respect to the first image sequence 114 may be more feasible or more accurate if each first image 118 has a corresponding second image 120.

[0046] One method of filling the second image sequence 116 (matching the first image sequence 114 image by image) is to duplicate the second image 120. That is, for any given first image 118 that does not have a correspondingly captured second image 120, the closest (in time) captured second image 120. In other words, the second image sequence 116 may be filled to match the first image sequence 114 by filling the gaps in the first image sequence 114 with a copy of the closest (in time) second image 120. While this approach is effective, if a first image 118 is captured at a given time and its corresponding second image 120 is captured at a significantly different time (e.g., a few or many frames away), the pair of images may differ in a manner that makes subsequent image processing less accurate. For example, if the first image capture device 122 and the second image capture device 124 are part of the same subject (e.g., a camera or endoscope), and the subject is moving while the first image 118 and the second image 120 are captured, features in the duplicated second image 120 may not align with features in the first image 118. Additionally, light and surgical scene conditions may change during the time between capturing the various images.Techniques may be used to generate a composite second image 120A that may approximate the second image taken by second image capture device 124 when first image 118 was captured.

[0047] One such technique is to generate a corresponding transformation for any given first image 118 that does not have a time corresponding second image 120. The transformation is applied to the time corresponding second image 120 to generate a synthesized second image 120A. The synthesized second image 120A can approximate the image that would be captured when the second image capture device 124 captures an image when the given first image 118 is captured. The transformation can be a geometric transformation. For example, a geometric transformation can be generated based on information about the pose of the body or housing (e.g., camera) containing the first image capture device 122 and the second image capture device 124. The pose can be tracked directly by a separate system. Alternatively, the pose can be estimated by building a map of the scene from the captured images. Other techniques can be used. For example, the transformation can be a linear interpolation between the two second images 120. Color transformations can also be generated using interpolation. Regardless of how the synthesized second image 120A is generated, the result is that for each originally captured second image 120, there will be multiple corresponding synthesized second images 120A.

[0048] refer to Figure 5 In the illustrated example, for each time (t1, ..., t10) shown, a transformation is obtained 180 for the corresponding first image 118 and applied 182 to the corresponding second image 120 to generate a composite second image 120A for that time. For times where the first image 118 has a temporally corresponding second image 120 (e.g., times t3 and t8), the transformation may be an identity operation, or the transformation may be omitted and the temporally corresponding second image 120 may be copied (or used). The resulting composite second image sequence 116A may be well suited for use with the operations 154 described herein. As used herein, "second image sequence" refers to both composite and non-composite variants unless the context indicates otherwise.

[0049] While generating the synthesized second image 120A is a useful technique, in other embodiments, object recognition / detection may be performed on the original second image 120 rather than the synthesized image sequence. The detected / recognized objects may be propagated to a subsequence of the first images 118. For example, referring to Figure 5 , an object detected / identified in the second image 120 at time t3 can be propagated to the first image 118 at times t1 to t5. This technique can be used for any of the embodiments described herein involving performing operations on the first image sequence 114 based on objects found in the second image sequence 116.

[0050] Figure 6An embodiment for marking one or more first images 118 of a first image sequence 114 based on detecting and / or identifying an object 200 in one or more second images 120 in a second image sequence 114 is shown. In this embodiment, the second image sequence 116 passes through an object detection / recognition module 202. The object detection / recognition module 202 may implement any algorithm for detecting an object in an image and / or identifying an object in an image. A temporal algorithm using inter-frame analysis may be used to detect and / or identify the object 200. An indication of the detected / recognized object 200 is passed to an image marking module 204 (described below). In the case of object detection, the indication may include information about the extent and location of the object detected in the second image 120. In some embodiments, the indication may include a bit mask representing the detected object. In the case of object recognition, the indication may also (or alternatively) include information about the type or category of the object recognized in the second image 120. In summary, the object detection / recognition module 202 may output an indication of the detected / recognized object 200 to the image marking module 204.

[0051] In some embodiments, due to the difference in imaging modalities of the first image sequence 114 and the second image sequence 116, the second image sequence 116 may be input into the object detection / recognition module 202 instead of the first image sequence 114. In such embodiments, performing object detection / recognition using the second image sequence 116 may produce more accurate labels and / or require fewer computational resources than performing similar object detection / recognition using the first image sequence 114. For example, consider an embodiment in which the first image sequence 114 includes visible light images and the second image sequence 116 includes fluorescent images. In such an embodiment, fluorescent signal regions in images in the second image sequence 116 may correspond to certain objects, such as perfused organs. Therefore, the fluorescent signal regions may be used to detect and / or identify organs in the images without relying on more computationally expensive operations that may be required to detect and / or identify the same organs in the corresponding visible light images.

[0052] The image marking module 204 can use the indications of the detected / recognized objects 200 to perform operations on the first image sequence 114. The image marking module 204 also receives the first image sequence 114 and synchronizes the indications with the respective first images 118. In some embodiments, the synchronization can be inherent, for example, the image marking module 204 can be driven by a common timer or clock (as part of the image processing pipeline). In other embodiments, the synchronization can be based on a timestamp included in the indication; the timestamp can be used to align the corresponding detected / recognized object 200 with the respective first image 118.

[0053] The image tagging module 204 can tag the first image 118 with an indication. If the indication indicates an object or category that is recognized, the first image 118 can be tagged accordingly. The tag 206 can be added to the metadata of the first image 118, or can be saved in a separate data structure linked to the first image 118. If the indication includes the location (and / or shape) of the object, the location can be included in the metadata or data structure. In summary, the objects detected / recognized in the second image sequence 116 can be synchronously associated with the first image sequence 114. The resulting tagged image sequence 208 can be used alone or in combination with the second image sequence 116 for other purposes. For example, one or more tagged images in the tagged image sequence 208 can be used alone for supervised machine learning. The computer-assisted medical system can also use the tags 206 to make decisions, control movement, render sound or graphics, generate models of the scene 102, etc.

[0054] Figure 7 An embodiment for segmenting the first image sequence 114 is shown. The feature extraction module 220 receives the second image sequence 116 and executes any known feature extraction algorithm to extract features 222 from the second image sequence 116. In one embodiment, the feature extraction algorithm may be the object detection / recognition module 202, and the features 222 are detected / recognized objects. In other embodiments, the feature extraction algorithm may extract other types of features from the second image sequence 116. For example, the feature detection algorithm may extract features 222, such as one or more of the following: edges, segments (e.g., using background foreground segmentation), color histograms (or other statistics about the second image 120), gradient maps, patches, key points or points of interest, or other known image features.

[0055] The features 222 are passed to an image segmentation module 224, which also receives the first image sequence 114. As with the object detection / recognition embodiments described above, indications of the extracted features 222 may be synchronized with the first image 118. In one embodiment, the image segmentation module may apply segments from the second image sequence 116 directly, i.e., segment the first image 118 in the same manner as segmenting their respective counterpart second images 120. In another embodiment, the features 222 may provide additional information for segmenting the first image 118. In another embodiment, the segmentation module 224 may segment the first image 118 and then concatenate the segments with segments from the second image 120. The segmentation module 224 outputs a segmented image sequence 226 that may be used for further image processing, as input to a computer-assisted medical system, or the like.

[0056] Figure 8 An embodiment for modifying the first image sequence 114 is shown. Figure 6As discussed, object detection / recognition may be performed on the second image sequence 116. Indications thereof may be provided to the image editing module 240, as may the first image sequence 114. Indications of detected / recognized objects may be synchronized with the second image sequence 116. The image editing module 240 may modify any given first image 118 based on any detected / recognized objects that are temporally associated with that given first image 118. For example, the detected / recognized objects may be merged with the first image 118. In another embodiment, objects in the first image 118 may be associated with objects detected in the second image 120, which objects may be colored, highlighted, etc., based on the objects of the first image 118. In another embodiment, objects recognized in the second image 120 may be associated with objects in the first image 118, such as by comparing location, shape similarity, etc. The objects in the first image 118 are then modified based on the labels of the objects recognized in the second image 120. For example, if the object is recognized as a specific anatomical feature, such as an artery, a graphical "artery" label may be added to the corresponding first image 118. As another example, a shape in the first image 118 that has been associated with an object recognized in the second image 120 may be colored according to the label of the recognized object. As another example, a graphical element representing the recognized object may be added (superimposed) in the first image 118, but may be colored according to the color in the superimposed position of the first image 118 (before the recognized object is superimposed). The image editing module 240 may graphically alter the first image sequence 114 in any manner with the object recognized / detected in the second image sequence 116, thereby generating a modified first image sequence 242.

[0057] Fig. 9 An embodiment for reconstructing at least a portion of a scene as a 3D model 260 and using the 3D model 260 is shown. Although this embodiment may be useful when implemented by a computer-assisted medical system 262, any image processing system 126 may be used. As described above, the object detection / recognition module 202 detects and / or recognizes objects in the second image sequence 116 and provides an output 264 regarding the detected / recognized objects. The 3D model generation module 266 receives the output 264 and may also receive the first image sequence 114 and / or the second image sequence 116. The 3D model generation module 266 may generate the 3D model 260 from the image sequence using a known model generation algorithm, possibly utilizing information about the pose of the image capture device when capturing the images in the image sequence.

[0058] In one embodiment, a 3D model may be generated from each respective image sequence, and the models may be merged to form the 3D model 260. In another embodiment, the 3D model data from the second image sequence 116 may be used to generate a 3D model of the internal (subsurface) structure of a tissue, organ, or other anatomical feature that may not be visible in the first image sequence 114. Any of the 3D models described may be rendered and displayed by the computer-assisted medical system 262. In some examples, displaying the 3D model may include superimposing a perspective-aligned view of the 3D model on the first image sequence 114, or otherwise compositing the perspective-aligned view of the 3D model with the first image sequence 114 to produce an enhanced view of the scene. Any model may be supplemented with objects detected / identified in the second image sequence 116. For example, if an object is identified as an anatomical feature, the feature may be added to the model as linked metadata or a graphical tag (to be rendered). The detected / identified object may indicate which textures are applied to the 3D model, etc. In some embodiments, the movement of components of the computer-assisted medical system 262 may be controlled based on one or more of the 3D models. For example, the 3D model 260 may include a representation of a key anatomical feature (e.g., a nerve), and the computer-aided medical system 262 may prohibit its movable elements from contacting (or approaching) the key anatomical feature. This can be accomplished, for example, by using the generated 3D model to generate a no-fly zone at a location corresponding to the physical object represented by the 3D model. A movable element (e.g., the distal end of an instrument mounted to a manipulator arm) can be controlled to prevent entry into the no-fly zone. The model generation process can also be used to estimate anatomical dimensions based on the identified objects.

[0059] In some embodiments, object detection / recognition may involve techniques that are more specific to the type of content expected in the second image sequence 116. For example, the pixel values ​​in the second image 120 may indicate the type of molecule. Thus, pixels, regions, patches, objects, etc. may be identified as corresponding to the type of molecule and may be identified accordingly. Furthermore, identifiers of anatomical features identified in the second image sequence 116 may be used by other algorithms (e.g., algorithms for identifying the stage of a surgical procedure, algorithms for identifying the type of medical procedure, or algorithms for evaluating the accuracy of molecule identification).

[0060] As described above, in some examples, imaging device 112 and / or image processing system 126 may be associated with a computer-assisted medical system for performing medical procedures on a body (whether alive or not). Fig.10An example of a computer-assisted medical system 262 that can be used to perform various types of medical procedures (including surgical and / or non-medical procedures) is shown. The imaging device 112 and the image processing system 126 can be part of or supplement the computer-assisted medical system 262. The computer-assisted medical system 262 can utilize the image sequence as it is captured in real time during the surgical procedure. For example, by triggering the recording of a video (image sequence), updating a user interface, rendering (audio / video) notifications during the surgical procedure, etc.

[0061] As shown, the computer-assisted medical system 262 may include a manipulator assembly 1002 ( Fig.10 1008 ), a manipulator cart is shown in the figure), a user control device 1004 and an auxiliary device 1006, all of which are communicatively coupled to each other. The medical team can use the computer-assisted medical system 262 to perform computer-assisted medical procedures or other similar operations on the body of the patient 1008 or any other subject that can serve a specific embodiment. As shown, the medical team can include a first user 1010-1 (e.g., a surgeon for a medical procedure), a second user 1010-2 (e.g., a patient-side assistant), a third user 1010-3 (e.g., another assistant, a nurse, an intern, etc.) and a fourth user 1010-4 (e.g., an anesthesiologist for a medical procedure), all of which can be collectively referred to as users 1010, and each user can control the computer-assisted medical system 262, interact with the computer-assisted medical system 262, or otherwise become a user of the computer-assisted medical system 262. During the medical procedure, there may be more, fewer, or alternative users, which can serve a specific embodiment. For example, the team composition for different medical procedures or non-medical procedures can be different and include users with different roles.

[0062] Some embodiments may be implemented in the context of components of computer-assisted medical system 262. For example, image capture device 122 may be part of an endoscope mounted to one of manipulator arms 1012, or may be held by a user (e.g., user 1010-2). Any one or more of first image sequence 114, second image sequence 116, or results of operations 154 / 154A performed on first image sequence 114 may be displayed using a display in user control device 1004 and / or using display monitor 1014. Image processing system 126 may use an endoscope, auxiliary device 1006, control device 1004, or Fig.10 The invention may be implemented by a computer system at any one of the other computing systems not shown.

[0063] Although Fig.10A minimally invasive medical procedure (e.g., a minimally invasive medical procedure) is shown being performed, but it should be understood that the computer-assisted medical system 262 can be similarly used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations can also be performed.

[0064] like Fig.10 As shown, manipulator assembly 1002 can include one or more manipulator arms 1012 (e.g., manipulator arms 1012-1 to 1012-4) to which one or more instruments can be coupled. The instruments can be used for computer-assisted medical procedures on patient 1008 (e.g., in the surgical example, by being at least partially inserted into and manipulated within patient 1008). Although manipulator assembly 1002 is depicted and described herein as including four manipulator arms 1012, manipulator assembly 1002 can include a single manipulator arm 1012 or any other number of manipulator arms. Although Fig.10 The example of the manipulator arm 1012 is shown as a robotic manipulator arm, but it should be understood that in some examples, one or more instruments can be partially or fully manually controlled, such as by being held and manually controlled by a person. For example, these partially or fully manually controlled instruments can be coupled to Fig.10 The computer-assisted instrumentation of the manipulator arm 1012 is shown for use in conjunction with, or as an alternative to, it.

[0065] During a medical procedure, the user control device 1004 may be configured to facilitate remote operation control of the manipulator arm 1012 and instruments attached to the manipulator arm 1012 by the user 1010-1. To this end, the user control device 1004 may provide the user 1010-1 with an image of the operating area associated with the patient 1008 captured by the imaging device. To facilitate control of the instruments, the user control device 1004 may include a set of master controls. These master controls may be manipulated by the user 1010-1 to control the movement of the manipulator arm 1012 or any instrument coupled to the manipulator arm 1012.

[0066] The auxiliary device 1006 may include one or more computing devices configured to perform auxiliary functions to support the medical procedure, such as providing insufflation, electrocautery energy, irradiation or other energy for the coordinating components of the imaging device, image processing, or computer-assisted medical system 262. In some examples, the auxiliary device 1006 may be configured with a display monitor 1014 configured to display one or more user interfaces or graphical or textual information to support the medical procedure. In some cases, the display monitor 1014 may be implemented by a touch screen display and provide user input functions. The enhanced content provided by the zone-based augmentation system may be similar to or different from the content associated with the display monitor 1014 or one or more display devices (not shown) in the operating area.

[0067] Manipulator assembly 1002, user control device 1004, and auxiliary device 1006 may be communicatively coupled to one another in any suitable manner. Fig.10 As shown, the manipulator assembly 1002, the user control device 1004, and the auxiliary device 1006 can be communicatively coupled via a control line 1016, which can represent any wired or wireless communication link that can serve a particular embodiment. To this end, the manipulator assembly 1002, the user control device 1004, and the auxiliary device 1006 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0068] In some embodiments, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. In general, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0069] Computer-readable media (also referred to as processor-readable media) include any non-transient media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take a variety of forms, including, but not limited to, non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memories. Volatile media can include, for example, dynamic random access memory ("DRAM"), which typically constitutes main memory. Common forms of computer-readable media include, for example, disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital video disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM, any other memory chip or cartridge memory, or any other tangible medium from which a computer can read.

[0070] Fig.11 An exemplary computing device 1100 is shown, which may be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 1100 .

[0071] like Fig.11 As shown, computing device 1100 may include a communication interface 1102, a processor 1104, a storage device 1106, and an input / output ("I / O") module 1108 that are communicatively coupled to one another via a communication infrastructure 1110. Fig.11 An illustrative computing device 1100 is shown in FIG. Fig.11 The components shown in are not intended to be limiting. Additional or alternative components may be used in other embodiments. The computing device 1100 may be a virtual machine or may include virtualized components. Fig.11 Components of computing device 1100 shown in .

[0072] The communication interface 1102 may be configured to communicate with one or more computing devices. Examples of the communication interface 1102 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (such as a wireless network interface card, etc.), a modem, an audio / video connection, and any other suitable interface.

[0073] The processor 1104 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1104 may perform operations by executing computer-executable instructions 1112 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1106.

[0074] Storage device 1106 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1106 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 1106. For example, data representing computer executable instructions 1112 configured to direct processor 1104 to perform any of the operations described herein may be stored within storage device 1106. In some examples, the data may be housed in one or more databases within storage device 1106.

[0075] The I / O module 1108 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1108 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1108 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0076] I / O module 1108 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, I / O module 1108 is configured to provide graphics data to the display to present to the user. Graphics data may represent one or more graphical user interfaces and / or may serve any other graphics content of a particular embodiment.

[0077] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, certain features of one embodiment described herein may be combined or replaced with features of another embodiment described herein. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory for storing instructions; and One or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process including: accessing a first sequence of images including a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second sequence of images including a second image based on illumination of the scene using light in a non-visible spectrum; detecting an object in the second image sequence; and An operation is performed on the first sequence of images based on the detected object.

2. The system of claim 1, wherein the non-visible spectrum light comprises fluorescent light. The system of claim 1 , wherein the non-visible spectrum light comprises infrared light.

4. The system of claim 1, wherein the first images are captured at a first rate of images per second, wherein the second images are captured at a second rate of images per second, and wherein the second rate is less than the first rate.

5. The system of claim 1, wherein the process further comprises: A third image sequence including a third image is accessed, a second object in the third image sequence is detected, and the operation is performed based on the detected object and the detected second object.

6. A system according to claim 1, wherein the image capture device captures the first image sequence and the second image sequence when the image capture device and the scene move relative to each other, wherein the first image in the first image sequence is captured at a first time and the second image in the second image sequence is captured at a second time, and wherein the second image is geometrically transformed based on data indicating movement of the image capture device between the first time and the second time.

7. The system of claim 1, wherein the process further comprises identifying the object detected in the second image sequence, and wherein the operation comprises applying a label to the first image sequence based on the identification of the object.

8. The system of claim 1, wherein the operation comprises modifying pixels in a first image based on the object detected in the second sequence of images.

9. The system of claim 1, wherein the process further comprises: A third image sequence including a third image is accessed, a second object in the third image sequence is detected, and a second operation is performed on the first image sequence based on the detected second object.

10. The system of claim 1 , wherein a subsequence of first images in the first image sequence corresponds in chronological order to second images in the second image sequence, wherein the process further comprises identifying the object detected in the second image sequence, wherein the identified object comprises an identified anatomical feature, and wherein an indication of the identified anatomical feature is applied to the subsequence of the first images.

11. The system of claim 10, wherein the applying comprises marking or modifying a subsequence of the first image.

12. The system of claim 1, wherein light provided by the light source corresponding to capturing the second sequence of images is imperceptible to human vision.

13. The system of claim 1, wherein the second image is captured intermittently during capturing of the first image such that some first images are captured during a period of time when no second image is captured.

14. The system of claim 13, wherein the first image and the second image are captured by the same sensor, and wherein the first image and the second image are captured at mutually exclusive times.

15. The system of claim 1, wherein the process further comprises identifying the object detected in the second image sequence, and wherein a label corresponding to the identified object is displayed with the first image sequence.

16. The system of claim 1, wherein a segmentation of the first image is displayed based on the detected object.

17. The system of claim 1, wherein a computer-assisted medical system is controlled based on the detected object.

18. The system of claim 17, wherein the computer-assisted medical system includes a manipulator arm, and wherein the controlling includes inhibiting movement of the manipulator arm based on the detected object.

19. The system of claim 1, wherein the detected object corresponds to an anatomical feature of the scene that is internal to tissue in the scene, and wherein the process further comprises constructing a three-dimensional model of the anatomical feature.

20. The system of claim 1, wherein the process further comprises: identifying the object detected in the second sequence of images; Wherein performing the operation is also based on determining that the identified object corresponds to a particular type of anatomical feature.

21. The system of claim 20, wherein the operation comprises one or more of triggering video recording, updating a user interface, or rendering a notification.

22. The system of claim 1, wherein the process further comprises identifying the object as an anatomical feature, and wherein an identifier corresponding to the identified anatomical feature is provided to an algorithm that (i) identifies a segment of a surgical procedure, (ii) identifies a type of medical procedure, or (iii) evaluates a molecular identity.

23. The system of claim 1, wherein the process further comprises: identifying the object detected in the second sequence of images; and Anatomical dimensions are estimated based on the identified object.

24. The system of claim 1, wherein the first image sequence is displayed while the object is detected in the second image sequence.

25. The system of claim 1, wherein the process further comprises causing the first sequence of images to be displayed but not the second sequence of images to be displayed.

26. The system of claim 1, wherein the first sequence of images and the second sequence of images are captured by an imaging device that senses visible light and invisible light.

27. The system of claim 1, wherein the operation is performed on the first sequence of images during a surgical procedure.

28. A system according to claim 27, wherein accessing the first image sequence and the second image sequence includes receiving an image stream including the first image and the second image from an imaging device of a computer-assisted medical system used to perform the surgical procedure during the surgical procedure, and wherein the operation is performed in real time when the image stream is received.

29. The system of claim 1, wherein the process further comprises inputting the second image into a machine learning model.

30. A method performed by one or more computing devices, the method comprising: accessing a first sequence of images including a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second sequence of images including a second image based on illumination of the scene using light in a non-visible spectrum; detecting an object in the second image sequence; and An operation is performed on the first sequence of images based on the detected object.

31. The method of claim 30, wherein the non-visible spectrum light comprises fluorescent light.

32. The method of claim 30, wherein the non-visible spectrum light comprises infrared light.

33. The method of claim 30, wherein the first images are captured at a first rate of images per second, wherein the second images are captured at a second rate of images per second, and wherein the second rate is less than the first rate.

34. The method of claim 30, further comprising: A third image sequence including a third image is accessed, a second object in the third image sequence is detected, and the operation is performed based on the detected object and the detected second object.

35. A method according to claim 30, wherein the image capture device captures the first image sequence and the second image sequence when the image capture device and the scene move relative to each other, wherein a first image in the first image sequence is captured at a first time and a second image in the second image sequence is captured at a second time, and wherein the second image is geometrically transformed based on data indicating movement of the image capture device between the first time and the second time.

36. The method of claim 30, further comprising identifying the object detected in the second sequence of images, and wherein the operation comprises applying a label to the first sequence of images based on the identification of the object.

37. The method of claim 30, wherein the operation comprises modifying pixels in a first image based on the object detected in the second sequence of images.

38. The method of claim 30, further comprising: A third image sequence including a third image is accessed, a second object in the third image sequence is detected, and a second operation is performed on the first image sequence based on the detected second object.

39. A method according to claim 30, wherein a subsequence of first images in the first image sequence corresponds in chronological order to second images in the second image sequence, wherein the method further includes identifying the object detected in the second image sequence, wherein the identified object includes an identified anatomical feature, and wherein an indication of the identified anatomical feature is applied to the subsequence of the first image.

40. The method of claim 39, wherein the applying comprises marking or modifying a subsequence of the first image.

41. The method of claim 30, wherein light provided by the light source corresponding to capturing the second sequence of images is imperceptible to human vision.

42. The method of claim 30, wherein the second image is captured intermittently during capturing of the first image such that some first images are captured during a period of time when no second images are captured.

43. The method of claim 42, wherein the first image and the second image are captured by the same sensor, and wherein the first image and the second image are captured at mutually exclusive times.

44. The method of claim 30, further comprising identifying the object detected in the second sequence of images, and wherein a label corresponding to the identified object is displayed with the first sequence of images.

45. The method of claim 30, wherein a segmentation of the first image is displayed based on the detected object.

46. ​​The method of claim 30, wherein a computer-assisted medical system is controlled based on the detected object.

47. The method of claim 46, wherein the computer-assisted medical system comprises a manipulator arm, and wherein the controlling comprises inhibiting movement of the manipulator arm based on the detected object.

48. The method of claim 30, wherein the detected object corresponds to an anatomical feature of the scene that is internal to tissue in the scene, and wherein the method further comprises constructing a three-dimensional model of the anatomical feature.

49. The method of claim 30, further comprising: identifying the object detected in the second sequence of images; and Wherein performing the operation is also based on determining that the identified object corresponds to a particular type of anatomical feature.

50. The method of claim 49, wherein the second operation comprises triggering video recording, updating a user interface, or rendering a notification.

51. The method of claim 30 further comprising identifying the object as an anatomical feature, and wherein an identifier corresponding to the identified anatomical feature is provided to an algorithm that (i) identifies a segment of a surgical procedure, (ii) identifies a type of medical procedure, or (iii) evaluates a molecular identity.

52. The method of claim 30, further comprising: identifying the object detected in the second sequence of images; and Anatomical dimensions are estimated based on the identified object.

53. The method of claim 30, wherein the first image sequence is displayed while the object is detected in the second image sequence.

54. The method of claim 30, further comprising causing the first sequence of images to be displayed but not the second sequence of images to be displayed.

55. The method of claim 30, wherein the first sequence of images and the second sequence of images are captured by an imaging device that senses visible light and invisible light.

56. The method of claim 30, wherein the operation is performed on the first sequence of images during a surgical procedure.

57. A method according to claim 56, wherein accessing the first image sequence and the second image sequence includes receiving an image stream including the first image and the second image from an imaging device of a computer-assisted medical system used to perform the surgical procedure during the surgical procedure, and wherein the operation is performed in real time when the image stream is received.

58. The method of claim 30 further comprising inputting the second image into a machine learning model.

59. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform a process comprising: accessing a first sequence of images including a first image based on illumination of a scene associated with a medical procedure using visible spectrum light; accessing a second sequence of images including a second image based on illumination of the scene using light in a non-visible spectrum; detecting an object in the second image sequence; and An operation is performed on the first sequence of images based on the detected object.

Images