TECHNIQUES FOR SIMULTANEOUS EXECUTION OF INDIVIDUAL IMAGING MODALITIES

Abstract

A system and processing techniques for the simultaneous analysis of multiple image modalities. The described method can include the acquisition of a combined image stream containing a first modality stream and at least one second modality stream. The technique proceeds with the analysis of the first modality stream using a first analysis module and simultaneously with the analysis of a second modality stream using a second analysis module. The combined image stream can be generated using time-synchronized illumination modalities to create nested image streams, or by extracting narrow color bands from a primary color sensor.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 364,814, filed May 17, 2022, the contents of which are incorporated by reference in their entirety. BACKGROUND

[0002] During a typical endoscopic procedure, a healthcare provider (HCP) may use several different imaging modalities to gain a complete understanding of the anatomy being examined. During a colonoscopy, for example, a physician may guide a colonoscope through a patient's colon to detect polyps or other target structures. When palpating seemingly healthy tissue, the healthcare provider may primarily rely on one 1 stimaging modality such as white light endoscopy (WLE). With WLE, images of the colon are captured while the full spectrum of visible light exits the colonoscope. During the colonoscopy, the healthcare provider can manually switch from the imaging modality 1 at various times. st to an imaging modality 2 nd For example, if an area of ​​interest, such as a polyp, is identified, the healthcare provider can manually press a button on the colonoscope to switch to a narrowband imaging (NBI) modality. Narrowband imaging captures images of the colon while a specific range of visible light (but not the full spectrum) is emitted by the colonoscope.

[0003] Background: NBI is an optical imaging technology that enhances the visibility of vessels and other tissue on the mucosal surface. NBI works by selectively emitting only certain wavelengths of light that are absorbed by hemoglobin and penetrate only the surface of human tissue. Therefore, with NBI, capillaries on the mucosal surface are displayed in brown and veins in the submucosa in cyan on the monitor. NBI is not intended as a substitute for histopathological sampling as a means of diagnosis.

[0004] While CADe (Computer Aided Detection) algorithms can detect polyps or other abnormalities by analyzing WLE images, many CADx (Computer Aided Diagnosis) algorithms that classify polyps or other abnormalities (e.g., malignant, benign, etc.) are designed to analyze NBI images. OVERVIEW

[0005] Modern endoscopy devices operate discretely in one imaging modality until manually switched to another. Accordingly, such devices are not capable of simultaneously generating and displaying real-time image feeds for multiple discrete imaging modalities. Furthermore, modern endoscopy devices that operate discretely in one imaging modality until manually switched to another are not capable of simultaneously delivering suitable images for CADe and CADx algorithms if these different algorithms require different image types as inputs. Furthermore, there are systems that can provide multiple imaging modalities simultaneously for operator display.However, these systems do not provide the operator with sufficient mechanisms to utilize the different image streams, resulting in the operator focusing on only one of the different streams.

[0006] The inventors herein have developed systems and techniques to solve these problems with conventional endoscopic instruments. The systems and techniques discussed herein involve acquiring a composite image stream comprising multiple illumination modalities and simultaneously processing images extracted from the composite image stream using various computer-aided detection and diagnostic modules (e.g., multiple machine learning algorithms trained to utilize different illumination modalities).

[0007] The inventors have developed several different techniques for acquiring a composite image stream, extracting illumination modality-specific image streams from the composite image stream, and simultaneously feeding the illumination modality-specific image streams to different algorithms (e.g., CADe and CADx modules). The inventors have also developed techniques for displaying one of the illumination modality-specific image streams, e.g., a white light image stream, and then overlaying the results of CADe and CADx modules running simultaneously on the different illumination modality-specific image streams extracted from the composite image stream.

[0008] The techniques presented here eliminate the need for medical personnel to manually switch between different illumination modalities and offer greater flexibility in presenting results to medical personnel during an endoscopic procedure. The techniques also enable the simultaneous analysis of an endoscopic procedure using multiple computer-assisted analysis algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings, which are not necessarily to scale, like numbers may describe similar components in different views. Like numbers with different letter suffixes may represent different variations of similar components. The drawings generally illustrate, by way of example but not limitation, various embodiments covered in this document. Fig. 1 shows a block diagram of an example system in accordance with at least one example of this disclosure. Fig. 2 shows an example of a composite image acquisition technique in accordance with at least one example of this disclosure. The Fig. show an example of a composite image capture technique using a color image sensor in accordance with at least one example of this disclosure. Fig. 4 is a flowchart illustrating a technique for simultaneously analyzing multiple image streams acquired with different illumination modalities in accordance with at least one example of this disclosure. Fig. 5A is a flowchart illustrating a technique for simultaneously analyzing multiple image streams acquired with different illumination modalities in accordance with at least one example of this disclosure. Fig. 5B-5C are partial flow diagrams illustrating variations of a technique for generating and extracting various composite image streams in accordance with at least one example of this disclosure. Fig. shows a block diagram of an example machine on which one or more of the techniques discussed herein may be performed in accordance with at least one example of this disclosure. DETAILED DESCRIPTION

[0010] An endoscopy system configured to automatically and continuously switch between multiple imaging modalities and generate discrete image streams corresponding to individual ones of the imaging modalities. Thus, the endoscopy system is configured to simultaneously generate and display video streams of anatomy imaged in two or more discrete imaging modalities (and is not limited to operating in only one modality at a time).

[0011] In another example, an endoscopy system is configured to use time-sequential imaging to sequentially read a color image sensor to generate a plurality of narrowband images that can be selectively combined to generate images representative of a plurality of imaging modalities and can then be used to display video streams of anatomy imaged in two or more discrete imaging modalities (see discussion with reference to Fig. 2-3B below).

[0012] Fig. 1 shows a block diagram of an example system 100 in accordance with at least one example of this disclosure. The system 100 represents a minimal system implementation that enables simultaneous imaging with multiple illumination modalities and simultaneous computer-assisted analysis of the image streams with multiple illumination modalities. Other examples of a suitable system may include additional processors, dedicated video processors, and / or multiple systems that simultaneously receive the composite image stream. In this example, the system 100 may include an instrument 110 and a clinical support system 102. The clinical support system 102 includes a processor 104, a storage device 106, and an output device 108. In this example, the clinical support system 102 may include bidirectional communication with the instrument 110.The instrument 110 may include a camera 112 and a lighting system 114. In certain examples, the instrument 110 is an endoscopic instrument used in procedures such as colonoscopy, among others.

[0013] In this example, instrument 110 communicates with clinical support system 102, e.g., by transmitting a composite image stream captured by camera 112 using illumination system 114. Illumination system 114 enables camera 112 to capture a composite image containing two or more image streams, each image stream generated using a selected illumination modality. For example, the composite image stream may include a white light image stream and an NBI image stream. In this example, clinical support system 102 receives the composite image stream and extracts each of the embedded image streams (e.g., a white light image stream and an NBI image stream). Further details regarding the operation of camera 112 and illumination system 114 are described below with reference to Fig. 2-3B discussed.

[0014] The clinical support system 102 may include various CADe and CADx modules stored in the storage device 106 and executed by the processor 104 based on the extracted image streams. The output of the CADe and CADx modules may be displayed on the output device 108 along with one or more of the extracted image streams. In some examples, the Fig. 6 may be used as clinical support system 102.

[0015] Fig. 2 illustrates an example of a composite image acquisition technique 200 in accordance with at least one example of this disclosure. In this example, the composite image stream acquisition technique 200 synchronizes different illumination modalities with different sequences of frames within the composite image stream. In the present disclosure, the term "interleaved" is used as a shorthand to describe the multiple image streams created by synchronizing the illumination modality with different sequences of frames within the composite image stream (e.g., a 30 Hz white light image stream interleaved with a 30 Hz NBI image stream in a 60 Hz composite image stream).

[0016] In Fig. 2, individual images within a continuous image sequence are labeled as a first modality 212, a second modality 214, and a third modality 216, shown with distinct cross-hatching and in the format AB, where A represents a discrete image modality and B represents a successive image of this image modality within the continuous image sequence (1.1, 2.1, 3.1, 1.2, 2.2, ... etc.).

[0017] As in Fig. 2, the endoscopy system (e.g., system 100) may generate a composite data stream 210 containing images corresponding to a plurality of different imaging modalities. In particular, the composite stream 210 includes image 1.1 (a first image in a sequence of images acquired with imaging modality 1 st were taken), followed by Figure 2.1 (a first image in a sequence of images taken with an imaging modality 2 ndwere acquired), etc. The endoscopy system may be operated at a full cycle frequency 230, which represents the time period in which a single image of each imaging modality is acquired. Since multiple images are acquired within each full cycle frequency 230, the endoscopy system may also be characterized by a frame rate 232, which represents the time period between the frames of any illumination modality type. For the purposes of this example, it is assumed that the full cycle frequency 230 is 60 Hz, the endoscopy system continuously switches between three illumination modalities, and that the same time period exists between each image in the composite data stream 210. Based on these parameters, the frame rate 232 is therefore 180 Hz. Let us further assume that the system is powered by the 1 st Imaging modality 212 to 2 nd Imaging modality 214 to 3 rdImaging modality 216 to 1 st Imaging modality 212 switches, and so on. Other sequences may also be implemented. In some examples, the composite stream 210 may include any number of embedded individual modality streams. Because illumination modalities such as white light and NBI are commonly used in typical endoscopic procedures, the examples discussed here are limited to two interleaved image streams within the composite image stream.

[0018] The endoscopy system may further include a video processor (e.g., processor 104) communicatively coupled to the colonoscope (or other type of image acquisition device), such as instrument 110. After receiving the composite data stream 210, which includes sequences of images from all three imaging modalities (e.g., interleaved within the composite data stream 210), the video processor analyzes the images according to the imaging modality to generate discrete data streams uniquely corresponding to each imaging modality. In this example, a 1 st Modality stream 220 images that are transmitted over the 1 st imaging modality 212 in the order in which they were acquired (e.g., 212(1), 212(2), 212(3), 212(4), ... 212(N)), while a 2 nd Modality stream contains 230 images that are transmitted over the 2 ndImaging modality 214 were acquired in the order in which they were acquired (e.g., 214(1), 214(2), 214(3), 214(4), ... 214(N)). Since in this example the Full Cycle Hz 230 of the composite data stream 210 is 60 Hz, the refresh rate of each imaging modality-specific data stream (e.g., 1 st modality stream 220, 2 nd modality stream 230 and N thmodality stream (not shown) is also 60 Hz—which is a suitable refresh rate for real-time intraprocedural video display. Within each illumination modality stream, the frame rate 234 is the same as the frame rate 236 (since there are no longer any interleaved additional image streams). In one example, the illumination system 114 for the instrument 100 may include a color wheel to implement the various imaging modalities. The color wheel may be tuned to the image sensor frequency of the camera 112 to align the imaging modality with the timing of the acquisition sequence.

[0019] In another example, the illumination system 114 may include an array of light-emitting diodes (LEDs) or laser diodes that can be synchronized to a frequency of the camera 112 to create various illumination modalities. An LED or laser diode illumination system may be configured to generate a narrowband light output through selective color activation within the array. The selective activation may be synchronized to subfrequencies of the camera frequency to create a combined image stream with multiple image modalities (similar to other examples).

[0020] After the generation of the discrete image streams 220, 230 by the modality stream N thThe video processor may deliver the streams simultaneously to a display (e.g., output device 108) so that the healthcare professional can view the video streams of several different image modalities simultaneously without having to manually switch between the image modalities. Furthermore, in some embodiments, the video processor may also forward individual ones of the discrete image streams to different image analysis tools configured to analyze different types of images. For example, in an embodiment where the composite data stream 210 contains blended (interlaced) images acquired via both WLE and NBI, a discrete image stream corresponding to WLE (e.g., 1 st modality data stream 220), to a CADe tool (e.g., an AI / ML processing module focused on recognition), while another discrete image stream corresponding to NBI (e.g., 2 ndModality Stream 230), which can be made available to a CADx tool (e.g., an AI / ML processing module focused on classification). In this example, artificial intelligence / machine learning models trained on different image modalities can be applied simultaneously to an image stream. The results are merged into annotated displays that are made available to medical personnel, e.g., the 1 st Modality stream 220, which is overlaid with the results of the CADe and CADx tools on the output device 108.

[0021] In this way, an endoscopy display can simultaneously display image streams in multiple imaging modalities, while annotating each of these image streams in real time with intelligent annotations provided by specialized CAD tools that have different requirements for the input image modality. In some embodiments, the endoscopy display (e.g., output device 108) can provide medical personnel with a video in a 1 st Modality 220 (e.g., full-spectrum white light imaging) and composite or overlay information (e.g., bounding boxes, classifications, etc.) obtained by an AI model that processes images of a 2 ndModality 230 (e.g., narrowband imaging). In this way, information generated by analyzing an image stream from a non-preferred imaging modality can be provided to the healthcare provider, and this information can be directly overlaid or otherwise displayed with another image stream of their preference (e.g., viewing preference). For example, a healthcare provider who prefers to view an endoscopic image stream in full-spectrum white light can still gain valuable insight by enabling the system to simultaneously analyze an image stream corresponding to one or more predefined narrower light bands.

[0022] In the Fig. In the example shown, the different imaging modalities are created by timing the light generation for each modality with the corresponding image acquisition time for that modality. Accordingly, in the example described above, full-spectrum light generation would occur at 60 Hz, while the NBI imaging light generation would also occur at 60 Hz but would be interleaved with the full-spectrum light, and so on.

[0023] In another example, a different but related technique may be used to generate a variety of image modalities, as in Fig. 3A-3B. Fig. 3A-3B illustrate an example of a composite image capture technique using a color image sensor in accordance with at least one example of the present disclosure. This example utilizes the concept of time-sequential images, where the images are captured by an image sensor (e.g., a CCD or CMOS sensor). Time-sequential imaging typically uses a monochrome sensor to capture three consecutive images illuminated with red, blue, and green light, and then uses an overlay of these three images to create an RGB image.

[0024] In this additional embodiment, the use of a color sensor instead of a black and white sensor is provided in order to read out narrower spectral bands in each image by using a color sensor: a. In the case of a primary color sensor, the sensor has three primary bands [Red], [Green], [Blue] - and we can distribute several narrow bands within each main color b. In the case of a complementary color sensor, the sensor has the primary bands [Yellow], [Cyan], [Magenta], [Green] - and we can distribute several narrow bands within each main color

[0025] In the further embodiment, the sensitive area of ​​each primary band is divided into several narrow bands, which are recorded sequentially over time. In one example, the full-spectrum light 302 is passed through various filters 304 to generate output patterns on each part of the RGB sensor 306.

[0026] In the Fig. The example shown uses an RGB sensor with a total of 9 bands. (3 frames * 3 bands per exposure = 9 bands). In this example, the bands are labeled 312 to 336. It is now intended that each of these 9 resulting spatial-spectral data sets is stored in an electrical memory and can then be selectively combined into different imaging modalities. For example, the combination of all 9 data sets can be used to generate a "normal" RGB image (e.g., a full-spectrum white light image stream). Alternatively, selected narrow bands in the green and blue ranges can be used to generate an NBI image. Alternatively, selected narrow bands in the red range can be used to generate an RDI image (see Fig. for illustration purposes).

[0027] In this example, frame 1 (310) includes the red band 312, the green band 314, and the blue band 316; frame 2 (320) includes the red band 322, the green band 324, and the blue band 326; and frame 3 (330) includes the red band 332, the green band 334, and the blue band 336. In this example, each of the red bands (312, 322, 332) encompasses a narrow range of color frequency within the sensitivity of the red portion of the RGB sensor. Similarly, the green (314, 324, 334) and blue (316, 326, 336) bands are each narrow frequency bands within the green and blue ranges of the RGB sensor, respectively. All narrow bands within images 310, 320 and 330 are combined at operation 340 to form a composite image stream 350 (shown in Fig. 3B).

[0028] Once the composite image stream 350 is generated, the processor 104 within the clinical support system 100 can be used to generate a variety of different modality-specific image streams. In this example, the processor 104 can extract a standard WLI (white light image) image stream 360 by selecting all bands within the composite image stream 350. At the same time, the processor 104 can select fewer bands to generate other specialized image modalities, such as the NBI stream 370 or the Dual Red Imaging (DRI) stream 380. Other hyperspectral subsets can be selected to generate a variety of image streams (represented by image stream 390). As shown in Fig. As shown in Figure 3B, each of the different image streams can be generated simultaneously and used simultaneously for various specific computer-aided processing for anomaly detection and diagnosis.

[0029] It is intended that multiple imaging modalities can be provided in parallel, e.g., for different displays, for different areas of a display, to be overlaid with false color representations, or to be submitted to different algorithms for further analysis, e.g., for CADe or CADx applications. As described in Fig. For example, as shown in Figure 3B, the different image modalities can be processed by different dedicated neural networks (DNNs) to achieve different results, such as polyp detection, polyp classification, blood vessel identification, and / or bleeding risk identification, as well as a neural network for determining tissue oxygenation, among other types of processing.

[0030] Fig. 4 is a flowchart illustrating a method 400 for simultaneously analyzing multiple image streams acquired with different illumination modalities, according to at least one example of this disclosure. In this example, the technique 400 may include operations such as acquiring a composite image stream at 402, analyzing a first modality stream at 404, and simultaneously analyzing a second modality stream at 406, and optionally outputting analysis results at 408. The technique 400 is described with reference to the Fig. 1 system 100 is discussed.

[0031] In one example, the method 400 may begin at 402 with the camera 112 and the lighting system 114 collaborating to acquire a composite image stream. At 404, the method 400 continues with the clinical support system 102 using the processor 104 to analyze a first modality stream extracted from the composite image stream acquired at 402. At 406, the method 400 operates concurrently with operation 404 with the clinical support system 102 using the processor 104 to analyze a second modality stream extracted from the composite image stream. At 408, the method 400 may optionally conclude with the clinical support system 102 outputting the analysis results of operations 404 and 406 via the output device 108.

[0032] In some examples, operation 408 may include outputting the first modality stream and / or the second modality stream on output device 108 to a healthcare professional. Furthermore, the analysis results of operations 404 and 406 may include identifying and / or classifying anomalies associated with modality image stream 1. st or 2 nd or adjacent parts of a screen. For example, a CADe analysis of modality stream 1 st at 404, generate a bounding box around the identified anomaly while simultaneously displaying a classification of the anomaly generated by a CADx analysis of the second modality stream at 406.

[0033] Fig. 5A is a flowchart illustrating a method 500 for simultaneously analyzing multiple image streams acquired with different illumination modalities, in accordance with at least one example of this disclosure. In this example, the method 500 may include operations such as acquiring a composite image stream at 502, extracting a 1 st Modality stream at 504, extracting a 2 nd Modality stream at 506 and simultaneously analyzing the 1 st and 2 nd Modality streams at 508 and 510, respectively. The method 500 may optionally also include operations for generating an output at 512 and for overlaying analysis results at 514. Note: Although not depicted as parallel or concurrent operations, extracting the modality streams 1 st and 2 ndat operations 504 and 506 in a system such as system 100, may be performed concurrently. Generating an output at 512 may also include either automatically selecting the modality stream to be displayed on an output device based on the analysis results or suggesting to the operator that a particular modality stream be selected for display. For example, if a detection algorithm detects a potential finding in the first stream and the characterization algorithm classifies that finding as disease, the system may switch to the second modality stream or suggest to the operator that the second modality stream be displayed.Alternatively, the system can also produce a hybrid or enhanced output, where the image is displayed in a first modality stream but certain regions of the image are overlaid with the corresponding region of the second modality stream, or where a cropped and magnified region of the second modality stream is displayed in a picture-in-picture configuration, even if not explicitly shown. For example, the system can display a white-light image but then overlay the region of a detected adenoma with the corresponding NBI image, or it can display the white-light image and a magnified view of a region of interest corresponding to the location of a detected polyp in a smaller image adjacent to the white-light image.

[0034] In this example, the method 500 may begin at 502 with the camera 112 and the lighting system 114 capturing a composite image stream. At 504, the method 500 may continue with the processor 104 generating a 1 st Modality stream is extracted from the composite image stream received from camera 112. The method 500 may begin by extracting a 2 nd modality stream at 506. In operations 508 and 510, the technique 500 may simultaneously st Modality stream and the 2 nd Analyze modality stream. In one example, the 1 st Modality stream analyzed at 508 using a CADe module operating on the clinical support system 102. The 2 nd Modality stream may be analyzed at 510 using a CADx module operating concurrently on the clinical support system 102.

[0035] At 512, the method 500 may optionally continue with the clinical support system 102 generating an output that includes one or more of the modality streams 1 st and 2 nd and may include analysis results from operations 508 and 510. At 514, method 500 may optionally conclude with clinical support system 102 overlaying the analysis results from the CADe and / or CADx operations at 508 and 510 on an output display.

[0036] Fig. are partial flow diagrams illustrating variations of the technique 500 for generating and extracting various composite image streams in accordance with at least one example of this disclosure. Fig. 5B shows a variant of the method 500 that is used with the Fig. 2. Method 500B may include specific implementations of operations 502, 504, and 506 of method 500. In this example, method 500B may begin at 502 with operations including operating camera 112 at 60 Hz at 520, synchronizing an illumination modality 1 st at 522 and the synchronization of a lighting modality 2 nd at 524. In this example, the camera 112 is operating at a full frame rate of 60 Hz and includes two interleaved modality streams of 30 Hz each. At 504, the method 500B may continue by the 1 st modality stream is extracted from the composite stream by selecting odd images from the composite stream at 526 (where the odd images are each selected using a 1 stlighting modality in operation 520). At 506, method 500B continues with processor 104 detecting the 2 nd Modality stream extracted from the composite image stream by extracting even images from the composite stream at 528.

[0037] Fig. Figure 5C shows a second variant of the technique 500, which is used with the Fig. 3A and Fig. 3B. Method 500C may include specific implementations of operations 502, 504, and 506 that are similar to method 500B described above. In this example, technique 500C may begin at 502 with operations to acquire a composite image, including acquiring a 1 st image with 3 narrow frequency bands at 530, capturing a 2 nd image with a second set of 3 narrow frequency bands at 532 and the acquisition of a 3 rdimage with a third set of 3 narrow frequency bands at 534. As in the Fig. As described above, the frequency band sets in this example refer to the red, green, and blue components of an RGB camera sensor. The acquisition of the three different images in operations 530, 532, and 534 is repeated at the camera frame rate to generate the composite image stream at 502. At 504, method 500C may continue with the clinical support system 102 generating a modality stream 1 st extracted from the composite stream acquired at 502. In this example, operation 504 may include extracting a first set of selected bands from the composite image stream to obtain a 1 stmodality stream. In one example, the first set of selected bands may include all nine bands acquired in operations 530, 532, and 534 to form a WLI modality. At 506, method 500C may continue with the clinical support system 102 selecting a 2 nd Modality stream extracted from the composite stream. In this example, operation 506 may include extracting a second set of selected bands from the composite image, such as narrow bands within red and green frequency ranges, to form an NBI modality stream. As described above with respect to Fig. 3B, method 500C can be modified to extract more than two different modality streams by simultaneously selecting additional sets of selected bands. The different modality streams can be generated simultaneously and fed to different analysis modules, which, as described above, can also be executed simultaneously.

[0038] Fig. 6 shows a block diagram of an example machine 600 on which one or more of the techniques (processes) discussed herein may be performed in accordance with some embodiments. The example device 600 may also be implemented within the framework described above with reference to Fig.1 may be used as a clinical support system. In alternative embodiments, machine 600 may operate as a standalone device and / or be connected (e.g., networked) to other machines. In a networked deployment, machine 600 may operate as a server, a client, or both in a server-client network environment. In one example, machine 600 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 600 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, switch, or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be performed by that machine.Although only a single machine is depicted, the term "machine" also includes any collection of machines that individually or collectively execute a set (or sets) of instructions to perform one or more of the methods discussed herein, e.g., cloud computing, software as a service (SaaS), or other computer cluster configurations.

[0039] The machine (e.g., the computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 604, and static memory 606, which may communicate partially or entirely with each other via an interconnect (e.g., a bus) 608. The machine 600 may further include a display unit 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In one example, the display unit 610, the input device 612, and the UI navigation device 614 may be a touchscreen display. The machine 600 may additionally include a storage device (e.g., drive) 616, a signal generating device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 621, such asa Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensor. The machine 600 may include an output controller 628, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near-field communication (NFC), etc.) connection for communicating with and / or controlling one or more peripheral devices (e.g., a printer, card reader, etc.).

[0040] The storage device 616 may include a machine-readable medium 622 storing one or more sets of data structures or instructions 624 (e.g., software) that embody or are used by one or more of the techniques or functions described herein. The instructions 624 may also be located entirely or at least partially in the main memory 604, the static memory 606, or the hardware processor 602 when executed by the machine 600. In one example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the storage device 616 may constitute a machine-readable medium.

[0041] While machine-readable medium 622 is illustrated as a single medium, the term "machine-readable medium" may encompass a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store the one or more instructions 624. The term "machine-readable medium" may encompass any medium capable of storing, encoding, or carrying instructions for execution by machine 600 that cause machine 600 to perform one or more of the techniques of the present disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state storage, as well as optical and magnetic media.

[0042] The instructions 624 may further be transmitted or received over a communications network 626 using a transmission medium via the network interface device 620 using any transmission protocol (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Data Packet Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Examples of communications networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, and others. In one example, the network interface device 620 may include one or more physical sockets (e.g.,Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communications network 626. In one example, the network interface device 620 may include a plurality of antennas to communicate wirelessly using at least one of the SIMO (Single-Input Multiple-Output), MIMO (Multiple-Input Multiple-Output), or MISO (Multiple-Input Single-Output) techniques. The term "transmission medium" encompasses any intangible medium capable of storing, encoding, or transmitting instructions for execution by the machine 600, and includes digital or analog communication signals or other intangible media for facilitating communication of such software.

[0043] The methods described herein may be at least partially machine- or computer-implemented. Some examples may include a computer-readable medium or a machine-readable medium encoded with instructions that configure an electronic device to perform methods as described in the above examples. Implementation of such methods may include code, such as microcode, assembly language code, high-level language code, or the like. Such code may include computer-readable instructions for performing various methods. The code may form portions of computer program products. Furthermore, in one example, the code may be stored on one or more transient, non-transitory, or non-transitory tangible computer-readable media, for example, during execution or at other times.Examples of these tangible computer-readable media include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact discs and digital video disks), magnetic cassettes, memory cards or sticks, RAMs (Random Access Memories), ROMs (Read Only Memories), and the like. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 364,814

[0001]

Claims

[1] A method for simultaneously analyzing multiple imaging modalities, the method comprising: Acquiring a combined image stream including a first modality stream and at least one second modality stream; analyzing the first modality stream with a first analysis module; and Analyzing the second modality stream with a second analysis module simultaneously with analyzing the first modality stream. [2] The method of claim 1, wherein the first analysis module is a computer-aided identification module and wherein the second analysis module is a computer-aided classification module. [3] The method of claim 2, wherein analyzing the first modality stream with the computer-assisted identification module comprises applying a first machine learning model to the first modality stream. [4] The method of claim 3, wherein applying the first machine learning model comprises outputting an identification of an abnormality, and wherein analyzing the second modality stream comprises simultaneously outputting an indication of the classification of the abnormality. [5] The method of claim 4, wherein outputting the identification comprises outputting a region of interest that identifies a location of the abnormality. [6] The method of claim 2, wherein analyzing the second modality stream with the computer-assisted classification module comprises applying a second machine learning model to the second modality stream. [7] The method of claim 2, wherein the first modality is white light, and wherein the method further comprises displaying the first modality on a screen. [8] The method of claim 7, further comprising simultaneously superimposing results of the computer-aided identification module and the computer-aided classification module on the screen. [9] The method of claim 1, wherein detecting the combined image stream comprises detecting at least one 50 Hz image stream including a first 25 Hz image stream and a second 25 Hz image stream interleaved with the first 25 Hz image stream. [10] The method of claim 9, wherein detecting the combined image stream comprises parsing the first 25 Hz image stream as the first modality stream and the second 25 Hz image stream as the second modality stream. [11] The method of claim 1, wherein capturing the combined image stream comprises capturing a first portion of the combined image stream with a first illumination modality and capturing a second portion of the combined image stream with a second illumination modality, the first portion and the second portion being interleaved. [12] The method of claim 1, wherein acquiring the combined image stream comprises synchronizing a color wheel to a sensor frequency of a sensor used to acquire the combined image stream, the color wheel comprising a first set of filters for generating a first imaging modality corresponding to the first modality stream and a second set of filters for generating a second imaging modality corresponding to the second modality stream. [13] The method of claim 1, wherein acquiring the combined image stream comprises synchronizing light-emitting diodes or laser diodes to a sensor frequency of a sensor used to acquire the combined image stream, the light-emitting diodes or laser diodes being configured to generate a first image modality corresponding to the first modality stream and a second image modality corresponding to the second modality stream. [14] The method of claim 1, further comprising simultaneously outputting the first modality stream and the second modality stream to a user interface device. [15] The method of claim 2, further comprising simultaneously outputting: an identification output generated by the computer-based identification module; and a classification output generated by the computer-aided classification module. [16] The method of claim 15, wherein the simultaneous output comprises overlaying at least one of the identification output and the classification output over at least one of the first modality stream and the second modality stream. [17] The method of claim 1, wherein capturing the combined image stream comprises: Capturing a full frame rate image using a primary color sensor; Distributing a plurality of narrow color bands within three primary bands generated by the primary color sensor; Selecting a first subset of bands from the plurality of narrow color bands to generate the first modality stream; and Selecting a second subset of bands from the plurality of narrow color bands to generate the second modality stream. [18] The method of claim 17, wherein acquiring the combined image stream comprises selecting a third subset of bands from the plurality of narrow color bands to generate a third modality stream. [19] The method of claim 1, wherein capturing the combined image stream comprises: Capturing a full frame rate image using a complementary color sensor; Distributing a plurality of narrow color bands within four primary bands generated by the complementary color sensor; Selecting a first subset of bands from the plurality of narrow color bands to generate the first modality stream; and Selecting a second subset of bands from the plurality of narrow color bands to generate the second modality stream. [20] System comprising: an endoscope having a camera and an illumination system, wherein the camera and the illumination system are configured to generate a first modality stream and a second modality stream; a processor and a memory device, the memory device containing instructions that, when executed by the processor, cause the processor to concurrently access the first modality stream and the second modality stream and to perform a method according to any one of claims 1 to 19. [21] A computer-readable medium containing instructions which, when executed by a clinical support system, cause the system to perform any of the methods of claims 1 to 19. [22] A method according to any one of claims 1 to 18, further comprising: Selecting, based on analyzing the first modality stream and analyzing the second modality stream, a display modality stream from the first modality stream and the second modality stream; and Displaying the display modality stream on a display device. [23] A method according to any one of claims 1 to 18, further comprising: Output a recommended image stream based on the analysis of the first modality stream and the analysis of the second modality stream. [24] A method according to any one of claims 1 to 18, further comprising: Generating, based on the analysis of the first modality stream and the analysis of the second modality stream, a hybrid image that selectively combines outputs of the first modality stream and the second modality stream.

Images