Techniques for simultaneously executing discrete imaging modalities

Abstract

Systems and processing techniques are provided for simultaneously analyzing multiple imaging modalities. The techniques discussed herein can include capturing a combined image stream that includes a first modality stream and at least a second modality stream. The techniques then analyze the first modality stream using a first analysis module and, in parallel, analyze the second modality stream using a second analysis module. The combined image stream can be generated by using a synchronized illumination modality to generate an interleaved image stream or by extracting narrow color bands from a primary color sensor.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 364,814, filed May 17, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002] During typical endoscopic procedures, healthcare providers (HCPs) may utilize multiple different imaging modalities to gain a complete understanding of the anatomical structures being examined. For example, during a colonoscopy, the HCP may manipulate the colonoscope across the patient's entire colon to identify polyps or other structures of interest. When scanning seemingly healthy tissue, the HCP may primarily use a first imaging modality such as white - light endoscopy (WLE). WLE can involve capturing an image of the colon while visible light across the entire spectrum is emitted from the colonoscope. As various cases throughout a colonoscopy, the HCP may choose to manually toggle from a first imaging modality to a second imaging modality. For example, when identifying an area of interest such as a polyp, the HCP may manually press a button on the colonoscope to toggle to the narrow - band imaging (NBI) modality. NBI can involve capturing an image of the colon while a predetermined range of visible light (but not the entire spectrum) is emitted from the colonoscope.

[0003] As background, NBI is an optical imaging technique that enhances the visibility of blood vessels and other tissues on the mucosal surface. NBI functions by selectively emitting only specific wavelengths of light that are absorbed by hemoglobin and penetrate only the surface of human tissue. As a result, when using NBI, capillaries on the mucosal surface are displayed in brown and blood vessels within the submucosa are displayed in cyan on the monitor. NBI is not intended to be a means of diagnosis in place of histopathological sampling.

[0004] Furthermore, while a CADe (computer-aided detection) algorithm can identify polyps or other abnormalities by analyzing images captured through WLE, many CADx (computer-aided diagnosis) algorithms designed to classify polyps or other abnormalities (such as malignant type, benign type, etc.) are designed to analyze images captured through NBI. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0005] Latest endoscopy devices operate in one imaging modality discretely until manually toggled to another imaging modality. Therefore, such devices are unable to simultaneously generate and display real-time image feeds for multiple discrete imaging modalities. Furthermore, latest endoscopy devices that operate in one imaging modality discretely until manually toggled to another imaging modality are unable to simultaneously provide images suitable for these different algorithms in situations where CADe and CADx algorithms require different image types as inputs. Furthermore, there are certainly systems that can provide multiple simultaneous imaging modalities for an operator to view. However, those systems do not provide the operator with a mechanism sufficient to utilize multiple image streams, and as a result, the operator will focus on only one of the multiple streams.

[0006] The inventors have developed systems and techniques for addressing these issues using conventional endoscopic equipment. The systems and techniques discussed herein capture a synthetic image stream that includes multiple illumination modalities, and then process in parallel the frames extracted from the synthetic image stream using different computer-aided detection modules and computer-aided diagnosis modules (e.g., multiple machine learning algorithms trained to utilize different illumination modalities).

[0007] The inventors have developed different techniques for capturing a synthetic image stream, extracting image streams for each illumination modality from the synthetic image stream, and feeding the image streams for each illumination modality in parallel to various algorithms (e.g., CADe modules and CADx modules). The inventors have also developed techniques for displaying one of the image streams for each illumination modality, such as a white light image stream, and then overlaying the results from both the CADe modules and CADx modules that were executed in parallel for the different image streams for each illumination modality extracted from the synthetic image stream.

[0008] The techniques discussed herein eliminate the need for the HCP to manually toggle between different illumination modalities and improve the freedom in the way the results can be presented to the HCP during the endoscopic procedure. These techniques also enable parallel analysis of the endoscopic procedure by multiple computer-aided analysis algorithms.

[0009] In drawings that are not necessarily drawn to scale, like numerals may represent similar components in different figures. Like numerals with different suffixes may represent different examples of similar components. The drawings generally illustrate, by way of example and not limitation, the various embodiments discussed in this document.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6

DETAILED DESCRIPTION OF THE INVENTION

[0011] The endoscopic examination system is configured to automatically and continuously toggle between multiple imaging modalities and to generate discrete image streams corresponding to the individual modalities of those imaging modalities. In this way, the endoscopic examination system is configured to generate and display in parallel video streams of anatomical structures imaged in two or more discrete imaging modalities (rather than being limited to operating in only one modality at a time).

[0012] In a further example, the endoscopic examination system is configured to use time-sequential imaging to sequentially read out a color image sensor, thereby generating a plurality of narrow-band images, and selectively combine these narrow-band images to create images representing various imaging modalities, and then use those images to display video streams of anatomical structures imaged in two or more discrete imaging modalities (see the discussion in FIGS. 2-3B below).

[0013] FIG. 1 shows a block diagram of an exemplary system 100 according to at least one example of the present disclosure. System 100 represents a minimal system implementation form to enable parallel multi-illumination modality imaging and parallel computer-aided analysis of multi-illumination modality image streams. Other examples of suitable systems can include additional processors, dedicated video processors, and / or multiple systems that receive synthetic image streams in parallel. In this example, system 100 can include device 110 and clinical support system 102. Clinical support system 102 includes processor 104, memory device 106, and output device 108. In this example, clinical support system 102 can include two-way communication with device 110. Device 110 can include camera 112 and illumination system 114. In some examples, device 110 is an endoscopic device used in procedures such as, among other things, colonoscopy.

[0014] In this example, device 110 communicates with clinical support system 102, such as by transmitting a synthetic image stream captured using illumination system 114 by camera 112. Illumination system 114 operates to enable camera 112 to capture a synthetic image that includes two or more image streams in which each individual image stream is generated using a selected illumination modality. For example, the synthetic image stream can include a white light image stream and an NBI image stream. In this example, clinical support system 102 receives the synthetic image stream and extracts the embedded image streams (e.g., the white light image stream and the NBI image stream) respectively. For further details regarding the operation of camera 112 and illumination system 114, reference is made to FIGS. 2 - 3B below.

[0015] Clinical support system 102 can include various CADe modules and CADx modules stored in memory device 106 for execution by processor 104 on the extracted image streams. The outputs from the CADe modules and CADx modules can be displayed on output device 108, along with one or more of the extracted image streams. In some examples, a more detailed computing device, described with reference to FIG. 6, can be utilized as clinical support system 102.

[0016] FIG. 2 shows an exemplary synthetic image capture technique 200 according to at least one example of the present disclosure. In this example, the synthetic image stream capture technique 200 synchronizes different illumination modalities with different frame sequences within the synthetic image stream. In the present disclosure, the term "alternately arranged" is used as a shorthand for describing a plurality of image streams (e.g., a 30 Hz white light image stream alternately arranged with a 30 Hz NBI image stream within a 60 Hz synthetic image stream) created by synchronizing illumination modalities with different frame sequences within the synthetic image stream.

[0017] In FIG. 2, individual frames within a continuous frame sequence are labeled as a first modality 212, a second modality 214, and a third modality 216, which are shown using distinct cross-hatching and in the format A.B, where A represents a discrete imaging modality and B represents sequential frames of that imaging modality within the continuous frame sequence (1.1, 2.1, 3.1, 1.2, 2.2,... etc.).

[0018] As shown in FIG. 2, an endoscopic inspection system (e.g., system 100) can generate a composite stream 210 that includes image frames corresponding to a plurality of different imaging modalities. Specifically, what the composite stream 210 includes is frame 1.1 (the first frame within the frame sequence captured through the first imaging modality 212), followed by frame 2.1 (the first frame within the frame sequence captured through the second imaging modality 214), and so on. The endoscopic inspection system can operate at a Full Cycle Frequency 230 that represents the amount of time in which a single image of each imaging modality is captured. Since a plurality of images are captured within each Full Cycle Frequency 230, the endoscopic inspection system can also be characterized in terms of a Single Frame Frequency 232 that represents the amount of time between individual frames of any illumination modality type. In this example, it is assumed that the Full Cycle Frequency 230 is 60 Hz, the endoscopic inspection system is continuously toggling between three imaging modalities, and there is equal timing between each frame of the composite stream 210. Therefore, based on these parameters, the Single Frame Frequency 232 is 180 Hz. It is further assumed that the system toggles from the first imaging modality 212 to the second imaging modality 214, the third imaging modality 216, the first imaging modality 212, and so on. Other sequences can also be implemented. In some examples, the composite stream 210 can include any number of embedded individual modality streams. Since illumination modalities such as white light and NBI are commonly used in typical endoscopic procedures, other examples discussed herein are limited to the discussion of two image streams arranged alternately within the composite image stream.

[0019] The endoscopic examination system can further include a video processor (such as processor 104) communicatively coupled to a colonoscope (or other type of image capture device) such as device 110. When the video processor receives a composite stream 210 that includes a sequence of frames from all three imaging modalities that are mixed (e.g., interleaved within the composite stream 210), the video processor parses the frames according to the imaging modality to generate discrete streams that uniquely correspond to each imaging modality. In this example, the first modality stream 220 includes the frames captured through the first imaging modality 212 in the order in which they were captured (e.g., 212(1), 212(2), 212(3), 212(4),...212(N)), while the second modality stream 230 includes the frames captured through the second imaging modality 214 in the order in which they were captured (e.g., 214(1), 214(2), 214(3), 214(4),...214(N)). In this example, since the full cycle Hz 230 of the composite stream 210 is 60 Hz, the refresh rate of each imaging modality-specific stream (e.g., the first modality stream 220, the second modality stream 230, and the Nth modality stream [not shown]) is also 60 Hz, which corresponds to a refresh rate suitable for real-time video display during the procedure. Within an individual illumination modality stream, the full frame rate 234 is the same as the individual frame rate 236 (because there are no longer any additional interleaved image streams). In one example, the illumination system 114 of device 110 can include a color wheel for implementing different imaging modalities. The color wheel can be timed to the image sensor frequency of camera 112 to align the imaging modality with the capture order timing.

[0020] In another example, the illumination system 114 can include an array of light-emitting diodes (LEDs) or laser diodes, and this array can be synchronized with the frequency of the camera 112 to generate different illumination modalities. The LED or laser diode illumination system can be configured to generate a narrow-band light output through selective color activation within the array. Synchronizing the selective activation with a sub-frequency of the camera frequency can result in a combined image stream having multiple image modalities (similar to other examples).

[0021] When the video processor generates from the discrete imaging streams 220, 230 up to the Nth modality stream, the HCP can view the video streams of a plurality of different imaging modalities in parallel without manually toggling between the imaging modalities, and these streams can be provided in parallel to a display (e.g., output device 108). Further, in some embodiments, the video processor can also provide individual ones of the discrete image streams to different image analysis tools configured to analyze different types of images. For example, in one embodiment where the composite stream 210 includes a mixed (interleaved) image captured through both WLE and NBI, the discrete image stream corresponding to WLE (e.g., the first modality stream 220) can be provided to a CADe tool (e.g., an AI / ML processing module focused on detection), while another discrete image stream corresponding to NBI (e.g., the second modality stream 230) can be provided to a CADx tool (e.g., an AI / ML processing module focused on classification). In this example, artificial intelligence / machine learning models trained from different image modalities can be applied to the image stream simultaneously (in parallel). The results are merged and provided to the HCP as an annotated display, such as an overlaid first modality stream 220 from the results of both the CADe tool and the CADx tool on the output device 108.

[0022] In this way, the endoscopic display can present, in parallel, image streams of multiple imaging modalities with insights provided by specialized CAD tools having different requirements for the input image modalities, with each of these image streams being intelligently annotated in real time. In some embodiments, the endoscopic display (e.g., output device 108) can present a video of a first modality 212 (e.g., full-spectrum white light observation) to the HCP and can synthesize or overlay information (e.g., bounding boxes, classifications, etc.) obtained by an AI model that analyzes an image of a second modality 214 (e.g., narrow-band light observation). In this way, information generated by analyzing an image stream of an un-preferred imaging modality can be provided to the HCP, and this information can be directly synthesized on top of another image stream that the HCP prefers (e.g., browses and prefers), or otherwise associated and displayed with that image stream. For example, an HCP who prefers to browse an endoscopic image stream of full-spectrum white light can still obtain valuable insights obtained by enabling the system to analyze, in parallel, one or more image streams corresponding to pre-defined, narrower bands of light.

[0023] In the example shown in FIG. 2, different image modalities are generated by timing the light generation associated with each modality to the associated frame capture timing for that modality. Thus, in the example described above, full-spectrum light generation occurs at 60 Hz, while NBI imaging light generation also occurs at 60 Hz, but is interleaved with the full-spectrum light.

[0024] In yet another example, as shown in FIGS. 3A-3B, another but related technique can be used to generate a number of image modalities. FIGS. 3A-3B show an exemplary synthetic image capture technique using a color image sensor according to at least one example of the present disclosure. The approach in this example uses the concept of time-sequential images, where the images are recorded by an image sensor (e.g., a CCD or CMOS sensor). In time-sequential imaging, typically a B / W sensor is used to capture three successive image frames illuminated with red light, blue light, and green light, and then an overlay of the three images is used to generate an RGB image.

[0025] In this further embodiment, it is contemplated that instead of a B / W sensor, a color sensor is used to read out a narrower spectral band in each frame using the color sensor. a. In the case of a primary color sensor, the sensor has three primary bands [Red], [Green], [Blue], and the inventors can disperse several narrow bands within each primary color b. In the case of a complementary color sensor, the sensor has four primary bands [Yellow], [Cyan], [Magenta], [Green], and the inventors can disperse several narrow bands within each primary color

[0026] This further embodiment includes dividing the sensitive area of each primary band of the primary bands into several narrow bands and recording them time-sequentially. In one example, full-spectrum light 302 passes through various filters 304 to generate an output pattern on each part of the RGB sensor 306.

[0027] In the example outlined in FIG. 3A, an RGB sensor is used with a total of nine bands. (3 frames * 3 bands per exposure = 9 bands). In this example, the bands are labeled from 312 to 336. Here, it is envisioned that each of the resulting nine spatial spectral data sets will be stored in an electrical memory and then selectively combined to enable different imaging modalities. For example, using all nine data sets combined, a “normal” RGB image (e.g., a full-spectrum white light image stream) can be generated. Using alternatively selected narrow bands within the green and blue areas, an NBI image can also be generated. Using alternatively selected narrow bands within the red area, an RDI image can also be generated (see FIG. 3B for an example).

[0028] In this example, frame 1 (310) includes a red band 312, a green band 314, and a blue band 316, frame 2 (320) includes a red band 322, a green band 324, and a blue band 326, and frame 3 (330) includes a red band 332, a green band 334, and a blue band 336. In this example, each red band (312, 322, 332) includes a narrow color frequency range within the sensitivity of the red portion of the RGB sensor. Similarly, each green band (314, 324, 334) and each blue band (316, 326, 336) are each narrow frequency bands within the green and blue portions of the RGB sensor, respectively. When all of the narrow bands within frames 310, 320, and 330 are combined in operation 340, a composite image stream 350 (shown in FIG. 3B) is formed.

[0029] After the composite image stream 350 is generated, the processor 104 within the clinical support system 102 can be used to generate image streams for a wide variety of different modalities. In this example, the processor 104 can extract a standard WLI (white light image) stream 360 by selecting all the bands within the composite image stream 350. In parallel, the processor 104 can select a smaller number of bands to generate other specialized image modalities such as an NBI stream 370 or a dual-red imaging (DRI) stream 380. Other hyperspectral subsets can also be selected to generate a wide variety of image streams (represented by the image stream 390). As shown in FIG. 3B, the different image streams can be generated in parallel and used in parallel for various specific computer-aided processes for detecting and diagnosing abnormalities.

[0030] It is envisioned that several imaging modalities can be provided in parallel for, for example, overlaying on different displays, different areas of a display, wrong-color representations, or for further analysis, for example, for application of CADe or CADx to different algorithms. For example, as shown in FIG. 3B, different image modalities can be processed by different dedicated neural networks (DNNs) among various types of processing to generate different outputs such as polyp detection, polyp classification, blood vessel identification, and / or hemorrhage risk identification, and can also be processed by a neural network to determine tissue oxygenation.

[0031] Figure 4 is a flowchart showing a technique 400 for parallel analysis of a plurality of image streams captured using different illumination modalities, according to at least one example of the present disclosure. In this example, the technique 400 can include operations such as capturing a composite image stream at 402, analyzing a first modality stream at 404, and analyzing a second modality stream in parallel at 406, and optionally, outputting the analysis results at 408. The technique 400 will be discussed with reference to the system 100 shown in FIG. 1.

[0032] In one example, the technique 400 can begin at 402 where the camera 112 and the illumination system 114 operate together to capture a composite image stream. At 404, the technique 400 continues where the clinical support system 102 uses the processor 104 to analyze a first modality stream extracted from the composite image stream captured at 402. At 406, the technique 400 functions by, in parallel with operation 404, the clinical support system 102 using the processor 104 to analyze a second modality stream extracted from the composite image stream. At 408, the technique 400 can optionally end with the clinical support system 102 outputting the analysis results from operations 404 and 406 through the output device 108.

[0033] In some examples, operation 408 can include outputting the first modality stream and / or the second modality stream on the output device 108 for the HCP. Additionally, the analysis results from operations 404 and 406 can also include identification information and / or classification of anomalies overlaid on or adjacent to the first or second modality image streams on the display screen. For example, a bounding box can be generated around the identified anomaly by a CADe analysis of the first modality stream at 404, and the classification of the anomaly generated by a CADx analysis of the second modality stream at 406 can also be displayed in parallel.

[0034] Figure 5A is a flowchart showing a technique 500 for parallel analysis of multiple image streams captured using different illumination modalities, according to at least one example of the present disclosure. In this example, the technique 500 can include operations such as capturing a composite image stream at 502, extracting a first modality stream at 504, extracting a second modality stream at 506, and parallelly analyzing the first modality stream and the second modality stream at 508 and 510 respectively. The technique 500 can optionally also include operations for generating an output at 512 and for overlaying the analysis results at 514. Although not shown as parallel or concurrent operations, it should be noted that extracting the first modality stream and the second modality stream at operations 504 and 506 can be performed in parallel within a system such as system 100. The generation of the output at 512 can also include an automatic selection of any of the modality streams to be displayed on an output device based on the analysis results, or a proposal to the operator to select a particular modality stream for display. For example, if a detection algorithm detects a potential finding within the first stream and a characterization algorithm characterizes this finding as a disease, the system can propose to the operator to switch to the second modality stream or to display the second modality stream. Alternatively, although not explicitly shown, it is also contemplated that the system can generate a hybrid or enhanced output where an image is shown as the first modality stream but the corresponding area of the second modality stream is overlaid on a particular area of the image, or an area of the second modality stream that is trimmed and enlarged is shown as a picture-in-picture configuration adjacent to the white light image. For example, the system can display a white light image but then overlay the corresponding NBI image on the area of the detected adenoma, or the system can show the white light image and display an enlarged view of the area of interest corresponding to the location of the detected polyp as a smaller picture adjacent to the white light image.

[0035] In this example, technique 500 can start at 502 where camera 112 and illumination system 114 capture a synthetic image stream. At 504, technique 500 can continue by processor 104 extracting a first modality stream from the synthetic image stream received from camera 112. Technique 500 can then extract a second modality stream at 506. At operations 508 and 510, technique 500 can analyze the first modality stream and the second modality stream in parallel. In one example, the first modality stream is analyzed at 508 using a CADe module operating on clinical support system 102. The second modality stream can be analyzed at 510 using a CADx module operating in parallel on clinical support system 102.

[0036] At 512, technique 500 can optionally continue with clinical support system 102 generating an output that can include one or more of the first modality stream, the second modality stream, and the analysis results from operations 508 and 510. At 514, technique 500 can optionally end with clinical support system 102 overlaying the analysis results from the CADe operation and / or the CADx operation at 508 and 510 on an output display.

[0037] Figures 5B - 5C are partial flowcharts showing variations of technique 500 for another generation and extraction of a synthetic image stream, according to at least one example of the present disclosure. Figure 5B shows one variation of technique 500 that is consistent with the imaging technique discussed with reference to Figure 2. Technique 500B can include specific implementations of operations 502, 504, and 506 from technique 500. In this example, technique 500B can start with operations including, at 502, operating camera 112 at 520 at 60 Hz, synchronizing the first lighting modality at 522, and synchronizing the second lighting modality at 524. In this example, camera 112 is operated at a full frame rate of 60 Hz and includes two interleaved modality streams, each at 30 Hz. At 504, technique 500B can then extract the first modality stream from the synthetic stream by selecting the odd frames from the synthetic stream at 526 (where the odd frames are each captured using the first lighting modality in operation 520). At 506, technique 500B continues by extracting the second modality stream from the synthetic image stream by the processor 104 extracting the even frames from the synthetic stream at 528.

[0038] Figure 5C shows a second variant of technique 500 that is consistent with the imaging techniques discussed with reference to FIGS. 3A and 3B. Technique 500C can include specific implementations of operations 502, 504, and 506 similar to those of technique 500B discussed above. In this example, technique 500C can start with operations for capturing a composite image, including, at 502, capturing a first frame at 530 using three narrow frequency bands, capturing a second frame at 532 using a second set of three narrow frequency bands, and capturing a third frame at 534 using a third set of three narrow frequency bands. As discussed with reference to FIGS. 3A and 3B, the sets of frequency bands in this example were related to the red, green, and blue portions of an RGB camera sensor. The capture of three different frames at operations 530, 532, and 534 is repeated at the camera's frame rate to generate a composite image stream at 502. At 504, technique 500C can continue with the clinical support system 102 extracting a first modality stream from the composite stream captured at 502. In this example, operation 504 can include extracting a first set of selected bands from the composite image stream to form a first modality stream. In one example, the first set of selected bands can include all nine bands captured at operations 530, 532, and 534 to form a WLI modality. At 506, technique 500C can continue with the clinical support system 102 extracting a second modality stream from the composite stream. In this example, operation 506 can include extracting a second set of selected bands, such as narrow bands within the red and green frequency ranges, from the composite image to form an NBI modality stream. As discussed above with reference to FIG. 3B, technique 500C can be modified to extract more than three different modality streams by selecting additional sets of selected bands in parallel. The various modality streams can be generated in parallel and supplied to various analysis modules that can also be executed in parallel as discussed above.

[0039] Figure 6 shows a block diagram of an exemplary machine 600 on which one or more of the techniques (processes) discussed herein can be implemented, according to some embodiments. The exemplary machine 600 can also be used as a clinical support system within the system 100 discussed above with reference to FIG. 1. In an alternative embodiment, the machine 600 can operate as a stand-alone device and / or can be connected (e.g., networked) to other machines. In a network-connected deployment, the machine 600 can operate as a server machine, a client machine, or both, in a server-client network environment. In one example, the machine 600 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 600 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a network switch, or a network bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, although only a single machine is shown, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to implement any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, and the like.

[0040] A machine (e.g., a computer system) 600 can include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 604, and a static memory 606, and some or all of them can communicate with each other via an interlink (e.g., a bus) 608. The machine 600 can further include a display unit 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In one example, the display unit 610, the input device 612, and the UI navigation device 614 can be a touch screen display. In addition, the machine 600 can also include a storage device (e.g., a drive unit) 616, a signal generation device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 621 such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 600 can include an output controller 628 for communicating with and / or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.), such as a serial (e.g., a universal serial bus (USB) connection, a parallel connection, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection).

[0041] The storage device 616 can include a machine-readable medium 622 on which one or more sets of data structures or instructions 624 (e.g., software) are stored that embody or are utilized by any one or more of the techniques or functions described herein. The instructions 624 can be present, fully or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution thereof by the machine 600. In one example, one of the hardware processor 602, the main memory 604, the static memory 606, or the storage device 616, or any combination thereof, can constitute a machine-readable medium.

[0042] The machine-readable medium 622 is shown as a single medium, but the term "machine-readable medium" can include a single medium or a plurality of media (e.g., a centralized database or a distributed database, and / or associated caches and servers) configured to store one or more instructions 624. The term "machine-readable medium" can include any medium that can store, encode, or carry instructions for causing the machine 600 to perform any one or more of the techniques of the present disclosure, or that can be used by or associated with such instructions to store, encode, or carry a data structure. Non-limiting examples of machine-readable media can include solid-state memory, as well as optical and magnetic media.

[0043] Command 624 can further use any one of several transfer protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.) to transmit or receive via a communication network 626 using a transmission medium through a network interface device 620. Exemplary communication networks include, among others, local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile telephone networks (e.g., cellular networks), plain old telephone service (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi (registered trademark), the IEEE 802.16 family of standards known as WiMax (registered trademark), the IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks). In one example, network interface device 620 can include one or more physical jacks (e.g., Ethernet jack, coaxial jack, or telephone jack) or one or more antennas for connecting to communication network 626. In one example, network interface device 620 can include multiple antennas for wireless communication using at least one of single input multiple output (SIMO) techniques, multiple input multiple output (MIMO) techniques, or multiple input single output (MISO) techniques. The term "transmission medium" is construed to include any non-transitory medium capable of storing, encoding, or carrying instructions for execution by machine 600 and includes digital or analog communication signals or other non-transitory media for facilitating such software communication.

[0044] The method examples described in this specification can be implemented, at least in part, by a machine or a computer. Some examples can include a computer-readable medium or a machine-readable medium encoded with instructions capable of configuring an electronic device to perform the methods described in the above examples. One implementation of such a method can include code such as microcode, assembly language code, higher-level language code, etc. Such code can include computer-readable instructions for performing various methods. The code can form part of a computer program product. Further, in one example, the code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of such tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), and the like.

Description of Signs

[0045] 100 System, Device, Clinical Support System 102 Clinical Support System 104 Processor 106 Memory Device 108 Output Device 110 Device 112 Camera 114 Lighting System 200 Synthetic Image Capture Technique, Synthetic Image Stream Capture Technique 210 Synthetic Stream 212 First Modality, First Imaging Modality 214 Second Modality, Second Imaging Modality 216 Third Modality, Third Imaging Modality 220 First modality stream, discrete imaging stream, first modality 230 Full-cycle frequency, second modality stream, full-cycle Hz, discrete imaging stream, second modality 232 Single-frame frequency 234 Full-frame rate 236 Individual frame rate 302 Full-spectrum light 304 Filter 306 RGB sensor 310 Frame 1, frame 312 Red band 314 Green band 316 Blue band 320 Frame 2, frame 322 Red band 324 Green band 326 Blue band 330 Frame 3, frame 332 Red band 334 Green band 336 Blue band 340 Calculation 350 Composite image stream 360 WLI (white light image) stream 370 NBI stream 380 Dual red imaging (DRI) stream 390 Image stream 400 Technique 500 Technique 500B Technique 500C Technique 600 Machine 602 Hardware processor 604 Main memory 606 Static memory 608 Interlink 610 Display unit 612 Alphanumeric input device 614 User interface (UI) navigation device 616 Storage device 618 Signal generation device 620 Network interface device 621 Sensor 622 Machine-readable medium 624 Instruction 626 Communication network 628 Output controller

Claims

1. A method for simultaneously analyzing a plurality of imaging modalities, comprising: capturing a combined image stream including a first modality stream and at least a second modality stream; analyzing the first modality stream using a first analysis module; analyzing the second modality stream using a second analysis module in parallel with the step of analyzing the first modality stream .

2. The method according to claim 1, wherein the first analysis module is a computer-aided identification module and the second analysis module is a computer-aided classification module.

3. The method according to claim 2, wherein the step of analyzing the first modality stream using the computer-aided identification module includes applying a first machine learning model to the first modality stream.

4. The step of applying the first machine learning model includes outputting abnormal identification information, and the step of analyzing the second modality stream includes outputting, in parallel, labels for classifying the abnormality, The method according to claim 3.

5. The method according to claim 4, wherein the step of outputting the identification information includes outputting a region of interest identifying the location of the abnormality.

6. The method according to claim 2, wherein the step of analyzing the second modality stream using the computer-aided classification module includes applying a second machine learning model to the second modality stream.

7. The method according to claim 2, wherein the first modality is white light, and the method further includes displaying the first modality on a display screen.

8. The method according to claim 7, further including overlaying the results from the computer-aided identification module and the computer-aided classification module on the display screen in parallel.

9. The method according to claim 1, wherein the step of capturing the combined image stream includes capturing at least a 50Hz image stream including a first 25Hz image stream and a second 25Hz image stream arranged alternately with the first 25Hz image stream.

10. The method of claim 9, wherein the step of capturing the combined image stream includes parsing the first 25 Hz image stream as the first modality stream and parsing the second 25 Hz image stream as a second modality.

11. The method of claim 1, wherein the step of capturing the combined image stream includes capturing a first portion of the combined image stream using a first illumination modality and capturing a second portion of the combined image stream using a second illumination modality, the first portion and the second portion being arranged alternately.

12. The method of claim 1, wherein the step of capturing the combined image stream includes synchronizing a color wheel with a sensor frequency of a sensor used to capture the combined image stream, the color wheel including a first set of filters for generating a first imaging modality corresponding to the first modality stream and a second set of filters for generating a second imaging modality corresponding to the second modality stream.

13. The method of claim 1, wherein the step of capturing the combined image stream includes synchronizing a light emitting diode or a laser diode with a sensor frequency of a sensor used to capture the combined image stream, the light emitting diode or the laser diode being configured to generate a first imaging modality corresponding to the first modality stream and a second imaging modality corresponding to the second modality stream.

14. The method of claim 1, further comprising the step of outputting the first modality stream and the second modality stream in parallel to a user interface device.

15. Output of identification information generated by the computer-assisted identification module, and Classification output generated by the computer-assisted classification module The method of claim 2, further comprising the step of outputting in parallel.

16. The method according to claim 15, wherein the step of outputting in parallel includes overlaying at least one of the identification information output and the classification output on at least one of the first modality stream and the second modality stream.

17. The step of capturing the combined image stream includes capturing a full-frame rate image using a primary color sensor; dispersing a plurality of narrow color bands within three main bands generated by the primary color sensor; selecting a first subset of bands from the plurality of narrow color bands to generate the first modality stream; and selecting a second subset of bands from the plurality of narrow color bands to generate the second modality stream. The method according to claim 1.

18. The method according to claim 17, wherein the step of capturing the combined image stream includes selecting a third subset of bands from the plurality of narrow color bands to generate a third modality stream.

19. The step of capturing the combined image stream includes capturing a full-frame rate image using a complementary color sensor; dispersing a plurality of narrow color bands within four main bands generated by the complementary color sensor; selecting a first subset of bands from the plurality of narrow color bands to generate the first modality stream; and selecting a second subset of bands from the plurality of narrow color bands to generate the second modality stream. The method according to claim 1.

20. An endoscope comprising a camera and an illumination system, wherein the camera and the illumination system are configured to generate a first modality stream and a second modality stream; a processor and a memory device, the memory device including instructions that, when executed by the processor, cause the processor to access the first modality team stream and the second modality stream in parallel and perform the method according to any one of claims 1 to 19. A system comprising the above.

21. A computer-readable medium containing instructions that, when executed by a clinical support system, cause the system to perform any one of the methods according to claims 1 to 19.

22. Analyzing the first modality stream and, based on analyzing the second modality stream, selecting a display modality stream from the first modality stream and the second modality stream; Displaying the display modality stream on a display device The method according to any one of claims 1 to 18, further comprising.

23. Outputting a recommended image stream based on analyzing the first modality stream and analyzing the second modality stream The method according to any one of claims 1 to 18, further comprising.

24. Generating a hybrid image that selectively combines the output of the first modality stream and the output of the second modality stream based on analyzing the first modality stream and analyzing the second modality stream The method according to any one of claims 1 to 18, further comprising.

Images