Computer-assisted diagnostic system

Abstract

A system is provided for intelligent monitoring of attention and recognition of computer-aided diagnostic system outputs. The system may comprise an endoscope, a computer-aided diagnostic module, a camera, memory, and a controller. The endoscope may comprise an elongated member that may include a distal portion and a process camera attached to the distal portion. The process camera may capture a video stream during the procedure. The computer-aided diagnostic module may be configured to detect anomalies in the video stream using a diagnostic algorithm and transmit a signal. The controller may be configured to determine the physician's gaze position during the procedure using a gaze algorithm, to determine whether the physician saw the detected anomaly by comparing the signal from the computer-aided diagnostic module with the physician's gaze position, and to trigger an action based on the physician's determination that they did not see the detected anomaly.

Description

Technical Field

[0001] Priority Claim This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 488,564, filed Mar. 6, 2023, which is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to endoscope systems, and more particularly to endoscope systems for monitoring attention and recognizing computer-aided diagnostic system outputs.

Background Art

[0003] Conventional endoscopes can be used in a variety of clinical procedures. For example, an endoscope can be used to elucidate, image, detect, and diagnose one or more medical conditions, provide fluid delivery (e.g., saline or other preparations via a fluid channel) to an anatomical region, provide a passage for one or more treatment devices for sampling or treating an anatomical region (e.g., via a working channel), provide a suction channel for collecting fluid (e.g., saline or other preparations), and the like. Such anatomical regions can include the gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary ducts, intestine, colon, and the like), renal regions (e.g., kidneys, ureters, bladder, urethra), other internal organs (e.g., reproductive organs, sinuses, submucosal regions, respiratory system), and the like.

Summary of the Invention

Means for Solving the Problems

[0004] Various examples are illustrated in the figures of the accompanying drawings. Such examples are illustrative and are not intended to be comprehensive or exclusive examples of the subject matter.

Brief Description of the Drawings

[0005] [Figure 1]This is a schematic diagram of an endoscopic system, as an example of the disclosure. [Figure 2] Figure 1 is a schematic diagram of an imaging and control system connected to an endoscope, illustrating an example of an imaging and control system according to this disclosure. [Figure 3] This is a block diagram illustrating a system as an example of this disclosure. [Figure 4] This is a schematic diagram illustrating a system as an example of this disclosure. [Figure 5] This is a flowchart illustrating a method according to an example of this disclosure. [Figure 6] This flowchart further illustrates the method shown in Figure 5, as an example of this disclosure. [Figure 7] This is a schematic diagram illustrating an example of a display based on the gaze location of a medical professional, as described in this disclosure. [Figure 8] This is a schematic diagram illustrating an example of a display based on the observational placement of medical professionals, as shown in this disclosure. [Figure 9] This flowchart further illustrates the method shown in Figure 5, as an example of this disclosure. [Figure 10] This is a schematic diagram illustrating an example of a display based on the observational placement of medical professionals, as shown in this disclosure. [Figure 11] This is a schematic diagram illustrating an example of a display based on the observational placement of medical professionals, as shown in this disclosure. [Figure 12] This is a schematic diagram illustrating an example of an annotated image that may be displayed on a display in an operating room, according to an example of the present disclosure. [Figure 13] This flowchart further illustrates the method shown in Figure 5, as an example of this disclosure. [Figure 14] This is a block diagram illustrating an example of a machine in which one or more examples may be implemented. [Modes for carrying out the invention]

[0006] A medical examination may involve the use of various medical imaging techniques to capture images of specific parts of a patient's body for visual inspection by a physician. For example, a colonoscopy may involve an endoscopic examination of the patient's colon using a camera positioned at the distal tip of a maneuverable probe. The maneuverable probe may include working channels through which instruments can be passed to access the area being examined. Such an examination can facilitate a visual diagnosis of abnormal tissue (e.g., polyps) and provide an opportunity for biopsy or resection of suspected cancerous tissue.

[0007] Computer-aided detection (CADe) systems or computer-aided diagnosis (CADx) systems (collectively referred to as “CAD systems”) may be deployed in real time during a consultation to enhance both the efficiency and effectiveness of the consultation. CAD systems can assist physicians in interpreting medical images by processing medical images and generating annotations (e.g., in the form of virtual bounding boxes) that highlight abnormal tissues. The inventors of this disclosure have found at least the following problems in current systems, including CAD systems: There is a risk that, even when such annotations are presented, the physician may not examine carefully, or even notice, all abnormal tissues. Furthermore, even when the physician examines individual abnormal tissues and determines whether a procedure (e.g., polyp removal, biopsy, or recording a diagnosis in a laboratory report) is justified, there may be inefficiencies associated with documenting these determined procedures related to individually identified abnormal tissues.

[0008] Generally speaking, this disclosure includes a system that can generate a signal for the operator of a CAD system based on the length of time the operator spends examining individual CAD-identified tissue anomalies. In one example, the CAD system may detect when the operator may have missed a tissue anomaly (e.g., a colon polyp) displayed on the monitor during an examination (e.g., a colonoscopy). Then, when the operator appears to have missed the tissue anomaly, the system can generate a user-perceptible signal that can be designed to draw the operator's attention to the tissue anomaly.

[0009] For example, a system for intelligent monitoring of attention and recognition of computer-aided diagnostic system outputs may include an endoscope, a computer-aided diagnostic module, a camera, memory, and a controller. The endoscope may comprise an elongated member that includes a distal portion and a process camera attached to the distal portion. The process camera may capture a video stream during the procedure. The computer-aided diagnostic module may be configured to detect anomalies in the video stream using a diagnostic algorithm and transmit signals. The camera may capture an operator video stream. The operator video stream may include at least the eyes of the physician during the procedure.

[0010] The memory can store instructions. The controller may include a processing circuit configured, when in operation, to determine the physician's gaze position during the procedure using a gaze algorithm, to determine whether the physician saw a detected anomaly by comparing the physician's gaze position with signals from a computer-aided diagnostic module, and to trigger countermeasures based on the physician's determination that they did not see a detected anomaly. The gaze position can indicate the position on the monitor that the physician is looking at during the procedure.

[0011] Figure 1 is a schematic diagram of an endoscopic system 10 which may include an imaging and control system 12 and an endoscope 14. System 10 is an exemplary example of an endoscopic system suitable for use with the systems, devices, and methods described herein, such as a colonoscopy system for automatic annotation of endoscopic video.

[0012] The endoscope 14 is insertable into an anatomical region for imaging, or can provide a passage or attachment (e.g., via tethering) to one or more sampling devices for biopsy or therapeutic devices for the treatment of a medical condition associated with the anatomical region. The endoscope 14 can interface with and connect to an imaging and control system 12. The endoscope 14 may also include a colonoscope, but other types of endoscopes may also be used with the features and teachings of this disclosure. The imaging and control system 12 may comprise a control unit 16, an output unit 18, an input unit 20, a light source unit 22, a fluid source 24, and a suction pump 26.

[0013] The imaging and control system 12 may have various ports for coupling with the endoscope system 10. For example, the control unit 16 may have data input / output ports for receiving data from and transmitting data to the endoscope 14. The light source unit 22 may include an output port for transmitting light to the endoscope 14, for example, via a fiber optic link. The fluid source 24 may include a port for supplying fluid to the endoscope 14. The fluid source 24 may include, for example, a pump and a fluid tank, or may be connected to an external tank, container, or storage unit. The suction pump 26 may include a port for drawing a vacuum from the endoscope 14 to generate suction, such as for drawing fluid out of the anatomical region into which the endoscope 14 is inserted. The output unit 18 and input unit 20 may be used by the operator of the endoscope system 10 to control the functions of the endoscope system 10 and to view the output of the endoscope 14. The control unit 16 may also generate signals or other outputs from the anatomical region into which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical, acoustic, fluid, and similar outputs for treating anatomical regions using operations such as cauterization, cutting, freezing, and similar methods.

[0014] The endoscope 14 can include an insertion section 28, a function section 30, and a handle section 32, which can be coupled to a cable section 34 and a coupler section 36. The insertion section 28 extends distally from the handle section 32, and the cable section 34 can extend proximally from the handle section 32. The insertion section 28 can be elongated and can include a bending section and a distal end to which the function section 30 can be attached. The bending section can be controllable (e.g., by a control knob 38 on the handle section 32) to maneuver the distal end through a tortuous anatomical passage (e.g., the stomach, duodenum, kidney, ureter, etc.). The insertion section 28 can also be elongated and can include one or more working channels (e.g., lumens) that can support the insertion of one or more treatment instruments of the function section 30, such as a choledochoscope. The working channels can extend between the handle section 32 and the function section 30. Additional functionality, such as fluid passages, guide wires, and pull wires, can also be provided by the insertion section 28 (e.g., via a suction or irrigation passage or the like).

[0015] The coupler section 36 is connected to the control unit 16, thereby enabling the endoscope 14 to be connected to a plurality of features of the control unit 16, such as an input unit 20, a light source unit 22, a fluid source 24, and a suction pump 26.

[0016] The handle section 32 can include a knob 38 and a port 40A. The knob 38 can be connected to a pull wire or other actuating mechanism that can penetrate the insertion section 28. Other ports, such as the port 40A and further the port 40B (FIG. 2), can be configured to couple various electrical cables, guide wires, auxiliary scopes, tissue collection devices, fluid tubes, and the like to the handle section 32, such as for coupling to the insertion section 28.

[0017] According to an example, the imaging and control system 12 can be provided on a mobile platform (e.g., cart 41) having shelves for housing a light source unit 22, a suction pump 26, an image processing unit 42 (FIG. 2), etc. Alternatively, some components of the imaging and control system 12 (shown in FIGS. 1 and 2) can be provided directly on the endoscope 14, thereby making the endoscope "self - contained".

[0018] The functional section 30 can include components for treating and diagnosing a patient's anatomical structure. The functional section 30 can include an imaging device, a lighting device, and an elevator. The functional section 30 can further include devices for collecting and removing optically enhanced biological materials and tissues, as described herein. For example, the functional section 30 can include one or more electrodes conductively connected to the handle section 32 and functionally connected to the imaging and control system 12, whereby biological materials contacting the electrodes can be analyzed based on comparative biological data stored in the imaging and control system 12.

[0019] Figure 2 is a schematic diagram of the endoscopic system 10 of Figure 1, including the imaging and control system 12 and the endoscope 14. Figure 2 schematically illustrates the components of the imaging and control system 12 coupled to the endoscope 14, including a colonoscope in the illustrated example. The imaging and control system 12 may include a control unit 16, which may include or be coupled with an image processing unit 42, a treatment generator 44, and a drive unit 46, as well as a light source unit 22, an input unit 20, and an output unit 18. The control unit 16 may include or be able to communicate with the endoscope, surgical instruments 48, and the endoscopic system, which may include a device configured to engage with tissue, collect and store a portion of that tissue, through which imaging equipment (e.g., a camera) can view target tissue by including optically enhanced materials and components. The control unit 16 may be configured to activate the camera to view target tissue distal to the endoscopic system. Similarly, the control unit 16 is configured to activate the light source unit 22 to illuminate the surgical instrument 48, which may include selected components configured to reflect light in a specific manner, such as an enhanced tissue cutter with reflective particles.

[0020] The coupler section 36 is connected to the control unit 16, thereby allowing connection to the endoscope 14 to several features of the control unit 16, such as the image processing unit 42 and the treatment generator 44. In this example, port 40A may be used to insert another surgical instrument 48 or device, such as a daughter scope or auxiliary scope, into the endoscope 14. Such instruments and devices may be independently connected to the control unit 16 via cable 47. In this example, port 40B may be used to connect the coupler section 36 to various inputs and outputs, such as video, air, light, and electricity.

[0021] The image processing unit 42 and the light source unit 22 can each interface with the endoscope 14 (for example, in the functional section 30) by wired or wireless electrical connections. The imaging and control system 12 can accordingly illuminate an anatomical region, collect signals representing the anatomical region, process the signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 may include a light source unit 22 for illuminating the anatomical region with light of a desired spectrum (e.g., broadband white light, narrowband imaging using preferred electromagnetic wavelengths, and so on). The imaging and control system 12 can be connected to the endoscope 14 (for example, via an endoscope connector) for signal transmission (e.g., light output from the light source, video signals from the imaging system at the distal end, diagnostic and sensor signals from diagnostic devices, and so on).

[0022] A fluid source 24 (shown in Figure 1) can communicate with the control unit 16 and may include one or more sources of air, saline, or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels, or similar) and connectors (barb fittings, fluid seals, valves, or similar). The fluid source 24 may be used as activation energy for the biasing or pressure-applying devices of this disclosure. The imaging and control system 12 may also include a drive unit 46 which may include an electric drive for advancing the distal section of the endoscope 14.

[0023] Figure 3 is a block diagram illustrating a system 300 according to an example of the present disclosure. The system 300 may comprise an endoscope 302, a computer-aided diagnostic module 312, a camera 318, a memory 322, and a controller 336. The elongated member 304 may include a distal portion 306 and a process camera 308 attached to the distal portion 306. The memory 322 may store instructions 324. As best shown in Figure 2, the elongated member 304 (e.g., an insertion section 28 and a functional section 30 (Figures 1 and 2)) may extend from a proximal portion 305 to a distal portion 306. The elongated member 304 may be insertable into the patient's cavity.

[0024] The process camera 308 may be configured to capture a process video stream 310 during a medical procedure. An image processing unit 42 (Figure 2) can process the video stream 310 and display it on a display unit 18 (Figures 1 and 2), so that a physician or other medical professional can see the front of the distal portion 306 of the elongated member 304 during a medical procedure. The camera 308 can also transmit the video stream 310 to multiple components simultaneously. For example, the camera 308 can transmit the video stream 310 to a display unit to provide a live feed of the video stream 310 on a display for the physician, an image processing unit or controller 336 for processing, and memory 322 for storing a raw version of the video stream 310. Any example of the video stream 310 may include a first timestamp 334 to help synchronize the video stream 310 with other signals in the system 300.

[0025] As will be described in more detail below with reference to Figure 4, the computer-aided diagnostic module 312 may be configured to use a diagnostic algorithm to detect anomalies or abnormalities (detected anomalies 314) in the process video stream 310 and transmit signals 316. In one example, the computer-aided diagnostic module 312 may be a module that communicates with a controller 336, which can automatically perform detection of anomalies 314 and diagnose the video stream 310 during the performance of a medical procedure. In another example, the computer-aided diagnostic module 312 may be another controller of the system 300. As previously described, the computer-aided diagnostic module 312 may be deployed in real time during a consultation to enhance the efficiency and effectiveness of the consultation. The computer-aided diagnostic module 312 can assist physicians in interpreting medical images by processing medical images and generating annotations (e.g., in the form of virtual bounding boxes) that highlight abnormal tissue.

[0026] Camera 318 may be one or more cameras mounted around System 300 to capture at least a physician performing a medical procedure. For example, Camera 318 may be configured to capture the eyes of a physician performing a medical procedure. Camera 318 can capture an operator video stream 320. The operator video stream 320 may be a video stream showing a physician completing a medical procedure. The operator video stream 320 may be used for training purposes and for processing purposes. As such, Camera 318 can transmit the operator video stream 320 to several components of System 300. For example, Camera 318 can transmit the operator video stream 320 to a display unit to display a live stream of the operator video stream 320. Camera 318 can also transmit the operator video stream 320 to a video processor, which can then analyze the operator video stream 320. In yet another example, camera 318 can transmit the operator video stream 320 to controller 336 or to any other component of system 300. The operator video stream 320 may include a second timestamp 338 to help synchronize the operator video stream 320 with the video stream 310 of system 300 or any other signal.

[0027] Memory 322 may be main memory, static memory, or a mass storage device (for example, main memory 1404, static memory 1406, or mass storage device 1408, as described with reference to Figure 14). Memory 322 may store instructions 324, which may include programs, processes, or other operations that can configure controller 336 to complete tasks that assist system 300 in completing attention monitoring and recognition of computer-aided diagnostic system outputs.

[0028] The controller 336 (for example, control unit 16) may be one or more controllers configured to operate the system 300. Instructions 324 on memory 322 can cause the processing circuit of the controller 336 to perform an operation or complete a procedure. For example, the processing circuit of the controller 336 may be configured, in response to instruction 324, to determine the physician's gaze position during an endoscopic procedure using a gaze algorithm, to determine whether the physician saw a detected anomaly by comparing the physician's gaze position with signals from a computer-aided diagnostic module, and to trigger an action based on the determination that the physician did not see a detected anomaly.

[0029] For example, a decision that an operator may have overlooked a tissue anomaly may include analyzing medical images used with the CAD system to identify the tissue anomaly, assigning an anomaly ID to the tissue anomaly (e.g., a unique identifier specifically corresponding to an individual tissue anomaly), monitoring the operator's gaze to determine the gaze duration associated with the anomaly ID (e.g., the length of time the operator focused on the tissue anomaly on the monitor), and comparing the gaze duration to one or more gaze duration thresholds. In some examples, a gaze duration lower than a gaze duration threshold may indicate that the operator overlooked a tissue anomaly. System 300 is described in more detail herein with reference to Figures 4 to 14.

[0030] Figure 4 is a schematic diagram illustrating a system according to an example of the present disclosure. System 400 may be an example of one implementation of system 300 from Figure 3. Systems 300 and 400 may be described together as a system in this specification. In addition to the components of system 300, system 400 may include a user interface module 410 and a display 412. System 400 may also include a CAD algorithm 402, an endoscopy physician detection algorithm 404, a gaze placement detection algorithm 406, and a comparison algorithm 408 stored as an instruction 324 (Figure 3) on memory 322 (Figure 3) which can be executed by controller 336 to complete a task.

[0031] The CAD algorithm 402 can be initiated by the computer-aided diagnostic module 312, the controller 336, or any other controller or processor of the system or communicating with the system. The CAD algorithm 402 may be configured to use and analyze the video stream 310 (Figure 3) from the process camera 308 (Figure 3) to find detected anomalies 314 (Figure 3), which may be, for example, polyps, abnormalities, diseases, other undesirable features, or similar in the patient's body.

[0032] In the example, the CAD algorithm 402 can determine the location of the detected anomaly 314 and label the location of the detected anomaly 314. For example, the CAD algorithm 402 can store the location of the anomaly 420 on the video stream 310, which may be stored on memory 322 (Figure 3), transmitted to any other component of the system, or shared, stored, or transmitted to a device in the cloud. Furthermore, the CAD algorithm 402 can label the detected anomaly 314 with a unique identifier that makes it easier to identify the detected anomaly 314 in future procedure executions and during pathological examination of the detected anomaly 314. The identification of the detected anomaly 314 may also be stored along with the location of the anomaly 420 so that any system receiving the location of the anomaly 420 also has a unique identifier for the detected anomaly 314.

[0033] In the example, the CAD algorithm 402 can place a bounding box (further described in Figure 7) around the detected anomaly 314, creating a visual marker of the detected anomaly 314 on the analyzed video stream 414. In the example, the bounding box may also include a unique identification of the detected anomaly 314 such that a unique identification may be displayed or indicated around the bounding box. The CAD algorithm 402 can instruct a computer-aided diagnostic module 312, a controller 336, or another processor to transmit the analyzed video stream 414 to a user interface module 410 for further processing, as described below.

[0034] The endoscopist detection algorithm 404 can be initiated by the controller 336, or any other processor in system 300, system 400, or any other processor communicating with system 300, system 400. The endoscopist detection algorithm 404 can be configured to receive an operator video stream 320 (Figure 3) from camera 318 (Figure 3) and analyze the operator video stream 320 to determine the location of the endoscopist 424. The location of the endoscopist 424 may be the location of the person holding the endoscope. The endoscopist detection algorithm 404 can be trained via machine learning techniques to find the location of the endoscopist 424 by learning the physical appearance of all physicians using the system. In another example, the endoscopist detection algorithm 404 can find the endoscope and determine the person holding it.

[0035] In the example, after the endoscopist detection algorithm 404 has determined the operator of the endoscope, the endoscopist detection algorithm 404 may instruct the processor to focus the camera 318 on the endoscopist. In particular, the endoscopist detection algorithm 404 may instruct the processor to focus the camera 318 on the endoscopist's face to ensure that the operator's eyes are within the frame of the operator video stream 320.

[0036] The accuracy of the endoscopist detection algorithm 404 can help prevent false alarms in the system. For example, the endoscopist detection algorithm 404 can ensure that the system is looking at the eyes of the medical professional performing the medical procedure, and not a nurse, medical assistant, or any other person in the operating room during the medical procedure.

[0037] In the example, after the endoscopist detection algorithm 404 finds the position of the endoscopist operator 424, the endoscopist detection algorithm 404 can determine whether the endoscopist operator's eyes are within the operator video stream 320. If the operator's eyes are not within the operator video stream 320, the endoscopist detection algorithm 404 can generate an alarm, signal, or warning prompting the endoscopist to move to a new position, the camera 318 to move, or an object obstructing the camera 318's view to move.

[0038] The gaze placement detection algorithm 406 may be executed by the controller 336, or any other processor in system 300, system 400, or any other processor communicating with system 300, system 400. For example, the gaze placement detection algorithm 406 may be executed after the endoscopist detection algorithm 404 has found the placement of the endoscopist operator 424. The gaze placement detection algorithm 406 may be configured to receive the operator video stream 320 (Figure 3) from the camera 318 (Figure 3), analyze the operator video stream 320, and determine the gaze placement 426 (Figure 3) of the medical professional performing the medical procedure. For example, the endoscopist detection algorithm 404 may be used to find an attention area 422, which may be the placement on the display 412 that the eyes of the medical professional performing the medical procedure are looking at during the performance of the medical procedure.

[0039] In another example, the endoscopist may wear glasses, a headset, or any other sensors that can help the system determine the endoscopist's gaze direction. For example, the endoscopist may wear glasses that may have a camera 318 mounted on them. As such, the operator video stream 320 may then be stored and used for further machine learning of the CAD algorithm 402, or for any other algorithm or system that can benefit from seeing where the physician is looking during the performance of the medical procedure. Furthermore, when the physician performing the medical procedure is wearing glasses, the endoscopist detection algorithm 404 may be used to find the position of the physician performing the medical procedure relative to the display 412. For example, the glasses may have sensors that can detect position, orientation, distance, or acceleration, thereby helping to determine the position of the physician performing the medical procedure relative to the display 412.

[0040] The glasses may incorporate augmented reality (AR) or virtual reality (VR) technology that allows any of the system's video streams to be displayed on the lenses of the glasses. Furthermore, the glasses may transmit warnings, signals, or other visual cues to a physician performing a medical procedure. As such, the glasses may have a controller that communicates with either the system's controller or processor.

[0041] The comparison algorithm 408 may be executed by the computer-aided diagnostic module 312, the controller 336, or any other controller or processor of the system or communicating with the system. The comparison algorithm 408 may be configured to compare one or more signals, characteristics, or other parameters generated, captured, or detected by components of the system. For example, the comparison algorithm 408 may be configured to compare the location of an anomaly 420 with the attention area 422. For example, the location of the anomaly 420 and the attention area 422 may be compared with the location of the anomaly 420 and the attention area 422 being displayed on the display 412.

[0042] In one example, if the location of the anomaly 420 and the attention area 422 do not overlap for the shortest possible time (e.g., threshold time) while the detected anomaly 314 is displayed on the display 412, the comparison algorithm 408 may generate a signal, warning, or other instruction indicating that the detected anomaly 314 may have been missed by a medical professional performing a medical procedure. In the example, the comparison algorithm 408 may generate an audible warning signal, a visual warning signal, a still image showing the unrecognized disease area, or similar to generate a warning video stream 428. The comparison algorithm 408 will be described in more detail with reference to Figures 5–11.

[0043] The user interface module 410 can receive video stream 310, operator video stream 320, analyzed video stream 414, and warning video stream 428 from the comparison algorithm 408. In an example, the user interface module 410 can combine, modify, or edit video stream 310, analyzed video stream 414, or warning video stream 428. For example, the user interface module 410 can overlay the warning video stream 428 onto video stream 310 to generate an overlaid video stream 430. The user interface module 410 can overlay video stream 310 with the analyzed video stream 414 to generate a detected video stream 432. Furthermore, the user interface module 410 can overlay the detected video stream 432 with the warning video stream 428 to generate a detected warning video stream 434.

[0044] The detected warning video stream 434 may include both the detected anomaly 314 from the CAD algorithm 402 and the warning video stream 428, which may contain the warning generated by the comparison algorithm 408. In such an example, the user interface module 410 can continuously update the detected warning video stream 434 when it receives the video stream 310, the analyzed video stream 414, or the detected warning video stream 434.

[0045] The user interface module 410 can transmit any combination of video stream 310, operator video stream 320, analyzed video stream 414, or detected warning video stream 434 to the display 412, operator video stream 320, or any other component of the system or any other component communicating with the system. For example, the user interface module 410 can transmit and store video stream 310, operator video stream 320, analyzed video stream 414, or detected warning video stream 434 in memory 322.

[0046] The display 412 (for example, output unit 18 (Figure 1) or display unit 18 (Figure 2)) may be placed in the room where the medical procedure is being performed. The display can communicate with the user interface module 410 or any other component of the system to communicate with people in the room during the medical procedure.

[0047] Figure 5 is a flowchart illustrating Method 500 according to an example of the present disclosure. Method 500 enables a system (for example, system 300 from Figure 3 or system 400 from Figure 4) to complete attention monitoring and recognition of computer-aided diagnostic system outputs. Method 500 includes operations that trigger countermeasures when the system detects an anomaly that was not examined by an endoscopist. In the example, Method 500 may include any of steps 510 to 560.

[0048] In step 510, method 500 may include using the controller's processing circuitry to receive a video stream captured during the endoscopic procedure from a process camera mounted on the endoscope. For example, the controller 336's processing circuitry may receive a video stream 310 from a process camera 308. As described herein, the process camera 308 may be mounted on the endoscope 302. The process camera 308 may be mounted on the elongated member 304 of the endoscope 302. The video stream 310 may be captured during the endoscopic procedure. The controller 336 may simultaneously transmit the video stream 310 to multiple components of the system 300. For example, the controller 336 may send the video stream 310 to one or more of the following: memory 322 for storage, computer-aided diagnostic module 312 for processing, in-room display (e.g., display 412 in Figure 4), or any other components of the system for further storage or processing.

[0049] In step 520, method 500 may include receiving a signal from a computer-aided diagnostic module, the signal indicating an anomaly detected in the video stream by a diagnostic algorithm. For example, method 500 may include the controller 336 receiving a signal 316 from the computer-aided diagnostic module 312, or any other component of the system or any other component communicating with the system. The signal 316 may include one or more detected anomalies 314 detected by the computer-aided diagnostic module 312 or the controller 336 executing the CAD algorithm 402. In this example, the computer-aided diagnostic module 312, the controller 336, or any other processor of the system or any other processor communicating with the system may execute the diagnostic algorithm 402, which may be stored on the computer-aided diagnostic module 312, on memory 322, or directly on any other component of the system or any other component communicating with the system.

[0050] In step 530, method 500 may include receiving an operator video stream from a camera installed in the room during the endoscopic procedure, the operator video stream may include at least the eyes of the physician who completed the endoscopic procedure. For example, method 500 may include the controller 336 receiving an operator video stream 320 from a camera 318 installed in the room during the endoscopic procedure. The operator video stream 320 may include at least the eyes of the physician who completed the endoscopic procedure. The controller 336 can simultaneously transmit the operator video stream 320 to various components of the system 300. For example, the controller 336 can transmit the operator video stream 320 to memory 322 for storage, to the user interface module 410 (Figure 4), or to any other component of the system for processing.

[0051] In step 540, method 500 may include determining the physician's gaze position while performing an endoscopic procedure using a gaze algorithm, the gaze position may indicate a position on a monitor that the physician views while performing the endoscopic procedure. For example, the controller 336 may, in operation, be configured by command 324 to determine the physician's gaze position 426 during the procedure using a gaze algorithm (e.g., gaze position detection algorithm 406 in Figure 4). The gaze position 426 may indicate a position 342 on a display 412 (e.g., display unit 18 or any other monitor or display that the physician views while performing the procedure).

[0052] In step 550, method 500 may include determining whether the physician saw the detected anomaly by comparing the signal from the computer-aided diagnostic module with the physician's gaze position. For example, controller 336 may determine whether the physician saw the detected anomaly 314 by comparing the signal 316 from computer-aided diagnostic module 312 with the physician's gaze position 426. For example, the comparison algorithm 408 (Figure 4) may be executed by controller 336, computer-aided diagnostic module 312, or any other processor in the system, or any other processor communicating with the system.

[0053] In step 560, method 500 may include triggering an action based on the physician's determination that they did not see the detected anomaly. For example, controller 336 may trigger action 350 based on the physician's determination that they did not see the detected anomaly 314 over a threshold time. Triggering action 350 may include transmitting an alert signal 352. The alert signal may indicate that the physician's attention placement 426 was not directed to the detected anomaly over a first set threshold time. In one example, the alert signal may include an anomaly identification specific to each anomaly detected by the computer-aided diagnostic module.

[0054] Figure 6 is a flowchart illustrating a further example of the method from Figure 5, according to an example of the present disclosure. In one example, when triggering the countermeasure 350 in step 560, method 500 may optionally include steps 610-630.

[0055] In step 610, step 560 of method 500 may optionally include generating a perceptible signal 360. The perceptible signal 360 may be configured to notify a physician of a detected anomaly (e.g., detected anomaly 314). The perceptible signal 360 may be an acoustic signal, a tactile signal, a visual signal, any other type of signal to attract the attention of a physician performing a medical procedure, or any combination thereof, as described above.

[0056] In step 620, step 560 of method 500 may optionally include overlaying the video stream with a bounding box by associating the arrangement of anomalies detected from a computer-aided diagnostic module with the arrangement of anomalies detected on the video stream, thereby creating an overlaid video stream. For example, a controller 336 or an image processor (e.g., an image processing unit 42 or a user interface module 410 (Figure 4)) may overlay the video stream 310 with a bounding box 362 by associating the arrangement of anomalies 314 detected from a computer-aided diagnostic module 312 with the arrangement of anomalies 314 detected on the video stream 310, thereby creating an overlaid video stream 366.

[0057] In step 630, method 500 may include displaying an overlaid video stream 366 to show a physician or other medical professional the arrangement 342 of the detected anomaly 314.

[0058] In one example, the perceptible signal 360, or user-perceptible signal 360, may include (i) adjusting bounding box parameters associated with a tissue anomaly to make the bounding box appear or more prominent, (ii) generating an acoustic or tactile feedback signal, or (ii) displaying a still image of a tissue anomaly that the operator might have overlooked, or similar. The system can direct the attention of the medical professional performing the procedure to individual tissue anomalies that have not received sufficient attention duration.

[0059] Figure 7 is a schematic diagram illustrating an exemplary display based on the gaze position of a healthcare professional, according to an example of this disclosure. Figure 8 is a schematic diagram illustrating an exemplary display based on the gaze position of a healthcare professional, according to an example of this disclosure. Figures 7 and 8 are described together below.

[0060] In one example, the bounding box 702 may be selectively modulated (e.g., change color, flash) in response to individual tissue anomalies. For example, the bounding box 702 may start with a first color, and if the medical professional completing the medical procedure does not see the detected anomaly within the bounding box 702 over a threshold time, the bounding box 702 may change to a second color, begin flashing on the screen, increase in size, or perform any other operation of the bounding box 702, or similar operations, which can help attract the attention of the medical professional completing the medical procedure. Such an implementation can prevent the medical professional from inadvertently missing CAD-identified or classified tissue anomalies.

[0061] As shown in Figure 7, the CAD system can identify or diagnose three tissue abnormalities 314A, 314B, and 314C in the medical image 704. As described above, the bounding boxes 702 may initially be displayed in a first color, for example, green. For example, bounding box 702A may enclose tissue abnormality 314A, bounding box 702B may enclose abnormality 314B, and bounding box 702C may enclose tissue abnormality 314C.

[0062] With respect to the purpose of the example illustrated in Figure 7, assume that the gaze duration of the medical professional completing the medical procedure for each of anomalies 314A and 314B exceeds the gaze duration threshold 708, and that the gaze duration of the medical professional completing the medical procedure does not gaze at anomaly 314C for at least the gaze duration threshold. Here, the bounding boxes 702A and 702B surrounding tissue anomalies 314A and 314B, respectively, may be a first color, for example, green, and the bounding box 702C surrounding tissue anomaly 314C may be a second color, for example, red. In another example, a pattern, shape, other visual marker, or similar may be used to indicate whether the gaze duration of the medical professional completing the medical procedure exceeds or does not exceed the gaze duration threshold. As such, the CAD system may selectively modulate the bounding box 702C of anomaly 314C to draw the attention of the medical professional completing the medical procedure to it.

[0063] In some cases, the CAD system may have system parameters that prevent the bounding box 702 from modulating until certain conditions are met, in addition to the gaze duration not meeting the gaze duration threshold. For example, if the operator is still intently focused on anomaly 314A or 314B, the CAD system cannot modulate the bounding box 702C around anomaly 314C. Then, if the operator stops focusing on anomaly 314A or 314B and begins scanning the screen, looking around the room, or moving the scope so that anomaly 314C is out of the field of view, the bounding box 702C around anomaly 314C may be updated.

[0064] In the example shown in Figure 8, the CAD system can display a bounding box 702 around or near the detected anomaly only if the anomaly is displayed on the screen for a set duration of time and the medical professional completing the medical procedure is not gazing at the detected anomaly for a threshold duration (e.g., gaze duration threshold 708). Such an implementation can mitigate the negative operator experience resulting from the bounding box 702 being placed around a false positive. For example, some gastroenterologists have pointed out that having the bounding box placed over a false positive (i.e., a tissue area incorrectly identified by the CADe / CADx system as being of interest) can be distracting and cumbersome during examination.

[0065] For example, as shown in Figure 8, the CAD system can identify or diagnose anomalies 314A, 314B, and 314C in the medical image 704. The display parameters of the CAD system can refrain from displaying any bounding boxes (or other information associated with them) around individual tissue anomalies among the tissue anomalies unless the gaze duration associated with such tissue anomalies does not meet the gaze duration threshold. For example, suppose the gaze duration for each of anomalies 314A and 314B exceeds the gaze duration threshold, while it is estimated that the operator has not gazed at anomaly 314C for at least the gaze duration threshold. As such, the CAD system will not place bounding boxes 702A or 702B around either anomaly 314A or 314B, respectively, but will place a bounding box 702C around anomaly 314C.

[0066] The gaze duration threshold 708 can be adjusted via system parameters. For example, the gaze duration threshold 708 can be adjusted based on the type of medical procedure the system may be used in. The gaze duration threshold 708 can vary from 50 milliseconds to 5 seconds. For example, the gaze duration threshold 708 can vary from 100 milliseconds to 1 second. In yet another example, the gaze duration threshold 708 can be any length of time that can be used to determine whether a physician or other medical professional saw the detected anomaly or may have had difficulty identifying the detected anomaly.

[0067] In another example, the system may include one or more gaze duration thresholds 708, and each of these thresholds may trigger the system to output or generate a signal for a different action. For example, if a healthcare professional observes an anomaly detected across a gaze duration threshold, that anomaly may be labeled as complex and flagged for review by either the healthcare professional completing the medical procedure or a third-party healthcare professional. In another example, there may be a minimum value for the gaze duration threshold 708 indicating different levels of review for detected anomalies. In yet another example, the gaze duration threshold 708 may be adjusted according to the preferences of the healthcare professional performing the medical procedure.

[0068] In yet another example, the gaze duration threshold (e.g., gaze duration threshold 708) can be modified based on the identification of the CAD system. For example, the threshold for an identified tissue abnormality can be determined based on classification data output from the CADx system. For instance, a colon polyp analyzed by the CADx system and classified as an inflammatory colon polyp may be assigned a first gaze duration threshold, while another colon polyp analyzed and classified as a chorioadenoma may be assigned a second gaze duration that is longer than the first. Such examples can ensure that sufficient operator attention is given to tissue abnormalities that are more likely to be harmful to the patient.

[0069] Figure 9 is a flowchart further illustrating Method 500 from Figure 5, in an example of the present disclosure. For example, in addition to generating a perceptible signal from step 610 from Figure 6, Method 500 may optionally also include any of steps 910-930.

[0070] In step 910, step 610 of method 500 may also include determining that the physician's gaze position did not correspond to the position of the detected anomaly while the detected anomaly was displayed on the monitor. For example, step 610 of method 500 may also include determining that the physician's gaze position 426 did not correspond to the position 342 of the detected anomaly 314 while the detected anomaly 314 was displayed on the monitor (e.g., display unit 18, display 412, or similar). For example, the endoscopist can move the process camera so that the detected anomaly 314 is no longer shown on the display.

[0071] In step 920, step 610 of method 500 may optionally include creating a missed anomaly video stream 902 by overlaying the video stream with a visual graphic indicating that the physician missed a detected anomaly that is no longer visible on the monitor. For example, in step 920, step 610 of method 500 may include overlaying the video stream 310 with a visual graphic 904. The visual graphic 904 may be an object, text, or any other indicator that may appear on the display to gain the attention of a healthcare professional performing a medical procedure. In this example, the visual graphic 904 may be accompanied by secondary alerts, such as audible, tactile, or any other alerts that may accompany the visual graphic 904 to gain the attention of a healthcare professional performing a medical procedure. The visual graphic 904 may indicate that the physician, or any other healthcare professional performing a medical procedure, missed a detected anomaly 314 that is no longer visible on the monitor, even if an attempt is made to create a missed anomaly video stream 902.

[0072] In step 930, step 610 of method 500 may optionally include displaying a missed anomaly video stream to alert a physician to a detected anomaly that is no longer displayed on the monitor. For example, step 610 of method 500 may optionally include displaying a missed anomaly video stream 902 to alert a physician to a detected anomaly 314 that is no longer displayed on the monitor. In this example, the controller 336 (Figure 3) or the computer-aided diagnostic module 312 (Figure 3) may transmit the missed anomaly video stream 902 to the user interface module 410 (Figure 4), which may overlay the missed anomaly video stream 902 on the video stream 310. In another example, the controller 336, the computer-aided diagnostic module 312, or any other processor in the system or any other processor communicating with the system may transmit the missed anomaly video stream 902 to any other component in the system or any other component communicating with the system for storage, classification, or use as training data for future machine learning iterations.

[0073] Figure 10 is a schematic diagram illustrating an exemplary display based on the gaze positioning of medical professionals, according to an example of this disclosure. Figure 11 is a schematic diagram illustrating an exemplary display based on the gaze positioning of medical professionals, according to an example of this disclosure. Figures 10 and 11 will be described together below.

[0074] In one example, an acoustic or visual signal (e.g., a visual graphic 904) may instruct a medical professional to return the attention of a medical professional to complete a medical procedure to a detected anomaly 314 that was seen but not sufficiently examined before moving outside the displayed area (e.g., of a video stream 310 displayed on a display unit 18 or display 412).

[0075] As shown in Figures 10 and 11, a CAD system (e.g., system 300 or system 400) can identify or diagnose anomalies 314A, 314B, and 314C in the medical image 704. The operator may be able to focus on each of anomalies 314A and 314B for at least their respective gaze duration thresholds (e.g., gaze duration threshold 708), but may fail to see anomaly 314C for the respective gaze duration thresholds before adjusting the endoscope so that anomaly 314C is no longer visible on the monitor or display (e.g., display 412). In response to this decision, the computer-aided diagnostic module 312, the controller 336, or any other processor connected to the system may generate a signal (e.g., a visual graphic 904) designed to draw the operator's attention to anomaly 314C.

[0076] As shown in Figure 10, an anomaly 314C may move outside the displayed area, and the computer-assisted diagnostic module 312, the controller 336, or any other processor communicating with the system may generate a visual graphic 904 to draw the endoscopist's attention back in the direction the detected anomaly 314 is off-screen. As shown in Figure 10, the visual graphic 904 may include a warning label and a directional arrow. In another example, the warning label may include a unique identifier for the missed anomaly, which can inform the healthcare professional of the unique identification, classification, and any other information that the detected anomaly 314C generates. In the example, any other visual markers may be used to alert the endoscopist to the missed anomaly.

[0077] As shown in Figure 11, the anomaly 314C may reappear on the screen when the endoscopist moves the camera towards the anomaly 314C after the visual graphic 904 has drawn the endoscopist's attention to it. In one such implementation, the bounding box (e.g., bounding box 702 (Figure 7)) may change from red to green after the operator's gaze duration on the anomaly 314C reaches an appropriate gaze duration threshold (e.g., gaze duration threshold 708).

[0078] Figure 12 illustrates a schematic diagram of an example of an annotated image 1200 that may be optionally displayed on a display in the operating room. The annotated image 1200 may be, for example, any of the annotated images described herein and may include image 1210, annotation 1220, marking box 1230, polyp identification box 1240, and process identification box 1250. Furthermore, the annotated image 1200 may be a symbolic representation of any video stream described herein. For example, as described above, any combination of overlaid video stream 366, analyzed video stream 414, warning video stream 428, overlaid video stream 430, detected video stream 432, detected warning video stream 434, or missed anomaly video stream 902 may be combined on or overlaid on video stream 310 by the user interface module 410 or any other controller or processor communicating with the system.

[0079] Image 1210 may be a continuous feed from a video stream captured by a camera during an endoscopic procedure. Image 1210 may be from timestamps corresponding to timestamps of indicators of abnormalities found by a computer-aided diagnostic system (e.g., computer-aided diagnostic module 312 (Figure 3)). In another example, the system may automatically send still image examples or aggregates of Image 1210 to the physician for review immediately after a medical procedure. For example, Image 1210 may be sent for review if the physician has not looked at the detected abnormality for longer than a threshold time, indicating that the physician may have looked at the abnormality for too long and therefore become confused or struggled to identify it, or for other reasons programmed by the system.

[0080] Annotation 1220 may be placed on image 1210, as shown in Figure 12. In another example, annotation 1220 may be shifted laterally to image 1210, for example, in a polyp identification box 1240, a process identification box 1250, or in any area around image 1210. Annotation 1220 may be a unique identifier generated for an anomaly. Annotation 1220 can help identify the location of an anomaly encountered during the performance of a medical procedure.

[0081] A marking box 1230 (for example, a bounding box 702 (Figure 7)) may be overlaid on the image 1210 to help identify any anomalies found. For example, the marking box 1230 could help a physician performing a procedure find an anomaly detected by a CAD system, or it could help direct the physician to an anomaly that may have been missed during a medical procedure. In another example, the marking box 1230 could help a machine learning algorithm focus on anomalies to improve the quality of learning.

[0082] The polyp identification box 1240 may include information about anomalies from a medical procedure or from a physician's review after a medical procedure. For example, the polyp identification box 1240 may include annotations of utterances made by a medical professional before and after the timestamp of a spoken keyword. In another example, the polyp identification box 1240 may include notes entered by a physician after the physician has reviewed the annotated image 1200. The information provided to the polyp identification box 1240 helps to improve machine learning by providing additional information about the annotated image 1200, which can help to improve the information provided for machine learning by sorting the annotated image 1200 into groups of similar findings.

[0083] The process identification box 1250 may include process information relating to a medical procedure. For example, the process identification box 1250 may include a timestamp of the video stream in which the image was captured, a timestamp when an anomaly was recognized, a confidence level or polyp identification, and other processing information of the medical procedure that may be useful to know after the procedure is completed. The process identification box 1250 may also include manufacturing information or model number of the equipment used to perform the medical procedure.

[0084] The example of an annotated image 1200 shown in Figure 12 is merely one example of an annotated image 1200. This example, including the information presented therein, is not intended in any way to limit the scope of the present invention. Rather, the information provided is intended to be an example of an annotated image 1200 that the system described herein can generate.

[0085] Figure 13 is a flowchart illustrating the method 500 from Figure 5, using an example from the present disclosure. For example, the method 500 may optionally include steps 1302-1316 for generating a confirmed anomaly report 1320.

[0086] In step 1302, method 500 may include recognizing a physician who will complete the endoscopic procedure using a related person algorithm (e.g., endoscopist detection algorithm 404 (Figure 4)). The related person algorithm may be configured to identify the physician holding the endoscope in order to confirm that the gaze position is the gaze position of the physician performing the endoscopic procedure.

[0087] In step 1304, method 500 may include measuring the possible duration of gaze placement at each placement on the monitor. As described above, the system may use an operator video stream, glasses, or any other sensor to determine a physician's gaze placement and measure the length of time the physician's gaze coincides with a placement on the display where an anomaly has been detected. In an example, the system may determine that a medical professional gazed at a placement on the display over a threshold value. If that placement on the display does not have a pre-detected anomaly, i.e., an anoloy, that placement on the screen may be flagged for future review. For example, this may indicate a false negative, e.g., an anomaly that the medical professional found but the CAD system did not find in the video stream.

[0088] In step 1306, method 500 may include labeling the detected anomaly with a complex anomaly label, which indicates that the time for which a gaze placement may be directed towards the detected anomaly may exceed a third threshold 1322. For example, if a physician is looking at an anomaly for a time exceeding the third threshold 1322, the anomaly may be automatically labeled as a complex anomaly 1326, as the extended length of time the physician's gaze is placed on the anomaly suggests that the physician may be having difficulty identifying or classifying the detected anomaly.

[0089] In step 1308, method 500 may include labeling the detected anomaly with a review anomaly label 1328, which indicates that the time for which a gaze placement may be directed to the detected anomaly may be shorter than a fourth threshold time 1324. For example, the fourth threshold time 1324 may be longer than a gaze duration threshold 708, but still a threshold number that raises concerns about whether the physician has fully analyzed the anomaly.

[0090] In step 1310, method 500 may include extracting detected anomalies having complex anomaly labels 1326 and review anomaly labels 1328 to generate an anomaly review report 1330.

[0091] In step 1312, method 500 may include transmitting the abnormal review report 1330 to one or more physicians for review. For example, the abnormal review report 1330 may be sent for review to the physician who completed the medical procedure, to one of their colleagues or supervisors, or to an unbiased third party.

[0092] In step 1314, method 500 may include receiving reviewed anomaly review reports 1332 from one or more physicians in order to generate a confirmed anomaly report 1334, the confirmed anomaly report 1334 indicating that one or more physicians have confirmed the detected anomaly.

[0093] In step 1316, method 500 may include storing the confirmed anomaly report 1334 in a database. In another example, the confirmed anomaly report 1334 may be transmitted to a convolutional neural network for training machine learning. In yet another example, the confirmed anomaly report 1334 may be transmitted to the patient's medical file, which may remain with the patient as the patient ages. In yet another example, the confirmed anomaly report 1334 may be stored and later combined with pathological results of anomalies or polyps removed during the performance of a medical procedure.

[0094] In some cases, gaze duration thresholds (e.g., gaze duration threshold 708, third threshold 1322, or fourth threshold time 1324) can be modified based on the identification of the CAD system. For example, thresholds for identified tissue abnormalities can be determined based on classification data output from the CADx system. For instance, a colon polyp analyzed by the CADx system and classified as an inflammatory colon polyp may be assigned a first gaze duration threshold, while another colon polyp analyzed and classified as a chorioadenoma may be assigned a second gaze duration that is longer than the first. Such examples can ensure that sufficient operator attention is given to tissue abnormalities that are more likely to be harmful to the patient.

[0095] In the example, images of the placement that the physician gazed upon at the threshold time, even if they did not have the corresponding detected abnormality or abnormality, may also be saved. For example, these images may be saved and reviewed. If an abnormality or abnormality is confirmed, these images may be used to further train the CAD system and reduce false negatives in future medical procedures.

[0096] Figure 14 is a block diagram illustrating an example of a machine in which one or more examples may be implemented. The example may include, or be operated by, logic or a number of components or mechanisms within the machine 1400, as described herein. A circuit (e.g., a processing circuit) is a collection of circuits implemented in a tangible entity of the machine 1400, including hardware (e.g., simple circuits, gates, logic, etc.). Member relationships between circuits can change flexibly over time. A circuit includes members that can perform a specified operation, either individually or in combination when operating. In one example, the hardware of a circuit can be designed immutably to perform a particular operation (e.g., hardwired). In one example, the hardware of a circuit may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) that include a machine-readable medium that is physically modified to encode instructions for a particular operation (e.g., a movable arrangement of magnetic, electrical, or invariant mass particles, etc.). When connecting physical components, the basic electrical properties of the hardware components are changed, for example, from insulator to conductor, or from conductor to insulator. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of a circuit in the hardware via variable connections to perform a specific part of an operation during operation. Thus, in one example, a machine-readable medium element is either part of a circuit or communicatively coupled to other components of a circuit when the device is operating. In one example, any of the physical components can be used as multiple members of multiple circuits. For example, during operation, an execution unit can be used at one point in a first circuit of a first set of circuits, and at a different point in time to be reused by a second circuit within the first set of circuits, or a third circuit in a second set of circuits. An example of additional these components relating to machine 1400 is shown below.

[0097] In alternative examples, machine 1400 can operate as a standalone device or be connected to other machines (e.g., network-connected). In a network-connected deployment, machine 1400 can operate as a server machine, a client machine, or both in a server-client network environment. In one example, machine 1400 can operate as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 1400 can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term “machine” or “collection of machines” shall be interpreted to include machines or collections of machines that individually or in conjunction execute a set of instructions (or more instruction sets) to perform one or more of the methods described herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0098] The machine (e.g., a computer system) 1400 may include a hardware processor 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 1404, static memory (e.g., memory or storage for firmware, microcode, basic input / output (BIOS), Unified Extensible Firmware Interface (UEFI), etc.) 1406, and mass storage device 1408 (e.g., a hard drive, tape drive, flash storage, or other block device), some or all of which can communicate with each other via an interlink (e.g., a bus) 1430. The machine 1400 may further include a display unit 1410, an alphanumeric input device 1412 (e.g., a keyboard), and a user interface (UI) navigation device 1414 (e.g., a mouse). In one example, the display unit 1410, the input device 1412, and the UI navigation device 1414 may be touchscreen displays. The machine 1400 may further include a storage device (e.g., a drive unit) 1408, a signal generating device 1418 (e.g., a speaker), a network interface device 1420, and one or more sensors 1416 such as a Global Positioning System (GPS) sensor, compass, accelerometer, or other sensor. The machine 1400 may also include an output controller 1428 such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near-field communication (NFC)) connection for communicating with or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0099] The registers of processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 may be, or include, a machine-readable medium 1422 on which one or more sets of data structures or instructions 1424 (e.g., software) that are embodied or utilized by one or more of the techniques or functions described herein are stored. The instructions 1424 may also reside, fully or at least partially, in any of the registers of processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 during their execution by machine 1400. In one example, one or any combination of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 may constitute the machine-readable medium 1422. Although the machine-readable medium 1422 is exemplified as a single medium, the term “machine-readable medium” may include a single or multiple mediums configured to store one or more instructions 1424 (for example, a centralized or distributed database, and / or associated caches and servers).

[0100] The term “machine-readable medium” can include any medium that can store, encode, or carry instructions for execution by machine 1400, which cause machine 1400 to perform one or more of the technologies of the present disclosure, or that can store, encode, or carry data structures used by or associated with such instructions. Examples of non-limiting machine-readable mediums can include solid memory, optical mediums, magnetic mediums, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In one example, a non-temporary machine-readable medium includes a machine-readable medium having a plurality of particles having an immutable (e.g., stationary) mass, and is therefore a composition of a material. Thus, a non-temporary machine-readable medium is a machine-readable medium that does not contain a transient propagating signal. Specific examples of non-temporary machine-readable media include non-volatile memory such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0101] For example, information stored on machine-readable medium 1422 or otherwise provided in any way can represent instruction 1424 itself, or instruction 1424 in a format from which instruction 1424 can be derived. This format from which instruction 1424 can be derived may include source code, encoded instructions (e.g., in a compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), or similar. Information representing instruction 1424 in machine-readable medium 1422 can be processed by a processing circuit into instructions for performing any of the operations described herein. For example, deriving instruction 1424 from information (e.g., processing by a processing circuit) may include compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decrypting, encrypting, decrypting, packaging, unpackaging, or otherwise manipulating the information into instruction 1424.

[0102] For example, the derivation of instruction 1424 may involve assembling, compiling, or interpreting information (e.g., by a processing circuit) to create instruction 1424 from some intermediate or pre-processed format provided by machine-readable medium 1422. When the information is provided in multiple parts, it can be combined, decompressed, and modified to create instruction 1424. For example, the information may take the form of multiple compressed source code packages (or object code, or binary executable code, etc.) on one or more remote servers. The source code packages can be encrypted and decrypted as they are transferred over a network, decompressed if necessary, assembled (e.g., linked), compiled or interpreted on the local machine (e.g., into a library, a standalone executable, etc.), and executed by the local machine.

[0103] Instruction 1424 can also be transmitted or received over a communication network 1426 using a transmission medium via a network interface device 1420 that utilizes any one of numerous transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Illustrative communication networks may include, but are not limited to, local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), LoRa / LoRaWAN, or satellite communication networks, mobile phone networks (e.g., cellular networks such as those conforming to 3G, 4G LTE / LTE-A, or 5G standards), legacy telephone service (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 family of standards, the IEEE 802.15.4 family of standards, and peer-to-peer (P2P) networks, as well as Wi-Fi®). In one example, the network interface device 1420 may include one or more plug jacks (e.g., Ethernet, coaxial, or telephone plug jacks) or one or more antennas for connecting to the communication network 1426. In one example, the network interface device 1420 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term “transmission medium” is to be interpreted as including any intangible medium on which instructions for execution by machine 1400 can be stored, encoded, or carried, including digital or analog communication signals, or other intangible medium for facilitating such software communication. The transmission medium is a machine-readable medium.

[0104] The following are non-limiting examples, but in particular, specific aspects of the subject matter will be described in detail to address the problems described herein and to provide advantages.

[0105] Example 1 is a method for intelligent monitoring of attention and recognition of computer-aided diagnostic system outputs, the method comprising: receiving a video stream from an endoscope equipped with a camera, the video stream being captured during the endoscopic procedure; receiving a signal from a computer-aided diagnostic module, the signal indicating an anomaly detected in the video stream by a diagnostic algorithm; receiving an operator video stream from a camera installed in the room during the endoscopic procedure, the operator video stream including at least the eyes of a physician completing the endoscopic procedure; determining the physician's gaze position while performing the endoscopic procedure using a gaze algorithm, the gaze position indicating the position on the monitor that the physician views during the endoscopic procedure; determining whether the physician saw the detected anomaly by comparing the signal from the computer-aided diagnostic module with the physician's gaze position; and triggering an action based on the determination that the physician did not see the detected anomaly.

[0106] In Example 2, the subject of Example 1 is to recognize a physician who will complete an endoscopic procedure using a related person algorithm, which is configured to identify the physician holding the endoscope in order to confirm that the gaze position is the gaze position of the physician performing the endoscopic procedure.

[0107] In Example 3, the subject of Example 2 is to measure the time that a gaze position is at each position on the monitor, to label the detected anomalies with a complex anomaly label, the complex anomaly label indicating that the time the gaze position is directed at the detected anomaly exceeds a third threshold, to label the detected anomalies with a review anomaly label, the review anomaly label indicating that the time the gaze position is directed at the detected anomaly is less than a fourth threshold time, and to extract the detected anomalies along with the complex anomaly labels and review anomaly labels to generate an anomaly review report.

[0108] In Example 4, the subject of Example 3 is transmitted to one or more physicians for review of the abnormal review report, and received the reviewed abnormal review report from one or more physicians in order to generate a confirmed abnormal report, the confirmed abnormal report indicating that one or more physicians have confirmed the detected abnormality, and stored the confirmed abnormal report in a database.

[0109] In Example 5, the subject matter of Examples 1-4 includes the transmission of a warning signal to trigger the countermeasure.

[0110] In Example 6, the subject of Example 5 is further described by an alert signal indicating that the physician's gaze position was not directed towards the detected anomaly over a first set threshold time.

[0111] In Example 7, the subject matter of Examples 5-6 is expanded to include a warning signal that includes an anomaly identification specific to each anomaly detected by the computer-assisted diagnostic module.

[0112] In Example 8, the subject matter of Examples 1-7 includes generating a perceptible signal, which is configured to notify a physician of the detected anomaly, and the trigger for the countermeasure is the generation of a perceptible signal.

[0113] In Example 9, the subject of Example 8 is carried out by the controller's processing circuit processing the video stream along with the detected anomalies, by overlaying the video stream with a bounding box by associating the placement of detected anomalies from the computer-aided diagnostic module with the placement of detected anomalies on the video stream, thereby creating an overlaid video stream, and by displaying the overlaid video stream to show the placement of detected anomalies to the physician.

[0114] In Example 10, the subject of Example 9 is further described by the bounding box being a first color before the physician's gaze position is directed towards the detected anomaly over a set threshold time, and the bounding box being a second color after the physician's gaze position is directed towards the detected anomaly over a set threshold time.

[0115] In Example 11, the subject matter of Examples 8-10 includes the inclusion of a bounding box surrounding the detected anomaly when the physician's gaze position does not align with the position of the detected anomaly.

[0116] In Example 12, the subject of Example 11 is carried out by the controller's processing circuitry processing the video stream along with the detected anomaly, which includes determining that the physician's gaze position did not correspond to the position of the detected anomaly while the detected anomaly was displayed on the monitor, creating a missed anomaly video stream by overlaying the video stream with a visual graphic indicating that the physician missed the detected anomaly which is no longer displayed on the monitor, and displaying the missed anomaly video stream to alert the physician to the detected anomaly which is no longer displayed on the monitor.

[0117] Embodiment 13 is a system for intelligent monitoring of attention and recognition of computer-aided diagnostic system output, the system comprising an endoscope comprising an elongated member including a distal portion, the elongated member including a process camera attached to the distal portion for capturing a video stream during a procedure, a computer-aided diagnostic module configured to detect anomalies in the video stream using a diagnostic algorithm and transmit signals, a camera for capturing an operator video stream, the operator video stream including at least the physician's eyes during a procedure, a memory including commands, and a controller including a processing circuit, the processing circuit being configured to, when in operation, determine by command, the physician's gaze position during a procedure using a gaze algorithm, the gaze position indicating the position on the monitor the physician sees during the procedure, determine whether the physician saw a detected anomaly by comparing the signal from the computer-aided diagnostic module with the physician's gaze position, and trigger countermeasures based on the determination that the physician did not see a detected anomaly.

[0118] In Example 14, the subject of Example 13 is to recognize a physician who completes a procedure using a related person algorithm, the related person algorithm comprising recognition, configured to identify the physician holding the endoscope in order to confirm that the gaze position is the gaze position of the physician performing the procedure.

[0119] In Example 15, the subject of Example 14 is configured such that the controller's processing circuit measures, by command, the time a gaze placement is at each placement on the monitor; labels the detected anomaly with a complex anomaly label, the complex anomaly label indicating that the time the gaze placement is directed at the detected anomaly exceeds a third threshold; labels the detected anomaly with a review anomaly label, the review anomaly label indicating that the time the gaze placement is directed at the detected anomaly is less than a fourth threshold time; and extracts the detected anomaly along with the complex anomaly label and the review anomaly label to generate an anomaly review report.

[0120] In Example 16, the subject of Example 15 is configured such that the controller's processing circuit is configured to transmit an anomaly review report to one or more physicians for review of the anomaly review report by command, receive the reviewed anomaly review report from one or more physicians in order to generate a confirmed anomaly report, the confirmed anomaly report indicating that one or more physicians have confirmed the detected anomaly, and store the confirmed anomaly report in a database.

[0121] In Example 17, the subject matter of Examples 13-16 includes triggering an action by transmitting a warning signal, and the warning signal indicating that the physician's attention placement was not directed to the detected anomaly over a first set threshold time.

[0122] In Example 18, the subject of Example 17 is expanded to include a warning signal that includes an anomaly identification specific to each anomaly detected by the computer-assisted diagnostic module.

[0123] In Example 19, the subject matter of Examples 13-18 is further configured such that, in order to trigger countermeasures, the controller's processing circuit generates a perceptible signal by command, and the perceptible signal is configured to notify a physician of the detected anomaly.

[0124] In Example 20, the subject of Example 19 is configured such that, in order to generate a perceptible signal, the controller's processing circuitry is configured to, by command, overlay a video stream with a bounding box by corresponding to the detected anomaly arrangement from a computer-aided diagnostic module and the detected anomaly arrangement on the video stream, and to create an overlaid video stream and display the overlaid video stream to show the detected anomaly arrangement to a physician.

[0125] In Example 21, the subject of Example 20 is further described by the bounding box being a first color before the physician's gaze position is directed towards the detected anomaly over a set threshold time, and the bounding box being a second color after the physician's gaze position is directed towards the detected anomaly over a set threshold time.

[0126] In Example 22, the subject matter of Examples 19-21 includes the provision that when the physician's gaze position does not align with the position of the detected anomaly, the perceptible signal includes a bounding box surrounding the detected anomaly.

[0127] In Example 23, the subject of Example 22 is configured such that, in order to generate a perceptible signal, the controller's processing circuit determines by command that the physician's gaze position did not correspond to the position of the detected anomaly while the detected anomaly was displayed on the monitor; creates a missed anomaly video stream by overlaying the video stream with a visual graphic indicating that the physician missed the detected anomaly which is no longer displayed on the monitor; and displays the missed anomaly video stream to alert the physician to the detected anomaly which is no longer displayed on the monitor.

[0128] Example 24 is at least one machine-readable medium containing instructions that, when executed by a processing circuit, cause the processing circuit to perform an operation to implement any of Examples 1 to 23.

[0129] Example 25 is an apparatus that includes means for implementing any of Examples 1 to 23.

[0130] Example 26 is a system for implementing any of Examples 1 to 23.

[0131] Example 27 is a method for implementing any of Examples 1 to 23.

[0132] The description detailed above includes references to the accompanying drawings, which form part of the detailed description. The drawings illustrate specific examples that may be implemented. These embodiments are also referred to herein as “Examples.” Such embodiments may include elements in addition to those illustrated or described. However, the inventors also intend examples in which only those elements illustrated or described are provided. Furthermore, the inventors also intend examples in which any combination or permutation of those elements illustrated or described (or one or more aspects thereof) is used with respect to a particular embodiment (or one or more aspects thereof) or with respect to other embodiments (or one or more aspects thereof) illustrated or described herein.

[0133] All publications, patents, and patent documents referenced herein are incorporated herein by reference in their entirety, as if they were incorporated individually by reference. In the event of any conflict between the usage herein and those documents incorporated by reference, the usage of the incorporated documents shall be considered supplementary to the usage herein, and in the event of an incompatible conflict, the usage herein shall prevail.

[0134] In this specification, the word "one" (which may not be used; in the original English, it is "a" or "an") includes one or more, as is common in patent literature, regardless of any other examples or uses of "at least one" or "one or more". In this specification, the word "or" is used to refer to non-exclusive "or" such that "A or B" includes "A but not B", "B but not A", and "A and B". In the appended claims, "including" and "in which" in the original English are used as plain English paraphrases of "comprising" and "wherein", respectively. Also, in the following claims, "including" and "comprising" in the original English are open-ended; that is, a system, device, article, or process that includes elements in addition to those listed after such words in the claim is still considered to fall within the scope of that claim. Furthermore, in the following claims, terms such as "first," "second," and "third" are used merely as labels and are not intended to impose numerical requirements on the subject.

[0135] As used herein, the term “approximately” means roughly, within a range, about, or around that range. When the term “approximately” is used with a numerical range, it modifies that range by extending the boundary above and below the specified numerical value. Generally, the term “approximately” is used herein to modify a numerical value by a 10% variance above and below the stated number. In one embodiment, the term “approximately” means plus or minus 10% of the numerical value of the number in which it is used. Thus, approximately 50% means within the range of 45% to 55%. Numerical ranges described by endpoints herein include all numbers and fractions that fall within that range (for example, 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, 4.24, and 5). Similarly, numerical ranges described herein by endpoints include subranges that fall within that range (for example, 1 to 5 includes 1 to 1.5, 1.5 to 2, 2 to 2.75, 2.75 to 3, 3 to 3.90, 3.90 to 4, 4 to 4.24, 4.24 to 5, 2 to 5, 3 to 5, 1 to 4, and 2 to 4). It should also be understood that all numbers and fractions are presumed to be modified by the word “approximately”.

[0136] The above description is intended to be illustrative and not restrictive. For example, the embodiments (or one or more embodiments thereof) described above can be used in combination with each other. Other embodiments can be used by those skilled in the art who have reviewed the above description. The abstract is submitted with the understanding that it is intended to allow the reader to quickly confirm the nature of the technical disclosure and is not to be used to interpret or limit the claims or their meaning. Also, in the forms for carrying out the above invention, various features may be grouped together in order to streamline the disclosure. This should not be interpreted as meaning that the disclosed features not claimed are essential to any claim. Rather, the subject matter of the invention may lie in fewer features than all the features of a particular disclosed embodiment. Accordingly, the following claims are incorporated into forms for carrying out the invention, such that each claim stands on its own as a separate embodiment. The scope of the embodiments should be determined with reference to the supplementary claims, along with the entire scope of the equivalents that are the subject of the supplementary claims. [Explanation of symbols]

[0137] 10 Endoscopy Systems 12. Imaging and control systems 14 Endoscopy 16 Control Unit 18 Output unit, display unit 20 Input Units 22 Light source units 24 Fluid source 26 Suction pump 28 Insertion Section 30 Functional Sections 32 Handle section 34 Cable Sections 36 Coupler Sections 38 Control knob 40A Port 40B Port 41 Cart 42 Image Processing Unit 44 Treatment Generator 46 Drive Unit 47 Cables 48 Surgical instruments 300 Systems 302 Endoscope 304 Slender member 305 Proximal portion 306 Distal portion 308 Process Camera 310 Process Video Stream 312 Computer-Aided Diagnostic Module 314 Abnormality 314A, 314B, 314C Tissue abnormalities 316 Signal 318 Camera 320 Operator Video Stream 322 memory 324 command 334 First timestamp 336 Controllers 338 Second timestamp 342 Placement 350 countermeasures 352 Warning signal 360 Perceptible Signals 362 Bounding Boxes 366 overlaid video streams 400 System 402 CAD Algorithms 404 Endoscopist Detection Algorithm 406 Gaze Placement Detection Algorithm 408 Comparison Algorithms 410 User Interface Module 412 displays 414 video streams analyzed 420 Abnormal 422 Area of ​​attention 424 Endoscopy Operators 426 Gaze arrangement 428 Warning Video Stream 430 Overlaid video streams 432 video streams detected 434 Detected Warning Video Streams 500 ways 702 Bounding Box 702A Bounding Box 702B Bounding Box 702C Bounding Box 704 Medical Images 708 Gaze duration threshold 902 Abnormal video stream 904 Visual Graphics 1200 annotated images 1210 images 1220 annotations 1230 Marking Box 1240 Polyp Identification Box 1250 Process Identification Box 1320 Confirmed anomaly report 1322 Third threshold 1324 The fourth threshold time 1326 Complex Anomaly 1328 Reviews Abnormal Label 1330 Anomaly Review Report 1332 reviewed. Report anomaly review. 1334 Confirmed anomaly report 1400 machines 1402 Hardware Processors 1404 Main Memory 1406 Static Memory 1408 Mass storage devices 1410 Display Unit 1412 Alphanumeric input device 1414 User Interface (UI) Navigation Devices 1416 Sensor 1418 Signal Generating Devices 1420 Network Interface Device 1422 Machine-readable media 1424 Instructions 1426 Communication Network 1428 Output Controller 1430 Interlink

Claims

1. A method for intelligent monitoring of attention and recognition of computer-aided diagnostic system outputs, A step of receiving a video stream from an endoscope equipped with a camera using a controller's processing circuit, wherein the video stream is captured during the endoscopic procedure, A step of receiving a signal from a computer-aided diagnostic module, wherein the signal indicates an anomaly detected in the video stream by a diagnostic algorithm, The steps include receiving an operator video stream from a camera installed in the room during the endoscopic procedure, wherein the operator video stream includes at least the eyes of the physician completing the endoscopic procedure, A step of determining the gaze position of the physician while performing the endoscopic procedure using a gaze algorithm, wherein the gaze position represents the arrangement on the monitor that the physician views while performing the endoscopic procedure, The steps include determining whether the physician saw the detected abnormality by comparing the signal from the computer-assisted diagnostic module with the physician's gaze position, A step in which the physician determines that he did not see the detected abnormality, triggers countermeasures, Methods that include...

2. The method according to claim 1, comprising the step of recognizing the physician who will complete the endoscopic procedure using a related person algorithm, wherein the related person algorithm is configured to identify the physician holding the endoscope in order to confirm that the gaze position is the gaze position of the physician performing the endoscopic procedure.

3. The steps include measuring the time the gaze placement is in each placement on the monitor, A step of labeling the detected anomaly with a complex anomaly label, wherein the complex anomaly label indicates that the time the gaze placement is directed towards the detected anomaly exceeds a third threshold. A step of labeling the detected anomaly with a review anomaly label, wherein the review anomaly label indicates that the time the attention placement is directed towards the detected anomaly is less than a fourth threshold time. The steps include: extracting the detected anomalies along with the complex anomaly label and the review anomaly label, and generating an anomaly review report; The method according to claim 2, including the method described in claim 2.

4. The steps include transmitting the abnormal review report to one or more physicians for review, A step of generating a confirmed anomaly report, wherein the confirmed anomaly report indicates that the one or more physicians have confirmed the detected anomaly, The steps include saving the confirmed anomaly report to the database, The method according to claim 3, including the method described in claim 3.

5. The method according to claim 1, wherein the step of triggering the countermeasure includes the step of transmitting a warning signal.

6. The method according to claim 5, wherein the warning signal indicates that the physician's gaze position was not directed to the detected anomaly for a first set threshold time.

7. The method according to claim 5, wherein the warning signal includes an anomaly identifier specific to each anomaly detected by the computer-assisted diagnostic module.

8. The step that triggers the aforementioned countermeasure is: The method according to claim 1, comprising the step of generating a perceptible signal, wherein the perceptible signal is configured to notify the physician of the detected abnormality.

9. The perceptible signal indicates that the processing circuit of the controller processes the video stream together with the detected anomaly. The process involves associating the placement of detected anomalies from the computer-aided diagnostic module with the placement of detected anomalies on the video stream, thereby overlaying the video stream with a bounding box and creating an overlaid video stream. Displaying the overlaid video stream to show the physician the arrangement of the detected anomalies, The method according to claim 8, including being carried out by...

10. The method according to claim 9, wherein the bounding box is a first color before the physician's gaze position is directed to the detected anomaly over a set threshold time, and the bounding box is a second color after the physician's gaze position is directed to the detected anomaly over a set threshold time.

11. The method according to claim 8, wherein the perceptible signal includes a bounding box surrounding the detected anomaly when the gaze position of the physician does not match the position of the detected anomaly.

12. The perceptible signal indicates that the processing circuit of the controller processes the video stream together with the detected anomaly. It is determined that the physician's gaze position did not correspond to the position of the detected anomaly while the anomaly was displayed on the monitor, The process involves overlaying the video stream with a visual graphic indicating that the physician missed the detected anomaly, which is no longer displayed on the monitor, to create a missed anomaly video stream. Displaying the missed abnormal video stream to warn the physician of the detected abnormality that is no longer displayed on the monitor, The method according to claim 11, comprising performing the action by...

13. A system for intelligent monitoring of attention and recognition of computer-aided diagnostic system outputs, It is an endoscope, An elongated member including a distal portion, An endoscope comprising an elongated member attached to the distal portion, including a process camera that captures a video stream during the procedure, A computer-aided diagnostic module configured to detect anomalies in the video stream using a diagnostic algorithm and transmit signals, A camera for capturing an operator video stream, wherein the operator video stream includes at least the eyes of the physician during the procedure, Memory containing instructions, A controller including a processing circuit, wherein the processing circuit, when in operation, responds to the command, The determination of the physician's gaze position while performing the procedure using a gaze algorithm, wherein the gaze position indicates the arrangement of the monitor that the physician views during the procedure. The determination of whether the physician saw the detected abnormality is made by comparing the signal from the computer-assisted diagnostic module with the physician's gaze position. The countermeasures are triggered based on the fact that the aforementioned physician determined not to have observed the aforementioned abnormality, A controller configured to do the following, A system equipped with these features.

14. The system according to claim 13, comprising recognizing the physician who completes the procedure using a related person algorithm, wherein the related person algorithm is configured to identify the physician holding the endoscope in order to confirm that the gaze position is the gaze position of the physician performing the procedure.

15. The processing circuit of the controller, according to the instruction, The process involves measuring the time that the gaze placement is in each placement on the monitor, The process involves labeling the detected anomaly with a complex anomaly label, wherein the complex anomaly label indicates that the time the gaze placement is directed towards the detected anomaly exceeds a third threshold. The process involves labeling the detected anomaly with a review anomaly label, wherein the review anomaly label indicates that the time the gaze placement is directed towards the detected anomaly is less than a fourth threshold time. The detected anomalies are extracted along with the complex anomaly labels and the review anomaly labels, and an anomaly review report is generated. The system according to claim 14, configured to do the following.

16. The processing circuit of the controller, in accordance with the instruction, Transmitting the aforementioned abnormal review report to one or more physicians for review, To generate a confirmed anomaly report, the system receives the reviewed anomaly review report from one or more physicians, the confirmed anomaly report indicating that the one or more physicians have confirmed the detected anomaly. The confirmed anomaly reports mentioned above are stored in a database, The system according to claim 15, configured to do the following.

17. The system according to claim 13, wherein triggering the countermeasures includes transmitting a warning signal, the warning signal indicating that the physician's gaze position was not directed to the detected anomaly for a first set threshold time.

18. The system according to claim 17, wherein the warning signal includes an anomaly identifier specific to each anomaly detected by the computer-assisted diagnostic module.

19. In order to trigger the aforementioned countermeasure, the processing circuit of the controller, by the instruction, To generate a perceptible signal, wherein the perceptible signal is configured to notify the physician of the detected abnormality. The system according to claim 13, configured to do the following.

20. In order to generate the perceptible signal, the processing circuit of the controller, by the command, The process involves overlaying the video stream with a bounding box by corresponding to the placement of the detected anomalies from the computer-aided diagnostic module and the placement of the detected anomalies on the video stream, thereby creating an overlaid video stream. Displaying the overlaid video stream to show the physician the arrangement of the detected anomalies, The system according to claim 19, configured to do the following.

21. The system according to claim 20, wherein the bounding box is a first color before the physician's gaze position is directed to the detected anomaly over a set threshold time, and the bounding box is a second color after the physician's gaze position is directed to the detected anomaly over a set threshold time.

22. The system according to claim 19, wherein the perceptible signal includes a bounding box surrounding the detected anomaly when the physician's gaze position does not match the position of the detected anomaly.

23. In order to generate the perceptible signal, the processing circuit of the controller, by the command, It is determined that the physician's gaze position did not correspond to the position of the detected anomaly while the anomaly was displayed on the monitor, The process involves overlaying the video stream with a visual graphic indicating that the physician missed the detected anomaly, which is no longer displayed on the monitor, to create a missed anomaly video stream. Displaying the missed abnormal video stream to warn the physician of the detected abnormality that is no longer displayed on the monitor, The system according to claim 22, configured to do the following.

Images