Device, system, and method for identifying uninspected areas during a medical procedure

Abstract

At least one exemplary embodiment is directed to a device including a memory including instructions and a processor that executes the instructions to generate image data and depth data of an internal region during a medical procedure being performed by a clinician on the internal region of a patient, generate a depth model of the internal region of the patient based on the depth data during the medical procedure, determine based on the depth model that the image data of the medical procedure does not include image data of a section of the internal region, and generate one or more alerts alerting the clinician that the section of the internal region has not been examined.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority under 35 U.S.C. §119(e) to U.S. patent application Ser. No. 17 / 012,974, filed Sep. 4, 2020, entitled "Devices, Systems, and Methods for Identifying Uninspected Areas During a Medical Procedure," which is incorporated herein by reference in its entirety.

[0002] The present disclosure is generally directed to devices, systems, and methods for identifying uninspected areas during a medical procedure. [Background technology]

[0003] Modern medical procedures can be camera-assisted, with video and / or still images of the procedure displayed in real time to assist the clinician in navigating the anatomy and performing the procedure. In some cases, regions of the anatomy are left unexamined because, for example, large areas of the anatomy look very similar. In other cases, confusion due to a lack of distinct features in the anatomy can lead to portions of the anatomy being left unexamined. Summary of the Invention

[0004] At least one exemplary embodiment is directed to a device including a memory including instructions and a processor that executes the instructions to generate image data and depth data of an internal region during a medical procedure being performed by a clinician on the internal region of a patient, generate a depth model of the internal region of the patient based on the depth data during the medical procedure, determine based on the depth model that the image data of the medical procedure does not include image data of a section of the internal region, and generate one or more warnings alerting the clinician that the section of the internal region has not been examined.

[0005] At least one exemplary embodiment is directed to a system including a display, a medical instrument, and a device, the system including a memory including instructions, and a processor that executes the instructions to generate, during a medical procedure being performed by a clinician on the internal region of a patient, image data and depth data of the internal region, generate, during the medical procedure, a depth model of the patient's internal region based on the depth data, determine, based on the depth model, that the image data of the medical procedure does not include image data of a section of the internal region, and generate one or more alerts alerting the clinician that the section of the internal region has not been examined.

[0006] At least one exemplary embodiment is directed to a method of generating image data and depth data of an internal region of a patient during a medical procedure being performed by a clinician on the internal region, generating a depth model of the patient's internal region based on the depth data during the medical procedure, determining based on the depth model that the image data does not include image data for a section of the internal region, and generating one or more warnings alerting the clinician that the section of the internal region has not been examined. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a system in accordance with at least one exemplary embodiment. [Figure 2] 1 illustrates an exemplary structure for a medical device in accordance with at least one exemplary embodiment. [Figure 3] 1 illustrates a method in accordance with at least one example embodiment. [Figure 4] 1 illustrates a method in accordance with at least one example embodiment. [Figure 5] 1 illustrates a workflow for a medical procedure in accordance with at least one exemplary embodiment. [Figure 6] 1 illustrates an exemplary output device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Endoscopes and other medical instruments for imaging anatomical structures have a limited field of view and require the user to manipulate the endoscope to image a larger field of view within the anatomy. In some cases, regions of the anatomy are left unexamined because, for example, the regions of the anatomy appear similar to one another. Confusion due to a lack of distinct features in the anatomy is another example that can lead to portions of the anatomy being left unexamined.

[0009] The inventive concepts relate to anatomical imaging systems and diagnostic procedures in which it is important to ensure that targeted regions of an anatomical structure are completely inspected. For example, the inventive concepts are directed to a system that assists in the visual inspection of an anatomical structure during an endoscopy or other medical procedure by identifying and indicating uninspected regions. The system can be used to generate information about how much of the anatomical structure has been inspected or imaged by an imaging sensor, as well as to provide information (e.g., location, shape, size, etc.) about regions of interest that were not inspected. For example, the system can assist a surgeon or clinician in ensuring that all anatomical structures intended to be inspected for abnormalities have actually been inspected. This system is useful for procedures such as colonoscopy, bronchoscopy, laryngoscopy, etc.

[0010] In general, the inventive concept involves generating depth data and visual imaging data, combining them (using known alignment and / or mapping operations) to generate visualizations and trigger alerts that can be used to indicate portions of the anatomy that were and were not imaged by the imaging system. The visualizations and alerts provide information (actual or relative location, area, shape, etc.) about areas that were not inspected or imaged to help mitigate the risk of leaving areas intended to be inspected uninspected.

[0011] By graphing / constructing the 3D depth model, regions within the anatomy that are not imaged in color by the imaging sensor can be identified. For example, discontinuities (e.g., missing data) in the 3D depth model itself can indicate that the area surrounding the discontinuity is not imaged by the color image sensor.

[0012] In at least one exemplary embodiment, a general 3D model may be pre-generated (e.g., from depth data acquired from other patients) and used to select the general region of the anatomy to be examined. The alert may then indicate whether the selected region has been completely examined, as well as provide a measurement and / or indication of how much of the overall region was left unexamined, the number of regions left unexamined (blind spots), and where those missing regions are located.

[0013] Even in the absence of a pre-generated general 3D model, a user or clinician can have the ability to indicate what the general region of interest is and have the mapping and measurement system of the present invention begin generating the necessary information once the endoscope reaches that general region of interest. The user can also indicate when the endoscope reaches the end of the general region of interest, which can then enable the generation of additional metrics that can be used to indicate whether the region of interest has been completely inspected, how much has been inspected, how many sub-regions have not been inspected (blind spots), etc.

[0014] An additional feature of the inventive concept is to help the user navigate the 3D space to reach areas that have not yet been inspected or that have been missed. This can be done by adding graphics (arrows) to the display monitor, audio instructions (e.g., "keep moving forward," "uninspected area on the left," etc.).

[0015] For medical endoscopy, relying solely on 2D image data to determine whether a region within the anatomy has been completely inspected is less reliable than also using depth data. For example, using 2D visual data is susceptible to reflections, overexposure, smoke, etc. However, because there are very few or no flat surfaces in the human anatomy that extend beyond the field of view of a depth camera, depth data is useful for mapping the anatomy and reduces or eliminates cases where depth data cannot reliably determine whether a region has been completely inspected.

[0016] Thus, the inventive concepts relate to systems, methods, and / or devices that generate visual imaging data and depth data simultaneously or in a time-aligned manner, align the visual imaging data with the depth data and / or map one data set to the other, generate a 3D model of the scene using the depth data, identify discontinuities (missing data) in the 3D model as parts of the model for which no depth data is present, and infer those parts of the scene that were not imaged by the imaging sensor (i.e., identify areas where image / visual data is missing). The inventive concepts can create visualizations, alerts, measurements, etc. to provide a user with information about imaged areas, unimaged areas, their locations, their shapes, number of missed areas, etc.

[0017] Additionally, concepts of the present invention can use depth and image / visual data to create a 3D reproduction or composite 3D model of the visual content of a scene, for example, by projecting or overlaying 2D visual or image data onto the 3D depth model using corresponding depth data information. At least one exemplary embodiment provides a user with the option to interactively rotate the 3D composite model and / or the depth model. A depth map representation for the depth model can be created using the depth data and, optionally, the image / visual data. As an example, for a particular scene of interest, four corresponding depth map views can be generated having views from the north, south, east, and west, all four views derived from a single 3D depth model. As described above, a system according to an exemplary embodiment identifies regions in the 3D model that are missing data to identify portions of the anatomy that have not been imaged or examined.

[0018] In at least one exemplary embodiment, the depth and / or current position of an endoscope or other instrument with an imaging camera can be shown on the 3D depth model to help the user navigate to portions of the anatomy that are not being imaged (i.e., a "you're here" feature). Optionally, the system can calculate and report to the user metrics describing which regions of the anatomy are not being imaged. For example, this can be based on multiple gaps in the imaged region (e.g., the number of subregions that were not imaged, the size of each subregion, the shortest distance between subregions, etc.). These metrics can then be used to generate alerts for the user, e.g., audio and / or visual alerts, that a particular region has not been examined.

[0019] In at least one exemplary embodiment, the system uses information about the missed areas to generate a visualization on a primary or secondary live-view monitor that can help the user navigate to areas of the anatomy that have not been imaged / inspected. For example, graphics such as arrows can be overlaid on the live video data and used to indicate the direction (e.g., arrow direction) and distance (e.g., arrow length or arrow color) of the uninspected area. Here, the user can have the option to select one of the missed / uninspected areas and instruct the system to help navigate the user to that particular area. In at least one exemplary embodiment, a 3D pre-generated model of the anatomy (e.g., a generic model) can be used to allow the user to indicate areas of the anatomy that need to be inspected before the surgical procedure is performed. Displays, visualizations, and / or warnings can then be generated based on the difference between the areas intended to be inspected and the areas actually inspected.

[0020] In at least one exemplary embodiment, the system uses neural networks or other machine learning algorithms to learn (using multiple manually performed procedures) which areas should be inspected with a particular type of procedure. This can help determine whether any area has not been imaged / inspected or should be inspected. The system can generate an alert / notification to the user whenever an uninspected area is determined to be an area that should be inspected.

[0021] In at least one exemplary embodiment, a robotic arm may be included in an endoscopy system of devices that can use, for example, 3D depth map information to guide / navigate a camera-equipped endoscope or other device to a region of interest that has not yet been imaged. For example, using the robotic arm, a user can select a region of interest for the system to navigate, and a machine learning algorithm can be pre-trained and used to automatically perform this navigation without or with minimal user intervention. The system can use a second machine learning algorithm to create and recommend the most efficient navigation path so that all unimaged regions of interest can be imaged in the shortest possible time. The user can then approve this path, and the robotic arm will automatically navigate / move the camera-equipped endoscope or other device along this path. Otherwise, the user can override the recommendation for the most efficient path and select a region of interest for the robotic arm to navigate.

[0022] As described above, depth and image / visual information can be generated by using an imaging sensor that simultaneously captures both depth and visual imaging data. This is a useful approach that simplifies aligning the two types of data or mapping one type of data to the other. An example of this type of sensor is the AR430 CMOS sensor. In at least one exemplary embodiment, a system collects depth data using a first depth sensor (e.g., LIDAR) and generates image data using a second imaging sensor. The image data and depth data are then combined by either aligning them or inferring a mapping from one data set to the other. This allows the system to infer which image data corresponds to the acquired depth data and, optionally, project the visual data onto a 3D depth model. In at least one other exemplary embodiment, the system uses two imaging sensors in a stereoscopic configuration to generate a stereo capture of visual / image data of a scene, which the system can then use to infer depth information from the stereo image data.

[0023] As described above, exemplary embodiments can identify regions of the anatomy that have not been imaged or examined. This is accomplished by creating a 3D (depth) model and identifying discontinuities or missing depth information within the model (i.e., identifying regions of the 3D model for which no depth data exists). Optionally, based on such discontinuities in the depth information, the system can also infer what image / visual information is missing, for example, by generating a visualization on a display showing the regions with missing data. If the depth data is less sparse than the image data, the resolution of the depth sensor is taken into account to determine whether image data is indeed missing.

[0024] To determine whether missing / uninspected regions should be inspected, the system can identify gaps in the region the user is inspecting (e.g., occlusion regions). Optionally, the system uses input from the user about the entire region of interest. User input may be in the form of identifying regions of interest on a general 3D depth model. In at least one exemplary embodiment, the system uses machine learning algorithms or deep neural networks to learn the regions that should be inspected for each specific type of procedure over time and across multiple manually performed procedures. The system can then compare this information to the regions actually inspected by the user to determine whether any regions not imaged / inspected should be inspected.

[0025] When using a 3D pre-generated model of the anatomical structure, the system may determine the position of the endoscope or other camera device using known direct or inferred mapping methods to map the 3D pre-generated model to the 3D model being generated in real time. In this case, the endoscope may include additional sensors that provide current position information. These additional sensors may include, but are not limited to, a magnetometer, a gyroscope, and / or an accelerometer. The accuracy of the information from these sensors does not need to be precise, for example, when the user indicates a general region of interest on the 3D pre-generated model. Even if the estimated position is not sufficiently accurate, the selected region in the pre-generated model may be expanded to ensure that the region of interest is still fully inspected. Additionally or alternatively, specific features within the anatomical structure can be identified and used as interest points to determine when the endoscope enters and exits the region of interest established using the 3D pre-generated model. These features provide the user with some known indication / cues as to where the region of interest should begin and end (e.g., the beginning and end of a lumen). In addition to depth data, the 3D pre-generated model may also include reference image / visual data to assist in feature matching operations that align the live depth model with the pre-generated depth model. Thus, both depth and image data may be used for feature matching and mapping operations.

[0026] As described above, the 3D pre-generated model may be obtained from multiple other depth models generated from previously performed medical procedures. However, exemplary embodiments are not limited thereto, and alternative methods can be used to create the 3D pre-generated model. For example, the pre-generated model may be derived from a different modality, such as a computed tomography (CT) scan of the anatomical structure prior to the surgical procedure. In this case, if the CT scans are of the same patient, the 3D pre-generated model may be custom or specific to the patient. Here, the CT scan data is correlated with real-time depth data and / or image data from the medical procedure.

[0027] In consideration of the foregoing and the following description, it should be appreciated that exemplary embodiments provide a system that assists in the visual inspection of anatomical structures, for example, during endoscopic procedures and other procedures within anatomical structures. For example, a system according to exemplary embodiments identifies and indicates uninspected areas during a medical procedure, which may take the form of gaps in the entire inspected area or gaps in predetermined or preselected areas. The system can generate alerts, metrics, and / or visualizations to provide the user with information about the uninspected areas. For example, the system can provide navigation instructions to navigate the user to the uninspected area. A 3D pre-generated model can assist the user in specifying the area of ​​interest. The system can use deep learning algorithms to learn the areas to be inspected for each specific surgical procedure and then generate live alerts for the user whenever an area is missed by comparing the learned areas with the area actually being imaged / inspected. In at least one exemplary embodiment, the system uses a robotic arm that holds the endoscope, a machine learning algorithm that controls the robotic arm, and another machine learning algorithm that identifies the robotic arm and instructs it to follow the most efficient / fastest path from the current position of the endoscope to an uninspected region of interest or to and through a set of uninspected regions of interest. These and other advantages will become apparent from the description below.

[0028] 1 illustrates a system 100 according to at least one exemplary embodiment. System 100 includes an output device 104, a robotic device 108, a memory 112, a processor 116, a database 120, a neural network 124, an input device 128, a microphone 132, a camera 136, and a medical instrument or tool 140.

[0029] The output device 104 may include a display such as a liquid crystal display (LCD), a light emitting diode (LED) display, etc. The output device 104 may be a standalone display or a display integrated as part of another device such as a smartphone, laptop, tablet, etc. Although a single output device 104 is shown, the system 100 may include more output devices 104 according to the system design.

[0030] The robotic device 108 includes known hardware and / or software capable of robotically assisting medical procedures within the system 100. For example, the robotic device 108 may be a robotic arm mechanically attached to the instrument 140 and electrically communicating with and controllable by the processor 116. The robotic device 108 may be an optional element of the system 100 that can consume or receive 3D depth map information and guide / navigate the instrument 140 to regions of interest (ROIs) that have not yet been imaged. For example, if the robotic device 108 is a robotic arm, a user (e.g., a clinician) can select an ROI for the system to navigate, and the robotic arm can be pre-trained using a machine learning-based algorithm and used to automatically perform this navigation with little or no user involvement. Additionally, a second machine learning algorithm can be used to create and recommend the most efficient navigation path, so that all ROIs that have not yet been imaged can be imaged in the shortest possible time. The user can then approve this path, and the robotic arm automatically navigates / moves the instrument 140 along this path. If not, the user can override the recommendation for the most efficient path and select an area of ​​interest through which the robot arm should navigate.

[0031] Memory 112 may be a computer-readable medium containing instructions executable by processor 116. Memory 112 may include any type of computer memory device and may be volatile or non-volatile in nature. In some embodiments, memory 112 may include multiple different memory devices. Non-limiting examples of memory 112 include random access memory (RAM), read-only memory (ROM), flash memory, electrically erasable programmable ROM (EEPROM), dynamic RAM (DRAM), etc. Memory 112 may contain instructions that enable processor 120 to control various elements of system 100 and to store data in, for example, database 120 and retrieve information from database 120. Memory 112 may be local to (e.g., integrated with) processor 116 and / or separate from processor 116.

[0032] Processor 116 may correspond to one or many computer processing devices. For example, processor 116 may be provided as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any other type of integrated circuit (IC) chip, a collection of IC chips, a microcontroller, a collection of microcontrollers, etc. As a more specific example, processor 116 may be provided as a microprocessor, a central processing unit (CPU), and / or a graphics processing unit (GPU), or multiple microprocessors configured to execute a set of instructions stored in memory 112. Processor 116, when executing the instructions stored in memory 112, enables various functions of system 100.

[0033] Database 120 includes the same or similar structure as memory 112 described above. In at least one exemplary embodiment, database 120 is included on a remote server and stores training data for training neural network 124. The training data included in database 120 and used to train neural network 124 is described in more detail below.

[0034] Neural network 124 may perform functions related to artificial intelligence (AI) and may be an artificial neural network (ANN) implemented by one or more computer processing devices having the same or similar structure as processor 116, executing instructions on a memory having the same or similar structure as memory 112. For example, neural network 124 may use machine learning or deep learning to improve over time the accuracy of a set of outputs based on a set of inputs (e.g., similar sets of inputs). Thus, neural network 124 may utilize supervised learning, unsupervised learning, reinforcement learning, self-learning, and / or any other type of machine learning to generate a set of outputs based on a set of inputs. The role of neural network 124 is discussed in more detail below. It should be understood that database 120 and neural network 124 may be implemented by a server or other computing device separate from the remaining elements of system 100.

[0035] Input device 128 includes hardware and / or software that allows for user input to system 100. Input device 128 may include a keyboard, a mouse, a touch-sensitive pad, a touch-sensitive button, a touch-sensitive portion of a display, a mechanical button, a switch, and / or other control elements that provide user input to system 100 that allows for user control of certain functions of system 100.

[0036] The microphone 132 includes hardware and / or software to enable detection and collection of audio signals within the system 100. For example, the microphone 132 may enable collection of the voice of a clinician, activation of a medical instrument (e.g., medical instrument 140), and / or other sounds within the operating room.

[0037] The camera 136 includes hardware and / or software to enable collection of video, image, and / or depth information of a medical procedure. In at least one exemplary embodiment, the camera 136 captures video and / or still images of a medical procedure being performed on a patient's body. As known in endoscopy, arthroscopy, and the like, the camera 136 may be designed to enter the body and capture real-time video of the procedure to assist a clinician in performing the procedure and / or making a diagnosis. In at least one other exemplary embodiment, the camera 136 remains outside the patient's body to capture video of the external medical procedure. More cameras 136 can be included depending on the system design. For example, according to at least one exemplary embodiment, the camera 136 includes a camera for capturing image data (e.g., two-dimensional color images) and a camera for capturing depth data to create a three-dimensional depth model. Details of the camera 136 are discussed in more detail below with reference to FIG. 2.

[0038] The instrument or tool 140 may be a medical instrument or tool that can be controlled by a clinician and / or the robotic device 108 to assist in performing a medical procedure on a patient. The camera 136 may be integrated with the instrument 140, for example, in the case of an endoscope. However, the exemplary embodiment is not limited in this respect, and the instrument 140 may be separate from the camera 136, depending on the medical procedure. While one instrument 140 is shown, additional instruments 140 may be present in the system 100, depending on the type of medical procedure. Furthermore, it should be understood that the instrument 140 may be for use externally and / or internally to a patient's body.

[0039] 1 depicts various elements in system 100 as separate from one another, it should be understood that some or all of the elements may be integrated with one another, if desired. For example, a single desktop or laptop computer may include output device 104 (e.g., a display), memory 112, processor 116, input device 128, and microphone 132. In another example, neural network 124 may be included with processor 116 such that AI operations are performed locally rather than remotely.

[0040] Furthermore, it should be understood that each element in system 100 includes one or more communication interfaces that enable communication with other elements in system 100. These communication interfaces include wired and / or wireless communication interfaces for exchanging data and control signals with each other. Examples of wired communication interfaces / connections include Ethernet connections, HDMI connections, connections conforming to PCI / PCIe and SATA standards, etc. Examples of wireless interfaces / connections include Wi-Fi connections, LTE connections, Bluetooth connections, NFC connections, etc.

[0041] FIG. 2 illustrates an exemplary configuration for a medical instrument 140 including one or more cameras 136 mounted thereon, according to at least one exemplary embodiment. As described above, the medical instrument 140 may include one or more cameras or sensors for collecting image data for generating color images and / or depth data for generating a depth image or depth model. FIG. 2 illustrates a first exemplary configuration of the medical instrument 140a including two cameras 136a and 136b disposed at one end 144 of the medical instrument 140a. The camera 136a may be an imaging camera having an image sensor for generating and providing image data, and the depth camera 136b may be a depth sensor for generating and providing depth data. The camera 136a may generate color images including color information (e.g., RGB color information), while the camera 136b may generate depth images without color information.

[0042] 2 shows another exemplary configuration of a medical instrument 140b in which cameras 136a and 136b are positioned on the tip or end face 148 of the end 144 of the medical instrument 140b. In both exemplary medical instruments 140a and 140b, cameras 136a and 136b are positioned on the medical instrument to have overlapping fields of view. For example, cameras 136a and 136b are aligned with each other vertically or horizontally as shown, as desired. Also, imaging camera 136a may be swapped with depth camera 136b depending on design preference. The amount of overlapping fields of view may be a design parameter set based on empirical evidence and / or preference.

[0043] It should be understood that the depth camera 136b includes hardware and / or software for enabling distance or depth detection. The depth camera 136b can operate according to the time-of-flight (TOF) principle. Thus, the depth camera 136b includes a light source that emits light (e.g., infrared (IR) light) that reflects off an object and is then sensed by pixels of the depth sensor. For example, the depth camera 136b can operate according to the direct TOF or indirect TOF principle. A device operating according to the direct TOF principle measures the actual time delay between the emitted light and the reflected light received from the object, while a device operating according to the indirect TOF principle measures the phase difference between the emitted light and the reflected light received from the object, and the time delay is then calculated from the phase difference. In either case, the time delay between the emission of light from the light source and the reception of the reflected light at the sensor corresponds to the distance between the pixels of the depth sensor and the object. A specific example of the depth camera 136b is one that uses LIDAR. A depth model of the object can then be generated according to known techniques.

[0044] FIG. 2 illustrates a third exemplary configuration of a medical instrument 140c that includes a combined camera 136b capable of capturing image data and depth data. The combined image and depth sensor 136c may be located on the tip 148 of the instrument 140c. The camera 136c may include depth pixels that provide depth data and imaging pixels that provide image data. Like the medical instrument 140b, the medical instrument 140c further includes a light source that emits light (e.g., IR light) to enable collection of depth data by the depth pixels. One exemplary arrangement of imaging and depth pixels includes a camera 136c having a 2x2 array of pixels in a Bayer filter configuration, where one of the pixels in each 2x2 array, typically with a green color filter, is replaced with a depth pixel that is sensitive to IR light. Each depth pixel may have a filter that passes IR light and blocks visible light. However, exemplary embodiments are not limited thereto, and other configurations for the depth and image pixels are possible depending on design preference.

[0045] Although not explicitly shown for medical instrument 140a, it should be understood that a single camera 136c having image and depth sensing capabilities may be used on tip 144 of instrument 140a in place of cameras 136a and 136b. Additionally, in at least one exemplary embodiment, depth data may be derived from image data, for example, in a scenario where camera 136b in instrument 140a or instrument 140b is replaced with another camera 136a to form a stereoscopic camera from two cameras 136a that collect only image data. Depth data may be derived from a stereoscopic camera in accordance with known techniques, for example, by generating a disparity map from a first image from one camera 136a and a second image from the other camera 136a, acquired simultaneously.

[0046] It should be understood that additional cameras 136a, 136b, and / or 136c may be included on the medical instrument 140 and in any location depending on design preference. It should also be understood that various other sensors may be included on the medical instrument 140. Such other sensors include, but are not limited to, magnetometers, accelerometers, and / or the like, which may be used to estimate the direction of position and / or orientation of the medical instrument 140.

[0047] 3 illustrates a method 300 according to at least one example embodiment. In general, method 300 may be performed by one or more elements from FIG. 1. For example, method 300 may be performed by processor 116 based on various inputs from other elements of system 100. However, method 300 may be performed by additional or alternative elements within system 100, for example, under the control of processor 116 or another element, as will be appreciated by those skilled in the art.

[0048] At operation 304, method 300 includes generating image data and depth data of an internal region of a patient during a medical procedure being performed by a clinician on the internal region. The image data and depth data are generated according to any known technique. For example, as described above, the image data can include color images and / or video of the medical procedure captured by camera 136a or camera 136c, and the depth data can include depth images and / or video of the medical procedure captured by camera 136b or camera 136c. In at least one exemplary embodiment, the depth data is derived from the image data, for example, from image data of two cameras 136a in a stereoscopic configuration (or even a single camera 136a). The depth data may be derived from the image data in any known manner.

[0049] At operation 308, the method 300 includes generating a depth model or depth map of the interior region based on the depth data. For example, the depth model is generated during the medical procedure using depth data received by the processor 116 from one or more cameras 136. Any known method can be used to generate the depth model. In at least one exemplary embodiment, the depth model is generated in response to determining that a medical instrument 140 used in the medical procedure enters the general region of interest.

[0050] At operation 312, the method 300 includes determining, based on the depth model, that the image data of the medical procedure does not include image data for a section of the internal region. For example, the processor 116 determines that the image data does not include image data for a section of the internal region when the region of the depth model lacks depth data exceeding a threshold amount. The threshold amount of depth data and the size of the region in the depth model are design parameters based on empirical evidence and / or preference. A region with missing depth data may be a single region in the depth model. In at least one exemplary embodiment, a region with missing depth data may include regions with depth data interspersed among regions with no depth data. The parameters of the region (e.g., size, shape, continuity) and / or the threshold amount of depth data may be variable and / or selectable during the medical procedure and may change automatically depending on, for example, the position of the medical instrument 140 within the internal region. For example, as the medical instrument 140 approaches or enters a known region of interest in the internal region, the threshold amount of depth data and / or the region parameters may be adjusted to be more sensitive to missing data than regions of general interest. This may further ensure that regions of interest are thoroughly inspected while reducing unnecessary alarms and / or processing resources for regions of no interest.

[0051] In at least one exemplary embodiment, a clinician can confirm or doubt that image data is missing based on a simultaneously displayed composite 3D model that includes image data overlaid or mapped onto the depth model. For example, the clinician can see whether the composite 3D model includes image data in areas where the system detected a lack of depth data. Details of composite 3D models are discussed in more detail below.

[0052] In operation 316, method 300 examines another depth model, which may be general to the patient's internal region, specific to the patient's internal region, or both. The another depth model may be a 3D model with or without overlaid image data. For example, if the internal region is the patient's esophagus, the general depth model may be a general model of the esophagus received from a database. The general model may be modeled based on depth and / or image data acquired from one or more other patient internal regions (e.g., esophagus) during other medical procedures. Thus, the general model may be an approximation of a depth model generated during a medical procedure based on the patient's anatomy. In at least one exemplary embodiment, the general depth model may include depth data of the current patient's internal region, for example, if depth and / or image data of the current patient exists from a previous medical procedure on the internal region.

[0053] If a previous medical procedure on the current patient generated depth and / or image data, the separate depth model in operation 316 may be entirely specific to the current patient (i.e., not based on data from other patients). In at least one exemplary embodiment, the separate depth model includes patient-specific image and / or depth data as well as generic image and / or depth data. For example, if patient-specific data exists but is incomplete, data from the generic model may be applied to fill in gaps in the patient-specific data. The separate depth model may be received and / or generated in operation 316 or at some other point within or before operations 304 through 312.

[0054] The alternative depth model examined in operation 316 may have a pre-selected region of interest to assist in identifying unimaged regions of the interior region during the medical procedure. As discussed in more detail below with reference to FIG. 4 , the region of interest may be selected by a clinician before the medical procedure or during the medical procedure (e.g., using a touch display displaying the alternative depth model). The region of interest may be selected with or without the assistance of labeling or direction on the alternative depth model, where such labeling or direction is generated using the neural network 124 and / or using input from the clinician. For example, using the historical image and / or depth data that generated the alternative depth model, the neural network 124 may assist in identifying known problem areas (e.g., existing lesions, growths, etc.) and / or known potential problem areas (e.g., areas where lesions, growths, etc. often appear) by analyzing the historical data and known conclusions drawn therefrom to arrive at one or more other conclusions that may assist method 300. The region of interest may be identified by the neural network 124 with or without clinician assistance, allowing the method to be fully automated or user-controlled.

[0055] In operation 320, method 300 determines whether the section determined in operation 312 to have no image data is a region of interest. For example, method 300 can determine the location of medical instrument 140 within the interior region using the additional sensors described above and compare the determined location to the location of the region of interest from another depth model. If the location of medical instrument 140 is within a threshold distance of the region of interest in the other depth model, the section of the interior region is determined to be a region of interest, and method 300 proceeds to operation 324. Otherwise, the section of the interior region is determined not to be a region of interest, and method 300 returns to operation 304 and continues to generate image and depth data. The threshold distance may be a design parameter set based on empirical evidence and / or preference.

[0056] The location of the medical instrument 140 may be determined with the aid of a depth model, image data, and / or one or more other sensors commonly known to be useful in detecting locations within an anatomical structure. For example, in at least one exemplary embodiment, the depth model may be incomplete early in a medical procedure and may be compared to another depth model. Knowledge of which portions of the depth model are complete or incomplete compared to another (complete) depth model may be used to estimate the location of the medical instrument 140 in the interior region. For example, a completed portion of the depth model may be overlaid on another depth model to estimate the location of the medical instrument 140 as the location where the depth model becomes incomplete compared to the other completed depth model. However, exemplary embodiments are not limited thereto, and any known method for determining the location of the medical instrument 140 may be used. Such methods include algorithms for simultaneous localization and mapping (SLAM) techniques that can simultaneously map an environment (e.g., an interior region) while tracking a current location within the environment (e.g., the current location of the medical instrument 140). SLAM algorithms may be further assisted by a neural network 124.

[0057] In at least one exemplary embodiment, even if a section of the interior region is determined to be not of interest, the section may still be flagged and / or recorded in memory 112, allowing the clinician to potentially revisit the uninspected region at a later time. For example, the system may present an audio and / or visual notification to the clinician that a particular section was determined to be missing image data but not of interest. The notification may include a visual notification on the depth model and / or composite model (including image data overlaid on the depth model), as well as directions for navigating the medical instrument 140 to the section determined to be missing image data.

[0058] It should be appreciated that, if desired, operations 316 and 320 can be omitted and method 300 can proceed from operation 312 directly to operation 324 to alert the clinician. The omission or inclusion of operations 316 and 320 can be presented as an option to the clinician at any time before or during the medical procedure.

[0059] In operation 324, the method 300 generates one or more warnings to alert the clinician that a section of the interior region has not been inspected. The warnings may be audio and / or video in nature. For example, the output device 104 outputs an audio warning, such as a beep or other noise, and / or a visual warning, such as a warning message on a display or a warning light.

[0060] As shown in FIG. 3, the method 300 may further perform optional operations 328 and 332, for example, in parallel with other operations of FIG.

[0061] For example, in operation 328, the method 300 can generate a composite model of the interior region based on the image data of the medical procedure and the depth model. The composite model includes a three-dimensional model of the interior region with the image data of the medical procedure projected or overlaid onto the depth model. The projection or overlay of the image data onto the depth model can be performed according to known techniques, for example, by registering the depth model with a color image to obtain color information for each point on the depth model.

[0062] In operation 332, method 300 causes the display to display the composite model and information regarding the section of the internal region. The information may include a visualization of the section of the internal region on the composite model. If the section of the internal region in the composite model is determined to be an area of ​​interest in operation 320, the information may include audio and / or visual cues and instructions for the clinician to navigate the medical instrument 140 to the section of the internal region. The composite model may be interactive on the display. For example, the composite model may be rotatable on the x-, y-, and / or z-axes, may undergo zoom-in / zoom-out operations, may undergo selection of specific regions, and / or may undergo other operations commonly known to exist for interactive 3D models. Interactions may be performed by the clinician via the input device 128 and / or directly on the touch display.

[0063] It should be understood that the operations of Figure 3 may be fully automated. For example, no user or clinician input is required throughout operations 304-332, other than guiding the medical instrument 140 or other device with the image and / or depth camera, if desired. In this case, the alternative depth model in operation 316 is automatically generated and applied, and the region of interest is automatically selected. The automatic generation and application of the alternative depth model and the automatic selection of the region of interest may be assisted by the neural network 124, the database 120, the processor 116, and / or the memory 112.

[0064] FIG. 4 illustrates a method 400 according to at least one exemplary embodiment. For example, FIG. 4 illustrates additional operations that may be performed in addition to or instead of the operations shown in FIG. 3 according to at least one exemplary embodiment. Operations shown in FIG. 4 that have the same reference numbers as FIG. 3 are performed in the same manner as described above with reference to FIG. 3. Therefore, these operations will not be described in detail below. FIG. 4 differs from FIG. 3 by including operations 302, 310, and 314. FIG. 4 relates to an example in which a clinician identifies a region of interest for testing and identifies a time point at which the region of interest is expected to be tested.

[0065] In operation 302, method 400 receives a first input from a clinician identifying a region of interest within the patient's interior region. The first input may be entered by the clinician at input device 128 to indicate where the region of interest begins and ends. For example, the clinician may identify a start point and an end point or otherwise mark (e.g., surround) the region of interest on the alternative depth model discussed in operation 316, where the alternative depth model may be a patient-general model, a patient-specific model, or a combination of both. As described above, the region of interest may be determined or aided by neural network 124, which uses historical data regarding other regions of interest in other medical procedures to conclude that the same region in the patient's interior region is also a region of interest. In this case, neural network 124 identifies areas on the alternative depth model that may be of interest, and the clinician can confirm or doubt that each area is a region of interest using input at input device 128.

[0066] In at least one exemplary embodiment, the first input can identify a region of interest within a patient's interior region without using a separate depth model. In this case, the first input can use the clinician's general knowledge of the interior region and the tracked position of the medical instrument 140 within the interior region to flag start and end points within the interior region itself. In other words, the start point of the region of interest can be a known or estimated distance from the camera's 136 entry point into the patient, and the end point of the region of interest can be another known or estimated distance from the entry point (or alternatively, the start point of the region of interest). Tracking the position of the camera 136 within the interior region according to known techniques (e.g., SLAM) can determine when the camera 136 enters the start and end points of the region of interest. For example, if the clinician knows that the region of interest begins 15 cm from the camera's 136 entry point and ends 30 cm from the entry point, other sensors on the camera 136 can provide information to the processor 116 to estimate when the camera 136 enters and exits the region of interest. The clinician can trigger the start and end points, for example, by pressing a button on an external control portion of the medical instrument 140 .

[0067] Although operation 302 is shown as being performed before operation 304, operation 302 may be performed at any time before operation 310.

[0068] 3 above to generate image data and depth data, and generate a depth model from the depth data. Operation 302 may also be performed at more than one point prior to operation 310, for example, at a first point during the medical procedure to indicate the start of the region of interest and at a second point during the medical procedure to indicate the end of the region of interest. Additionally, indications of the start and end points of multiple regions of interest may be configured.

[0069] In operation 310, method 400 receives a second input from a clinician during the medical procedure indicating that a region of interest has been examined in the interior region. The second input may be entered on input device 128 in the same or similar manner as the first input in operation 302. For example, during the medical procedure, the clinician is informed of the region of interest selected in operation 302 through a display of the depth model, another depth model, and / or a composite model. When the clinician believes that the region of interest in the interior region has been examined, the clinician provides the second input during the medical procedure. Operation 310 serves as a trigger to proceed to operation 314.

[0070] In operation 314, after receiving second input from the clinician in operation 310, method 300 determines that the region of interest includes a section of the interior region that is missing image data. In other words, operation 344 serves as a double check on the clinician's confidence that the entire region of interest has been inspected. If, in operation 314, method 400 determines that a section of the interior region that is missing data exists within the region of interest, the method proceeds to operation 324, which is performed according to the description of FIG. 3 . Otherwise, method 400 returns to operation 304 and continues to generate image data and depth data for the interior region. If method 400 proceeds to operation 324, the one or more warnings may include a warning informing the clinician that at least a portion of the region of interest remained uninspected.

[0071] Act 314 can be performed in the same or similar manner as act 312 of Figure 3. For example, to determine whether the region of interest includes a section of an interior region that is missing image data, method 400 evaluates whether the depth model generated in act 308 is missing more than a threshold amount of depth data, where the missing depth data is in an area that corresponds to a portion of the region of interest. Similar to the method of Figure 3, method 400 includes mapping the region of interest selected in act 302 to the depth model generated in act 308 according to known techniques.

[0072] It should be appreciated that method 400 provides a clinician or other user with the ability to provide input for selecting an area of ​​interest and / or to double-check the clinician's confidence that the area of ​​interest has been completely examined.

[0073] FIG. 5 illustrates a workflow 500 for a medical procedure in accordance with at least one exemplary embodiment. The operations of FIG. 5 are described with reference to FIGS. 1-4 to illustrate how the elements and operations of FIGS. 1-4 fit within the workflow of a medical procedure for a patient. While the operations of FIG. 5 are described in numerical order, it should be understood that one or more operations may occur at different times than illustrated and / or may occur simultaneously with other operations. As with FIGS. 3 and 4, the operations of FIG. 5 may be performed by one or more elements within system 100.

[0074] In operation 504, workflow 500 includes generating another model, for example, a 3D depth model, with a pre-selected region of interest (see, e.g., operations 302 and 316). Operation 504 may include generating information regarding the relative position, shape, and / or size of the region of interest and passing that information to operation 534, which is described in more detail below.

[0075] In operation 508, the camera system (eg, cameras 136a and 136b) collects image data and depth data of a medical procedure being performed by a clinician, according to the discussion of FIGS.

[0076] In operation 512, the depth and time data is used to construct a 3D depth model, while in operation 516 the image data and time data are used along with the depth model to align the depth data with the image data. For example, the time data for each of the image data and depth data may include a timestamp for each of the frames or still images taken by camera 136 such that in operation 516 processor 116 can match the timestamp of the image data with the timestamp of the depth data, thereby ensuring that the image data and depth data are aligned with each other at their respective instants in time.

[0077] The image data is projected onto the depth model to form a composite model as a 3D color image model of the interior region in act 520. The 3D composite model and the temporal data can be used to aid in navigation of the camera 136 and / or medical instrument 140 in act 524 and displayed in a display user interface in act 528.

[0078] At operation 524, the workflow 500 performs a navigation operation, which may include generating a direction from the current position of the camera 136 to the nearest and / or largest uninspected area. The instructions may be generated as audio and / or visual instructions on a user interface at operation 528. Exemplary audio instructions include audible "left, right, up, down" instructions, while exemplary video instructions include visual left, right, up, and down arrows on the user interface. The length and / or color of the arrows may change as the clinician navigates toward the uninspected area. For example, the arrows may become shorter and / or change color as the camera 136 approaches the uninspected area.

[0079] In operation 528, the user interface displays or generates various information related to the medical procedure. For example, the user interface may include a warning that an area has not been inspected, statistics about the uninspected area (e.g., how likely the uninspected area is to contain something of interest), a visualization of the uninspected area, an interactive 3D model of the interior area, navigation graphics, audio instructions, and / or any other information related to the medical procedure that may be potentially useful to a clinician.

[0080] Operation 532 includes receiving the depth model from operation 512 and detecting one or more uninspected regions of the interior region based on depth data missing from the depth model, e.g., as in operation 312 above.

[0081] Act 534 includes receiving information about the uninspected region, e.g., information about the relative position, shape, and / or size of the uninspected region. Act 534 further includes using this information to perform feature matching with the depth model from act 512 and another model from act 534. Feature matching between the models may be performed according to any known technique, which may utilize mesh modeling concepts, point cloud concepts, scale invariant feature transform (SIFT) concepts, and / or the like.

[0082] Next, workflow 500 proceeds to operation 536, where it is determined, based on feature matching, whether the uninspected regions are regions of interest at operation 534. This determination may be performed, for example, according to operation 320 described above. Information about the uninspected regions and whether they are of interest is passed to operations 524 and 528. For example, if the uninspected regions are regions of interest, that information is used in operation 524 to generate information guiding the clinician to the largest uninspected region closest to the current location. The direction generated in operation 524 may be displayed in a user interface at operation 528. Additionally or alternatively, if it is determined that the uninspected region is not of interest, a notification thereof may be sent to the user interface along with information about the location of the uninspected region that is not of interest. This allows the clinician to double-check whether the region is indeed not of interest. The clinician can then indicate that the region is of interest, and instructions to that region may be generated as in operation 524.

[0083] 6 shows examples of output devices 104A and 104B as displays, e.g., flat panel displays. Although two output devices are shown, more or fewer output devices may be included, as desired.

[0084] In at least one exemplary embodiment, output device 104A displays a live depth model of the current medical procedure. Various functions may be available for interacting with the depth model, which may include zooming (in and out), rotation (x-, y-, and / or z-axis rotation), region selection, and / or the like. Output device 104A may further display a live 2D video or still image feed of the interior region from camera 136a. Output device 104A may further display one or more warnings, such as warnings about missing data in the live depth model, warnings indicating that a region has not been inspected, etc. Output device 104A may also display various information, such as graphics for navigating medical instrument 140 to uninspected regions, statistics about the medical procedure and / or uninspected regions, etc.

[0085] Output device 104B can display an interactive composite 3D model with image data overlaid or projected onto the depth model. Various functions, which may include zooming (in and out), rotation (x-, y-, and / or z-axis rotation), region selection, and / or the like, may be available for interacting with the composite 3D model. Similar to output device 104A, output device 104B can display warnings and / or other information related to the medical procedure. Displaying a live depth and image feed and the 3D composite model during a medical procedure can help ensure that all areas are examined.

[0086] Output devices 104A and / or 104B may further display the real-time location of medical instrument 140 and / or other devices with camera 136 within the depth model and / or composite model. When detecting an uninspected area, the aforementioned navigation arrows may be displayed on the model and may change color, the rate at which they may flash, and / or the length according to how close or far the camera is to the uninspected area.

[0087] It should be understood that the operations in Figures 3-5 do not necessarily have to be performed in the order shown and described, and those skilled in the art should understand that other operations in Figures 3-5 may be rearranged according to design preference.

[0088] Although the exemplary embodiments have been described with respect to medical procedures occurring inside a patient's body, the exemplary embodiments may also be applied to camera-assisted non-medical procedures of internal body areas (e.g., inspection of the trachea or other structures that are difficult to inspect from an external perspective).

[0089] In light of the foregoing, it should be appreciated that the exemplary embodiments provide an efficient method for automatically identifying potential uninspected areas of the anatomy and providing appropriate warnings and / or instructions to guide the clinician-user through the uninspected areas, thereby ensuring that all intended areas are inspected.

[0090] At least one exemplary embodiment is directed to a device including a memory including instructions and a processor that executes the instructions to generate, during a medical procedure being performed by a clinician on a patient's internal region, image data and depth data of the internal region; generate, during the medical procedure, a depth model of the patient's internal region based on the depth data; determine, based on the depth model, that the image data of the medical procedure does not include image data of a section of the internal region; and generate one or more warnings alerting the clinician that the section of the internal region has not been examined.

[0091] According to at least one exemplary embodiment, the instructions include instructions to cause a processor to generate a composite model of the internal region based on image data and a depth model of the medical procedure, and to display information about the composite model and sections of the internal region.

[0092] According to at least one exemplary embodiment, the composite model includes a three-dimensional model of the interior region with image data of the medical procedure projected onto the depth model.

[0093] According to at least one exemplary embodiment, the one or more alerts include an alert displayed on a display.

[0094] According to at least one exemplary embodiment, the information includes a visualization of a section of an interior region of the composite model.

[0095] According to at least one exemplary embodiment, the information includes visual and / or audio cues and instructions for the clinician to navigate the medical instrument to a section of the interior region.

[0096] According to at least one exemplary embodiment, the instructions include instructions that cause the processor to determine that a section of the interior region is a region of interest based on another depth model that is general to or specific to interior regions, and the one or more warnings include a warning that notifies a clinician that the section of the interior region should be examined.

[0097] According to at least one exemplary embodiment, the instructions include instructions to cause the processor to receive a first input from a clinician identifying a region of interest within an interior region of a patient, and to receive a second input from the clinician during a medical procedure indicating that the region of interest has been examined.

[0098] According to at least one exemplary embodiment, the instructions include instructions for causing the processor to determine, after receiving a second input from the clinician, that the region of interest includes a section of the interior region that is missing data, and the one or more warnings include a warning informing the clinician that at least a portion of the region of interest was left unexamined.

[0099] According to at least one exemplary embodiment, the processor generates the depth model in response to determining that a medical instrument used in a medical procedure enters the region of interest.

[0100] According to at least one example embodiment, the processor determines that the image data does not include image data for a section of an interior region when the region of the depth model lacks depth data in excess of a threshold amount.

[0101] According to at least one exemplary embodiment, the instructions include instructions to cause the processor to execute a first machine learning algorithm to determine a region of interest within the interior region and determine a path for navigating the medical instrument to the region of interest, and to execute a second machine learning algorithm to cause the robotic device to navigate the medical instrument to the region of interest within the interior region.

[0102] At least one exemplary embodiment is directed to a system including a display, a medical instrument, and a device including a memory including instructions and a processor that executes the instructions to generate, during a medical procedure being performed by a clinician on the internal region of a patient, image data and depth data of the internal region, generate, during the medical procedure, a depth model of the patient's internal region based on the depth data, determine, based on the depth model, that the image data of the medical procedure does not include image data of a section of the internal region, and cause / generate one or more alerts to alert the clinician that the section of the internal region has not been examined.

[0103] In accordance with at least one exemplary embodiment, the medical instrument includes a stereoscopic camera that provides image data, and the depth data is derived from the image data.

[0104] According to at least one exemplary embodiment, the medical instrument includes a depth sensor that provides depth data and an image sensor that provides image data, the depth sensor and the image sensor being positioned to have overlapping fields of view on the medical instrument.

[0105] According to at least one exemplary embodiment, a medical instrument includes a sensor including depth pixels that provide depth data and imaging pixels that provide image data.

[0106] According to at least one exemplary embodiment, a system includes a robotic device for navigating a medical instrument within an interior region, the instructions including instructions for causing a processor to execute a first machine learning algorithm to determine a region of interest within the interior region, determine a path for navigating the medical instrument to the region of interest, and execute a second machine learning algorithm to cause the robotic device to navigate the medical instrument to the region of interest within the interior region.

[0107] According to at least one exemplary embodiment, the system includes an input device that receives input from a clinician to approve a path for navigating the medical instrument to the region of interest before the processor executes the second machine learning algorithm.

[0108] At least one exemplary embodiment is directed to a method that includes generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model of the patient's internal region based on the depth data during the medical procedure; determining, based on the depth model, that the image data does not include image data for a section of the internal region; and generating one or more warnings that alert the clinician that the section of the internal region has not been examined.

[0109] According to at least one exemplary embodiment, a method includes generating an interactive three-dimensional model of the internal region using image data of the medical procedure projected onto the depth model, and displaying on a display the interactive 3D model and visual and / or audio cues and instructions to guide a clinician performing the medical procedure through sections of the internal region.

[0110] Any one or more of the aspects / embodiments substantially as disclosed herein.

[0111] Any one or more aspects / embodiments substantially disclosed herein may be optionally combined with any one or more other aspects / embodiments substantially disclosed herein.

[0112] One or more means adapted to implement any one or more of the above aspects / embodiments substantially as disclosed herein.

[0113] The terms "at least one," "one or more," "or," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B, and / or C," and "A, B, or C" means A alone, B alone, C alone, A and B in combination, A and C in combination, B and C in combination, or A, B, and C in combination.

[0114] An occurrence of the term "a" or "an" refers to one or more of that occurrence. Thus, the terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" may be used interchangeably.

[0115] Aspects of the present disclosure may take the form of entirely hardware embodiments, entirely software (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which may be generally referred to herein as "circuits," "modules," or "systems," and any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0116] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0117] As used herein, the terms "determine," "calculate," "compute," and variations thereof are used interchangeably and include any type of methodology, process, mathematical operation, or technique.

[0118] An exemplary embodiment is configured as follows. (1) A device comprising a memory containing instructions and a processor that executes the instructions to generate image data and depth data of an internal region of a patient during a medical procedure being performed by a clinician on the internal region, generate a depth model of the patient's internal region based on the depth data during the medical procedure, determine based on the depth model that the image data of the medical procedure does not include image data of a section of the internal region, and generate one or more alerts alerting the clinician that the section of the internal region has not been examined. (2) the instructions to the processor: generating a composite model of the interior region based on the image data of the medical procedure and the depth model; The device of (1), further comprising instructions for causing a display to display information regarding the composite model and the section of the interior region. (3) One or more devices of (1) to (2), wherein the composite model includes a three-dimensional model of the internal region with the image data of the medical procedure projected onto the depth model. (4) One or more devices among (1) to (3), wherein the one or more warnings include a warning displayed on the display. (5) One or more devices of (1) to (4), wherein the information includes a visualization of the section of the interior region of the composite model. (6) One or more of the devices of (1) to (5), wherein the information includes visual and / or audible cues and instructions for the clinician to navigate medical instruments to the section of the internal area. (7) the instructions to the processor: One or more devices of (1) to (6), including instructions for determining that the section of the internal region is a region of interest based on another depth model that is general to or specific to the internal region, and the one or more warnings include a warning to notify the clinician that the section of the internal region should be examined. (8) the instructions to the processor: receiving a first input from the clinician identifying a region of interest within the interior region of the patient; and receiving a second input from the clinician during the medical procedure indicating that the region of interest has been examined; A device according to one or more of (1) to (7), comprising instructions to cause the device to perform the following. (9) the instructions include instructions for causing the processor, after receiving the second input from the clinician, to determine that the region of interest includes the section of the interior region; The one or more devices of (1) to (8), wherein the one or more warnings include a warning informing the clinician that at least a portion of the region of interest remained unexamined. (10) One or more devices of (1) to (9), wherein the processor generates the depth model in response to a determination that a medical instrument used in the medical procedure enters the region of interest. (11) One or more devices of (1) to (10), wherein the processor determines that the image data does not include image data of the section of the internal region if more than a threshold amount of depth data is missing in the region of the depth model. (12) One or more devices of (1) to (11), wherein the instructions include instructions to cause the processor to: execute a first machine learning algorithm to determine a region of interest within the internal region and determine a path for navigating a medical instrument to the region of interest; and execute a second machine learning algorithm to cause a robotic device to navigate the medical instrument to the region of interest within the internal region. (13) a display; Medical equipment and Device and the device comprises: a memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model of the interior region of the patient based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data for a section of the interior region; and generating one or more warnings alerting the clinician that the section of the interior region has not been examined. A system comprising a processor for executing said instructions. (14) The system of (13), wherein the medical instrument includes a stereoscopic camera that provides the image data, and the depth data is derived from the image data. (15) One or more of the systems of (13) to (14), wherein the medical instrument includes a depth sensor that provides the depth data and an image sensor that provides the image data, and the depth sensor and the image sensor are positioned on the medical instrument so as to have overlapping fields of view. (16) One or more of the systems of (13) to (15), wherein the medical instrument includes a sensor including depth pixels that provide the depth data and imaging pixels that provide the image data. (17) One or more of the systems of (13) to (16), further comprising a robotic device for navigating a medical instrument within the internal region, wherein the instructions include instructions to cause the processor to execute a first machine learning algorithm to determine a region or set of regions of interest within the internal region and determine a path for navigating to the region of interest, and execute a second machine learning algorithm to cause the robotic device to navigate to the region of interest within the internal region. (18) One or more of the systems of (13) to (17), further comprising an input device that receives input from the clinician to approve the path for navigating to the region of interest before the processor executes the second machine learning algorithm. (19) generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model of the interior region of the patient based on the depth data during the medical procedure; determining, based on the depth model, that the image data does not include image data for a section of the interior region; and and causing one or more warnings to alert the clinician that the section of the interior area has not been examined. (20) The method of (19), further comprising: generating an interactive three-dimensional model of the internal region using the image data of the medical procedure projected onto the depth model; and displaying on the display the interactive three-dimensional model and visual and / or auditory cues and directions to guide a clinician performing the medical procedure to the section of the internal region.

Claims

1. A memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with the image data and the depth data have an overlapping area; The instructions cause the processor to: generating a composite model of the interior region based on the image data of the medical procedure and the depth model; displaying information about the composite model and the section of the interior region on a display; and instructions for determining that the section of the interior region is a region of interest based on another depth model that is general to the interior region or specific to the interior region; The one or more warnings include a warning to notify the clinician that the section of the interior area has not been inspected.

2. A memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with the image data and the depth data have an overlapping area; The instructions cause the processor to: generating a composite model of the interior region based on the image data of the medical procedure and the depth model; displaying information about the composite model and the section of the interior region on a display; receiving a first input from the clinician identifying a region of interest within the interior region of the patient; and receiving a second input from the clinician during the medical procedure indicating that the region of interest has been examined; A device containing instructions.

3. The instructions include instructions for causing the processor to determine, after receiving the second input from the clinician, that the region of interest includes the section of the interior region; The device of claim 2 , wherein the plurality of warnings includes a warning informing the clinician that the at least a portion of the region of interest remained unexamined.

4. The device of claim 2 , wherein the processor generates the depth model in response to determining that a medical instrument used in the medical procedure enters the region of interest.

5. A memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with the image data and the depth data have an overlapping area; The device, wherein the processor determines that the image data does not include image data for the section of the interior region if more than a threshold amount of depth data is missing in that region of the depth model.

6. A memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with the image data and the depth data have an overlapping area; The instructions cause the processor to: executing a first machine learning algorithm to determine a region of interest within the interior region and to determine a path for navigating a medical instrument to the region of interest; and causing a robotic device to execute a second machine learning algorithm to navigate the medical instrument to the region of interest within the interior region. A device containing instructions.

7. A display; Medical equipment and a device, The device is a memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with The system, wherein the medical instrument includes a stereoscopic camera that provides the image data, and the depth data is derived from the image data.

8. A display; Medical equipment and a device, The device is a memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with The system, wherein the medical instrument includes a depth sensor that provides the depth data and an image sensor that provides the image data, the depth sensor and the image sensor being positioned on the medical instrument such that they have overlapping fields of view.

9. A display; Medical equipment and a device, The device is a memory containing instructions; generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model, which is a three-dimensional model of an interior region of the patient, based on the depth data during the medical procedure; determining, based on the depth model, that the image data of the medical procedure does not include image data of the section of the interior region; and generating one or more warnings alerting the clinician that a section of the interior region has not been examined; A processor that executes instructions Equipped with The system, wherein the medical instrument includes a sensor including depth pixels that provide the depth data and imaging pixels that provide the image data.

10. further comprising a robotic device for navigating the medical instrument within the interior region; The instruction tells the processor: Executing a first machine learning algorithm to determine a region or set of regions of interest within the interior region and to determine a path for navigating to the region of interest; and Executing a second machine learning algorithm to cause the robotic device to navigate to a region of interest within the interior region. The system of claim 9 including instructions.

11. 11. The system of claim 10, further comprising an input device that receives input from the clinician to approve the path for navigating to the region of interest before the processor executes the second machine learning algorithm.

12. generating image data and depth data for an internal region of a patient during a medical procedure being performed by a clinician on the internal region; generating a depth model of the interior region of the patient based on the depth data during the medical procedure; determining, based on the depth model, that the image data does not include image data for a section of the interior region; and generating one or more warnings alerting the clinician that the section of the interior region has not been examined; The system according to any one of claims 7 to 11 is implemented method.

13. moreover, generating an interactive three-dimensional model of the interior region using the image data of the medical procedure projected onto the depth model; and The display includes the interactive three-dimensional model and visual and / or audio cues and directions to guide a clinician performing the medical procedure through the section of the interior area. Display The method of claim 12.

Images