Lung lobule segmentation and measurement of nodule distance to lobe boundaries.

Abstract

A method for segmenting an anatomical feature into a plurality of parts may include receiving an image of the anatomical feature; segmenting the image into a plurality of parts based on a trained image segmentation neural network, wherein each part of the plurality of parts is assigned a part label; determining a nodule location associated with a nodule within the anatomical feature; assigning the nodule to a part of the plurality of parts in the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the part and a second point away from the boundary of the part; and generating a distance-coded image based on the distance metric, wherein the distance-coded image indicates a distance from the boundary of the part to points outside the boundary based on a color scheme.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 476,147, filed December 19, 2022, entitled "LOBE SEGMENTATION OF LUNG AND MEASUREMENT OF NODULE DISTANCE TO LOBE BOUNDARY," the disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION The present disclosure relates to the field of medical procedures. [Background technology]

[0003] Various medical procedures involve the use of one or more machine-generated images, which may be utilized to provide visualization that confers particular benefits during the medical procedure. Certain operational processes may be guided based at least in part on the machine-generated images. Summary of the Invention [Means for solving the problem]

[0004] Described herein are systems, devices, and methods for facilitating the identification and segmentation of various anatomical features based on images of such features acquired using a scope device or other medical instrument. Such feature identification and / or segmentation can facilitate the manipulation of particular anatomical features relevant to a medical procedure, such as, for example, a bronchoscopy, a lung biopsy, a lung nodule treatment, or other procedures that access the respiratory system.

[0005] In some aspects, the techniques described herein relate to a computer-implemented method that includes receiving an image of an anatomical feature; segmenting the image into a plurality of portions based on a trained image segmentation neural network, wherein each portion of the plurality of portions is assigned a portion label; determining a nodule location associated with a nodule within the anatomical feature; assigning the nodule to a portion of the plurality of portions in the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point away from the boundary of the portion; and generating a distance-coded image based on the distance metric, wherein the distance-coded image indicates a distance from the boundary of the portion to points outside the boundary based on a color scheme.

[0006] In some aspects, the technology described herein relates to a method, wherein determining a nodule location associated with a nodule within an anatomical feature includes receiving input from a user identifying the nodule location.

[0007] In some aspects, the technology described herein relates to a method, wherein determining a nodule location associated with a nodule in an anatomical feature includes analyzing an image to identify the nodule location.

[0008] In some aspects, the techniques described herein relate to a method, further comprising generating a partial image excluding other portions of the plurality of portions based on portion labels assigned to the portions, wherein generating the distance-coded image involves assigning colors to the partial images based on a color scheme.

[0009] In some aspects, the techniques described herein relate to a method, wherein generating a distance-coded image includes assigning a color to pixels in the distance-coded image based on a respective shortest distance between a first pixel away from a boundary of a portion in the sub-image and a second pixel on the boundary.

[0010] In some aspects, the techniques described herein relate to a method, further comprising providing a distance from a location to a boundary of the portion based on the distance-coded image.

[0011] In some aspects, the technology described herein relates to a method, wherein providing a distance from a location to a boundary of a portion based on a distance-coded image includes determining a distance between the location and the boundary of the portion, wherein the distance is based on at least one of Euclidean coordinates or polar coordinates.

[0012] In some aspects, the techniques described herein relate to a method, wherein generating a partial image based on a part label assigned to the part further includes assigning a single color to points outside the boundary of the part.

[0013] In some aspects, the techniques described herein relate to a method, further comprising assigning different colors to portions within the sub-image to generate a binary image, wherein the anatomical feature is a lung, the portion is a lobe, and the multiple portions are multiple lobes.

[0014] In some aspects, the techniques described herein relate to a method, wherein determining a nodule location associated with a nodule in an anatomical feature includes determining a nodule centroid, and assigning the nodule to a portion of a plurality of portions in the image includes determining a portion label associated with the nodule centroid.

[0015] In some aspects, the techniques described herein relate to a method, wherein assigning a nodule to a portion of a plurality of portions in an image further includes determining a pixel value associated with the nodule centroid.

[0016] In some aspects, the technology described herein relates to a method, wherein generating a distance-coded image based on a distance metric includes passing the sub-image through a distance map filter.

[0017] In some aspects, the technology described herein relates to a method, wherein the color scheme comprises grayscale colors.

[0018] In some aspects, the technology described herein relates to a method, wherein a trained image segmentation neural network identifies one or more pleura or fissures, and part labels are assigned to a plurality of parts based on the pleura or fissures.

[0019] In some aspects, the technology described herein relates to a method, wherein the trained image segmentation neural network is a UNet convolutional neural network.

[0020] In some aspects, the techniques described herein relate to a system including a processor and a memory storing computer-executable instructions that cause the processor to perform steps including: segmenting an anatomical feature image into a plurality of portions based on a neural network trained to label at least a portion of the anatomical feature image; selecting a portion of the plurality of portions based on coordinates of a nodule location corresponding to a pixel associated with the portion; mapping pixel distances to a range of values; and assigning a value to the first pixel based on (i) a distance between a first pixel away from a boundary of the portion and a second pixel on the boundary of the portion and (ii) the range of values.

[0021] In some aspects, the techniques described herein relate to a system further including determining a second pixel based on the second pixel having the shortest distance to the first pixel among pixels on a boundary of a portion.

[0022] In some aspects, the technology described herein relates to a system or claim 17, wherein the anatomical feature is a lung, the portion is a lung lobe, and the plurality of portions are a plurality of lung lobes, and mapping the pixel distance to a range of values ​​includes mapping the pixel distance to a range of grayscale values.

[0023] In some aspects, the technology described herein relates to a medical system including an endoscope having a position sensor associated with a distal end thereof, a robotic medical system including a plurality of articulating arms, and control circuitry communicatively coupled to the endoscope and the robotic medical system, the control circuitry being configured to: receive a distance-coded image associated with a treatment site, wherein colors in the distance-coded image are assigned values ​​based on a distance of pixels to a boundary of a lobe encompassing the treatment site; determine that the distal end is below a threshold distance to the boundary based on referencing the color of the distance-coded image; and generate a notification that the distal end is at risk of contacting the boundary.

[0024] In some aspects, the technology described herein relates to a medical system, wherein the control circuitry is further configured to limit control of the endoscope based on a risk of the distal tip contacting a boundary. For purposes of summarizing this disclosure, certain aspects, advantages, and novel features have been described. It is understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein. [Brief explanation of the drawings]

[0025] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an embodiment of a robotic medical system, according to one or more embodiments. [Figure 2]2 illustrates an example device that may be implemented in the medical system of FIG. 1, according to one or more embodiments. [Figure 3] 1 illustrates a bronchoscope positioned in a portion of the respiratory system, according to one or more embodiments. [Figure 4] 1 illustrates lobular segments of a patient's respiratory system. [Figure 5-1] FIG. 5 is a flow diagram illustrating a process for providing a distance-coded image according to one or more embodiments (represented in section 5-1). [Figure 5-2] FIG. 5 is a flow diagram illustrating a process for providing a distance-coded image according to one or more embodiments (represented in section 5-2). [Figure 6-1] 6-1) illustrates certain images corresponding to various blocks, states, and / or operations associated with the process of FIG. 5, according to one or more embodiments. [Figure 6-2] 6-2) illustrates certain images corresponding to various blocks, states, and / or actions associated with the process of FIG. 5, according to one or more embodiments. [Figure 7] FIG. 1 is a flow diagram illustrating a process for assigning nodes to leaves in accordance with one or more embodiments. [Figure 8] 8 illustrates certain images corresponding to various blocks, states, and / or operations associated with the process of FIG. 7, in accordance with one or more embodiments. [Figure 9] 1 illustrates a lobule segmentation framework, according to one or more embodiments. [Figure 10] 1 illustrates an exemplary distance-coded image generation architecture, according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0026] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. While certain preferred embodiments and examples are disclosed below, the inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Accordingly, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described sequentially as multiple separate operations in a manner that may be helpful in understanding a particular embodiment; however, the order of description should not be construed to imply that these operations are order-dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments will be described. Not necessarily all such aspects or advantages will be realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0027] Certain standard anatomical terms of location are used herein to refer to animal, i.e., human, anatomical structures with respect to preferred embodiments. While certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another, it should be understood that these terms are used herein for ease of description to describe positional relationships between elements / structures, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures during use or operation in addition to the orientation shown in the drawings. For example, an element / structure described as being "above" another element / structure may represent a position below or to the side of such other element / structure, and vice versa, relative to the intended patient or alternative orientations of the element / structure.

[0028] The present disclosure relates to systems, devices, and methods for identifying and segmenting target anatomical features within a patient's body cavity / region to assist in certain medical procedures. While certain aspects of the present disclosure are described in detail herein in the context of an endoscopic procedure, such as a bronchoscopy procedure, it should be understood that such context is provided for convenience and clarity, and that the anatomical feature identification and segmentation concepts disclosed herein are applicable to any suitable medical procedure. Furthermore, certain embodiments of robot-enabled medical procedures are disclosed herein in the context of pulmonary nodule treatment. However, while certain principles disclosed herein are particularly applicable to pulmonary and respiratory system anatomy, it should be understood that the anatomical feature identification and segmentation concepts disclosed herein may be implemented or configured for implementation in any suitable or desired anatomy.

[0029] In medical procedures involving the respiratory system, the risk of pneumothorax can pose a significant challenge. As described in more detail below, a typical lung generally includes fissures and pleura that segment or otherwise divide the lung into lobes. Separated lobes can provide respiratory function redundancy and help prevent the spread of disease under normal circumstances. Accidental puncture of the lobe boundaries formed by such fissures and pleura can result in pneumothorax, which is often associated with a higher likelihood of complications and slower recovery. Therefore, accurate, sometimes real-time, measurement of the distance from lobe boundaries to target anatomical features, such as nodules, during pulmonary nodule treatment procedures can significantly reduce the risk of pneumothorax.

[0030] However, relying on clinicians to manually measure the distance from the nodule to the lung border can be difficult and cumbersome. Typically, clinicians display 3D computed tomography (CT) images in 2D slice multiplanar reconstruction (MPR) views and take several measurements in different views to determine where the nodule is located relative to the fissure and pleura. This manual process is often too time-consuming and inefficient when real-time issues arise.

[0031] Disclosed herein are concepts for identifying and segmenting specific anatomical features that can perform such measurements automatically and provide distance-coded images that readily encapsulate distance information within an image as visual information. All, or substantially all, of the processes involved in generating a distance-coded image can be performed by a computing system utilizing some form of artificial intelligence framework, such as a deep learning architecture. For example, the framework can employ a neural network trained to identify fissures and pleura within a received CT lung image and semantically segment the image into lobes based on the fissures and pleura. The architecture can then calculate the distance from the lobe boundary to a given pixel within the CT lung image and generate a distance-coded image. Because the distance-coded information visually encapsulates distance information, clinicians are freed from performing manual distance measurements and can instead spend more time focusing on surgical planning and execution.

[0032] Further disclosed herein are certain robotic systems that can be used to perform various medical procedures, such as endoscopic and laparoscopic procedures. During certain procedures, a medical instrument, such as a robotically controlled medical instrument (e.g., a working instrument, such as an endoscope, an access sheath, or a needle instrument), is inserted into a patient's body. Within the patient's body, the instrument may be positioned within the patient's luminal network or other anatomical structure. As used herein, the term "luminal network" refers to any hollow structure within the body, whether it includes multiple lumens or branches (e.g., multiple branching lumens or blood vessels, such as in the lungs) or a single lumen or branch (e.g., in the digestive tract). During such procedures, the instrument may be moved (e.g., advanced, retracted, navigated, guided, driven, etc.) through the luminal network to one or more regions of interest, such as the location of a lesion (e.g., a tumor, nodule, etc.).

[0033] medical system FIG. 1 illustrates an example medical system 10 for performing various medical procedures according to aspects of the present disclosure. The medical system 10 includes a robotic system 11 configured to engage and / or control a medical instrument 32 and perform a procedure on a patient 13. The medical system 10 also includes a control system 50 configured to interface with the robotic system 11, provide information regarding the procedure, and / or perform various other operations. For example, the control system 50 may include a display 42 for presenting certain information and assisting the physician 5. The medical system 10 may include a platform 15 configured to hold the patient 13. The medical system 10 may further include an electromagnetic (EM) field generator (not shown, see FIG. 3). The EM field generator may be carried by one or more of the robotic arms 12 of the robotic system 11 or may be a stand-alone device.

[0034] The robotic-enabled medical system 10 may be configured in various ways depending on the particular procedure. During an endoscopic procedure (e.g., bronchoscopy), the medical system 10 may utilize the robotic arm 12 to deliver a medical instrument, such as a steerable endoscope 32 (e.g., a bronchoscope), through a natural orifice access point (e.g., the mouth 9 of a patient 13, positioned on a table 15 in this example) to deliver diagnostic and / or therapeutic tools / instruments. As shown, the robotic system 11 may be positioned adjacent to the patient's upper torso to provide access to the access point. Similarly, the robotic arm 12 may be actuated to position the steerable endoscope 32 relative to the access point. While described in the context of a bronchoscopy procedure, it should be understood that the robotic system 11 may be implemented for other types of procedures, such as gastrointestinal (GI) procedures involving a gastroscope or other specialized endoscope.

[0035] Once the robotic system 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 32 into the patient robotically, manually, or a combination thereof. The steerable endoscope 32 can be advanced within the outer access sheath 40, which in some implementations can be coupled to and / or controlled by one or more robotic arms. For example, the steerable endoscope 32 and the sheath 40 can each be coupled to a separate instrument driver / manipulator from a set of instrument drivers / manipulators 28, with each instrument driver coupled to the distal end of a respective robotic arm 12. The instrument drivers 28 can be configured in a linear arrangement that facilitates coaxial alignment of the steerable endoscope 32 and the sheath 40 along a “virtual rail” 33 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. Translation of the instrument driver 28 along the virtual rail 33 can nest the steerable endoscope 32 relative to the outer sheath 40 or advance or retract the steerable endoscope 32 from the patient.

[0036] In some embodiments, the medical system 10 may be used to perform locally targeted procedures, such as bronchoscopy-assisted lung nodule treatment. Bronchoscopes typically include an endoscope at their distal end configured to allow visualization of the airways. The terms "scope" and "endoscope" are used herein according to their broad and ordinary meanings and are understood to refer to any type of elongated medical instrument having imaging, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space in the body. For example, references herein to a scope or endoscope may refer to a bronchoscope, cystoscope, nephroscope, bronchoscope, arthroscope, colonoscope, laparoscope, borescope, or the like. A scope / endoscope may, in some cases, comprise a rigid or flexible tube and may be sized to be passed through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device.

[0037] By way of example, a patient 13 is undergoing a bronchoscopy. An endoscope 32 may be robotically navigated within the patient's 13 respiratory system. To enhance navigation through the patient's pulmonary network and / or reach a desired target, the endoscope 32 may be manipulated to telescope the steerable endoscope 32 out of the outer access sheath 40 to provide enhanced articulation and a larger bend radius. The use of a separate instrument driver 28 allows the steerable endoscope 32 and sheath 40 to be driven independently of one another.

[0038] Generally, the respiratory system includes certain passageways, blood vessels, organs, and muscles that aid the body in gas exchange between air and blood, and between blood and body cells. The respiratory system includes the upper respiratory tract, including the nose / nasal cavity, pharynx (i.e., throat), and larynx (i.e., voice box). The respiratory system also includes the lower respiratory tract, including the trachea 6, lungs 4 (4r and 4l), and various portions of the bronchial tree, including the alveoli and alveolar ducts, which contain clusters of small air sacs responsible for gas exchange between the lungs and pulmonary blood vessels. The bronchial tree is an exemplary luminal network that a robotically controlled instrument may navigate and utilize in accordance with the inventive solutions presented herein. However, although aspects of the present disclosure are presented in the context of a luminal network, including the bronchial network of the airways (e.g., lumens, branches) of a patient's lungs, some embodiments of the present disclosure may be implemented in other types of luminal networks, such as renal networks, cardiovascular networks (e.g., arteries and veins), the digestive tract, the urinary tract, etc. In general, luminal networks include three-dimensional structures. 1 represents the luminal network as a two-dimensional structure solely for ease of illustration. The organs of the lower respiratory tract are located inside the thoracic cavity, bounded by the sternum (i.e., chest bone) and rib cage in front, and the vertebrae (i.e., spine) in back, which collectively protect the lungs and other organs within the chest.

[0039] The trachea 6 may provide the main entrance to the lungs 4. Bronchi 7 branch from the trachea 6 to each lung 4, i.e., the left lung 4l and the right lung 4r. The trachea 6 is located just below the larynx (not shown) and provides the main airway to the lungs 4. The left 4l and right 4r lungs are responsible for providing oxygen to the capillaries and exhaling carbon dioxide. The bronchi 7 branch from the trachea 6 to each lung 4, forming an intricate network of passages that supply air to the lungs 4. The diaphragm is the primary respiratory muscle that contracts and relaxes to force air into the lungs. The trachea 6 is a tube that carries air in and out of the lungs 4. Each lung 4 is associated with tubes 7 called bronchi that connect to the trachea. The trachea and bronchi form the bronchial tree. The bronchial tree includes primary bronchi 71, which branch into smaller secondary bronchi 78 and tertiary bronchi 75, which terminate in even smaller tubes called bronchioles 77. Each bronchiole is connected to a cluster of alveoli (not shown). During the inspiratory phase of the respiratory cycle, air enters through the mouth and nose, down the throat into the trachea 6, passes through the left and right main bronchi 71 into the lungs 4, enters the smaller bronchial airways 78, 75, enters the smaller bronchiole 77, and enters the alveoli where the exchange of oxygen and carbon dioxide takes place.

[0040] The robotic system 11 can be coupled to any component of the medical system 10, such as the control system 50, the stage 15, the EM field generator (not shown, see FIG. 3 ), the steerable endoscope 32, and / or a surgical instrument (e.g., a needle). In some embodiments, the robotic system 11 is communicatively coupled to the control system 50. For example, the robotic system 11 can be configured to receive control signals from the control system 50 and perform actions such as positioning the robotic arm 12 in a particular manner, manipulating the steerable endoscope 32, etc. In response, the robotic system 11 can control components of the robotic system 11 and perform actions. In some embodiments, the robotic system 11 is configured to receive images and / or image data from the steerable endoscope 32 representing the internal anatomy of the patient 13, i.e., the respiratory system for the particular depiction of FIG. 1 , and / or transmit the image / image data to the control system 50 (which can then be displayed on the display 42 or other output device). Furthermore, in some embodiments, the robotic system 11 is coupled to components of the medical system 10, such as the control system 50, in a manner such that it can receive fluids, optics, power, etc. from the components. Additional exemplary details of the robotic system are discussed in more detail below with reference to FIG. 2.

[0041] The control system 50 can be configured to provide various functions to assist in the performance of a medical procedure. In some embodiments, the control system 50 is coupled to the robotic system 11 and can operate in cooperation with the robotic system 11 to perform a medical procedure on the patient 13. For example, the control system 50 can communicate with the robotic system 11 via a wireless or wired connection (e.g., to control the robotic system 11 and / or the steerable endoscope 32, receive images captured by the steerable endoscope 32, etc.), provide fluid to the robotic system 11 via one or more fluid channels, provide power to the robotic system 11 via one or more electrical connections, provide optics to the robotic system 11 via one or more optical fibers or other components, etc. Additionally, in some embodiments, the control system 50 can communicate with the needle and / or endoscope to receive positional data therefrom. Additionally, in some embodiments, the control system 50 can communicate with the table 15 to position the table 15 in a particular orientation or otherwise control the table 15. Additionally, in some embodiments, the control system 50 can communicate with an EM field generator (not shown) to control the generation of an EM field in an area surrounding the patient 13.

[0042] The control system 50 may include various I / O devices configured to assist the physician 5 or others in performing a medical procedure. For example, the control system 50 may include certain input / output (I / O) components configured to allow user input for controlling the steerable endoscope 32, such as navigating the steerable endoscope 32 within the patient 13. For example, joysticks, buttons, and / or other types of user input controls 312 may be used to control the articulation of the robotically controlled medical instruments, including the steerable endoscope 32, the sheath 40, and the working channel instrument assembly (e.g., a needle assembly, not shown, which may be controlled by a robotic driver / end effector associated with the robotic system 11). Depending on the drive mode, inputs received from the user controls 312 may be mapped to one or more of the instrument drivers 28 at a given time.

[0043] In some embodiments, the physician 5 can provide input to the control system and / or the robotic system, which in response can send control signals to the robotic system 11 to operate the steerable endoscope 32. As also shown in FIG. 1 , the control system 50 can include a display 42 to provide various information related to the procedure. For example, the display 42 can provide information related to the steerable endoscope 32. For example, the control system 50 can receive real-time images captured by the steerable endoscope 32 and display the real-time images via the display 42. Additionally or alternatively, the control system 50 can receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with the patient 13, and the display 42 can present information related to the health or environment of the patient 13. Such information may include, for example, information displayed via a medical monitor, such as heart rate (e.g., ECG, HRV, etc.), blood pressure / blood flow velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), ambient temperature and / or local or core body temperature.

[0044] To facilitate the function of the control system 50, the control system may include various components (sometimes referred to as "subsystems"). For example, the control system 50 may include control electronics / circuitry as well as one or more power sources, pneumatic devices, light sources, actuators, data storage devices, and / or communication interfaces. In some embodiments, the control system 50 includes control circuitry, including a computer-based control system, configured to store executable instructions that, when executed, cause various operations to be implemented. In some embodiments, the control system 50 is mobile, while in other embodiments, the control system 50 is a substantially stationary system. Although various functions and components are discussed as being implemented by the control system 50, any of such functions and / or components may be integrated with and / or performed by other systems and / or devices, such as, for example, the robotic system 11, the table 15, etc. Components of an exemplary robotic system are discussed in further detail below with reference to FIG. 2.

[0045] The medical system 10 can provide various benefits, such as providing guidance (e.g., instrument tracking, instrument alignment information, etc.) to assist physicians in performing procedures, allowing physicians to perform procedures from ergonomic positions without requiring awkward arm movements and / or positions, allowing a single physician to perform procedures using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy), allowing procedures to be performed in a single operating setting, and providing continuous suction to more efficiently remove objects (e.g., removing kidney stones). For example, the medical system 10 can provide guidance information to assist physicians in accessing target anatomical features with various medical instruments while minimizing bleeding and / or damage to anatomical structures (e.g., pneumothorax, critical organs, blood vessels, etc.). Furthermore, the medical system 10 can provide non-radiation-based navigation and / or localization techniques to reduce physician and patient radiation exposure and / or reduce the amount of equipment in the operating room. Furthermore, the medical system 10 can provide distributed functionality between at least the control system 50 and the robotic system 11, which may be independently mobile. Such a distribution of functions and / or mobility allows the control system 50 and / or robotic system 11 to be placed in an optimal location for a particular medical procedure, which can maximize the working area around the patient and / or provide an optimized location for the physician to perform the procedure.

[0046] The various components of the medical system 10 may be communicatively coupled to one another via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), internet area networks (IANs), cellular networks, the internet, etc. Additionally, in some embodiments, the various components of the medical system 10 may be connected for data communication, fluid / gas exchange, power exchange, etc. via one or more supporting cables, conduits, or the like.

[0047] FIG. 2 provides a detailed illustration of an embodiment of a robotic system 11 (e.g., a cart-based robotic-enabled system) and control system 50 shown in FIG. 1 . The robotic system 11 generally includes an elongated support structure 14 (also referred to as a “column”), a robotic system base 25, and a console 16 at the top of the column 14. The column 14 may include one or more arm supports 17 (also referred to as “carriages”) for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2 ). The arm supports 17 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arms 12 for better positioning relative to the patient. The arm supports 17 also include a column interface 19 that allows the arm supports 17 to translate vertically along the column 14.

[0048] The column interface can connect to the column 14 through slots, such as slot 20, positioned on opposite sides of the column 14 to guide the vertical translation of the arm support 17. Slot 20 accommodates the vertical translation interface for positioning and holding the arm support 17 at various vertical heights relative to the robotic system base 25. The vertical translation of the arm support 17 allows the robotic system 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, an individually configurable arm mount on the arm support 17 allows the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0049] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of linkages 23 connected by a series of joints 24, each joint including one or more independent actuators. Each actuator may include an independently controllable motor. Each independently controllable joint 24 may provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints, thus providing seven degrees of freedom, including a "redundant" degree of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its respective end effector 22 at a specific position, orientation, and trajectory in space using different linkage positions and joint angles. This allows the system to position and orient a medical instrument from a desired point in space, while also allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0050] The robotic system base 25 balances the weight of the column 14, arm support 17, and arm 12 across a surface, such as a floor. Thus, the robotic system base 25 can house heavier components such as electronics, motors, power supplies, and components that selectively enable movement or immobilize the robotic system. For example, the robotic system base 25 includes casters 28 in the form of wheels that allow the robotic system to be easily moved around a room before a procedure. After reaching the appropriate position, the casters 28 can be locked using wheel locks to hold the robotic system 11 in place during a procedure.

[0051] When positioned at the top of column 14, console 16 allows for both a user interface for receiving user input and a display screen (or dual-purpose device, such as, for example, a touchscreen 26) for providing both pre-operative and intra-operative data to a physician user. Potential pre-operative data on touchscreen 26 may include pre-operative planning, navigation and mapping data derived from a pre-operative computed tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data on the display may also include vital patient statistics such as respiration, heart rate, and / or pulse, along with optical information provided by tools, sensor information from sensors, and coordinate information. Console 16 can be positioned and tilted to allow a physician to access the console from the side of column 14 opposite arm support 17. From this position, the physician can view console 16, robotic arm 12, and the patient while operating console 16 from behind robotic system 11. As shown, console 16 also includes a handle 27 to assist in maneuvering and stabilizing robotic system 11.

[0052] Each end effector 22 of the robotic arm 12 may include an instrument device manipulator (IDM), which may be attached using a mechanism changer interface (MCI). In some embodiments, the IDM may be removed and replaced with a different type of IDM; for example, a first type of IDM may operate an endoscope, while a second type of IDM may operate a laparoscope. The MCI may include connectors for transmitting air pressure, power, electrical signals, and / or optical signals from the robotic arm 12 to the IDM. The IDM may be configured to operate a medical instrument (e.g., a surgical tool / instrument), such as the steerable endoscope 32, using technologies including, for example, direct drive, harmonic drive, gear drive, belt and pulley drive, magnetic drive, and the like.

[0053] 2 can serve as a command console for the exemplary surgical robotic system 11. The control system 50 can include a console base 51 and one or more display devices 42.

[0054] The medical system 10 may include particular control circuitry 60 configured to perform certain of the functions described herein. The control circuitry 60 may be part of the robotic system, the control system 50, or both. That is, references herein to control circuitry may refer to circuitry embodied in the robotic system, the control system, or any other component of a medical system, such as the medical system 10 shown in FIG. 1. The term “control circuitry” is used herein according to its broad and ordinary meaning and may refer to any collection of processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connectivity circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coded and / or operational instructions in the circuitry. The control circuitry referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuitry referred to herein may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments in which the control circuitry comprises hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage devices / registers that store any relevant operating instructions may be embedded within or external to the circuitry comprising the state machines, analog circuits, digital circuits, and / or logic circuits.

[0055] 2, control circuitry 60 may comprise hard-coded and / or computer-readable media storing operational instructions corresponding to at least some of the steps and / or functions illustrated in one or more of the figures and / or described herein. Such computer-readable media may, in some cases, be included in an article of manufacture. Control circuitry 60 may be maintained / located entirely locally or may be at least partially remotely located (e.g., communicatively coupled indirectly via a local area network and / or wide area network).

[0056] In some embodiments, at least a portion of the control circuitry 60 is integrated with the robotic system 11 (e.g., within the base 25, the column 14, and / or the console 16) or another system communicatively coupled to the robotic system 11. In some embodiments, at least a portion of the control circuitry 60 is integrated with the control system 50 (e.g., within the console base 51 and / or the display unit 42). Thus, any description of functional control circuitry herein may be understood to be embodied in either the robotic system 11, the control system 50, or both, and / or at least partially in one or more other local or remote systems / devices.

[0057] The medical system 10 further includes specific user controls 65, which may comprise any type of user input (and / or output) device or device interface and / or interface / connector therefor, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game-style controllers), computer mice, trackpads, trackballs, control pads, and / or sensors (e.g., motion sensors or cameras) that capture hand and finger gestures. The user controls 65 are communicatively and / or physically coupled to at least a portion of the control circuitry 60.

[0058] In some embodiments, the user controls 65 and / or control circuitry 60 are configured to receive user inputs to allow a user to control a medical instrument, such as an endoscope or other instrument, such as an instrument operable at least in part by a robotic system, in a velocity mode or a position control mode. In velocity mode, the user may be allowed to directly control, for example, pitch and yaw movement of the distal end of the endoscope or other instrument based on direct manual control using the controls 65. For example, movement on a joystick may be mapped to yaw and pitch movement at the distal end of the scope / device. In some embodiments, the user controls 65 are configured to provide tactile feedback to the user. For example, the joystick or other control mechanism may vibrate to indicate invalid or potentially problematic input. In some embodiments, the control system 50 and / or robotic system 11 may also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate a problem related to the robotic operation.

[0059] In the position control mode, the control circuitry 60 may control a surgical instrument (e.g., an endoscope) using a three-dimensional (3D) map of the patient and / or a predetermined computer model of the patient. For example, the control circuitry 60 may be configured to provide control signals to the robotic arm 12 of the robotic system 11 to manipulate and position an associated instrument at a target location, position, and / or orientation / alignment. For embodiments implementing 3D mapping, the position control mode may require sufficiently accurate mapping of the patient's anatomy.

[0060] In some embodiments, a user can manually manipulate the robotic arm 12 of the robotic system 11 without using the user control device 65. For example, during a surgical operating room setup, a user may move the robotic arm 12 and / or any other medical instruments to provide desired access to a patient. The robotic system 11 may rely on force feedback and inertial control from the user to determine the appropriate configuration of the robotic arm 12 and associated instrumentation.

[0061] The display device 42 of the control system 50 may be integrated with the user controls 65, for example, as a tablet device with a touchscreen for providing user input. The display device 42 may be configured to provide data and input commands to the robotic system 11 using the integrated display touch controls. The display device 42 may be configured to display a graphical user interface that shows information regarding the position and orientation of the patient and / or various instruments operating within the system based on information provided by one or more position sensors. In some embodiments, a position sensor associated with a medical instrument (e.g., an endoscope) may be configured to generate a signal indicative of its position and transmit the signal over a wire and / or transmitter coupled to the sensor. Such connectivity components may be configured to transmit the position information to the console base 51 for processing by the control circuitry 60 and presentation via the display device.

[0062] FIG. 3 illustrates a bronchoscope 340, which may be referred to as a scope, endoscope, medical instrument, etc., depending on the context, positioned in a portion of a patient's respiratory system, in accordance with one or more embodiments of the present disclosure. As referenced above, bronchoscopy procedures can be performed to investigate and / or treat abnormalities in a person's lungs. For example, bronchoscopy procedures can be performed to treat and / or remove lesions or nodules. Such procedures can be performed at least partially manually and / or at least partially using robotic technology, such as the robotic system 11 shown in FIGS. 1 and 2 . For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively greater precision, control, and / or coordination compared to strictly manual procedures. In some embodiments, the scope 340 includes a working channel 344 for deploying medical instruments (e.g., lithotriptors, basket devices, forceps, etc.), irrigation, and / or suction to a working area at the distal end of the scope.

[0063] The scope 340 may be articulatable, for example, relative to at least a distal portion of the scope, so that the scope can be maneuvered within the human anatomy. In some embodiments, the scope 340 is configured to be articulated with five degrees of freedom, including, for example, X, Y, and Z coordinate translation as well as pitch and yaw. The position sensors of the scope 340 may similarly have similar degrees of freedom with respect to the position information they generate / provide. Figure 3 illustrates multiple degrees of motion of the scope 340, according to some embodiments. As shown in Figure 3, the tip or distal end 342 of the scope 340 can be oriented with zero deflection relative to its longitudinal axis 306 (also referred to as the "roll axis").

[0064] To capture images at different orientations of the tip 342, the robotic system may be configured to deflect the tip 342 on the positive yaw axis 302, the negative yaw axis 303, the positive pitch axis 304, the negative pitch axis 305, or the roll axis 306. The tip 342 or the body 345 of the scope 340 may be extended or translated on the longitudinal axis 306, the x-axis 308, or the y-axis 309. The scope 340 may include a reference structure (not shown) for calibrating the position of the scope. For example, the robotic system may measure the deflection of the scope 340 relative to the reference structure. The reference structure may be located, for example, on the proximal end of the endoscope 340 and may include a key, slot, or flange. The reference structure may be coupled to a first drive mechanism for initial calibration and to a second drive mechanism for performing the surgical procedure.

[0065] For robotic implementations, the robotic arm of the robotic system can be configured / configurable to manipulate the scope 340 using elongated movement members. The elongated movement members can include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or flexible shafts. For example, the robotic arm can be configured to actuate multiple pull wires (not shown) coupled to the scope 340 to deflect the tip 342 of the scope 340. The pull wires can include any suitable or desirable material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope 340 is configured to exhibit nonlinear behavior in response to forces applied by the elongated movement members. The nonlinear behavior can be based on the stiffness and compressibility of the scope and the variability in slack or stiffness between different elongated movement members.

[0066] The scope (e.g., endoscope / bronchoscope) 340 may comprise a tubular, flexible medical instrument configured to be inserted into a patient's anatomy to capture images of the anatomy. In some embodiments, the scope 340 may house an optical assembly, which may include an imaging device 348, such as an optical camera, and wires and / or optical fibers for transmitting signals to / from the distal end 342 of the scope 340.

[0067] The camera / imaging device 348 can be used to capture images of internal anatomical spaces, such as targeted portions of the bronchi 7 (e.g., primary bronchi 71, secondary bronchi 78, and tertiary bronchi 75, and bronchioles 77). The scope 340 may be further configured to accommodate an optical fiber for carrying light from a proximally located light source, such as a light-emitting diode, to the distal end 342 of the scope. The distal end 342 of the scope 340 may include a port for a light source for illuminating the anatomical space when the camera / imaging device is in use. In some embodiments, the scope 340 is configured to be controlled by a robotic system similar in one or more respects to the robotic system 11 shown in FIGS. 1 and 2. The imaging device may comprise an optical fiber, a fiber array, and / or a lens. The optical components move with the tip of the scope 340, such that movement of the tip of the scope results in changes in the images captured by the imaging device 348.

[0068] In some embodiments, the medical instrument (e.g., scope) 340 includes a sensor configured to generate and / or transmit sensor position data to another device. The sensor position data may indicate the position and / or orientation of the medical instrument 340 (e.g., its distal end 342) and / or may be used to determine / estimate the position / orientation of the medical instrument. For example, the sensor (sometimes referred to as a "position sensor") may include an electromagnetic (EM) sensor with a coil of conductive material or other form / embodiment of an antenna.

[0069] 3 shows an EM field generator 315 configured to broadcast an EM field 90 that is detected by an EM sensor on the medical instrument. The EM field 90 may induce a small current in the coil of the EM position sensor, which may be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator 315. Additionally, the medical instrument / scope 340 may include other types of sensors, such as shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., global positioning system (GPS) sensors), radio frequency transceivers, etc. In some embodiments, the sensors on the medical instrument can provide sensor data to a control system, which is then used to determine the position and / or orientation of the medical instrument. In some embodiments, the position sensor is positioned on the distal end 342 of the medical instrument 340, while in other embodiments, the sensor is positioned elsewhere on the medical instrument. The bronchoscope may be driven to a position proximate to a target portion of the bronchus 7.

[0070] In some implementations, as described in further detail below, the distal end of the bronchoscope 340 may be advanced through the chest 6 into the bronchi of the lungs 4l, 4r to contact or otherwise reach a target anatomical feature, which may be a nodule 201. When a position sensor associated with the distal end of the scope 340 is in contact with and / or proximity to the target anatomical feature, the position of the distal end of the scope 340 may be recorded as a target access position at which a surgical instrument (e.g., a needle) may be directed through the bronchus 7 to access the nodule 201.

[0071] The scope 340 may be oriented to deliver an injection needle to a target, such as a nodule 201 in a patient's lung. For example, a needle may be deployed down a working channel 344 extending the length of the endoscope 340 to inject a cancer treatment agent / drug directly into the target nodule 201 and / or surrounding area. In some implementations, the endoscope 340 may endoscopically deliver tools to ablate potentially cancerous tissue. In some cases, diagnostic and therapeutic procedures can be delivered in separate procedures. In these situations, the endoscope 340 may also be used to deliver fiducials to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic procedures may be delivered during the same procedure. While certain descriptions herein are presented in the context of robotic end effectors and other instrument manipulators associated with a robotic arm and / or mounted on a cart, it should be understood that the robotic control / manipulation described herein may be with any type of end effector / manipulator, such as rail-based and / or pedestal-based robotic end effectors / manipulators.

[0072] Certain embodiments of the present disclosure advantageously help automate and guide a physician through the process of accessing and treating a target anatomical feature. For example, electromagnetic positioning and scope imaging can be used together to guide needle insertion into a patient. Such a solution can enable a physician to access and treat pulmonary nodules within the lung 4.

[0073] Certain embodiments of the present disclosure involve position sensor-guided access to a target treatment site, such as the location of a nodule 201 in a lung 4. For example, if the scope 340 is fitted with one or more electromagnetic sensors and the bronchoscope 340 further includes one or more electromagnetic sensors, and such sensors are exposed to the electromagnetic field 90 created by the electric field generator 315, the associated system control circuitry can be configured to detect and track their location. In some embodiments, the tip of the bronchoscope 340 acts as a guidance beacon while the user is inserting the bronchoscope 340. Such a solution can allow the user to reach the target site from various approaches, thereby eliminating the need to rely on fluoroscopy or ultrasound imaging.

[0074] In some embodiments, a control system (not shown in FIG. 3 ) associated with the scope 340 is configured to implement localization / positioning techniques to determine and / or track the location / position of the scope 340 and / or a medical instrument, such as a drug injection needle (not shown). In some examples, the EM field generator 315 is configured to provide an EM field 90 within the patient's environment, as described above. The scope 340 and / or the drug injection needle may include an EM sensor configured to detect EM signals and transmit sensor data related to the detected EM signals to the control system. The control system can analyze the sensor data to determine the position and / or orientation of the scope 340 (e.g., the distance and / or angle / orientation between the EM sensor and the EM field generator 315). Alternatively, or in addition, in some examples, the control system can use other techniques to determine the position and / or orientation of the scope 340. For example, the scope 340 (and / or the needle) may include a shape-sensing fiber, an accelerometer, a gyroscope, an accelerometer, a satellite-based positioning sensor (e.g., a global positioning system (GPS)), a radio frequency transceiver, etc. The control system can receive sensor data from and determine the position and / or orientation of the scope 340. In some embodiments, the control system can track the position and / or orientation of the scope 340 in real time relative to the patient's coordinate system and / or anatomy.

[0075] The scope 340 may be controllable in any suitable or desirable manner, either based on user input or automatically. The controllers 311, 312 provide examples that may be used to receive user input. In some embodiments, the controls for the scope 340 are located on the proximal handle of the scope, which may be relatively difficult to grasp in some procedural postures / positions as the orientation of the bronchoscope changes. In some embodiments, the scope 340 is controlled using two-handed controllers 312. While the controllers 311, 312 are shown as handheld controllers, user input may be received using any type of I / O device, such as a touchscreen / pad, mouse, keyboard, microphone, etc.

[0076] Lobular pulmonary segments FIG. 4 illustrates lobule segments of a patient's respiratory system 400. In a clinical context, the lungs 4 are segmented into lobes through a process that can be referred to as lobe segmentation. Lobe segmentation can be particularly important during the process of assessing the location (e.g., location of nodules 201) and progression of symptoms, complications, or diseases, as well as when selecting the most appropriate treatments. For example, emphysema quantification and lung nodule detection are among the clinical applications that can benefit from lung segmentation. Correct lobe segmentation and determination of lobe boundaries can prevent pleural injuries, such as pneumothorax, during examination and treatment.

[0077] Two human lungs 4 are divided into five lobes. The lungs 4 include pulmonary fissures, which are folds (e.g., double folds) of the visceral pleura that form boundaries between sections of the lung and segment the lungs 4 into lobes. For example, both lungs have oblique fissures 81, 85 that separate the upper and lower lobes, and the right lung further has a horizontal fissure 84 that separates the right middle lobe from the upper lobe. In the left lung, an oblique fissure (left greater interlobular fissure) 81 separates the left lung 41 into two lobes: an upper left lobe (upper) 82 and a lower left lobe (lower) 83. In the right lung 4r, a horizontal fissure (right greater interlobular fissure) 84 and an oblique fissure (right lesser interlobular fissure) 85 separate the lung into three lobes: an upper (upper) right lobe 86, a middle right lobe 87, and an inferior (lower) right lobe 88. That is, each lobe has its own pleural covering formed by fissures.

[0078] Biologically, separated lobes offer various advantages. For example, lobes can function to limit the spread of bronchopulmonary infections to the affected lobe. Therefore, as discussed above, identifying fissures that function as boundaries is an important part of the diagnosis and treatment planning of lung malignancies and lung diseases. This is particularly true in the context of surgical procedures, including CT-guided lung biopsies and lobectomy, which carry the risk of crossing pulmonary fissures and incurring the risk of pneumothorax. Pneumothorax often requires the insertion of a chest tube, which results in a longer hospital stay and a higher risk of respiratory failure and mechanical ventilation. Pneumothorax risk can vary depending on the lobe and the location of the nodule within the lobe. For example, the upper lobes 82, 86 are associated with a significantly higher risk of pneumothorax than the lower lobes 83, 88. Accurate identification of lobes and their respective boundaries can help avoid crossing and puncturing pulmonary fissures during procedures, significantly facilitating patient recovery after the procedure. For example, in peripheral lung biopsy, measuring the distance from the nodule to the identified border can aid in pre-operative planning of the biopsy and avoidance of the border during the biopsy. If the distance of the surgical tool to the lobe border falls below a threshold level, a warning / error can be generated to notify the clinician or to stop or otherwise limit control of the surgical tool.

[0079] However, accurate identification of the lobes can be difficult. On CT scans, these fissures typically appear as translucent bands with hypovascularity (missing blood vessels), but can also appear as thin white lines or dark bands. The variable appearance of fissures makes detecting them a challenging task, even for experienced clinicians. In many cases, the appearance of the fissures can be ambiguous and difficult to track with traditional computer vision techniques. Further complicating the problem is patient-to-patient variability, including differences in anatomy and deformed or incomplete fissures.

[0080] 4 illustrates a respiratory system 400 and its lungs 4 and lobes 82, 83, 86, 87, 88, it is contemplated that the present disclosure may be applied to any anatomical feature in place of the lungs 4, and to any portion of such an anatomical feature in place of the lobes 82, 83, 86, 87, 88. For example, the present disclosure may be applied to another anatomical feature of a different system, such as the heart of the cardiovascular system, along with its portions (e.g., the atria and ventricles).

[0081] Distance-encoded image generation Figure 5 (represented in sections 5-1 and 5-2) is a flow diagram illustrating a process 500 for providing distance-coded images, according to one or more embodiments. Figure 6 (represented in sections 6-1 and 6-2) illustrates specific images corresponding to various blocks, states, and / or operations associated with the process of Figure 5, according to one or more embodiments. The distance-coded images generated by process 500 can aid in the diagnosis, pre-operative planning, and treatment of respiratory disorders, such as lung malignancies and pulmonary diseases.

[0082] At block 502, process 500 includes receiving an image of the lungs (lung image). The lung image may be a CT scan, an MRI scan, or other clinical image of the lungs, and it may be an original image or a modeled image constructed based on one or more original images capturing the patient's lungs. The lung image may be 2D or 3D, composed of pixels or voxels, raster or vector images, or any variant thereof. In particular, the lung image may not yet be segmented into lobes. That is, the lung image is an unlabeled image without lobe labels associated with its pixels or voxels. It will be appreciated that pixels and voxels are the smallest units of visual representation to which specific values, such as color values, can be assigned.

[0083] For example, lung images 602a, 602c, and 602d represent 2D images, while lung image 602b represents a 3D modeled image. Image 602 may include more or fewer anatomical features, such as blood vessels / bronchi 604 in lung image 602a or chest 606 in lung image 602d, depending on the various image capture / generation techniques used for image 602. Lung image 602 may include pixels / voxels associated with nodules 201a-201d, which may or may not be readily discernible by the human eye as indicating the presence of a nodule.

[0084] At block 504, process 500 involves performing lobule segmentation. Lobule segmentation may be performed using one or more artificial intelligence frameworks. For example, a machine learning framework utilizing deep learning, such as a deep neural network, a deep belief network, a deep reinforcement learning, a recurrent neural network, or a convolutional neural network, may be trained and configured to perform lobule segmentation on lung images 602. In some embodiments, lobule segmentation may involve labeling some or all of the pixels or voxels in lung images 602a-602d with a lobe label indicating which lobe the pixel or voxel belongs to. Further details regarding machine learning frameworks are described in connection with FIGS. 9 and 10.

[0085] Segmented lung image 612 shows the image resulting from block 504. Lung lobes (A, B, C, D, and E) are reflected in segmented lung image 612. The outermost pixels / voxels associated with a particular lobe label can define the boundary of the lobe associated with the particular lobe label, and the boundary indicates the fissure and pleura.

[0086] In block 506, process 500 involves assigning at least one nodule to a lobe. In some implementations, a clinician may provide a nodule location to assign the nodule to a lobe. The assignment may involve tagging one or more pixels / voxels on the segmented lung images 612a-612d, such as by clicking on the images 612a-612d or by other methods that provide coordinates associated with the images 612a-612d, to assign the nodule location. In some other implementations, an artificial intelligence algorithm / machine may automatically tag the nodule location. For example, the algorithm / machine may automatically analyze the lung image 602 associated with block 502 to identify nodules 201 and tag those pixels / voxels. Further details regarding the assignment of nodules to lobes are described in connection with FIGS. 7 and 8.

[0087] Assigning nodules to lobes allows the specific lobe containing the nodule to be identified. For example, as shown on segmented lung images 612a-612d, lobe "A" contains each of nodules 201a-201d. It will be understood that the association between nodules 201a-201d and lobe "A" is for ease of explanation only, and that nodules may be assigned to any of the remaining lobes "B," "C," "D," or "E," if applicable.

[0088] While nodules 201 have been described thus far as objects of interest, this block 506 and the concepts described herein can be applied to any anatomical feature, such as a tumor, to assign the anatomical feature to a lobe. It will further be understood that blocks 504 and 506 can be performed independently or in reverse order. That is, it is possible to assign nodules to pixels / voxels in lung image 602 before or simultaneously with segmenting lung image 602, and later determine the lobe that contains the pixel / voxel.

[0089] At block 508, process 500 may optionally involve generating leaf images 622a-622d. Generating leaf images 622a-622d may involve generating new images or converting existing images, such as converting segmented lung images 612a-612d of block 506 into leaf images 622a-622d. Leaf images 622a-622d may include the leaf label of the assigned leaf (leaf "A") and exclude the leaf labels of other leaves (leaf "B," "C," "D," and "E"). That is, leaf images 622a-622d may include pixels / voxels associated with the assigned leaf from block 506 and exclude pixels / voxels associated with other leaves.

[0090] As shown in leaf image 622, inclusion of an assigned leaf may involve assigning a bias (first) color value to pixels / voxels associated with the assigned leaf. That is, pixels / voxels associated with the leaf label of the assigned leaf may be assigned the bias color value. Exclusion of other leaves may involve filtering, masking, or otherwise removing color from pixels / voxels associated with other leaves. That is, pixels / voxels associated with the leaf labels of other leaves may be assigned an intermediate (second) color value. The intermediate color value may be a designated value in a coloring scheme different from the bias color value, such as an RGB color scheme value #000000 or a CMYK color scheme value 0, 0, 0, 100.

[0091] The assignment of bias and / or neutral color values ​​can selectively pass the assigned leaf to the leaf images 622a-622d, to the exclusion of other leaves. In some implementations, the leaf images can be binary images that include pixels / voxels associated with the assigned leaf and exclude all other pixels / voxels. For example, leaf image 622b illustrates a binary leaf image that excludes all pixels / voxels not associated with the leaf label of the leaf assigned in block 506. In other implementations, the leaf images can distinguish the assigned leaf from other leaves, while leaving pixels / voxels associated with other leaves and anatomical features, such as leaf images 622a, 622c, and 622d, substantially unchanged.

[0092] In some implementations in which block 508 is performed, the generated leaf images 622a-622d may be passed to block 510 to be distance coded. In some other implementations, this block 506 may be optional in the sense that leaf labels labeling assigned leaf pixels / voxels may be passed to block 508 along with the segmented images 612a-612d without generating leaf image 622, thereby leaving the assignment of bias color values ​​as a step to be performed in block 510.

[0093] At block 510, process 500 involves generating a distance-coded image. Block 510 receives assigned leaf and leaf segmentation information from a previous block. In some implementations, the received information may include leaf labels for the assigned leaf passed from the previous block and segmented lung image 612. In some other implementations, the received information may be in the form of a generated / transformed image, such as leaf image 622, that distinguishes the assigned leaf from other leaves.

[0094] The assigned lobe has a boundary that surrounds or encompasses the lobe. In human anatomy, the lobe boundary is defined by the pleura and fissures. In the received information, the lobe boundary may include a set of pixels / voxels that are on the edge in 2D or on the surface in 3D of the assigned lobe. Various computational algorithms, such as edge detection algorithms and 3D equivalents, may be utilized to identify the boundary.

[0095] As part of block 510, a distance metric may be calculated for pixels / voxels outside the boundary. In some embodiments, the distance metric may be the Euclidean distance between a given pixel / voxel outside the boundary and the nearest pixel / voxel on the edge / surface of the boundary. In some embodiments, the distance metric may be the polar distance between a given pixel / voxel outside the boundary and the nearest pixel / voxel on the edge / surface of the boundary. It will be understood that other distance metrics may also be used.

[0096] The calculated distance metric for a pixel / voxel can be used to code (distance code) a lung image, such as segmented lung images 612a-612d or lobe images 622a-622d, to generate distance-coded images 632a-632d. The distance coding can take various formats. For example, distance-coded image 632a may represent a grayscale image in which closer distances from the boundary are assigned darker colors (color indices) and farther distances from the boundary are assigned lighter colors (color indices). An applicable set of color index ranges can be calculated based on the shortest and longest distances from the boundary (e.g., on or adjacent to the boundary) within the lung image, and the color index can be mapped to distance accordingly. Thus, distance-coded image 632a may represent the gradient of the color index when considering pixels / voxels outside the boundary as a whole. Although grayscales ranging between black and white are typically described, it will be understood that any range of color indices is contemplated, including color spectrum (e.g., red to blue), contrast (low contrast to high contrast), and inverse color indices (e.g., lighter to darker as opposed to closer to farther), other values, or any combination of color indices.

[0097] Additionally, other distance coding schemes can be applied to the distance-coded images. By way of example, distance-coded images 632b, 632d illustrate the use of contour lines (isolines, isopleths, or isarithms) to indicate pixel / voxel distance from a boundary. Distance-coded image 632b further utilizes a variety of patterns to indicate distance, such as a denser pattern indicating closer distances. As another example, distance-coded image 632c utilizes a point cloud with denser points indicating closer distances and sparser points indicating farther distances. Many variations on distance coding schemes are possible.

[0098] Distance-coded images 632a-632d may be generated by passing leaf images through a process that calculates a distance metric, determines a distance coding scheme, and applies the scheme to the leaf images. In some implementations, the process may apply a filter (distance map filter) that converts the calculated distance metric to a range of values, maps the range of values ​​to a color index, and assigns the mapped color index to pixels / voxels outside the boundary.

[0099] Although not shown, it will be understood that in some embodiments, the interior of the leaf (within the leaf boundary) may be distance coded as well. For example, a similar distance coding scheme may be used to assign a color to the interior of the leaf from a given point within the interior to the nearest point on the boundary. Interior distance coding may be performed on distance-coded images 632a-632d instead of, or in addition to, the illustrated exterior distance coding.

[0100] At block 512, process 500 may involve providing a distance-coded image 632. Providing the distance-coded image 632 may involve displaying the image 632 to a clinician or transmitting the image 632 to a medical system (e.g., medical system 10). A clinician may use the image 632 during pre-operative planning of a medical procedure to determine how best to perform the operation to reduce the risk of pneumothorax. The medical system may integrate the image 632 into its interoperability software to provide distance information in real time and alert the clinician to such risk.

[0101] Nodule assignment Figure 7 is a flow diagram illustrating a process 700 for assigning nodes to leaves, according to one or more embodiments. Figure 8 illustrates certain images corresponding to various blocks, states, and / or actions associated with the process of Figure 7, according to one or more embodiments. Process 700 describes block 506 of Figure 5 in more detail.

[0102] At block 710, process 700 includes determining a nodule location. While nodules and their locations are described herein, it will be understood that process 700 may be applied to any target anatomical feature. The nodule location determination associated with block 710 may be performed in any suitable or desirable manner, such as using an at least partially manual determination subprocess 711 or an at least partially automated determination subprocess 712, described below in association with blocks 713 and 714, respectively.

[0103] With respect to a particular manual determination process, in block 713, process 700 involves receiving or obtaining a nodule location from a clinician. For example, the clinician may access a lung image, such as a CT scan or a 3D model of the lung constructed from such an image, and identify the nodule location. The clinician can then provide input in some manner to inform an associated computing / medical system of the nodule location, such as by clicking on the centroid 715 of the nodule in the lung image or by entering the coordinates of the centroid 715. In some embodiments, other manual user input methods include a drag selection or selection gesture around the nodule 201. Once the clinician provides the area or volume associated with the nodule 201, the coordinates of the centroid 715 can be calculated and determined as the nodule location.

[0104] With respect to a particular automated determination process, in block 714, process 700 involves determining the location of the nodule location using image data input (e.g., lung image 602) and an artificial intelligence framework, as described in detail below with respect to Figures 9 and 10. In some implementations, the artificial intelligence framework may be a deep learning framework, such as a convolutional neural network framework.

[0105] The artificial intelligence framework can receive image data input including the nodule 201. Depending on whether the image data is 2D or 3D, the framework can analyze the pixels / voxels based on various image segmentation techniques to determine the set of pixels / voxels associated with the nodule 201 from the surrounding pixels / voxels. In some embodiments, the nodule pixels / voxels can be further analyzed to provide a convenient location metric representing the nodule location. For example, the nodule location may be represented as a centroid 715 on a coordinate system such as the illustrated Euclidean coordinate system in XYZ axes.

[0106] Sub-processes 711, 712 can replace or complement each other, i.e., nodule locations can be determined manually, automatically, or automatically and then manually confirmed. Furthermore, the nodule locations determined by each sub-process 711, 712 can be compared, and if there is a discrepancy above a threshold level, the discrepancy can be notified to the clinician as a warning or error.

[0107] At block 720, if the nodule location does not readily identify pixels / voxels that correspond to the nodule location on the lung image, process 700 involves determining such pixels / voxels. For example, the acquired or calculated coordinates may be based on a coordinate system used in connection with the preoperative planning model and may need to be converted to pixel / voxel coordinates of the lung image. In some embodiments, the acquired or calculated coordinates may need to be rounded up or down to adequately map to pixel / voxel coordinates of the lung image.

[0108] At block 730, process 700 involves determining the lobe associated with the pixel / voxel. Referring back to block 720, the pixel / voxel of the nodule location has been determined. In the segmented lung image (e.g., segmented lung image 622), the pixel / voxel has an associated lobe label to which it corresponds. Based on the lobe label, the nodule is assigned a particular lobe (731).

[0109] Neural network based lobule segmentation and labeling 9 illustrates a lobule segmentation framework 900 according to one or more embodiments of the present disclosure. The lobule segmentation framework 900 may be embodied in specific control circuitry, including one or more processors, data storage devices, connectivity features, substrates, passive and / or active hardware circuit devices, chips / dies, and / or the like. For example, the framework 900 may be embodied in the control circuitry 60 shown in FIG. 2 and described above. The framework 900 may use machine learning capabilities to perform lobule segmentation and labeling of lung images.

[0110] The framework 900 may be configured to operate on a data structure of a particular image type, such as image data representing at least a portion of a lung, including a CT scan, an MRI scan, or other clinical image, which may be an original image or a modeled image constructed based on one or more original images. Such input data / data structure may be manipulated in some manner by a particular segmentation / labeling network 920 associated with the image processing portion of the framework 900. The segmentation / labeling network 920 is an example of any suitable or desirable artificial intelligence architecture capable of performing lobule segmentation / labeling. The framework 900 may involve a training process 901 and an operating process 902.

[0111] With respect to the training process 901, the segmentation / labeling network 920 may be trained according to the known anatomical image 912 and the known segmented image 932 corresponding to each image 912 as an input / output pair, and the segmentation / labeling network 920 is configured to adjust one or more parameters or weights associated therewith to correlate the known input and output image data. For example, the segmentation / labeling network 920 (e.g., a convolutional neural network) may be trained using a labeled dataset and / or machine learning. In some implementations, the machine learning framework may be configured to perform the learning / training in any suitable or desirable manner.

[0112] The known segmented image 932 may be generated, at least in part, by manually labeling anatomical features in the known anatomical image 912. For example, the manual labels may be determined and / or applied by a relevant medical professional, e.g., to label or otherwise indicate where each lobule segment is located on the known anatomical image 912. The known input / output pairs may indicate parameters of the segmentation / labeling network 920, which may be dynamically updatable in some embodiments.

[0113] The known segmented image 932 may depict the boundaries of the segmented lobes therein. In some embodiments, the framework 900 may be configured to generate the segmented image 935 in a manner that indicates in a binary fashion whether a particular lung image in the unlabeled anatomical image 915 contains a lobe (segment), and further processing may be performed on images identified as containing one or more instances of a lobe to further identify the location, boundaries, and / or other aspects of the lobe. In some embodiments, further processing may be performed to determine a nodule location in one of the segmented lobes.

[0114] With respect to operational process 902, lobule segmentation framework 900 may be further configured to use a trained version of segmentation / labeling network 920 to generate a segmented image 935 associated with unlabeled anatomical image 915. For example, during a medical procedure, real-time lung images of a treatment site may be processed using segmentation / labeling network 920 to generate segmented image 935 that identifies the presence and / or location of one or more lobes within the real-time lung images.

[0115] In some embodiments, framework 900 may be configured to identify fissures and / or pleura separating lobes, as well as segment the lobes within the image. Framework 900 may include an artificial neural network (e.g., segmentation / labeling network 920), such as a convolutional neural network. For example, framework 900 may implement a deep learning architecture that takes an input image and assigns learnable weights / biases to various aspects / objects within the image to distinguish one from another.

[0116] The network 920 may include multiple neurons (e.g., a layer of neurons as shown in FIG. 9 ) corresponding to overlapping regions of the input image that cover the visual region of the input image. The network 920 may further operate to flatten the input image or portions thereof in some manner. The network 920 may be configured to capture spatial and / or temporal dependencies within the input image 915 through the application of specific filters. Such filters may be implemented in various convolution operations to achieve desired output data and may be manually designed or learned by machine learning. Such convolution operations may be used to extract features such as edges, contours, and the like. The network 920 may include any number of convolutional layers, with more layers providing higher levels of feature discrimination. The network 920 may further include one or more pooling layers that may be configured to reduce the spatial size of the convolved features, which may be useful for extracting features that are rotationally and / or positionally invariant, as well as certain anatomical features. Once prepared through flattening, pooling, and / or other processes, the image data may be processed by a multilevel perceptron and / or a feedforward neural network. Additionally, backpropagation may be applied at each iteration of training. The framework may be capable of distinguishing between dominant features and specific low-level features in the input image and classifying them using any suitable or desirable technique. In some embodiments, the neural network architecture comprises any of the following known convolutional neural network architectures: UNet, LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, ZFNet, or any suitable architecture.

[0117] In particular, the network 920 can be modeled based on the UNet semantic segmentation deep learning model. The model can be developed to perform lung-to-lobe segmentation, as described in connection with FIG. 4 . Initially, the model can be trained on segmented images already associated with lobe labels. For example, the known anatomical image 912 is an original image with corresponding known segmented images 932 that have already segmented the known anatomical image 912 into lobes with leaf labels. In some examples, the known anatomical images 912 and their corresponding known segmented images 932 may be public data, such as available for public use in the Slicer Chest Imaging Extension. In some examples, the images 912, 932 may be by-products of a previous medical procedure in which a clinician identified lobes in the images, allowing the images to not violate patient privacy concerns. The images may be provided as CT images, MRI images, or other 2D or 3D medical images, and can form the training dataset.

[0118] In some embodiments, the model can be tuned and validated with cross-validation. Cross-validation is a resampling procedure used to evaluate machine learning models against limited data samples. For example, in five-fold cross-validation, the training dataset can be randomly reshuffled and divided into five groups. One group is selected as the validation dataset, and the remaining group is used to train the model, and the validation dataset is used to evaluate the trained model. Any evaluation metric can be used, including an error metric such as a global average Dice score. Training and evaluation are repeated for each group. Cross-validation can effectively use limited training datasets to estimate how the model is generally expected to perform when used to make predictions on data not used during model training. In some embodiments, transfer learning techniques can be utilized along with cross-validation to further fine-tune the model.

[0119] Distance-encoded image generation architecture 10 illustrates an exemplary distance-encoded image generation architecture 1000 according to one or more embodiments. Architecture 1000 may represent a convolutional neural network architecture and may include one or more of the illustrated components, which may represent specific functional components, each of which may be embodied in one or more portions or components of control circuitry associated with any of the systems, devices, and / or methods of the present disclosure. Architecture 1000 may implement an embodiment of lobule segmentation framework 900 of FIG. 9 or portions thereof.

[0120] The architecture 1000 may include a trained segmentation network component 1002, such as a UNet neural network. The trained segmentation network component 1002 may be the segmentation / labeling network 920 of FIG. 9 and may perform some or all of the functions described in blocks 502 and 504 of FIG. 5-1. That is, the trained segmentation network component 1002 may receive lung images and perform lobule segmentation to provide a segmented image 1004, such as segmented lung images 632a-632d. The segmented image may assign a lobe label to every pixel / voxel in the CT image, such that the pixel / voxel value indicates which lobe the pixel / voxel belongs to.

[0121] The architecture 1000 may further include a nodule classifier component 1006. The nodule classifier component 1006 may be configured to determine a nodule location within an image and assign a lobe (segment) containing pixels / voxels corresponding to the nodule location. The nodule classifier component 1006 may perform some or all of the functions described in block 506 of FIG. 5-1 and the blocks of the nodule assignment process 700 of FIG. 7. In some embodiments, the nodule classifier component 1006 may be configured to assign a lobe containing a nodule. In some embodiments, the lobe assignment may involve selecting a lobe to assign to the nodule. For example, the pixel / voxel corresponding to the nodule's centroid may be determined, and the lobe containing the pixel / voxel may be selected as the lobe to assign to the nodule.

[0122] One or more additional components 1016 of the architecture 1000 can further process the image based on the selected (assigned) leaves. The leaf image generation component 1010 can be configured to generate a leaf image that includes the selected leaves and excludes other leaves. That is, the leaf image generation component 1010 can be configured to perform some or all of the functions described in block 508 of FIG. 5-2, such as generating a binary image.

[0123] The distance calculation component 1012 may be configured to calculate a distance metric between a given pixel / voxel and the nearest pixel / voxel on the boundary of the selected leaf. The color assignment component 1014 may be configured to assign a color to the pixel / voxel based on the distance metric. The assigned color may be determined based on a range of colors, including grayscale, that maps color to distance. The distance calculation component 1012 and the color assignment component 1014 may be configured to perform some or all of the functions described in block 510.

[0124] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, added, merged, or omitted entirely. Thus, in a particular embodiment, not all of the described acts or events are necessary to perform a process.

[0125] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like are synonymous and used in their ordinary sense, inclusively in a non-limiting manner, and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, each be present.

[0126] In the foregoing description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the invention(s) disclosed herein and claimed below should not be limited by the specific embodiments described above, but should be determined solely by a fair reading of the following claims.

[0127] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply any physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from other elements having a similar or identical name (apart from the use of the ordinal terminology). Additionally, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an action performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0128] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0129] Spatially relative terms such as "outside," "inside," "upper," "lower," "below," "upper," "vertical," "horizontal," and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device shown in the figures were inverted, a device positioned "below" or "under" another device would be disposed "above" another device. Thus, the illustrative term "below" can include both lower and upper positions. Devices may also be oriented in other directions, and thus spatially relative terms may be interpreted differently depending on the orientation.

[0130] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," etc. are intended to encompass the notion of equality. For example, "less" can mean "less than" in the strict mathematical sense as well as "less than or equal to."

[0131] [Embodiment] (1) A computer-implemented method comprising: receiving an image of an anatomical feature; Segmenting the image into a plurality of portions based on a trained image segmentation neural network, each portion of the plurality of portions being assigned a portion label; determining a nodule location associated with a nodule within the anatomical feature; assigning the nodule to a portion of the plurality of portions within the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point away from the boundary of the portion; generating a distance-coded image based on the distance metric, the distance-coded image indicating distances from a boundary of the portion to points outside the boundary based on a color scheme; 11. A computer-implemented method comprising: (2) determining a nodule location associated with a nodule within the anatomical feature; 2. The method of claim 1, comprising receiving input from a user identifying the nodule location. (3) determining a nodule location associated with a nodule within the anatomical feature; 2. The method of claim 1, comprising analyzing the image to identify the nodule location. (4) generating a partial image excluding other parts of the plurality of parts based on the part label assigned to the part; A method according to any one of claims 1 to 3, wherein generating a distance-coded image involves assigning colors to the partial images based on the color scheme. (5) generating a distance-coded image, A method according to any one of embodiments 1 to 3, comprising assigning a color to a pixel in the distance-coded image based on the respective shortest distances between a first pixel away from the boundary of the portion in the partial image and a second pixel on the boundary.

[0132] (6) A method according to any one of claims 1 to 3, further comprising providing a distance from a location to the boundary of the portion based on the distance-coded image. (7) Providing a distance from a location to the boundary of the portion based on the distance-coded image includes: 7. The method of claim 6, further comprising determining a distance between the location and the boundary of the portion, the distance being based on at least one of Euclidean coordinates or polar coordinates. (8) generating a partial image based on a partial label assigned to the portion, 5. The method of claim 4, further comprising assigning a single color to the points outside the boundary of the portion. (9) further comprising assigning different colors to the portions within the sub-images to generate a binary image; the anatomical feature is a lung; the portion is a lobe of the lung; 9. The method of embodiment 8, wherein the multiple portions are multiple lung lobes. (10) determining a nodule location associated with a nodule within the anatomical feature includes determining a nodule centroid; A method according to any one of claims 1 to 3, wherein the assigning of the nodule to a part among the plurality of parts in the image includes determining the part label associated with the nodule centroid.

[0133] (11) The assigning of the nodule to a portion of the plurality of portions in the image includes: 11. The method of claim 10, further comprising determining a pixel value associated with the nodule centroid. (12) A method according to any one of embodiments 1 to 3, wherein generating a distance-coded image based on the distance metric includes passing the subimage through a distance map filter. (13) The method of any one of claims 1 to 3, wherein the color scheme includes grayscale colors. (14) A method according to any one of claims 1 to 3, wherein the trained image segmentation neural network identifies one or more pleura or fissures, and the part labels are assigned to the plurality of parts based on the pleura or the fissures. (15) The method of embodiment 1, wherein the trained image segmentation neural network is a UNet convolutional neural network.

[0134] (16) A system comprising: a processor; and a memory storing computer-executable instructions that cause the processor to: Segmenting the anatomical feature image into a plurality of portions based on a neural network trained to label at least a portion of the anatomical feature image; selecting a portion of the plurality of portions based on coordinates of a nodule location corresponding to a pixel associated with the portion; Mapping pixel distances to a range of values; (i) assigning a value to a first pixel based on a distance between a first pixel away from a boundary of the portion and a second pixel on the boundary of the portion, and (ii) the range of values. (17) The system of embodiment 16, further comprising determining the second pixel based on the second pixel having the shortest distance to the first pixel among the pixels on the boundary of the portion. (18) The anatomical feature is a lung; the portion is a lobe of the lung; the plurality of portions being a plurality of lobes; 18. The system of claim 16 or 17, wherein the mapping of pixel distances to the range of values ​​includes mapping the pixel distances to a range of grayscale values. (19) A health care system, an endoscope having a position sensor associated with a distal end thereof; a robotic medical system including a plurality of articulating arms; a control circuit communicatively coupled to the endoscope and the robotic medical system, the control circuit comprising: receiving a distance-coded image associated with a treatment site, wherein colors in the distance-coded image are assigned values ​​based on a pixel's distance to a lobe boundary encompassing the treatment site; determining that the distal end is below a threshold distance to the boundary based on a reference to a color of the distance-coded image; generating a notification that the distal end is at risk of contacting the boundary. (20) The medical system of embodiment 19, wherein the control circuit is further configured to limit control of the endoscope based on the risk of the distal end contacting the boundary.

Claims

1. 1. A system comprising: a processor; and a memory storing computer-executable instructions that cause the processor to: Segmenting the anatomical feature image into a plurality of portions based on a neural network trained to label at least a portion of the anatomical feature image; selecting a portion of the plurality of portions based on coordinates of a nodule location corresponding to a pixel associated with the portion; Mapping pixel distances to a range of values; (i) assigning a value to the first pixel based on a distance between a first pixel away from a boundary of the portion and a second pixel on the boundary of the portion, and (ii) the range of values.

2. The system of claim 1 , further comprising determining the second pixel based on the second pixel having the shortest distance to the first pixel among the pixels on the boundary of the portion.

3. the anatomical feature is a lung; the portion is a lobe of the lung; the plurality of portions being a plurality of lobes; The system of claim 1 or 2, wherein the mapping of pixel distances to the range of values ​​comprises mapping the pixel distances to a range of grayscale values.

4. 1. A healthcare system comprising: an endoscope having a position sensor associated with a distal end thereof; a robotic medical system including a plurality of articulating arms; a control circuit communicatively coupled to the endoscope and the robotic medical system, the control circuit comprising: receiving a distance-coded image associated with a treatment site, wherein colors in the distance-coded image are assigned values ​​based on a pixel's distance to a lobe boundary encompassing the treatment site; determining that the distal end is below a threshold distance to the boundary based on a reference to a color of the distance-coded image; generating a notification that the distal end is at risk of contacting the boundary.

5. The medical system of claim 4 , wherein the control circuitry is further configured to limit control of the endoscope based on the risk of the distal tip contacting the boundary.

6. 1. A computer-implemented method comprising: receiving an image of an anatomical feature; Segmenting the image into a plurality of portions based on a trained image segmentation neural network, each portion of the plurality of portions being assigned a portion label; determining a nodule location associated with a nodule within the anatomical feature; assigning the nodule to a portion of the plurality of portions within the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point away from the boundary of the portion; generating a distance-coded image based on the distance metric, the distance-coded image indicating distances from a boundary of the portion to points outside the boundary based on a color scheme; 11. A computer-implemented method comprising:

7. Determining a nodule location associated with a nodule within the anatomical feature includes: The method of claim 6 , comprising receiving input from a user identifying the nodule location.

8. Determining a nodule location associated with a nodule within the anatomical feature includes: The method of claim 6 , further comprising analyzing the image to identify the nodule location.

9. generating a partial image excluding other parts of the plurality of parts based on the part labels assigned to the parts; The method of claim 6 , wherein the generating a distance-coded image involves assigning colors to the sub-images based on the color scheme.

10. generating a distance coded image includes:

9. The method of claim 6, comprising assigning a color to pixels in the distance-coded image based on each shortest distance between a first pixel in the partial image away from the boundary of the portion and a second pixel on the boundary.

11. The method of claim 6 , further comprising providing a distance from a location to the boundary of the portion based on the distance-coded image.

12. providing a distance from a location to the boundary of the portion based on the distance coded image, The method of claim 11 , comprising determining a distance between the location and the boundary of the portion, the distance being based on at least one of Euclidean or polar coordinates.

13. generating a partial image based on a partial label assigned to the portion, The method of claim 9 , further comprising assigning a single color to the points outside the boundary of the portion.

14. further comprising assigning different colors to the portions within the sub-images to generate a binary image; the anatomical feature is a lung; the portion is a lobe of the lung; The method of claim 13 , wherein the plurality of portions are multiple lobes of the lung.

15. determining a nodule location associated with a nodule within the anatomical feature includes determining a nodule centroid; 9. The method of claim 6, wherein the assigning the nodule to a portion of the plurality of portions in the image comprises determining the portion label associated with the nodule centroid.

16. The assigning of the nodule to a portion of the plurality of portions in the image may include: The method of claim 15 , further comprising determining pixel values ​​associated with the nodule centroids.

17. The method of claim 6 , wherein generating a distance-coded image based on the distance metric comprises passing the sub-image through a distance map filter.

18. The method of claim 6 , wherein the color scheme comprises grayscale colors.

19. 9. The method of claim 6, wherein the trained image segmentation neural network identifies one or more pleura or fissures, and the part labels are assigned to the multiple parts based on the pleura or the fissures.

20. The method of claim 6 , wherein the trained image segmentation neural network is a UNet convolutional neural network.

Images