Lung leaflet segmentation and distance measurement of nodule from leaflet boundary

Abstract

A method of segmenting an anatomical feature portion into a plurality of portions may involve: receiving an image of the anatomical feature portion; segmenting the image into a plurality of parts based on a trained image segmentation neural network, wherein each part in the plurality of parts is allocated with a part label; determining a nodule location associated with a nodule in the anatomical feature; assigning a nodule to a portion of the plurality of portions in the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point remote from the boundary of the portion; and generating a distance coded image based on the distance metric, wherein the distance coded image indicates a distance from a boundary of the portion to a point outside the boundary based on a color scheme.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 476,147, filed on December 19, 2022, entitled “LOBUAR SEGMENTATION OF LUNG AND MEASUREMENT OF NODULE DISTANCE TO LOBE BOUNDARY,” the disclosure of which is hereby incorporated by reference in its entirety. Background Art Technical Field

[0003] The present disclosure relates to the field of medical surgery.

[0004] Related technologies

[0005] Various medical procedures involve the use of one or more machine-generated images, which can be used to provide visualization that confers certain benefits during the medical procedure. Certain surgical procedures can be guided at least in part based on machine-generated images. Summary of the Invention

[0006] Described herein are systems, devices, and methods for facilitating the identification and segmentation of various anatomical features based on images of such features obtained using a scopic device or other medical instrument. For example, such feature identification and / or segmentation can facilitate manipulation of certain anatomical features in conjunction with medical procedures such as bronchoscopy, lung biopsy, lung nodule treatment, or other procedures approaching the respiratory system.

[0007] In some aspects, the technology described herein relates to a computer-implemented method comprising: receiving an image of an anatomical feature; segmenting the image into a plurality of portions based on a trained image segmentation neural network, wherein each of the plurality of portions is assigned a portion label; determining a nodule location associated with a nodule in the anatomical feature; assigning the nodule to a portion of the plurality of portions in the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point away from the boundary of the portion; and generating a distance-coded image based on the distance metric, wherein the distance-coded image indicates a distance from the boundary of the portion to a point outside the boundary based on a color scheme.

[0008] In some aspects, the technology described herein relates to a method wherein determining a nodule location associated with the nodule in the anatomical feature includes receiving input from a user identifying the nodule location.

[0009] In some aspects, the technology described herein relates to a method wherein determining a nodule location associated with the nodule in the anatomical feature includes analyzing the image to identify the nodule location.

[0010] In some aspects, the technology described herein relates to a method, further comprising: generating a partial image based on the portion label assigned to the portion, the partial image not including other portions of the plurality of portions, wherein generating the distance-coded image involves assigning a color to the partial image based on the color scheme.

[0011] In some aspects, the technology described herein relates to a method in which generating a distance-coded image includes assigning a color to a pixel in the distance-coded image based on each shortest distance between a first pixel in the portion of the image away from the boundary and a second pixel on the boundary of the portion.

[0012] In some aspects, the technology described herein relates to a method that also includes providing a distance from a location to the boundary of the portion based on the distance-coded image.

[0013] In some aspects, the technology described herein relates to a method wherein providing a distance from a location to the boundary of the portion based on the distance-coded image includes determining a distance between the location and the boundary of the portion, wherein the distance is based on at least one of Euclidean coordinates or polar coordinates.

[0014] In some aspects, the technology described herein relates to a method wherein generating the partial image based on the partial label assigned to the portion further comprises assigning a single color to the point outside the boundary of the portion.

[0015] In some aspects, the technology described herein relates to a method further comprising assigning a different color to the portion in the partial image to generate a binary image, wherein the anatomical feature is a lung, the portion is a lung lobe, and the plurality of portions are a plurality of lung lobes.

[0016] In some aspects, the technology described herein relates to a method wherein determining a nodule location associated with the nodule in the anatomical feature comprises determining a nodule centroid; and assigning the nodule to a portion of the plurality of portions in the image comprises determining a portion label associated with the nodule centroid.

[0017] In some aspects, the technology described herein relates to a method wherein assigning the nodule to a portion of the plurality of portions in the image further comprises determining a pixel value associated with a centroid of the nodule.

[0018] In some aspects, the technology described herein relates to a method wherein generating a distance-encoded image based on the distance metric includes passing the portion of the image through a distance map filter.

[0019] In some aspects, the technology described herein relates to a method wherein the color scheme includes grayscale colors.

[0020] In some aspects, the technology described herein relates to a method in which the trained image segmentation neural network identifies one or more pleurae or interlobar fissures and assigns the portion labels to the plurality of portions based on the pleurae or interlobar fissures.

[0021] In some aspects, the technology described herein relates to a method in which the trained image segmentation neural network is a UNet convolutional neural network.

[0022] In some aspects, the technology described herein relates to a system comprising: a processor; and a memory storing computer-executable instructions to cause the processor to perform steps comprising: segmenting an image of an anatomical feature into a plurality of portions based on a neural network trained to label at least a portion of the image of the anatomical feature; selecting a portion of the plurality of portions based on coordinates corresponding to a nodule location of a pixel associated with the portion; mapping pixel distances to a range of values; and assigning a value to a first pixel away from a boundary of the portion based on: (i) a distance between the first pixel and a second pixel on the boundary of the portion, and (ii) the range of values.

[0023] In some aspects, the technology described herein relates to a system that further includes determining the second pixel based on the second pixel having a shortest distance to the first pixel among the pixels on the boundary of the portion.

[0024] In some aspects, the technology described herein relates to a system or claim 17, wherein the anatomical feature is a lung, the portion is a lung lobe, the multiple portions are multiple lung lobes, and mapping the pixel distance to the value range includes: mapping the pixel distance to a grayscale value range.

[0025] In some aspects, the technology described herein relates to a medical system comprising: an endoscope having a position sensor associated with a distal end of the endoscope; a robotic medical system comprising a plurality of articulated arms; and a control circuit communicatively coupled to the endoscope and the robotic medical system, the control circuit being configured to: receive a distance-coded image associated with a treatment site, wherein colors in the distance-coded image are assigned values ​​based on distances of pixels to a boundary of a lobe surrounding the treatment site; determine, based on a reference to the colors of the distance-coded image, that the distal end is below a threshold distance from the boundary; and generate a notification that there is a risk of the distal end contacting the boundary.

[0026] In some aspects, the technology described herein relates to a medical system in which the control circuitry is further configured to limit control of the endoscope based on the risk of the distal tip contacting the boundary. For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features have been described. It will be understood that not necessarily all such advantages may be achieved according to any particular embodiment. Thus, the disclosed embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be construed as limiting the scope of the present invention. In addition, the various features of the different disclosed embodiments may be combined to form additional embodiments as part of the present disclosure. Throughout the accompanying drawings, reference numerals may be reused to indicate the corresponding relationship between reference elements.

[0028] Figure 1 Implementations of a robotic medical system in accordance with one or more embodiments are shown.

[0029] Figure 2 Shows that according to one or more embodiments, Figure 1 An example apparatus implemented in a medical system.

[0030] Figure 3 A bronchoscope is shown positioned in various portions of a respiratory system according to one or more embodiments.

[0031] Figure 4 The lobular segments of a patient's respiratory system are shown.

[0032] FIG5 (designated as 5-1 and 5-2) is a flow chart illustrating a process for providing a distance-coded image according to one or more embodiments.

[0033] FIG. 6 (shown as 6 - 1 and 6 - 2 ) illustrates certain images corresponding to various blocks, states, and / or operations associated with the process of FIG. 5 , according to one or more embodiments.

[0034] Figure 7 is a flow chart illustrating a process for assigning nodes to leaves according to one or more embodiments.

[0035] Figure 8 According to one or more embodiments, the Figure 7 Certain images of the various boxes, states, and / or operations associated with a process.

[0036] Figure 9 A leaflet segmentation framework is shown in accordance with one or more embodiments.

[0037] Figure 10 An example distance-coded image generation architecture is shown in accordance with one or more embodiments. DETAILED DESCRIPTION

[0038] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although specific preferred embodiments and examples are disclosed below, the subject matter of the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any specific disclosed sequence. The various operations may then be described as multiple discrete operations in a manner that may help understand certain embodiments; however, the order of description should not be interpreted as implying that these operations are dependent on the order. In addition, the structures, systems and / or devices described herein may be embodied as integrated components or separate components. For the purpose of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Therefore, for example, the various embodiments may be performed in a manner that achieves or optimizes one advantage or a group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0039] Specific standard positional anatomical terms are used herein to refer to the anatomical structures of an animal, and the animal is a human with respect to the preferred embodiment. Although specific spatial relative terms such as "external," "internal," "upper," "lower," "below," "above," "vertical," "horizontal," "top," "bottom," and similar terms are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, as shown in the accompanying drawings. It should be understood that spatial relative terms are intended to encompass different orientations of elements / structures in use or operation other than the orientation depicted in the accompanying drawings. For example, an element / structure described as being "above" another element / structure may represent a position below or beside such other element / structure relative to a subject patient or an alternative orientation of the element / structure.

[0040] The present disclosure relates to systems, devices and methods for identifying and segmenting target anatomical features within a patient's body cavity / region to assist in certain medical procedures. Although certain aspects of the present disclosure are described in detail herein in the context of endoscopic procedures (such as bronchoscopic procedures), it should be understood that this background is provided for the purposes of convenience and clarity, and the anatomical feature identification and segmentation concepts disclosed herein are applicable to any suitable medical procedure. In addition, certain embodiments of robotic-enabled medical procedures are disclosed herein in the context of pulmonary nodule treatment. However, although certain principles disclosed herein are particularly applicable to the anatomical structures of the lungs and respiratory system, it should be understood that the anatomical feature identification and segmentation concepts disclosed herein can be implemented or configured for implementation in any suitable or desired anatomical structure.

[0041] In medical procedures involving the respiratory system, the risk of pneumothorax can pose a significant challenge. As will be described in more detail below, a typical lung typically includes interlobar fissures and pleura that segment or otherwise divide the lung into lobes. Under normal circumstances, the separated lobes can help provide redundancy in respiratory function and prevent the spread of disease. Accidental puncture of the border of the lobe formed by these interlobar fissures and pleura can result in pneumothorax, which is generally associated with a greater likelihood of complications and a slower recovery. Therefore, during the course of lung nodule treatment, accurate, sometimes real-time, measurement of the distance from the border of the lobe to the target anatomical feature (such as a nodule) can significantly reduce the risk of pneumothorax.

[0042] However, relying on clinicians to manually measure the distance from the nodule to the lung border can be challenging and cumbersome. Typically, clinicians display 3D computed tomography (CT) images in 2D slice multiplanar reconstruction (MPR) views and perform several measurements in different views to determine where the nodule is located relative to the interlobar fissures and pleura. When real-time problems arise, such manual work is often too time-consuming and ineffective.

[0043] Disclosed herein are certain anatomical feature recognition and segmentation concepts that can automatically perform such measurements and provide distance-coded images that easily encapsulate distance information within an image as visual information. All or substantially all of the processes involved in generating distance-coded images can be performed by a computing system that utilizes some form of artificial intelligence framework, such as a deep learning architecture. For example, the framework can employ a neural network that is trained to identify interlobar fissures and pleura in a received CT lung image and semantically segment the image into lobes based on the interlobar fissures and pleura. The architecture can then calculate the distance from the boundary of the lobe to a given pixel in the CT lung image and generate a distance-coded image. Because the distance-coded information visually encapsulates the distance information, clinicians can eliminate manual distance measurements and instead focus more time on surgical planning and execution.

[0044] The present invention further discloses a certain robotic system that can be used to perform various medical operations (such as endoscopic and laparoscopic operations). During certain operations, medical devices, such as robotically controlled medical devices (e.g., endoscopes, access sheaths, working instruments, such as needle-type instruments) are inserted into the patient's body. Within the patient's body, the instruments can be positioned within the patient's luminal network or other anatomical structures. As used herein, the term "luminal network" refers to any cavity structure within the body, whether comprising multiple lumens or branches (e.g., multiple branch lumens, such as in the lungs or blood vessels) or a single lumen or branch (e.g., within the gastrointestinal tract). During such operations, the instrument can be moved (e.g., advanced, retracted, navigated, guided, driven, etc.) through the luminal network to one or more regions of interest, such as the location of a lesion (e.g., a tumor, a nodule, etc.).

[0045] Medical system

[0046] Figure 1An example medical system 10 for performing various medical procedures in accordance with aspects of the present disclosure is shown. The medical system 10 includes a robotic system 11 configured to engage and / or control a medical instrument 32 to perform a procedure on a patient 13. The medical system 10 also includes a control system 50 configured to interface with the robotic system 11, provide information about the procedure, and / or perform various other operations. For example, the control system 50 may include a display 42 to present certain information to assist the physician 5. The medical system 10 may include a table 15 configured to hold the patient 13. The medical system 10 may also include an electromagnetic (EM) field generator (not shown; see Figure 3 ), the electromagnetic field generator may be held by one or more of the robotic arms 12 of the robotic system 11, or may be a stand-alone device.

[0047] The robotic-enabled medical system 10 can be configured in a variety of ways, depending on the particular procedure. During an endoscopic procedure (e.g., bronchoscopy), the medical system 10 can utilize the robotic arm 12 to deliver a medical device, such as a steerable endoscope 32 (e.g., a bronchoscope), through a natural opening entry point (e.g., the mouth 9 of the patient 13 positioned on the table 15 in this example) to deliver diagnostic and / or therapeutic tools / instruments. As shown, the robotic system 11 can be positioned near the patient's upper torso to provide access to the entry point. Similarly, the robotic arm 12 can be actuated to position the steerable endoscope 32 relative to the entry point. Although described in the context of a bronchoscopy procedure, it should be understood that the robotic system 11 can be implemented for other types of procedures, such as gastrointestinal (GI) procedures involving a gastroscope or other specialized endoscope.

[0048] With the robotic system 11 properly positioned, the robotic arm 12 can robotically, manually, or a combination thereof, insert the steerable endoscope 32 into the patient's body. In some embodiments, the steerable endoscope 32 can be advanced within an external access sheath 40, which can be coupled to and / or controlled by one or more robotic arms. For example, the steerable endoscope 32 and sheath 40 can each be coupled to a separate instrument driver / manipulator from a set of instrument drivers / manipulators 28, each of which is coupled to the distal end of a respective robotic arm 12. The instrument driver 28 can facilitate a linear arrangement of the steerable endoscope 32 and sheath 40 in coaxial alignment along a "virtual track" 33, which can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. Translation of the instrument driver 28 along the virtual track 33 may telescope the steerable endoscope 32 relative to the outer sheath 40 or advance or retract the steerable endoscope 32 relative to the patient.

[0049] In some embodiments, the medical system 10 can be used to perform locally targeted surgery, such as pulmonary nodule treatment using a bronchoscope. Typically, a bronchoscope includes an endoscope at its distal end that is configured to enable visualization of the respiratory tract. It should be understood that the terms "scope" and "endoscope" are used herein in accordance with their broad and ordinary meanings and may refer to any type of elongated medical device having image generation, viewing and / or capture functionality and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. For example, reference herein to a scope or endoscope may refer to a bronchoscope, cystoscope, nephroscope, bronchoscopy, arthroscope, colonoscope, laparoscope, laparoscope, or the like. In some cases, a scope / endoscope may include a rigid or flexible tube and may be sized to pass within an external sheath, catheter, introducer, or other lumen-type device, or may be used without such a device.

[0050] For illustration purposes, patient 13 is undergoing a bronchoscopy. Endoscope 32 can be robotically navigated within the respiratory system of patient 13. To enhance navigation through the patient's pulmonary network and / or to reach a desired target, steerable endoscope 32 is telescopically extended from an external access sheath 40 to achieve enhanced articulation and a larger bending radius. Using a separate instrument driver 28 allows steerable endoscope 32 and sheath 40 to be driven independently of each other.

[0051] In general, the respiratory system includes certain passages, blood vessels, organs, and muscles that help the body exchange gases between air and blood, and between blood and body cells. The respiratory system includes the upper respiratory tract, which includes the nose / nasal cavity, pharynx (i.e., larynx), and larynx (i.e., vocal cord box). The respiratory system also includes the lower respiratory tract, which includes the trachea 6, the lungs 4 (4r and 4l), and various segments of the bronchial tree, which include alveoli and alveolar ducts, which include clusters of small air sacs responsible for gas exchange between the lungs and the pulmonary blood vessels. The bronchial tree is an example luminal network in which robotically controlled instruments can be navigated and utilized according to the inventive solutions presented herein. However, although aspects of the present disclosure are presented in the context of a luminal network of bronchial networks including the airways (e.g., lumens, branches) of a patient's lungs, some embodiments of the present disclosure may be implemented in other types of luminal networks, such as the renal network, cardiovascular network (e.g., arteries and veins), gastrointestinal tract, urinary tract, etc. Typically, the luminal network includes a three-dimensional structure; for ease of illustration only, Figure 1 The luminal network is represented as a two-dimensional structure. The organs of the lower respiratory tract are located inside the thoracic cavity, which is surrounded by the sternum (ie, chest bone) and rib cage in front and the vertebrae (ie, spine) in the back, which together protect the lungs and other organs in the chest.

[0052] The trachea 6 provides the main entrance to the lungs 4. The bronchi 7 branch from the trachea 6 into each of the lungs 4, namely the left lung 41 and the right lung 4r. The trachea 6 is located just below the larynx (not shown) and provides the main airways for the lungs 4. The left lung 41 and the right lung 4r are responsible for supplying oxygen to the capillaries and removing carbon dioxide. The bronchi 7 branch from the trachea 6 into each lung 4, forming a complex network of channels that supply air to the lungs 4. The diaphragm is the primary respiratory muscle that contracts and relaxes to allow air into the lungs. The trachea 6 is the tube that carries air in and out of the lungs 4. Each lung 4 has an associated tube 7, called a bronchus, connected to the trachea. The trachea and bronchi form a bronchial tree. The bronchial tree includes a main bronchi 71, which branches into smaller secondary bronchi 78 and tertiary bronchi 75, terminating in even smaller tubes called bronchioles 77. Each bronchiole is connected to a cluster of alveoli (not shown). During the inhalation phase of the respiratory cycle, air enters through the mouth and nose and travels down the throat into the trachea 6, through the right and left mainstem bronchi 71 into the lungs 4, into the smaller bronchial airways 78, 75, into the smaller bronchioles 77, and into the alveoli where the exchange of oxygen and carbon dioxide occurs.

[0053] The robotic system 11 may be coupled to any component of the medical system 10, such as the control system 50, the table 15, the EM field generator (not shown; see Figure 3 ), steerable endoscope 32, and / or surgical instruments (e.g., needles). In some embodiments, the robotic system 11 is communicatively coupled to the control system 50. For example, the robotic system 11 may be configured to receive control signals from the control system 50 to perform operations, such as positioning the robotic arm 12 in a particular manner, manipulating the steerable endoscope 32, and the like. In response, the robotic system 11 may control components of the robotic system 11 to perform operations. In some embodiments, the robotic system 11 is configured to receive signals from the steerable endoscope 32 representing the internal anatomy of the patient 13 (i.e., information about the internal anatomy of the patient 13). Figure 1 4 (a particular depiction of the respiratory system) and / or transmits the images / image data to the control system 50 (which may then be displayed on the display 42 or other output device). In addition, in some embodiments, the robotic system 11 is coupled to components of the medical system 10, such as the control system 50, in a manner that allows it to receive fluids, optics, power, etc. therefrom. Figure 2 Additional example details of the robotic system are discussed in more detail.

[0054] The control system 50 can be configured to provide various functions to assist in performing a medical procedure. In some embodiments, the control system 50 can be coupled to the robotic system 11 and operate in conjunction with the robotic system 11 to perform a medical procedure on the patient 13. For example, the control system 50 can communicate with the robotic system 11 via a wireless or wired connection (e.g., to control the robotic system 11 and / or the steerable endoscope 32, receive images captured by the steerable endoscope 32, etc.), provide fluids to the robotic system 11 via one or more fluid channels, provide power to the robotic system 11 via one or more electrical connections, provide optics to the robotic system 11 via one or more optical fibers or other components, etc. Furthermore, in some embodiments, the control system 50 can communicate with the needle and / or endoscope to receive position data therefrom. Furthermore, in some embodiments, the control system 50 can communicate with the table 15 to position the table 15 in a particular orientation or otherwise control the table 15. Furthermore, in some embodiments, the control system 50 can communicate with an EM field generator (not shown) to control the generation of an EM field in the area surrounding the patient 13.

[0055] The control system 50 may include various I / O devices configured to assist the physician 5 or other individuals in performing a medical procedure. For example, the control system 50 may include certain input / output (I / O) components configured to allow user input to control the steerable endoscope 32, such as to navigate the steerable endoscope 32 within the body of the patient 13. For example, joystick-type, button-type, and / or other types of user input controls 312 may be used to control the articulation of a robotically controlled medical instrument, including the steerable endoscope 32, the sheath 40, and a working channel instrument assembly (e.g., a needle assembly, not shown; which may be controlled by a robotic drive / end effector associated with the robotic system 11). Depending on the drive mode, input received from the user controls 312 may be mapped to one or more of the instrument drives 28 at a given time.

[0056] In some embodiments, the physician 5 may provide input to the control system and / or the robotic system, wherein in response to the input, control signals may be sent to the robotic system 11 to steer the steerable endoscope 32. Figure 1As shown in FIG, control system 50 may include display 42 to provide various information related to the procedure. For example, display 42 may provide information about steerable endoscope 32. For example, control system 50 may receive real-time images captured by steerable endoscope 32 and display the real-time images via display 42. Additionally or alternatively, control system 50 may receive signals (e.g., analog, digital, electrical, acoustic / sound wave, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with patient 13, and display 42 may present information about the health status or environment of patient 13. Such information may include information displayed via medical monitors, including, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / rate, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), ambient and / or local or core body temperature, etc.

[0057] To facilitate the functionality of the control system 50, the control system may include various components (sometimes referred to as "subsystems"). For example, the control system 50 may include control electronics / circuitry, as well as one or more power supplies, pneumatics, light sources, actuators, data storage devices, and / or communication interfaces. In some embodiments, the control system 50 includes control circuitry that includes a computer-based control system that is configured to store executable instructions that, when executed, cause various operations to be performed. In some embodiments, the control system 50 is movable, while in other embodiments, the control system 50 is a substantially stationary system. Although various functions and components are discussed as being performed by the control system 50, any of such functions and / or components may be integrated into and / or performed by other systems and / or devices, such as the robotic system 11, the workbench 15. Reference is made below to Figure 2 Components of an example robotic system are discussed in greater detail.

[0058] The medical system 10 can provide various benefits, such as providing guidance to assist a physician in performing a procedure (e.g., instrument tracking, instrument alignment information, etc.), enabling a physician to perform a procedure from an ergonomic position without requiring awkward arm motions and / or positions, enabling a single physician to perform a procedure with one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), enabling a procedure to be performed in a single operating setting, providing continuous suction for more efficient object removal (e.g., removal of kidney stones), etc. For example, the medical system 10 can provide guidance information to assist a physician in using various medical instruments to access a target anatomical feature while minimizing bleeding and / or damage to anatomical structures (e.g., pneumothorax, critical organs, blood vessels, etc.). Furthermore, the medical system 10 can provide non-radiation-based navigation and / or positioning technology to reduce radiation exposure for the physician and patient and / or reduce the number of devices in the operating room. Furthermore, the medical system 10 can provide functionality distributed between at least the control system 50 and the robotic system 11, which can move independently. Such distribution of functionality and / or mobility can enable the control system 50 and / or robotic system 11 to be placed in a location optimal for a particular medical procedure, which can maximize the working area around the patient and / or provide an optimized position for the physician to perform the procedure.

[0059] The various components of the medical system 10 can be communicatively coupled to each other via a network, which can include wireless and / or wired networks. Example networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), internet area networks (IANs), cellular networks, the internet, and the like. In addition, in some embodiments, the various components of the medical system 10 can be connected via one or more supporting cables, tubes, and the like for data communication, fluid / gas exchange, power exchange, and the like.

[0060] Figure 2 Provided in Figure 1 Detailed illustration of an embodiment of a robotic system 11 (e.g., a cart-based robot-enabled system) and a control system 50 is shown in FIG. The robotic system 11 generally includes an elongated support structure 14 (also referred to as a "column"), a robotic system base 25, and a console 16 at the top of the column 14. The column 14 may include a base for supporting one or more robotic arms 12 (e.g., a robot-enabled system). Figure 2 The arm supports 17 may include one or more arm supports 17 (also referred to as "brackets") that are deployed in a manner similar to that of the arm support 17 (three are shown in FIG). The arm supports 17 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arm 12 for better positioning relative to the patient. The arm supports 17 also include a column interface 19 that allows the arm supports 17 to translate vertically along the column 14.

[0061] The column interface can be connected to the column 14 through slots, such as slot 20, located on opposite sides of the column 14 to guide vertical translation of the arm support 17. The slot 20 comprises a vertical translation interface to position and maintain the arm support 17 at various vertical heights relative to the robotic system base 25. Vertical translation of the arm support 17 allows the robotic system 11 to adjust the reach of the robotic arm 12 to accommodate a variety of table heights, patient sizes, and physician preferences. Similarly, individually configurable arm mounts on the arm support 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0062] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including one or more independent actuators. Each actuator may include an independently controllable motor. Each independently controllable joint 24 may provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints and therefore provides seven degrees of freedom, including "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different linkage mechanism positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0063] The robotic system base 25 balances the weight of the column 14, the arm support 17, and the arm 12 on a surface such as a floor. Thus, the robotic system base 25 can accommodate heavier components such as electronics, motors, a power supply, and components that selectively enable the robotic system to be moved and / or immobilized. For example, the robotic system base 25 includes wheel-shaped casters 28 that allow the robotic system to be easily moved around the room before surgery. Once in position, the casters 28 can be secured using wheel locks to hold the robotic system 11 in place during surgery.

[0064] The console 16 positioned at the upper end of the column 14 allows for both a user interface for receiving user input and a display screen (or dual-purpose device, such as a touch screen 26) that provides preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors and coordinate information from the sensors as well as important patient statistics such as respiration, heart rate and / or pulse. The console 16 can be positioned and tilted to allow the physician to approach the console from the side of the column 14 opposite the arm support 17. From this position, the physician can view the console 16, the robotic arm 12 and the patient while operating the console 16 from behind the robotic system 11. As shown, the console 16 may also include a handle 27 that assists in manipulating and stabilizing the robotic system 11.

[0065] The end effector 22 of each of the robotic arms 12 may include an instrument device manipulator (IDM) that can be attached using a mechanism converter interface (MCI). In some embodiments, the IDM can be removed and replaced with a different type of IDM, for example, a first type of IDM can steer an endoscope, while a second type of IDM can steer a laparoscope. The MCI may include a connector for transmitting pneumatic pressure, power, electrical signals, and / or optical signals from the robotic arm 12 to the IDM. The IDM can be configured to steer a medical device (e.g., surgical tool / instrument) such as a steerable endoscope 32 using technologies including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, and the like.

[0066] exist Figure 2 The control system 50 shown in FIG. 5 may be used as a command console for the example surgical robotic system 11 . The control system 50 may include a console base 51 and one or more display devices 42 .

[0067] The medical system 10 may include specific control circuitry 60 configured to perform the specific functions described herein. The control circuitry 60 may be part of the robotic system, the control system 50, or both. That is, references herein to a control circuit may refer to a control circuitry embodied in the robotic system, the control system, or a control circuitry such as in a medical system. Figure 1The term "control circuitry" is used herein in its broad and ordinary sense and may refer to any collection of: a processor, a processing circuit, a processing module / unit, a chip, a die (e.g., a semiconductor die comprising one or more active and / or passive devices and / or connectivity circuits), a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine (e.g., a hardware state machine), a logic circuit, an analog circuit, a digital circuit, and / or any device that manipulates signals based on hard coding of circuits and / or operating instructions. The control circuitry referred to herein may also include one or more circuit substrates (e.g., printed circuit boards), conductive traces and through-holes and / or mounting pads, connectors, and / or components. The control circuitry referred to herein may also include one or more memory devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of the device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments where the control circuitry includes hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage devices / registers storing any associated operating instructions may be embedded within the circuitry including the state machines, analog circuits, digital circuits, and / or logic circuits, or external to the circuitry.

[0068] Further references Figure 2 , the control circuit 60 may include a computer-readable medium storing hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions depicted in one or more of the figures and / or described herein. In some cases, such computer-readable medium may be included in an article of manufacture. The control circuit 60 may be entirely maintained / located locally or may be at least partially remotely located (e.g., indirectly communicatively coupled via a local area network and / or a wide area network).

[0069] In some embodiments, at least a portion of the control circuitry 60 is integrated with the robotic system 11 (e.g., integrated into the base 25, column 14, and / or console 16) or integrated with another system communicatively coupled to the robotic system 11. In some embodiments, at least a portion of the control circuitry 60 is integrated with the control system 50 (e.g., integrated into the console base 51 and / or display unit 42). Thus, any description herein of functional control circuitry can be understood as embodied in the robotic system 11, the control system 50, or both, and / or at least partially embodied in one or more other local or remote systems / devices.

[0070] The medical system 10 also includes specific user controls 65, which may include any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game-type controllers), computer mice, trackpads, trackballs, control pads, and / or sensors that capture hand gestures and finger postures (e.g., motion sensors or cameras), and / or interfaces / connectors thereto. The user controls 65 are communicatively and / or physically coupled to at least some of the control circuits in the control circuitry 60.

[0071] In some embodiments, the user controls 65 and / or control circuitry 60 are configured to receive user input to allow the user to control a medical device, such as an endoscope or endoscope, such as an instrument that can be at least partially manipulated by the robotic system in a velocity mode or a position control mode. In velocity mode, the user may be permitted to directly control the pitch and yaw motions of, for example, the distal end of an endoscope or other instrument using controls 65 based on direct manual control. For example, movements on a joystick may be mapped to yaw and pitch motions in the distal end of the endoscope / device. In some embodiments, the user controls 65 are configured to provide tactile feedback to the user. For example, a joystick or other control mechanism may vibrate to indicate invalid or potentially problematic input. In some embodiments, the control system 50 and / or robotic system 11 may also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate problems associated with robotic operation.

[0072] In the position control mode, the control circuit 60 can use a three-dimensional (3D) map of the patient and a predetermined computer model of the patient to control a medical instrument (e.g., an endoscope). For example, the control circuit 60 can be configured to provide control signals to the robotic arm 12 of the robotic system 11 to manipulate the associated instrument to position the associated instrument at a target position, location, and / or orientation / alignment. For embodiments implementing 3D mapping, the position control mode may require a sufficiently accurate mapping of the patient's anatomy.

[0073] In some embodiments, a user can manually manipulate the robotic arm 12 of the robotic system 11 without using the command controls 65. For example, during setup in a surgical operating room, a user can move the robotic arm 12 and / or any other medical instruments to provide desired access to a patient. The robotic system 11 can rely on force feedback and inertial control from the user to determine the appropriate configuration of the robotic arm 12 and associated instruments.

[0074] The display device 42 of the control system 50 can be integrated with the user controls 65, for example, as a tablet computer device with a touch screen for user input. The display device 42 can be configured to provide data and input commands to the robotic system 11 using the integrated display touch controls. The display device 42 can be configured to display a graphical user interface that shows information about the position and orientation of various instruments operating within the patient and / or the system based on information provided by one or more position sensors. In some embodiments, a position sensor associated with a medical instrument (e.g., an endoscope) can be configured to generate a signal indicating the position and transmit the signal on a wire and / or transmitter coupled to the sensor. Such connectivity components can be configured to transmit the position information to the console base 51 so that the position information is processed by the control circuit 60 and presented via the display device.

[0075] Figure 3 A bronchoscope 340, which may be referred to as a scope, endoscope, medical device, etc., depending on the context, is shown positioned in a portion of a patient's respiratory system in accordance with one or more embodiments of the present disclosure. As mentioned above, bronchoscopic procedures may be performed to examine an abnormality of a person's lungs and / or to treat the abnormality. For example, bronchoscopic procedures may be performed to treat and / or remove lesions or nodules. Such procedures may be performed at least partially manually and / or at least partially using robotic technology, such as in Figure 1 and Figure 2 1. For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively greater precision, control, and / or coordination than fully manual procedures. In some embodiments, the endoscope 340 includes a working channel 344 for deploying medical instruments (e.g., lithotripsy, basketing devices, forceps, etc.), irrigation, and / or suction to the operative area at the distal end of the endoscope.

[0076] The scope 340 can be articulated (such as with respect to at least the distal portion of the scope) so that the scope can be manipulated within the human anatomy. In some embodiments, the scope 340 is configured to articulate with, for example, five degrees of freedom, including XYZ coordinate movement, as well as pitch and yaw. The position sensor of the scope 340 can also have similar degrees of freedom with respect to the position information generated / provided by the position sensor. Figure 3 340 according to some embodiments. Figure 3 As shown, the tip or distal end 342 of the scope 340 can be oriented with zero deflection relative to its longitudinal axis 306 (also referred to as the "axis of rotation").

[0077] To capture images along different orientations of the endoscope 342, the robotic system can be configured to deflect the endoscope 342 along a positive yaw axis 302, a negative yaw axis 303, a positive pitch axis 304, a negative pitch axis 305, or a rotation axis 306. The endoscope 340's end 342 or body 345 can extend or translate along the longitudinal axis 306, the x-axis 308, or the y-axis 309. The endoscope 340 may include a reference structure (not shown) to calibrate the position of the endoscope. For example, the robotic system can measure the deflection of the endoscope 340 relative to the reference structure. The reference structure can be located, for example, on the proximal end of the endoscope 340 and can include a key, a slot, or a flange. The reference structure can be coupled to a first drive mechanism for initial calibration and to a second drive mechanism for performing the surgical procedure.

[0078] For robotic implementations, the robotic arm of the robotic system may be configured / capable of being configured to manipulate the endoscope 340 using an elongated moving member. The elongated moving member may include one or more pull wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. For example, the robotic arm may be configured to actuate a plurality of pull wires (not shown) coupled to the endoscope 340 to deflect the end 342 of the endoscope 340. The pull wires may comprise any suitable or desired material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the endoscope 340 is configured to exhibit nonlinear behavior in response to the force applied by the elongated moving member. The nonlinear behavior may be based on the stiffness and compressibility of the endoscope, as well as the variability in slack or stiffness between different elongated moving members.

[0079] The scope (e.g., endoscope / bronchoscope) 340 may include a tubular, flexible medical device configured to be inserted into a patient's anatomy to capture images of the anatomy. In some embodiments, the scope 340 may house electrical wires and / or optical fibers to transmit signals to / from the optical assembly and a distal end 342 of the scope 340, which may include an imaging device 348, such as an optical camera.

[0080] The camera / imaging device 348 can be used to capture images of an internal anatomical space, such as a target portion of the bronchi 7 (e.g., the main bronchus 71, the secondary 78 and tertiary 75 bronchi, and the bronchioles 77). The scope 340 can also be configured to accommodate an optical fiber to carry light from a proximally located light source (such as a light emitting diode) to the distal end 342 of the scope. The distal end 342 of the scope 340 can include a port for the light source to illuminate the anatomical space when the camera / imaging device is used. In some embodiments, the scope 340 is configured to be controlled by a robotic system that is similar in one or more respects to the robotic system used in the present invention. Figure 1 and Figure 2340. The imaging device may include an optical fiber, an optical fiber array, and / or a lens. The optical components move with the end of the endoscope 340 so that movement of the end of the endoscope causes the image captured by the imaging device 348 to change.

[0081] In some embodiments, a medical device (e.g., a scope) 340 includes a sensor configured to generate sensor position data and / or transmit the sensor position data to another device. The sensor position data may indicate the position and / or orientation of the medical device 340 (e.g., its distal end 342) and / or may be used to determine / infer the position / or orientation of the medical device. For example, the sensor (sometimes referred to as a "position sensor") may include an electromagnetic (EM) sensor having a coil of conductive material or other forms / embodiments of an antenna.

[0082] Figure 3 An EM field generator 315 is shown, configured to broadcast an EM field 90, which is detected by an EM sensor on the medical device. The EM field 90 may induce a small current in the coil of the EM position sensor, which can be analyzed to determine the distance and / or angle / or orientation between the EM sensor and the EM field generator 315. Furthermore, the medical device / scope 340 may include other types of sensors, such as shape sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS) sensors), radio frequency transceivers, and the like. In some embodiments, the sensors on the medical device may provide sensor data to a control system, which is then used to determine the position and / or orientation of the medical device. In some embodiments, the position sensor is located on the distal end 342 of the medical device 340, while in other embodiments, the sensor is located at another location on the medical device. The bronchoscope may be driven to a position proximate to the target portion of the bronchus 7.

[0083] In some implementations, as described in further detail below, the distal end of the bronchoscope 340 can be advanced through the thoracic cavity 6 and into the bronchus of the lungs 41, 4r to contact or otherwise reach a target anatomical feature, which can be the nodule 201. With a position sensor associated with the distal end of the scope 340 in contact with and / or proximate to the target anatomical feature, the position of the distal end of the scope 340 can be recorded as a target entry position at which a surgical instrument (e.g., a needle) can be guided through the bronchus 7 into the nodule 201.

[0084] For example, endoscope 340 can be guided to deliver an injection needle to a target, such as, for example, a nodule 201 within a patient's lung. For example, a needle can be deployed down a working channel 344, which extends along the length of endoscope 340, to inject a cancer therapeutic / drug directly into the target nodule 201 and / or surrounding area. In some implementations, endoscope 340 can be used to deliver tools through the endoscope to remove potentially cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered during separate surgeries. In these cases, endoscope 340 can also be used to deliver fiducials to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same surgery. Although some descriptions herein are presented in the context of robotic end effectors and other instrument manipulators associated with robotic arms and / or attached to carts, it should be understood that the robotic control / manipulation described herein can be used with any type of end effector / manipulator, such as track-based and / or table-based robotic end effectors / manipulators.

[0085] Certain embodiments of the present disclosure advantageously help automate and guide physicians through the process of accessing and treating target anatomical features. For example, electromagnetic positioning and endoscope images can be used together to guide the insertion of a needle into a patient. Such a solution can allow a physician to access the lungs 4 and perform lung nodule treatment.

[0086] Certain embodiments of the present disclosure relate to position sensor-guided access to a target treatment site, such as a location in a nodule 201 in a lung 4. For example, where the endoscope 340 is equipped with one or more electromagnetic sensors and the bronchoscope 340 also includes one or more electromagnetic sensors and such sensors are subjected to the electromagnetic field 90 generated by the field generator 315, the associated system control circuitry can be configured to detect and track the location of these electromagnetic sensors. In some embodiments, the tip of the bronchoscope 340 acts as a guidance beacon when the user inserts the bronchoscope 340. Such a solution can allow the user to reach the target site from multiple approaches, thereby avoiding the need to rely on fluoroscopy or ultrasound imaging modalities.

[0087] In some embodiments, the control system associated with the scope 340 (in Figure 3340 and / or a medication injection needle (not shown). In some examples, the EM field generator 315 is configured to provide an EM field 90 within the patient's environment, as described above. The scope 340 and / or medication injection needle may include an EM sensor configured to detect EM signals and transmit sensor data regarding the detected EM signals to a control system. The control system may analyze the sensor data to determine the position and / or orientation of the scope 340 (e.g., the distance and / or angle / orientation between the EM sensor and the EM field generator 315). Alternatively or additionally, in some examples, the control system may use other techniques to determine the position and / or orientation of the scope 340. For example, the scope 340 (and / or needle) may include shape sensing fibers, accelerometers, gyroscopes, accelerometers, satellite-based positioning sensors (e.g., a Global Positioning System (GPS)), radio frequency transceivers, etc. The control system may receive the sensor data from the scope 340 and determine the scope's position and / or orientation. In some embodiments, the control system can track the position and / or orientation of the scope 340 in real time relative to the patient's coordinate system and / or anatomy.

[0088] The endoscope 340 can be controlled in any suitable or desired manner based on user input or automatically. Controls 311, 312 provide examples of controls that can be used to receive user input. In some embodiments, the controls for the endoscope 340 are located on a proximal handle of the endoscope, which can be relatively difficult to grasp in some surgical postures / positions when the orientation of the bronchoscope changes. In some embodiments, the endoscope 340 is controlled using a two-handed controller 312. Although the controllers 311, 312 are shown as handheld controllers, any type of I / O device (such as a touch screen / pad, mouse, keyboard, microphone, etc.) can be used to receive user input.

[0089] Lobular segment

[0090] Figure 4 A lobular segment of a patient's respiratory system 400 is shown. In a clinical setting, the lungs 4 are segmented into lobes by a process that may be referred to as lobar segmentation. Lobar segmentation can be particularly important during the process of assessing the location and progression of symptoms, complications, or diseases (e.g., the location of nodules 201), as well as in selecting the most appropriate treatment for them. For example, emphysema quantification and lung nodule detection are among the clinical applications that can benefit from lung segmentation. Correct lobar segmentation and determination of lobar boundaries can prevent pleural damage, such as pneumothorax, during examination and treatment.

[0091] The two human lungs 4 are divided into five lobes. The lungs 4 contain interlobar fissures, which are folds (e.g., bifolds) of the visceral pleura that form boundaries between portions of the lungs and divide the lungs 4 into lobes. For example, both lungs have oblique interlobar fissures 81, 85 that separate the upper and lower lobes, and the right lung also has a horizontal interlobar fissure 84 that separates the right middle lobe from the upper lobe. In the left lung, an oblique interlobar fissure (left primary interlobar fissure) 81 divides the left lung 41 into two lobes: the upper (superior) left lobe 82 and the lower (lower) left lobe 83. In the right lung 4r, a horizontal interlobar fissure (right primary interlobar fissure) 84 and an oblique interlobar fissure (right secondary interlobar fissure) 85 divide the lung into three lobes: the upper (superior) right lobe 86, the middle right lobe 87, and the lower (lower) right lobe 88. That is, each lobe has its own pleural covering formed by the interlobar fissures.

[0092] Biologically, separated lobes offer various benefits. For example, lobes can be used to limit the spread of bronchopulmonary infections to the affected lobe. Therefore, as mentioned above, identifying interlobar fissures as boundaries is an important part of the diagnosis and treatment planning of lung malignancies and lung diseases. This is particularly true in the context of surgical procedures, including CT-guided lung biopsies and lobectomies, which carry the risk of traversing interlobar fissures and presenting the risk of pneumothorax. Pneumothorax often requires chest tube insertion, which leads to longer hospital stays and a higher risk of respiratory failure and mechanical ventilation. The risk of pneumothorax can vary depending on the lobe and the location of the nodule within the lobe. For example, the upper lobes 82 and 86 are associated with a significantly higher risk of pneumothorax than the lower lobes 83 and 88. Correctly identifying the lobes and their corresponding boundaries can help avoid traversing and puncturing interlobar fissures during surgery and significantly speed up patient recovery after surgery. For example, in peripheral lung biopsies, measuring the distance from the nodule to the identified boundary can aid in preoperative biopsy planning and boundary avoidance during the biopsy. In the event that the distance of the surgical tool to the boundary of the lobe falls below a threshold level, a warning / error may be generated to notify the clinician or to halt or otherwise limit control of the surgical tool.

[0093] However, correctly identifying the lobes can be challenging. On CT scans, these fissures typically appear as clear bands with hypovascularization (defects in blood vessels), but can also appear as thin white lines or dense bands. The variable appearance of the fissures makes detecting them a challenging task even for experienced clinicians. In many cases, the appearance of the fissures can be ambiguous and difficult to track using traditional computer vision techniques. Further complicating the problem are patient-to-patient variations, including anatomical differences and deformed or incomplete fissures.

[0094] Although Figure 4A respiratory system 400 is shown with its lung 4 and lobes 82, 83, 86, 87, 88, but it is contemplated that the present disclosure may be applied to any anatomical feature in place of the lung 4 and any portion of such an anatomical feature in place of the lobes 82, 83, 86, 87, 88. For example, the present disclosure may be applied to another anatomical feature of a different system, such as the heart and portions thereof (e.g., the atria and ventricles) of the cardiovascular system.

[0095] Distance-coded image generation

[0096] FIG5 (designated as 5-1 and 5-2) is a flow chart illustrating a process 500 for providing distance-coded images, according to one or more embodiments. FIG6 (designated as 6-1 and 6-2) illustrates specific images corresponding to various blocks, states, and / or operations associated with the process of FIG5, according to one or more embodiments. The distance-coded images generated by process 500 can aid in the diagnosis, preoperative planning, and treatment of respiratory disorders, such as lung malignancies and lung diseases.

[0097] At box 502, process 500 involves receiving an image of the lungs (lung image). The lung image can be a CT scan, MRI scan, or other clinical image of the lungs, which can be a raw image or a modeled image constructed based on one or more raw images capturing the patient's lungs. The lung image can be a 2D or 3D, raster or vector image, or any variation thereof, composed of pixels or voxels. Specifically, the lung image may not have been segmented into lobes. That is, the lung image is an unlabeled image that has no lobe labels associated with its pixels or voxels. It should be understood that pixels and voxels are the smallest units of visual representation that can be assigned a specific value, such as a color value.

[0098] For example, lung images 602a, 602c, and 602d illustrate 2D images, while lung image 602b illustrates a 3D modeled image. Depending on the various image capture / generation techniques used for images 602, images 602 may include more or fewer anatomical features, such as blood vessels / bronchi 604 in lung image 602a or thoracic cavity 606 in lung image 602d. Lung image 602 may include pixels / voxels associated with nodules 201a to 201d, which may or may not be readily discernible by the human eye as indicating the presence of nodules.

[0099] At block 504, process 500 involves performing lobule segmentation. One or more artificial intelligence frameworks may be used to perform lobule segmentation. For example, a machine learning framework utilizing deep learning (such as a deep neural network, a deep belief network, a deep reinforcement learning, a recurrent neural network, a convolutional neural network, etc.) may be trained and configured to perform lobule segmentation on lung image 602. In some embodiments, lobule segmentation may be performed by labeling some or all of the pixels or voxels in lung images 602a-602d with a leaf label indicating to which leaf the pixel or voxel belongs. Figure 9 and Figure 10 Describes more details about the machine learning framework.

[0100] Segmented lung image 612 shows the resulting image of block 504. The lobes (A, B, C, D, and E) are reflected in the segmented lung image 612. The outermost pixels / voxels associated with a particular lobe label may define the boundary of the lobe associated with the particular lobe label, where the boundary indicates the interlobar fissures and the pleura.

[0101] At block 506, process 500 involves assigning at least one nodule to a lobe. In some implementations, a clinician may provide a nodule location to assign the nodule to a lobe. The assignment may involve marking one or more pixels / voxels on the segmented lung images 612a-612d, such as by clicking on the images 612a-612d or by other methods of providing coordinates associated with the images 612a-612d, to assign the nodule location. In some other implementations, an artificial intelligence algorithm / machine may automatically mark the nodule location. For example, the algorithm / machine may automatically analyze the lung image 602 associated with block 502 to identify the nodule 201 and mark its pixels / voxels. Figure 7 and Figure 8 Describes more details about the assignment of nodes to lobes.

[0102] Assigning nodules to lobes allows identification of the specific lobe containing the nodule. For example, as shown in segmented lung images 612a to 612d, lobe "A" contains corresponding nodules 201a to 201d. The association between nodules 201a to 201d and lobe "A" is merely for simplification of the description, and it should be understood that the nodule can be assigned to any of the remaining lobes "B," "C," "D," or "E," where applicable.

[0103] While nodules 201 have been described as targets of interest so far, block 506 and the concepts described herein can be applied to any anatomical feature, such as a tumor, to assign anatomical features to lobes. Additionally, it should be understood that blocks 504 and 506 can be performed independently or in reverse order. That is, nodules can be assigned to pixels / voxels of lung image 602 before or simultaneously with segmentation of lung image 602, and the lobe containing the pixels / voxels can then be determined.

[0104] At block 508, process 500 may optionally involve generating leaf images 622a through 622d. The generation of leaf images 622a through 622d may involve the generation of new images or the conversion of existing images, such as converting the segmented lung images 612a through 612d of block 506 into leaf images 622a through 622d. The leaf images 622a through 622d may include a leaf label for the assigned leaf (leaf "A") and exclude leaf labels for other leaves (leafs "B," "C," "D," and "E"). That is, the leaf images 622a through 622d may include pixels / voxels associated with the assigned leaf from block 506 and exclude pixels / voxels associated with other leaves.

[0105] As shown in leaf image 622, including the assigned leaf may involve assigning a bias (first) color value to pixels / voxels associated with the assigned leaf. That is, pixels / voxels associated with the leaf label of the assigned leaf may be assigned the bias color value. Excluding other leaves may involve filtering, masking, or otherwise removing color from pixels / voxels associated with other leaves. That is, pixels / voxels associated with the leaf labels of other leaves may be assigned a neutral (second) color value. The neutral color value may be a specified value in a coloring scheme that is different from the bias color value, such as an RGB color scheme value #000000 or a CMYK color scheme value 0,0,0,100.

[0106] The assignment of bias color values ​​and / or neutral color values ​​can cause leaf images 622a through 622d to selectively pass through the assigned leaf while excluding other leaves. In some implementations, the leaf image can be a binary image that includes pixels / voxels associated with the assigned leaf and excludes all other pixels / voxels. For example, leaf image 622b illustrates a binary leaf image that excludes all pixels / voxels not associated with the leaf label of the assigned leaf at block 506. In other implementations, the leaf image can distinguish the assigned leaf from other leaves while leaving pixels / voxels associated with other leaves and anatomical features substantially unchanged, such as leaf images 622a, 622c, and 622d.

[0107] In some implementations of performing block 508, the generated leaf images 622a-622d may be passed to block 510 for distance encoding. In some other implementations, this block 506 may be optional in the sense that leaf labels, which mark the pixels / voxels of the assigned leaves, may be passed to block 508 along with the segmented images 612a-612d without generating the leaf images 622, thereby leaving the assignment of the deviation color values ​​as a step to be performed at block 510.

[0108] At block 510, process 500 involves generating a distance-coded image. Block 510 receives information about the assigned lobe and lobe segmentation from the previous block. In some implementations, the received information may include the lobe label of the assigned lobe and the segmented lung image 612 passed from the previous block. In some other implementations, the received information may be in the form of a generated / converted image, such as a lobe image 622, that distinguishes the assigned lobe from other lobes.

[0109] The assigned lobe has a boundary that surrounds or encloses the lobe. In human anatomy, the lobe boundary is defined by the pleura and interlobar fissures. In the received information, the lobe boundary may include a set of pixels / voxels on the 2D edge or 3D surface of the assigned lobe. Various computational algorithms, such as edge detection algorithms and 3D equivalents, may be used to identify the boundary.

[0110] As part of block 510, a distance metric may be calculated for pixels / voxels outside the boundary. In some embodiments, the distance metric may be the Euclidean distance between a given pixel / voxel outside the boundary and the nearest pixel / voxel on the edge / surface of the boundary. In some embodiments, the distance metric may be the epipolar distance between a given pixel / voxel outside the boundary and the nearest pixel / voxel on the edge / surface of the boundary. It will be appreciated that other distance metrics may also be used.

[0111] The calculated distance metric for pixels / voxels can be used to encode (distance encoding) a lung image (such as the segmented lung images 612a to 612d or the lobe images 622a to 622d) to generate distance-coded images 632a to 632d. Distance encoding can be in various formats. For example, the distance-coded image 632a shows a grayscale image in which closer distances to the boundary are assigned darker colors (color indices) and farther distances from the boundary are assigned lighter colors (color indices). A set of applicable color index ranges can be calculated based on the shortest (e.g., on or near the boundary) distance and the farthest distance away from the boundary in the lung image, and the color indices are mapped to these distances accordingly. Therefore, when considering pixels / voxels outside the boundary as a whole, the distance-coded image 632a can show a gradient of color indices. It should be understood that while grayscale is generally described between black and white, any range of color indices is contemplated, including color spectra (e.g., red to blue), contrast (low contrast to high contrast), and inverted color indices (e.g., lighter to darker for closer to farther), other values, or any combination of color indices.

[0112] Additionally, other distance encoding schemes may be applied to the distance-encoded images. As an example, distance-encoded images 632b and 632d illustrate using contour lines (isolines, contours, or isoscale lines) to indicate pixel / voxel distances from a boundary. Distance-encoded image 632b additionally utilizes various patterns to indicate distances, such as a closer knit pattern indicating a closer distance. As another example, distance-encoded image 632c utilizes a point cloud with denser points indicating a closer distance and less dense points indicating a greater distance. Many variations in distance encoding schemes are possible.

[0113] The distance-coded images 632a to 632d can be generated by passing the leaf images through a process of calculating a distance metric, determining a distance coding scheme, and applying the scheme to the leaf images. In some implementations, the process can apply a filter (a distance map filter) that converts the calculated distance metric into a value range, maps the value range to a color index, and assigns the mapped color index to pixels / voxels outside the boundary.

[0114] Although not shown, it should be understood that in some embodiments, the interior of the leaf (within the leaf's boundary) can be similarly distance-coded. For example, a similar distance-coding scheme can be used to assign a color to the interior of the leaf, from a given point in the interior to the nearest point on the boundary. Alternatively or in addition to the external distance coding shown, the distance-coded images 632a to 632d can be internally distance-coded.

[0115] At block 512, process 500 may involve providing a distance-coded image 632. Providing the distance-coded image 632 may involve displaying the image 632 to a clinician or transmitting the image 632 to a medical system (e.g., medical system 10). The clinician may use the image 632 during preoperative planning of their medical procedure to determine how best to perform the procedure with a reduced risk of pneumothorax. The medical system may integrate the image 632 into its interoperability software to provide distance information in real time, thereby alerting the clinician to such risks.

[0116] Nodule allocation

[0117] Figure 7 is a flow chart illustrating a process 700 for assigning nodes to leaves according to one or more embodiments. Figure 8 According to one or more embodiments, the Figure 7 5. Process 700 describes block 506 of FIG. 5 in more detail.

[0118] At block 710, process 700 involves determining a nodule location. It should be understood that while nodules and their locations are described herein, process 700 is applicable to any target anatomical feature. The nodule location can be determined in conjunction with block 710 in any suitable or desired manner, such as using an at least partially manually determined sub-process 711 or an at least partially automatically determined sub-process 712, described below in conjunction with blocks 713 and 714, respectively.

[0119] With respect to certain manual determination processes, at box 713, process 700 involves receiving or obtaining a nodule location from a clinician. For example, a clinician may access a lung image, such as a CT scan or a 3D model of the lungs constructed from such an image, and identify the nodule location. The clinician may then provide input to inform the relevant computing / medical system of the nodule location in some manner, such as by clicking on the center of mass 715 of the nodule in the lung image or entering the coordinates of the center of mass 715. In some embodiments, other manual user input methods include a drag selection or selection gesture around the nodule 201. When the clinician provides an area or volume associated with the nodule 201, the coordinates of the center of mass 715 may be calculated and determined as the nodule location.

[0120] With respect to a specific automated determination process, at block 714, process 700 involves determining the location of a nodule using an image data input (e.g., lung image 602) and an artificial intelligence framework, as described below with respect to Figure 9 and Figure 10 In some implementations, the artificial intelligence framework can be a deep learning framework, such as a convolutional neural network framework.

[0121] The artificial intelligence framework can receive image data input containing nodule 201. Depending on whether the image data is 2D or 3D, the framework can analyze pixels / voxels based on various image segmentation techniques to determine a set of pixels / voxels associated with nodule 201 from surrounding pixels / voxels. In some embodiments, the nodule pixels / voxels can be further analyzed to provide a convenient positional metric representing the nodule's location. For example, the nodule's location can be represented as a centroid 715 on a coordinate system, such as the illustrated Euclidean coordinate system based on XYZ axes.

[0122] The sub-processes 711 and 712 may replace or complement each other. That is, the nodule location may be determined manually, automatically, or automatically and then manually confirmed. Furthermore, the nodule locations determined by each sub-process 711 and 712 may be compared, and when there is a difference above a threshold level, the difference may be notified to the clinician as a warning or error.

[0123] At block 720, if the nodule location does not readily identify pixels / voxels corresponding to the nodule location on the lung image, process 700 involves determining such pixels / voxels. For example, the acquired or calculated coordinates may be based on a coordinate system used in connection with the preoperative planning model and may need to be converted to pixel / voxel coordinates of the lung image. In some embodiments, the acquired or calculated coordinates may need to be rounded up or down to adequately map to pixel / voxel coordinates of the lung image.

[0124] At block 730, process 700 involves determining the leaf associated with the pixel / voxel. Referring to block 720, the pixel / voxel of the nodule location has been determined. In the segmented lung image (e.g., segmented lung image 622), the pixel / voxel has an associated leaf label corresponding to it. Based on the leaf label, the nodule is assigned a specific leaf 731.

[0125] Neural network-based lobule segmentation and labeling

[0126] Figure 9 A lobule segmentation framework 900 is shown in accordance with one or more embodiments of the present disclosure. The lobule segmentation framework 900 may be embodied in a specific control circuit comprising one or more processors, data storage devices, connectivity features, substrates, passive and / or active hardware circuit devices, chips / dies, etc. For example, the framework 900 may be embodied in Figure 2 , and in the control circuit 60 described above. Framework 900 may employ machine learning functionality to perform lobule segmentation and labeling of lung images.

[0127] The framework 900 can be configured to operate on certain image-type data structures, such as image data representing at least a portion of the lung, including CT scans, MRI scans, or other clinical images, which can be raw images or modeled images constructed based on one or more raw images. Such input data / data structures can be manipulated in some manner by a specific segmentation / labeling network 920 associated with the image processing portion of the framework 900. The segmentation / labeling network 920 is an example of any suitable or desired artificial intelligence architecture that can perform lobule segmentation / labeling. The framework 900 can involve a training process 901 and an operation process 902.

[0128] Regarding the training process 901, a segmentation / labeling network 920 can be trained based on a known anatomical image 912 and a known segmented image 932 corresponding to the corresponding image 912 as an input / output pair, wherein the segmentation / labeling network 920 is configured to adjust one or more parameters or weights associated therewith to correlate the known input and output image data. For example, the segmentation / labeling network 920 (e.g., a convolutional neural network) can be trained using a labeled dataset and / or machine learning. In some implementations, the machine learning framework can be configured to perform learning / training in any suitable or desired manner.

[0129] The known segmented image 932 can be generated, at least in part, by manually marking anatomical features in the known anatomical image 912. For example, manual markings can be determined and / or applied by a relevant medical expert to mark or otherwise indicate, for example, where each leaflet segment is on the known anatomical image 912. The known input / output pairs can indicate parameters of the segmentation / labeling network 920, which can be dynamically updated in some embodiments.

[0130] The known segmented image 932 may depict the boundaries of the lobes segmented therein. In some embodiments, the framework 900 may be configured to generate a segmented image 935 by binaryly indicating whether a particular lung image of the unlabeled anatomical image 915 includes a lobe (segment), wherein further processing may be performed on the image identified as containing one or more instances of a lobe to further identify the location, boundaries, and / or other aspects of the lobe. In some embodiments, further processing may be performed to determine the location of a nodule in one of the segmented lobes.

[0131] With respect to operational process 902, the lobule segmentation framework 900 can also be configured to use the trained version of the segmentation / labeling network 920 to generate a segmented image 935 associated with the unlabeled anatomical image 915. For example, during a medical procedure, the segmentation / labeling network 920 can be used to process a real-time lung image of a treatment site to generate a segmented image 935 that identifies the presence and / or location of one or more lobes in the real-time lung image.

[0132] In some embodiments, the framework 900 can be configured to identify interlobar fissures and / or pleura, and segment the lobes in the image. The framework 900 can include an artificial neural network (e.g., a segmentation / labeling network 920), such as a convolutional neural network. For example, the framework 900 can implement a deep learning architecture that accepts an input image and assigns learnable weights / biases to various aspects / objects in the image to distinguish them from each other.

[0133] The network 920 may include a plurality of neurons (e.g., layers of neurons, such as Figure 9). Network 920 may also be operable to flatten the input image or portion thereof in some manner. Network 920 may be configured to capture spatial and / or temporal dependencies in input image 915 by applying specific filters. Such filters may be performed in various convolution operations to achieve the desired output data and may be manually designed or learned through machine learning. Such convolution operations may be used to extract features such as edges, contours, and the like. Network 920 may include any number of convolutional layers, with more layers providing for the recognition of higher-level features. Network 920 may also include one or more merging layers that may be configured to reduce the spatial size of the convolved features, which may be used to extract rotationally and / or positionally invariant features, such as specific anatomical features. Once prepared through flattening, merging, and / or other processes, the image data may be processed using a multi-stage perceptron and / or feedforward neural network. Furthermore, backpropagation may be applied to each iteration of training. The framework may be able to distinguish between primary and specific low-level features in the input image and classify them using any suitable desired technique. In some embodiments, the neural network architecture comprises any of the following known convolutional neural network architectures: UNet, LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, ZFNet, or any suitable architecture.

[0134] Specifically, the network 920 can be modeled based on the UNet semantic segmentation deep learning model. The model can be developed to perform Figure 4 Segmentation of the lungs into lobes described. Initially, the model can be trained on segmented images that have been associated with lobe labels. For example, the known anatomical image 912 is an original image that has a corresponding known segmentation image 932 that has divided the known anatomical image 912 into lobes with lobe labels. In some cases, the known anatomical image 912 and its corresponding known segmentation image 932 can be public data, such as available for public use with the Slicer chest imaging extension module. In some cases, the images 912, 932 can be byproducts of previous medical procedures in which a clinician has already identified the lobes in the image, allowing the image to not violate patient privacy issues. The images can be provided as CT images, MRI images, or other medical images in 2D or 3D, and can form a training data set.

[0135] In some embodiments, cross-validation can be used to tune and validate the model. Cross-validation is a resampling process used to evaluate machine learning models on a limited sample of data. For example, in a 5-fold cross-validation, the training dataset can be randomly reshuffled and divided into 5 groups. One group is selected as the validation dataset, and the remaining groups are used to train the model, and the validation dataset is used to evaluate the trained model. Any evaluation metric can be used, including error metrics such as the global average dice score. Training and evaluation are repeated for each group. Cross-validation can effectively use a limited training dataset to estimate how the expected model will generally perform when used to make predictions on data that was not used during model training. In some embodiments, transfer learning techniques can be used along with cross-validation to further fine-tune the model.

[0136] Distance-encoded image generation architecture

[0137] Figure 10 An example distance coded image generation architecture 1000 is shown according to one or more embodiments. The architecture 1000 may represent a convolutional neural network architecture and may include one or more of the components shown, which may represent specific functional components, each of which may be embodied in one or more parts or components of the control circuitry associated with any of the systems, devices, and / or methods of the present disclosure. The architecture 1000 may implement Figure 9 An embodiment of the leaflet segmentation frame 900 or a portion thereof.

[0138] The architecture 1000 may include a trained segmentation network component 1002, such as a UNet neural network. The trained segmentation network component 1002 may be Figure 9 The segmentation / labeling network 920 can be executed Figure 5-1 504. That is, the trained segmentation network component 1002 can receive a lung image and perform lobule segmentation to provide a segmented image 1004, such as the segmented lung images 632a to 632d. The segmented image can assign a leaf label to each pixel / voxel in the CT image in a manner such that the pixel / voxel value indicates to which leaf the pixel / voxel belongs.

[0139] The architecture 1000 may also include a nodule classifier component 1006. The nodule classifier component 1006 may be configured to determine nodule locations in the image and assign leaves (segments) containing pixels / voxels corresponding to the nodule locations. The nodule classifier component 1006 may perform Figure 5-1 Box 506 and Figure 7The nodule assignment process 700 of FIG. 1006 may include some or all of the functionality described in the blocks of FIG. 1006 . In some embodiments, the nodule classifier component 1006 may be configured to assign a leaf that contains a nodule. In some embodiments, the assignment of a leaf may involve selecting a leaf to assign to a nodule. For example, a pixel / voxel corresponding to a nodule centroid may be determined, and a leaf containing the pixel / voxel may be selected as the leaf to be assigned to the nodule.

[0140] One or more additional components 1016 of the architecture 1000 may further process the image based on the selected (assigned) leaves. The leaf image generation component 1010 may be configured to generate a leaf image that includes the selected leaves and excludes other leaves. That is, the leaf image generation component 1010 may be configured to perform Figure 5-2 Some or all of the functionality described in block 508 of , such as generating a binary image, may be performed.

[0141] Distance calculation component 1012 can be configured to calculate a distance metric between a given pixel / voxel and the nearest pixel / voxel on the boundary of the selected leaf. Color assignment component 1014 can be configured to assign a color to the pixel / voxel based on the distance metric. The assigned color can be determined based on a color range (including grayscale) that maps color to distance. Distance calculation component 1012 and color assignment component 1014 can be configured to perform some or all of the functions described in block 510.

[0142] Additional Implementation Plans

[0143] Depending on the implementation, some actions, events or functions of any one of the processes or algorithms described herein may be performed in a different order, added, combined or completely ignored. Thus, in some embodiments, not all described actions or events are necessary for the practice of the process.

[0144] Unless otherwise specifically stated or understood otherwise within the context of use, conditional language used herein, such as "may," "can," "might," "could," "for example," etc., refers to its ordinary meaning and is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that one or more embodiments require a feature, element, and / or step in any way, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or will be performed in any particular embodiment. The terms "comprise," "include," "have," etc. are synonymous and are used in their ordinary sense and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Moreover, the term "or" is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term "or" refers to one, some, or all of the elements in the series. Unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood in the context of general usage to convey that an item, term, element, etc. can be X, Y, or Z. Thus, such conjunctive language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0145] It should be understood that in the above description of the embodiments, in order to simplify the disclosure and assist in understanding one or more of the various inventive aspects, various features are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be understood to reflect the intention that any claim requires more features than those expressly recited in that claim. In addition, any component, feature, or step shown and / or described in a particular embodiment herein may be applied to or used with any other embodiment. In addition, no component, feature, step, or group of components, features, or steps is required or indispensable for each embodiment. Therefore, it is expected that the scope of the invention disclosed herein and claimed below is not limited to the specific embodiments described above, but should be determined solely by a fair reading of the appended claims.

[0146] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element such as a structure, component, operation, etc. do not necessarily indicate the priority or order of the element relative to any other element, but rather may generally distinguish the element from another element having a similar or identical name (but for which the ordinal term is used). Additionally, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events that are not explicitly stated.

[0147] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and not in an idealized or overly formal sense, unless expressly defined as such herein.

[0148] For ease of description, the spatially relative terms "outside," "inside," "upper," "lower," "below," "above," "vertical," "horizontal," and similar terms may be used herein to describe the relationship between one element or component and another element or component shown in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation other than the orientation depicted in the drawings. For example, where the device shown in the drawings is turned over, a device that is "below" or "beneath" another device may be placed "above" the other device. Thus, the illustrative term "below" may include both a lower position and an upper position. The device may also be oriented in another direction, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0149] Unless expressly stated otherwise, comparative and / or quantitative terms such as "less," "more," "greater," etc., are intended to encompass equivalent concepts. For example, "less" may mean not only "less" in the strictest mathematical sense, but also "less than or equal to."

Claims

1. A computer-implemented method comprising: receiving an image of the anatomical feature; segmenting the image into a plurality of parts based on a trained image segmentation neural network, wherein each of the plurality of parts is assigned a part label; determining a nodule location associated with the nodule in the anatomical feature; assigning the nodule to a portion of the plurality of portions in the image based on the nodule location; calculating at least one distance metric between a first point on a boundary of the portion and a second point away from the boundary of the portion; as well as A distance-coded image is generated based on the distance metric, wherein the distance-coded image indicates a distance from a boundary of the portion to a point outside the boundary based on a color scheme.

2. The method according to claim 1, wherein Determining a nodule location associated with the nodule in the anatomical feature includes: Input is received from a user identifying the location of the nodule.

3. The method according to claim 1, wherein Determining a nodule location associated with the nodule in the anatomical feature includes: The images are analyzed to identify the nodule location.

4. The method according to any one of claims 1 to 3, further comprising: generating a partial image based on the partial label assigned to the portion, the partial image excluding other portions of the plurality of portions, Wherein generating the distance-coded image involves assigning colors to the partial images based on the color scheme.

5. The method according to any one of claims 1 to 3, wherein Generating a distance-coded image involves: A color is assigned to a pixel in the distance-coded image based on each shortest distance between a first pixel in the portion of the image that is far from the boundary and a second pixel on the boundary of the portion.

6. The method according to any one of claims 1 to 3, further comprising: A distance from a location to the boundary of the portion is provided based on the distance-coded image.

7. The method according to claim 6, wherein: Providing a distance from a location to the boundary of the portion based on the distance-coded image includes: A distance between the location and the boundary of the portion is determined, wherein the distance is based on at least one of Euclidean coordinates or polar coordinates.

8. The method according to claim 4, wherein Generating the partial image based on the partial label assigned to the portion further comprises: The points outside the boundary of the portion are assigned a single color.

9. The method according to claim 8, further comprising: Different colors are assigned to the portion in the partial image to generate a binary image, wherein: The anatomical feature is a lung, The portion is a lung lobe, and The plurality of parts are a plurality of lung lobes.

10. The method according to any one of claims 1 to 3, wherein: Determining a nodule location associated with the nodule in the anatomical feature includes: determining a nodule centroid; and Assigning the nodule to a portion of the plurality of portions in the image includes determining the portion label associated with the nodule centroid.

11. The method according to claim 10, wherein: Assigning the nodule to a portion of the plurality of portions in the image further comprises: A pixel value associated with the nodule centroid is determined.

12. The method according to any one of claims 1 to 3, wherein Generating a distance-encoded image based on the distance metric comprises passing the portion of the image through a distance map filter.

13. The method according to any one of claims 1 to 3, wherein The color scheme includes grayscale colors.

14. The method according to any one of claims 1 to 3, wherein The trained image segmentation neural network identifies one or more pleurae or interlobar fissures and assigns the portion labels to the plurality of portions based on the pleurae or interlobar fissures.

15. The method according to claim 1, wherein The trained image segmentation neural network is a UNet convolutional neural network.

16. A system comprising: processor; and A memory storing computer-executable instructions for causing the processor to perform the following steps: segmenting the image of the anatomical feature into a plurality of portions based on a neural network trained to label at least a portion of the image of the anatomical feature; selecting a portion of the plurality of portions based on coordinates of a nodule location corresponding to pixels associated with the portion; Map pixel distances to value ranges; as well as A value is assigned to the first pixel away from a boundary of the portion based on: (i) a distance between the first pixel and a second pixel on the boundary of the portion, and (ii) the value range.

17. The system of claim 16, further comprising: The second pixel is determined based on the second pixel having the shortest distance to the first pixel among pixels on the boundary of the portion.

18. A system according to claim 16 or claim 17, wherein: The anatomical feature is a lung, The part is a lobe of a lung, The plurality of parts are a plurality of lung lobes, and Mapping the pixel distance to the value range includes mapping the pixel distance to a grayscale value range.

19. A medical system comprising: an endoscope having a position sensor associated with a distal end of the endoscope; A robotic medical system comprising a plurality of articulated arms; and a control circuit communicatively coupled to the endoscope and the robotic medical system, the control circuit configured to: receiving a distance-coded image associated with a treatment site, wherein colors in the distance-coded image are assigned values ​​based on distances of pixels to a boundary of a lobe surrounding the treatment site; determining that the distal end is less than a threshold distance from the boundary based on reference to a color of the distance-coded image; and A notification is generated that there is a risk that the distal end portion contacts the boundary.

20. The medical system of claim 19, wherein: The control circuitry is further configured to limit control of the endoscope based on the risk of the distal end contacting the boundary.

Images