System and method for segmenting image data

Abstract

Systems and methods for segmenting medical image data are described. The method may include (i) obtaining a plurality of image data depicting a portion of the subject; (ii) inputting the plurality of image data into a deep learning model trained to generate a confidence map; (iii) generating a first binary image based on the confidence map and a first confidence threshold, and generating a second binary image based on the confidence map and a second confidence threshold; (iv) discarding one or more connected components below a threshold size in the second binary image; and (v) causing the display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image data, where the portion of the plurality of image data is marked based on the first binary image and the second binary image. The method may be implemented by a system.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit from the filing date of U.S. Provisional Patent Application No. 63 / 516,726, entitled "SYSTEMS AND METHODS FOR SEGMENTINGIMAGE DATA," filed July 31, 2023. The entire contents of that U.S. Provisional Patent Application are hereby expressly incorporated herein by reference. Technical Field

[0003] The disclosed examples relate to image data segmentation. Specifically, the disclosed examples relate to systems and methods for segmenting image data associated with a medical procedure performed on a subject. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, physicians can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, and / or biopsy instruments) to reach target tissue locations. One such technique utilizes flexible and / or steerable elongated devices (e.g., flexible catheters) that can be inserted into an anatomical channel and navigated toward a region of interest within the subject's anatomy.

[0005] The subject's image data is used to plan paths toward the region of interest and to ensure proper navigation during the procedure. As an example, image data can be analyzed to segment sensitive structures to be avoided (e.g., blood vessels, pleura, fissures, etc.) from other objects in the image. Therefore, accurate segmentation of sensitive structures is necessary to ensure efficient execution of the procedure. Summary of the Invention

[0006] The following is a simplified overview of the various examples described herein and is not intended to identify key or important elements or to depict the scope of the claims.

[0007] In some examples, a system for segmenting medical image data is provided. The system may include: (i) one or more processors; and (ii) one or more non-transitory computer-readable media storing instructions. When executed by one or more processors, the instructions cause the system to: (1) acquire multiple image data depicting a portion of a subject; (2) input the multiple image data into a deep learning model trained to generate a confidence map indicating the confidence level of one or more portions of the image data depicting an anatomical structure of the subject; (3) generate a first binary map based on the confidence map and a first confidence threshold; (4) generate a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discard one or more connected components in the second binary map that are smaller than a threshold size; and (6) cause a display device to display a graphical user interface (GUI) based on at least a portion of the multiple image data, wherein the portions of the multiple image data are labeled based on the first and second binary maps.

[0008] In other examples, a method for segmenting medical image data is provided. The method includes: (1) obtaining multiple image data depicting a portion of a subject via one or more processors; (2) inputting the multiple image data into a deep learning model via one or more processors, the deep learning model being trained to generate a confidence map indicating the confidence level of one or more portions of the image data depicting an anatomical structure of the subject; (3) generating a first binary map via one or more processors based on the confidence map and a first confidence threshold; (4) generating a second binary map via one or more processors based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discarding one or more connected components in the second binary map that are smaller than a threshold size via one or more processors; and (6) causing a display device via one or more processors to display a graphical user interface (GUI) based on at least a portion of the multiple image data, wherein the portions of the multiple image data are labeled based on the first binary map and the second binary map.

[0009] In yet another example, a non-transitory computer-readable medium is provided on which instructions are stored. When executed by one or more processors, the instructions cause one or more processors to: (1) acquire multiple image data depicting a portion of a subject; (2) input the multiple image data into a deep learning model trained to generate a confidence map indicating the confidence level of one or more portions of the image data depicting an anatomical structure of the subject; (3) generate a first binary map based on the confidence map and a first confidence threshold; (4) generate a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discard one or more connected components in the second binary map that are smaller than a threshold size; and (6) cause a display device to display a graphical user interface (GUI) based on at least a portion of the multiple image data, wherein the portions of the multiple image data are labeled based on the first and second binary maps.

[0010] It should be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of this disclosure, without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] Figure 1 A graphical user interface that can be generated and displayed to the user is described based on some examples.

[0012] Figure 2 An example workflow for applying a deep learning model to generate confidence maps based on input image data is described, based on several examples.

[0013] Figure 3 Example processing for generating binary maps based on confidence maps is described based on some examples.

[0014] Figure 4 Example processing for combining binary images to produce an output segmentation map is described based on some examples.

[0015] Figure 5 A graphical user interface that can be generated and displayed to the user is described based on some examples.

[0016] Figure 6 This is an example flowchart for segmenting medical image data, based on some examples.

[0017] Figure 7 This is a simplified diagram of a medical system that can implement the techniques disclosed in this article, based on some examples.

[0018] Figure 8A This is a simplified diagram of a medical device system including a flexible elongation device, which can be used in conjunction with the techniques disclosed herein, based on some examples.

[0019] Figure 8B It is based on some examples. Figure 8A A simplified diagram of a medical instrument within a flexible elongation device.

[0020] Figure 9A and Figure 9B It is a simplified diagram of a patient coordinate space side view based on some examples, including a medical device mounted on an insertion component.

[0021] Examples of this disclosure and its advantages can be better understood by referring to the following detailed description. It should be understood that similar reference numerals are used to identify similar elements shown in one or more figures, wherein the illustrations in the figures are for illustrative purposes and not for limiting the scope of this disclosure. Detailed Implementation

[0022] In the following description, specific details are set forth in relation to some examples conforming to this disclosure. Numerous specific details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be illustrative and not restrictive. Other elements within the scope and spirit of this disclosure can be implemented by those skilled in the art, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless otherwise specifically described or if one or more features would render the example inoperable. In some cases, well-known methods, processes, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the examples.

[0023] This disclosure describes various instruments and parts thereof based on their state in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of that object or part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term "shape" refers to the set of poses, positions, and / or orientations measured along an object. As used herein, the term "distal" refers to a position closer to the process site, and the term "proximal" refers to a position further away from the process site. Thus, when an instrument is designed to perform a process, the distal portion or distal end of the instrument is closer to the process site than the proximal portion or proximal end of the instrument.

[0024] This disclosure occasionally refers to the application of the disclosed technology to a "patient" undergoing a "medical procedure." It should be understood that these references are not intended to limit the application of the disclosed technology to the specific medical context in which it is applied. For example, the described technology can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Therefore, any reference to the term "patient" is for illustrative purposes only, and it is also contemplated that the described technology can be applied to a "subject" in general.

[0025] This disclosure describes various types of image data. In some embodiments, the image data includes 2D images and / or 3D images. While the term "pixel" is used herein to refer to a specific point within the image data, the term "pixel" should not be construed as implying that the corresponding image data is a 2D image. In embodiments where the image data is 3D image data, the term "pixel" should be understood to refer to a "voxel" of the 3D image.

[0026] This disclosure relates to segmenting image data to overcome problems associated with existing segmentation algorithms. For example, deep learning models can be applied to generate confidence maps that indicate the level of confidence that any given pixel is associated with a blood vessel or other anatomical feature. However, deep learning models have several drawbacks. For instance, they rely on error-prone training data. Therefore, techniques relying on simple confidence thresholds produce a large number of false positives. Additionally, these inaccuracies tend to manifest in groupings of smaller, discrete connectivity components belonging to the same blood vessel. Therefore, techniques that filter out connectivity components tend to inappropriately remove these smaller connectivity components corresponding to blood vessels.

[0027] Additionally, while advanced algorithms can be applied to grow connected components to reduce discontinuities, these techniques are computationally intensive and may not be applicable within the time constraints of intraoperative settings. Other techniques that completely abandon deep learning models may not be generalizable to detecting other types of structures (e.g., lesions, ablation, tissue, etc.) within the subject's anatomy.

[0028] For example, the systems and methods described herein can apply two different confidence thresholds to the output of a deep learning model to generate two different binary maps (e.g., a matrix of values ​​corresponding to pixels in image data, where values ​​of "1" correspond to pixels with a confidence level above the threshold confidence level, and values ​​of "0" correspond to pixels with a confidence level below the threshold confidence level). The systems and methods can then detect connectivity components within the binary maps (e.g., consecutive sets of pixels associated with values ​​of "1"). It should be understood that although this disclosure uses "1" and "0" to refer to binary classifications associated with pixels, other binary classification representations are also conceivable.

[0029] To reduce false positives, the system and method discard connected components smaller than a threshold size from the binary map associated with the lower confidence level (“lower confidence binary map”) (e.g., reassigning the values ​​of the corresponding pixels to “0”). The system and method can then combine the binary map associated with the higher confidence threshold (“higher confidence binary map”) with the modified lower confidence binary map (e.g., by multiplication or performing an AND operation) to produce an output binary map.

[0030] Typically, higher-confidence binary maps provide more accurate boundaries of the target anatomical structure, while lower-confidence binary maps provide better information about the spread, length, and / or size of the target anatomical structure. Removing smaller connectivity components from only the lower-confidence binary map has several advantages. First, it maintains a higher confidence threshold in boundary regions near the connectivity components. That is, if pixels bordering the connectivity component in the higher-threshold binary map are contained within the connectivity component in the lower-threshold binary map, the disclosed technique is able to remove false positives that might occur when relying solely on the lower-threshold threshold. When the two binary maps are combined, pixels are zeroed based on the "0" assigned to pixels in the higher-threshold binary map. This ensures that the higher-confidence threshold is applied to the boundary regions of the connectivity components, where improved delineation between different types of anatomical structures (e.g., airway and blood vessel, blood vessel and lesion, lung boundary) can improve surgical efficiency.

[0031] Additionally, if two closely connected components exist in a higher-confidence binary graph, in many cases, these two connected components are connected in the lower-confidence binary graph. In some cases, the size of an individual connected component in the higher-confidence binary graph may be below a size threshold, while the combined connected components in the lower-confidence binary graph may be above the size threshold. Therefore, if the size threshold is applied only when relying on the higher-confidence binary graph, smaller connected components that actually exist may be inappropriately discarded. In other words, applying the size threshold to discard connected components only in the lower-confidence binary graph can help ensure that smaller connected components are not inappropriately discarded.

[0032] On the other hand, false positives isolated from connected components in a higher-threshold binary map are typically kept isolated in a lower-threshold binary map. Therefore, when a size threshold is applied to a lower-confidence binary map, false positives are discarded in the lower-threshold binary map and are absent when the two binary maps are combined. Thus, the system and method are able to remove false positives based on the size of the connected components and prevent inappropriately connecting closely spaced connected components at lower confidence levels.

[0033] According to certain aspects, the segmentation technique disclosed herein can be executed by a typical medical computing device within one minute of collecting an image dataset from an intraoperative image sensor. Therefore, the instantaneous segmentation technique can be executed quickly enough to provide the aforementioned benefits to the intraoperative procedure.

[0034] In some implementations, the systems and methods adjust the confidence level associated with the binary map based on the specific procedure being performed. For example, having a high confidence threshold during segmentation may be more important when performing a biopsy procedure compared to when performing therapeutic ablation. Therefore, in these implementations, the systems and methods can enable the operator to adjust the confidence threshold to the specific procedure being performed.

[0035] It should be understood that this improvement does not constitute an exhaustive list, and other improvements will be clear based on the various examples discussed in this article.

[0036] First refer to Figure 1 The GUI 100 provides a sample GUI (Guideline User 100) to users (or multiple users) to facilitate robot-assisted medical procedures. Specifically, the GUI 100 enables users to visualize, consider, and decide on actions for moving / guiding and / or manipulating minimally invasive medical devices (e.g., flexible elongating devices and therapeutic tools capable of extending from them) within the patient's anatomy. The flexible elongating device can be maneuverable using various controls, such as physical controls manipulated by the user (e.g., trackballs, scroll wheels, mice, etc., or virtual controls on the GUI 100 or another GUI). Figure 1In the example, the medical procedure is an endovascular ablation of a lesion in the patient's lung, and the flexible elongation device is a catheter that carries / contains an ablation probe capable of extending from the catheter. The ablation probe (e.g., a needle, balloon, and / or other structure) can be used to perform ablation using radiofrequency ablation, microwave ablation, cryoablation, electroporation, thermal ablation, or any other suitable ablation technique. See below. Figures 7 to 9B Example systems and devices / tools for endovascular ablation procedures are discussed in more detail. It should be understood that GUIs similar to GUI 100 can be used alternatively for other parts of the patient's anatomy (e.g., gastrointestinal surgery, cardiac surgery, etc.) and / or alternatively for medical procedures other than ablation, such as treatments involving injections into the target lesion and / or biopsies.

[0037] GUI 100 can be generated by one or more processors of one or more computing devices and / or systems (e.g., one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs)), which in turn enables a display device (e.g., a dedicated or general-purpose monitor, or a head-mounted display unit, etc.) to display GUI 100. For example, the processor can render GUI 100 and send corresponding signals / data to the display device for display. For ease of illustration, the following description refers to operation by the term "system," which can be any suitable system (controller, etc.) or (commonly) a system comprising one or more processors. References below... Figure 7 and Figure 8A Discuss specific examples of such systems, including systems or subsystems that can generate and render GUIs such as GUI 100.

[0038] like Figure 1 As seen, the example GUI 100 typically includes a visualization section 102 and a control section 104. The visualization section 102 depicts a model 110 of the lung airways in a patient's body, wherein the model 110 includes a visual representation 112a of the patient's blood vessels, a visual representation 112b of the patient's airways, and a virtual representation 114 of the target lesion. For ease of illustration, the visual representation 112a of the blood vessels is referred to herein as blood vessel 112a, the visual representation 112b of the airways is referred to herein as airway 112b, and the virtual representation 114 of the target lesion is referred to herein as target lesion 114. Although referred to in the singular herein, it should be understood that model 110 may consist of only a single model or may be a mixture of multiple models. For example, the system may model the lung airways and the target lesion separately (possibly based on different imaging modalities) and register the two models with each other for proper relative placement within the visualization section 102.

[0039] The system can generate model 110 based on preoperative imaging data and / or intraoperative imaging data. Preoperative and / or intraoperative imaging data can be captured using any suitable imaging technique / modality or multiple imaging techniques / modalities, such as computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. In some examples, the system generates an initial model 110 based on preoperative imaging data and then validates or updates the model 110 based on intraoperative imaging data (e.g., to correct inaccuracies in the initial model 110, such as inaccuracies in the structure of the lung airways or the size and / or location of lesions, which may be due to changes that have occurred since the preoperative images were captured). The process of updating the initial model 110 may include registering the intraoperative imaging data with the preoperative imaging data and / or with the model 110 itself. In some examples, different imaging modalities are used to capture preoperative and intraoperative imaging data. For instance, a CT imaging device can be used to capture preoperative imaging data, and a CBCT or fluoroscopic imaging device can be used to capture intraoperative imaging data.

[0040] In the example ablation navigation workflow, with the catheter within the patient's anatomy, an ablation treatment plan is created intraoperatively using GUI 100 during the ablation procedure. Initially, an imaging device (e.g., a CT imaging device) captures preoperative imaging data of the patient before the catheter is in the patient's body. Based on the preoperative imaging data, the system generates model 110 and identifies the target lesion (based on user segmentation or automatic segmentation using the preoperative imaging data) to be included in model 110 as target lesion 114. Additionally, the system identifies blood vessels (e.g., the pulmonary vascular system) to be included in model 110 as vessel 112a. When the catheter is in the patient's body, the system registers the catheter to model 110. Optionally (e.g., before the catheter is in the patient's body), a user (or another person) can use model 110 to plan a route / path to the target lesion 114 via airway 112b. In some cases, the user can perform a biopsy using the catheter, and the catheter can be repositioned near the target lesion, or left in place if already near the target lesion.

[0041] During the intraoperative procedure, the system can capture additional imaging data using an intraoperative imaging device (e.g., a CBCT imaging device) and use the intraoperative imaging data to verify and / or update the pose of the anatomical structures included within model 110. Therefore, the system can perform the disclosed segmentation techniques on the intraoperative imaging data using the segmentation techniques described herein.

[0042] As shown, the visualization portion 102 may also include 2D image data 120 for points selected in model 110. In the example shown, the selected points are associated with target lesion 114. The 2D image data 120 may include a dataset of images of the selected points captured from different axes. For example, the 2D image data 120 may include a first image 120a of the target lesion 114 along the axial axis, a second image 120b of the target lesion 114 along the coronal axis, and a third image 120c of the target lesion 114 along the sagittal axis. This provides the user with additional context to help develop a suitable ablation navigation workflow.

[0043] If a user touches or otherwise activates a control in control section 104, the system alters the visual appearance and other properties of the blood vessel 112. As an example, control 122 may allow the user to define a blood vessel size (e.g., thickness) threshold to change the visual appearance of portions of the blood vessel 112 in model 110 that meet the blood vessel size threshold. Thus, blood vessels 112a that meet the blood vessel size threshold are depicted in a first manner (e.g., in red), while blood vessels 112a that do not meet the blood vessel size threshold are depicted in a second manner (e.g., colorless).

[0044] As another example, control 122 allows a user to define the dimensions of a volume 116 associated with the target lesion 114, in which a virtual representation of the blood vessel 112a will be depicted. In this example, the virtual representation of the volume 116 can also change in size as the user changes control 122. In these implementations, the system can display only the blood vessels 112a within the volume. This allows the system to indicate the location of the blood vessels 112a for the portion of model 110 to be processed without obscuring the rest of model 110.

[0045] As shown, control portion 104 also includes control 124, which allows a user to change the opacity associated with the depiction of a visual representation of the patient's anatomy. Controls 122 and 124 allow a user to identify, for example, blood vessels 112a without obscuring the visual representation of the patient's airway 112b.

[0046] Given the controls provided by the system, errors during the segmentation process may result in blood vessels 112a and / or airways 112b being incorrectly represented in model 110. This could lead to unexpected situations when the user plans the following ablation navigation workflow, where the user may encounter unexpected situations when actually steering / driving the catheter toward the target lesion 114. Therefore, improving segmentation accuracy makes the execution of the workflow more efficient.

[0047] To improve segmentation accuracy, the system can implement the disclosed techniques for segmenting image data used to form model 110. Figure 2 An example workflow 200 is described for applying a deep learning model 225 to generate a confidence map 230 based on input image data 220 (e.g., preoperative imaging data and / or intraoperative imaging data that form the basis of model 110).

[0048] The deep learning model 225 can have any model architecture suitable for classifying image data 220. For example, deep learning models can include convolutional neural networks (CNNs), fully convolutional neural networks (FCNs), U-net models, Transformer models, etc. For some types of image data, U-net has been found to provide better segmentation performance than other types of deep learning models because image features detected at different resolutions can be propagated to the output.

[0049] Labeled image data can be used to train a deep learning model 225. In some embodiments, the deep learning model 225 is a single classifier model, such as a model trained to recognize only the presence of blood vessels. In other embodiments, the deep learning model 225 is a multi-classifier model trained to apply multiple labels to the image data 220. For example, the multi-classifier model may include classifiers specific to anatomical features (e.g., blood vessels, lung airways, lesions, etc.), classifiers associated with the shape of a structure (e.g., tubularity, size, etc.), classifiers associated with appearance (e.g., brightness), etc. In either case, the labeled image data may include labels associated with a label of interest. In some embodiments, the labeled image data is obtained from publicly available datasets (e.g., the National Institutes of Health's National Lung Screening Trial (NLST) dataset or the Cancer Imaging Archive's Non-Small Cell Lung Cancer (NSCLC) dataset). It should be understood that while the listed datasets specifically relate to lung anatomy segmentation, other public datasets can also be applied to systems that segment other parts of a patient's anatomy.

[0050] As shown, the output of deep learning model 225 is a confidence map 230, which includes confidence levels assigned to pixels in image data 220. These confidence levels indicate the confidence level of the deep learning model 225 to which a pixel should be classified using its corresponding label. In the example shown, the confidence levels included in confidence map 230 may indicate the confidence level at which a pixel should be classified as a blood vessel. In implementations where deep learning model 225 is a multi-classifier model, the confidence map may output a vector of confidence values ​​assigned to pixels by each classifier.

[0051] In these multi-classifier implementations, the system can perform a sanity check on the labels assigned to pixels to detect sets of classification labels that include inconsistent labels. One example of an inconsistent label set arises if the deep learning model 225 classifies a pixel as a blood vessel but located outside the lung. In this example, the pixel might be a false positive for the blood vessel classifier. The system can correct false positives by lowering the confidence level associated with the blood vessel classifier (e.g., setting the confidence level to a predetermined level, subtracting a predetermined value from the output confidence level, subtracting the value relative to the confidence level of the "outside the lung" classifier from the output confidence level, etc.). Another example of an inconsistent label set arises if the deep learning model 225 classifies a pixel as tubular and bright but not a blood vessel. In this example, the pixel might be a false negative for the blood vessel classifier. The system can correct false negatives by increasing the confidence level associated with the blood vessel classifier (e.g., setting the confidence level to a predetermined level, adding a predetermined value to the output confidence level, adding a value relative to the confidence level of the “tubular” and / or “brightness” classifiers, etc.).

[0052] It should be understood that in some implementations, the system for training the deep learning model 225 may be a different system from the system for applying the deep learning model to the image data 220. That is, the deep learning model 225 may be a pre-trained model to which the disclosed segmentation technique is applied to further improve the classification provided by the pre-trained model. It should be understood that when verifying the consistency of the label set, the system can still apply a default confidence threshold associated with the pre-trained deep learning model 225 when determining whether a label has been applied to a pixel. By checking for inconsistent label sets using the default confidence threshold, the system can remove false positives and / or false negatives from the dataset before applying the disclosed segmentation technique to the confidence map 230. This can further improve the ability of the disclosed segmentation technique to remove false positives that occur during the segmentation process.

[0053] Figure 3 Example processing for generating a binary map 335 based on a confidence map 330 (e.g., a confidence map 230 output by a deep learning model 225) is depicted. It should be understood that... Figure 3 Only a subset of the confidence graph 330 is depicted, and the confidence graph 330 may include additional values ​​extended across all three dimensions.

[0054] According to the technique described herein, the system can generate a first binary image 335a based on a higher confidence threshold and a second binary image 335b based on a lower confidence threshold. To generate a binary image, the system determines whether the confidence value is greater than or equal to the confidence threshold. If so, the system sets the value to "1"; otherwise, the system sets the value to "0". It should be understood that although the example shown uses .5 for the first confidence threshold and .4 for the second confidence threshold, this is only an example of suitable confidence thresholds, and specific values ​​can be adjusted based on the user's specific needs.

[0055] After generating the binary map 335, the system can identify the connectivity components 337 within it. As used herein, the term "connectivity component" refers to a contiguous set of pixels or voxels that is above a confidence threshold associated with the binary map. That is, a contiguous set of "1" values ​​in the binary map. In some implementations, a single pixel with a value of "1" that is not adjacent to any other pixel with a value of "1" can be considered a connectivity component. As shown, the binary map includes three connectivity components 337a, 337b, and 337c. In the example shown, connectivity component 337c is associated with the false positive output of the deep learning model. Therefore, if the application relies on processing that depends only on a single binary map, the false positive component 337c can be included in the output segmentation map.

[0056] Regarding the second binary image 335b, in addition to identifying connected components, the system also discards connected components that do not meet a size threshold (e.g., a predetermined number of pixels, the number of pixels relative to the size of a user-defined anatomical structure, etc.). For example, the threshold size can be defined such that the resulting segmented image data does not include a threshold number of connected components below the threshold size. To discard connected component 337, the system changes the value of the corresponding pixel from "1" to "0". Therefore, false positive connected component 337c does not have a corresponding connected component in the binary image 335b.

[0057] Continue to refer to Figure 4 Example processing for combining binary images 335a and 335b to produce an output binary image 335c is described. For example, the system can combine binary image 335a and binary image 335b by performing an AND operation 336. In other words, for any given pixel, the system multiplies the value in binary image 335a by the value in binary image 335b to calculate the corresponding value in binary image 335c. The system can then utilize binary image 335c when generating model 110 and / or indicating the location of, for example, blood vessel 112 in model 110.

[0058] As shown, when a lower confidence threshold is used to generate binary map 335b, connected components 337a and 337b are now connected via additional pixels that satisfy the lower confidence threshold. Therefore, connected component 337d indicates that a path exists between connected components 337a and 337b. Thus, if binary map 335b is relied upon in isolation, then... Figure 1 The ablation navigation workflow may unsuitably rely on connected component 337d, thus still generating paths through discontinuous pixels between connected components 337a and 337b. In other words, by combining the first binary map 335a and the second binary map 335b using the disclosed technique, the system is able to remove false positives 337c while still keeping the discrete connected components 337a and 337b separated. Based on experimental testing, the disclosed technique related to combining binary maps 335a and 335b enables the system to improve the confidence threshold associated with typical pre-trained deep learning models by 4%.

[0059] Next, refer to Figure 5 The diagram depicts an example graphical user interface 500 for selecting an operation type. The graphical user interface 500 can be displayed in the control section 104 of the graphical user interface 100.

[0060] As discussed above, different procedures have different accuracy requirements and therefore different tolerances for false positive errors. For example, an ablation procedure may require highly accurate segmentation of sensitive tissue (e.g., blood vessels) to ensure that the lesion is treated while minimizing damage to the sensitive tissue. On the other hand, biopsy treatment may cause less damage to sensitive tissue compared to an ablation procedure. Therefore, the segmentation of sensitive tissue (e.g., blood vessels) may not need to be as accurate as for an ablation procedure. Therefore, the confidence thresholds associated with binary plot 335a and / or binary plot 335b can be set higher for biopsy procedures than for ablation procedures.

[0061] If a user touches or otherwise activates control 526 in control section 102, the system allows the user to select a process type. Based on this selection, the system can adjust the confidence threshold used to generate binary images 335a, 335b, or in some cases, use a single binary image instead of combining them. In some implementations, the system can then obtain the latest image data 220 and reapply the disclosed segmentation technique using the updated threshold.

[0062] Figure 6An example flowchart 600 is depicted for segmenting medical image data (e.g., image data 220 on which generation model 110 depends). Flowchart 600 may be executed by one or more processors that execute instructions stored in one or more computer-readable media (e.g., non-volatile memory), such as those combined below. Figures 7 to 9B The various processors that execute the system or subsystem under discussion.

[0063] At box 602, the system acquires multiple image data (e.g., image data 120, 220) depicting a portion of the subject. The image data may include preoperative or intraoperative image data. In some embodiments, the image data includes 3D image data, such as computed tomography (CT) image data, cone-beam computed tomography (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRI) image data.

[0064] At box 604, the system inputs multiple image data sets into a deep learning model (e.g., deep learning model 225), which is trained to generate confidence maps (e.g., confidence map 235) that indicate the confidence level of one or more portions of the image data in depicting an anatomical structure of the subject. For example, the anatomical structure may be a sensitive structure such as blood vessels, pleura, lung fissures, etc. In some implementations, the deep learning model includes the U-net model.

[0065] In some implementations, the deep learning model is a multi-classifier model configured to apply two or more classifiers to each portion of the image data. In these implementations, box 604 may include identifying a set of inconsistent labels applied to a specific portion of the image data by two or more classifiers, and adjusting the confidence level for that portion. As an example, the inconsistent set of labels includes a set of labels indicating that the portion of the image data is a blood vessel and is outside the lung. In this example, the system may decrease the confidence level for that portion. As another example, the inconsistent set of labels includes a set of labels indicating at least one of the following: tubular features above a threshold, brightness features above a threshold, or blood vessel features below a threshold. In this example, the system may increase the confidence level for that portion.

[0066] At box 606, the system generates a first binary map based on the confidence map and a first confidence threshold. For example, the system may apply the techniques described in section 335a regarding the generation of the binary map.

[0067] At points 608 and 610, the system generates a second binary map based on the confidence map and a second confidence threshold, respectively, and discards one or more connected components in the second binary map that are smaller than the threshold size. For example, the system may apply the technique described in section 335b regarding the generation of the binary map. The second confidence threshold may be lower than the first confidence threshold.

[0068] At box 612, the system causes the display device to display a graphical user interface (GUI) (e.g., GUI 100) based on at least a portion of multiple image data. This portion of the multiple image data can be labeled based on a first binary image and a second binary image. To label this portion of the image data, the system can combine the first binary image and the second binary image, for example, by multiplying the first binary image and the second binary image.

[0069] like Figure 1 As shown, the GUI can change the visual representation of the anatomical structure represented by model 110 based on labels. As an example, the GUI may include a thickness selection control and / or a volume size adjustment control (e.g., control 122). In response to interaction with the thickness selection control and / or the volume size adjustment control, the system can apply thickness and / or volume size standards to update the display of portions of the image data accordingly. As another example, the GUI may include a process type control for indicating the process type (e.g., control 526). In response to interaction with the process selection control, the system can adjust a first threshold and a second threshold based on the indicated process type.

[0070] Figures 7 to 9B A diagram is depicting a medical system that, in some examples, can be used to manipulate a medical device according to any of the methods and systems described above.

[0071] Figure 7 This is a simplified diagram of a medical system 700 based on some examples. The medical system 700 can be applied to procedures such as surgery, diagnosis (e.g., biopsy), or treatment (e.g., ablation, electroporation, etc.). While some examples of these procedures are provided herein, any references to medical or surgical instruments and methods are non-limiting. The systems, instruments, and methods described herein can be used with animals, human cadavers, animal carcasses, parts of human or animal anatomy, for non-surgical diagnosis, and for industrial systems, general-purpose or special-purpose robotic systems, general-purpose or special-purpose remote operating systems, or robotic medical systems.

[0072] like Figure 7As shown, the medical system 700 may include a manipulator assembly 702 that controls the operation of a medical device 704 during various procedures performed on a patient P. The medical device 704 may extend into an internal part of the patient P through an opening within the patient P. The manipulator assembly 702 may be a remotely operated, non-remotely operated, or hybrid remotely and non-remotely operated assembly, having one or more degrees of freedom of motion that can be electrically operated and / or one or more degrees of freedom of motion that can be non-electrically operated (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near the patient table T. A master assembly 706 enables an operator O (e.g., a surgeon, clinician, internist, or other user) to control the manipulator assembly 702. In some examples, the master assembly 706 enables the operator O to view the procedure site or other graphical or information displays. In some examples, the manipulator assembly 702 may be excluded from the medical system 700, and the device 704 may be directly controlled by the operator O. In some examples, the manipulator assembly 702 can be manually controlled by operator O. Direct operator control may include various handles and operator interfaces for handheld operation of the instrument 704.

[0073] The main component 706 may be located at a surgeon's console near the patient table T where the patient P is located (e.g., in the same room as the patient table T), such as at the side of the patient table T. In some examples, the main component 706 is located away from the patient table T, such as in a different room or a different building. The main component 706 may include one or more control devices for controlling the manipulator component 702. The control devices may include any number of various input devices, such as joysticks, trackballs, rollers, steering pads, buttons, data gloves, trigger guns, manual controllers, voice recognition devices, motion or presence sensors, etc. In some examples, the main component 706 may be or include an extended reality (XR) device, such as a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or any other such device described herein.

[0074] Manipulator assembly 702 supports medical device 704 and may include a kinematic structure of links providing a setting structure. Links may include one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place) and / or one or more servo-controlled links (e.g., one or more links that can be controlled in response to commands, for example, from control system 712). Manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on medical device 704 in response to commands, for example, from control system 712. Actuators may include a drive system that moves medical device 704 in various ways when coupled to it. For example, one or more actuators may advance medical device 704 into a natural or surgically generated anatomical opening. Actuators may control engagement of medical device 704, for example, by moving the distal end (or any other part) of medical device 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control the rotation of the medical device about its longitudinal axis. The actuators may also be used to move the engageable end effector of the medical device 704 (e.g., for grasping tissue in the jaws of a biopsy device, etc.), or may be used to move or otherwise control tools inserted within the medical device 704 (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.). According to the example, the manipulator assembly 702 may include, or as described above regarding... Figures 1 to 4 A more detailed description of the robot-assisted platform. Similarly, medical device 704 can be or includes the features described above. Figures 1 to 4 The components of the medical device described.

[0075] The medical system 700 may include a sensor system 708 having one or more subsystems for receiving information about the manipulator assembly 702 and / or the medical device 704. Such subsystems may include: a position sensor system (e.g., using an electromagnetic (EM) sensor or other type of sensor for detecting position or location); a shape sensor system for determining the position, orientation, velocity, rate, pose, and / or shape along one or more segments and / or distal ends of the flexible body of the medical device 704; a visualization system (e.g., using a color imaging device, infrared imaging device, ultrasound imaging device, X-ray imaging device, fluorescence imaging device, computed tomography (CT) imaging device, magnetic resonance imaging (MRI) imaging device, or some other type of imaging device) for capturing images, for example, from the distal end of the medical device 704 or from some other location; and / or an actuator position sensor (e.g., a resolver, encoder, potentiometer, etc.) describing the rotation and / or orientation of the actuator controlling the medical device 704.

[0076] Medical system 700 may include a display system 710 for displaying images or representations of the procedure site and medical device 704. The display system 710 and main component 706 may be oriented such that a physician O can use telepresent perception to control the medical device 704 and main component 706. In some examples, although the display system 710 and main component 706 are in... Figure 7 While depicted as a separate box, the display system 710 and the main component 706 can both be part of the same device and / or operating control system.

[0077] In some examples, medical device 704 may include a visualization system that includes an image capture component that records simultaneous or real-time images of the procedure site and provides the images to an operator O via one or more displays of display system 710. The image capture component may include various types of imaging devices. The simultaneous images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the anatomical procedure site. In some examples, the visualization system may include an endoscope component that may be integrally or detachably coupled to medical device 704. Additionally or alternatively, a separate endoscope attached to a separate manipulator assembly may be used with medical device 704 to image the procedure site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as control system 712.

[0078] Display system 710 can also display images of the process site and medical device, which can be captured by a visualization system. In some examples, medical system 700 provides operator O with a telepresented perception. For example, an image captured by an imaging device at the distal portion of medical device 704 can be presented by display system 710 to provide operator O with a perception of the distal portion of medical device 704. Inputs provided by operator O to master component 706 can move the distal portion of medical device 704 in a manner corresponding to the nature of the input (e.g., the distal end turns to the right when the trackball is rolled to the right), and cause a corresponding change in the viewing angle of the image captured by the imaging device at the distal portion of medical device 704. Thus, operator O's telepresent perception is maintained when medical device 704 is moved using master component 706. Operator O can manipulate the hand controls of master component 706 and medical device 704 as if viewing a workspace in a substantially real-world setting, simulating the experience of physically manipulating medical device 704 from within the patient's anatomy.

[0079] In some examples, the display system 710 may present virtual images of the procedure site created using image data recorded preoperatively (e.g., before the procedure performed by the medical device system 800) or intraoperatively (e.g., simultaneously with the procedure performed by the medical device system 800), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluorescence microscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The virtual images may include two-dimensional, three-dimensional, or higher-dimensional images (e.g., including information based on time or rate). In some examples, one or more models are created based on the preoperative or intraoperative image dataset, and the virtual image is generated using one or more models.

[0080] In some examples, for the purpose of image-guided medical procedures, the display system 710 can display virtual images generated based on the positioning of the tracking medical device 704. For example, the tracking positioning of the medical device 704 can be registered with a model generated using preoperative or intraoperative images (e.g., a dynamic reference), where different parts of the model correspond to different positions of the patient's anatomy. As the medical device 704 moves through the patient's anatomy, the registration is used to determine the parts of the model corresponding to the positioning and / or viewpoint of the medical device 704, and virtual images are generated using the determined parts of the model. This presents the operator O with a virtual image of the internal process site corresponding to the tracking positioning of the medical device 704, based on the viewpoint of the medical device 704.

[0081] The medical system 700 may further include a control system 712, which may include processing circuitry to implement some or all of the methods or functions discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operation of the manipulator assembly 702, medical device 704, main assembly 706, sensor system 708, and / or display system 710. The control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing instructions) that, when executed by at least one processor, configure one or more processors to implement some or all of the methods or functions discussed herein. Although the control system 712... Figure 7 While shown as a single block, the control system 712 may include two or more separate data processing circuits, with some processing performed at the manipulator component 702, others at the main component 706, and so on. In some examples, the control system 712 may include other types of processing circuitry systems, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). The control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0082] In some examples, the control system 712 may receive feedback from the medical device 704, such as force and / or torque feedback. In response to this feedback, the control system 712 may transmit a signal to the main component 706. In some examples, the control system 712 may transmit a signal instructing one or more actuators of the manipulator component 702 to move the medical device 704. In some examples, the control system 712 may transmit information about the feedback to a display system 710 for presentation or to perform other types of actions based on the feedback.

[0083] Control system 712 may include a virtual visualization system to provide navigation assistance to operator O when controlling medical device 704 during image-guided medical procedures. Virtual navigation using the virtual visualization system may be based on a preoperative or intraoperative dataset of the acquired anatomical pathways of patient P. Control system 712 or a separate computing device may, alone or in combination with operator input, use programmed instructions to convert recorded images into a model of the patient's anatomy. This model may include a segmented two-dimensional or three-dimensional synthetic representation of parts or entire anatomical organs or regions. The image dataset may be associated with the synthetic representation. The virtual visualization system may obtain sensor data from sensor system 708 for calculating (e.g., approximate) positioning of medical device 704 relative to the anatomical structures of patient P. Sensor system 708 may be used to register and display medical device 704 and images recorded preoperatively or intraoperatively. For example, PCT disclosure WO 2016 / 161298 (published December 1, 2016, entitled "Systems and Methods of Registration for Image Guided Surgery") discloses an example system, which is incorporated herein by reference in its entirety.

[0084] During the virtual navigation process, sensor system 708 can be used to calculate (e.g., approximate) localization of medical device 704 relative to the anatomical structure of patient P. This localization can be used to generate both macroscopic (e.g., external) tracking images of the anatomical structure of patient P and virtual internal images of the anatomical structure of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display the medical device and preoperatively recorded medical images. For example, U.S. Patent No. 8,300,131 (filed May 13, 2011, entitled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated herein by reference in its entirety, discloses an example system.

[0085] The medical system 700 may also include operating and support systems (not shown), such as lighting systems, steering and maneuvering control systems, flushing systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on factors such as the medical procedure and space constraints within the operating room. Multiple master assemblies may be located in the same location or may be positioned in separate locations. Multiple master assemblies allow more than one operator to control one or more manipulator assemblies in various combinations.

[0086] Figure 8A This is a simplified diagram of a medical device system 800 based on some examples. The medical device system 800 includes a flexible elongation device 802 (also referred to as elongation device 802), a drive unit 804, and a medical tool 826, which together serve as an example of a medical device 704 in the medical system 700. The medical system 700 can be a remote operating system, a non-remote operating system, or a hybrid of remote and non-remote operating systems, as shown in reference... Figure 7 As described above, the visualization system 831, tracking system 830, and navigation system 832 are also included. Figure 8A The diagram shows an example component of the control system 712 of the medical system 700. In some examples, the medical device system 800 can be used in non-remotely operated exploration procedures or in procedures involving routine manual operation of medical devices (e.g., endoscopy). The medical device system 800 can be used to collect (e.g., measure) a set of data points corresponding to locations within the anatomical passage of a patient (e.g., patient P).

[0087] The elongation device 802 is coupled to the drive unit 804. The elongation device 802 includes a channel 821 through which a medical instrument 826 can be inserted. The elongation device 802 navigates within the patient's anatomy to deliver the medical instrument 826 to the procedure site. The elongation device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters.

[0088] Medical device system 800 may include a tracking system 830 for determining the position, orientation, velocity, rate, pose, and / or shape of a flexible body 816 at its distal end 818 and / or along one or more segments 824 of the flexible body 816, as will be described in further detail below. Tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816 (e.g., the length between the distal end 818 and the proximal end 817) may include multiple segments 824. Tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, tracking system 830 is... Figure 7 Part of the control system 712 shown.

[0089] The tracking system 830 can use a shape sensor 822 to track the distal end 818 and / or one or more segments 824 of the flexible body 816. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., disposed within an internal channel of the flexible body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter of approximately 800 μm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 can form an optical fiber bending sensor for determining the shape of the flexible body 816. Optical fibers including fiber Bragg gratings (FBGs) can be used to provide strain measurements of the structure in one or more dimensions. Various systems and methods applicable to monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. Patent Application Publication No. 8006 / 0013523 (filed July 13, 2005, entitled "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent No. 7,772,541 (filed March 12, 2008, entitled "Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter"), and U.S. Patent No. 8,773,350 (filed September 2, 2010, entitled "Optical Position and / or Shape Sensing"), all of which are incorporated herein by reference in their entirety. In some examples, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering.

[0090] In some examples, other techniques may be used to determine the shape of the flexible body 816. For example, the history of the position and / or pose of the distal end 818 of the flexible body 816 may be used to reconstruct the shape of the flexible body 816 over time intervals, such as when the flexible body 816 advances or retracts within a patient's anatomy. In some examples, the tracking system 830 may alternatively and / or additionally use a position sensor system 820 to track the distal end 818 of the flexible body 816. The position sensor system 820 may be a component of an EM sensor system, wherein the position sensor system 820 includes one or more position sensors. Although the position sensor system 820 is shown proximity to the distal end 818 of the flexible body 816 to track the distal end 818, the number and positioning of the position sensors in the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that can withstand externally generated electromagnetic fields. Each coil of the position sensor system 820 may generate an induced electrical signal having characteristics that depend on the position and orientation of the coil in relation to the externally generated electromagnetic field. The position sensor system 820 can measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of the flexible body 816. In some examples, the position sensor system 820 can be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, and Z, and three orientation angles indicating pitch, yaw, and roll of a reference point. In some examples, the position sensor system 820 can be configured and positioned to measure five degrees of freedom, such as three position coordinates X, Y, and Z, and two orientation angles indicating pitch and yaw of a reference point. Further description of the position sensor system applicable to some examples is provided in U.S. Patent No. 6,380,432 (filed August 11, 1999, entitled "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0091] In some examples, the tracking system 830 may alternatively and / or additionally rely on a set of pose, position, and / or orientation data stored at points for the elongation device 802 and / or medical instrument 826, captured during one or more cycles of alternating movement (e.g., breathing). This stored data can be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown)—such as EM sensors like those in position sensor 820 or some other type of position sensor—can be positioned along the flexible body 816 and used for shape sensing. In some examples, the history of data acquired during the process from one or more of these position sensors can be used to represent the shape of the elongation device 802, particularly where the anatomical passage is typically static.

[0092] Figure 8B This is a simplified diagram of a medical tool 826 within an elongation device 802, based on some examples. The flexible body 816 of the elongation device 802 may include a channel 821 sized and shaped to accommodate the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as diagnostics, imaging, surgery, biopsy, ablation, illumination, irrigation, aspiration, electroporation, etc. The medical tool 826 can be deployed through the channel 821 of the flexible body 816 and operate at the procedure site within an anatomical structure. The medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, gripper, brush, etc.), an ablation tool (e.g., a laser ablation tool, a radiofrequency (RF) ablation tool, a cryoablation tool, a thermal ablation tool, a heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector with a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, etc. Other end effector types can include, for example, forceps, grippers, scissors, sutures, clamps, etc. Other end effectors can also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc.

[0093] Medical tool 826 may be a biopsy tool for removing sample tissue or cell samples from a targeted anatomical location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may also include a sheath that can surround the flexible needle to protect the needle and the inner surface of the channel 821 when the biopsy tool is within the channel 821. Medical tool 826 may be an image capture probe that includes a distal portion having a stereo or single-field-of-view camera that can be positioned at or near the distal end 818 of the flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by visualization system 831 for display and / or provided to tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and / or one or more segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting the captured image data, the cable being coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a bundle of optical fibers, such as a fiber optic endoscope, coupled to a closer imaging device, such as the visualization system 831. The image capture probe may be monospectral or multispectral, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectra. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, X-ray, fluorescein, CT, MRI, or other types of imaging techniques.

[0094] In some examples, an image capture probe is inserted within the flexible body 816 of the elongation device 802 to facilitate visual navigation of the elongation device 802 to the procedure site, and then the image capture probe is replaced within the flexible body 816 with another type of medical instrument 826 for performing the procedure. In some examples, the image capture probe may be located together with another type of medical instrument 826 within the flexible body 816 of the elongation device 802 to facilitate simultaneous image capture and tissue intervention, for example, within the same channel 821 or in different channels. The medical instrument 826 may advance from an opening in the channel 821 to perform the procedure (or some other function) and then retract into the channel 821 upon completion of the procedure. The medical instrument 826 may be removed from the proximal end 817 of the flexible body 816 or along the flexible body 816 from another optional instrument port (not shown).

[0095] In some examples, the extension device 802 may include integrated imaging capabilities instead of utilizing a removable image capture probe. For example, the imaging device (or fiber bundle) and light emitter may be located at the distal end 818 of the extension device 802. The flexible body 215 may include one or more dedicated channels carrying cables and / or optical fibers between the distal end 818 and the visualization system 831. Here, the medical device system 800 can perform imaging and tooling operations simultaneously.

[0096] In some examples, the medical tool 826 is capable of controlled engagement. The medical tool 826 may house a cable (also referred to as a traction cable), linkage, or other actuation controls (not shown), extending between its proximal and distal ends to controllably bend the distal end of the medical tool 826, such as those discussed herein with respect to the flexible elongation device 802. The medical tool 826 may be coupled to a drive unit 804 and a manipulator assembly 702. In these examples, the elongation device 802 may be excluded from the medical device system 800, or may be a flexible device without controlled engagement. The steerable maneuvering apparatus or tool applicable to some examples is further described in detail in U.S. Patent No. 7,916,681 (filed October 4, 2005, entitled "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent No. 9,259,274 (filed September 30, 2008, entitled "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0097] The flexible body 816 of the elongation device 802 may also, or alternatively, accommodate a cable, linkage, or other steering control (not shown) extending between the drive unit 804 and the distal end 818 to controllably bend the distal end 818, such as, for example, by Figure 8A The distal end 818 is depicted by a dashed line as shown in Figure 819. In some examples, at least four cables are used to provide independent up-and-down steering maneuvers to control the pitch of the distal end 818 and left-and-right steering maneuvers to control the yaw of the distal end 281. In these examples, the flexible elongation device 802 may be a steerable maneuverable conduit. Examples of steerable maneuverable conduits suitable for some examples are described in detail in PCT Publication WO 2019 / 018436 (published January 24, 2019, entitled "Flexible Elongate Device Systems and Methods"), which is incorporated herein by reference in its entirety.

[0098] In examples where the elongation device 802 and / or medical tool 826 is actuated by a remotely operated component (e.g., manipulator component 702), the drive unit 804 may include a drive input detachably coupled to and receiving power from a drive element (e.g., an actuator) of the remotely operated component. In some examples, the elongation device 802 and / or medical tool 826 may include a grasping feature, a manual actuator, or other components for manually controlling the movement of the elongation device 802 and / or medical tool 826. The elongation device 802 may be steerable, or alternatively, it may be non-steerable, without an integrated mechanism for operator control of bending of the distal end 818. In some examples, one or more channels 821 (which may also be referred to as lumens) may be defined by the inner wall of the flexible body 816 of the elongation device 802, through which the medical tool 826 may be deployed and used at a target anatomical location.

[0099] In some examples, medical device system 800 (e.g., extension device 802 or medical tool 826) may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes, for the examination, diagnosis, biopsy, and / or treatment of the lungs. Medical device system 800 may also be adapted to navigate and treat other tissues in any anatomical system of a variety of anatomical systems via naturally or surgically generated access channels, including the colon, intestine, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, etc.

[0100] Information from tracking system 830 can be sent to navigation system 832, where it can be combined with information from visualization system 831 and / or a preoperatively acquired model to provide real-time location information to physicians, clinicians, surgeons, or other operators. In some examples, the real-time location information can be displayed on display system 710 for use in the control of medical device system 800. In some examples, navigation system 832 can utilize the location information as feedback for locating medical device system 800. U.S. Patent No. 8,300,131 (filed May 13, 2011, and entitled “Medical System Providing Dynamic Registration of a Model of an Anatomical Structure for Image-Guided Surgery”) provides various systems applicable to some examples for using fiber optic sensors to register and display surgical instruments and surgical images, which are incorporated herein by reference in their entirety.

[0101] Figure 9A and Figure 9BThis is a simplified diagram based on some examples, including a side view of a medical device mounted on an insertion assembly in patient coordinate space. (e.g.) Figure 9A and Figure 9B As shown, the surgical environment 900 may include a patient P positioned on a patient table T. Patient P may be stationary within the surgical environment 900 because overall patient movement is restricted by sedation, restraint, and / or other means. Periodic anatomical movements of patient P (including respiratory and cardiac movements) may continue. Within the surgical environment 900, a medical device 904 is used to perform medical procedures, which may include, for example, surgery, biopsy, ablation, illumination, irrigation, aspiration, or electroporation. The medical device 904 may also be used to perform other types of procedures, such as a registration process that associates position, orientation, and / or pose data captured by a sensor system 708 with a desired (e.g., anatomical or systemic) reference frame. The medical device 904 may be, for example, a medical device 704. In some examples, the medical device 904 may include an elongation device 910 (e.g., a catheter) coupled to an instrument body 912. The elongation device 910 includes one or more channels sized and shaped to accommodate medical instruments.

[0102] The elongation device 910 may also include one or more sensors (e.g., components of sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may move with the instrument body 912, and the positioning of the proximal point 916 relative to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure the shape from the proximal point 916 to another point (e.g., the distal end 918 of the elongation device 910). The shape sensor 914 may be aligned with the elongation device 910 (e.g., disposed within an internal channel or mounted externally). In some examples, the shape sensor 914 may use optical fibers to generate shape information of the elongation device 910.

[0103] In some examples, position sensors (e.g., EM sensors) may be incorporated into medical device 904. A series of position sensors may be positioned along the flexible elongation device 910 and used for shape sensing. Position sensors may be used in place of or in conjunction with shape sensor 914, for example, to improve the accuracy of shape sensing or to verify shape information.

[0104] The extension device 910 may accommodate cables, linkages, or other steering control mechanisms that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up-and-down steering control to control the pitch of the distal end 918 and to provide left-and-right steering control to control the yaw of the distal end 918. The instrument body 912 may include a drive input that is removably coupled to and receives power from a drive element (e.g., an actuator) of the manipulator assembly.

[0105] The instrument body 912 may be coupled to the instrument holder 906. The instrument holder 906 may be mounted to an insertion stage 908 fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but has a known position within the surgical environment 900 (e.g., via a tracking sensor or other tracking device). The instrument holder 906 may be part of a manipulator assembly (e.g., manipulator assembly 702) coupled to the medical device 904 to control the movement of the distal end 918 of the extension device 910 in multiple directions (e.g., yaw, pitch, and / or roll) and / or insertion movement (e.g., movement along the insertion axis A). The instrument holder 906 or the insertion stage 908 may include actuators, such as servo motors, for controlling the movement of the instrument holder 906 along the insertion stage 908.

[0106] Sensor device 920 (which may be a component of sensor system 708) can provide information about the position of instrument body 912 as it moves along insertion axis A relative to insertion stage 908. Sensor device 920 may include one or more rotary transformers, encoders, potentiometers, and / or other sensors that measure the rotation and / or orientation of actuators controlling the movement of instrument carriage 906, thereby indicating the movement of instrument body 912. In some examples, insertion stage 908 has, for example, […]. Figure 9A and Figure 9B The linear track is shown. In some examples, the insertion stage 908 may have a curved track or a combination of curved track segments and linear track segments.

[0107] Figure 9A The instrument body 912 and instrument holder 906 are shown in the retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is located at position L0 on the insertion axis A. The positioning of the proximal point 916 can be set to zero and / or other reference values ​​to provide a basic reference (e.g., corresponding to the origin of the desired reference system) to describe the position of the instrument holder 906 along the insertion stage 908. In the retracted position, the distal end 918 of the extension device 910 can be positioned precisely within the inlet orifice of the patient P. Also in the retracted position, data captured by the sensor device 920 can be set to zero and / or other reference values ​​(e.g., I=0). Figure 9BIn this configuration, the instrument body 912 and instrument holder 906 have advanced along the linear track of the insertion stage 908, and the distal end 918 of the extension device 910 has advanced into the patient P. At this advanced position, the proximal point 916 is at position L1 on the insertion axis A. In some examples, rotation and / or orientation of the actuator measured by a sensor device 920 indicating the movement of the instrument holder 906 along the insertion stage 908 and / or by one or more position sensors associated with the instrument holder 906 and / or the insertion stage 908 can be used to determine the position L1 of the proximal point 916 relative to position L0. In some examples, position L1 can also serve as an indicator of the distance or insertion depth of the distal end 918 of the extension device 910 into the channel of the anatomical structure of the patient P.

[0108] One or more components of the examples discussed in this disclosure (e.g., control system 712) can be implemented in software to execute on one or more processors of a computer system. The software may include code that, when executed by one or more processors, configures the processors to perform the various functions discussed herein. The code may be stored in a non-transitory computer-readable storage medium (e.g., memory, magnetic storage device, optical storage device, solid-state storage device, etc.). The computer-readable storage medium may be part of a computer-readable storage device, such as electronic circuitry, a semiconductor device, a semiconductor memory device, a read-only memory (ROM), flash memory, an erasable programmable read-only memory (EPROM), a floppy disk, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded to the computer-readable storage medium for storage via a computer network such as the Internet, an intranet, etc. The code may be executed by any of a variety of centralized or distributed data processing architectures. The programming instructions of the code may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the system described herein. Components of the computing system discussed herein may be connected using wired and / or wireless connections. In some examples, wireless connectivity can use wireless communication protocols such as Bluetooth, Near Field Communication (NFC), Infrared Data Association (IrDA), Home RF, IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and Wireless Medical Telemetry Service (WMTS).

[0109] Various general-purpose computer systems can be used to perform one or more processes, methods, or functions described herein. Additionally or alternatively, various special-purpose computer systems can be used to perform one or more processes, methods, or functions described herein. Furthermore, various programming languages ​​can be used to implement one or more processes, methods, or functions described herein.

[0110] While certain examples and illustrations have been described above and shown in the accompanying drawings, it should be understood that these examples and illustrations are merely exemplary and not limited to the specific constructions and arrangements shown and described, as various other alternatives, modifications and equivalents will be understood by those skilled in the art.

Claims

1. A system for segmenting medical image data, the system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to: obtain a plurality of image data depicting a portion of a subject; input the plurality of image data into a deep learning model trained to generate a confidence map indicating a level of confidence that one or more portions of the image data depict an anatomical structure of the subject; generate a first binary map based on the confidence map and a first confidence threshold; generate a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; discard one or more connected components in the second binary map that are below a threshold size; and cause a display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image data, wherein the portion of the plurality of image data is labeled based on the first binary map and the second binary map. The image data comprises 3D image data.

2. The system of claim 1, wherein, The 3D image data comprises computed tomography (CT) image data, cone beam CT (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRI) image data.

3. The system of claim 1 or 2, wherein, The anatomical structure comprises a blood vessel.

4. The system of any one of claims 1 to 3, wherein, The deep learning model is a multi-classifier model configured to apply two or more classifiers to each portion of the image data.

5. The system of any one of claims 1 to 4, wherein, The instructions, when executed, cause the system to:

6. The system of claim 5, wherein, identify an inconsistent set of labels applied to a portion of the image data by the two or more classifiers and adjust a level of confidence for the portion of the image data.

7. The system of claim 6, wherein: the inconsistent set of labels comprises a set of labels indicating that the portion of the image data is a blood vessel and is outside of a lung; and the instructions, when executed, cause the system to decrease the level of confidence.

8. The system of claim 6, wherein: the inconsistent set of labels comprises a set of labels indicating one or more of: a tubular feature is above a threshold, a brightness feature is above a threshold, or a blood vessel feature is below a threshold; and the instructions, when executed, cause the system to increase the level of confidence. To label the portion of the plurality of image data based on the first binary map and the second binary map, the instructions, when executed, cause the system to:

9. The system of any one of claims 1 to 8, wherein, combine the first binary map and the second binary map. To combine the first binary map and the second binary map, the instructions, when executed, cause the system to:

10. The system of claim 9, wherein, multiply the first binary map and the second binary map.

11. The system of any of claims 1-10, wherein: the GUI comprises a procedure type control for indicating a type of procedure; and the procedure type control is a drop-down menu. the instructions, when executed, cause the system to adjust the first confidence threshold and the second confidence threshold based on the type of procedure.

12. The system of any of claims 1-11, wherein: the GUI includes a thickness selection control or a volume size adjustment control; and the instructions, when executed, cause the system to: apply a thickness criterion or a volume size criterion to update a display of the portion of the image data.

13. The system of any one of claims 1 to 12, wherein, the deep learning model includes a U-net model.

14. The system of any one of claims 1 to 13, wherein, the one or more connected components include a set of contiguous pixels or voxels above a confidence threshold associated with a binary map.

15. A method for segmenting medical image data, the method comprising: obtaining, via one or more processors, a plurality of image data depicting a portion of a subject; inputting, via the one or more processors, the plurality of image data into a deep learning model trained to generate a confidence map indicating a level of confidence that one or more portions of the image data depict an anatomical structure of the subject; generating, via the one or more processors, a first binary map based on the confidence map and a first confidence threshold; generating, via the one or more processors, a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; discarding, via the one or more processors, one or more connected components in the second binary map that are below a threshold size; and causing, via the one or more processors, a display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image data, wherein the portion of the plurality of image data is labeled based on the first binary map and the second binary map.

16. The method of claim 15, wherein, the image data includes 3D image data.

17. The method of claim 15 or 16, wherein, the 3D image data includes computed tomography (CT) image data, cone beam CT (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRI) image data.

18. The method of any one of claims 15-17, wherein, the anatomical structure includes a blood vessel.

19. The method of any one of claims 15 to 18, wherein, the deep learning model is a multi-classifier model configured to apply two or more classifiers to each portion of the image data.

20. The method of claim 19, further comprising: identifying, via the one or more processors, a set of inconsistent labels applied to a portion of the image data by the two or more classifiers; and adjusting, via the one or more processors, a confidence level for the portion of the image data.

21. The method of claim 20, wherein: the set of inconsistent labels includes a set of labels indicating that the portion of the image data is a blood vessel and outside of a lung; and adjusting a confidence level for the portion includes decreasing, via the one or more processors, the confidence level.

22. The method of claim 20, wherein: The set of inconsistent labels includes a set of labels indicating one or more of: a tubular feature above a threshold, a brightness feature above a threshold, or a blood vessel feature below a threshold; And Adjusting the confidence level for the portion includes increasing, via the one or more processors, the confidence level.

23. The method of any of claims 15-22, labeling the portion of the plurality of image data based on the first binary map and the second binary map includes: combining, via the one or more processors, the first binary map and the second binary map.

24. The method of claim 23, wherein, Combining the first binary map and the second binary map includes: multiplying, via the one or more processors, the first binary map and the second binary map.

25. The method of any of claims 15-24, wherein: the GUI includes a procedure type control for indicating a procedure type; and the method further includes adjusting, via one or more processors, the first threshold and the second threshold based on the procedure type.

26. The method of any of claims 15-25, wherein: the GUI includes a thickness selection control or a volume size adjustment control; and the method further includes: applying, via the one or more processors, a thickness criterion or a volume size criterion to update a display of the portion of the image data.

27. The method of any one of claims 15 to 26, wherein, The deep learning model includes a U-net model.

28. The method of any one of claims 15 to 27, wherein, The connected component includes a set of contiguous pixels or voxels above a confidence threshold associated with a binary map.

29. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 15-28.

Images