Convolutional Neural Networks for Effective Tissue Segmentation

Abstract

An imaging system for pixel-level segmentation of an image is provided, comprising: a camera that captures an image of an anatomical object and represents the image in a two-dimensional (2D) pixel arrangement; one or more processors and a non-transitory computer-readable medium having information including CNN instructions and multiple sets of weights, the CNN instructions causing the one or more processors to implement the CNN, the CNN being configured to associate an anatomical object classification with pixels in the 2D pixel arrangement, the multiple sets of weights being configured differently for the CNN based on different camera image training data; and a display screen configured to display the classified two-dimensional (2D) pixel arrangement and the anatomical object classification.

Description

[0001] Priority requirements

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 767,351, filed November 14, 2018, which is incorporated herein by reference in its entirety. Background Technology

[0003] During minimally invasive surgical procedures or diagnostic procedures, the surgical scene can be viewed via camera. The procedure requires accurate identification of anatomical tissue types. Convolutional neural networks (CNNs) have been used to perform per-pixel segmentation of image pixel data. Classification can be more challenging due to the variations in appearance within the same tissue type and the similarities in appearance between different tissue types. Attached Figure Description

[0004] Figure 1 This is an illustrative block diagram representing an example convolutional neural network (CNN).

[0005] Figure 2 This is an illustrative function block diagram representing the image processing information flow within an example region suggestion CNN (R-CNN) configured to perform pixel-level image segmentation.

[0006] Figure 3 This is an illustrative drawing representing patient P, who is positioned face up on the operating table.

[0007] Figures 4A-4D It is an illustrative drawing showing a device display screen showing unclassified images of a first and second anatomical object viewed in four different example poses.

[0008] Figures 4E-4H It shows the display. Figures 4A-4D An illustrative drawing of the device display screen showing the classification images of the first and second anatomical objects viewed in the same four different example poses.

[0009] Figures 5A-5D It is an illustrative drawing showing four different patient positions in which a camera can observe the anatomical object during minimally invasive surgery or diagnostic procedures.

[0010] Figure 6 This is an illustrative drawing of an operating room table on which a patient can be placed. Dashed lines indicate different possible table tilt positions.

[0011] Figures 7A-7D This is an illustrative block diagram representing individual CNNs (CNN1-CNN4) trained to classify images based on corresponding ranges of image pixel data for the pose of the anatomical object relative to the camera.

[0012] Figure 8 This is an illustrative function block diagram representing the segmentation of object images provided to a CNN as input images. The CNN is trained to segment object images based on the input images provided to the CNN. Figures 7A-7D Image pixel data of one of the anatomical object poses is used to classify images.

[0013] Figure 9 This is an illustrative block diagram representing a first pixel segmentation system according to some embodiments.

[0014] Figure 10 This is an illustrative block diagram representing a second pixel segmentation system according to some embodiments.

[0015] Figure 11 This is an illustrative block diagram representing a third pixel segmentation system according to some embodiments.

[0016] Figure 12 This is an illustrative block diagram representing a fourth pixel segmentation system according to some embodiments.

[0017] Figure 13 This is an illustrative drawing representing a fifth pixel segmentation system according to some embodiments.

[0018] Figure 14 These are illustrative block diagrams of a computer system according to some embodiments. Detailed Implementation

[0019] A. Convolutional Neural Networks

[0020] Figure 1 This is an illustrative block diagram representing an example region proposal convolutional neural network (R-CNN) 100. A computer system can be configured with executable instructions stored in a computer-readable storage medium to implement CNN 100. CNNs have been used for image classification, which typically involves taking an input image and outputting a category (e.g., car, airplane, boat, bicycle, etc.) or a probability that best describes the category of the object shown in the input image. CNN 100 receives a two-dimensional pixel value array (PX) 102 as input and provides a set of classification scores as output. CNN 100 includes multiple processing layers. Specifically, CNN 100 includes convolutional (C) layers 104, pooling (P) layers 108, and fully connected (FC) network layers 110 combined with non-linear activation function (A) layers (e.g., ReLU layers) 106, and classifier layers (CL) 103 (such as a softmax classifier). Each layer is associated with array values, called weights. During the training of CNN 100, training data is provided as input to CNN 100 in the form of various training data input images, so as to train CNN 100 to accurately classify object images by adjusting the weights in the CNN layers.

[0021] More specifically, CNN 100 performs image classification by finding low-level image features such as edges and curves, and then constructs more complex combinations of image features through a series of convolutional layers. CNN 100 includes multiple convolutional layers in a sequence for constructing complex image feature maps from the input image. Each convolutional layer 104 identifies certain image features; different convolutional layers can identify different image features. Each convolutional layer applies a filter corresponding to the image feature that the layer is to identify. The filter can be configured with a two-dimensional array of numbers, commonly called weights, applied to a two-dimensional array of input values ​​input to the convolutional layer to produce an array of output values ​​from that layer. After the convolutional layers is typically a non-linear (or activation) layer 106, typically a rectified linear unit (ReLU), but other functions (e.g., tanh, sigmoid) can also be used. At each convolutional layer 104, the filter, including the array of weights, slides or convolves around the array of input values ​​to align the filter weights with the input values ​​at each of multiple consecutive locations in the array of input values. At each location, a single weight of the filter is multiplied by a single input value aligned with that weight at that location, and the resulting products are summed to produce the output value corresponding to that location. The output value is provided at a location within activation layer 106 that corresponds to the location of the array of input values ​​used to generate it.

[0022] Activation layers 106 that receive output values ​​from preceding convolutional layers 104 can serve as input layers to subsequent convolutional layers 104. For example, the input image 102 provided to the initial convolutional layer 104 of the CNN 100 typically comprises a two-dimensional array of pixel values ​​representing a visual image captured by a camera. Each pixel value in the two-dimensional array can include RGB color information. Therefore, separate RGB color information is provided for each pixel in the two-dimensional array. The sequence of convolutional layers 104 performs a convolution sequence, with each convolutional layer recognizing different low-level image features, such as edges and curves. Successive convolutional layers 104 and activation layers 106 result in combinations of image features that together represent higher-level image features, such as a semicircle (a combination of curves and straight edges) or a square (a combination of several straight edges). Generally, deeper convolutional layers 104 within the CNN 100 have a larger receptive field, meaning they can consider information from a larger area of ​​the original input volume. In other words, deeper convolutional layers may be more sensitive to larger regions of pixel space.

[0023] Activation layers 106 can be spread among convolutional layers 104 to provide nonlinearity and maintain size to control overfitting. More specifically, a nonlinear activation function layer can be included after each convolutional layer to add nonlinearity. For example, the nonlinear activation function layer can include tanh, sigmoid, or ReLU functions. Pooling layers 108 can be included after some ReLU activation function layers 106 to perform a pooling function that can include downsampling multiple values ​​in the ReLU layers. The reason behind pooling layers 108 is that once it is known that a particular image feature exists in the input image, its exact location is less important than its relative position to other image features. Max pooling is a well-known example of a pooling technique that downsamples by selecting the largest ReLU value in each pooling filter (e.g., a 2x2 filter) as the filter slides through an array of ReLU values. Pooling layers 108 can significantly reduce the spatial dimension of the input activation layers 106, which can reduce the number of weights required for convolution, thereby reducing computational cost. Pooling layer 108 can also control overfitting, which occurs when the model is tuned so much to the training examples that it cannot generalize well to the validation and test sets.

[0024] One or more fully connected network (FCN) layers 110 may be included near the end of CNN 100 to perform classification. One or more FC layers 110 may receive an array of input values ​​from preceding convolutional layers 104, activation layers 106, or pooling layers 108 and output an N-dimensional vector, where N is the number of classes that CNN 100 must select from. Each number in the N-dimensional vector can represent the probability of a particular class. Essentially, one or more FC layers 110 determine the probability that the high-level image features represented by the received array of input values ​​are associated with that class for each class. See A. Garcia-Garcia et al., A Review on Deep Learning Techniques Applied to Semantic Segmentation, April 22, 2017, arXiv:1704.06857 [cs.CV].

[0025] B. Object recognition using R-CNN

[0026] Figure 2This is an illustrative functional block diagram representing the image processing information flow within an example region proposal CNN (R-CNN) 200 configured to perform object detection. A computer system may be configured with executable instructions stored in a computer-readable storage medium to implement the CNN 200. For simplicity, details of the various scattering layers, which may include convolutional layers, activation layers, pooling layers, and fully connected network (FCN) layers, such as those described above, are omitted. The CNN 200 receives a two-dimensional pixel value array 202 as input and provides an image feature map 210 as output. The Region Proposal Network (RPN) 212 defines Region of Interest (ROI) proposals (indicated within dashed lines) 213. For example, based on the image feature map 210, the ROI proposals may include corresponding first, second, and third bounding boxes 221, 222, and 223 corresponding to first, second, and third ROIs 231, 232, and 233 within the pixel image 202. The image feature map 210 and the bounding box information 221, 222, and 223 may be provided to the pooling layer 214. The first object classifier 241 generates a first classification score C corresponding to the feature information associated with the first bounding box 221. O1 The second object classifier 242 generates a second classification score C corresponding to the image feature information associated with the second bounding box 222. O2 The third object classifier 243 generates a third classification score C corresponding to the image feature information associated with the third bounding box 223. O3 Therefore, the multiple different object images 231, 232, and 233 within the input image can be classified into C respectively. O1 C O2 C O3 Furthermore, the pixels of the input image 202 corresponding to the classification object can be spatially segmented based on the classification of regions that define boundaries for the classification object. It should be understood that, according to embodiments herein, the region of interest comprises individual pixels rather than a bounding box containing multiple pixels.

[0027] Therefore, in response to receiving the input image 202, the R-CNN 200 proposes separate regions of interest 221, 222, 223 for displaying individual objects in the image 202. In some embodiments, image feature information within each proposed region is individually provided to object classifiers 241, 242, 243 to determine its classification individually or a set of possible classifications. In some embodiments, the respective object classifiers 241, 242, 243 extract separate image feature vector information for each individual corresponding bounding box region 221, 222, 223 and provide the image feature vector information to a classifier, such as a set of support vector machines (SVMs) trained for each category, for output classification. Thus, individual objects within individually proposed regions can be classified individually. See also: S. Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:l506.01497v3[csCV], January 6, 2016; V. Badrinalanan et al., SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, arXiv:1511oo561 v3[csCV], October 10, 2016; K. He et al., Mask R-CNN, arXiv:1.703.06870v2[csCV], April 5, 2017.

[0028] C. Align the anatomical atlas with the patient's anatomy to determine the location of anatomical objects within the anatomy.

[0029] Figure 3 This is an illustrative drawing showing a patient 302 positioned face-up on an operating table 304, aligned with an anatomical atlas 306 representing internal organs. The anatomical atlas 306 is shown aligned with the patient's anatomy to provide indications of tissue types, such as lungs (L), heart (H), stomach (S), kidneys (K), and intestines (I) within the patient's body 302. It should be understood that the anatomical atlas 306 may include a generalized three-dimensional model of relevant portions of the human anatomy stored in a computer-readable storage device, which can be virtually aligned with the patient's anatomy to provide indications of tissue types desired within the field of view of a camera 308 (such as camera fields of view from different viewing angles).

[0030] D. Patient position and camera position

[0031] As used in this article, the term "pose" refers to the position of an object in a six-degree-of-freedom transformation (3 translations and 3 rotations). The pose of an anatomical object (such as anatomical tissue) can refer to the object's position in a six-degree-of-freedom transformation. The pose of a camera can refer to the camera's position in a six-degree-of-freedom transformation.

[0032] Anatomical objects being examined during surgical or diagnostic procedures can be difficult to distinguish from other anatomical objects displayed in camera images. Differentiating tissue types in a surgical setting can sometimes be challenging. In some cases, certain anatomical objects and tissue types may have similar appearances. Different tissue types can often be obscured by a layer of fat during surgical or diagnostic procedures. Furthermore, the appearance of tissue types can change as the surgical or diagnostic procedure progresses (due to blood, cauterization, manipulation, etc.). During minimally invasive procedures, the patient's awareness of their position relative to the camera can be used to improve tissue type identification by limiting the selection of possible tissue types that may be present in the camera images.

[0033] Anatomical objects visible within the field of view of an endoscopic camera during minimally invasive surgical or diagnostic procedures (also known as minimally invasive surgical procedures (MIS procedures) or laparoscopic procedures) can include, for example, one or more of a variety of different tissue types, such as blood vessels, adipose tissue, nerve fibers, or organ tissues (such as the lungs, kidneys, and liver). The relative posture between the camera and the patient's anatomy can be a factor in the ability to distinguish similar images of different anatomical objects, as camera images of anatomical objects (such as organ tissues) can vary significantly depending on the camera posture relative to the anatomical object posture.

[0034] Figures 4A-4D This is an illustrative drawing of a device display screen 1402 showing unclassified images of a first anatomical object 404 and a second anatomical object 406 viewed in four different example postures. The display screen 1402 may be a computer display screen or other display devices for displaying camera images of anatomical scenes, such as camera images captured by an endoscopic camera during minimally invasive surgery or diagnostic procedures, such as those from within a patient's anatomy. For example, the first anatomical object 404 and the second anatomical object 406 may be the spleen and liver, respectively. Figure 4A It is an illustrative drawing showing a first anatomical object 404 and a second anatomical object 406 in a first position, which may involve a patient positioned face up on an operating table. Figure 4B It is an illustrative drawing showing a first anatomical object 404 and a second anatomical object 406 in a second position, which may involve a patient positioned face down on an operating table. Figure 4C It is an illustrative drawing showing a first anatomical object 404 and a second anatomical object 406 in a third position, which may involve a patient located on the operating table with his or her right side. Figure 4D This is an illustrative drawing showing a first anatomical subject 404 and a second anatomical subject 406 in a fourth position, which may involve a patient positioned on an operating table with his or her left side. From Figures 4A-4D As can be understood from the diagram, two anatomical objects 404 and 406 may look similar, and visual examination of anatomical objects 404 and 406 in their original appearance may be difficult to distinguish without more information. For example, the challenge of distinguishing two anatomical objects may be increased during medical procedures due to blood, smoke from burns, fat, or tissue deformation.

[0035] Figures 4E-4H This is an illustrative drawing showing the device display screen 1402, which displays... Figures 4A-4D The classification images of the first anatomical object 404 and the second anatomical object 406 are shown in the same four different example poses. Figures 4E-4H In a display, the shadows of the first object 404 and the second object 406 indicate their different classifications. For example, pixels within a portion of display 1402 displaying the first object (spleen) 404 can be colored blue, while pixels within a portion of display 1402 displaying the second object (liver) 406 can be colored yellow. Without such shadows, the boundaries between tissue types may be difficult to discern with the naked eye. Therefore, shadows help medical personnel distinguish different tissue types. To accurately classify the images within display 1402, the pixels used to illuminate the images must be classified individually. As explained more fully below, a CNN can be used to classify individual pixels based on the type of anatomical object. Pixel classification can then be displayed on display 1402 as shadows or colors on the images of classified anatomical objects displayed on display 1402. Therefore, as described below, pixel-level classification can be displayed on display 1402 (e.g., as shadows or colors).

[0036] Figures 5A-5D It is an illustrative drawing showing four different example patient postures during a MIS procedure. Figures 5A-5D The illustration shows different possible camera poses and their corresponding camera fields of view. Figure 5A This is an illustrative drawing showing patient 502 in the first patient position, corresponding to the face-up position in the above diagram. Figure 5B This is an illustrative drawing showing patient 502 in the second patient position, corresponding to the face-down position in the diagram above. Figure 5C This is an illustrative drawing showing patient 502 in the third patient position, corresponding to the right position in the diagram above. Figure 5D This is an illustrative drawing showing patient 502 in the fourth patient position, which corresponds to the left position in the diagram above. Figures 5A-5DCamera 504 and camera reference frame 506 are shown in different alternative example camera poses relative to patient 502 in different patient poses.

[0037] The position of the operating table 602 can be a factor in determining the posture of the anatomical object. Figure 6 This is an illustrative drawing showing an operating room table where patients can be arranged. Dashed line 602 indicates different possible table tilt positions. Although not shown, the table height can also be adjusted.

[0038] E. Camera Change

[0039] The camera pose can be determined by assuming the camera is located at (0,0,0), with +Y upward and looking down along the -Z axis. The camera's focal length and image center define its projection transformation. The model-view transformation of the anatomical object can be encoded as a translation plus a quaternion in the camera reference frame. As used herein, the term "camera reference frame" refers to the coordinate system associated with the camera. For example, in some embodiments, the camera reference frame is centered between the two eyes, with +X to the left, +Y upward, and +Z outside the camera. See A. Kendall et al., PoseNet: A convolutional network for Real-Time 6-DOF Camera Relocalization, IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2938-2946.

[0040] According to some embodiments, the pose of the anatomical object relative to the camera reference frame can be determined based on a combination of the pose of the (operating room) table relative to the camera reference frame, the pose of the patient relative to the table reference frame, and the pose of the anatomical object relative to the patient reference frame.

[0041] More specifically, the pose of the stage relative to the camera reference frame can be defined as a 4x4 homogeneous transformation.

[0042] The patient's posture relative to the table reference frame can be defined as a 4x4 homogeneous transformation.

[0043] The posture of the anatomical object relative to the patient reference frame can be defined as a 4x4 homogeneous transformation.

[0044] The pose of the anatomical object relative to the camera reference frame can be defined as a composite camera transformation chain.

[0045] More specifically, for example, the transformation of the table relative to the camera reference frame can be derived from the reference marks on the table base plus knowledge of the table tilt and the Trend-Lomber position, or inferred from knowledge of the position of the instrument cannula sensed by the robot manipulator (see Methods and devices for table pose tracking using fiducial markers, WO2015142802A1, and System and method for integrated surgicaltable, WO2016069648 A1).

[0046] More specifically, for example, the patient's transformation relative to the table can be estimated, for example, by a CNN, using an external view (e.g., an RGB and / or depth camera mounted on a surgical trolley in the operating room), or specified by the operator at the start of the procedure.

[0047] More specifically, for example, the transformation of an anatomical object (such as internal tissue structures) relative to the patient reference frame can be estimated based on data from an anatomical atlas aligned with the patient's body (a model of the anatomical object's position relative to a common reference frame), an estimate of the patient's size used to scale the anatomical atlas (e.g., generated by the CNN above), and an estimate of deformations caused by gravity, inflation, etc. (if important).

[0048] F. Train different CNNs to segment anatomical objects for different poses of the anatomical objects relative to the camera reference frame.

[0049] Figures 7A-7D This is an illustrative block diagram representing individual CNNs (CNN1-CNN4) trained to classify images of anatomical objects based on varying ranges of image pixel data relative to the pose of the anatomical object relative to a camera reference frame. Refer to the discussion below. Figure 14 The computer system 1400 can be configured with executable instructions 1424 stored in storage media 1404, 1406, 1416 to implement CNNs (CNN1-CNN4). For example, a camera looking down at the pelvis (not shown) is a different anatomical object pose than a camera looking up at the chest. CNNs can be trained using images of anatomical objects that vary naturally for different patients. Furthermore, for example, CNNs can be trained using images of anatomical objects at different points in a MIS procedure, images of anatomical objects being manipulated surgically, and images of anatomical objects exhibiting burns, smoke, and / or blood. CNNs can also be trained using images enhanced by adjusting colors (e.g., hue, saturation) and / or by image rotation and / or cropping.

[0050] Each CNN1-CNN4 receives pixel information arranged in a two-dimensional (2D) array of pixel data, which includes each pixel of an image captured by a camera, potentially including RGB, depth / distance, and / or additional color channels such as infrared (IR) wavelengths. Each CNN1-CNN4 outputs a set of classification scores C1-C4. N Each CNN1-CNN4 includes convolutional and activation layers 602A-602D to generate image feature maps and region of interest (ROI) proposals. Each CNN1-CNN4 includes RPNs 603A-603D to define the ROI proposals. Each CNN1-CNN4 also includes one or more classification layers 604A-604D, which may include one or more FCNs to generate image classification scores for classifying images within the ROI proposals. The image feature maps and classifications generated by CNN1-CNN4 in response to the input image are based on the corresponding feature weight sets [W]. F1 ]-[W F4 These feature weight sets can be stored in the storage medium, and the corresponding classification weight sets [WC1]-[WC4] can be stored in the storage medium.

[0051] Each CNN1-CNN4 is trained to segment one or more tissue types. More specifically, each distinct CNN1-CNN4 can be trained to segment camera images of one or more tissue types for different viewpoints. Each different viewpoint corresponds to a different pose of one or more specific tissue types relative to the camera reference frame. Some tissue types may appear in camera images in multiple poses relative to the camera reference frame. Furthermore, each corresponding CNN1-CNN4 can be trained to identify one or more corresponding tissue types using training data images under various different conditions, such as during surgical manipulation, during smoke and / or blood leakage. For example, during the execution of an MIS procedure, the pose of the anatomical object relative to the camera reference frame can be used to determine a set of tissue types expected to be within the camera's field of view, and one or more CNN1-CNN4s can be selected to perform classification to identify one or more tissue types within the camera's field of view. More specifically, for example, the anatomical model can be converted to at least roughly match the patient's pose and size. The model can then be queried for a list of expected tissue types within the camera's field of view (i.e., using a virtual camera, using the same viewpoint viewed when the endoscope is examining the patient—this can provide a set of expected tissue types that can be used as the basis for selecting the CNN).

[0052] Training data from different poses can be used to train different CNN1-CNN4 to distinguish different tissue types (such as the tissue types in the figure below).

[0053] CNN / Organization Classification Chart

[0054] CNN# Classified organization types CNN1 Lungs, heart, liver CNN2 Spleen, kidney, gallbladder CNN3 Stomach, lungs, liver CNN4 Stomach, omentum, fat, small / large intestine

[0055] The first training dataset was used to train the first CNN1. The first training dataset included multiple different training images showing a first set of tissue types (such as lung, heart, and liver tissue), for example, within a first pose range of one or more anatomical structures relative to a camera reference frame (i.e., the relative pose between the anatomical body and the camera, such as...). Figure 5A The images shown are multiple different training images of a first group of tissue types captured by a camera under various conditions (e.g., surgical manipulation, smoke, blood). Based on the first training dataset, the configuration of convolutional and activation layers 602A and RPN 603A, and the classification weights [W] of a classification layer 604A configured as CNN1, are determined. C1 Feature weights [W] F1 The second training dataset is used to train the second CNN2. The second training dataset includes multiple different training images of a second set of tissue types (such as spleen, kidney, and gallbladder), for example, within a second pose range of one or more anatomical structures relative to a camera reference frame (such as...). Figure 5B The images shown are multiple different training images of a second group of tissue types captured by a camera at a patient (facing down) under various different conditions. Based on the second training dataset, the configuration of convolutional and activation layers 602B and RPN 603B, and the classification weights of a classification layer 604B configured as CNN2, were determined. C2 Feature weights [W] F2 The third training dataset is used to train the third CNN3. The third training dataset includes multiple different training images of a third group of tissue types (such as stomach, lung, and cervical vertebrae), for example, within a third pose range of one or more anatomical structures relative to a camera reference frame (such as...). Figure 5C The images shown are of the patient lying on the right side, and multiple different training images of the third group of tissue types captured by a camera under various different conditions. Based on the third training dataset, the configuration of convolutional and activation layers 602C and RPN 603C, and the classification weights [W] of the classification layer 604C configured with CNN3, were determined. C3 The feature weights [WF3]. A fourth training dataset is used to train the fourth CNN4. The fourth training dataset includes a fourth pose range (e.g., for one or more anatomical structures relative to a camera reference frame) within a fourth pose range. Figure 5DThe images shown are multiple different training images captured by a camera under various conditions, depicting a fourth group of tissue types (such as stomach, omentum, fat, and small / large intestine). Based on this fourth training dataset, classification weights [W] were determined for the configuration of convolutional and activation layers 602D and RPN 603D, and a classification layer 604D configured with CNN4. C4 Feature weights [W] F4 ].

[0056] Figure 8 This is an illustrative functional block diagram illustrating the use of a first CNN1 to segment an object image. Pixel RGB color information 701 can be provided as input to the first CNN1. Additionally, or alternatively, color channel information such as infrared (IR) wavelengths can be provided as input to the first CNN1. Depth / distance information can also be provided as input to the first CNN1. For example, suppose pixel image data 701 includes first object image data 702 representing lung tissue adjacent to second object image data 704 representing heart tissue. The convolutional and activation layers 602A of the first CNN1 generate feature maps 720; the RPN segment of the first CNN1 generates a first region of interest proposal 722 corresponding to the first object image data 702 and a second region of interest proposal 724 corresponding to the second object image data 704. The classification segment 604A of the first CNN1 classifies the content of the first region of interest proposal 722 to generate a first classification score 742 corresponding to the first object image data 702. The classification segment 604A of the first CNN1 classifies the content of the second region of interest proposal 724 to generate a second classification score 744 corresponding to the second object image data 704.

[0057] The classification information determined by the first CNN1 can be used, for example, as added to the user interface (UI) display showing the pixel image data 701, the classification information (such as the label 'lung' adjacent to the first object image data 702), and as the basis for adding the label 'heart' near the second object image data 704. Alternatively, for example, the determined classification information can be used as the basis for aligning a segmented preoperative image of the lung (not shown) with the first object image data 702 and a segmented preoperative image of the heart (not shown) with the second object image data 704. In the case where the preoperative image data is aligned with the object image data 702 and / or 704, the classification result can be used as the basis for improving the temporal procedure segmentation, although it is not directly displayed or reported to the user.

[0058] G. Pose-based pre-filtering to select CNNs trained to segment specific tissue types

[0059] Figure 9This is an illustrative block diagram representing a first pixel segmentation system 902 according to some embodiments. A camera 504 is configured to capture camera images comprising a two-dimensional array of camera pixel data 904 representing an anatomical object of a patient 502 lying on an operating table 602, for storage in a storage medium 906. For example, the pixel data 904 may include motion picture data such as video data. The first pixel segmentation system 902 includes multiple CNNs, namely CNN1-CNN4. As described above, reference... Figures 7A-7D and Figure 8 Different CNNs in CNN1-CNN4 can be trained and configured to classify tissue types expected to appear in the camera's field of view under different poses of the patient's anatomy relative to the camera reference frame 506. Computer systems (such as the following reference...) Figure 14 The computer system 1400 described may be configured with executable instructions (such as instruction 1424) for implementing a pre-filtered CNN selection block 908 to select one or more of CNN1-CNN4 to classify a two-dimensional arrangement of camera pixel data.

[0060] The pre-filter CNN selection block 908 receives stage pose information, patient pose information, anatomical atlas information, and camera pose information. Based on the aforementioned camera transformations, the pre-filter CNN selection block 908 determines the pose of the patient anatomy 502 relative to the camera. The pose of the patient anatomy relative to the camera 504 is used to determine which tissue types are expected to be within the camera's field of view, and therefore, the tissue types expected to be represented by the captured pixel data 904. The pre-filter CNN selection block 908 selects one or more of CNN1-CNN4 that best suit the pixel segmentation for the expected tissue types.

[0061] One or more CNN1-CNN4 CNNs are selected and trained to segment pixel data based on tissue type. More specifically, one or more CNN1-CNN4 CNNs are selected and trained to classify individual pixels represented within the pixel data according to tissue type. See reference... Figures 4A-4GThe explained pixel classification based on tissue type can be used to display different tissue types appearing on the user interface display screen 1402 in different colors. Alternatively, for example, pixel classification can be used to label different tissue types appearing on the display screen 1402 as, for example, representing different tissue types. For example, a first CNN1 can be trained and configured to segment pixel data into pixel data corresponding to lung, heart, and liver tissues; a second CNN2 can be trained and configured to segment pixel data into pixel data corresponding to spleen, kidney, and gallbladder tissues; a third CNN3 can be trained and configured to segment pixel data into pixel data corresponding to stomach, lung, and liver tissues; and a fourth CNN4 can be trained and configured to segment pixel data into pixel data corresponding to stomach, omentum, fat, and small / large intestine tissues.

[0062] H. Pose-based post-filtering for identification by pixel classification using a CNN trained to segment tissue types. Invalid classification

[0063] Figure 10 This is an illustrative block diagram representing a second pixel segmentation system 1002 according to some embodiments. A camera 504 is configured to capture camera images comprising a two-dimensional array of camera pixel data 1004 representing an anatomical object of a patient 502 lying on an operating table 602, for storage in a storage medium 1006. For example, the pixel data 1004 may include motion picture data such as video data. The second pixel segmentation system 1002 includes a CNN 1003 comprising convolutional and activation layers, RPN and FCN layers (not shown), which, as described above, can be trained to segment a wider range of different tissue types than a single CNN (e.g., CNN1-CNN4 of the first pixel segmentation system). A computer system 1400 may be configured with executable instructions 1424 stored in storage media 1404, 1406, 1416 for implementing the CNN 1003 and a post-filtering classification correction logic block 1008 to identify invalid classifications based on the pose of the patient's anatomy relative to the camera. CNN 1003 provides suggested pixel classification data 1005 to post-filtering classification correction block 1008, which filters the suggested pixel classification data 1005 based on the pose of the anatomical object in the patient 502's anatomy relative to the camera reference frame 506 to identify pixel classification errors in the suggested pixel classification data 1005.

[0064] More specifically, in some embodiments, CNN 1003 is trained and configured to output a proposed classification for each pixel in a 2D permutation of pixel values ​​captured by camera 504. The proposed pixel classification values ​​may include probability values ​​for each of several possible different classifications of the pixel (e.g., liver, spleen, heart, lung, retina, nerve, fat, blood vessel, etc.), and post-filter 1008 may modify the pixel classification probability values ​​of one or more pixels based on pose information. For example, the probability of seeing lungs is low when the camera is positioned looking down at the pelvis. Therefore, for example, when pose information indicates that the camera is looking down at the pelvis, the post-filter may reduce the high probability CNN output classification probability value indicating that the object in the camera's field of view is a lung. For each pixel, the highest error-corrected classification probability value obtained after post-filtering by post-filter 1008 is selected as the final pixel classification. In some embodiments, low classification values ​​may be filtered out based on a threshold. For example, classification values ​​that do not meet the threshold may be ignored or treated as unknown.

[0065] According to some embodiments, the post-filtering classification correction logic block 1008 receives table posture information, patient posture information, anatomical atlas information, and camera posture information. The post-filtering classification correction block 1008 determines the posture of one or more anatomical objects of the patient 502 relative to the camera 504 based on the aforementioned camera transformation. The posture of the patient's anatomical objects relative to the camera determines the expected tissue type within the camera's field of view, and thus the different tissue types represented by different pixels in the 2D arrangement of the received pixel data 1004. The post-filtering classification correction block 1008 filters pixel classifications to identify and correct erroneous pixel classifications based on the patient's anatomical object's posture relative to the camera as described above.

[0066] Pose-based fully connected layers in a CNN trained to segment tissue types

[0067] Figure 11This is an illustrative block diagram representing a third pixel segmentation system 1102 according to some embodiments. A camera 504 is configured to capture camera images comprising a two-dimensional array of camera pixel data 1104 representing an anatomical object of a patient 502 lying on an operating table 602, for storage in a storage medium 1106. For example, the pixel data 1104 may include motion picture data such as video data. The third pixel segmentation system 1102 includes a CNN 1103, which includes convolutional and activation layers 1105, an RPN 1107, and a fully connected layer (FCN) 1109 trained based on the anatomical object pose relative to a camera reference frame 506. A computer system 1400 may be configured with executable instructions 1424 stored in storage media 1404, 1406, 1416 for implementing the CNN 1103. The CNN 1103 is trained and configured to classify tissues across various anatomical object poses relative to the camera reference frame 504. Therefore, CNN 1103 can be trained to distinguish a wider range of different tissue types than individual CNNs (e.g., CNN1-CNN4 of the first pixel segmentation system 902). FCN layer 1109 receives information indicating the relative pose between the anatomical object within patient 502 and camera reference frame 506 as input (encoded as quaternions + translation), and also receives camera calibration parameters (e.g., one or more focal lengths, image center) as input. FCN layer 1109 assigns feature information indicating the anatomical object pose relative to the camera pose to the feature map produced by the convolutional and activation layers 1105. CNN 1103 receives a 2D arrangement of camera pixel data 1104, information indicating the relative pose between the anatomical object within patient 502 and camera reference frame 506 (encoded as quaternions + translation), and camera calibration parameters (e.g., focal length, image center) as input, and produces a two-dimensional arrangement of pixel-level classification data. It should be understood that, essentially, the training of CNN 1103 incorporates anatomical atlases into CNN 1103, where the training of CNN 1103 includes fully connected layers 1109 containing camera models. In other words, the third-pixel segmentation system 1102 learns 3D models of the overall anatomy and the anatomical objects within it. Therefore, anatomical atlas information may not be required as input.

[0068] J. Preoperative model alignment provides cues to a CNN trained to segment tissue types, utilizing model alignment to... Drive pixel-level segmentation

[0069] Figure 12This is an illustrative block diagram representing a fourth pixel segmentation system 1202 according to some embodiments. A camera 504 is configured to capture camera images comprising a two-dimensional arrangement of camera pixel data 1203 representing an anatomical object of a patient 502 lying on an operating table 602, for storage in a storage medium 1205. For example, the pixel data 1203 may include motion picture data such as video data. A computer system 1400 may be configured with executable instructions 1424 stored in storage media 1404, 1406, 1416 for implementing the fourth pixel segmentation system 1202, which includes a coarse alignment logic block 1204, a fine alignment logic block 1206, a 3D-to-2D renderer block 1208, and a CNN 1210. In some embodiments, the CNN 1210 includes convolutional and activation layers, RPN and FCN layers (not shown) trained as described above to segment tissue types. In some embodiments, the CNN 1210 is trained based on camera images captured from multiple different anatomical object poses relative to one or more camera reference frames and on two-dimensional (2D) renderings of a three-dimensional (3D) preoperative model of the anatomical object, to classify, for example, pixels in a 2D arrangement of pixel values ​​captured by the camera during a MIS or laparoscopic procedure according to the anatomical object type. Thus, the CNN 1210 is trained to classify individual pixels within a 2D arrangement of pixel values ​​according to object type, and is based at least in part on cues provided by the two-dimensional preoperative model of the tissue structure as the basis for pixel-level segmentation classification. In some embodiments, during training of the CNN 1210, labeled 2D renderings of the preoperative model and corresponding labeled endoscopic images including 2D arrangements of pixel values ​​of the patient's anatomy are provided as training inputs. Thus, the CNN 1210 is trained to predict the classification of pixels within a captured 2D arrangement of pixels representing an image of the patient's anatomy using the 2D preoperative model rendering as cues.

[0070] In operation, a preoperative model 1212 stored in storage medium 1214 can be obtained, which includes a three-dimensional (3D) image scan data model comprising a three-dimensional image representation of relevant portions of the patient's anatomy, including scan data models of individual anatomical tissue objects. For example, an image scanning system (not shown) can generate the preoperative model 1212 prior to a diagnostic or surgical procedure. Preoperative three-dimensional (3D) image scan data can be generated using scanning systems such as computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound technologies, indicating physical tissue constituencies at discrete three-dimensional volume locations within anatomical tissue structures. Alternatively, according to some embodiments, the preoperative model 1212 may include a generalized anatomical atlas (not shown). The physical anatomical tissue structures (not shown) within the patient 502 to be scanned may include one or more anatomical objects, such as kidney tissue, liver tissue, blood vessels, bones, and nerve fibers. The three-dimensional locations within the preoperative image scan data model 1212 correspond to three-dimensional locations within the physical anatomical tissue structures of the patient 502. The smallest 3D location unit can be referred to as a voxel. A 3D preoperative image scan data model 1212 may include individual voxels corresponding to individual three-dimensional locations within a physical tissue structure. Pixels in a camera image may correspond to 2D slices of voxels within the 3D preoperative model. Individual voxel values ​​(grayscale or color values) can represent the density of the physical tissue at the corresponding location within the physical tissue structure, and thus can serve as a basis for cues regarding segmentation of different tissue types represented in a captured camera image of a patient's anatomy in a 2D arrangement of pixels. Different types of tissue, such as bone and blood vessels, have different densities, and therefore the voxel values ​​at 3D locations within the image scan data can indicate the tissue type at the corresponding 3D location within the physical tissue structure. Preoperative image scan data can be converted into a preoperative 3D model 1212 of the patient's anatomy and stored as a mesh (e.g., a set of triangles for computer graphics, etc.). See W. Lorenesn et al., Marching Cubes: A High Resolution 3D Surface Construction Algorithm, July 1987, Computer Graphics, Vol. 21, No. 4. For example, different tissue types can be represented by different shades of gray or different colors. Furthermore, different tissue types can be labeled to indicate, for example, the tissue type; grayscale or color itself can serve as a label.

[0071] In operation, coarse alignment logic block 1204 receives information indicating the pose of the anatomical object relative to a camera reference frame as input. This information may include table pose information, patient pose information, and camera pose information. Coarse alignment logic block 1204 also receives a preoperative model 1212, which may include an anatomical atlas. Coarse alignment logic block 1204 aligns the preoperative model 1212 with the camera reference frame based on the pose of the anatomical object relative to the camera reference frame. For example, the pose of the anatomical object relative to the camera reference frame can be determined based on the aforementioned camera transformation. More specifically, coarse alignment logic block 1204 determines a view or orientation of the 3D model 1212 that substantially matches the anatomical view represented by a 2D arrangement of captured pixel values, and designates the matching view or orientation as the aligned preoperative model 1212. Coarse alignment block 1204 generates a 3D coarse alignment view 1216 of the preoperative model relative to camera reference frame 506, which can be stored in storage device 1218.

[0072] Fine alignment logic block 1206 receives a 3D coarse alignment preoperative model view 1216 and pixel depth data 1220 as input. In some embodiments, camera 504 is implemented as a stereo camera and pixel depth data 1220 includes stereo pixel depth data. Known techniques based on stereo images can be used to determine depth information for individual pixels of the captured 2D pixel data 1203, which can in turn be used to closely align individual pixels within the captured 2D pixel data 1203 with individual voxels (or vertices) of slices from different depths of the 3D preoperative model 1212. More specifically, according to some embodiments, fine alignment block 1206 receives a stereo 2D arrangement of pixel data 1203 captured by camera 504 and determines pixel depth information 1220 based on the distance between corresponding pixels in two stereo images. The fine alignment block 1206 can be configured to use an iterative nearest neighbor (ICP) process based on camera pixel depth data 1220 to finely align the surface geometry of the 3D preoperative model 1212 with the geometric features imaged within the captured 2D pixel data 1203. It is worth noting that fine alignment using ICP is generally more suitable for rigid pose changes than for tissue deformation changes. For example, fine alignment may be required due to displacement of internal anatomical objects caused by gravity or inflation. The fine alignment block 1206 produces a finely aligned 3D preoperative model view 1222, which is stored in a memory device 1224.

[0073] The 3D-to-2D rendering logic block 1208 receives a finely aligned 3D preoperative model view 1222 as input and produces a rendered 2D preoperative model 1226 as output. The rendered 2D preoperative model 1226 includes pixels corresponding to and aligned with the captured 2D pixel data 1203. The 2D preoperative model is stored in a memory device 1228.

[0074] CNN 1210 receives captured 2D pixel data 1203 and a rendered 2D preoperative model 1226 as input. Due to coarse and fine alignment stages, the pixels of the rendered 2D preoperative model 1226 are substantially aligned with the pixels of the captured 2D pixel data 1203. Labels provided for the pixels of the 3D preoperative model 1212 can be passed to the pixels of the rendered 2D preoperative model 1226 to generate a per-pixel label for the rendered 2D preoperative model 1226. CNN 1210 is trained to segment tissue types within the captured 2D pixel data 1203 on a per-pixel basis and also uses pixel data from the rendered 2D preoperative model 1226 as cues for performing per-pixel segmentation. CNN 1210 provides pixel-level classification 1211 as output, which indicates pixel-level segmentation of the anatomical object image that can be used to label individual pixels within the display screen 1402.

[0075] Understandably, in a stereo vision system, the perspectives of the L-eye and R-eye (left and right eyes) observers are slightly different. Therefore, we could choose to align and render an aligned preoperative model to match either view, and then run it through a CNN. Alternatively, we could choose to align the 3D preoperative model based on the pixel depth of the L and R views, render aligned preoperative models for both L and R views, and run both through a CNN sequentially. Or, alternatively, we could choose to align the 3D preoperative model based on the pixel depth of the L and R views, render aligned preoperative models for both L and R views, and run them through a CNN simultaneously by, for example, wrapping the L and R views side-by-side, top-bottom, or in a staggered manner.

[0076] K. Preoperative model alignment provides cues to a CNN trained to segment tissue types, utilizing pixel-level segmentation. Driving model alignment

[0077] Figure 13This is an illustrative drawing representing a fifth pixel segmentation system 1302 according to some embodiments. A camera 504 is configured to capture camera images comprising a two-dimensional array of camera pixel data 1303 representing an anatomical object of a patient 502 lying on an operating table 602, for storage in a storage medium 1305. For example, the pixel data 1303 may include motion picture data such as video data. A computer system 1400 may be configured with executable instructions 1424 stored in storage media 1404, 1406, 1416, and may be configured to implement the fifth pixel segmentation system 1302, which includes an alignment logic block 1304, a 3D-to-2D renderer 1308, an alignment error correction feedback block 1342, and a CNN 1310. In some embodiments, the CNN 1310 includes convolutional and activation layers, RPN and FCN layers (not shown) trained as described above to segment tissue types. Training data may include a 2D arrangement of labeled pixel data representing anatomical objects within a patient's anatomy, captured by an endoscopic camera, and a 2D rendering of the corresponding labeled 3D preoperative model of the anatomical object. Thus, the CNN 1310 is trained to use reference preoperative images as cues when performing per-pixel classification of 2D arranged pixel data representing anatomical structures captured during surgical or diagnostic procedures. The CNN 1310 is also configured to base its segmentation and classification, at least in part, on cues provided by the corresponding rendered 2D preoperative model of the tissue structure.

[0078] In operation, a preoperative model 1312 stored in storage medium 1314 can be obtained. The preoperative model 1312 includes a three-dimensional (3D) image scan data model, which includes three-dimensional image representations of relevant parts of the patient's anatomy, and the three-dimensional image representations of relevant parts of the patient's anatomical structures include scan data models of individual anatomical tissue objects. As described above, the preoperative model may include a three-dimensional (3D) image scan data model, which includes 3D image representations of relevant parts of the patient's anatomy, and the relevant parts of the patient's anatomical structures include anatomical tissue objects. Alternatively, the preoperative model may include a generalized anatomical atlas.

[0079] In operation, the coarse alignment logic block 1304 receives information indicating the pose of the anatomical object relative to a camera reference frame as input. This information may include table pose information, patient pose information, and camera pose information. The coarse alignment logic block 1304 also receives a preoperative model 1312, which may include an anatomical atlas. In some embodiments, the operation of the alignment block 1304 is similar to... Figure 12The coarse alignment block 1204. Specifically, the coarse alignment logic block 1304 aligns the preoperative model 1312 with the camera reference frame based on the pose of the anatomical object relative to the camera reference frame. For example, the pose of the anatomical object relative to the camera reference frame can be determined based on the camera transformation described above. More specifically, the coarse alignment logic block 1304 determines a view or orientation of the 3D model 1312 that substantially matches the view of the anatomical structure represented by a 2D arrangement of captured pixel values, and designates the matching view or orientation as the aligned preoperative model 1312. The coarse alignment block 1304 generates a 3D coarse alignment view of the preoperative model 1316 relative to the camera reference frame 506, which can be stored in the storage device 1318.

[0080] The 3D coarse alignment view of the preoperative model 1316 is provided as input to the 3D-to-2D rendering logic block 1308. The rendered 2D preoperative model pixel data 1326, aligned with the camera reference frame, is generated based on the alignment surface geometry of the 3D model 1316 and is stored in the storage device 1328 as the output of the 3D-to-2D rendering block 1308.

[0081] CNN 1310 receives a captured 2D arrangement of camera pixel data 1303 and a rendered 2D preoperative model 1326 as input. Due to the alignment phase including an alignment error correction feedback block 1342, the rendered 2D preoperative model pixel data 1326 is substantially aligned with the captured 2D arrangement of camera pixel data 1303. Per-voxel labels provided for the 3D preoperative model 1312 can be used to generate corresponding per-pixel labels for the rendered 2D preoperative model 1326. CNN 1310 is configured, based on training, to perform pixel-level segmentation of tissue types represented in the captured 2D arrangement of camera pixel data 1303, which may represent one or more anatomical objects, and is also configured to use the corresponding rendered 2D preoperative model 1326 as cues for pixel-level segmentation. CNN 1310 provides pixel-level classification 1311 as output, which can be used to label individual pixels within the display screen 1402.

[0082] Alignment error correction feedback logic block 1342 receives the rendered 2D preoperative model 1326 and pixel-level output classification 1311 as input. Alignment error correction feedback logic block 1342 generates pixel alignment error information 1344 as output. In some embodiments, this pixel alignment error information 1344 may include a pose correction transformation between the nominal anatomical object position (c') in the camera reference frame 506 as anticipated from the previously described camera transformation chain and the actual anatomical object position (c) observed by camera 504.

[0083] The pose correction transformation information can be determined by minimizing pixel-level classifications of mismatches / misalignments between the corresponding pixel labels of the rendered 2D preoperative model. For example, a mismatch / misalignment between a threshold number of pixel labels from a region of the rendered 2D preoperative model and the pixel classifications generated by the CNN might indicate misalignment. A sufficiently large threshold level can be selected so that system noise or minute pixel misalignments involving only a few pixels do not trigger the detection of alignment errors. Pixel alignment error information 1344 can provide the alignment logic block 1304 with an indication of the location of the 3D preoperative model 1302 that is misaligned with the pose of the actual anatomical object.

[0084] In some embodiments, alignment logic block 1304 receives pose correction transformation information provided by alignment error correction feedback block and links it to the previously described transformation chain as follows:

[0085]

[0086] To generate an updated alignment transformation.

[0087] L. Computer System Implementation

[0088] Figure 14 This is an illustrative block diagram of computer system 1400. Computer system 1400 or variations thereof can be configured to act as a system to implement the disclosed CNN and imaging system according to executable instructions 1424. In some embodiments, computer system 1400 operates as a stand-alone device or can be connected (e.g., networked) to other computers. For example, in a networked deployment, the computer system can operate as a server or client computer in a server-client network environment, or as a peer computer in a peer-to-peer (or distributed) network environment.

[0089] Example computer system 1400 includes one or more hardware processors 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), main memory 1404, and static memory 1406, which communicate with each other via bus 1408. Computer system 1400 may also include a video display unit 1410 (e.g., a plasma display, liquid crystal display (UCD), or cathode ray tube (CRT)). For example, display unit 1410 may be used to implement... Figures 4A-4GThe computer system 1400 also includes an alphanumeric input device 1412 (e.g., a keyboard), a user interface (UI) navigation device 1414 (e.g., a mouse, a touchscreen, or the like), an SSD or disk drive unit 1416, a signal generation device 1418 (e.g., a speaker), and a network interface device 1420.

[0090] DRAM, SSD, or disk drive units 1404, which can function as storage memory devices, include computer-readable storage devices 1404, 1406 on which one or more sets of instructions and data structures (e.g., software 1424) embodying or used by any one or more methods or functions described herein are stored. Software 1424 may also reside wholly or at least partially within computer-readable storage devices (such as main memory 1404 and / or processor 1402) during execution by computer system 1400, which also constitute non-transitory computer-readable media. As described above, one or more CNNs may be stored in DRAM, SSD, or disk drive units 1416 or on an external server. Software 1424 may also be sent or received on network 1420 via network interface device 1420 using any of a variety of well-known transport protocols (e.g., HTTP). The storage memory device 1416 can be configured to store feature weights and classification weights, as well as instructions executable by one or more processors, such as instructions that, when executed, cause one or more processors to implement a CNN.

[0091] Example

[0092] Example 1 includes an imaging system for a surgical or diagnostic medical procedure, comprising: a camera that captures an image of an anatomical object and represents the image in a two-dimensional (2D) pixel arrangement; one or more processors and a non-transitory computer-readable medium operatively coupled thereto, the non-transitory computer-readable medium including: CNN instructions executable by the one or more processors, which, when executed, cause the one or more processors to implement a CNN configured, according to an optional set of weights, to associate anatomical object classifications with the pixels in the 2D pixel arrangement; and a plurality of weight sets configured differently for training the CNN implemented by the one or more processors based on different camera image training data; and a display screen configured to display the 2D pixel arrangement of the classifications and the associated anatomical object classifications.

[0093] Example 2 includes the subject of Example 1, and further includes: CNN selection logic that selects a weight set from multiple weight sets based on the pose of the anatomical object relative to a camera reference frame to configure a CNN implemented by one or more processors.

[0094] Example 3 includes the subject of Example 1, wherein each weight set corresponds to a training dataset comprising camera images captured for pose ranges of one or more anatomical objects relative to one or more camera reference frames; and wherein each training dataset comprises camera images captured for different pose ranges of one or more anatomical objects relative to one or more camera reference frames, different from the other training datasets.

[0095] Example 4 includes the subject of Example 1, wherein a CNN implemented by one or more processors includes one or more convolutional and activation layers, a region proposal network layer, and one or more classification layers.

[0096] Example 5 includes the subject of Example 4, wherein each of the different weight sets includes a different set of feature weights for configuring convolutional and activation layers in a CNN implementation and a different set of classification weights for configuring one or more classification layers in a CNN implementation.

[0097] Example 6 includes the subject of Example 1, with the display showing a two-dimensional (2D) pixel arrangement of categories with associated anatomical object classifications.

[0098] Example 7 includes an imaging system for surgical or diagnostic medical procedures, comprising: a camera that captures an image of an anatomical object and represents the image in a 2D pixel arrangement; one or more processors and a non-transitory computer-readable medium operatively coupled thereto, the non-transitory computer-readable medium including instructions executable by the one or more processors, including: CNN instructions that, when executed, cause the one or more processors to implement a CNN configured to associate an anatomical object classification with pixels in the 2D pixel arrangement; CNN classification selection logic that adjusts one or more classification values ​​generated by the CNN for the 2D pixel arrangement based on the pose of the anatomical object relative to a reference frame of the camera; and a display screen configured to display the two-dimensional (2D) pixel arrangement of the classification and the associated anatomical object classification.

[0099] Example 8 includes the subject of Example 7, wherein a CNN implemented by one or more processors includes one or more convolutional and activation layers, a region proposal network layer, and one or more classification layers.

[0100] Example 9 includes the topic of Example 7, where the CNN classification selection logic adjusts one or more classifications based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

[0101] Example 10 includes the topic of Example 7, where the CNN classification selection logic is based on a composite camera transform chain. To adjust one or more categories; where The change in the orientation of the display platform relative to the camera reference frame; whereby... This represents the patient's transformation relative to the table reference frame; and where... This indicates the change in the posture of the anatomical object relative to the patient's reference frame.

[0102] Example 11 includes an imaging system for a surgical or diagnostic medical procedure, comprising: a camera that captures an image of an anatomical object and represents the image in a 2D pixel arrangement; one or more processors and a non-transitory computer-readable medium operatively coupled thereto, the non-transitory computer-readable medium including: CNN instructions executable by the one or more processors, which, when executed, cause the one or more processors to implement a CNN, wherein the CNN is configured to associate an anatomical object classification with pixels in a 2D pixel arrangement; wherein the CNN implemented by the one or more processors includes one or more convolutional and activation layers and fully connected layers, the one or more convolutional and activation layers being used to generate one or more image feature maps based on one or more 2D pixel arrangements, the fully connected layers being used to associate the anatomical object classification with pixels in one or more 2D pixel arrangements based on one or more feature maps, relative pose between patient anatomy, and camera calibration information; and a display screen configured to display the categorized two-dimensional (2D) pixel arrangement and the associated anatomical object classification.

[0103] Example 12 includes the subject of Example 11, wherein a CNN implemented by one or more processors includes a region proposal network layer that generates regions of interest based on feature maps.

[0104] Example 13 includes the subject of Example 11, wherein the CNN implemented by one or more processors includes: a region proposal network layer to identify one or more regions of interest within one or more feature maps; wherein a fully connected layer associates the anatomical object classification with one or more 2D pixel arrangements of pixels based on the feature maps within one or more regions of interest and the relative pose between the patient anatomy and the camera, as well as camera calibration information.

[0105] Example 14 includes an imaging system for a surgical or diagnostic medical procedure, comprising: a camera that captures images of an anatomical object and represents the images in a 2D pixel arrangement; one or more processors and a non-transient computer-readable medium operatively coupled thereto, the non-transient computer-readable medium including instructions executable by the one or more processors, including: coarse alignment instructions, which, upon execution, cause the one or more processors to implement a coarse alignment block configured to align voxels of a three-dimensional (3D) preoperative model with pixels of one or more two-dimensional (2D) pixel arrangements based on the pose of the anatomical object relative to a camera reference frame; and fine alignment instructions, which, upon execution, cause the one or more processors to implement fine alignment. The system comprises: a block, the fine alignment block being configured to align voxels of the 3D preoperative model with pixels in one or more 2D pixel arrangements based on depth information associated with pixels in one or more pixel arrangements and the corresponding voxel depth within the 3D preoperative model; a rendering instruction, when executed, causing one or more processors to render the aligned 3D preoperative model to produce a rendered 2D pixel arrangement; a CNN instruction, when executed, causing one or more processors to implement a CNN, the CNN being configured to associate anatomical object classifications with pixels in the 2D pixel arrangement based at least in part on cues provided by the rendered 2D pixel arrangement; and a display screen configured to display the categorized 2D pixel arrangement and the associated anatomical object classifications.

[0106] Example 15 includes the subject of Example 14, wherein the coarse alignment block is configured to align the voxels of the three-dimensional (3D) preoperative model with one or more two-dimensional (2D) pixel arrangements based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

[0107] Example 16 includes the subject of Example 14, where the coarse alignment block is configured based on the composite camera transform chain. Align the voxels of the three-dimensional (3D) preoperative model with one or more pixels arranged in a two-dimensional (2D) pixel array; where The change in the orientation of the display platform relative to the camera reference frame; whereby... This indicates the patient's transformation relative to the reference frame; and This indicates the change in the posture of the anatomical object relative to the patient's reference frame.

[0108] Example 17 includes the subject of Example 14, where fine alignment blocks are configured to align voxels of a 3D preoperative model with pixels in one or more 2D pixel arrangements based on an iterative nearest-point process.

[0109] Example 18 includes the subject of Example 14, wherein a camera is configured to capture 2D stereo pixel arrangements; and wherein a fine alignment block is configured to align voxels of a 3D preoperative model with pixels in one or more 2D pixel arrangements based on pixel depth information associated with the distance between corresponding pixels in two 2D stereo pixel arrangements.

[0110] Example 19 includes the subject of Example 14, wherein a CNN implemented by one or more processors includes one or more convolutional and activation layers, region proposal network layers, and one or more classification layers.

[0111] Example 20 includes the subject of Example 14, wherein multiple voxels of a 3D preoperative model are labeled to indicate the type of anatomical object; wherein multiple pixels in a rendered 2D pixel arrangement are labeled to indicate the type of anatomical object; and wherein a CNN implemented by one or more processors is configured to use the labels as classification cues.

[0112] Example 21 includes the subject of Example 14, wherein multiple voxels of a 3D preoperative model are labeled to indicate the type of anatomical object; wherein a rendering block is configured to pass the voxel labels of the 3D preoperative model to the corresponding pixels in a rendered 2D pixel arrangement; and wherein a CNN implemented by one or more processors is configured to use the labels as classification cues.

[0113] Example 22 includes an imaging system for a surgical or diagnostic medical procedure, comprising: a camera that captures an image of an anatomical object and represents the image in one or more 2D pixel arrangements; one or more processors and a non-transitory computer-readable medium operatively coupled thereto, the non-transitory computer-readable medium including instructions executable by the one or more processors, including: coarse alignment instructions that, when executed, cause the one or more processors to implement a coarse alignment block configured to align voxels of a three-dimensional (3D) preoperative model with pixels in one or more two-dimensional (2D) pixel arrangements based on a nominal pose of the anatomical object relative to a camera reference frame and based on pixel alignment error correction information; rendering instructions that, when executed, cause the one or more processors to render the aligned 3D preoperative model to produce a rendered 2D pixel arrangement; and CNN instructions executable by the one or more processors that, when executed, cause... One or more processors implement a CNN configured to associate anatomical object classification with pixels in a 2D pixel arrangement based at least in part on cues provided by a rendered 2D pixel arrangement; wherein multiple voxels of a 3D preoperative model are labeled to indicate anatomical object types; wherein multiple pixels in a rendered 2D pixel arrangement are labeled to indicate anatomical object types; wherein the CNN implemented by one or more processors is configured to use the labels as classification cues; instructions executable by one or more processors further include: alignment correction instructions, which, when executed, cause one or more processors to implement an alignment error correction block configured to identify a mismatch between a label associated with a pixel in the rendered 2D pixel arrangement and a classification associated with the corresponding classification pixel, and generate pixel alignment error correction information based on the identified mismatch; and a display screen configured to display the categorized 2D pixel arrangement and the associated anatomical object classification.

[0114] Example 23 includes the subject of Example 22, wherein the coarse alignment block is configured to align one or more pixels in a three-dimensional (3D) preoperative model voxels and a two-dimensional (2D) pixel arrangement based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

[0115] Example 24 includes the subject of Example 22, wherein a CNN implemented by one or more processors includes one or more convolutional and activation layers, a region proposal network layer, and one or more classification layers.

[0116] Example 25 includes the subject of Example 22, wherein the alignment correction block is configured to identify a threshold level of mismatch between the label associated with a pixel in the rendered 2D pixel arrangement and the classification associated with the corresponding pixel classified by the CNN pixel classification.

[0117] Example 26 includes the subject of Example 22, wherein the alignment correction block is configured to generate pixel alignment error correction information indicating the difference between the nominal pose of the anatomical object relative to a camera reference frame indicated by a rendered 2D pixel arrangement and the pose of the anatomical object captured by the camera based on a classified 2D pixel arrangement.

[0118] The above description is presented to enable those skilled in the art to create and use imaging systems for pixel-level segmentation of anatomical tissue images. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the scope of the invention. In the foregoing description, numerous details have been set forth for purposes of explanation. However, those skilled in the art will recognize that the embodiments of this disclosure can be practiced without using these specific details. In other instances, well-known processes are shown in block diagram form to avoid obscuring the description of the invention with unnecessary detail. The same reference numerals may be used to denote different views of the same or similar items in different figures. Therefore, the foregoing description and figures of embodiments of the invention are merely illustrative of the principles of the invention. Consequently, it should be understood that those skilled in the art can make various modifications to the embodiments without departing from the scope of the invention, which is defined by the appended claims.

Claims

1. An imaging system for surgical or diagnostic medical procedures, comprising: A camera that captures images of the anatomical object and represents the images in a two-dimensional pixel arrangement, i.e., a 2D pixel arrangement; One or more processors and a tangible computer-readable medium operatively coupled thereto, the tangible computer-readable medium including the following information: CNN instructions executable by the one or more processors, when executed, cause the one or more processors to implement a CNN, which is configured, according to an optional set of weights, to associate anatomical object classifications with pixels in the 2D pixel arrangement; And multiple weight sets, which are configured differently by the one or more processors based on different camera image training data, the camera image training data including different poses of one or more anatomical objects relative to the camera's camera reference frame, to train the CNN to distinguish different tissue types at least based on the camera poses relative to the anatomical object poses; as well as The display screen is configured to display a two-dimensional pixel arrangement of categories and an associated anatomical object category indicating tissue type; and The posture of the anatomical object relative to the camera reference frame is determined based on a combination of the posture of the stage relative to the camera reference frame, the posture of the patient relative to the stage reference frame, and the posture of the anatomical object relative to the patient reference frame.

2. The imaging system according to claim 1, Each weight set corresponds to a training dataset, which includes camera images captured for the pose range of one or more anatomical objects relative to one or more camera reference frames; and Each training dataset includes camera images captured for different pose ranges of the one or more anatomical objects relative to one or more camera reference frames, different from the other training datasets.

3. The imaging system according to claim 1, The CNN implemented by the one or more processors includes one or more convolutional and activation layers, region proposal network layers, and one or more classification layers.

4. The imaging system according to claim 3, Each of the different weight sets includes a different set of feature weights for configuring the convolutional and activation layers of the CNN implementation and a different set of classification weights for configuring the one or more classification layers of the CNN implementation.

5. The imaging system according to claim 1, further comprising: The display screen shows the two-dimensional pixel arrangement of the classification and the associated anatomical object classification.

6. An imaging system for surgical or diagnostic medical procedures, comprising: A camera that captures images of the anatomical object and represents the images in a two-dimensional pixel arrangement, i.e., a 2D pixel arrangement; One or more processors and a tangible computer-readable medium operatively coupled thereto, the tangible computer-readable medium including instructions executable by the one or more processors, the instructions including: The CNN instruction, when executed, causes the one or more processors to implement a CNN, the CNN being configured to associate anatomical object classification with pixels in the 2D pixel arrangement; A CNN classification selection logic, which adjusts one or more classification values ​​generated by the CNN for the 2D pixel arrangement based on the pose of the anatomical object relative to the camera's camera reference frame; and The display screen is configured to display the classification of the two-dimensional pixel arrangement, i.e., the 2D pixel arrangement, and the associated anatomical object classification indicating tissue type; and The one or more classification values ​​are adjusted based on a composite camera transformation, which is based on the transformation of the posture of the table relative to the camera reference frame, the transformation of the patient relative to the camera reference frame, and the transformation of the posture of the anatomical object relative to the patient reference frame.

7. The imaging system according to claim 6, The CNN implemented by the one or more processors includes one or more convolutional and activation layers, region proposal network layers, and one or more classification layers.

8. The imaging system according to claim 6, The CNN classification selection logic adjusts one or more classifications based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

9. The imaging system according to claim 6, The CNN classification selection logic is based on the composite camera transform chain. To adjust one or more of the categories; in The change in the orientation of the display platform relative to the camera reference frame; in This indicates the patient's transformation relative to the reference frame; and in This indicates the change in the posture of the anatomical object relative to the patient's reference frame.

10. An imaging system for surgical or diagnostic medical procedures, comprising: A camera that captures images of the anatomical object and represents the images in a 2D pixel arrangement; as well as One or more processors and a tangible computer-readable medium operatively coupled thereto, the tangible computer-readable medium including the following information: CNN instructions executable by the one or more processors, when executed, cause the one or more processors to implement a CNN, the CNN being configured to associate anatomical object classification with pixels in the 2D pixel arrangement; The CNN implemented by the one or more processors includes, One or more convolutional and activation layers, which generate one or more image feature maps based on one or more 2D pixel arrangements, and A fully connected layer that associates anatomical object classification with pixels in one or more 2D pixel arrangements based on the one or more feature maps, the relative pose between the patient's anatomy and the camera reference frame of the camera, and camera calibration information. as well as The display screen is configured to display a two-dimensional pixel arrangement (i.e., a 2D pixel arrangement) of classifications and associated anatomical object classifications indicating tissue types; and The relative posture of the patient's anatomy with respect to the camera reference system is determined based on a combination of the posture of the stage relative to the camera reference system, the posture of the patient relative to the stage reference system, and the posture of the patient's anatomy relative to the patient reference system.

11. The imaging system according to claim 10, The CNN implemented by the one or more processors includes, A region proposal network layer generates regions of interest based on the feature maps.

12. The imaging system according to claim 10, The CNN implemented by the one or more processors includes, A region proposal network layer that identifies one or more regions of interest within the one or more feature maps; The fully connected layer associates the anatomical object classification with pixels in the one or more 2D pixel arrangements based on the feature mapping in the one or more regions of interest, the relative pose between the patient anatomy and the camera, and camera calibration information.

13. An imaging system for surgical or diagnostic medical procedures, comprising: A camera that captures images of the anatomical object and represents the images in a two-dimensional pixel arrangement, i.e., a 2D pixel arrangement; One or more processors and a tangible computer-readable medium operatively coupled thereto, the tangible computer-readable medium including instructions executable by the one or more processors, the instructions including: A coarse alignment instruction, when executed, causes one or more processors to implement a coarse alignment block, which is configured to align the voxels of the three-dimensional preoperative model (i.e., the 3D preoperative model) with pixels in one or more two-dimensional pixel arrangements (i.e., the 2D pixel arrangements) based on the pose of the anatomical object relative to the camera reference frame. A fine alignment instruction, when executed, causes the one or more processors to implement a fine alignment block, the fine alignment block being configured to align the voxels of the 3D preoperative model with the pixels of the one or more 2D pixel arrangements based on depth information associated with pixels in the one or more pixel arrangements and the corresponding voxel depth within the 3D preoperative model; Rendering instructions, when executed, cause the one or more processors to render the aligned 3D preoperative model to produce a rendered 2D pixel arrangement; CNN instructions, when executed, cause the one or more processors to implement a CNN, the CNN being configured to associate anatomical object classifications with pixels in the 2D pixel arrangement based at least in part on cues provided by the rendered 2D pixel arrangement; and The display screen is configured to display a two-dimensional pixel arrangement (i.e., a 2D pixel arrangement) of classifications and associated anatomical object classifications indicating tissue types; and The posture of the anatomical object relative to the camera reference frame is determined based on a combination of the posture of the stage relative to the camera reference frame, the posture of the patient relative to the stage reference frame, and the posture of the anatomical object relative to the patient reference frame.

14. The imaging system according to claim 13, The coarse alignment block is configured to align the voxels of the three-dimensional preoperative model (i.e., the 3D preoperative model) with one or more pixels in the two-dimensional pixel arrangement (i.e., the 2D pixel arrangement) based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

15. The imaging system according to claim 13, The coarse alignment block is configured based on the composite camera transform chain. Align the voxels of the three-dimensional preoperative model with one or more pixels in the two-dimensional pixel arrangement, i.e., the 2D pixel arrangement. in The change in the orientation of the display platform relative to the camera reference frame; in This indicates the patient's transformation relative to the reference frame; and in This indicates the change in the posture of the anatomical object relative to the patient's reference frame.

16. The imaging system according to claim 13, The fine alignment block is configured to align the voxels of the 3D preoperative model with pixels in one or more of the 2D pixel arrangements based on an iterative nearest-point process.

17. The imaging system according to claim 13, The camera is configured to capture a 2D stereo pixel arrangement; and The fine alignment block is configured to align the voxels of the 3D preoperative model with one or more pixels in the 2D pixel arrangement based on pixel depth information associated with the distance between corresponding pixels in two 2D stereo pixel arrangements.

18. The imaging system according to claim 13, The CNN implemented by the one or more processors includes one or more convolutional and activation layers, region proposal network layers, and one or more classification layers.

19. The imaging system according to claim 13, In the 3D preoperative model, multiple voxels are labeled to indicate the type of anatomical object; In the rendered 2D pixel arrangement, multiple pixels are marked to indicate the type of anatomical object; The CNN implemented by the one or more processors is configured to use labels as classification cues.

20. The imaging system according to claim 13, In the 3D preoperative model, multiple voxels are labeled to indicate the type of anatomical object; The rendering block is configured to pass the voxel markers of the 3D preoperative model to the corresponding pixels in the rendered 2D pixel arrangement; and The CNN implemented by the one or more processors is configured to use the labels as classification cues.

21. An imaging system for surgical or diagnostic medical procedures, comprising: A camera that captures images of the anatomical object and represents the images in one or more two-dimensional pixel arrangements, i.e., one or more 2D pixel arrangements; One or more processors and a tangible computer-readable medium operatively coupled thereto, the tangible computer-readable medium including instructions executable by the one or more processors, including: A coarse alignment instruction, when executed, causes one or more processors to implement a coarse alignment block, which is configured to align the voxels of the three-dimensional preoperative model (i.e., the 3D preoperative model) with pixels in one or more two-dimensional pixel arrangements (i.e., the 2D pixel arrangements) based on the nominal pose of the anatomical object relative to the camera reference frame and based on pixel alignment error correction information. The nominal pose of the anatomical object relative to the camera reference frame is determined based on a combination of the pose of the table relative to the camera reference frame, the pose of the patient relative to the table reference frame, and the pose of the anatomical object relative to the patient reference frame. Rendering instructions, when executed, cause the one or more processors to render the aligned 3D pre-operative model to produce a rendered 2D pixel arrangement; CNN instructions executable by the one or more processors, which, when executed, cause the one or more processors to implement a CNN, the CNN being configured to associate anatomical object classification with pixels in the 2D pixel arrangement based at least in part on cues provided by the rendered 2D pixel arrangement; In the 3D preoperative model, multiple voxels are labeled to indicate the type of anatomical object; In the rendered 2D pixel arrangement, multiple pixels are marked to indicate the type of anatomical object; The CNN implemented by the one or more processors is configured to use labels as classification cues; the instructions executable by the one or more processors also include: An alignment correction instruction, when executed, causes one or more processors to implement an alignment error correction block, the alignment error correction block being configured to identify a mismatch between a marker associated with a pixel in the rendered 2D pixel arrangement and a category associated with the corresponding categorized pixel, and to generate pixel alignment error correction information based on the identified mismatch; and The display screen is configured to display a two-dimensional pixel arrangement of the classification, i.e., a 2D pixel arrangement of the classification, and an associated anatomical object classification indicating the tissue type.

22. The imaging system according to claim 21, The coarse alignment block is configured to align the voxels of the three-dimensional preoperative model (i.e., the 3D preoperative model) with one or more pixels in the two-dimensional pixel arrangement (i.e., the 2D pixel arrangement) based on a combination of the pose of the stage relative to the camera reference frame, the pose of the patient relative to the stage reference frame, and the pose of the anatomical object relative to the patient reference frame.

23. The imaging system according to claim 21, The CNN implemented by the one or more processors includes one or more convolutional and activation layers, region proposal network layers, and one or more classification layers.

24. The imaging system according to claim 21, The alignment correction block is configured to identify a threshold level of mismatch between a marker associated with a pixel in the rendered 2D pixel arrangement and a classification associated with the corresponding pixel classified by the CNN pixels.

25. The imaging system according to claim 21, The alignment correction block is configured to generate pixel alignment error correction information indicating the difference between the nominal pose of the anatomical object relative to the camera reference frame, as indicated by the rendered 2D pixel arrangement, and the pose of the anatomical object captured by the camera, as indicated by the classified 2D pixel arrangement.

Images