Medical image analysis using machine learning and anatomical vectors

Abstract

The present disclosure relates to a computer-implemented method comprising registering a tracking imaging device (such as a microscope) with a known viewing direction and an atlas to a patient space so that a transformation can be established between the atlas space and a reference frame for defining the positions in the image of the patient's anatomical structures. Labels are associated with certain components of the image and are input into a learning algorithm together with the medical image and anatomical vectors and, for example, the atlas to train the learning algorithm (such as a machine learning algorithm, e.g., a convolutional neural network) to automatically segment patient images generated with the tracking imaging device. The trained learning algorithm then allows for efficient segmentation and / or labeling of the patient images without having to register the patient images to the atlas each time, thereby saving computational effort.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method of training a learning algorithm to determine a relationship between, on the one hand, a location or type of anatomical structures in a medical image and, on the other hand, a label indicating the location or type of anatomical structures in the medical image, as well as a method of using the trained learning algorithm to segment and / or label medical patient images, a corresponding computer program, a computer-readable storage medium storing such a program, a computer executing such a program, and a system comprising an electronic data storage device and the aforementioned computer. Background Art

[0002] Anatomical images can be used to segment or label medical patient images. This requires registering the patient images to the atlas data, which involves a large amount of computational effort. Summary of the invention

[0003] It is an object of the present invention to provide a method for more efficient segmentation and / or labeling of medical patient images.

[0004] The following discloses various aspects, examples and exemplary steps of the invention and its embodiments.Different exemplary features of the invention can be combined according to the invention whenever technically suitable and feasible.

[0005] In the following, a brief description of specific features of the present invention is given, which should not be construed as limiting the present invention to only the features or combinations of features described in this section.

[0006] The method of the present disclosure comprises registering a tracking imaging device, such as a microscope with a known viewing direction and an atlas, to patient space so that a transformation can be established between the atlas space and a reference frame for defining the positions in the image of the patient's anatomical structures. Labels are associated with certain components of the image and are input into a learning algorithm together with the medical image and anatomical vectors and, for example, the atlas to train the learning algorithm for automatic segmentation of patient images generated with the tracking imaging device, the learning algorithm being, for example, a machine learning algorithm, such as a convolutional neural network. The trained learning algorithm then allows for efficient segmentation and / or labeling of patient images without having to register the patient images to the atlas each time, thereby saving computational effort.

[0007] In this summary, a description of the general features of the invention is given, for example, by reference to possible embodiments of the invention.

[0008] In general, in a first aspect, the present invention achieves the above-mentioned object by a computer-implemented method (e.g., a medical method) of training a learning algorithm to determine a relationship between, on the one hand, an indication of the location or type of an anatomical structure in a medical image and, on the other hand, an indication of the location or type of an anatomical structure in a medical image. The method according to the first aspect comprises executing, on at least one processor of at least one computer (e.g., at least one computer as part of a navigation system), the following exemplary steps performed by at least one processor.

[0009] In a (e.g. first) exemplary step of the method according to the first aspect, patient training image data are acquired, the patient training image data describing digital medical images of anatomical structures of a plurality of patients. For example, the medical images are two-dimensional images such as microscopic images, for example parts of videos taken with a microscope, wherein the imaging device is for example a microscope. The patient training image data have for example been generated by images or videos taken with an imaging device that generates two-dimensional images (such as an electron microscope or a camera or an endoscope equipped with a digital camera) or with an X-ray device that generates two-dimensional projection images or is configured to generate two-dimensional projection images.

[0010] In a (eg second) exemplary step of the method according to the first aspect, atlas data are acquired, the atlas data describing an image-based model of a body anatomical region including anatomical structures.

[0011] In an exemplary step (e.g., the third) of the method according to the first aspect, viewing direction data is acquired, which describes the viewing direction of the imaging device toward the anatomical structure at a point in time when the medical image is generated using the imaging device. For example, the viewing direction has been determined by tracking the imaging device with a tracking system, which is based on the principle of marker tracking (i.e., optically detecting a retroreflective marker that is attached to the imaging device in a predetermined and known relationship with its viewing direction), video tracking, or electromagnetic tracking. The viewing direction can be further based on or defined by the geometry of the imaging device, and optically, for example, based on the field of view or focal axis of the imaging device, or for example, the focal point, such as the position of the focal spot of the imaging device. The viewing direction, for example, includes a direction perpendicular to the imaging plane. The viewing direction, for example, includes or is defined by the position of the camera and the orientation of the camera. In addition, the viewing direction data, for example, includes information defining the focal length, zoom, or magnification of the imaging device. Additionally or alternatively, it, for example, includes information defining the position of an edge or corner of the image or field of view, such as information about the position of the imaging device. For an imaging device using X-rays and comprising a substantially flat X-ray detector, the viewing direction data can, for example, comprise or be defined by a normal to the detector area, or it can, for example, comprise information defining the center of the detector area, or the center of an area defined, for example, by collimator blades near the X-ray detector or the X-ray source.

[0012] In an (e.g. fourth) exemplary step of the method according to the first aspect, anatomical vector data is determined based on viewing direction data and atlas data, wherein the anatomical vector data describes an anatomical vector that is a result of transforming the viewing direction into a reference system in which a position in an image-based model is defined.

[0013] This transformation from viewing direction data to anatomical vector data is performed, for example, by the following steps a) to c):

[0014] a) transforming the coordinates of the position and orientation of the tracking imaging device (which are, for example, defined in the coordinate system of the tracking system, such as relative to a stereo tracking camera of an optical tracking system) into the same coordinate system as the patient (e.g., via optical or other tracking markers attached to the patient), such as the patient's intraoperative coordinate system.

[0015] b) transforming positions in the planning image data (e.g. a computed tomography image, e.g. an X-ray or magnetic resonance tomography from the patient) into a patient coordinate system of the patient (e.g. using, e.g., radiopaque artificial markers or natural landmarks in the preoperative image data). The coordinate system of the planning image data is, e.g., defined by the coordinate system of an imaging device (e.g., a computed tomography scanner or a magnetic resonance imaging scanner) used to acquire the planning image data. The planning image data is, e.g., acquired preoperatively.

[0016] c) The positions that have been transformed to the patient coordinate system are then transformed to the coordinate system of the anatomical atlas, using a registration of the planning image data to the atlas, which registration can be, for example, a rigid transformation, such as an affine transformation, or an elastic transformation, such as a deformation of the image data. After the registration, the deformed image data is consistent with at least a portion of the atlas, or at least a portion of the deformed atlas is consistent with the image data.

[0017] As a result of the above transformation steps a) to c), all coordinates given in any coordinate system, such as coordinates in preoperative image data, coordinates of the viewing direction of an imaging device, coordinates of a tracking system and all tracking devices tracked by the tracking system, and patient coordinates, can be expressed in an atlas coordinate system, which is not specific to an individual patient. This transformation process is also called "registration"; the tracked imaging device is thereby registered to the atlas coordinate system. The anatomical vector data, for example, include viewing direction data of the imaging device expressed in the atlas coordinate system, such as the viewing direction coordinates transformed into coordinates of the atlas coordinate system.

[0018] In a (eg, fifth) exemplary step, label data is acquired that describes labels that describe the location or type of anatomical structures in the image-based model.

[0019] In an (e.g., sixth) exemplary step of the method according to the first aspect, anatomical indicator data is determined based on patient training image data and anatomical vector data and label data, wherein the anatomical indicator data describes model parameters (e.g., weights) of a learning algorithm for establishing a relationship (e.g., relative position or assignment) between the position or type of the anatomical structure described by the medical image and the label, wherein the anatomical indicator data is determined by inputting the patient training image data and the label data into a function that establishes the relationship. For example, the learning algorithm includes or consists of a machine learning algorithm. For example, the learning algorithm includes or consists of a convolutional neural network. For example, the model parameters define learnable parameters of the learning algorithm, such as weights. For example, the anatomical indicator data is determined by additionally inputting a subset of the atlas data that has been determined based on the atlas data and the anatomical vector data into the function that establishes the relationship. The subset is, for example, a real subset, i.e., has fewer elements, i.e., less data, than the atlas data.

[0020] In an example of the method according to the first aspect, additional data is acquired as a function of the anatomical vectors. The anatomical indicator data is then determined by additionally inputting the additional data into the function establishing the relationship. For example, the additional data includes or consists of the anatomical vector data.

[0021] In a second aspect, the present invention is directed to a computer-implemented method (e.g., a medical method) of determining a relationship between, on the one hand, a location or type of an anatomical structure in a medical image and, on the other hand, a label indicating the location or type of the anatomical structure in the medical image. The method according to the second aspect comprises executing, on at least one processor of at least one computer (e.g., at least one computer as part of a navigation system), the following exemplary steps performed by at least one processor.

[0022] In a (e.g. first) exemplary step of the method according to the second aspect, individual patient image data is acquired, which describes a digital individual medical image of the anatomical structure of the individual patient. For example, the individual medical image has been generated using the same imaging mode as used to generate the patient training image data.

[0023] In an (e.g. second) exemplary step of the method according to the second aspect, label relationship data is determined, which label relationship data describes the relationship (e.g. relative position or assignment) between labels and anatomical structures in individual medical images, wherein the label relationship data is determined by inputting individual patient image data into a function that establishes the relationship between the anatomical structures described by the individual medical images and the labels, the function being part of a learning algorithm that has been trained by performing the method according to the first aspect (as long as it involves inputting only patient training image data and label data as training data into the function that establishes the relationship).

[0024] In a third aspect, the present invention is directed to a computer-implemented method (e.g., a medical method) of determining a relationship between, on the one hand, a location or type of an anatomical structure in a medical image and, on the other hand, a label indicating the location or type of the anatomical structure in the medical image. The method according to the third aspect comprises executing, on at least one processor of at least one computer (e.g., at least one computer as part of a navigation system), the following exemplary steps performed by at least one processor.

[0025] In a (e.g. first) exemplary step of the method according to the third aspect, individual patient image data is obtained, which describes a digital individual medical image of the anatomical structure of the individual patient. For example, the individual medical image has been generated using the same imaging mode as used to generate the patient training image data. The individual medical image is a two-dimensional image such as a microscopic image, for example a part of a video taken with a microscope, wherein the imaging device is, for example, a microscope.

[0026] In a (eg second) exemplary step of the method according to the third aspect, atlas data is acquired, the atlas data describing an image-based model of a body anatomical region including anatomical structures.

[0027] In an (e.g. third) exemplary step of the method according to the third aspect, individual viewing direction data are acquired, which describe the viewing direction of the imaging device towards the anatomical structure at a point in time when the medical image is generated using the imaging device. The viewing direction has been determined, for example, by tracking the imaging device with a tracking system based on the principles of marker tracking (i.e. optically detecting a retroreflective marker that is attached to the imaging device in a predetermined and known relationship to its viewing direction), video tracking or electromagnetic tracking.

[0028] In a (e.g. fourth) exemplary step of the method according to the third aspect, individual anatomical vector data are determined based on the individual viewing direction data and the atlas data, wherein the anatomical vector describes an anatomical vector which is a result of transforming the viewing direction into a reference system in which positions of the image-based model are defined, e.g. a relative position between an imaging device, e.g. a microscope, used to generate the individual patient image data and the individual anatomical vector data, e.g. predetermined, e.g. known and e.g. acquired by the method.

[0029] In a (eg fifth) exemplary step of the method according to the third aspect, additional data is acquired, the additional data being a function of the individual anatomical vectors. For example, the additional data comprises or consists of the individual anatomical vector data.

[0030] In an (e.g. sixth) exemplary step of the method according to the third aspect, label relationship data describing a relationship (e.g. a relative position or assignment) between a label and an anatomical structure described by the individual medical image is determined, wherein the label relationship data is determined by inputting the individual patient image data and the additional data into a function establishing a relationship between the position or type of the anatomical structure in the individual medical image and the label, the function having been part of a learning algorithm trained by executing the method according to the first aspect (insofar as it comprises inputting only patient training image data and label data as training data into the function establishing the relationship). The function used to generate the anatomical vector of the acquired additional data is the same as the function used to generate the anatomical vector of the additional data input into the function establishing the anatomical indicator data.

[0031] In an example of the method according to the third aspect, the additional data comprises or consists of individual anatomical vector data and the learning algorithm has been trained by additionally inputting the additional data into the function establishing the relationship.

[0032] In an example of the method according to the third aspect, label relationship data is additionally determined based on the atlas data by additionally inputting a subset of the atlas data into a function that establishes a relationship between an anatomical structure described by an individual medical image and a label, and the learning algorithm has been trained by additionally inputting a subset of the atlas data determined based on the atlas data and the anatomical vector data into the function that establishes the relationship. The subset is, for example, a real subset, i.e., has fewer elements, i.e., less data, than the atlas data.

[0033] In the method according to the second and third aspects, the learning algorithm for example comprises or consists of a machine learning algorithm, such as a convolutional neural network. In the method according to the second and third aspects, the model parameters define learnable parameters of the learning algorithm, such as weights.

[0034] In one example of the method according to the first, second and third aspects, the learning algorithm may be a random forest algorithm. See Antonio Criminisi, Jamie Shotton and Ender Konukoglu, Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning, 2011, available at https: / / www.microsoft.com / zh-us / research / wp-content / uploads / 2016 / 02 / CriminisiFore sts_FoundTrends_2011.pdf , the explanation of random forest is as follows: Random forest is an ensemble learning method for classification or regression, which constructs a large number of decision trees during training and outputs the pattern of classes (classification) or the average prediction of individual trees (regression).

[0035] The basic building block of a random forest is a single decision tree. A decision tree is a set of questions organized in a hierarchical manner and represented graphically as a tree. A decision tree estimates an unknown property ("label") of an object by asking consecutive questions about its known properties (so-called "features"). The next question to be asked depends on the answer to the previous question, and this relationship is represented graphically as a path through the tree that the object follows. Decisions are then made based on the terminal nodes on the path (so-called "leaf nodes"). Each question corresponds to an internal node of the tree (a so-called "split node").

[0036] Each split node has a so-called test function associated with it. The test function at a split node j is formulated as a function with binary output:

[0037]

[0038] Among them, 0 and 1 can be interpreted as "false" and "true" respectively. Represents the parameters of the test function at the jth split node.

[0039] v is therefore the vector represents the current object ("data point"), where the component x i Represents certain attributes (features) of data points, all of which form a feature space

[0040] In its simplest form, the test function is a linear model that selects a feature axis in the feature space and classifies each data point according to whether the value of the respective feature is below or above a learnable threshold. For other more complex forms, non-linear test functions are possible.

[0041] To train a decision tree, a set of training data points is used, for which the features as well as the required labels are known. The goal of the training is to automatically learn a suitable test function at all split nodes that is best suited to determine the label from the features of the data points. Later, this trained decision tree can be evaluated for new data points with unknown labels by sending the data points through the tree trained based on its features.

[0042] To understand the training process, it is helpful to represent subsets of training points as being associated with different branches. For example, S1 represents the subset of training points that reach node 1 (nodes are numbered starting from 0 in breadth-first order to the root F), while Represents a subset of the left and right children going to node 1.

[0043] Training is responsible for selecting a test function h(v,θ) associated with each split node (indexed by j) by optimizing a chosen objective function defined on the available training set. j ) types and parameters.

[0044] The optimization of the split function is performed in a greedy manner. At each node j, we learn to transform S into j The "best" split is and This problem is formulated as the maximization of the objective function at the node:

[0045]

[0046] in

[0047]

[0048]

[0049]

[0050] As mentioned earlier, the symbol represents the set of training points before and after segmentation. The objective function is in abstract form here. Its precise definition and the meaning of "best" depend on the task at hand (e.g., whether it is supervised, continuous or discrete output). For example, for binary classification, the term "best" can be defined as the segmentation of the training subset S j , so that the resulting child nodes are as pure as possible, that is, only contain training points of a single class. In this case, the objective function can be defined as information gain, for example.

[0051] During training, the structure (shape) of the tree also needs to be optimized. Training starts from the root node j=0, where the optimal split parameters are found as described before. Therefore, two child nodes are constructed, each receiving a different disjoint subset of the training set. This process is then applied to all newly constructed nodes, and the training phase continues. The structure of the tree depends on how and when to decide to stop growing the various branches of the tree. Different stopping criteria can be applied. For example, the tree is usually stopped when a maximum number of levels D is reached. Alternatively, a maximum value can be applied. In other words, it stops when the attributes of the training points in the leaf nodes are similar to each other. The tree growth may also stop when the node contains too few training points. In terms of generalization, avoiding growing the entire tree has been shown to have a positive effect.

[0052] During training, randomness is injected into the tree: when training at the jth node, instead of optimizing over the entire parameter space of the test function, only a small random subset of the parameter values ​​is used. Therefore, under the randomness model, the training tree is completed by optimizing each split node j:

[0053]

[0054] Due to this random setting, it is later possible to train multiple decision trees in parallel, each utilizing a different set of attributes from the data points.

[0055] At the end of the training phase, we obtain: (i) the (greedy) optimal weak learner associated with each node; (ii) the learned tree structure; and (iii) a different set of training points at each leaf.

[0056] After training, each leaf node is still associated with a subset of the (labeled) training data. During testing, a previously unseen point traverses the tree until it reaches a leaf. Since the split nodes act on features, an input test point may end up in a leaf associated with a training point similar to itself. Therefore, it is reasonable to assume that the associated label must also be similar to the labels of the training points in that leaf. This justifies using the label statistics collected in that leaf to predict the label associated with the input test point.

[0057] In the most general sense, the leaf statistics can be gathered using the posterior distribution:

[0058] p(c|v) and p(y|v)

[0059] where c and y represent discrete or continuous labels, respectively. v is the data point tested in the tree, and the conditional representation distribution depends on the fact that the particular leaf node is reached by the test point. Different leaf predictor variables can be used. For example, in the discrete case, the maximum a posteriori (MAP) estimate can be obtained as c*=argmax c p(c|v).

[0060] Building on the construction principles of decision trees described above, we can now move on to decision forests, also known as random forests:

[0061] A random decision forest is a set of randomly trained decision trees. The key aspect of the forest model lies in the fact that its constituent trees are randomly different from each other. This leads to decorrelation between individual tree predictions, which in turn, leads to improved generalization and robustness.

[0062] In a forest of T trees, use the variable to index each constituent tree. All trees are trained independently (and possibly in parallel). During testing, each test point v is pushed into all trees simultaneously (starting from the root) until it reaches the corresponding leaf. Tree testing can also usually be done in parallel, thus achieving high computational efficiency on modern parallel CPU or GPU hardware. All tree predictions can be merged into a single forest prediction by a simple averaging operation. For example, in classification:

[0063]

[0064] where p t (c|v) represents the posterior distribution obtained by the tth tree. Alternatively, the tree outputs can be multiplied together (although the trees are not statistically independent):

[0065]

[0066] The partition function Z is used to ensure that the probabilities are normalized.

[0067] In one example of the method according to the first aspect, the second aspect and the third aspect, the learning algorithm may be a convolutional neural network. Figure 1 An explanation of a convolutional neural network is provided as an example of a machine learning algorithm to be used with the disclosed invention.

[0068] A convolutional network (also called a convolutional neural network, or CNN) is an example of a neural network for processing data with a known grid-like topology. Examples include time series data (which can be viewed as a one-dimensional grid sampled at regular time intervals) and image data (which can be viewed as a two-dimensional or three-dimensional grid of pixels). The name "convolutional neural network" indicates that the network employs the mathematical operation of convolution. Convolution is a linear operation. Convolutional networks are simple neural networks that use convolution instead of general matrix multiplication in at least one of their layers. There are many variations of the convolution function, which is widely used in neural networks in practice. In general, the operation used in convolutional neural networks does not correspond exactly to the definition of convolution used in other fields (such as engineering or pure mathematics).

[0069] The main component of a convolutional neural network is an artificial neuron. Figure 1 is an example of a single neuron depicted in Figure 1. The node in the middle represents the neuron, which accepts all inputs (x1,…,x n ), and multiply them by their specific weights (w1,…,w n ). The importance of the input depends on its weight value. The addition of these calculated values ​​is called the weighted sum, which will be plugged into the activation function. The weighted sum z is defined as:

[0070]

[0071] The bias b is a value that is independent of the input and modifies the boundaries of the threshold. The resulting value is processed by the activation function, which decides whether to pass the input to the next neuron.

[0072] CNNs usually take 1st-order or 3rd-order tensors as their input, for example, an image with H rows, W columns, and 3 channels (R, G, B color channels). However, CNNs can process higher-order tensor inputs in a similar way. The input then continues to undergo a series of processing. A processing step is usually called a layer, which can be a convolutional layer, a pooling layer, a normalization layer, a fully connected layer, a loss layer, etc. The details of these layers are described in the following sections.

[0073]

[0074] Equation 5 above explains how CNN operates layer by layer in the forward pass. The input is x 1 , usually an image (1st or 3rd order tensor). The parameters involved in the first layer of processing are collectively referred to as tensor w1 The output of the first layer is x 2 , which also serves as the input for the second layer of processing. This process continues until all layers in the CNN are processed, and its output x L . But add an additional layer for backward error propagation, which is a method for learning good parameter values ​​in CNN. Suppose the problem at hand is an image classification problem with class C. A common strategy is to convert x L Output as a C-dimensional vector whose i-th entry encodes the prediction (x 1 The posterior probability of comes from the i-th class). In order to make x L becomes a probability mass function, which can set the processing in the (L-1)th layer to x L-1 Softmax transformation (comparing distance measures and data transformation nodes). In other applications, the output x L can have other forms and interpretations. The last layer is the loss layer. Assume t is the input x 1 The corresponding target value (ground truth) of , then the cost or loss function can be used to measure the CNN prediction x L The difference between t and the target t. It should be noted that some layers may not have any parameters, i.e., for some i, w i May be empty.

[0075] In the example of CNN, ReLu is used as the activation function of the convolutional layer, while the softmax activation function provides information to give a classification output. The following section will explain the purpose of the most important layers.

[0076] The input image is input to the feature learning part which includes layers of convolution and ReLu, followed by layers including pooling, which are followed by further pairwise repetitions of layers of convolution and ReLu and pooling. The output of the feature learning part is input to the classification part which includes layers for flattening, fully connected and maximum softening.

[0077] In a convolutional layer, multiple convolution kernels are usually used. Assuming that D kernels are used and the spatial span of each kernel is H×W, all kernels are denoted as f. f is Similarly, using index variables 0≤i<H, 0≤j<W, 0≤d l <D l and 0≤d<D to determine the specific elements in the kernel. It should also be noted that the kernel set f is similar to the symbol w above L The same objects are referenced. The notation is changed slightly to simplify the derivation. It is also clear that the kernel remains the same even when the mini-batch strategy is used.

[0078] As long as the convolution kernel is larger than 1 × 1, the spatial amplitude of the output is smaller than the spatial amplitude of the input. Sometimes it is necessary for the input and output images to have the same height and width, and a simple padding trick can be used.

[0079] For each input channel, if padding (i.e., inserting) is done above the first row of input rows, fill below the last row (i.e., insert) Rows and fill Column to the left of the first column, fill Columns to the right of the last column, the size of the convolution output will be H l ×W l ×D, i.e., has the same spatial magnitude as the input. is the floor function. The elements of the padded rows and columns are typically set to 0, but other values ​​are possible.

[0080] Stride is another important concept in convolution. The kernel is convolved with the input at every possible spatial amplitude, which corresponds to a stride of s = 1. However, if s>1, each movement of the kernel skips s-1 pixel positions (i.e., a convolution is performed every s pixels in the horizontal and vertical directions).

[0081] In this section, we consider the simple case where the stride is 1 and no padding is used. There is y (or x l+1 ), where H l+1 =H l -H+1,W l+1 =W l -W+1, and D l+1 = D. In precise mathematics, the convolution process can be expressed as an equation:

[0082]

[0083] For all 0≤d≤D=D l+1 , and satisfy 0≤i l+1 <H l –H+1=H l+1 , 0≤j l+1 <W l –W+1=W l+1 Any spatial position (i l+1 ,j l+1 ) Repeat equation (2). In this equation, Refers to the triple (i l+1 +i,j l +1 +j,d l ) indexed by x l elements. Usually the bias term b dAdd to For greater clarity, this term is omitted.

[0084] Pooling functions replace the network output at a certain location with summary statistics of nearby outputs. For example, the max pooling operation reports the maximum output within a rectangular neighborhood of a table. Other popular pooling functions include the average of a rectangular neighborhood, the L2 norm of a rectangular neighborhood, or a weighted average based on the distance to the center pixel. In all cases, pooling helps make the representation approximately invariant to small translations of the input. Translation invariance means that if the input is translated by a small amount, the value of the pooled output will not change.

[0085] Since pooling aggregates responses over an entire neighborhood, it is possible to use fewer pooling units than detector units by reporting summary statistics for regions that are k pixels apart instead of one pixel. This improves the computational efficiency of the network because the next layer has approximately k times fewer inputs to process.

[0086] Assume that the CNN model w has been learned 1 ,…,w L-1 If we know all the parameters of , we can use the model to make predictions. Prediction involves only running the CNN model in the forward direction, that is, in the direction of the arrow in Equation 1. Take the image classification problem as an example. From the input x 1 To begin, pass it through the first layer (with parameter w 1 ) and get x 2 . In turn, x 2 Pass it to the second layer, and so on. Finally, receive Its estimated x 1 The posterior probability belongs to class C. The CNN prediction output can be:

[0087]

[0088] The question at this point is: how to learn the model parameters?

[0089] As in many other learning systems, the parameters of the CNN model are optimized to minimize the loss z, i.e., we want the predictions of the CNN model to match the ground truth labels. Suppose we are given a training example x 1 The training process involves running the CNN network in two directions. First, the network is run in a forward pass to obtain x L , to achieve predictions using the current CNN parameters. Instead of outputting predictions, we need to compare the predictions with the corresponding 1 The target t is compared, that is, the forward pass is continued until the last loss layer. Finally, the loss z is obtained. The loss z is a supervisory signal that guides how to correct (update) the parameters of the model.

[0090] There are several algorithms for optimizing loss functions, and CNNs are not limited to a specific algorithm. An example algorithm is called stochastic gradient descent (SGD). This means that the parameters are updated by using gradients estimated from a (usually) small subset of the training examples.

[0091]

[0092] In Equation 4, the ← symbol implicitly indicates the parameter w (of layer i) i Update from time t to t+1. If the time index t is used explicitly, the equation will be written as:

[0093]

[0094] In Equation 4, the partial derivative Measure z relative to w i The growth rate of the different dimensions of . This partial derivative vector is called a gradient in mathematical optimization. Therefore, in w i In a small local area near the current value of , move w in the direction determined by the gradient i The target value z will be increased. To minimize the loss function, w should be updated in the opposite direction of the gradient i . This update rule is called gradient descent.

[0095] However, if you move too far in the direction of the negative gradient, the loss function may increase. Therefore, in each update, the parameters are only changed by a small fraction of the negative gradient (controlled by η (learning rate)). η>0 is usually set to a small number (such as η=0.001). If the learning rate is not too high, then the loss function is based on x 1 An update of will make the loss for this particular training example smaller. However, it is likely to make the loss for some other training examples larger. Therefore, all training examples need to be used to update the parameters. When all training examples have been used to update the parameters, it is said that a learning cycle has been processed. Typically, a learning cycle will reduce the average loss of the training set until the learning system fits the training data. Therefore, the gradient descent update learning cycle can be repeated and terminated at a certain point to obtain the CNN parameters (for example, it can be terminated when the average loss of the validation set increases).

[0096] The partial derivatives of the last layer are easy to calculate. Since x L In the parameter w L is directly connected to z under the control of, so it is easy to calculate Only when w L This step is only required if it is not empty. Similarly, it is easy to calculate For example, if we use the squared L2 loss, then is empty, and

[0097] In fact, for each layer, two sets of gradients are computed: with respect to the layer parameters w i The partial derivative of z, and the input x of this layer i As shown in Equation 4, the term Can be used to update the parameters of the current (i-th layer). Can be used to update parameters backwards, for example, to update to the (i-1)th layer. The intuitive explanation is: x i is the output of the (i-1)th layer, and How should x be changed? i To reduce the loss function. Therefore, it can be regarded as layer by layer is considered as part of the “wrong” supervision information propagated back from z to the current layer. Therefore, the backpropagation process can be continued and used The error is back-propagated to the (i-1)th layer. This layer-by-layer backward update procedure can greatly simplify learning CNN.

[0098] Take the i-th layer as an example. When updating the i-th layer, the back propagation process of the (i+1)-th layer must have been completed. That is, the term and Both are stored in memory and are available for use. The task at this point is to calculate and Using the chain rule, we get:

[0099]

[0100]

[0101] Since it has been calculated and store it in memory, so only a matrix reshape operation (vec) and an additional transpose operation are required to obtain It is the first term on the right hand side (RHS) of both equations. As long as we can calculate and The expected value (the left side of both equations) can be easily obtained.

[0102] and Compared with direct calculation and Much easier, because x i By taking the parameter w i The function of x i+1 Directly related.

[0103] In the context of neural networks, activations act as transfer functions between neuron inputs and outputs. They define under which conditions a node is activated, i.e. the input value is mapped to an output which in turn is used as one of the inputs to subsequent neurons in the hidden layer. There are a large number of different activation functions with different characteristics.

[0104] The loss function quantifies how well the algorithm models the given data. In order to learn from the data and change the weights of the network, the loss function must be minimized. In general, a distinction can be made between regression loss and classification loss. In classification, one tries to predict an output from a limited set of categorical values ​​(class labels), while regression, on the other hand, deals with predicting continuous values.

[0105] In the following mathematical formula, the following parameters are defined as:

[0106] n is the number of training examples;

[0107] i is the i-th training example in the dataset;

[0108] ·y i is the ground truth label of the ith training example;

[0109] · is the prediction for the ith training example.

[0110] The most common setting for classification problems is the cross entropy loss. It increases as the predicted probability deviates from the actual label. The logarithm of the actual predicted probability is multiplied by the ground truth category. The important aspect is that the cross entropy loss severely penalizes confident but wrong predictions. The mathematical formula can be described as:

[0111]

[0112] A typical example of regression loss is the mean squared error or L2 loss. As the name suggests, the mean squared error is the average of the squared differences between the predicted values ​​and the actual observed values. It only concerns the average error magnitude, regardless of their direction. However, due to the squaring, predictions far from the actual values ​​suffer from severe flaws compared to predictions with smaller deviations. In addition, MSE has nice mathematical properties that make it easier to calculate the gradient. Its formula is as follows:

[0113]

[0114] For information on the capabilities of convolutional neural networks, see the following papers:

[0115] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning, chapter convolutional networks, 2016, available at http: / / www.deeplearningbook.org ;

[0116] J. Wu, Introduction to convolutional neural networks, see https: / / pdfs.semanticscholar.org / 450c / a19932fcef1ca6d0442cbf52fec38fb9d1e5.p df ;

[0117] "Common loss functions in machine learning", please see https: / / towardsdatascience.com / common-loss-functions-in-machine-learning- 46af0ffc4d23 , last visited: 2019-08-22;

[0118] Imagenet classification with deep convolutional neural networks by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, see http: / / papers.nips.cc / paper / 4824-imagenet-classification-with-deep-convolutional- neur al-networks.pdf ;

[0119] S. Ren, K. He, R. Girshick, and J. Sun, Faster r-CNN: Towards real-time object detection with region proposal networks, see h ttps: / / arxiv.org / pdf / 1506.01497.pdf ;

[0120] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, Convolutional posemachines, available at https: / / arxiv.org / pdf / 1602.00134.pdf ;

[0121] Jonathan Long, Evan Shelhamer and Trevor Darrell "Fully convolutional networks for semantic segmentation", please see https: / / www.cv-foundation.org / openaccess / content_cvpr_2015 / papers / Long_Fully_Convolutional_Networks_2015_ CVPR_paper.pdf .

[0122] In a fourth aspect, the present invention is directed to a program which, when run on or loaded onto a computer, causes the computer to execute the method steps of the method according to the first to third aspects, and / or a (e.g., non-transitory) program storage medium storing the program, or a program storage medium storing data defining model parameters and an architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or a data carrier signal carrying the above-mentioned program, and / or a data carrier signal carrying data defining model parameters and an architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or a data stream carrying the above-mentioned program, and / or a data stream carrying data defining model parameters and an architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or at least one computer comprising at least one processor and a memory, wherein the above-mentioned program runs on at least one processor or is loaded into the memory of the computer.

[0123] The present invention may alternatively or additionally relate to a signal wave (e.g., physically, e.g., electrically, generated by technical means) carrying information representing a program, e.g., the above-mentioned program, such as a digital signal wave, such as an electromagnetic carrier wave, which, for example, includes a code mechanism suitable for executing any or all steps of the method according to the first aspect. In one example, the signal wave is a data carrier signal carrying the above-mentioned computer program. The computer program stored on the disk is a data file, which, when read and transmitted, becomes a data stream in the form of a (e.g., physically, e.g., electrically, generated by technical means). The signal can be implemented as a signal wave, such as an electromagnetic carrier wave described herein. For example, the signal (e.g., signal wave) is constructed to be transmitted through a computer network (e.g., LAN, WLAN, WAN, mobile network (e.g., Internet)). For example, the signal (e.g., signal wave) is constructed to be transmitted through optical or acoustic data transmission. Therefore, the present invention according to the second aspect may alternatively or additionally relate to a data stream representing the above-mentioned program (i.e., including the program).

[0124] In a fifth aspect, the present invention is directed to a computer-implemented method of a system for determining a relationship between, on the one hand, a label indicating a location or type of an anatomical structure in a medical image and, on the other hand, a label indicating a location or type of an anatomical structure in the medical image, comprising:

[0125] a) A computer, wherein the program is run on the computer or loaded into the memory of the computer, causing the computer to perform the method steps according to the second or third aspect;

[0126] b) at least one electronic data storage device storing individual patient image data and causing the computer to perform the method according to the third aspect whenever the above program is run on said at least one processor or loaded onto the memory of the computer, the electronic data storage device storing additional data, atlas data, individual viewing direction data, individual anatomical vector data and additional data; and

[0127] c) the program storage medium according to the fourth aspect,

[0128] Wherein at least one computer is operably coupled to:

[0129] - at least one electronic data storage device for acquiring individual patient image data from the at least one electronic data storage device and for causing the computer to perform the method according to the third aspect, acquiring additional data, atlas data, individual viewing direction data, individual anatomical vector data and additional data from the at least one electronic data storage device, and for storing at least label relationship data in the at least one electronic data storage device, whenever the above program is run on the at least one processor or loaded onto the memory of the computer; and

[0130] -A program storage medium for obtaining data defining the model parameters and the architecture of the learning algorithm from the program storage medium.

[0131] For example, the present invention does not involve or particularly does not include or encompass invasive procedures that represent substantial physical interference with the body requiring professional medical care and measures that may expose the body to significant health risks even if the necessary professional care and measures are taken.

[0132] definition

[0133] Definitions of specific terms used in this disclosure are provided in this section and also constitute a part of this disclosure.

[0134] The method according to the invention is, for example, a computer-implemented method. For example, all or only some of the steps (i.e., less than the total number of steps) of the method according to the invention can be performed by a computer (e.g., at least one computer). An embodiment of the method implemented by a computer is a purpose of a computer for performing a data processing method. An embodiment of the method implemented by a computer is a method involving computer operation, such that the computer is operated to perform one, more or all of the steps of the method.

[0135] The computer comprises, for example, at least one processor and, for example, at least one memory, in order to (technically) process data, for example electronically and / or optically process data. The processor is, for example, made of a semiconductor substance or composition, for example, at least partially n-type and / or p-type doped semiconductor, for example, at least one of type II, type III, type IV, type V, type VI semiconductor materials, for example (doped) silicon arsenide and / or gallium arsenide. The described calculation steps or determination steps are, for example, performed by a computer. The determination step or calculation step is, for example, a step of determining data within the framework of a technical method (for example, within the framework of a program). The computer is, for example, any type of data processing device, for example, an electronic data processing device. The computer can be a device that is generally regarded as a computer, such as a desktop personal computer, a laptop, a netbook, etc., but can also be any programmable device, such as a mobile phone or an embedded processor. The computer can, for example, include a "sub-computer" system (network), in which each sub-computer represents its own computer. The term "computer" includes cloud computers, such as cloud servers. The term "computer" includes server resources. The term "cloud computer" includes a cloud computer system, which includes, for example, a system of at least one cloud computer, and, for example, includes a plurality of operably interconnected cloud computers, such as a server cluster. Such a cloud computer is preferably connected to a wide area network such as the World Wide Web (WWW) and is located in a so-called cloud of computers all connected to the World Wide Web. Such infrastructure is used for "cloud computing", which describes those computing, software, data access and storage services that do not require the end user to know the physical location and / or configuration of the computer providing a particular service. For example, the term "cloud" is used metaphorically for the Internet (World Wide Web). For example, the cloud provides a computing infrastructure as a service (IaaS). A cloud computer can be used as a virtual host for an operating system and / or a data processing application for executing the method of the present invention. A cloud computer is, for example, provided by Amazon Web Services. TM) provided by Elastic Compute Cloud (EC2). The computer, for example, includes an interface to receive or output data and / or perform analog-to-digital conversion. The data, for example, represents physical properties and / or is generated from technical signals. The technical signal is generated, for example, by a (technical) detection device (for example, a device for detecting a marking device) and / or a (technical) analysis device (for example, a device for performing a (medical) imaging method), wherein the technical signal is, for example, an electrical signal or an optical signal. The technical signal, for example, represents data received or output by the computer. The computer is preferably operably coupled to a display device that allows information output by the computer to be displayed to, for example, a user. An example of a display device is a virtual reality device or an augmented reality device (also known as virtual reality glasses or augmented reality glasses), which can be used as "goggles" for navigation. A specific example of such augmented reality glasses is Google Glass (Google Glass, a trademark brand of Google, Inc.). The augmented reality device or virtual reality device can be used to input information into the computer through user interaction and to display information output by the computer. Another example of a display device is a standard computer monitor, for example comprising a liquid crystal display, operatively connected to a computer for receiving display control data from a computer for generating signals for displaying the image information content on the display device. A specific embodiment of such a computer monitor is a digital light box. An example of such a digital light box is the product 1000x1000 from Brainlab AG. The monitor may also be a handheld portable device such as a smart phone or a personal digital assistant or a digital media player, for example.

[0136] The present invention also relates to a computer program, which includes instructions that, when executed on a computer, cause the computer to perform one or more methods, such as one or more methods described herein, and / or to a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing the program, and / or a computer comprising the above-mentioned program storage medium, and / or a signal wave (physical, such as electrical, such as technically generated) with information representing the program (e.g., the above-mentioned program), such as a digital signal wave, such as an electromagnetic carrier wave, the program for example including code means suitable for performing any or all of the method steps described herein. In one example, the signal wave is a data carrier signal carrying the above-mentioned computer program. The present invention also relates to a computer, which includes at least one processor and / or the above-mentioned computer-readable storage medium and, for example, a memory, wherein the program is executed by a processor.

[0137] Within the framework of the present invention, a computer program element may be embodied by hardware and / or software (this includes firmware, resident software, microcode, etc.). Within the framework of the present invention, a computer program element may take the form of a computer program product, which may be implemented by a computer-usable, e.g. computer-readable data storage medium, which includes computer-usable, e.g. computer-readable program instructions, the "code" or "computer program" embodied in the data storage medium being used on or in conjunction with an instruction execution system. Such a system may be a computer; a computer may be a data processing device comprising means for executing a computer program element and / or a program according to the present invention, e.g. a data processing device comprising a digital processor (central processing unit or CPU) executing the computer program element, and optionally a data processing device comprising a volatile memory (e.g. random access memory or RAM) for storing data used for and / or generated by executing the computer program element. Within the framework of the present invention, a computer-usable, e.g. computer-readable data storage medium may be any data storage medium that may contain, store, communicate, propagate or transmit programs for use on or in conjunction with those instruction execution systems, devices or apparatuses. Computer-usable, for example, computer-readable data storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or communication media such as the Internet. Computer-usable or computer-readable data storage media can even be, for example, paper or other suitable media on which the program can be printed, because the program can be captured electronically, for example, by optically scanning the paper or other suitable media, and then compiled, decoded or otherwise processed in an appropriate manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product described herein and any software and / or hardware form various mechanisms for performing the functions of the present invention in the example embodiments. The computer and / or data processing device can, for example, include a guidance information device, which includes a mechanism for outputting guidance information. The guidance information can, for example, be output to the user visually by a visual indication mechanism (e.g., a monitor and / or a light) and / or audibly by an auditory indication mechanism (e.g., a speaker and / or a digital voice output device) and / or tactilely by a tactile indication mechanism (e.g., a vibration element or a vibration element incorporated into an instrument). For the purpose of this document, a computer is a technical computer, for example comprising technical components such as tangible components, for example mechanical components and / or electronic components. Any device mentioned in this document is a technical device and is for example a tangible device.

[0138] The expression "obtaining data" for example includes (within the framework of the computer-implemented method) a scenario in which data is determined by a computer-implemented method or program. Determining data for example includes measuring a physical quantity and transforming the measured value into data, such as digital data, and / or calculating (such as outputting) the data with the aid of a computer and for example within the framework of the method according to the invention. The "determining" step as described herein for example includes or consists of issuing a command to perform the determination described herein. For example, the step includes or consists of issuing a command to cause a computer (such as a remote computer, such as a remote server, such as in the cloud) to perform the determination. Alternatively or additionally, the "determining" step as described herein for example includes or consists of the following steps: receiving the result data of the determination described herein, such as receiving the result data from a remote computer (such as from a remote computer that causes it to perform the determination). The meaning of "obtaining data" also includes, for example, the following scenario: receiving or retrieving data by (such as inputting) a method or program implemented by a computer, such as from another program, a previous method step or a data storage medium, such as for further processing by a method or program implemented by a computer. The generation of the data to be acquired can, but does not have to be, part of the method according to the invention. Therefore, the expression "obtaining data" can also, for example, mean waiting to receive data and / or receiving data. The received data can be input, for example, via an interface. The expression "obtaining data" can also mean that a method or program implemented by a computer performs some steps in order to (actively) receive or retrieve data from a data source such as a data storage medium (e.g., ROM, RAM, database, hard disk drive, etc.) or via an interface (e.g., from another computer or network). The data obtained by the method or device of the present disclosure, respectively, can be obtained from a database located in a data storage device, which is operably connected to a computer for data transmission between the database and the computer, such as data transmission from a database to a computer. The computer obtains data for use as input in the "determine data" step. The determined data can be output to the same or another database for storage for subsequent use. The database or the database for implementing the disclosed method can be located in a network data storage device or a network server (e.g., a cloud data storage device or a cloud server) or a local data storage device (e.g., a mass storage device operably connected to at least one computer performing the disclosed method). The data can be "ready" by performing an additional step before the acquisition step. According to this additional step, data is generated for acquisition. For example, the data are detected or captured (eg, by an analysis device). Alternatively or additionally, the data are input according to an additional step, such as via an interface. For example, the generated data may be input (eg, into a computer).According to an additional step (which is performed before the acquisition step), the data can also be provided by performing an additional step of storing the data on a data storage medium (such as a ROM, RAM, CD and / or hard disk drive), so that the data are ready within the framework of the method or program according to the invention. Therefore, the step of "acquiring data" can also involve commanding the device to acquire and / or provide the data to be acquired. In particular, the acquisition step does not involve an invasive step, which represents a substantial physical interference with the body, requiring professional medical measures, and even if the required professional care and measures are taken when performed, the body may be exposed to significant health risks. In particular, the step of acquiring data, such as determining data, does not involve a surgical step, in particular does not involve a step of treating the human or animal body with surgery or therapy. In order to distinguish between the different data used by the present method, the data are denoted (i.e. referred to as) "XY data" etc. and are defined according to the information they describe, and then preferably referred to as "XY information" etc.

[0139] The function of the marker is to be detected by a marker detection device (e.g., a camera or an ultrasound receiver, or an analysis device such as a CT or MRI device) in a manner that can determine the spatial position of the marker (i.e., its spatial position and / or alignment). The detection device is, for example, part of a computer-based navigation system and can be a stereoscopic camera that is sensitive to electromagnetic waves within a predetermined wavelength range, such as an infrared wavelength range, or any other wavelength range that can be used to irradiate the marker and can be reflected by the marker. The marker can be an active marker. An active marker can, for example, emit electromagnetic radiation and / or waves within the infrared, visible and / or ultraviolet spectrum. However, the marker can also be passive, i.e., it can, for example, reflect electromagnetic radiation within the infrared, visible and / or ultraviolet spectrum, or it can block X-ray radiation. For this purpose, the marker can be provided with a surface having corresponding reflective properties, or it can be made of metal so as to block X-ray radiation. The marker can also reflect and / or emit electromagnetic radiation and / or waves within the radio frequency range or at ultrasonic wavelengths. The marker preferably has a spherical and / or spheroidal shape and may therefore be referred to as a marker sphere; however, the marker may also have an angular (eg cubic) shape.

[0140] The marker device may for example be a reference star or a pointer or a single marker, or a plurality of (separate) markers, which are then preferably in a predetermined spatial relationship. The marker device comprises one, two, three or more markers, wherein two or more of these markers are in a predetermined spatial relationship. This predetermined spatial relationship is for example known to the navigation system and is for example stored in a computer of the navigation system.

[0141] In another embodiment, the marker device comprises an optical pattern, for example, on a two-dimensional surface. The optical pattern may comprise a plurality of geometric shapes, like circles, rectangles, and / or triangles. The optical pattern may be identified in an image captured by a camera, and the position of the marker device relative to the camera may be determined based on the size of the pattern in the image, the orientation of the pattern in the image, and the distortion of the pattern in the image. This allows relative positions in up to three rotational dimensions and up to three translational dimensions to be determined from a single two-dimensional image.

[0142] The position of the marker device can be determined, for example, by a medical navigation system. If the marker device is attached to an object such as a bone or a medical instrument, the position of the object can be determined based on the position of the marker device and the relative position between the marker device and the object. Determining this relative position is also called registering the marker device and the object. The marker device or object can be tracked, which means that the position of the marker device or object is determined two or more times over time.

[0143] Preferably, atlas data describing (e.g. defining, more particularly representing and / or serving as) the general three-dimensional shape of a body anatomical part is obtained. Thus, the atlas data represents an atlas of a body anatomical part. The atlas is typically composed of a plurality of object general models, wherein these object general models together form a composite structure. For example, the atlas constitutes a statistical model of a patient's body (e.g. a part of the body), which has been generated based on anatomical information collected from a plurality of human bodies, for example, based on medical image data containing images of these human bodies. Thus, in principle, the atlas data represents the results of a statistical analysis of such medical image data of a plurality of human bodies. This result can be output as an image - thus the atlas data contains or is equivalent to the medical image data. This comparison can be performed, for example, by applying an image fusion algorithm, wherein the image fusion algorithm performs image fusion between the atlas data and the medical image data. The comparison result can be a measure of similarity between the atlas data and the medical image data. The atlas data includes image information (e.g., position image information) that can be matched with image information (e.g., position image information) contained in, for example, medical image data (e.g., by applying an elastic or rigid image fusion algorithm) so that, for example, the atlas data is compared with the medical image data in order to determine the position of anatomical structures in the medical image data that correspond to anatomical structures defined by the atlas data.

[0144] Multiple human bodies (whose anatomical structures are used as inputs to generate atlas data) advantageously share common features, such as at least one of gender, age, race, physical measurements (e.g., height and / or weight), and pathological conditions. Anatomical information describes, for example, human anatomical structures and is extracted, for example, from medical image information about the human body. For example, an atlas of the femur may include the femoral head, femoral neck, body, greater trochanter, lesser trochanter, and lower limbs as objects that together constitute a complete structure. For example, an atlas of the brain may include the telencephalon, cerebellum, diencephalon, pons, midbrain, and medulla oblongata as objects that together constitute a complex structure. One application of such an atlas is in medical image segmentation, where the atlas is matched to medical image data and the image data is compared to the matched atlas so that points (pixels or voxels) of the image data are assigned to objects of the matched atlas, thereby segmenting the image data into objects.

[0145] For example, the atlas data includes information about the anatomical part of the body. The information is, for example, at least one of patient-specific, non-patient-specific, indication-specific, or non-indication-specific. Therefore, the atlas data describes, for example, at least one of a patient-specific, non-patient-specific, indication-specific, or non-indication-specific atlas. For example, the atlas data includes movement information indicating the freedom of movement of the anatomical part of the body relative to a given reference (e.g., another anatomical part of the body). For example, the atlas is a multimodal atlas that defines atlas information for multiple (i.e., at least two) imaging modes and includes mappings between atlas information in different imaging modes (e.g., mappings between all modalities), so that the atlases can be used to transform medical image information from its image depiction in a first imaging mode to its image depiction in a second imaging mode different from the first imaging mode, or to compare different imaging modes with each other (e.g., matching or registration).

[0146] In the medical field, imaging methods (also referred to as imaging modalities and / or medical imaging modalities) are used to generate image data (e.g. two-dimensional or three-dimensional image data) of human anatomical structures (e.g. soft tissue, bones, organs, etc.). The term "medical imaging method" is to be understood as meaning (advantageously device-based) imaging methods (e.g. so-called medical imaging modalities and / or radiographic imaging methods), such as computed tomography (CT) and cone beam computed tomography (CBCT, such as volumetric CBCT), x-ray tomography, magnetic resonance tomography (MRT or MRI), conventional x-ray, ultrasound verification and / or ultrasound verification and positron emission tomography. The medical imaging method is performed, for example, by an analysis device. Examples of medical imaging modalities applied by medical imaging methods are: X-ray radiography, magnetic resonance imaging, medical ultrasound or ultrasound, endoscopy, elastic imaging, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques such as positron emission tomography (PET) and single-photon emission computed tomography (SPECT). The image data generated thereby are also referred to as "medical imaging data". The analysis device is used, for example, to generate image data in an imaging method based on an apparatus. The imaging method is used, for example, for medical diagnosis of an anatomical structure of the body to generate an image described by the image data. The imaging method is also used, for example, to detect pathological changes in the human body. However, some changes in the anatomical structure, such as pathological changes in the structure (tissue), may not be detected and may not be visible, for example, in an image generated by the imaging method. A tumor represents an example of a change in the anatomical structure. If a tumor grows, it can be considered to represent an expanded anatomical structure. Such an expanded anatomical structure may not be detected, for example, only a part of the expanded anatomical structure can be detected. For example, early / late stage brain tumors are often visible in MRI scans when contrast agents are used to infiltrate the tumor. MRI scans represent one example of an imaging method. In the case of an MRI scan of such a brain tumor, the signal enhancement in the MRI image (caused by the contrast agent infiltrating the tumor) is believed to represent a solid tumor mass. Therefore, the tumor is detectable and, for example, discernible in an image generated by the imaging method. In addition to these tumors, which are referred to as "enhancing" tumors, it is believed that approximately 10% of brain tumors are not discernible in a scan and are not visible, for example, to a user viewing the image generated by the imaging method. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] In the following, the present invention is described with reference to the accompanying drawings, which give a background description of the present invention and show specific embodiments of the present invention. However, the scope of the present invention is not limited to the specific features disclosed in the context of the accompanying drawings, in which:

[0148] Figure 1 showing neurons of a neural network;

[0149] Figure 2 The basic flow of the method according to the first aspect is shown;

[0150] Figure 3 The basic flow of the method according to the second aspect is shown;

[0151] Figure 4 The basic flow of the method according to the third aspect is shown;

[0152] Figure 5 illustrating the application of the method according to the first to third aspects; and

[0153] Figure 6 is a schematic diagram of a system according to the fifth aspect. DETAILED DESCRIPTION

[0154] Figure 1 Elucidate the structure of a neuron as part of a neural network, such as a convolutional neural network, where inputs are assigned certain weights to be processed by an activation function that generates the neuron's output.

[0155] Figure 2 The basic flow of the method according to the first aspect is described, which starts from step S21, obtaining patient training image data, continues to step S22, which includes obtaining atlas data, and then continues to obtain viewing direction data in step S23. On this basis, step S24 calculates anatomical vector data, and then obtains label data in step S25. Finally, anatomical index data is determined in step S26.

[0156] Figure 3 The basic steps of the method according to the second aspect are shown, where step S31 comprises acquiring individual patient image data, and step 32 determines label relationship data.

[0157] Figure 4 The basic steps of the method according to the third aspect are shown, where step S41 comprises acquiring individual patient image data, and step 42 acquires atlas data. A subsequent step 43 acquires individual viewing direction data, followed by determining individual anatomical vector data in step S44. Additional data is acquired in step S45. Then, step S46 determines label relationship data.

[0158] Figure 5An overview of the application of the method according to the first to third aspects is given. A generic patient model (atlas) 1 and a tracking imaging device 6 are registered to a patient space 4 via a registration 5 between an atlas 1 and a patient space 4 (using, for example, 3D tomographic image data as planning data for a specific patient and elastically registered to the atlas) and via a registration 1 between the patient space 4 and a tracking imaging device 6. Registrations 5 and 13 can be combined into a registration 14 between the atlas 1 and the tracking imaging device 6. The tracking imaging device 6 generates medical images 7 and generates 8 labels 9 for these images, for example according to the atlas 1 or manually. Anatomical vectors are determined by registration 14, which is based on 5 and 13 and the known viewing direction of the tracking imaging device 6.

[0159] During learning, the medical image 7 is input 15 into the learning algorithm 12. The anatomical vectors determined using the atlas 1 are input 2 into the learning algorithm 12. The labels 9 are input 10 into the learning algorithm 12. Optionally, the atlas 1 is input 3 into the learning algorithm 12. Thus, during learning, the relationship between the labels 9 and the anatomical structures depicted by the medical image 7 is determined based on the inputs 2, 10, 15 and optionally 3.

[0160] During testing or use of the learning algorithm, a medical image 7 is input 15 into the learning algorithm 12. Anatomical vectors determined using the atlas 1 are input 2 into the learning algorithm 12. Optionally, the atlas 1 is input 3 into the learning algorithm 12. Labels 9 are determined 11 by the learning algorithm based on the inputs 2 and 15 and optionally based on 3.

[0161] Figure 6 is a schematic diagram of a medical system 61 according to the fifth aspect. The system is generally indicated with reference numeral 61 and comprises a computer 62, an electronic data storage device (e.g. a hard disk) 63 for storing at least data stored by the system according to the fifth aspect. The components of the medical system 1 have the functions and characteristics described above in relation to the fifth aspect of the present disclosure.

Claims

1. A computer-implemented method of training a learning algorithm to determine a relationship between a label indicating a location or type of an anatomical structure in a medical image on the one hand and a label indicating a location or type of the anatomical structure in the medical image on the other hand, the method comprising the following steps: a) acquiring patient training image data, the patient training image data describing digital medical images of anatomical structures of a plurality of patients; b) acquiring atlas data, the atlas data describing a model of a body anatomical part including the anatomical structure, wherein the atlas data is comparable to the medical image data; c) acquiring viewing direction data by tracking an imaging device used to generate a medical image, the viewing direction data describing the viewing direction of the imaging device toward the anatomical structure at a point in time when the medical image is generated using the imaging device; d) determining anatomical vector data based on the viewing direction data and the atlas data, wherein the anatomical vector data describes an anatomical vector, and the anatomical vector describes the viewing direction of the imaging device in the coordinate system of the atlas data; e) acquiring label data describing a label describing a location or type of the anatomical structure in the model of the body anatomical part; and f) determining anatomical indicator data based on the patient training image data and the anatomical vector data and the label data, wherein the anatomical indicator data describes model parameters of a learning algorithm for establishing the relationship between the position or type of the anatomical structure described by the medical image and the label, wherein the anatomical indicator data is determined by inputting the patient training image data and the label data and a subset of the atlas data into a function establishing the relationship, wherein the subset of the atlas data has been determined based on the atlas data and the anatomical vector data.

2. The method according to claim 1, wherein: The medical image is a two-dimensional image.

3. The method according to claim 2, wherein: The medical image is a microscopic image forming part of a video captured with a microscope, wherein the imaging device is a microscope, or an endoscope equipped with a digital camera, or an X-ray device configured to produce a two-dimensional projection image.

4. The method according to claim 3, comprising the steps of: acquiring additional data as a function of the anatomical vector; as well as The anatomical index data are determined by additionally inputting the additional data into the function establishing the relationship.

5. The method according to claim 4, wherein: The additional data comprises the anatomical vector data or consists of the anatomical vector data.

6. The method according to claim 1, wherein: The learning algorithm comprises a machine learning algorithm or consists of the machine learning algorithm.

7. The method according to claim 1, wherein: The learning algorithm comprises a convolutional neural network or consists of the convolutional neural network.

8. The method according to claim 1, wherein: The model parameters define the learnable parameters of the learning algorithm.

9. The method according to claim 8, wherein: The model parameters define the weights of the learning algorithm.

10. A computer-implemented method of determining a relationship between a label indicating a location or type of an anatomical structure in a medical image on the one hand and a label indicating a location or type of the anatomical structure in the medical image on the other hand, the method comprising the steps of: a) acquiring individual patient image data, wherein the individual patient image data is a digital individual medical image describing the anatomical structure of the individual patient; as well as b) determining label relationship data describing the relationship between the label and the anatomical structure in the individual medical image, wherein the label relationship data is determined by inputting the individual patient image data into a function that establishes the relationship between the anatomical structure described by the individual medical image and the label, the function being part of a learning algorithm that has been trained by performing a method according to any one of claims 1 to 9.

11. The method according to claim 10, wherein: additionally determining the label relationship data based on the atlas data by additionally inputting a subset of the atlas data into a function establishing a relationship between the anatomical structures described by the individual medical images and the labels, and The learning algorithm has been trained by additionally carrying out a method according to any one of claims 4 to 9 .

12. The method according to claim 10, wherein: The individual medical images are two-dimensional images.

13. The method according to claim 12, wherein: The individual medical images are microscopic images forming part of a video taken with a microscope, wherein the imaging device is a microscope.

14. The method according to claim 13, wherein: The relative position between the microscope and the individual anatomical vector data for generating the individual patient image data is predetermined and acquired by the method.

15. The method according to claim 10, wherein: The learning algorithm comprises a machine learning algorithm or consists of the machine learning algorithm.

16. The method according to claim 10, wherein: The learning algorithm includes or consists of a convolutional neural network, and the learning algorithm serves as a part of a computational model.

17. The method according to claim 16, wherein: The parameters of the model define the learnable parameters of the learning algorithm.

18. The method according to claim 17, wherein: The parameters of the model define the weights of the learning algorithm.

19. A computer-readable program storage medium, wherein when the program stored thereon is executed on at least one computer, the program causes the at least one computer to execute the method steps of the method according to any one of claims 1 to 18.

20. A system for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image on the one hand and a label indicating a location or type of an anatomical structure in the medical image on the other hand, comprising: a) a computer having at least one processor and a memory, wherein the program, when executed on the computer or loaded into the memory of the computer, causes the at least one processor to perform the method steps of the method according to any one of claims 10 to 18; b) at least one electronic data storage device storing individual patient image data and causing the computer to perform the method of claim 10 whenever the program is run on the at least one processor or loaded onto the memory of the computer, the electronic data storage device storing atlas data, individual viewing direction data, individual anatomical vector data and additional data; and c) The computer readable program storage medium according to claim 19, Wherein the computer is operably coupled to - at least one electronic data storage device for retrieving the individual patient image data from the at least one electronic data storage device and causing the computer to perform the method according to claim 10, retrieving the atlas data, the individual viewing direction data, the individual anatomical vector data and the additional data from the at least one electronic data storage device and for storing at least the label relationship data in the at least one electronic data storage device, as long as the program is run on the at least one processor or loaded on the memory of the computer; and - The computer readable program storage medium is used to obtain architecture data defining model parameters and learning algorithms from the program storage medium.

Images