Medical image analysis using machine learning and anatomical vectors

Abstract

The present invention discloses a computer-implemented method comprising registering a tracking imaging device (such as a microscope) with a known viewing direction and an atlas to a patient space so that a transformation can be established between the atlas space and a reference frame to define the location of the patient's anatomical structures in the image. Labels are associated with certain components of the image and are input into a learning algorithm, such as a machine learning algorithm, such as a convolutional neural network, together with the medical image and anatomical vectors and, for example, the atlas, to train the learning algorithm to automatically segment patient images generated using the tracking imaging device. The trained learning algorithm then allows for efficient segmentation and / or labeling of the patient images without having to register the patient images to the atlas each time, thereby saving computational effort.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for training a learning algorithm for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, a method for segmenting and / or labeling a medical patient image using the trained learning algorithm, a corresponding computer program, a computer-readable storage medium storing such a program, a computer executing such a program, and a system comprising an electronic data storage device and the above-mentioned computer. Background Art

[0002] Anatomical images can be used to segment or label medical patient images. This requires registering the patient images to atlas data, which involves a lot of computational work.

[0003] Patent document US 2017 / 0213339 A1 discloses a method and system for segmenting medical images, which use a learning algorithm as an anatomical classifier for aligning patient images with atlas images.

[0004] Patent document US 2019 / 0320934 A1 discloses automatic sequence prediction for medical imaging sessions including a self-assessment mechanism. An initial scout sequence is performed on a patient or subject.

[0005] Validate initial scout sequence. Perform simplified acquisition protocol. Validate simplified acquisition protocol. Perform additional sequences. Sequences can also be configured using deep learning-based reasoning to select appropriate settings and procedures for the next step based on analysis of previous scans.

[0006] Patent document US 2017 / 0330325 A1 discloses a matching transformation for matching a patient image set of images of anatomical structures of a patient's body with an atlas image set of images of general anatomical structures including anatomical atlas elements. Atlas spatial information containing spatial information about general anatomical structures and element representation information is obtained. The element representation information describes obtaining a representation data set containing information about the representation of multiple atlas elements in the atlas image to be determined, and also describes a determination rule for determining each representation data set of each atlas element according to different respective parameter sets. Patient data is obtained by acquiring a patient image set and parameter sets respectively associated with images of the patient image set. The matching transformation is determined by matching images associated with the same parameter set with each other.

[0007] It is an object of the present invention to provide a method for more efficient segmentation and / or labeling of medical patient images.

[0008] Various aspects, examples and exemplary steps of the invention and their embodiments are disclosed below.Different exemplary features of the invention can be combined according to the invention whenever technically suitable and feasible. Summary of the invention

[0009]

[0013] The following gives a brief description of specific features of the present invention, which should not be construed as limiting the present invention to only the features or combinations of features described in this section.

[0010] The method of the present disclosure comprises registering a tracking imaging device (such as a microscope) with a known viewing direction and an atlas to a patient space so that a transformation can be established between the atlas space and a reference frame for defining locations in an image of an anatomical structure of the patient. Labels are associated with certain components of the image and are input into a learning algorithm, such as a machine learning algorithm, such as a convolutional neural network, together with the medical image and anatomical vectors and, for example, the atlas, to train the learning algorithm for automatically segmenting patient images generated using the tracking imaging device. The trained learning algorithm then allows the patient images to be effectively segmented and / or labeled without having to segment the patient images each time using the registered atlas, thereby saving computational effort.

[0011] In this summary, a description of the general features of the invention is given, for example, by reference to possible embodiments of the invention.

[0012] In general, to achieve the above-mentioned purpose, the first aspect of the present invention provides a computer-implemented method (e.g., a medical method) for training a learning algorithm for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image. The method according to the first aspect includes executing the following exemplary steps performed by at least one processor of at least one computer (e.g., at least one computer as part of a navigation system) on at least one processor.

[0013] In a (e.g., first) exemplary step of the method according to the first aspect, patient training image data is acquired, the patient training image data describing digital medical images of anatomical structures of multiple patients. For example, the medical image is a two-dimensional image, such as a microscope image, for example a portion of a video taken with a microscope, wherein the imaging device is, for example, a microscope. For example, the patient training image data has been generated from images or videos taken using an imaging device that generates two-dimensional images (such as a digital microscope, a camera, or an endoscope equipped with a digital camera) or using an X-ray device that generates or is configured to generate two-dimensional projection images. However, in other examples, the patient training image data can be generated by applying any other applicable imaging modality (e.g., magnetic resonance tomography, X-ray computed tomography, radiography, or ultrasound imaging) to the anatomical structure.

[0014] In an (e.g. second) exemplary step of the method according to the first aspect, atlas data is acquired, which describes an anatomical model of a body anatomical part including an anatomical structure. The anatomical model is, for example, a three-dimensional synthetic anatomical model defined by, for example, tissue classes. For example, the tissue classes have been generated from, for example, medical image data of a body anatomical part for a plurality of patients.

[0015] In an exemplary step (e.g., the third) of the method according to the first aspect, viewing direction data is obtained, which describes the viewing direction of the imaging device toward the anatomical structure at a point in time for generating a medical image. For example, the viewing direction has been determined by tracking the imaging device with a tracking system, the working principle of which is based on marker tracking (i.e., optical detection of a retroreflective marker, which is attached to the imaging device in a predetermined and known relationship relative to its viewing direction), video tracking, or electromagnetic tracking. The viewing direction can also be based on or defined by the geometry of the imaging device, and can be defined based on optical devices, such as the field of view or focal axis of the imaging device, or for example, the focus (focal point), such as the position of the focal spot of the imaging device. The viewing direction, for example, includes a direction perpendicular to the imaging plane. The viewing direction, for example, includes or is defined by the camera position and camera orientation. In addition, the viewing direction data, for example, includes information defining the focal length, zoom, or magnification of the imaging device. Additionally or alternatively, it, for example, includes information defining the position of an edge or corner of an image or field of view (e.g., relative to the imaging device position). For an imaging device using X-rays and comprising a substantially flat X-ray detector, the viewing direction data may, for example, comprise or be defined by a normal to a detector region, or it may, for example, comprise information defining a center of a detector region or a center of a region defined, for example, by collimator blades close to the X-ray detector or X-ray source. In one example, the viewing direction is generated by imaging a radiopaque marker using X-rays having a predetermined (e.g. known) geometry and / or a predetermined (e.g. known) position relative to an anatomical structure, and comparing the appearance of the marker in at least one resulting X-ray image with the predetermined (e.g. known) geometry or position of the anatomical structure, respectively. Where features of the invention are mentioned in the present disclosure in the context of video images, they may likewise be combined with any other type of imaging modality, such as still image digital photography, radiography, DRR, or a three-dimensional imaging modality, such as X-ray computed tomography, magnetic resonance tomography, or ultrasound tomography.

[0016] In an exemplary step (e.g., the fourth) of the method according to the first aspect, anatomical vector data is determined based on viewing direction data and atlas data, wherein the anatomical vector data describes an anatomical vector obtained by transforming the viewing direction into a reference system in which a position in the anatomical model is defined.

[0017] This transformation from viewing direction data to anatomical vector data is performed, for example, by the following steps a) to c):

[0018] a) transforming the coordinates of the position and orientation of the tracking imaging device (e.g., defined in the coordinate system of the tracking system, such as a stereo tracking camera relative to an optical tracking system) into the same coordinate system as the patient (e.g., via optical or other tracking markers attached to the patient), such as the patient's intraoperative coordinate system.

[0019] b) transforming positions in the planning image data (e.g. a computed tomography image, e.g. an X-ray or magnetic resonance tomography from the patient) into a patient coordinate system of the patient (e.g. using artificial markers or natural landmarks, e.g. radiopaque, in the preoperative image data). The coordinate system of the planning image data is e.g. defined by the coordinate system of an imaging device (e.g. a computed tomography scanner or a magnetic resonance imaging scanner) used to acquire the planning image data. The planning image data is acquired e.g. preoperatively.

[0020] c) The positions that have been transformed into the patient coordinate system are then transformed into the coordinate system of the anatomical atlas using a registration of the planning image data to the atlas, which registration may be, for example, a rigid transformation (e.g., an affine transformation) or an elastic transformation (e.g., a deformation of the image data). After the registration, the deformed image data is consistent with at least a portion of the atlas, or at least a portion of the deformed atlas is consistent with the image data.

[0021] As a result of the above transformation steps a) to c), all coordinates given in any coordinate system (e.g. coordinates in preoperative image data, coordinates of the viewing direction of the imaging device, coordinates of the tracking system and all tracking devices tracked by the tracking system, and patient coordinates) can be expressed in an atlas coordinate system that is not specific to an individual patient. This transformation process is also called "registration"; the tracking imaging device is thereby registered to the atlas coordinate system. The anatomical vector data, for example, includes viewing direction data of the imaging device expressed in the atlas coordinate system, for example viewing direction coordinates transformed into coordinates in the atlas coordinate system.

[0022] In a (eg, fifth) exemplary step, label data describing labels representing locations or types of anatomical structures in the anatomical model is acquired.

[0023] In an exemplary step (e.g., the sixth) of the method according to the first aspect, anatomical indicator data is determined based on patient training image data, anatomical vector data, and label data, wherein the anatomical indicator data describes a model parameter (e.g., weight) of a learning algorithm for establishing a relationship (e.g., relative position or assignment) between the position or type of the anatomical structure described by the medical image and the label, wherein the anatomical indicator data is determined by inputting the patient training image data and the label data into a relationship establishment function. For example, the learning algorithm includes or is composed of a machine learning algorithm. For example, the learning algorithm includes or is composed of a convolutional neural network. For example, the model parameters define learnable parameters of the learning algorithm, such as weights. For example, the anatomical indicator data is determined by additionally inputting a subset of the atlas data that has been determined based on the atlas data and the anatomical vector data into the relationship establishment function. The subset is, for example, a real subset, i.e., has fewer elements, i.e., less data, than the atlas data.

[0024] In an example of the method according to the first aspect, additional data is obtained as a function of the anatomical vector. The anatomical indicator data is then determined by additionally inputting the additional data into the relationship building function. For example, the additional data includes or consists of the anatomical vector data.

[0025] In a second aspect, the present invention is directed to a computer-implemented method (e.g., a medical method) for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image. The method according to the second aspect includes executing the following exemplary steps performed by at least one processor of at least one computer (e.g., at least one computer as part of a navigation system) on at least one processor.

[0026] In a (e.g. first) exemplary step of the method according to the second aspect, individual patient image data is acquired, the individual patient image data describing a digital individual medical image of the anatomical structure of the individual patient. For example, the individual medical image has been generated using the same imaging modality as used to generate the patient training image data.

[0027] In an (e.g. second) exemplary step of the method according to the second aspect, label relationship data is determined, which label relationship data describes the relationship (e.g. relative position or distribution) between the label and the anatomical structure in the individual medical image, wherein the label relationship data is determined by inputting the individual patient image data into a function for establishing the relationship between the anatomical structure described by the individual medical image and the label, which function, as long as it includes only inputting the patient training image data and the label data as training data into the relationship establishment function, is part of a learning algorithm that has been trained by executing the method according to the first aspect.

[0028] In a third aspect, the present invention is directed to a computer-implemented method (e.g., a medical method) for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image. The method according to the third aspect includes executing the following exemplary steps performed by at least one processor of at least one computer (e.g., at least one computer as part of a navigation system) on at least one processor.

[0029] In a (e.g. first) exemplary step of the method according to the third aspect, individual patient image data is acquired, the individual patient image data describing a digital individual medical image of the anatomical structure of the individual patient. For example, the individual medical image has been generated using the same imaging modality as used to generate the patient training image data. The individual medical image is a two-dimensional image, such as a microscope image, e.g. a portion of a video taken with a microscope, wherein the imaging device is, e.g., a microscope.

[0030] In an (e.g. second) exemplary step of the method according to the third aspect, atlas data is obtained, which describes an anatomical model of an anatomical part of a body including an anatomical structure. The above description of the anatomical model provided by the method according to the first aspect is also applicable to the anatomical model used in the method according to the third aspect.

[0031] In a (e.g. third) exemplary step of the method according to the third aspect, individual viewing direction data are acquired, the individual viewing direction data describing the viewing direction of the imaging device towards the anatomical structure at a point in time for generating the individual medical image. The viewing direction has been determined, for example, by tracking the imaging device with a tracking system, the working principle of which is based on marker tracking (i.e. optically detecting a retroreflective marker, which is attached to the imaging device in a predetermined and known relationship with respect to its viewing direction), video tracking or electromagnetic tracking.

[0032] In a (e.g. fourth) exemplary step of the method according to the third aspect, individual anatomical vector data is determined based on the individual viewing direction data and the atlas data, wherein the anatomical vector data describes an anatomical vector resulting from transforming the viewing direction into a reference system in which a position in the anatomical model is defined. For example, the relative position between an imaging device (e.g. a microscope) used to generate the individual patient image data and the individual anatomical vector data is predetermined (e.g. known) and is obtained, for example, by the above method.

[0033] In a (eg fifth) exemplary step of the method according to the third aspect, additional data is acquired, the additional data being a function of the individual anatomical vectors. For example, the additional data comprises or consists of the individual anatomical vector data.

[0034] In an (e.g. sixth) exemplary step of the method according to the third aspect, label relationship data is determined, the label relationship data describing the relationship (e.g. relative position or assignment) between the label and the anatomical structure described by the individual medical image, wherein the label relationship data is determined by inputting the individual patient image data and the additional data into a relationship establishment function between the position or type of the anatomical structure in the individual medical image and the label, the function being part of a learning algorithm that has been trained by executing the method according to the first aspect, as long as it includes inputting only the patient training image data and the label data and the additional data as training data into the relationship establishment function. The function used to generate the anatomical vector of the acquired additional data is the same as the function used to generate the anatomical vector of the additional data input into the relationship establishment function for determining the anatomical indicator data.

[0035] In an example of the method according to the third aspect, the additional data comprises or consists of individual anatomical vector data and the learning algorithm has been trained by additionally inputting the additional data into the relationship building function.

[0036] In an example of the method according to the third aspect, label relationship data is additionally determined based on the atlas data by additionally inputting a subset of the atlas data into a function for establishing a relationship between an anatomical structure described by an individual medical image and a label, and a learning algorithm has been trained by additionally inputting a subset of the atlas data determined based on the atlas data and the anatomical vector data into the relationship establishment function. The subset is, for example, a real subset, i.e., has fewer elements, i.e., less data, than the atlas data.

[0037] In the method according to the second aspect and the third aspect, the learning algorithm, for example, includes or consists of a machine learning algorithm (e.g., a convolutional neural network). In the method according to the second aspect and the third aspect, the model parameters define learnable parameters of the learning algorithm, such as weights.

[0038] In one example of the method according to the first aspect, the second aspect and the third aspect, the learning algorithm may be a random forest algorithm. See Antonio Criminisi, Jamie Shotton and Ender Konukoglu, Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning, 2011, see https: / / www.microsoft.com / zh-us / research / wp-content / uploads / 2016 / 02 / CriminisiForests_FoundTrends_2011.pdf, the explanation of random forest is as follows:

[0039] Random forests or ensemble learning methods for classification or regression work by constructing a large number of decision trees during training and outputting the pattern of classes (classification) or the average prediction of the individual trees (regression).

[0040] The basic building block of a random forest is a single decision tree. A decision tree is a set of questions organized in a hierarchical manner and represented graphically as a tree. A decision tree estimates an unknown property ("label") of an object by asking consecutive questions about its known properties (so-called "features"). The next question to be asked depends on the answer to the previous question, and this relationship is represented graphically as a path through the tree that the object follows. Decisions are then made based on the terminal nodes on the path (so-called "leaf nodes"). Each question corresponds to an internal node of the tree (a so-called "split node").

[0041] Each split node has a so-called test function associated with it. The test function at a split node j is formulated as a function with binary output:

[0042]

[0043] Among them, 0 and 1 can be interpreted as "false" and "true" respectively. Represents the parameters of the test function at the jth split node.

[0044] v is therefore the vector represents the current object ("data point"), where the component x i Represents certain attributes (features) of data points, all of which form a feature space

[0045] In its simplest form, the test function is a linear model that selects a feature axis in the feature space and classifies each data point according to whether the value of the respective feature is below or above a learnable threshold. For other more complex forms, non-linear test functions are possible.

[0046] To train a decision tree, a set of training data points is used, for which the features and the required labels are known. The goal of the training is to automatically learn a suitable test function at all split nodes that is best suited to determine the label from the features of the data point. Later, this trained decision tree can be evaluated for new data points with unknown labels by sending the data points through a tree trained on its features.

[0047] To understand the training process, it is helpful to represent subsets of training points as being associated with different branches. For example, S1 represents the subset of training points that reach node 1 (nodes are numbered starting from 0 in breadth-first order to the root F), while Represents a subset of the left and right children going to node 1.

[0048] Training is responsible for selecting a test function h(v,θ) associated with each split node (indexed by j) by optimizing a chosen objective function defined on the available training set. j ) types and parameters.

[0049] The optimization of the split function is performed in a greedy manner. At each node j, according to the introduction of the training set S j subset, learn to convert S j The "best" split is and This problem is formulated as the maximization of the objective function at the node:

[0050]

[0051] in

[0052]

[0053]

[0054]

[0055] As mentioned above, the symbol S j , represents the set of training points before and after segmentation. The objective function is in abstract form here. Its precise definition and the meaning of "best" depend on the task at hand (e.g., whether it is supervised, continuous or discrete output). For example, for binary classification, the term "best" can be defined as the segmentation of the training subset S j , so that the resulting child nodes are as pure as possible, that is, only contain training points of a single class. In this case, the objective function can be defined as information gain, for example.

[0056] During training, the structure (shape) of the tree also needs to be optimized. Training starts from the root node j=0, where the optimal split parameters are found as described before. Therefore, two child nodes are constructed, each receiving a different disjoint subset of the training set. This process is then applied to all newly constructed nodes, and the training phase continues. The structure of the tree depends on how and when to decide to stop growing the various branches of the tree. Different stopping criteria can be applied. For example, the tree is usually stopped when a maximum number of levels D is reached. Alternatively, a maximum value can be applied. In other words, it stops when the attributes of the training points in the leaf nodes are similar to each other. The tree growth may also stop when the node contains too few training points. In terms of generalization, it has been shown that avoiding growing the entire tree has a positive effect.

[0057] During training, randomness is injected into the tree: when training at the jth node, instead of optimizing over the entire parameter space of the test function, only a small random subset of the parameter values ​​is used. Therefore, under the randomness model, the training tree is completed by optimizing each split node j:

[0058]

[0059] Due to this random setting, it is later possible to train multiple decision trees in parallel, each utilizing a different set of attributes from the data points.

[0060] At the end of the training phase, we obtain: (i) the (greedy) optimal weak learner associated with each node; (ii) the learned tree structure; and (iii) a different set of training points at each leaf.

[0061] After training, each leaf node is still associated with a subset of the (labeled) training data. During testing, a previously unseen point traverses the tree until it reaches a leaf. Since the split nodes act on features, an input test point may end up in a leaf associated with a training point similar to itself. Therefore, it is reasonable to assume that the associated label must also be similar to the labels of the training points in that leaf. This justifies using the label statistics collected in that leaf to predict the label associated with the input test point.

[0062] In the most general sense, leaf statistics can be gathered using the posterior distribution:

[0063] p(c|v) and p(y|v),

[0064] where c and y represent discrete or continuous labels, respectively. v is the data point tested in the tree, and the conditional representation distribution depends on the fact that the particular leaf node is reached by the test point. Different leaf predictor variables can be used. For example, in the discrete case, the maximum a posteriori (MAP) estimate can be obtained as c*=arg max c p(c|v).

[0065] Based on the construction principle of the decision tree mentioned above, we can now enter the decision forest, also known as the random forest:

[0066] A random decision forest is a set of randomly trained decision trees. The key aspect of the forest model lies in the fact that its constituent trees are randomly different from each other. This leads to decorrelation between individual tree predictions, which in turn leads to improved generalization and robustness.

[0067] In a forest of T trees, use the variable to index each constituent tree. All trees are trained independently (and possibly in parallel). During testing, each test point v is pushed into all trees simultaneously (starting from the root) until it reaches the corresponding leaf. Tree testing can also usually be done in parallel, thus achieving high computational efficiency on modern parallel CPU or GPU hardware. All tree predictions can be merged into a single forest prediction by a simple averaging operation. For example, in classification:

[0068]

[0069] p t (c|v) represents the posterior distribution obtained by the tth tree. Alternatively, the tree outputs can be multiplied together (although the trees are not statistically independent):

[0070]

[0071] The partition function Z is used to ensure that the probabilities are normalized.

[0072] In one example of the method according to the first aspect, the second aspect and the third aspect, the learning algorithm may be a convolutional neural network. Figure 1 A convolutional neural network is explained as an example of a machine learning algorithm used in conjunction with the presently disclosed invention.

[0073] A convolutional network (also called a convolutional neural network or CNN) is an example of a neural network for processing data with a known grid-like topology. Examples include time series data (which can be viewed as a one-dimensional grid sampled at regular time intervals) and image data (which can be viewed as a two-dimensional or three-dimensional grid of pixels). The name "convolutional neural network" indicates that the network employs the mathematical operation of convolution. Convolution is a linear operation. Convolutional networks are simple neural networks that use convolution instead of general matrix multiplication in at least one of their layers. There are many variations of the convolution function, which are widely used in neural networks in practice. In general, the operation used in convolutional neural networks does not correspond exactly to the definition of convolution used in other fields (such as engineering or pure mathematics).

[0074] The main component of a convolutional neural network is an artificial neuron. Figure 1 is an example of a single neuron depicted in Figure 1. The node in the middle represents the neuron, which accepts all inputs (x1,…,x n ), and multiply them by their specific weights (w1,…,w n ). The importance of the input depends on its weight value. The addition of these calculated values ​​is called the weighted sum and will be plugged into the activation function. The weighted sum z is defined as:

[0075]

[0076] The bias b is a value that is independent of the input and modifies the boundaries of the threshold. The resulting value is processed by the activation function, which decides whether to pass the input to the next neuron.

[0077] CNNs usually take 1st or 3rd order tensors as their input, for example, an image with H rows, W columns, and 3 channels (R, G, B color channels). However, CNNs can process higher order tensor inputs in a similar way. The input then continues to undergo a series of processing. A processing step is usually called a layer, which can be a convolutional layer, a pooling layer, a normalization layer, a fully connected layer, a loss layer, etc.

[0078] The details of these layers are described in the following sections.

[0079]

[0080] Equation 5 above shows how CNN operates layer by layer in the forward pass. The input is x 1 , usually an image (1st or 3rd order tensor). The parameters involved in the first layer of processing are collectively referred to as tensor w i The output x of the first layer 2 , which also serves as the input for the second layer of processing. This process continues until all layers in the CNN are processed, and its output x L. But add an additional layer for backward error propagation, which is a method for learning good parameter values ​​in CNN. Suppose the problem at hand is an image classification problem with class C. A common strategy is to convert x L Output as a C-dimensional vector whose i-th entry encodes the prediction (x 1 The posterior probability of comes from the i-th class). In order to make x L Become a probability mass function, and the processing in the (L-1)th layer can be set to x L-1 Softmax transformation (comparing distance measures and data transformation nodes). In other applications, the output x L can have other forms and interpretations. The last layer is the loss layer. Assume t is the input x 1 The corresponding target value (ground truth) of , then the cost or loss function can be used to measure the CNN prediction x L The difference between t and the target t. It should be noted that some layers may not have any parameters, i.e., for some i, wi may be empty.

[0081] In the example of CNN, ReLu is used as the activation function of the convolutional layer, while the softmax activation function provides information to give a classification output. The following section will explain the purpose of the most important layers.

[0082] The input image is input to the feature learning part which includes layers of convolution and ReLu, followed by layers including pooling, which are followed by further pairwise repetitions of layers of convolution and ReLu and pooling. The output of the feature learning part is input to the classification part which includes layers for flattening, fully connected and maximum softening.

[0083] In a convolutional layer, multiple convolution kernels are usually used. Assuming that D kernels are used and the spatial span of each kernel is H×W, all kernels are denoted as f. f is Similarly, the index variables 0≤i<H, 0≤j<W, 0≤dd, and 0≤d<D are used to identify specific elements in the kernel. It should also be noted that the kernel set f is consistent with the above symbol w L The same objects are referenced. The notation is changed slightly to simplify the derivation. It is also clear that the kernel remains the same even when the mini-batch strategy is used.

[0084] As long as the convolution kernel is larger than 1 × 1, the spatial amplitude of the output is smaller than the spatial amplitude of the input. Sometimes it is necessary for the input and output images to have the same height and width, and a simple padding trick can be used.

[0085] For each input channel, if padding (i.e. insertion) is done above the first row Rows, fill below the last row (i.e. insert) Rows and fill Column to the left of the first column, fill Columns to the right of the last column, the size of the convolution output will be Hl×Wl×D, that is, it has the same spatial amplitude as the input. is a floor function. The elements of padded rows and columns are usually set to 0, but other values ​​are possible.

[0086] Stride is another important concept in convolution. The kernel is convolved with the input at every possible spatial amplitude, which corresponds to a stride of s = 1. However, if s > 1, each movement of the kernel skips s-1 pixel positions (i.e., a convolution is performed every s pixels in the horizontal and vertical directions).

[0087] In this section, we consider the simple case where the stride is 1 and no padding is used. There is y (or x l+1 ), where H l+1 =H l -H+1,W l+1 =

[0088] W l -W+1,D l+1 = D. In precise mathematics, the convolution process can be expressed as an equation:

[0089]

[0090] For all 0≤d≤D=D l+1 , and satisfy 0≤i l+1 <H l –H+1=H l+1 ,0≤j l+1 <W l –W+1=W l+1 Any spatial position (i l+1 ,j l+1 ) Repeat Equation 15. In this equation, It refers to the triple (i l+1 +i,j l +1 +j,d l ) indexed by x l elements. The bias term d is usually added to In order to make the description clearer, this term will not be described in detail.

[0091] Pooling functions replace the network output at a certain location with summary statistics of nearby outputs. For example, the max pooling operation reports the maximum output within a rectangular neighborhood of a table. Other popular pooling functions include the average of a rectangular neighborhood, the L2 norm of a rectangular neighborhood, or a weighted average based on the distance to the center pixel. In all cases, pooling helps make the representation approximately invariant to small translations of the input. Translation invariance means that the value of the pooled output will not change if the input is translated by a small amount.

[0092] Since pooling aggregates responses over an entire neighborhood, it is possible to use fewer pooling units than detector units by reporting summary statistics for regions that are k pixels apart instead of one pixel. This improves the computational efficiency of the network because the next layer has approximately k times fewer inputs to process.

[0093] Assume that the CNN model w has been learned 1 ,...,w L-1 If we know all the parameters of , we can use the model to make predictions. Prediction involves only running the CNN model in the forward direction, that is, in the direction of the arrow in Equation 1. Take the image classification problem as an example. From the input x 1 To begin, pass it through the first layer (with parameter w 1 ) and obtain x 2 . In turn, x 2 Pass it to the second layer, and so on. Finally, after receiving Its estimated x 1 The posterior probability belongs to class C. The CNN prediction output can be:

[0094]

[0095] The question at this point is: how to learn the model parameters?

[0096] As in many other learning systems, the parameters of the CNN model are optimized to minimize the loss z, i.e., we want the predictions of the CNN model to match the ground truth labels. Suppose we are given a training example x 1 The training process involves running the CNN network in two directions. First, the network is run in a forward pass to obtain x L , to use the current CNN parameters to reach a prediction. Instead of outputting the prediction, we need to compare the prediction with the corresponding 1 The target t is compared, that is, the forward pass is continued until the last loss layer. Finally, the loss z is obtained. The loss z is a supervision signal that guides how to correct (update) the parameters of the model.

[0097] There are several algorithms for optimizing loss functions, and CNNs are not limited to a specific algorithm. An example algorithm is called stochastic gradient descent (SGD). This means that the parameters are updated by using gradients estimated from a (usually) small subset of the training examples.

[0098]

[0099] In Equation 4, the ← symbol implicitly indicates the parameter w (of layer i) i Update from time t to t+1. If the time index t is used explicitly, the equation will be written as:

[0100]

[0101] In Equation 4, the partial derivative Measure z relative to w i The growth rate of the different dimensions of . This partial derivative vector is called a gradient in mathematical optimization. Therefore, in w i In a small local area near the current value of , move w in the direction determined by the gradient i The target value Z will be increased. To minimize the loss function, w should be updated in the opposite direction of the gradient i . This update rule is called gradient descent.

[0102] However, if you move too far in the direction of the negative gradient, the loss function may increase. Therefore, in each update, the parameters are only changed by a small fraction of the negative gradient (controlled by η (learning rate)). It is common to set η>0 to a small number (e.g. η=0.001). If the learning rate is not too high, then the loss function is based on x 1 An update will make the loss smaller for this particular training example. However, it is likely to make the loss larger for some other training examples. Therefore, all training examples need to be used to update the parameters. When all training examples have been used to update the parameters, it is said that a learning cycle has been processed. Typically a learning cycle will reduce the average loss of the training set until the learning system fits the training data. Therefore, the gradient descent update learning cycle can be repeated and terminated at a certain point to obtain the CNN parameters (for example, it can be terminated when the average loss of the validation set increases).

[0103] The partial derivatives of the last layer are easy to compute. L In the parameter w L is directly connected to z under the control of , so it is easy to calculate Only when w L This step is only required if it is not empty. Similarly, it is easy to calculate For example, if we use the squared L2 loss, then is empty, and

[0104] In fact, for each layer, two sets of gradients are computed: with respect to the layer parameters w i The partial derivative of z, and the input x of this layer i As shown in Equation 4, the term Can be used to update the parameters of the current (i-th layer). It can be used to update parameters backwards, for example, to update to the (i-1)th layer. The intuitive explanation is: x i is the output of the (i-1)th layer, and Should change x i In order to reduce the loss function. Therefore, it can be regarded as layer by layer The part of the “wrong” supervision information that is propagated backward from z to the current layer. Therefore, we can continue the backpropagation process and use The error is back-propagated to the (i-1)th layer. This layer-by-layer backward update procedure can greatly simplify learning CNN.

[0105] Take the i-th layer as an example. When updating the i-th layer, the back propagation process of the (i+1)-th layer must have been completed. That is, the term and Both are stored in memory and are available for use. The task at this point is to calculate and Using the chain rule, we get:

[0106]

[0107]

[0108] Since it has been calculated and store it in memory, so only matrix reshaping operations (vec) and an additional transpose operation are required to obtain This is the first term on the right hand side (RHS) of both equations. As long as we can calculate and The expected value (the left side of both equations) can be easily obtained.

[0109] and Compared with direct calculation and Much easier, because x i By taking the parameter w i The function of x i+1 Directly related.

[0110] In the context of neural networks, activations act as transfer functions between neuron inputs and outputs. They define under which conditions a node is activated, i.e., an input value is mapped to an output which in turn serves as one of the inputs to subsequent neurons in the hidden layer. There are a large number of different activation functions with different characteristics.

[0111] The loss function quantifies how well the algorithm models the given data. In order to learn from the data and change the weights of the network, the loss function must be minimized. In general, a distinction can be made between regression loss and classification loss. In classification, one tries to predict an output from a limited set of categorical values ​​(class labels), while regression, on the other hand, deals with predicting continuous values.

[0112] In the following mathematical formula, the following parameters are defined as:

[0113] n is the number of training examples;

[0114] i is the i-th training example in the dataset;

[0115] ·y i is the ground truth label of the ith training example;

[0116] · is the prediction for the ith training example.

[0117] The most common setting for classification problems is the cross entropy loss. It increases as the predicted probability deviates from the actual label. The logarithm of the actual predicted probability is multiplied by the ground truth class. The important aspect is that the cross entropy loss severely penalizes confident but wrong predictions. The mathematical equation can be described as:

[0118]

[0119] A typical example of regression loss is the mean squared error or L2 loss. As the name suggests, the mean squared error is the average of the squared differences between the predicted values ​​and the actual observed values. It only concerns the average error magnitude, regardless of their direction. However, due to the squaring, predictions far from the actual values ​​suffer from severe flaws compared to predictions with smaller deviations. In addition, MSE has nice mathematical properties that make it easier to calculate the gradient. Its equation is as follows:

[0120]

[0121] For information on the capabilities of convolutional neural networks, see the following papers:

[0122] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning, chapter convolutional networks, 2016, available at http: / / www.deeplearningbook.org;

[0123] J. Wu, “Introduction to convolutional neural networks”, available at https: / / pdfs.semanticscholar.org / 450c / a19932fcef1ca6d0442cbf52fec38fb9d1e5.pdf;

[0124] “Common loss functions in machine learning”, available at https: / / towardsdatascience.com / common-loss-functions-in-machine-learning-46af0ffc4d23, last accessed on 2019-08-22;

[0125] Imagenet classification with deep convolutional neural networks by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Available at http: / / papers.nips.cc / paper / 4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf.

[0126] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks”, available at https: / / arxiv.org / pdf / 1506.01497.pdf;

[0127] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, Convolutional posemachines, available at https: / / arxiv.org / pdf / 1602.00134.pdf;

[0128] Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully convolutional networks for semantic segmentation,” available at https: / / www.cv-foundation.org / openaccess / content_cvpr_2015 / papers / Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf.

[0129] In a fourth aspect, the present invention is directed to a program which, when run on or loaded onto a computer, causes the computer to execute the method steps of the methods according to the first to third aspects, and / or a (e.g., non-transitory) program storage medium storing the above-mentioned program, or a program storage medium storing data defining model parameters and the architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or a data carrier signal carrying the above-mentioned program, and / or a data carrier signal carrying data defining model parameters and the architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or a data stream carrying the above-mentioned program, and / or a data stream carrying data defining model parameters and the architecture of a learning algorithm that has been trained by executing the method according to the first aspect, and / or at least one computer comprising at least one processor and a memory, wherein the above-mentioned program runs on at least one processor or is loaded into the memory of the computer.

[0130] Alternatively or additionally, the present invention may relate to a signal wave (e.g., physically generated by technical means, such as electricity) carrying information representing a program, such as the above-mentioned program, such as a digital signal wave, such as an electromagnetic carrier wave, which includes, for example, a code mechanism suitable for executing any or all steps of the method according to the first aspect. In one example, the signal wave is a data carrier signal carrying the above-mentioned computer program. The computer program stored on the disk is a data file, and when the file is read and transmitted, the file becomes a data stream in the form of a (e.g., physically generated by technical means, such as electricity) signal. The signal can be implemented as a signal wave, such as the electromagnetic carrier wave described herein. For example, the signal (e.g., signal wave) is constructed to be transmitted via a computer network, such as a LAN, WLAN, WAN, a mobile network (e.g., the Internet). For example, the signal (e.g., signal wave) is constructed to be transmitted by optical or acoustic data transmission. Therefore, alternatively or additionally, the present invention may relate to a data stream representing the above-mentioned program (i.e., including the program) according to the second aspect.

[0131] In a fifth aspect, the present invention is directed to a system for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, comprising:

[0132] a) a computer, wherein the program is run on the computer or loaded into the memory of the computer, causing the computer to perform the method steps according to the second aspect or the third aspect;

[0133] b) at least one electronic data storage device storing individual patient image data and storing additional data, atlas data, individual viewing direction data, individual anatomical vector data and additional data, as long as the above program is run on at least one processor or loaded into the memory of a computer, causing the computer to perform the method according to the third aspect; and

[0134] c) the program storage medium according to the fourth aspect,

[0135] Wherein at least one computer is operably coupled to:

[0136] at least one electronic data storage device for acquiring individual patient image data from the at least one electronic data storage device and for causing the computer to perform the method according to the third aspect, whenever the above program is run on at least one processor of the computer or loaded onto the memory of the computer, to acquire additional data, atlas data, individual viewing direction data, individual anatomical vector data and additional data from the at least one electronic data storage device and for storing at least label relationship data in the at least one electronic data storage device; and

[0137] -A program storage medium for obtaining data defining model parameters and a learning algorithm architecture from the program storage medium.

[0138] For example, the present invention does not involve or particularly does not include or encompass invasive procedures that would represent a substantial physical disturbance of the body requiring professional medical intervention which could expose the body to significant health risks even if the required professional care or intervention were taken.

[0139] definition

[0140] This section provides definitions of certain terms used in this disclosure, which also constitute a part of this disclosure.

[0141] The method according to the invention is, for example, a computer-implemented method. For example, all or only some steps (i.e., less than the total number of steps) of the method according to the invention can be performed by a computer (e.g., at least one computer). An embodiment of the method implemented by a computer is a purpose of a computer for performing a data processing method. An embodiment of the method implemented by a computer is a method involving computer operation, such that the computer is operated to perform one, more or all steps of the method.

[0142] The computer comprises, for example, at least one processor and, for example, at least one memory, in order to (technically) process data, for example electronically and / or optically process data. The processor is, for example, made of a semiconductor substance or composition, for example, at least partially n-type and / or p-type doped semiconductor, for example, at least one of type II, type III, type IV, type V, type VI semiconductor materials, for example (doped) silicon arsenide and / or gallium arsenide. The described calculation steps or determination steps are, for example, performed by a computer. The determination step or calculation step is, for example, a step of determining data within the framework of a technical method (for example, within the framework of a program). The computer is, for example, any type of data processing device, for example, an electronic data processing device. The computer can be a device that is generally regarded as a computer, such as a desktop personal computer, a laptop, a netbook, etc., but can also be any programmable device, such as a mobile phone or an embedded processor. The computer can, for example, include a "sub-computer" system (network), in which each sub-computer represents its own computer. The term "computer" includes cloud computers, such as cloud servers. The term "computer" includes server resources. The term "cloud computer" includes a cloud computer system, which includes, for example, a system of at least one cloud computer, for example, a plurality of operably interconnected cloud computers, such as a server farm. Such a cloud computer is preferably connected to a wide area network such as the World Wide Web (WWW) and is located in a so-called cloud of computers all connected to the World Wide Web. Such an infrastructure is used for "cloud computing", which describes those computing, software, data access and storage services that do not require the end user to know the physical location and / or configuration of the computer providing a particular service. For example, the term "cloud" is used metaphorically for the Internet (World Wide Web). For example, the cloud provides a computing infrastructure as a service (IaaS). A cloud computer can be used as a virtual host for an operating system and / or a data processing application for executing the method of the present invention. A cloud computer is, for example, provided by Amazon Web Services. TM) provided by Elastic Compute Cloud (EC2). The computer, for example, includes an interface to receive or output data and / or perform analog-to-digital conversion. The data, for example, represents physical properties and / or is generated from technical signals. The technical signal is generated, for example, by a (technical) detection device (for example, a device for detecting a marker) and / or a (technical) analysis device (for example, a device for performing a (medical) imaging method), wherein the technical signal is, for example, an electrical signal or an optical signal. The technical signal, for example, represents data received or output by the computer. The computer is preferably operably coupled to a display device that allows information output by the computer to be displayed to, for example, a user. An example of a display device is a virtual reality device or an augmented reality device (also known as virtual reality glasses or augmented reality glasses), which can be used as "goggles" for navigation. A specific example of such augmented reality glasses is Google Glass (Google Glass, a trademark brand of Google, Inc.). The augmented reality device or virtual reality device can be used to input information into the computer through user interaction and to display information output by the computer. Another example of a display device is a standard computer monitor, for example including a liquid crystal display, which is operatively connected to a computer for receiving display control data from the computer for generating signals for displaying image information content on the display device. A specific embodiment of such a computer monitor is a digital light box. An example of such a digital light box is the product The monitor may also be a handheld portable device such as a smart phone or a personal digital assistant or a digital media player, for example.

[0143] The present invention also relates to a computer program comprising instructions, which, when executed by a computer, cause the computer to perform one or more methods described herein, such as steps of one or more methods; and / or a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing the above program; and / or a computer comprising the above program storage medium; and / or a signal wave (e.g., a physical, e.g., electrical, generated by technical means), such as a digital signal wave, such as an electromagnetic carrier wave, carrying information representing a program (e.g., the above program), the program for example comprising a code mechanism suitable for performing any or all of the method steps described herein. In one example, the signal wave is a data carrier signal carrying the above computer program. The present invention also relates to a computer comprising at least one processor and / or the above computer-readable storage medium and, for example, a memory, wherein the program is executed by a processor.

[0144] Within the framework of the present invention, a computer program element may be embodied as hardware and / or software (this includes firmware, resident software, microcode, etc.). Within the framework of the present invention, a computer program element may take the form of a computer program product, which may be embodied as a computer-usable, e.g. computer-readable data storage medium, which includes computer-usable, e.g. computer-readable program instructions, the "code" or "computer program" embodied in the data storage medium being used on or in conjunction with an instruction execution system. Such a system may be a computer; a computer may be a data processing device comprising means for executing a computer program element and / or a program according to the present invention, e.g. a data processing device comprising a digital processor (central processing unit or CPU) executing the computer program element, and optionally a data processing device comprising a volatile memory (e.g. random access memory or RAM) for storing data used for and / or generated by executing the computer program element. Within the framework of the present invention, a computer-usable, e.g. computer-readable data storage medium may be any data storage medium that may contain, store, communicate, propagate or transmit programs for use on or in conjunction with those instruction execution systems, devices or apparatuses. Computer-usable, for example, computer-readable data storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or communication media such as the Internet. Computer-usable or computer-readable data storage media can even be, for example, paper or other suitable media on which the program can be printed, because the program can be captured electronically, for example, by optically scanning the paper or other suitable media, and then compiled, decoded or otherwise processed in an appropriate manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product described herein and any software and / or hardware form various mechanisms for performing the functions of the present invention in the exemplary embodiments. The computer and / or data processing device can, for example, include a guidance information device, which includes a mechanism for outputting guidance information. The guidance information can be output to the user, for example, visually through a visual indication mechanism (e.g., a monitor and / or a light) and / or audibly through an auditory indication mechanism (e.g., a speaker and / or a digital voice output device) and / or tactilely through a tactile indication mechanism (e.g., a vibration element or a vibration element incorporated into an apparatus). For the purpose of this document, a computer is a technical computer, for example comprising technical components such as tangible components, for example mechanical components and / or electronic components. Any device mentioned in this document is a technical device and is for example a tangible device.

[0145] The expression "obtaining data" for example includes (within the framework of the computer-implemented method) a scenario in which data is determined by a computer-implemented method or program. Determining data for example includes measuring a physical quantity and transforming the measured value into data, such as digital data, and / or calculating (such as outputting) the data with the aid of a computer and for example within the framework of the method according to the invention. The "determining" step as described herein for example includes or consists of issuing a command to perform the determination described herein. For example, the step includes or consists of issuing a command to cause a computer (such as a remote computer, such as a remote server, such as in the cloud) to perform the determination. Alternatively or additionally, the "determining" step as described herein for example includes or consists of the following steps: receiving the result data of the determination described herein, such as receiving the result data from a remote computer (such as from a remote computer that causes it to perform the determination). The meaning of "obtaining data" also includes, for example, the following scenario: receiving or retrieving data by (such as inputting) a method or program implemented by a computer, such as from another program, a previous method step or a data storage medium, such as for further processing by a method or program implemented by a computer. The generation of the data to be acquired can, but does not have to be, part of the method according to the invention. Therefore, the expression "obtaining data" can also, for example, mean waiting to receive data and / or receiving data. The received data can be input, for example, via an interface. The expression "obtaining data" can also mean that a method or program implemented by a computer performs some steps in order to (actively) receive or retrieve data from a data source such as a data storage medium (e.g., ROM, RAM, database, hard disk drive, etc.) or via an interface (e.g., from another computer or network). The data obtained by the method or device of the present disclosure, respectively, can be obtained from a database located in a data storage device, which is operably connected to a computer so as to perform data transmission between the database and the computer, such as data transmission from a database to a computer. The computer obtains data to be used as input for the "determine data" step. The determined data can be output to the same or another database for storage for subsequent use. The database or the database for implementing the disclosed method can be located in a network data storage device or a network server (e.g., a cloud data storage device or a cloud server) or a local data storage device (e.g., a mass storage device operably connected to at least one computer that performs the disclosed method). The data can be "ready" by performing an additional step before the acquisition step. According to this additional step, data is generated for acquisition. For example, data are detected or captured (eg, by an analysis device). Alternatively or additionally, data are input according to an additional step, such as via an interface. For example, the generated data may be input (eg, into a computer).According to an additional step (which is performed before the acquisition step), the data can also be provided by performing an additional step of storing the data on a data storage medium (such as a ROM, RAM, CD and / or hard disk drive), so that the data are ready within the framework of the method or program according to the invention. Therefore, the step of "acquiring data" can also involve commanding the device to acquire and / or provide the data to be acquired. In particular, the acquisition step does not involve an invasive step, which represents a substantial physical interference with the body, requiring professional medical measures, and even if the required professional care and measures are taken when performed, the body may be exposed to significant health risks. In particular, the step of acquiring data, such as determining data, does not involve a surgical step, in particular does not involve a step of treating the human or animal body with surgery or therapy. In order to distinguish between the different data used by the present method, the data are denoted (i.e. referred to as) "XY data" etc. and are defined according to the information they describe, and then preferably referred to as "XY information" etc.

[0146] The function of the marker is to be detected by a marker detection device (e.g. a camera or an ultrasound receiver, or an analysis device such as a CT or MRI device) in such a way that the spatial position of the marker (i.e. its spatial position and / or alignment) can be determined. The detection device is, for example, part of a navigation system. The marker can be an active marker. An active marker can, for example, emit electromagnetic radiation and / or waves that can be in the infrared, visible and / or ultraviolet spectral range. However, the marker can also be a passive marker, i.e., it can, for example, reflect electromagnetic radiation in the infrared, visible and / or ultraviolet spectral range or can block X-ray radiation. To this end, the marker can be provided with a surface having corresponding reflective properties, or can be made of metal in order to block X-ray radiation. The marker can also reflect and / or emit electromagnetic radiation and / or waves in the radio frequency range or in an ultrasonic wavelength. The marker preferably has a spherical and / or spheroidal shape and can therefore be referred to as a marker sphere; however, the marker can also have an angular (e.g. cubic) shape.

[0147] The marker may be, for example, a reference star or a pointer or a single marker or a plurality of (individual) markers, which are then preferably in a predetermined spatial relationship. The marker comprises one, two, three or more markers, wherein two or more of these markers are in a predetermined spatial relationship. This predetermined spatial relationship is, for example, known to the navigation system and is, for example, stored in a computer of the navigation system.

[0148] In another embodiment, the marker comprises an optical pattern, for example, on a two-dimensional surface. The optical pattern may comprise a plurality of geometric shapes, such as circles, rectangles, and / or triangles. The optical pattern may be identified in an image captured by a camera, and the position of the marker relative to the camera may be determined based on the size of the pattern in the image, the orientation of the pattern in the image, and the distortion of the pattern in the image. This allows relative position in up to three rotational dimensions and up to three translational dimensions to be determined from a single two-dimensional image.

[0149] The position of the marker can be determined, for example, by a medical navigation system. If the marker is attached to an object (such as a bone or a medical device), the position of the object can be determined based on the position of the marker and the relative position between the marker and the object. Determining this relative position is also called registering the marker with the object. The marker or object can be tracked, which means determining the position of the marker or object two or more times over time.

[0150] Preferably, atlas data describing (e.g. defining, more particularly representing and / or serving as) the general three-dimensional shape of an anatomical part of a body is obtained. Thus, the atlas data represents an atlas of an anatomical part of a body. An atlas typically consists of a plurality of general models of objects, wherein these general models of objects together form a composite structure. For example, an atlas constitutes a statistical model of a patient's body (e.g. a part of a body), which has been generated based on anatomical information collected from a plurality of human bodies, for example based on medical image data containing images of these human bodies. Thus, in principle, the atlas data represents the result of a statistical analysis of such medical image data of a plurality of human bodies. This result can be output as an image - thus the atlas data contains or is equivalent to medical image data. Such a comparison can be performed, for example, by applying an image fusion algorithm, wherein the image fusion algorithm performs image fusion between the atlas data and the medical image data. The comparison result can be a measure of similarity between the atlas data and the medical image data. The atlas data includes image information (e.g., position image information) that can be matched with image information (e.g., position image information) contained in, for example, medical image data (e.g., by applying an elastic or rigid image fusion algorithm) so that, for example, the atlas data is compared with the medical image data in order to determine the position of anatomical structures in the medical image data that correspond to anatomical structures defined by the atlas data.

[0151] Multiple human bodies (whose anatomical structures are used as inputs to generate atlas data) advantageously share common features, such as at least one of gender, age, race, physical measurements (e.g., height and / or weight), and pathological conditions. Anatomical information, for example, describes the anatomical structure of a human body and is, for example, extracted from medical image information about the human body. For example, an atlas of a femur may include the femoral head, femoral neck, body, greater trochanter, lesser trochanter, and lower limbs as objects that together constitute a complete structure. For example, an atlas of the brain may include the telencephalon, cerebellum, diencephalon, pons, midbrain, and medulla oblongata as objects that together constitute a complex structure. One application of such an atlas is in medical image segmentation, where an atlas is matched to medical image data, and the image data is compared to the matched atlas so that points (pixels or voxels) of the image data are assigned to objects of the matched atlas, thereby segmenting the image data into objects.

[0152] For example, the atlas data includes information about the anatomical part of the body. The information is, for example, at least one of patient-specific, non-patient-specific, indication-specific, or non-indication-specific. Therefore, the atlas data describes, for example, at least one of a patient-specific, non-patient-specific, indication-specific, or non-indication-specific atlas. For example, the atlas data includes movement information indicating the freedom of movement of the anatomical part of the body relative to a given reference (e.g., another anatomical part of the body). For example, the atlas is a multimodal atlas that defines atlas information for multiple (i.e., at least two) imaging modalities and includes mappings between atlas information under different imaging modalities (e.g., mappings between all modalities), so that these atlases can be used to transform medical image information from its image depiction under a first imaging modality to its image depiction under a second imaging modality different from the first imaging modality, or to compare different imaging modalities with each other (e.g., matching or registration).

[0153] In the medical field, imaging methods (also referred to as imaging modalities and / or medical imaging modalities) are used to generate image data (e.g. two-dimensional or three-dimensional image data) of human anatomical structures (such as soft tissue, bones, organs, etc.). The term "medical imaging method" is to be understood as meaning (advantageously device-based) imaging methods (e.g. so-called medical imaging modalities and / or radiographic methods), such as computed tomography (CT) and cone beam computed tomography (CBCT, such as volume CBCT), X-ray tomography, magnetic resonance tomography (MRT or MRI), conventional X-ray, sonography and / or ultrasound verification and positron emission tomography. For example, the medical imaging method is performed by an analysis device. Examples of medical imaging modalities applied by the medical imaging method are: X-ray, magnetic resonance imaging, medical ultrasound scanning or ultrasound, endoscopy, elastic imaging, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques such as positron emission tomography (PET) and single photon emission computed tomography (SPECT). The image data thus generated is also referred to as "medical imaging data". The analysis device is used, for example, to generate image data in an imaging method based on an apparatus. The imaging method is used, for example, for medical diagnosis of the analysis of the anatomical structure of the body to generate an image described by the image data. The imaging method is also used, for example, to detect pathological changes in the human body. However, some changes in the anatomical structure, such as pathological changes in the structure (tissue), may not be detected and may not be visible, for example, in an image generated by the imaging method. A tumor represents an example of a change in the anatomical structure. If a tumor grows, it can be considered to represent an expanded anatomical structure. Such an expanded anatomical structure may not be detected, for example, only a part of the expanded anatomical structure can be detected. For example, early / late brain tumors are usually visible in MRI scans when a contrast agent is used to infiltrate the tumor. MRI scanning represents an example of an imaging method. In the case of an MRI scan of such a brain tumor, it is considered that the signal enhancement in the MRI image (caused by the contrast agent infiltrating the tumor) represents a solid tumor mass. Therefore, the tumor is detectable and can be distinguished, for example, in an image generated by the imaging method. In addition to these tumors, known as "enhancing" tumors, it is believed that approximately 10% of brain tumors are not discernible in scans and are not visible, for example, to a user viewing an image generated by an imaging method. BRIEF DESCRIPTION OF THE DRAWINGS

[0154] In the following, the present invention is described with reference to the accompanying drawings, which give a background description of the present invention and show specific embodiments of the present invention. However, the scope of the present invention is not limited to the specific features disclosed in the context of the accompanying drawings, in which:

[0155] Figure 1 showing neurons of a neural network;

[0156] Figure 2 The basic flow of the method according to the first aspect is shown;

[0157] Figure 3 The basic flow of the method according to the second aspect is shown;

[0158] Figure 4 The basic flow of the method according to the third aspect is shown;

[0159] Figure 5 The application of the method according to the first aspect to the third aspect is shown;

[0160] Figure 6 is a schematic diagram of a system according to the fifth aspect;

[0161] Figure 7a , Figure 7b and Figure 7c Explain three embodiments of using anatomical vectors to create ground truth values ​​according to the methods of the first and second aspects;

[0162] Figure 8a and Figure 8b Explain an embodiment of using anatomical vectors to create a segmentation mask prior according to the method of the first aspect and the second aspect;

[0163] Figure 9a and Figure 9b illustrating an embodiment of the method according to the first and second aspects using anatomical vectors as additional input (as raw numbers);

[0164] Fig.10a and Fig.10b illustrating an embodiment of the method according to the first and second aspects using anatomical vectors as additional input (as raw numbers);

[0165] Fig.11a and Fig.11b Explain an embodiment of using anatomical vectors to constrain an atlas according to the methods of the first and second aspects;

[0166] Fig.12a and Figure 12b Explain an embodiment of using anatomical vectors for classification / segmentation / localization according to the methods of the first and second aspects;

[0167] Fig.13a and Fig.13b An embodiment of using anatomical vectors for classification / segmentation / localization according to the methods of the first and second aspects is explained. DETAILED DESCRIPTION

[0168] Figure 1Illustration of the structure of a neuron as part of a neural network, such as a convolutional neural network, where inputs are assigned certain weights for processing by an activation function that generates the neuron's output.

[0169] Figure 2 The basic flow of the method according to the first aspect is described, which starts from step S21, obtaining patient training image data, continues to step S22, which includes obtaining atlas data, and then continues to obtain viewing direction data in step S23. On this basis, step S24 calculates anatomical vector data, and then in step S25, obtains label data. Finally, in step S26, anatomical index data is determined.

[0170] Figure 3 The basic steps of the method according to the second aspect are shown, wherein step S31 comprises acquiring individual patient image data, and step S32 determines label relationship data.

[0171] Figure 4 The basic steps of the method according to the third aspect are shown, wherein step S41 comprises acquiring individual patient image data, step 42 acquires atlas data. A subsequent step 43 acquires individual viewing direction data, followed by determining individual anatomical vector data in step S44. Additional data is acquired in step S45. Then, step S46 determines label relationship data.

[0172] Figure 5 The application of the method according to the first aspect to the third aspect is summarized. A generic patient model (atlas) 1 and a tracking imaging device 6 are registered to a patient space 4 via a registration 5 between the atlas 1 and the patient space 4 (using, for example, 3D tomographic image data as planning data for a specific patient and an elastic registration to the atlas) and via a registration 1 between the patient space 4 and the tracking imaging device 6. The registrations 5 and 13 can be combined into a registration 14 between the atlas 1 and the tracking imaging device 6. The tracking imaging device 6 generates medical images 7 and generates 8 labels 9 for these images, for example, according to the atlas 1 or manually. Anatomical vectors are determined by the registration 14, which is based on the registrations 5 and 13 and the known viewing direction of the tracking imaging device 6.

[0173] During learning, a medical image 7 is input 15 into a learning algorithm 12. Anatomical vectors determined using the atlas 1 are input 2 into the learning algorithm 12. Labels 9 are input 10 into the learning algorithm 12. Optionally, the atlas 1 is input 3 into the learning algorithm 12. Thus, during learning, a relationship between the labels 9 and the anatomical structures depicted by the medical image 7 is determined based on the inputs 2, 10, 15 and optionally based on the input 3.

[0174] During testing or use of the learning algorithm, a medical image 7 is input 15 into the learning algorithm 12. Anatomical vectors determined using the atlas 1 are input 2 into the learning algorithm 12. Optionally, the atlas 1 is input 3 into the learning algorithm 12. Labels 9 are determined 11 by the learning algorithm based on the inputs 2 and 15 and optionally based on the input 3.

[0175] Figure 6 is a schematic diagram of a medical system 61 according to the fifth aspect. The system is generally indicated with reference numeral 61 and comprises a computer 62 and an electronic data storage device (such as a hard disk) 63 for storing data stored by the system according to at least the fifth aspect. The components of the medical system 1 have the functions and characteristics explained above in relation to the fifth aspect of the present disclosure.

[0176] Figure 7a , Figure 7b and Figure 7c Embodiments 1 to 3 of creating ground truth using anatomical vectors according to the methods of the first and second aspects are explained.

[0177] Figure 7a Embodiment 1 is shown to use the method according to the second aspect to segment or locate image features, and to create ground truth segmentations using anatomical vectors from an atlas registered with the patient image. For training according to the method according to the first aspect, the video image and ground truth segmentation are input into a learning algorithm embodied as a machine learning algorithm. When testing the trained algorithm using the method according to the second aspect, the video image is input into the algorithm. The output of the algorithm is the segmentation or location of the image features. The training is performed in a supervised manner. Figure 5 The registration referred to in the context of is an elastic registration and is updated during the medical procedure performed on the patient.

[0178] Figure 7a Embodiment 2 is also shown to classify image features using the method according to the second aspect, and to create ground truth classes using anatomical vectors from an atlas registered with the patient image. For training according to the method according to the first aspect, the video images and ground truth classes are input into a learning algorithm embodied as a machine learning algorithm. When testing the trained algorithm using the method according to the second aspect, the video images are input into the algorithm. The output of the algorithm is the class of the image features. The training is performed in a supervised manner. Figure 5 The registration mentioned in the context of is either a strict or better registration, but usually depends on the image content that should be classified.

[0179] also, Figure 7aExample 3 is shown using the method according to the second aspect to segment or locate image features, and using anatomical vectors from an atlas registered with the patient image to create classes. For training according to the method according to the first aspect, the video image and the class are input into a learning algorithm embodied as a machine learning algorithm. When the trained algorithm is tested using the method according to the second aspect, the video image is input into the algorithm. The output of the algorithm is the segmentation or location of the image features. The training is performed in a weakly supervised manner. Figure 5 The registration mentioned in this context is a strict or better registration, but usually the claim should be about the image content being localized.

[0180] Figure 7b Embodiments 1 and 2 are shown to use anatomical vectors to create ground truth values ​​embodied as labels during algorithm training. The ground truth values ​​are fed into a machine learning (ML) algorithm along with the video data to generate labels using the ML algorithm.

[0181] Figure 7c Embodiment 3 shows that the anatomical vectors are used to create classes embodied as labels during algorithm training. The classes are fed into a machine learning (ML) algorithm along with the video data to generate labels using the ML algorithm.

[0182] Figure 8a and Figure 8b Example 4 of using anatomical vectors to create a segmentation mask prior according to the method of the first aspect and the second aspect is explained.

[0183] Figure 8a Embodiment 4 is shown to use the method according to the second aspect to segment or locate image features, and to create a segmentation mask prior using anatomical vectors. For training according to the method according to the first aspect, the video image, the segmentation mask prior, and the basic truth segmentation are input into a learning algorithm embodied as a machine learning algorithm. When the trained algorithm is tested using the method according to the second aspect, the video image and the prior are input into the algorithm. The output of the algorithm is the segmentation or location of the image features. The training is performed in a supervised manner. Figure 5 The registration referred to in the context of is an elastic registration and is updated during the medical procedure performed on the patient.

[0184] Figure 8b It is shown that the anatomical vector is used to create a segmentation mask prior, which is fed into a machine learning (ML) algorithm along with the ground truth and video data embodied as labels to generate labels using the ML algorithm.

[0185] Figure 9a and Figure 9b Example 5 using anatomical vectors as additional input (as raw numbers) according to the methods of the first and second aspects is explained.

[0186] Figure 9a Embodiment 5 is shown to use the method according to the second aspect to classify or segment or locate image features, and the anatomical vectors are used as additional inputs to the learning algorithm (for example, as raw numbers). For training according to the method according to the first aspect, the video images, the corresponding anatomical vectors, and the manually created labels are input into the learning algorithm embodied as a machine learning algorithm. When the trained algorithm is tested using the method according to the second aspect, the video images and the anatomical vectors are input into the algorithm. The output of the algorithm is the class of the image features or the segmentation or location of the image features. The training is performed in a supervised manner. In the above Figure 5 The context of registration depends on the type of application.

[0187] Figure 9b It is shown that the anatomical vectors are input into a machine learning (ML) algorithm together with the ground truth and video data embodied as labels to generate labels using the ML algorithm.

[0188] Fig.10a and Fig.10b Example 6 using anatomical vectors as additional input (as raw numbers) according to the methods of the first and second aspects is explained.

[0189] Fig.10a Embodiment 6 is shown to classify or segment or locate image features using the method according to the second aspect, and anatomical vectors are used as additional inputs to the learning algorithm (e.g., as raw numbers). For training according to the method according to the first aspect, video images, anatomical vectors, undeformed atlases, and manually created labels are input into a learning algorithm embodied as a machine learning algorithm. When testing the trained algorithm using the method according to the second aspect, the video images and anatomical vectors as well as the undeformed atlas are input into the algorithm. The output of the algorithm is the class of the image features or the segmentation or location of the image features. The training is performed in a supervised manner. In the above Figure 5 The context of registration depends on the type of application.

[0190] Fig.10b It is shown that the anatomical vectors are input into a machine learning (ML) algorithm together with the ground truth embodied as labels, video data, and an undeformed atlas to generate labels using the ML algorithm.

[0191] Fig.11a and Fig.11b Example 7 of using anatomical vectors to constrain an atlas according to the methods of the first and second aspects is explained.

[0192] Fig.11aEmbodiment 7 is shown to classify or segment or locate image features using the method according to the second aspect, and anatomical vectors are used to constrain the atlas. For training according to the method according to the first aspect, the video image and the constrained undeformed atlas are input into a learning algorithm embodied as a machine learning algorithm together with manually created labels. When the trained algorithm is tested using the method according to the second aspect, the video image and the constrained undeformed atlas are input into the algorithm. The output of the algorithm is the class of the image feature or the segmentation or location of the image feature. The training is performed in a supervised manner. In the above Figure 5 The registration referred to in the context of is a strict or better registration.

[0193] Fig.11b The anatomical vectors are shown to be used to generate a constraint atlas, which is input into a machine learning (ML) algorithm along with ground truth values ​​embodied as labels and video data to generate labels using the ML algorithm.

[0194] Fig.12a and Figure 12b Example 8 of using anatomical vectors for classification / segmentation / localization according to the methods of the first and second aspects is explained.

[0195] Fig.12a Embodiment 8 is shown to use the method according to the second aspect to classify or segment or locate image features, and the anatomical vector is used to create a ground truth value for the body part classifier in the first stage. For training according to the method according to the first aspect, in the first stage, the video image and the ground truth value are input into a learning algorithm embodied as a machine learning algorithm. In the second stage, the label is input into another algorithm that does not have to be a learning algorithm. When testing a trained algorithm using the method according to the second aspect, the video image is input into the algorithm in the first stage, and the label output by the learning algorithm is input into other algorithms in the second stage. For example, the label will undergo further data processing after being output by the learning algorithm. The other algorithm may, for example, be another learning algorithm for segmenting the anatomical parts of the body visible in the individual medical image identified by the label. The output of the algorithm is the class of the image feature or the segmentation or location of the image feature. The training is performed in a supervised manner. The above in Figure 5 The registration referred to in the context of is a strict or better registration.

[0196] Figure 12b The anatomical vectors are shown to be used to generate ground truth values ​​embodied as body part labels, which are input into a machine learning (ML) algorithm along with the video data to generate body part labels using the ML algorithm. In the second stage, the body part labels are input into another algorithm.

[0197] Fig.13a and Fig.13bExample 9 of using anatomical vectors for classification / segmentation / localization according to the methods of the first and second aspects is explained.

[0198] Fig.13a Embodiment 9 is shown to classify or segment or locate image features using the method according to the second aspect, and anatomical vectors are used to create a basic truth curriculum data pool. The curriculum represents the gradual increase in the complexity of the training data in different training steps. For training according to the method according to the first aspect, any applicable type of data is input into the learning algorithm embodied as a machine learning algorithm. The order in which the training data is input is controlled by the curriculum. When the trained algorithm is tested using the method according to the second aspect, another type of data is input into the algorithm. The output of the algorithm is the class of the image feature or the segmentation of the image feature. The training is performed in a supervised manner. The above Figure 5 The registration referred to in the context of is a strict or better registration.

[0199] Fig.13b It is shown that the anatomical vector is input into a curriculum generator along with the ground truth and video data, and the output of the curriculum generator is input into a machine learning (ML) algorithm to generate labels using the ML algorithm.

Claims

1. A computer-implemented method for training a learning algorithm for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, the method comprising the steps of: a) acquiring patient training image data (S21), wherein the patient training image data describes digital medical images of anatomical structures of a plurality of patients; b) acquiring atlas data ( S22 ), the atlas data being comparable to the image information and describing an anatomical model of the body anatomical part including the anatomical structure; c) acquiring viewing direction data (S23), the viewing direction data describing a viewing direction of an imaging device toward the anatomical structure at a point in time for generating the medical image; d) determining anatomical vector data based on the viewing direction data and the atlas data (S24), wherein the anatomical vector data describes an anatomical vector obtained by transforming the viewing direction into a reference system, and defines a position in the anatomical model in the reference system; e) acquiring label data ( S25 ), wherein the label data describes a label indicating a position or type of an anatomical structure in the anatomical model; and f) determining anatomical indicator data (S26) based on the patient training image data, the anatomical vector data and the label data, wherein the anatomical indicator data describes model parameters of a learning algorithm for establishing a relationship between the position or type of the anatomical structure described by the medical image and the label, wherein the anatomical indicator data is determined by inputting the patient training image data and the label data and a subset of atlas data into a relationship establishment function, wherein the subset of atlas data has been determined based on the atlas data and the anatomical vector data.

2. The method according to claim 1, wherein: The medical image is a two-dimensional image, including a microscope image or a portion of a video taken with a microscope, wherein the imaging device is a microscope, an endoscope equipped with a digital camera, or an X-ray device configured to produce a two-dimensional projection image.

3. The method according to claim 1, wherein: The anatomical index data are determined by additionally inputting into the relationship building function a real subset that has been determined based on the atlas data and the anatomical vector data.

4. The method according to claim 1, comprising the steps of: acquiring additional data as a function of the anatomical vector; as well as The anatomical index data are determined by additionally inputting the additional data into the relationship building function.

5. The method according to claim 4, wherein: The additional data comprises the anatomical vector data or consists of the anatomical vector data.

6. The method according to claim 1, wherein: The learning algorithm includes a machine learning algorithm or consists of a machine learning algorithm.

7. The method according to claim 1, wherein: The learning algorithm includes or consists of a convolutional neural network.

8. The method according to claim 1, wherein: The model parameters define the learnable parameters of the learning algorithm.

9. A computer-implemented method for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, the method comprising the steps of: a) acquiring individual patient image data (S31), wherein the individual patient image data describes a digital individual medical image of the anatomical structure of the individual patient; as well as b) determining label relationship data (S32), the label relationship data describing the relationship between the label and the anatomical structure in the individual medical image, wherein the label relationship data is determined by inputting the individual patient image data into a function that establishes the relationship between the anatomical structure described by the individual medical image and the label, the function being part of a learning algorithm that has been trained by executing the method according to claim 1 or 2.

10. A computer-implemented method for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, the method comprising the steps of: a) acquiring individual patient image data (S41), wherein the individual patient image data describes a digital individual medical image of the anatomical structure of the individual patient; b) acquiring atlas data (S42), the atlas data describing an anatomical model of a body anatomical part including the anatomical structure; and c) acquiring individual viewing direction data (S43), the individual viewing direction data describing the viewing direction of an imaging device towards the anatomical structure at a point in time for generating the individual medical image; d) determining individual anatomical vector data based on the individual viewing direction data and the atlas data (S44), wherein the anatomical vector data describes an anatomical vector obtained by transforming the viewing direction into a reference system, and defines a position in the anatomical model in the reference system; e) acquiring additional data (S45), the additional data being a function of the individual anatomical vectors; and f) determining label relationship data (S46), the label relationship data describing the relationship between the label and the anatomical structure described by the individual medical image, wherein the label relationship data is determined by inputting the individual patient image data and the additional data into a relationship establishing function between the position or type of the anatomical structure in the individual medical image and the label, the function being part of a learning algorithm that has been trained by executing the method according to claim 4, wherein the function used to generate the anatomical vector of the acquired additional data is the same as the function used to generate the anatomical vector of the additional data input into the relationship establishing function used to determine the anatomical indicator data.

11. The method according to claim 10, wherein: The additional data includes or consists of the individual anatomical vector data, and The learning algorithm has been trained by additionally carrying out the method according to claim 5 .

12. The method according to claim 11, wherein: determining the label relationship data additionally based on the atlas data by additionally inputting a subset of the atlas data into a function for establishing a relationship between the anatomical structure described by the individual medical image and the label, and The learning algorithm has been trained by additionally carrying out the method according to claim 3 .

13. The method according to any one of claims 9 to 12, wherein: The individual medical image is a two-dimensional image, including a microscope image or a portion of a video taken with a microscope, wherein the imaging device is a microscope.

14. The method according to claim 10, wherein: The relative position between the imaging device used to generate the individual patient image data and the individual anatomical vector data is predetermined and obtained by the method.

15. The method according to claim 9 or 10, wherein: The learning algorithm includes a machine learning algorithm or consists of a machine learning algorithm.

16. The method according to claim 9 or 10, wherein: The learning algorithm includes or consists of a convolutional neural network.

17. The method according to claim 11, wherein: The model parameters define the learnable parameters of the learning algorithm.

18. A program storage medium, wherein the program stored on the program storage medium, when executed on at least one processor of a computer (62) or loaded into a memory of the computer (62), causes the computer (62) to execute the method steps of the method according to any one of claims 1 to 17.

19. A system (61) for determining a relationship between a label indicating a location or type of an anatomical structure in a medical image and the location or type of the anatomical structure in the medical image, comprising: a) a computer (62), wherein the program, when executed on at least one processor of the computer (62) or loaded onto a memory of the computer (62), causes the computer (62) to perform the method steps of the method according to any one of claims 1 to 17; b) at least one electronic data storage device (63) storing individual patient image data and storing the atlas data, the individual viewing direction data, the individual anatomical vector data and the additional data as long as the program, when run on at least one processor of the computer (62) or loaded onto a memory of the computer (62), causes the computer to perform the method according to claim 10; and c) The program storage medium according to claim 18, Wherein, the computer (62) is operatively coupled to: the at least one electronic data storage device (63) being used to obtain the individual patient image data from the at least one electronic data storage device (63), and to obtain the atlas data, the individual viewing direction data, the individual anatomical vector data and the additional data from the at least one electronic data storage device (63) as long as the program, when executed on at least one processor of the computer (62) or loaded onto the memory of the computer (62), causes the computer to perform the method according to claim 10, and to store at least the label relationship data in the at least one electronic data storage device (63); and The program storage medium is used to obtain data defining model parameters and learning algorithm architecture from the program storage medium.

Images