Probabilistic regularization of convolutional neural networks for correlation-based multi-feature detection

Abstract

The invention relates to landmark and / or time event detection. The invention proposes to utilize previously learned spatial statistical correlations between multiple landmarks in order to regularize a convolutional neural network (CNN) as a post-processing step or during training in order to exploit anatomical priors, reduce false positive prediction rates, and / or increase the accuracy and stability of the algorithm. The proposed devices and methods can also be applied to improve the detection of related events in e.g. time series by exploiting prior knowledge.

Description

Technical Field

[0001] This invention relates to multi-feature detection. Specifically, this invention relates to an apparatus for feature recognition in medical images, a medical imaging system, a method for feature recognition in medical images, and a computer program unit. Background Technology

[0002] Convolutional neural networks (CNNs) have been used for anatomical landmark detection in medical images by predicting heatmaps of landmark locations. An example of using CNNs for anatomical landmark detection can be found in US2018 / 0116620A1. However, heatmaps can lead to false positive predictions, necessitating post-processing of the raw CNN results. Summary of the Invention

[0003] Improvements in feature detection in medical images may be needed.

[0004] The object of this invention is achieved through the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims. It should be noted that the following aspects of the invention also apply to devices, medical imaging systems, methods, and computer program units.

[0005] Therefore, a first aspect of the present invention provides an apparatus for feature recognition in medical images. The apparatus includes an input unit, a processing unit, and an output unit. The input unit is configured to receive at least one medical image of an object, wherein the at least one medical image includes a multidimensional image. The processing unit is configured to: i) generate feature heatmaps of at least two features of interest by applying a convolutional neural network (CNN) to the at least one medical image, wherein the CNN has been trained for feature recognition; ii) use the correlation between the at least two features of interest as a probabilistic prior for mutual regularization of the feature heatmaps, wherein the statistical correlation is estimated based on annotated medical image data different from the received at least one medical image of the object; and iii) identify at least two features of interest in the at least one medical image based on the regularized feature heatmaps. The output unit is configured to output the result of the feature recognition.

[0006] In other words, this invention proposes to utilize the correlations, i.e., statistical correlations, between multiple previously learned features of interest to regularize the CNNS as a post-processing step or during training, thereby leveraging prior knowledge to reduce false positive prediction rates and / or increase the accuracy and stability of the algorithm. The features of interest may include landmarks (e.g., points, voxels, contours, surfaces, and / or volumes) and / or temporal events (temporal activities of points, voxels, contours, surfaces, and / or volumes). The statistical correlations (e.g., spatial and / or temporal correlations) between the multiple features are previously learned from annotated medical image data (e.g., training data).

[0007] The statistical correlation between spatial landmarks refers to a statistical property derived from at least two landmarks on training data. This property can involve the conditional probability of detecting the first landmark at the first location given that the second landmark was detected at the second location. For efficient computation, this conditional probability can be approximated by the probability density of the relative vector between the locations of the first and second landmarks. The corresponding definition applies to the case of temporal events. The statistical correlation between spatial landmarks will be explained below, particularly regarding… Figures 3A to 3D The example shown.

[0008] The proposed device can be applied to all landmark detection applications based on CNN heatmap prediction and involving at least two landmarks. For example, the proposed device can be used to improve the detection of the medial end of the clavicle in chest X-ray images to estimate the patient's rotation around the head-to-foot axis for image quality assessment and control. While the absolute position of the medial end of the clavicle varies considerably between images, their relative distances can be anatomically constrained. The proposed device can allow this anatomical knowledge to be incorporated into the clavicle-head detector to improve its accuracy and robustness. Although landmark detection in images is illustrated below, the proposed device can also be applied to improve the detection of relevant events, such as in time series, by leveraging prior knowledge.

[0009] According to one embodiment of the invention, at least two features of interest include at least two landmarks. The feature heatmap includes landmark heatmaps of at least two landmarks generated by applying a first CNN to at least one medical image. The first CNN has been trained for landmark recognition. Statistical correlation includes spatial correlation between at least two landmarks, which serves as a first probabilistic prior for mutual regularization of the landmark heatmaps. The regularized landmark heatmap includes a regularized landmark heatmap.

[0010] Examples of landmarks may include points, voxels, contours, surfaces, and / or volumes in medical images. In some examples, landmarks may include anatomical features such as folds, protrusions, ducts, blood vessels, etc.

[0011] According to one embodiment of the present invention, the processing unit is configured to estimate the spatial correlation between at least two landmarks based on the conditional probability of detecting a landmark at a certain spatial location given that another landmark has already been detected at a certain spatial location.

[0012] The representation of conditional probabilities allows the application of the prior probability to be represented as a convolution, thus making it easy to integrate into existing CNN architectures.

[0013] For example, these spatial correlations can be based on the fact that it is already known at location x j In the case where the j-th landmark is found at position x i The conditional probability of finding the i-th landmark is used to measure this:

[0014] p i|j (x i |x j )

[0015] Given the heatmap result q of the j-th landmark j (x j One can estimate the predicted heatmap for the i-th landmark as follows:

[0016]

[0017] Its heatmap q, used as the i-th landmark i (x i () prior.

[0018] Alternatively, when the conditional probabilities are for 2n-dimensional objects (where n represents the image dimension, i.e., n = 2, 3) and are therefore difficult to estimate, the transformation invariance of these binary correlations can be assumed, i.e.:

[0019] p i|j (x i +d|x j +d)=p i|j (x i |x j )

[0020] For all spatial displacements d, we can obtain:

[0021]

[0022] The right-hand side represents the probability distribution of the relative vector between the j-th and i-th landmarks. As an n-dimensional entity, the latter can be easily estimated from the training data. This will be explained below, especially regarding... Figures 3A to 3D Exemplary embodiments are shown below.

[0023] According to one embodiment of the present invention, the processing unit is configured to estimate the spatial correlation between a landmark and two or more other landmarks based on the product of conditional probabilities, each conditional probability being the conditional probability of detecting the landmark at a certain spatial location given that one of two or more other landmarks has already been detected at a certain spatial location.

[0024] For example, in order to impose mutual priors on the detection of more than two landmarks, spatial correlation can be based on the i-th landmark and N other landmarks, i.e., the j1, ..., j-th landmarks. N The approximation of the spatial correlation between several landmarks (e.g., their N anatomical nearest neighbors) is determined by decomposing them into binary correlations:

[0025]

[0026] Each factor on the right-hand side can be estimated using the probability density of the corresponding relative coordinates. Therefore, the proposed method avoids the curse of dimensionality when estimating and evaluating these correlations. This leads to the generation of information about the factors on the j1, ..., j... N The following priors are given to the heatmap of the i-th landmark induced by the heatmap of the i-th landmark:

[0027]

[0028] According to one embodiment of the present invention, a processing unit is configured to estimate a first probability prior of a landmark by convolving the conditional probability of a landmark at a certain spatial location with a heatmap result of another landmark, the heatmap result indicating the spurious probability of the other landmark appearing at one or more spatial locations.

[0029] According to one embodiment of the invention, at least two features of interest include at least two temporal events. The landmark heatmap includes a temporal event heatmap of at least two temporal events generated by applying a second CNN to at least one medical image. The second CNN has been trained for temporal event recognition. Statistical correlation includes the temporal correlation between the at least two temporal events to be detected, which serves as a second probabilistic prior for mutual regularization of the temporal event heatmaps. The regularized landmark heatmap includes a regularized temporal event heatmap.

[0030] In other words, it also proposes to improve the detection of relevant events, such as in time series, by leveraging prior knowledge.

[0031] Examples of temporal events may include temporal activities of points, voxels, contours, surfaces, and / or volumes. In some examples, temporal events are temporal activities that include one or more anatomical features. Temporal activities may include changes in intensity, changes in geometry, and / or movement of points, voxels, contours, surfaces, and / or volumes.

[0032] For example, in a functional magnetic resonance imaging (fMRI) setting, the focus is on activity in voxels following one or more stimuli (e.g., with a certain drug dose). In this case, the fMRI signal effectively constitutes a multidimensional (due to the spatial dimension) time series. In this setting, it is possible to predict, for example, the time following a stimulus when two or more activation patterns occur—e.g., the start of an increase in activity and the start of a decrease in activity—by using a CNN in the temporal dimension. Once the temporal correlation between two or more such events is understood, a priori information can be derived and the CNN predictions regularized. This can be particularly useful when the receptive domain of the CNN (i.e., the past time span considered for predicting the future) is too low to correctly capture the correlation between two events lagging behind those events.

[0033] According to one embodiment of the present invention, the processing unit is configured to estimate the temporal correlation between at least two time events based on the conditional probability of detecting a time event at a certain time location given that another time event has already been detected at a certain time location.

[0034] Similar to the landmark detection setup, the representation of conditional probabilities allows the application of that probability prior to be represented as a convolution with a fixed kernel, thus making it easy to integrate into existing CNN architectures.

[0035] Similar to landmark detection settings, these temporal correlations can be based on the fact that the data is already at a known time position x. j In the case of finding the j-th time event at time position x i The conditional probability of finding the i-th time event is used to measure:

[0036] p i|j (x i |x j )

[0037] Given the heatmap result q of the j-th time event j (x j One can estimate the prediction of the heatmap for the i-th time event as follows:

[0038]

[0039] Its heatmap q as the i-th time event i (x i() prior.

[0040] According to one embodiment of the present invention, the processing unit is configured to estimate the temporal correlation between a time event and two or more other time events based on the product of conditional probabilities, each conditional probability being the conditional probability of detecting the time event at a certain time position given that one of two or more other time events has already been detected at a certain time position.

[0041] Similar to the landmark detection setup, temporal correlation can be based on the i-th time event and N other time events, i.e., the j-th, ..., j-th events. N The approximation of the temporal correlation between N time events (e.g., their N nearest neighbors at time) is determined by decomposing them into binary correlations:

[0042]

[0043] Each factor on the right-hand side corresponds to the probability density of the corresponding relative coordinate. Therefore, this method avoids the curse of dimensionality when estimating and evaluating these temporal correlations. This leads to the generation of information about the j1, ..., j... factors. N The following priors are given to the heatmap of the i-th time event induced by the heatmap of the i-th time event:

[0044]

[0045] According to one embodiment of the present invention, the processing unit is configured to estimate a second probability prior of a time event by convolving the conditional probability of a time event at a certain time position with a heatmap result of another time event, the heatmap result indicating the pseudo probability of the other time event occurring at one or more time positions.

[0046] According to one embodiment of the invention, the processing unit is configured to determine statistical correlation based on training data using a kernel density estimation method for the relative vector between any pair of two features of interest.

[0047] According to one embodiment of the present invention, the computation of probabilistic priors is represented as additional convolutional layers of a CNN.

[0048] For example, the computation of probabilistic priors is represented as additional convolutional layers with fixed weights in a CNN.

[0049] In this way, probabilistic regularization schemes can be integrated into all existing CNN architectures by adding convolutional layers. This allows for two variations of applying the regularization layer: either applying it only at test time, i.e., as a post-processing step, or applying it during CNN training to stabilize predictions by learning features related to relevance.

[0050] According to one embodiment of the invention, a probabilistic prior is used to mutually regularize the landmark heatmap in at least one of the following ways: during a post-processing step, applying a regularization layer at test time, and applying a regularization layer during CNN training.

[0051] In other words, the construction of probabilistic priors can be used during training (as a fixed term) or as a post-processing step to improve the heatmap of predictions.

[0052] Once these statistical correlations are estimated from the training data, they can be kept fixed, thus preserving their interpretation, even when regularization layers are used during training. This framework allows for the approximate inclusion of statistical correlations between the target of interest and two or more additional targets due to factorization approximations of these correlations.

[0053] A second aspect of the invention provides a medical imaging system comprising a medical imaging device and the device described above. The medical imaging device is configured to acquire at least one medical image of a patient. The device is configured to identify at least two landmarks of interest in the at least one medical image.

[0054] Medical imaging equipment may include at least one of the following: magnetic resonance imaging (MRI) equipment, computed tomography (CT) imaging equipment, ultrasound imaging equipment, optical imaging equipment, positron emission tomography (PET) imaging equipment, and single-photon emission computed tomography (SPECT) imaging equipment.

[0055] A third aspect of the present invention provides a method for landmark recognition in medical images. The method includes:

[0056] Receive at least one medical image of an object, wherein the at least one medical image includes a multidimensional image;

[0057] By applying a convolutional neural network (CNN) to at least one medical image, feature heatmaps of at least two features of interest are generated, wherein the CNN has been trained for feature recognition.

[0058] The statistical correlation between at least two features of interest is used as a probabilistic prior for mutual regularization of the feature heatmap, wherein the statistical correlation is estimated based on annotated medical image data that differs from at least one medical image of the received object; and

[0059] Identify at least two features of interest in at least one medical image based on regularized feature heatmaps.

[0060] According to another aspect of the present invention, a computer program unit is provided for controlling the device described above and below, wherein when the computer program unit is executed by a processing unit, the computer program unit is adapted to perform the steps of the method described above and below.

[0061] As used herein, the term "object" includes, for example, patients, human subjects, and animal subjects.

[0062] As used herein, the term "feature" can refer to a spatial and / or temporal feature. Examples of features of interest may include landmarks and temporal events. The term "landmark" can refer to a point, voxel, contour, surface, and / or volume in a medical image. In some examples, a landmark may refer to anatomical features such as folds, protrusions, ducts, blood vessels, etc., which may always be present in the tissue to indicate a specific structure or location. The term "temporal event" can refer to the temporal activity of a point, voxel, contour, surface, and / or volume, for example, after one or more stimuli. Temporal activity can include the movement, intensity change, and / or geometrical change of a point, voxel, contour, surface, and / or volume over time. In some examples, a temporal event may include the temporal activity of one or more anatomical features.

[0063] As used herein, the term "unit" can refer to an application, software, firmware, and / or circuit configured to perform any of the operations described above. Software can be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware can be embodied as code, instructions, or instruction sets and / or data hard-coded (e.g., non-volatile) in a storage device. As used herein, "circuit" can include (e.g., individually or in any combination) hard-wired circuitry, programmable circuitry (such as a computer processor including one or more individual instruction processing cores), state machine circuitry, and / or firmware storing instructions executed by programmable circuitry. Units can be embodied collectively or individually as circuitry forming part of a larger system, such as integrated circuits (ICs), application-specific integrated circuits (ASICs), system-on-a-chip (SoCs), desktop computers, laptops, tablets, servers, smartphones, etc.

[0064] As used in this article, the term "device" can refer to ICs, ASICs, SoCs, desktop computers, laptops, tablets, servers, smartphones, etc.

[0065] Furthermore, the terms "first," "second," "third," etc., used in the specification and claims are used to distinguish similar elements, but not necessarily to describe order or chronological sequence. It should be understood that such terms are interchangeable where appropriate.

[0066] These and other aspects of the invention will become apparent and will be illustrated from the embodiments described below. Attached Figure Description

[0067] These and other aspects of the invention will become apparent and further elucidated by reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, wherein

[0068] Figure 1 An example of a network structure for estimating the occurrence of heatmaps of landmarks of interest is shown schematically.

[0069] Figure 2 A flowchart of a feature detection method according to some embodiments of the present disclosure is shown.

[0070] Figure 3A The anatomical knowledge regarding the relative vector between two landmarks is shown.

[0071] Figure 3B The results of the naked CNN are shown, corresponding to the heatmaps of the two landmarks.

[0072] Figure 3C This shows the mutual priors imposed on the heatmap by the corresponding other.

[0073] Figure 3D The diagram shows the regularized CNN result obtained by multiplying the raw CNN heatmap with the corresponding prior.

[0074] Figure 4 An apparatus for feature recognition in medical images according to some embodiments of the present disclosure is shown.

[0075] Figure 5 A medical imaging system according to some embodiments of the present disclosure is shown.

[0076] It should be noted that these figures are schematic only and not drawn to scale. In these figures, elements corresponding to those already described may have the same reference numerals. Examples, embodiments, or optional features, whether or not indicated as non-limiting, should not be construed as limiting the claimed invention. Detailed Implementation

[0077] Feature detection, such as landmark detection, is an important task in medical imaging, enabling various applications such as region-of-interest cropping, image registration, biomarker measurement, model-based segmentation, and image quality assessment. With the advent of deep learning, convolutional neural networks (CNNs) have been applied to landmark detection.

[0078] Besides direct regression of the spatial coordinates of the landmarks of interest, estimating a heatmap of landmark occurrences across the entire image using a CNN, i.e., pseudoprobability, has shown promise. This is due to the improved detection accuracy and the fact that the resulting heatmap allows for deeper insights into the "inference" of deep CNNs, thus aiding in understanding this algorithm. In this approach, landmark coordinates are estimated from the inferred heatmap, for example, through the argmax operation.

[0079] Figure 1 An example network architecture for estimating the heatmap of landmarks of interest occurrences is shown, where one or more medical images are considered as input and the heatmap of landmarks of interest occurrences is used as output. An exemplary network architecture could be a fully convolutional network (FCN), such as... Figure 1 Take the U-shaped network in the example. This network structure includes a shrinking path (left) and an expanding path (right). The shrinking path follows the typical structure of a convolutional network. It involves repeatedly applying two 3x3 convolutions (unpadded convolutions), each followed by a rectified linear unit (ReLU) and a 2x2 max-pooling operation, with a stride for downsampling. At each downsampling step, the number of channels is doubled. Each step in the expanding path involves upsampling the heatmap, followed by a 2x2 convolution that halves the number of channels (“upconvolution”), concatenated with the corresponding cropped heatmap from the shrinking path, and two 3x3 convolutions, each followed by a ReLU. Finally, the probability map of the landmark locations, i.e., the heatmap, can be estimated using the trained model.

[0080] However, in practice, heatmap-based landmark detection may be prone to false positive predictions because CNNs may react to image intensity structures of different anatomical structures that are similar to the image intensity structure near the target landmark.

[0081] To solve this problem, Figure 2A flowchart of a method 100 for feature detection according to some embodiments of the present disclosure is shown. In step 110, i.e., step a), for example, at least one medical image of an object (e.g., a patient or animal) is received from a medical imaging device (such as an ultrasound imaging device) or from a picture archiving and communication system (PACS). This at least one medical image includes multidimensional images, such as two-dimensional (2D) medical images, three-dimensional (3D) medical images, and four-dimensional (4D) medical images. Examples of medical images include, but are not limited to, images acquired by at least one of magnetic resonance imaging (MRI), computed tomography (CT), ultrasound imaging, optical imaging, positron emission tomography (PET), and single-photon emission computed tomography (SPET). An example of a 4D medical image could be a four-dimensional computed tomography (4DCT), a type of CT scan that records multiple images changing over time. It allows for playback of the scan in video form, enabling observation of physiological processes and tracking of internal movements. Another example of a 4D medical image could be fMRI, which measures brain activity by detecting changes associated with blood flow.

[0082] In step 120, i.e., step b), a feature heatmap of at least two features of interest is generated by applying a CNN to at least one medical image. The CNN has been trained for feature recognition.

[0083] In some examples, at least two features of interest include at least two landmarks, such as points, voxels, contours, volumes, anatomical features, etc. The feature heatmap includes a landmark heatmap of at least two landmarks generated by applying a first CNN to at least one medical image. The first CNN has been trained for landmark recognition. Statistical correlation includes the spatial correlation between at least two landmarks, which serves as a first probabilistic prior for mutual regularization of the landmark heatmaps. The regularized landmark heatmap includes a regularized landmark heatmap.

[0084] In some examples, at least two features of interest include at least two temporal events, such as the temporal activities of points, voxels, contours, volumes, anatomical features, etc. The landmark heatmap includes a temporal event heatmap of at least two temporal events generated by applying a second CNN to at least one medical image. The second CNN has been trained for temporal event recognition. Statistical correlation includes the temporal correlation between the at least two temporal events to be detected, which serves as a second probabilistic prior for mutual regularization of the temporal event heatmap. The regularized landmark heatmap includes a regularized temporal event heatmap.

[0085] In some examples, the same CNN can be used for landmark and temporal event detection. In other words, the first CNN and the second CNN are the same CNN. Therefore, the same CNN can be used to identify spatial-temporal features, such as multidimensional (due to spatial dimensions) time series in fMRI signals.

[0086] Figure 3B An example of a feature heatmap is shown.

[0087] In step 130, i.e., step c), the correlation between at least two features of interest is used as a probabilistic prior for mutual regularization of the feature heatmap. The statistical correlation is estimated based on annotated medical image data (which can be obtained from an annotated medical image database).

[0088] For illustrative purposes, Figures 3A to 3D An exemplary implementation and usage of statistical correlation as a probabilistic prior for the case of two landmarks x1 and x2 is shown. Regularization can be implemented through the following steps.

[0089] First, the spatial correlation between at least two landmarks x1 and x2 can be estimated based on the conditional probability of detecting one landmark at a certain spatial location given that another landmark has already been detected at a certain spatial location. For example, the probability density of the relative vector between landmarks x2 and x1 can be estimated, for example, by kernel density estimation methods:

[0090]

[0091] Estimated from annotated training data, with attention paid to giving rare events sufficiently high weights to accurately describe anomalies. For example, Figure 3A The anatomical knowledge of the relative vector between the two landmarks x1 and x2 is shown (in this case, it is assumed that there is a expected distance of about 5.5 mm and a narrow orientation distribution).

[0092] Secondly, the first probability prior of a landmark can be estimated by convolving the conditional probability of the landmark at a certain spatial location with the heatmap result of another landmark, which indicates the pseudo-probability of the other landmark appearing at one or more spatial locations.

[0093] For example, the outputs q1 and q2 of a naked CNN can be convolved with previously obtained conditional probabilities:

[0094]

[0095]

[0096] Where * denotes convolution, and

[0097]

[0098] For example, Figure 3B The results of the raw CNN for the heatmaps corresponding to the two landmarks are shown, namely the raw CNN outputs q1 and q2. The heatmaps for the two landmarks are drawn using solid and dashed outlines, respectively. Figure 3C The mutual priors imposed on the heatmap by the corresponding other are shown. The heatmaps of the two landmarks are drawn with solid and dashed outlines, respectively.

[0099] Finally, probabilistic priors were used to mutually regularize the landmark heatmaps.

[0100] For example, the following methods can be used to regularize CNN results:

[0101]

[0102]

[0103] Here, λ represents an adjustable regularization parameter, ranging between 0 and 1. In this way, this probabilistic regularization scheme can be integrated into all existing CNN architectures by attaching convolutional layers as described above. This allows for two variations of applying the regularization layer: either applying it only at test time (i.e., as a post-processing step), or applying it during CNN training to stabilize predictions by learning features related to relevance.

[0104] For example, Figure 3D The regularized CNN result is shown by multiplying the raw CNN heatmap by the corresponding prior. The regularized CNN results for the two landmarks are plotted using solid and dashed contour lines, respectively.

[0105] Optionally, in order to impose mutual priors on the detection of more than two landmarks, this regularization can be based on the i-th landmark and the other N landmarks, i.e., the j-th, ..., j-th landmarks. N The spatial correlation between several landmarks (e.g., their N anatomical nearest neighbors) is approximated by decomposing them into binary correlations:

[0106]

[0107] Each factor on the right-hand side corresponds to the probability density of the corresponding relative coordinate, as described above. Therefore, the proposed method avoids the curse of dimensionality when estimating and evaluating these correlations. This leads to the generation of information about the j1, ..., j... factors... N The following priors are given to the heatmap of the i-th landmark induced by the heatmap of the i-th landmark:

[0108]

[0109] This allows the computation of this prior to be represented as an additional convolutional layer of a given CNN. This prior can be used to regularize the CNN result in various ways, for example, as the raw CNN result q with respect to the i-th landmark. i The convexity of q, as described above, is used for post-processing or during training. Furthermore, q is adjusted using multiple sets of additional landmarks (and appropriate regularization weights). i Regularization can incorporate more or less rigid spatial correlations into a CNN. The more additional landmarks are included in the mutual prior, the more restrictive that prior becomes.

[0110] In step 140, i.e. step d), at least two features of interest in at least one medical image are determined based on the regularized feature heatmap.

[0111] It should be understood that the above operations can be performed in any suitable order, such as consecutively, simultaneously, or a combination thereof, following a specific order required, for example, by the input / output relationship, where applicable.

[0112] The above discussion can also be applied to the detection of time events. In other words, similar regularization schemes can be applied to time events. In the case of time event detection, the temporal correlation between at least two time events can be estimated based on the conditional probability of detecting a time event at a certain time position given that another time event has already been detected at a certain time position. Optionally, the temporal correlation between a time event and two or more other time events can be estimated based on the product of conditional probabilities, where each conditional probability is the conditional probability of detecting said time event at a certain time position given that one of two or more other time events has already been detected at a certain time position. Optionally, a second probability prior for a time event can be estimated by convolving the conditional probability of said time event at a certain time position with the heatmap result of another time event, which indicates the pseudo-probability of said other time event occurring at one or more time positions.

[0113] Figure 4 A device 10 for feature recognition in medical images according to some embodiments of the present disclosure is shown. The device 10 includes an input unit 12 and a processing unit 14.

[0114] The input unit 10 is configured to receive at least one medical image of an object. The at least one medical image includes multidimensional images, such as two-dimensional, three-dimensional, or four-dimensional medical images. For example, the input unit may be a communication interface that communicates wirelessly or via a wired connection with a medical imaging device (such as MRI or PACS) to obtain one or more medical images.

[0115] Processing unit 14 is configured to generate feature heatmaps of at least two features of interest by applying a CNN to the at least one medical image. The CNN has been trained for feature recognition.

[0116] In some examples, at least two features of interest may include at least two landmarks. The feature heatmap may include a landmark heatmap of at least two landmarks generated by applying a first CNN to at least one medical image. The first CNN has been trained for landmark recognition. Statistical correlation may include the spatial correlation between at least two landmarks, which serves as a first probabilistic prior for mutual regularization of the landmark heatmaps. The regularized landmark heatmap may include a regularized landmark heatmap.

[0117] In some examples, at least two features of interest may include at least two temporal events. A landmark heatmap may include a temporal event heatmap generated by applying a second CNN to at least one medical image, representing at least two temporal events. The second CNN has been trained for temporal event recognition. Statistical correlation may include the temporal correlation between at least two temporal events, serving as a second probabilistic prior for mutual regularization of the temporal event heatmaps. A regularized landmark heatmap may include a regularized temporal event heatmap.

[0118] Processing unit 14 is further configured to use the correlation between at least two features of interest as a probabilistic prior for mutual regularization of the feature heatmap. The statistical correlation is estimated based on annotated medical image data (e.g., training data).

[0119] For landmark detection, the processing unit can be configured to estimate the spatial correlation between at least two landmarks based on the conditional probability of detecting a landmark at a certain spatial location given that another landmark has already been detected at a certain spatial location. Optionally, the processing unit can be configured to estimate the spatial correlation between a landmark and two or more other landmarks based on the product of conditional probabilities, where each conditional probability is the conditional probability of detecting the first landmark at a certain spatial location given that one of two or more other landmarks has already been detected at a certain spatial location. Optionally, the processing unit can be configured to estimate a first probability prior of the first landmark by convolving the conditional probability of the first landmark at a certain spatial location with the heatmap result of another landmark, the heatmap result indicating the spurious probability of the other landmark appearing at one or more spatial locations.

[0120] For time event detection, processing unit 14 can be configured to estimate the temporal correlation between at least two time events based on the conditional probability of detecting a time event at a certain time position given that another time event has already been detected at a certain time position. Optionally, processing unit 14 can be configured to estimate the temporal correlation between a time event and two or more other time events based on the product of conditional probabilities, where each conditional probability is the conditional probability of detecting the time event at a certain time position given that one of two or more other time events has already been detected at a certain time position. Optionally, processing unit 14 can be configured to estimate a second probability prior of the time event by convolving the conditional probability of the time event at a certain time position with the heatmap result of another time event, the heatmap result indicating the pseudo-probability of the other time event occurring at one or more time positions.

[0121] Optionally, the processing unit 14 can be configured to determine statistical correlation based on the training data using a kernel density estimation method for the relative vector between any pair of features of interest.

[0122] Alternatively, the computation of probabilistic priors can be represented as additional convolutional layers in the CNN. This allows probabilistic regularization schemes to be integrated into all existing CNN architectures via additional convolutional layers.

[0123] Optionally, the probabilistic prior is used to mutually regularize the feature heatmap in at least one of the following ways: applying a regularization layer at test time during post-processing steps, and applying a regularization layer during CNN training.

[0124] The processing unit 14 is further configured to identify at least two features of interest in at least one medical image based on a regularized feature heatmap.

[0125] The device also includes an output unit 16, such as a communication interface, for outputting results to, for example, a display.

[0126] Figure 5 A medical imaging system 200 according to some embodiments of the present disclosure is shown, which is in the form of a C-arm X-ray imaging kit. Other examples of medical imaging systems may include, but are not limited to, MRI systems, ultrasound imaging systems, optical imaging systems, PET imaging systems, and SPECT imaging systems.

[0127] The C-arm imaging system 200 has a support structure 202 that can be translated about the object of interest 204 via azimuth and elevation axes. For example, the C-arm X-ray imaging system 200 can be supported by the ceiling of an X-ray facility. The support structure holds the rotating anode X-ray source 205 and the X-ray detector 206.

[0128] C-arm imaging systems (or CT imaging systems) may optionally be equipped with motion sensors (e.g., rotary encoders in the C-arm or CT gantry axis). This allows motion information to be fed back to the status detector of the X-ray imaging system. Alternatively, or in combination, the status detector of the X-ray imaging system is configured to receive a list of motion commands representing a pre-planned imaging scheme.

[0129] The C-arm X-ray imaging system is controlled, for example, by a console 208, which includes, for example, a display screen 210 and a computer device 212, which can be used as a stator control system and is controlled by a keyboard 214 and a mouse 216.

[0130] The C-arm 218 is configured to translate around the object of interest 204, not only in the sense of planar rotation (in the sense of a CT scanner), but also by tilting.

[0131] The aforementioned device 10 can be embodied as a computer device 212 for feature detection.

[0132] In operation, the object of interest 204 is placed between the detector 206 and the X-ray source 205 of the C-arm imaging system 200. The C-arm can rotate around the patient to acquire an image dataset, which is then used for 3D image reconstruction. The device 10, embodied as a computer device 212, is configured to identify at least two landmarks of interest in the image dataset.

[0133] In another exemplary embodiment of the present invention, a computer program or computer program unit is provided, characterized in that it is adapted to perform the method steps of the method according to one of the foregoing embodiments on a suitable system.

[0134] Therefore, a computer program unit can be stored on a computer unit, which can also be part of an embodiment of the present invention. The computing unit can be adapted to perform or induce the execution of the steps of the above-described method. Furthermore, it can be adapted to operate components of the above-described device. The computing unit can be adapted to automatically operate and / or execute user commands. The computer program can be loaded into the working memory of a data processor. Therefore, the data processor can be configured to execute the method of the present invention.

[0135] This exemplary embodiment of the invention covers both computer programs that use the invention from the outset and computer programs that convert existing programs into programs that use the invention through updates.

[0136] Furthermore, the computer program unit is capable of providing all the necessary steps to complete the operation of the exemplary embodiments of the method described above.

[0137] According to another exemplary embodiment of the present invention, a computer-readable medium, such as a CD-ROM, is provided, wherein the computer-readable medium has computer program units stored thereon, which are described in the foregoing portion.

[0138] Computer programs may be stored and / or distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0139] However, computer programs can also be provided via networks like the World Wide Web and can be downloaded from such networks to the working memory of a data processor. According to another exemplary embodiment of the invention, a medium is provided for making a computer program unit downloadable, the computer program unit being arranged to perform the method described in one of the foregoing embodiments of the invention.

[0140] It should be noted that embodiments of the invention have been described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the foregoing and hereinafter description that, unless otherwise indicated, any combination of features related to different subjects, in addition to any combination of features belonging to one type of subject, is also considered to be disclosed in this application. However, all features can be combined to provide a synergistic effect that is more than the simple sum of the features.

[0141] Although the invention has been illustrated and described in the accompanying drawings and the foregoing description, such illustrations and descriptions are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the dependent claims, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed invention.

[0142] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can perform the functions of several items recited in the claims. The recitation of certain measures only in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for feature recognition in medical images, comprising: Input unit (12); Processing unit (14); and Output unit (16); The input unit is configured to receive at least one medical image of an object, wherein the at least one medical image includes a multidimensional image; The processing unit is configured to: By applying a convolutional neural network (CNN) to the at least one medical image, feature heatmaps of at least two features of interest are generated, wherein the CNN has been trained for feature recognition. The statistical correlation between the at least two features of interest is used as a probabilistic prior for mutual regularization of the feature heatmap, wherein the statistical correlation is estimated based on annotated medical image data; and The at least two features of interest in the at least one medical image are identified based on the regularized feature heatmap; and The output unit is configured to output the result of the feature recognition.

2. The device according to claim 1, in, The at least two features of interest include at least two landmarks; The feature heatmap includes a landmark heatmap of the at least two landmarks generated by applying a first CNN to the at least one medical image, wherein the first CNN has been trained for landmark recognition. The statistical correlation includes the spatial correlation between the at least two landmarks, which serves as a first probabilistic prior for mutual regularization of the landmark heatmaps; and The regularized feature heatmap includes a regularized landmark heatmap.

3. The device according to claim 2, in, The processing unit is configured to estimate the spatial correlation between the at least two landmarks based on the conditional probability of detecting a landmark at a certain spatial location given that another landmark has already been detected at a certain spatial location.

4. The device according to claim 2 or 3, in, The processing unit is configured to estimate the spatial correlation between a landmark and two or more other landmarks based on the product of conditional probabilities, wherein each conditional probability is the conditional probability of detecting the landmark at a certain spatial location given that one of the two or more other landmarks has already been detected at a certain spatial location.

5. The device according to claim 3, in, The processing unit is configured to estimate the first probability prior of a landmark by convolving the conditional probability of a landmark at a certain spatial location with the heatmap result of another landmark, the heatmap result indicating the pseudo-probability of the other landmark appearing at one or more spatial locations.

6. The device according to any one of claims 1 to 3 and 5, in, The at least two features of interest include at least two temporal events; The feature heatmap includes a time event heatmap of the at least two time events generated by applying a second CNN to the at least one medical image, wherein the second CNN has been trained for time event recognition. The statistical correlation includes the temporal correlation between at least two time events to be detected, which serves as a second probabilistic prior for mutual regularization of the time event heatmap; and The regularized feature heatmap includes a regularized time event heatmap.

7. The device according to claim 6, in, The processing unit is configured to estimate the temporal correlation between the at least two time events based on the conditional probability of detecting a time event at a certain time location given that another time event has already been detected at a certain time location.

8. The device according to claim 6, in, The processing unit is configured to estimate the temporal correlation between a time event and two or more other time events based on the product of conditional probabilities, wherein each conditional probability is the conditional probability of detecting the time event at a certain time position given that one of the two or more other time events has already been detected at a certain time position.

9. The device according to claim 7, in, The processing unit is configured to estimate a second probability prior of a time event by convolving the conditional probability of a time event at a certain time location with a heatmap result of another time event, the heatmap result indicating the pseudo probability of the other time event occurring at one or more time locations.

10. The device according to any one of claims 1 to 3, 5, 7, and 9, in, The processing unit is configured to determine the statistical correlation based on the training data using a kernel density estimation method for the relative vector between any pair of two features of interest.

11. The device according to any one of claims 1 to 3, 5, 7, and 9, in, The calculation of the probability prior is represented as an additional convolutional layer of the CNN.

12. The device according to any one of claims 1 to 3, 5, 7, and 9, in, The probabilistic prior is used to mutually regularize the feature heatmap in at least one of the following ways: A regularization layer is applied during the post-processing step at test time; and A regularization layer is applied during the training of the CNN.

13. A medical imaging system (200), comprising: Medical imaging equipment; as well as The device according to any one of claims 1 to 12; The medical imaging device is configured to acquire at least one medical image of the patient; and The device is configured to identify the at least two landmarks of interest in the at least one medical image.

14. A method (100) for feature recognition in medical images, comprising: Receive at least one medical image of the object (110), wherein the at least one medical image includes a multidimensional image; By applying a convolutional neural network (CNN) to the at least one medical image, (120) feature heatmaps of at least two features of interest are generated, wherein the CNN has been trained for feature recognition. The statistical correlation between the at least two features of interest described in (130) is used as a probabilistic prior for mutual regularization of the feature heatmap, wherein the statistical correlation is estimated based on annotated medical image data; and Based on the regularized feature heatmap, identify (140) the at least two features of interest in the at least one medical image.

15. A computer program product for controlling a device according to any one of claims 1 to 12 and comprising a computer program, wherein when the computer program is executed by a processing unit, the computer program is adapted to perform the steps of the method according to claim 14.

Images