3D image processing

Abstract

A mechanism for determining the location of a desired plane (230) within a 3D image, using a machine learning algorithm to determine a first set of one or more offsets (251, 252) between an initial position (221, 222) within a 2D slice (210) of the 3D image and a predetermined position within the desired plane, and a second set of one or more offsets (261, 262) between the same initial position and the position of an anatomical landmark (240). The offsets are spatial offsets between the initial position and another position.

Description

[Technical Field]

[0001] The present invention relates to the field of 3D images, and in particular to the processing of 3D images. [Background technology]

[0002] In the medical field, there is growing interest in the use of 3D images (i.e., 3D medical images) to evaluate and / or analyze target anatomical structures. In particular, 3D images have been identified as providing additional contextual information to aid in the analysis of target anatomical structures. Summary of the Invention [Problem to be solved by the invention]

[0003] However, the large amount of information and detail contained in a 3D image can affect the ease of analysis by a clinician. Additionally, some standardized tests for analyzing a target anatomical structure require the use of a 2D image that includes certain views of the target anatomical structure and / or certain anatomical landmarks within the anatomical structure.

[0004] For example, in fetal screening, abdominal circumference (AC) is a standard measurement for estimating fetal size and growth. Existing screening guidelines define criteria that 2D images must meet to allow valid and comparable biometry measurements. For example, the stomach and umbilical veins must not reveal the heart and / or kidneys, the scan plane of the 2D image must be perpendicular to the head-to-toe axis, and the shape of the abdomen must be as circular as possible. [Means for solving the problem]

[0005] The invention is defined by the independent claims, the dependent claims defining advantageous embodiments.

[0006] According to an example according to one aspect of the present invention, a computer-implemented method for processing a 3D image of a target anatomical structure is provided.

[0007] The computer-implemented method includes acquiring 2D slices of the 3D image, processing the 2D slices using object detection techniques to identify a boundary region containing a representation of the target anatomical structure, defining a set of initial positions within the boundary region, and processing the 3D image using machine learning methods to predict, for each initial position, one or more first predicted offsets, each first predicted offset being a predicted spatial offset relative to the 3D image between the initial position and a predetermined position within a desired plane of the 3D image, and one or more second predicted offsets, each second predicted offset being a predicted spatial offset relative to the 3D image between the initial position and a position of an anatomical landmark of the target anatomical structure outside the desired plane.

[0008] The proposed approach provides a technique for identifying or defining a spatial offset between a 2D slice and a desired 2D plane or anatomical landmark within a 3D image. The defined spatial offset can be used, for example, to generate a 2D image containing a representation of the desired plane or landmark and / or to provide guidance on how to move / modify an image capture system to capture an image containing the desired plane or landmark.

[0009] Herein, advanced techniques utilize an initial position defined within a bounded region containing a representation of the target anatomy, which provides a consistent baseline for defining any spatial offsets to facilitate modification of the image generation and / or image capture system.

[0010] The bounding region may have a predetermined shape, for example, a rectangle, a circle, or a square. The bounding region defines a predicted region that contains a representation of the target anatomical structure.

[0011] Once a set of one or more first expected offsets is generated for each initial position, a set of one or more first expected offsets (ie, multiple sets) is generated.

[0012] The method may further include processing the set of initial positions, the set of one or more first predicted offsets, and the 3D image to generate at least one image smaller than the 3D image that includes one or more representations of the one or more desired positions. The present disclosure recognizes that the 3D image likely contains sufficient information to generate an image (e.g., a 2D image) that includes the desired positions or view planes. Thus, an automated approach is provided for providing an image of a desired view of an anatomical structure without requiring movement or additional input from a clinician or operator of the image capture system.

[0013] In some examples, each initial position is associated with a different respective desired position within a desired plane of the 3D image for viewing the target anatomical structure, and for each initial position, the set of one or more first predicted offsets comprises a single predicted offset that is a predicted spatial offset between the initial position and its respective desired position.

[0014] Optionally, the method further includes estimating a position of the desired plane in the 3D image by processing the set of initial positions and the set of first predicted offsets generated for each initial position, and processing the 3D image using the estimated position of the desired plane to generate a 2D image representing the desired plane.

[0015] In this manner, a 2D image is generated that includes a representation of the desired view of the anatomy, which is particularly advantageous when one or more 2D views are essential or necessary to perform a standardized analysis or monitoring technique.

[0016] Estimating the location of the desired plane within the 3D image may be performed using linear regression techniques by processing a set of initial positions and a first set of predicted offsets generated for each initial position.

[0017] Each second predicted offset may be a predicted spatial offset relative to the 3D image between the initial position and the position of the same first anatomical landmark of the target anatomical structure.

[0018] Preferably, each set of one or more second predicted offsets includes a plurality of second predicted offsets.

[0019] In some examples, defining a set of initial locations within the bounded region includes defining a grid of cells within the bounded region and identifying a single location within each cell to thereby define the multiple initial locations. Preferably, the grid is a regular grid (i.e., having cells of equal or approximately equal size). Each cell may be rectangular and / or square. The single location within each cell may be a center point of the cell, or an edge or corner point of the cell.

[0020] The 3D image may be defined by a stack of initial 2D slices, and the 2D slice may be closer to the central-most slice of the stack of initial 2D slices than either the first or last 2D slice in the stack of initial 2D slices. For example, the 2D slice may be the central-most slice of the stack of 2D slices.

[0021] One realization of this embodiment is that during capture of a 3D image by an operator of an image capture system, it can be assumed that the operator controls the image capture system so that the desired plane and / or anatomical landmark is approximately centered relative to the 3D image. This means that more central slices of the 3D image are more likely to be only partially offset relative to the desired plane and / or landmark. Therefore, the proposed technique is more accurate when operating under this valid assumption.

[0022] Processing the 2D slices using the machine learning method may further include generating, for each predicted offset, a confidence value that represents the reliability or certainty of the predicted offset.

[0023] In some examples, the method further includes using a linear regression technique to estimate the location of the desired plane within the 3D image by processing the set of initial positions and the set of first predicted offsets generated for each initial position, wherein values ​​of one or more weighting parameters of the linear regression technique are responsive to confidence values ​​of the one or more predicted offsets.

[0024] Also proposed is a computer-implemented method for training a machine learning method to predict the location of a desired plane in a desired 3D image.

[0025] The computer-implemented method includes obtaining a training dataset including a plurality of training entries, each training entry including an exemplary 3D image of a target anatomical structure; first information identifying a location of a desired plane within the exemplary 3D image; and second information identifying a location of one or more anatomical landmarks outside the desired plane within the exemplary 3D image; and using the training dataset, training a machine learning method to predict the information identifying the location of the desired plane within the desired 3D image and the information identifying the location of one or more anatomical landmarks outside the desired plane within the desired 3D image.

[0026] A computer program product is also proposed comprising computer program code which, when executed on a computing device having a processing system, causes the processing system to perform all of the steps of any of the methods described herein.

[0027] A processing system configured to carry out the methods described herein is also proposed.

[0028] Therefore, a processing system for processing a 3D image of a target anatomical structure is proposed, the processing system being configured to perform the steps of acquiring 2D slices of the 3D image, processing the 2D slices using an object detection technique to identify a boundary region containing a representation of the target anatomical structure, defining a set of initial positions within the boundary region, and processing the 3D image using a machine learning method to predict, for each initial position, a set of one or more first predicted offsets that are predicted spatial offsets between the initial position of the 3D image and a predetermined position within the desired plane, and a set of one or more second predicted offsets that are predicted spatial offsets between the initial position of the 3D image and a position of an anatomical landmark outside the desired plane.

[0029] In some examples, the processing system is further configured to process the set of initial positions, at least one set of one or more predicted offsets, and the 3D image to generate at least one image smaller than the 3D image that includes one or more representations of the one or more desired positions.

[0030] A processing system is proposed for training a machine learning method for predicting the location of a desired plane within a desired 3D image, the processing system being configured to acquire a training dataset including a plurality of training entries, each training entry configured to train the machine learning method using the training dataset to predict an exemplary 3D image of a target anatomical structure, first information identifying the location of the desired plane within the exemplary 3D image, second information identifying the location of one or more anatomical landmarks outside the desired plane within the exemplary 3D image, and information identifying the location of the desired plane within the desired 3D image.

[0031] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0032] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0033] [Figure 1] 1 is a flowchart illustrating a method according to one embodiment. [Figure 2] Shows the relationship between different locations in a 3D image. [Figure 3] 1 illustrates an architecture of a machine learning algorithm for use in embodiments. [Figure 4] 10 is a flowchart illustrating a method according to a further embodiment. [Figure 5] 10 is a flowchart illustrating a method according to another embodiment. [Figure 6] FIG. 2 is a schematic diagram of a processor according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0034] The present invention will now be described with reference to the drawings.

[0035] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the figures are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0036] The present invention provides a mechanism for determining the location of a desired plane within a 3D image. A machine learning algorithm is used to determine a first set of one or more offsets between an initial position within a 2D slice of the 3D image and a predetermined position within the desired plane, and a second set of one or more offsets between the same initial position and the positions of anatomical landmarks. The offsets are spatial offsets between the initial position and another position.

[0037] The present disclosure recognizes that a machine learning algorithm trained to simultaneously identify an offset to a desired plane and an offset to at least one anatomical landmark performs more accurate identification of the offset to the desired plane, which can be used, for example, to improve the accuracy of identification of the desired plane in a 3D image for generating a 2D image in the desired plane.

[0038] Embodiments can be used in any scenario where it is desirable to identify a 2D plane in a 3D medical image. One exemplary environment is fetal monitoring and / or analysis.

[0039] Figure 1 is a flow chart illustrating the overall approach for processing 3D images employed by the present invention. Figure 1 shows a computer-implemented method 100 by which 3D images of a target anatomical structure are processed.

[0040] The method 100 includes a step 110 of acquiring a 2D slice of the 3D image. A 2D slice is a subplanar selection of the 3D image. In particular, a 2D slice may be a slice perpendicular to an axis of the 3D image. Thus, if the 3D image has size X×Y×Z, the 2D slice has size X×Y.

[0041] The acquired 2D slice is preferably closer to the central-most slice of the 3D image. Thus, if the 3D image is defined by a stack of initial 2D slices, the acquired 2D slice is closer to the central-most slice of the stack of initial 2D slices than either the first or last 2D slice in the stack of initial 2D slices. Even more preferably, the acquired 2D slice is the central-most slice of the 3D image.

[0042] The method 100 also includes a step 120 of processing the 2D slices using an object detection technique to identify a boundary region that includes a representation of the target anatomy (i.e., includes a 2D image of the target anatomy).

[0043] A bounding region can have a defined shape (e.g., rectangular or circular) and is well established in the art. If it has a rectangular shape, the bounding region is commonly labeled a bounding box.

[0044] 2D object detection is a well-analyzed problem in computer vision, especially for generating boundary regions that identify desired objects or structures. Therefore, a wide variety of approaches exist for performing step 120. One suitable example is the "you-only-look-once" (YOLO) algorithm exemplified by Joseph Redmon and Ali Farhadi: YOLO9000: Better, Faster, Stronger, CVPR 2018. Another suitable approach is proposed by Wang, Guotai et al., "Interactive medical image segmentation using deep learning with image-specific fine tuning," IEEE Transactions on Medical Imaging 37.7 (2018): 1562-1573.

[0045] The size of the boundary region is smaller than the size of the 2D slice. In particular, the boundary region is sized and positioned to define the boundaries of the representation of the target anatomical structure within the 2D slice.

[0046] The method 100 also includes a step 130 of defining a set of initial positions within the bounded region.

[0047] Step 130 may be performed, for example, by defining a grid of cells within the bounding region and identifying a single location within each cell, thereby defining multiple initial locations. The grid may be a regular grid (i.e., having cells of equal or approximately equal size). Each cell may be rectangular and / or square, for example, depending on the size and / or shape of the bounding region. An initial location may be defined for each cell, for example, located at the center point of the cell or at a predetermined or predefined corner of the cell.

[0048] If used, the grid of cells may be of size N x M. Preferably, N = M.

[0049] Method 100 also performs step 140 of processing the 3D image using a machine learning method to predict a second position for each initial position, i.e., a set of one or more first predicted offsets and one or more second predicted offsets.

[0050] The predicted offset is the predicted spatial offset between the initial position and another position.

[0051] A spatial offset defines the distance and direction between two points in 3D space, i.e., relative to a 3D image. The spatial offset may, for example, be the formation of a vector that identifies the direction and distance between two related positions (e.g., between an initial position and a predetermined position, or between an initial position and the position of an anatomical landmark). The vector may, for example, be formatted in a Cartesian or polar format. Other suitable data formats for defining a spatial offset between two points in a 3D image will be apparent to those skilled in the art.

[0052] Each first predicted offset is a predicted spatial offset relative to the 3D image between the initial position and a predetermined position within a desired plane of the 3D image.

[0053] The desired plane may represent a plane of a 3D image that includes a desired view and / or cross-sectional representation of the target anatomy. For example, if the target anatomy is a fetus, the desired plane may be a plane perpendicular to the head-to-toe axis that shows the stomach and umbilical cord.

[0054] Each second predicted offset is a predicted spatial offset relative to the 3D image between the initial position and the position of an anatomical landmark outside the desired plane of the target anatomy.

[0055] In some embodiments, each set of one or more second predicted offsets comprises multiple second predicted offsets, each second predicted offset in the same set being a predicted spatial offset relative to the 3D image between the initial position and the position of the same anatomical landmark outside the desired plane of the target anatomical structure.

[0056] Thus, any given set of second predicted offsets may comprise multiple second predicted offsets between the same initial position and the position of the same anatomical landmark, with different sets associated with different initial positions.

[0057] Examples of anatomical landmarks may depend on the type of target anatomical structure. For example, if the target anatomical structure is a fetus, the anatomical landmarks may be, for example, the fetal eyes, the fetal fingers or toes, and / or the base of the fetal spine. As another example, if the target anatomical structure is the heart, the anatomical landmarks may be the entrance to particular blood vessels or chambers for the heart. Other suitable examples will be readily apparent to those skilled in the art.

[0058] Thus, the machine learning method is a machine learning method or technique that is trained to perform simultaneous identification of the relative position of the desired plane and the relative positions of the anatomical landmarks. It has been recognized that machine learning methods trained to perform simultaneous identification of both of these features have improved performance in identifying information about the desired plane. This means that there is improved performance for identifying the position of the desired plane (i.e., determining the first set of offsets).

[0059] In a preferred example, each initial position is associated with a different respective desired location within a desired plane of the 3D image for viewing the target anatomical structure, and for each initial position, the set of one or more first predicted offsets comprises a single predicted offset that is a predicted spatial offset between the initial position and its respective desired location.

[0060] Each desired location may effectively represent a particular portion or region of a desired plane of the 3D image.

[0061] To improve the performance of step 140, the input to the machine learning method for performing step 140 may be a portion or portions of the 3D image defined by the geometry of the bounding region. The geometry of the bounding region defines the size, shape, and location of the bounding region. Thus, the physical locations of the voxel values ​​supplied to the machine learning method for performing step 140 may depend on the bounding region.

[0062] FIG. 2 provides a visual representation of the procedure performed by steps 110-140.

[0063] 2 shows a 2D slice 210 in which a boundary region 220 has been identified by processing the 2D slice using a suitable object detection technique. The size of the boundary region 220 is smaller than the size of the 2D slice.

[0064] A bounding region 220 is used to define a number of initial positions 221, 222. In the illustrated example, each initial position 221, 222 is at the center point of a cell within a regular grid of cells.

[0065] The relative positions of the desired plane 230 and anatomical landmarks 240 are also shown conceptually.

[0066] The purpose of step 140 is to predict or determine one or more first offsets 251, 252 between each initial position 221, 222 and a predetermined position (for that initial position) in the desired plane 230, and one or more second offsets 261, 262 between each initial position 221, 222 and an anatomical landmark 240.

[0067] 2, each initial position may be associated with only a single first offset 251 that defines the offset between the initial position 221 and a corresponding predetermined position 231 in the desired plane 230. Thus, each initial position 221 maps to or corresponds to a predetermined position 231 in the desired plane 230.

[0068] It will be apparent that for clarity of explanation, only a subset of all possible initial positions, first offsets, and second offsets are shown and / or labeled, and those skilled in the art will readily understand how many more positions and / or offsets exist in practice.

[0069] The proposed embodiment utilizes a machine learning algorithm in processing the 3D image to generate the first and second set of offsets. A machine learning algorithm is any self-training algorithm that processes input data to generate or predict output data, where the input data includes the 3D image (or a portion thereof) and the output data includes the first and second set of offsets.

[0070] Machine learning algorithms suitable for use in the present invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms, such as logistic regression, support vector machines, or naive Bayesian models, are suitable alternatives.

[0071] However, one particularly advantageous form of machine learning algorithm for generating the first and second offsets (i.e., for use in step 140) is a neural network. Based on current understanding, neural networks appear to be by far the best existing choice because of their ability to perform difficult learning tasks while meeting near real-time performance.

[0072] One particularly advantageous approach utilizes multi-resolution neural networks as machine learning algorithms, which have been shown to yield state-of-the-art accuracy for several segmentation tasks in medical images.

[0073] Standard multiresolution neural networks utilize the concept of combining image patches of different resolutions for pixel-by-pixel classification. At each resolution level, image features are extracted using standard convolutional layers. The feature maps of the coarser levels are then successively upsampled and combined with the next finer level. In this way, a segmentation of the original image resolution is obtained while having a large perceptual field to take into account the broader image context.

[0074] For purposes of this disclosure, this general concept will be used for use in generating or predicting offsets. Thus, when used, a multi-resolution neural network is trained and / or configured to have an output layer of size N×M×P to hold the offsets, where P is the number of offsets generated for each initial position (N×M equals the number of initial positions).

[0075] It will be understood that, by design, the output layer containing the offset will have a much smaller resolution than the input layer to the multi-resolution neural network. Therefore, a standard multi-resolution neural network (discussed above) should be modified to utilize a downsampling strategy rather than an upsampling strategy. This means that instead of upsampling coarse levels and combining them with finer levels, as in the architecture described above, the fine levels are successively downsampled and combined with coarser levels.

[0076] Methods for training machine learning algorithms are well known. Typically, such methods involve obtaining a training dataset that includes training input data entries and corresponding training output data entries. This is commonly referred to as "ground truth data." An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error (e.g., mean squared difference) between the predicted output data entry and the corresponding training output data entry is used to correct the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entries are sufficiently similar (e.g., within ±1%) to the training output data entries. This is commonly known as a supervised learning technique.

[0077] FIG. 3 illustrates the architecture of a multi-resolution neural network 300 that can be used in embodiments of the present invention.

[0078] Neural network 300 generates output data 350 that includes a set of one or more first offsets for each initial position and a set of one or more second offsets for each initial position.

[0079] Where appropriate, blocks labeled "C" represent convolution processes performed on the data (e.g., a process involving convolution, batch normalization, and ReLU (rectified linear units), also known as CBR (Convolution-BatchNorm-ReLU)). Similarly, blocks labeled "P" represent pooling operations performed on the data. Blocks labeled D represent downsampling processes, which may themselves be pooling processes, such as max pooling or average pooling processes.

[0080] Image data 311 is processed to generate output data. The image data 311 includes 3D image data and / or portions thereof. The image data is successively downsampled to generate smaller resolution downsampled image data 312, 313, 314. At each resolution level, image features are extracted using one or more standard convolutional layers (in a convolution process C). The feature maps of the finer levels are then successively downsampled (e.g., using a pooling procedure P) and combined, e.g., summed, with the next coarser level.

[0081] FIG. 4 illustrates a method 400 for utilizing the generated offset, according to one embodiment.

[0082] The method 400 includes performing the method 100 described above to generate the offset.

[0083] The method 400 also includes a step 410 of estimating the location of the desired plane within the 3D image by processing the set of initial positions and the set of one or more first predicted offsets generated for each initial position.

[0084] Step 410 may be performed, for example, by processing the set of initial positions and the first set of predicted offsets generated for each initial position using linear regression techniques to identify the position / location of the desired plane within the 3D image.

[0085] For example, let each initial position in a 2D slice (in the plane x-y) be g xy Assuming that each initial position can be expressed as pxy The first offset o pxy represents the determined offset between the initial position and the corresponding predetermined position in the desired plane p. xy +o pxy Ao pxy :=(0,0,o pxy ) TThe desired plane can be located in the 3D image by interpolating with

[0086] Note that for each initial position, the corresponding predetermined position is the desired plane and the initial position g in the direction perpendicular to the plane of the 2D slice. xy Therefore, the single offset o pxy can be used to estimate the offset from the initial position to the desired plane.

[0087] An approach for performing linear regression to determine the location of the desired plane is disclosed in Schmidt-Richberg, Alexander et al., "Offset regression networks for view plane estimation in 3D fetal ultrasound," Medical Imaging 2019: Image Processing. Vol. 10949. SPIE, 2019. This same publication also discloses an approach for determining the offset.

[0088] In this scenario, each initial location is associated with a (single) different respective desired position within a desired plane of the 3D image for viewing the target anatomical structure, and thus, for each initial location, the set of one or more first predicted offsets comprises (only) a single predicted offset, which is the predicted spatial offset between the initial location and its respective desired position.

[0089] Using the estimated location of the desired plane, the method 400 can then perform step 420 of processing the 3D image to generate a 2D image representing the desired plane, which can effectively include extracting slices of the 3D image to form the 2D plane.

[0090] It will be appreciated that it is not necessary that step 420 be performed. Rather, other uses can be made of the determined location of the desired plane within the 3D image.

[0091] For example, the determined position may be used to guide an individual to move the image capture system to more accurately capture the desired plane.

[0092] In another use case scenario, the determined positions can be used to initialize a deformable segmentation model for performing segmentation of the 3D image. Thus, the method 400 can include a step 430 of processing the 3D image using an image segmentation technique, using the removed positions of the desired planes to initialize the segmentation.

[0093] It will be appreciated that one or more second sets of offsets may be used to similar effect to identify the location of anatomical landmark(s), which may be used, for example, to generate one or more images including representations of the anatomical landmarks.

[0094] This approach is conceptually illustrated by FIG. 5, which shows a method according to one embodiment.

[0095] Method 500 also includes performing method 100 described above to generate offsets. In this approach, each set of one or more second offsets generated for each initial position includes an offset between the corresponding initial position and the same first anatomical landmark. Thus, each set of one or more second offsets represents an offset to the same anatomical landmark.

[0096] The method 500 may then comprise processing 510 each set of one or more second offsets to estimate or identify the location of the anatomical landmark l.

[0097] For example, let each initial position in a 2D slice (in the plane x-y) be g xy, where each initial position is a second offset o (representing a spatial offset relative to a first anatomical landmark l) lxy The position of the first anatomical landmark in the 3D image can be related to, for example, g across all initial positions. xy + o lxy The location of the first anatomical landmark can be determined by calculating the mean or median of (instead of determining the mode or median), other interpolation procedures can be used to predict the location of the first anatomical landmark.

[0098] The method 500 can then perform a step 520 of processing the 3D image using the determined location of the first anatomical landmark to generate an output image (smaller than the 3D image) that includes a representation of the first anatomical landmark. This can include, for example, selecting a slice of the 3D image that includes the determined location of the first anatomical landmark. Alternatively, this can include extracting a smaller 3D image from the 3D image that includes the determined location of the first anatomical landmark therein.

[0099] Likewise, it will be appreciated that it is not necessary that step 520 be performed. Rather, other uses can be made of the determined locations of the anatomical landmarks.

[0100] For example, the determined positions may be used to guide an individual to move an image capture system to more accurately capture anatomical landmarks.

[0101] In another use case scenario, the determined locations can be used to initialize a deformable segmentation model for performing segmentation of the 3D image. Accordingly, the method 500 can include a step 530 of processing the 3D image using an image segmentation technique, using the determined locations of the anatomical landmarks to initialize the segmentation.

[0102] As mentioned above, in some embodiments, each set of one or more second predicted offsets comprises multiple second predicted offsets, and each second predicted offset in the same set is a predicted spatial offset relative to the 3D image between the initial position and the position of the same anatomical landmark outside the desired plane of the target anatomical structure.

[0103] The same averaging procedure can be used to determine the location of the first anatomical landmark in the 3D image.

[0104] It will be appreciated that the proposed technique can be employed to generate images for each of multiple anatomical landmarks.

[0105] In particular, the machine learning method may be configured to generate, for each initial position and each anatomical landmark, a set of one or more predicted offsets, where each offset in the same set represents an offset between the initial position and an additional anatomical landmark. For each anatomical landmark, the set of predicted offsets associated therewith may then be processed to determine or predict the position of the anatomical landmark in the 3D image, for example, using the approaches described above.

[0106] The determined locations can be used to similar effects as determining a single landmark, for example to initialize an image segmentation procedure or to guide the operator to generate new image data that more completely captures the landmark.

[0107] Of course, in some embodiments, both methods 400 and 500 may be performed.

[0108] Referring at least to FIG. 1 , in the aforementioned approach, the machine learning algorithm outputs, for each initial position, a plurality of offsets, including at least one or more first set of offsets and one or more second set of offsets.

[0109] In some examples, processing 140 the 2D slices using the machine learning method further includes generating, for each predicted offset, a confidence value that represents the reliability or certainty of the predicted offset.

[0110] Approaches for determining the confidence value of predicted data using machine learning methods are well known in the art. Exemplary approaches are described in Kendall, A., Gal, Y., 2017, "What uncertainties do we need in Bayesian deep learning for computer vision?", in: Advances in Neural Information Processing Systems, and van der Waa, Jasper et al., "ICM: an intuitive model independent and accurate certainty measure for machine learning." ICAART (2). 2018.

[0111] When the predicted offsets are used to determine the location of a desired plane and / or the location of a desired anatomical feature in a 3D image, the confidence values ​​may be used during the determination process, and in particular, the confidence values ​​may be used to weight the value of each offset used in determining the location.

[0112] Therefore, for each offset, a confidence value can be estimated to represent the confidence of the machine learning algorithm in its estimation. These confidence values ​​can then be used for weighted interpolation of planes / landmarks.

[0113] For example, if the location of the desired plane is found using linear regression, g xy + o pxy Ao pxy :=(0,0,o pxy ) TIf the confidence values ​​are located by interpolating with , the confidence values ​​may be used to perform a weighted linear regression, for example a weighted least squares method. The value of the weight or weighting factor may be equal to the confidence value.

[0114] As another example, the location of the first anatomical landmark in the 3D image is g across all initial positions. xy + o lxy If the mean or median is determined by calculating the mean or median of the data, the weighted mean or median may be determined using an appropriate formula (e.g., weighted arithmetic mean). The weighting used in determining the mean / median using such techniques may be a confidence value.

[0115] The previously described embodiments propose an approach that uses machine learning algorithms to determine or predict the offset from an initial position to a desired in-plane position and / or to the position of an anatomical landmark.

[0116] In particular, embodiments utilize the recognition that a machine learning algorithm trained to determine or predict information defining the location of a desired plane within a 3D image exhibits improved performance when simultaneously predicting the location of one or more anatomical landmarks outside the desired plane because, during the earlier training process, the locations of the one or more anatomical landmarks provide additional ground truth data for training the machine learning algorithm and reduce the likelihood of overfitting.

[0117] This same underlying inventive realization can be utilized to perform improved training of machine learning algorithms.

[0118] Therefore, a computer-implemented method for training a machine learning method for predicting the location of a desired plane in a desired 3D image is also proposed, which utilizes the same inventive realizations of the above-described method for using the machine learning method.

[0119] The computer-implemented method includes obtaining a training data set including a plurality of training entries, each training entry including an exemplary 3D image of a target anatomical structure, first information identifying a location of a desired plane within the exemplary 3D image, and second information identifying a location of one or more anatomical landmarks outside the desired plane within the exemplary 3D image.

[0120] The training data set can be obtained, for example, from a database or other storage system. The training data set can be initially generated by an appropriately trained or experienced clinician labeling differential instances or examples of 3D image data to identify the location of desired planes and / or positions of one or more anatomical landmarks.

[0121] The computer-implemented method also includes using the training data set to train a machine learning method to predict information identifying a location of the desired plane in the desired 3D image and information identifying a location of one or more anatomical landmarks in the desired 3D image outside the desired plane.

[0122] The information identifying the location of the desired plane in the desired 3D image may be, for example, a set of one or more first offsets. Each first predicted offset is a predicted spatial offset relative to the 3D image between an initial position and a predetermined location within the desired plane of the 3D image. Each predetermined location may be located, for example, within a cell of a grid of cells that overlays the 2D plane. The location of each predetermined location with each cell may be predetermined.

[0123] Each initial position may be defined by processing the 2D slice using an object detection technique to identify a boundary region containing a representation of the target anatomical structure, and defining a set of initial positions within the boundary region.

[0124] The information identifying the position of one or more anatomical landmarks outside the desired plane may include a set of one or more second predicted offsets, each second predicted offset being a predicted spatial offset relative to the 3D image between the initial position and the position of an anatomical landmark outside the desired plane of the target anatomical structure.

[0125] Suitable examples of machine learning algorithms have been described previously, but the machine learning algorithm is preferably a neural network, more preferably a multidimensional neural network.

[0126] Those skilled in the art can readily develop a processing system to perform any of the methods described herein, and thus each step in the flowchart may represent a different action performed by the processing system and may be performed by a respective module of the processing system.

[0127] Accordingly, embodiments may utilize a processing system. A processing system may be implemented in numerous ways using software and / or hardware to perform the various functions required. A processor is one example of a processing system that uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the necessary functions. However, a processing system may be implemented with or without a processor, and may also be implemented as a combination of dedicated hardware to perform some functions and processors (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

[0128] Examples of processing system components that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0129] In various implementations, a processor or processing system may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The storage media may be encoded with one or more programs that, when executed on the one or more processors and / or processing systems, perform the necessary functions. The various storage media may be fixed within the processor or processing system or may be portable, such that the one or more programs stored thereon can be loaded into the processor or processing system.

[0130] 6 illustrates a schematic diagram of a processor circuit 600, according to one embodiment. As shown, the processor circuit 600 may include a processor 606, a memory 603, and a communication module 608. These elements may communicate with each other directly or indirectly, for example, via one or more buses.

[0131] The processor 606 contemplated by this disclosure may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, a field-programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 606 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 606 may also implement various deep learning networks, which may include hardware or software implementations. The processor 606 may further include a preprocessor in either a hardware or software implementation.

[0132] The memory 603 contemplated by the present disclosure may be any suitable storage device, such as cache memory (e.g., cache memory of the processor 606), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), field programmable gate array read-only memory (PROM), erasable field programmable gate array read-only memory (EPROM), electrically erasable field programmable gate array read-only memory (EEPROM), flash memory, solid-state memory devices, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. The memory may be distributed across multiple memory devices and / or located remotely relative to the processor circuitry. In one embodiment, the memory 603 may store instructions 605. The instructions 605 may include instructions that, when executed by the processor 606, cause the processor 606 to perform the operations described herein.

[0133] The instructions 605 may also be referred to as code. The terms “instructions” and “code” should be broadly interpreted to include any type of computer-readable statement. For example, the terms “instructions” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc., and “instructions” and “code” may include a single computer-readable statement or many computer-readable statements. The instructions 605 may form an executable computer program or script. For example, the routines, subroutines, and / or functions may be defined in programming languages ​​including, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP script, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc. The instructions may also include instructions for machine learning, deep learning, and / or training a machine learning module.

[0134] The communications module 608 may include any electronic and / or logic circuitry for facilitating direct or indirect communication of data between the processor circuit 600 and, for example, an external display (not shown) and / or an imaging device or system, such as an X-ray imaging system. In this regard, the communications module 608 may be an input / output (I / O) device. Communications may be performed via any suitable means. For example, the communications means may be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, the communications means may be a wireless link, such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0135] It will be understood that the disclosed methods are preferably computer-implemented methods. Therefore, the concept of a computer program comprising code for implementing any of the methods described when said program is run on a processing system such as a computer is also proposed. Thus, different parts, lines or blocks of code of a computer program according to an embodiment may be executed by a processing system or a computer in order to perform the methods described herein.

[0136] A non-transitory storage medium for storing a computer program has also been proposed.

[0137] In some alternative implementations, the functions noted in the block diagrams or flowcharts may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

[0138] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0139] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0140] Where a computer program is described above, it may be stored / distributed on a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0141] It should be noted that when the term "adapted for" is used in the claims or description, it is intended to be equivalent to the term "configured for." When the term "apparatus" is used in the claims or description, it is intended to be equivalent to the term "system," and vice versa. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A computer-implemented method for processing a 3D image of a target anatomical structure, the computer-implemented method comprising: acquiring 2D slices of the 3D image; processing the 2D slices using object detection techniques to identify a boundary region containing a representation of the target anatomical structure; defining a set of initial positions within the boundary region; The 3D images are processed using machine learning methods to: one or more sets of first predicted offsets, each first predicted offset being a predicted spatial offset relative to the 3D image between an initial position in the set of initial positions and a predetermined position within a desired plane of the 3D image, the desired plane representing a plane of the 3D image that includes a representation of a desired view of the target anatomical structure; and one or more sets of second predicted offsets, each second predicted offset being a predicted spatial offset relative to the 3D image between the initial position and a position of an anatomical landmark of the target anatomical structure outside the desired plane; and and predicting 10. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, further comprising processing the set of initial positions, the set of one or more first predicted offsets, and the 3D image to generate at least one image smaller than the 3D image that includes a representation of one or more desired positions in the desired plane.

3. each initial position is associated with a different desired position within a desired plane of the 3D image for viewing the target anatomical structure; for each initial position, the set of one or more first predicted offsets comprises a single predicted offset that is a predicted spatial offset between the initial position and its respective desired position; 10. The computer-implemented method of claim 1.

4. estimating a location of a desired plane within the 3D image by processing the set of initial positions and a first set of predicted offsets generated for each initial position; processing the 3D image using the estimated location of the desired plane to generate a 2D image representing the desired plane; The computer-implemented method of claim 3, further comprising:

5. 5. The computer-implemented method of claim 4, wherein estimating the location of the desired plane within the 3D image is performed by processing the set of initial locations and the first set of predicted offsets generated for each initial location using a linear regression technique.

6. 2. The computer-implemented method of claim 1, wherein each second predicted offset is a predicted spatial offset relative to the 3D image between a position of a same first anatomical landmark and an initial position of the target anatomical structure.

7. 10. The computer-implemented method of claim 1, wherein each set of one or more second predicted offsets comprises a plurality of second predicted offsets.

8. The step of defining a set of initial positions within the boundary region comprises: defining a grid of cells within the bounding region; identifying a single location within each cell thereby defining said initial location; 2. The computer-implemented method of claim 1, comprising:

9. 2. The computer-implemented method of claim 1, wherein the 3D image is defined by a stack of initial 2D slices, the 2D slices being closer to a central-most slice of the stack of initial 2D slices than either the first or last 2D slice in the stack of initial 2D slices.

10. 2. The computer-implemented method of claim 1, wherein processing the 2D slices using a machine learning method further comprises generating a confidence value for the predicted offset, the confidence value representing a reliability or certainty of the predicted offset.

11. estimating a location of the desired plane within the 3D image by processing the set of initial positions and a first set of predicted offsets generated for each initial position using a linear regression technique; the values ​​of one or more weighting parameters of the linear regression technique are responsive to confidence values ​​of the one or more predicted offsets; 11. The computer-implemented method of claim 10.

12. 1. A computer-implemented method for training a machine learning method for predicting a location of a desired plane in a 3D image of a target anatomical structure relative to a 2D slice within the 3D image, the desired plane representing a plane of the 3D image that includes a representation of a desired view of the target anatomical structure, the computer-implemented method comprising: acquiring a training dataset having a plurality of training entries, each training entry having an exemplary 3D image of the target anatomical structure, first information identifying a location of a desired plane within the exemplary 3D image, and second information identifying a location of one or more anatomical landmarks outside the desired plane within the exemplary 3D image; using the training data set to train a machine learning method that predicts information identifying a location of a desired plane within the 3D image and information identifying a location of one or more anatomical landmarks outside the desired plane within the 3D image relative to a set of initial locations within a boundary region within the 2D slice, the boundary region including a representation of the target anatomical structure; and the information identifying the location of the desired plane within the 3D image comprises one or more sets of first offsets, each first offset being a spatial offset relative to the 3D image between an initial position of a set of initial positions and a predetermined position within the desired plane; the information identifying the location of one or more anatomical landmarks outside the desired plane within the 3D image comprises a set of one or more second offsets, each second offset being a spatial offset relative to the 3D image between the initial position and a location of the one or more anatomical landmarks; Computer-implemented methods.

13. 10. A computer program comprising computer program code that, when executed on a computing device having a processing system, causes the processing system to perform all of the steps of the method of claim 1.

14. 13. A computer program comprising computer program code that, when executed on a computing device having a processing system, causes the processing system to perform all of the steps of the method of claim 12.

15. A processing system for processing 3D images of a target anatomical structure, the processing system being configured to perform the method of claim 1.

Images