System and method for processing ultrasound imaging data

Abstract

A system and method for processing multi-plane ultrasound imaging data. A set of two-dimensional multi-plane images of near simultaneously acquired anatomical structures is processed using a machine learning algorithm to produce a synthetic three-dimensional image. The synthesized three-dimensional image is a complete three-dimensional volume anatomically consistent with the input two-dimensional multi-plane image.

Description

Technical Field

[0001] The present invention relates to the field of ultrasonic imaging. Background Art

[0002] When referring to ultrasound images, these images can be acquired in two or three dimensions. Two-dimensional (2D) ultrasound imaging remains a popular choice because it allows for higher frame rates than three-dimensional imaging. In the case of contrast-enhanced ultrasound, two-dimensional ultrasound imaging results in less destruction of the injected contrast agent (also called ultrasound enhancement agent, UEA) compared to three-dimensional (3D) ultrasound imaging.

[0003] However, many automatic image analysis algorithms are designed to operate on three-dimensional ultrasound images and cannot be used to analyze two-dimensional ultrasound images.

[0004] Therefore, there is a need for an improved method for generating three-dimensional information from ultrasound imaging data.

[0005] WO2021 / 209348A1 discloses an ultrasound roadmap image generation system. Summary of the invention

[0006] The invention is defined by the claims.

[0007] According to an example according to aspects of the present invention, a processing system for processing multi-plane ultrasound imaging data is provided. The processing system is configured to: obtain a first set of two-dimensional ultrasound images of an anatomical structure acquired by a multi-plane ultrasound scanning device (e.g., an imaging probe) at a first time frame, wherein at least two of the two-dimensional images have been acquired in planes of different orientations; obtain a three-dimensional ultrasound image of the anatomical structure acquired by a three-dimensional ultrasound scanning device at a third time frame, wherein the third time frame is different from the first time frame; and generate an estimated three-dimensional ultrasound image of the anatomical structure at the first time frame by processing at least the first set of two-dimensional ultrasound images and the three-dimensional ultrasound image of the anatomical structure using a machine learning algorithm.

[0008] The first set of two-dimensional ultrasound images are multi-plane images, i.e., images acquired / captured by an imaging probe at the same probe position and orientation and at the same time frame. The multi-plane images (and therefore at least two images in the first set of two-dimensional ultrasound images) are oriented differently from each other. In some examples, each image in the first set of two-dimensional ultrasound images has been acquired at a plane of different orientation. Preferably, one of the imaging axes of each multi-plane image is in the same position and orientation (relative to the imaging object / individual) as the imaging axis of at least one other multi-plane image. In other words, the multi-plane images can share an imaging axis.

[0009] Preferably, each two-dimensional image in the first set (ie, each multi-planar image) comprises image data representative of the same axis extending from the probe which captured the multi-planar image into the object / subject being imaged.

[0010] The inventors have recognized that multi-planar imaging provides some three-dimensional information and that the remaining information in the three-dimensional volume can be constructed by using machine learning algorithms.

[0011] The machine learning algorithm is a machine learning algorithm that has been trained to recognize three-dimensional features of anatomical structures based on two-dimensional multi-planar images. Therefore, the estimated three-dimensional image generated is a "complete" three-dimensional image, which can then be processed in the same way as a three-dimensional image obtained by three-dimensional imaging.

[0012] Since two-dimensional multi-plane ultrasound imaging has a higher frame rate than three-dimensional ultrasound imaging, an estimated three-dimensional ultrasound image can be generated at a higher frame rate than a "real" three-dimensional ultrasound image. In addition, in the case of using a contrast agent (or enhancement agent), two-dimensional multi-plane ultrasound imaging will result in less destruction of the contrast agent, thereby allowing a greater number of contrast-enhanced estimated three-dimensional ultrasound images to be generated.

[0013] The use of multi-plane images, rather than 2D images acquired sequentially by moving / rotating the imaging probe between acquisitions, means that the images are automatically aligned in space and time. This avoids the introduction of errors when aligning the images.

[0014] Temporal alignment is particularly valuable in applications that require three-dimensional images from specific points in time, such as cardiac imaging. Even when using cardiac gating, images from corresponding points in consecutive cardiac cycles will not be precisely aligned because different respiratory states at the same point in the cardiac cycle and irregularities in the cardiac cycle will result in different heart shapes.

[0015] Preferably, the first set of two-dimensional ultrasound images comprises less than 10 images, such as 5 or less images.The proposed method innovatively recognizes that 3D ultrasound images can be generated using only a limited number of multi-plane images by using a machine learning algorithm.

[0016] The combination of a three-dimensional ultrasound image acquired at one time frame and a two-dimensional multi-plane ultrasound image acquired at a different time frame can improve the accuracy of the estimated three-dimensional ultrasound image, while achieving a higher frame rate without acquiring a complete three-dimensional ultrasound image for each frame (using three-dimensional imaging). This can be particularly valuable for applications where it is desired to obtain three-dimensional imaging data over a period of time (e.g., within a cardiac cycle).

[0017] Furthermore, in the case of contrast-enhanced ultrasound, the acquisition of two-dimensional multi-plane ultrasound images for some time frames and three-dimensional images for other time frames allows the generation of three-dimensional imaging data for each time frame while reducing contrast agent destruction. This is particularly advantageous in contrast-enhanced cardiac ultrasound applications, such as left ventricular echocardiography, where it is desirable to obtain contrast-enhanced three-dimensional images at every point throughout the cardiac cycle.

[0018] In some examples, a machine learning algorithm has been trained using a training algorithm configured to receive a set of first training inputs and first known outputs, each first training input corresponding to a corresponding first known output. Each first training input includes a set of simulated two-dimensional multi-plane ultrasound images, wherein the set of simulated two-dimensional multi-plane ultrasound images is generated based on a three-dimensional ultrasound image. Each first known output includes a three-dimensional ultrasound image used to generate the set of simulated two-dimensional multi-plane ultrasound images for the corresponding first training input.

[0019] This allows easy generation of training sets of training inputs and associated known outputs.

[0020] The set of simulated two-dimensional multi-planar ultrasound images for each three-dimensional ultrasound image may be generated by resampling the three-dimensional image along a given two-dimensional cutting plane passing through the three-dimensional image.

[0021] In some examples, a machine learning algorithm has been trained using a training algorithm that is configured to receive a set of second training inputs and second known outputs, each second training input corresponding to a corresponding second known output; each second training input includes a set of two-dimensional multi-plane ultrasound images of an anatomical structure of an object; and each second known output includes a three-dimensional ultrasound image of the anatomical structure represented by the set of two-dimensional multi-plane ultrasound images of the corresponding second training input.

[0022] The set of two-dimensional multi-plane ultrasound images may be registered with the three-dimensional ultrasound image to form input-output pairs, each pair comprising a set of two-dimensional ultrasound images of anatomical structures of a subject and a three-dimensional ultrasound image of the same anatomical structures of the same subject.

[0023] The terms "first" and "second" are used herein purely to distinguish the first training input and known output from the second training input and known output, and are not intended to imply that the second training input and known output require the first training input and known output.

[0024] In some examples, the machine learning algorithm is a generative adversarial network.

[0025] A generative adversarial network (GAN) can be used to ensure that the estimated three-dimensional ultrasound image is consistent with the acquired two-dimensional multi-plane ultrasound image and is anatomically reasonable. The discriminator network of the GAN learns to distinguish between the real three-dimensional ultrasound image and the estimated three-dimensional ultrasound image generated from the two-dimensional multi-plane ultrasound image, while the generator network attempts to generate an estimated three-dimensional ultrasound image that the discriminator network cannot distinguish from the real three-dimensional ultrasound image.

[0026] In some examples, the processing system is further configured to provide the estimated three-dimensional ultrasound image to an image analysis algorithm developed for processing and / or analyzing three-dimensional imaging data.

[0027] Since the estimated 3D ultrasound image is a complete 3D image (rather than a set of multi-planar images with gaps between them), the estimated 3D ultrasound image can be analyzed using "standard" image analysis algorithms designed for analyzing 3D images.

[0028] This allows clinicians to more effectively utilize the three-dimensional information available in two-dimensional multi-plane ultrasound images.

[0029] The image analysis algorithm may be a model-based segmentation algorithm.

[0030] Model-based segmentation algorithms produce more accurate results when applied to 3D images because 3D objects can be identified more accurately from 3D images than from 2D views.

[0031] In particular, it has been found that measurements derived from segmentation of two-dimensional images, such as measurements of cardiac parameters, are often inaccurate.

[0032] Therefore, providing an estimated three-dimensional image generated from two-dimensional multiplanar images to a model-based segmentation algorithm designed for three-dimensional imaging data will produce a more accurate segmentation than that obtained by segmenting each multiplanar image using the algorithm for two-dimensional images.

[0033] In some examples, the processing system is further configured to provide a visual representation of the estimated three-dimensional ultrasound image of the anatomical structure at a display device.

[0034] Displaying the estimated three-dimensional ultrasound image allows the clinician to more easily identify three-dimensional features than simply displaying the two-dimensional multi-plane ultrasound image.

[0035] The visual representation of the estimated three-dimensional ultrasound image may distinguish between image regions comprising actual acquired data (ie, imaging data from the first set of two-dimensional ultrasound images) and estimated regions. For example, shading or color overlays may be used to indicate estimated regions.

[0036] In some examples, the processing system is configured to: obtain a second group of two-dimensional ultrasound images of the anatomical structure acquired by a multi-plane ultrasound scanning device in a second time frame, wherein, in the second group, at least two of the two-dimensional images have been acquired in planes of different orientations, and wherein the second time frame is different from the first time frame; and generate an estimated three-dimensional ultrasound image of the anatomical structure in the first time frame by processing at least the first group of two-dimensional ultrasound images and the second group of two-dimensional ultrasound images using the machine learning algorithm.

[0037] The use of additional imaging information, where available, can further improve the accuracy of the estimated three-dimensional image. Additional sets of multiplanar images can be obtained using a "spiral" acquisition mode, in which the orientation of the multiplanar images is rotated about a common axis during the acquisition period. Since these additional images will generally be oriented differently from the first set of multiplanar images and from each other, each set of multiplanar images will provide some three-dimensional information not included in the other set(s).

[0038] The output of the machine learning algorithm may be a single estimated three-dimensional ultrasound image at a first time frame, or an estimated three-dimensional ultrasound image may be generated for each time frame for which a set of multi-plane images have been acquired.

[0039] An ultrasound system is also provided, comprising a multi-plane ultrasound scanning device and the above processing system.

[0040] According to another aspect of the present invention, a computer-implemented method for processing multi-plane ultrasound imaging data is provided. The computer-implemented method includes: obtaining a first set of two-dimensional ultrasound images of an anatomical structure acquired by a multi-plane ultrasound scanning device at a first time frame, wherein at least two of the two-dimensional ultrasound images have been acquired in planes of different orientations; obtaining a three-dimensional ultrasound image of the anatomical structure acquired by a three-dimensional ultrasound scanning device at a third time frame, wherein the third time frame is different from the first time frame; and generating an estimated three-dimensional ultrasound image of the anatomical structure at the first time frame by processing at least the first set of two-dimensional ultrasound images and the three-dimensional ultrasound image of the anatomical structure using a machine learning algorithm.

[0041] In some examples, the computer-implemented method also includes providing the estimated three-dimensional ultrasound image to an image analysis algorithm developed for processing and / or analyzing three-dimensional imaging data.

[0042] In some examples, the computer-implemented method also includes providing a visual representation of the estimated three-dimensional ultrasound image of the anatomical structure at a display device.

[0043] In some examples, the computer-implemented method also includes: obtaining a second group of two-dimensional ultrasound images of the anatomical structure acquired by the multi-plane ultrasound scanning device in a second time frame, wherein, in the second group, at least two of the two-dimensional ultrasound images have been acquired in planes of different orientations, and wherein the second time frame is different from the first time frame; and wherein the step of generating the estimated three-dimensional ultrasound image of the anatomical structure in the first time frame includes processing at least the first group of two-dimensional ultrasound images and the second group of two-dimensional ultrasound images using the machine learning algorithm.

[0044] A computer program product is also proposed comprising computer program code means which, when executed on a computer device having a processing system, cause the processing system to perform all the steps of the above method.

[0045] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] For a better understanding of the invention, and in order to more clearly show how the invention may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0047] Figure 1 A system for acquiring and processing multi-plane ultrasound imaging data according to an embodiment of the present invention is shown; and

[0048] Figure 2 A computer-implemented method 200 for processing multi-plane ultrasound imaging data according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] The present invention will be described with reference to the accompanying drawings.

[0050] It should be understood that the detailed description and specific examples, although indicating exemplary embodiments of the device, system and method, are intended only for illustrative purposes and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood from the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used in all drawings to indicate the same or similar parts.

[0051] A system and method for processing multi-plane ultrasound imaging data is presented. A set of two-dimensional multi-plane images of anatomical structures acquired nearly simultaneously are processed using a machine learning algorithm to produce a synthetic three-dimensional image. The synthetic three-dimensional image is a complete three-dimensional volume that is anatomically consistent with the input two-dimensional multi-plane image.

[0052] Embodiments are based at least in part on the recognition that multiplanar images provide some three-dimensional information, and training a machine learning algorithm to estimate a complete three-dimensional volume based on the three-dimensional information contained in the multiplanar images enables the generation of three-dimensional images (which can then be input into an image analysis algorithm) at a higher frame rate than is possible using three-dimensional imaging alone (i.e., using a three-dimensional scanning device), and with reduced contrast agent destruction where applicable.

[0053] The illustrative embodiments may be used, for example, in ultrasound imaging systems (eg, cardiac ultrasound imaging systems), and particularly in ultrasound image analysis applications, such as the Philips QLAB.

[0054] Figure 1 A system 100 for acquiring and processing multi-plane ultrasound imaging data according to an embodiment of the present invention is shown. The system 100 includes a multi-plane ultrasound scanning device 110 and a processing system 120. The processing system itself is an embodiment of the present invention.

[0055] The multi-plane ultrasound scanning device may be any ultrasound scanning device capable of acquiring multi-plane imaging data (ie, an ultrasound scanning device comprising a two-dimensional transducer array). For example, the multi-plane ultrasound scanning device may comprise a matrix array transducer.

[0056] The multi-plane ultrasound scanning device 110 acquires a first set of two-dimensional ultrasound images 115 of the anatomical structure at a first time frame f1, and the processing system 120 obtains the first set of two-dimensional ultrasound images from the multi-plane scanning device.

[0057] The first set of two-dimensional ultrasound images 115 are multi-plane images: that is, images acquired by an ultrasound scanning device with the same probe position and orientation. At least two images in the first set of two-dimensional ultrasound images are oriented differently from each other. In some examples, each image in the first set can be oriented differently. For example, the first set of two-dimensional images can include two orthogonal images.

[0058] Preferably, one of the imaging axes of each multiplanar image is in the same position and orientation (relative to the imaged subject / individual) as the imaging axis of at least one other multiplanar image. In other words, the images forming the first set of two-dimensional ultrasound images may share a common imaging axis.

[0059] Preferably, each two-dimensional image in the first set (ie, each multi-planar image) comprises image data representing the same axis extending from the probe which captured the multi-planar image into the object / subject being imaged.

[0060] The first set of two-dimensional ultrasound images 115 is acquired by a multi-plane scanning device at the same time frame f1. In other words, the time between the acquisition of the ultrasound images acquired successively in the first set is lower than the frame rate of the multi-plane imaging. For example, the time between the consecutive acquisitions of the images in the first set may be less than 20 ms. Preferably, the time between the consecutive acquisitions of the images in the first set may be less than 10 ms.

[0061] The time between successive acquisitions of the images in the first set 115 can be low enough that the imaged anatomy is unlikely to change significantly across the first set of 2D multiplanar images. For example, at a heart rate of 80 bpm, a time of 10 ms between successive acquisitions would correspond to only 1.3% of the cardiac cycle.

[0062] exist Figure 1 In the embodiment, the multi-plane ultrasound scanning device 110 is an X-plane ultrasound scanning device, so the first set of two-dimensional ultrasound images includes two images. However, the present invention is not limited to X-plane imaging, and the first set of two-dimensional ultrasound images may include more than two images.

[0063] Preferably, the first set of two-dimensional ultrasound images comprises less than 10 images, such as 5 images or less.The proposed method innovatively recognizes that 3D ultrasound images can be generated using only a limited number of multi-plane images by using a machine learning algorithm.

[0064] The processing system 120 processes the first set of two-dimensional images 115 using a machine learning algorithm to generate an estimated three-dimensional image 125 of the anatomical structure at a first time frame f1.

[0065] A machine learning algorithm is any self-training algorithm that processes input data in order to produce or predict output data. Here, the input data comprises at least a first set of two-dimensional images 115, and the output data comprises an estimated three-dimensional image 125 corresponding to the first set of two-dimensional images. An estimated three-dimensional image is considered to correspond to a set of two-dimensional images if, for each two-dimensional image in a set, a two-dimensional cross-section of the estimated three-dimensional image having the same orientation and position as the two-dimensional image is identical to the two-dimensional image within a given accuracy.

[0066] Suitable machine learning algorithms for the present invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms such as logistic regression, support vector machines, or naive Bayes models are suitable alternatives.

[0067] The structure of an artificial neural network (or neural network for short) is inspired by the human brain. A neural network comprises layers, each layer comprising a plurality of neurons. Each neuron comprises a mathematical operation. Specifically, each neuron may comprise a different weighted combination of transformations of a single type (e.g., transformations of the same type, sigmoid transformations, etc., but with different weights). During the processing of input data, mathematical operations of each neuron are performed on the input data, a numerical output is generated, and the output of each layer in the neural network is sequentially fed into the next layer. The final layer provides the output.

[0068] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining a training data set, including training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the machine learning algorithm. The process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is generally referred to as a supervised learning technique.

[0069] For example, in the case where the machine learning algorithm is formed by a neural network, the mathematical operation (weight) of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, back propagation algorithms, etc.

[0070] In some examples, the training input data entries may correspond to a set of simulated two-dimensional multi-plane ultrasound images, wherein each set of simulated two-dimensional multi-plane ultrasound images is generated based on an example three-dimensional image. The training output data entries may correspond to example three-dimensional images based on which the training input data entries are generated. In other words, the training data set may include input-output pairs, each pair including an example three-dimensional image of an anatomical structure and a set of simulated two-dimensional multi-plane ultrasound images generated based on the example three-dimensional image.

[0071] Methods for generating simulated two-dimensional ultrasound images from three-dimensional ultrasound images will be apparent to those skilled in the art. For example, each three-dimensional ultrasound image may be resampled along one or more two-dimensional cutting planes through the image to generate (one or more) two-dimensional ultrasound images.

[0072] In other examples, the training input data entries may correspond to an example set of two-dimensional multi-plane ultrasound images of anatomical structures of a subject. The training output data entries may correspond to example three-dimensional ultrasound images, with each identical anatomical structure represented by a corresponding training input data entry.

[0073] Each example set of two-dimensional multi-plane ultrasound images can be registered with a corresponding three-dimensional image. Methods for registering two-dimensional ultrasound images to three-dimensional space will be apparent to those skilled in the art. Each example set of two-dimensional multi-plane ultrasound images can be acquired using the same ultrasound scanning device as the corresponding three-dimensional ultrasound image, with the same device position and orientation, but operating in a different imaging mode, so that each input-output pair is automatically aligned in space.

[0074] In addition to the estimated three-dimensional ultrasound images 125 corresponding to the input first set of two-dimensional ultrasound images 115, the machine learning algorithm is expected to output an anatomically plausible estimated three-dimensional ultrasound image. Preferably, the estimated three-dimensional ultrasound image should be similar to a true three-dimensional ultrasound image (i.e., an ultrasound image obtained using three-dimensional imaging) in terms of image features and appearance. When the image analysis algorithm is provided with the estimated three-dimensional ultrasound image, this improves the accuracy of the output of the image analysis algorithm that has been developed for three-dimensional images.

[0075] In some examples, this can be achieved by using a Generative Adversarial Network (GAN). A Generative Adversarial Network consists of a generator network and a discriminator network.

[0076] The generator network can be trained to generate estimated three-dimensional ultrasound images based on training inputs (ie, a set of two-dimensional multi-planar ultrasound images obtained from multi-planar imaging or simulated from three-dimensional ultrasound images, as described above).

[0077] The discriminator network can be trained to distinguish between real three-dimensional ultrasound images and estimated three-dimensional ultrasound images generated by the generator network. The discriminator can output a discriminator score whose value depends on the realism of the input three-dimensional ultrasound image (wherein a higher score indicates a more realistic image).

[0078] The generator network and the discriminator network are trained together, wherein the generator network is modified to the estimated three-dimensional ultrasound image generated by the generated network according to the discriminator score given by the discriminator network, and the discriminator network is modified according to whether the discriminator network correctly distinguishes between the real image and the estimated image. This causes the generator network to output a more realistic estimated three-dimensional ultrasound image (i.e., an estimated three-dimensional ultrasound image that will be given a high discriminator score by the discriminator network), and the discriminator network becomes increasingly accurate.

[0079] This process can continue until the loss function, defined as the difference between the discriminator scores for the estimated image and the discriminator scores for the real image, is sufficiently close to zero (i.e., within a predetermined margin). At this stage, the generator network can be considered to have learned to estimate a 3D ultrasound image that is indistinguishable from the real image.

[0080] In some examples, the processing system 120 is configured to provide the estimated three-dimensional ultrasound image 125 to an image analysis algorithm developed for processing and / or analyzing three-dimensional imaging data.

[0081] The processing system 120 may automatically provide the estimated three-dimensional ultrasound image to a predetermined image analysis algorithm. Alternatively, the processing system may select one of a plurality of image analysis algorithms and provide the estimated three-dimensional ultrasound image to the selected image analysis algorithm.

[0082] Examples of suitable image analysis algorithms include model-based segmentation algorithms and other algorithms that result in the measurement of intrinsic three-dimensional quantities. Model-based segmentation algorithms for three-dimensional images generally produce more accurate segmentation results than model-based segmentation algorithms for two-dimensional images. For example, in the case of cardiac ultrasound imaging, a segmentation algorithm may be used to estimate the volume of the left ventricle as a function of time. Since the volume of the left ventricle is an intrinsic three-dimensional quantity, applying a segmentation algorithm to a two-dimensional ultrasound image will not result in an accurate estimate of the volume of the left ventricle.

[0083] In some examples, system 100 also includes a display device 130. Processing system 120 may be configured to provide a visual representation of an estimated three-dimensional ultrasound image 125 at display device 130. Methods for providing a visual representation of a three-dimensional ultrasound image are well known and will be apparent to those skilled in the art.

[0084] The visual representation may indicate which areas of the estimated three-dimensional image are based on actual acquired ultrasound imaging data (i.e., which areas are included in the first set of two-dimensional ultrasound images) and which areas have been estimated. For example, shading or color overlays may be used to distinguish between estimated areas and areas where actual imaging data exists.

[0085] In the event that the estimated three-dimensional ultrasound image 125 has been provided to an image analysis algorithm, the processing system 120 may additionally or alternatively provide a visual representation of the output of the image analysis algorithm at the display device 130. For example, in the event that the estimated three-dimensional ultrasound image has been provided to a model-based segmentation algorithm, the display device may display a visual representation of the estimated three-dimensional ultrasound image with an overlay representing the segmentation results.

[0086] In some examples, the multiplane scanning device 110 may acquire a second set of two-dimensional ultrasound images of the anatomical structure at a second time frame f2 different from the first time frame f1, and the processing system 120 may obtain the second set of two-dimensional ultrasound images from the multiplane scanning device.

[0087] The processing system 120 may then generate an estimated three-dimensional ultrasound image 125 of the anatomical structure at the first time frame f1 by processing at least the first set of two-dimensional ultrasound images 115 and the second set of two-dimensional ultrasound images using a machine learning algorithm.

[0088] At least two images in the second set of two-dimensional ultrasound images are oriented differently from each other (e.g., each image in the second set can be oriented differently), and one or more images in the second set can be oriented differently from each image in the first set. In this way, the second set of two-dimensional ultrasound images can provide additional anatomical information (although at a different time frame) that can be used to improve the accuracy of estimating the three-dimensional ultrasound image.

[0089] In order to generate an estimated three-dimensional ultrasound image using a first set of two-dimensional ultrasound images and a second set of two-dimensional ultrasound images acquired at a first time frame f1, the training input data entries used to train the machine learning algorithm may each include a first set of two-dimensional ultrasound images of the anatomical structure (as described above) and a second set of two-dimensional ultrasound images of the anatomical structure acquired at a different time frame than the first set of two-dimensional ultrasound images. In the case where the first set of two-dimensional ultrasound images in the training data set are simulated images generated from three-dimensional ultrasound images, the second set of two-dimensional images are each acquired at a different time than the three-dimensional images used to generate the corresponding first set of simulated two-dimensional images, or are themselves simulated images generated from three-dimensional images acquired at a different time than the three-dimensional images used to generate the corresponding first set of simulated two-dimensional images.

[0090] In some examples, the system 100 may also include a three-dimensional ultrasound scanning device. The three-dimensional ultrasound scanning device may be the same device as the multi-planar ultrasound scanning device (operating in a three-dimensional imaging mode rather than a multi-planar imaging mode), or a separate ultrasound scanning device ( Figure 1 ). The three-dimensional ultrasound scanning device may include a matrix array transducer.

[0091] The 3D ultrasound scanning device can acquire a (real) 3D ultrasound image of the anatomical structure at a third time frame f3 different from the first time frame f1, and the processing system 120 can obtain the real 3D ultrasound image from the 3D scanning device. A "real" 3D image is an image of the complete 3D volume.

[0092] The processing system 120 can then generate an estimated three-dimensional ultrasound image 125 of the anatomical structure at the first time frame f1 by processing at least the first set of two-dimensional ultrasound images 115 and the real three-dimensional ultrasound image using a machine learning algorithm. The real three-dimensional ultrasound image acquired at the third time frame f3 will provide anatomical information of the area that was not imaged by the multi-plane scanning device at the first time frame f1. This information can improve the accuracy of the estimated three-dimensional ultrasound image of the first time frame f1, particularly because it will provide patient-specific information about the underlying 3D anatomical structure.

[0093] For example, in the case of cardiac ultrasound, a single real three-dimensional ultrasound image may be acquired by a three-dimensional scanning device at at least one point in the cardiac cycle, and a set of two-dimensional ultrasound images may be acquired by a multi-plane scanning device at each of one or more other points in the cardiac cycle. Preferably, the number of acquired real three-dimensional ultrasound images is small (for example, only one real three-dimensional ultrasound image may be acquired during the imaging process, or one real three-dimensional ultrasound image may be acquired for a given number of frames). For example, in the case of cardiac ultrasound, two real three-dimensional images may be acquired in each cardiac cycle (for example, one real three-dimensional image may be acquired at the end of diastole, and another real three-dimensional image may be acquired at the end of systole), wherein two-dimensional multi-plane images are acquired at other points in the cardiac cycle.

[0094] An estimated three-dimensional ultrasound image at one or more other points in the cardiac cycle can be generated by processing at least the real three-dimensional image(s) and a set of two-dimensional ultrasound images acquired at each of the one or more other points in the cardiac cycle using a machine learning algorithm. This allows accurate generation of three-dimensional images (each output by three-dimensional imaging or by a machine learning algorithm) throughout the cardiac cycle at a higher frame rate than by acquiring a real three-dimensional ultrasound image for each frame (i.e., using a three-dimensional ultrasound scanning device).

[0095] In the case of contrast-enhanced cardiac ultrasound (e.g., left ventricular echocardiography), three-dimensional ultrasound imaging results in more destruction of the injected contrast agent than two-dimensional multiplanar imaging. The combination of one or more true three-dimensional images with an estimated three-dimensional image at the time frame in which the two-dimensional multiplanar image was acquired allows for accurate three-dimensional images while reducing the amount of contrast agent destruction.

[0096] In an example where an estimated three-dimensional ultrasound image is generated using a real three-dimensional ultrasound image of a third time frame f3 in addition to a first set of two-dimensional ultrasound images, each training input data entry for the machine learning algorithm may additionally include an example three-dimensional ultrasound image of the same anatomical structure as a corresponding set of two-dimensional ultrasound images, acquired by a scanning device at a different time than the set of two-dimensional ultrasound images (or, where the two-dimensional ultrasound image is generated based on the three-dimensional image, the simulated image is at a different time than the three-dimensional image from which the simulated two-dimensional image is generated).

[0097] In some examples, the processing system 120 may generate an estimated three-dimensional ultrasound image 125 of the anatomical structure at the first time frame f1 by processing at least the first set of two-dimensional ultrasound images 115 acquired at the first time frame f1, the second set of two-dimensional ultrasound images acquired at the second time frame f2, and the real three-dimensional ultrasound image acquired at the third time frame f3 using a machine learning algorithm. It should be understood that the expressions "first time frame", "second time frame", and "third time frame" are purely used to distinguish between time frames, and the first set of two-dimensional ultrasound images, the second set of ultrasound images, and the real three-dimensional ultrasound images may be acquired in any order.

[0098] Figure 2 A computer-implemented method 200 for processing multi-plane ultrasound imaging data according to an embodiment of the present invention is shown.

[0099] The method starts at step 210, where a first set of two-dimensional ultrasound images of an anatomical structure is acquired by a multi-plane ultrasound scanning device at a first time frame. At least two of the two-dimensional ultrasound images are acquired at planes that are oriented differently from each other.

[0100] At step 240, an estimated three-dimensional image of the anatomical structure at a first time frame is generated by processing at least a first set of two-dimensional images using a machine learning algorithm.

[0101] In some examples, method 200 may also include step 250, where the estimated three-dimensional ultrasound image is provided to an image analysis algorithm. An image analysis algorithm is an image analysis algorithm developed for processing and / or analyzing three-dimensional imaging data.

[0102] In some examples, method 200 may also include step 260 , at which a visual representation of the estimated three-dimensional ultrasound image of the anatomical structure is provided at a display device.

[0103] In some examples, method 200 may also include step 220, at which a second set of two-dimensional ultrasound images of the anatomical structure is acquired by the multi-plane scanning device at a second time frame. At least two of the two-dimensional ultrasound images in the second set are acquired at planes of different orientations. The second time frame is different from the first time frame. At step 240, an estimated three-dimensional image of the anatomical structure at the first time frame is then generated by processing at least the first set of two-dimensional images and the second set of two-dimensional ultrasound images using a machine learning algorithm.

[0104] In some examples, method 200 may also include step 230, at which a three-dimensional ultrasound image of the anatomical structure is acquired by the three-dimensional scanning device at a third time frame. The third time frame is different from the first time frame. At step 240, an estimated three-dimensional image of the anatomical structure at the first time frame is then generated by processing at least the first set of two-dimensional images and the three-dimensional ultrasound image using a machine learning algorithm.

[0105] It should be understood that the disclosed methods are computer-implemented methods. Therefore, the concept of a computer program is also proposed, which includes code modules for implementing any described method when the program is run on a processing system.

[0106] Those skilled in the art will be able to easily develop a processor for executing any method described herein. Therefore, each step of the flowchart may represent a different action performed by the processor and may be performed by a corresponding module of the processor.

[0107] As described above, the system utilizes a processor to perform data processing. The processor can be implemented in a variety of ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuits for performing other functions. The processor can be located remotely (e.g., accessed using a cloud infrastructure).

[0108] Examples of circuits that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0109] In various embodiments, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the desired functions. The various storage media may be fixed within the processor, or the controller may be transportable so that one or more programs stored thereon can be loaded into the processor.

[0110] By studying the drawings, the disclosure and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.

[0111] The functions implemented by a processor may be implemented by a single processor or by multiple separate processing units, which may be collectively considered to constitute a “processor.” In some cases, such processing units may be remote from each other and communicate with each other in a wired or wireless manner.

[0112] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0113] The computer program may be stored / distributed on suitable media such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems.

[0114] If the term "suitable for" is used in the claims or the description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or the description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0115] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A processing system (120) for processing multi-plane ultrasound imaging data, the processing system being configured to: A first set of two-dimensional ultrasound images (115) of an anatomical structure acquired by a multi-plane ultrasound scanning device (110) in a first time frame (f1) is obtained, wherein: At least two of the two-dimensional images have been acquired at planes of different orientations; Obtaining a three-dimensional ultrasound image of the anatomical structure acquired by a three-dimensional ultrasound scanning device in a third time frame (f3), wherein the third time frame is different from the first time frame (f1); as well as An estimated three-dimensional ultrasound image (125) of the anatomical structure at the first time frame is generated by processing at least the first set of two-dimensional ultrasound images and the three-dimensional ultrasound image of the anatomical structure using a machine learning algorithm.

2. The processing system according to claim 1, wherein: Each two-dimensional image in the first set of two-dimensional images has been acquired at a differently oriented plane.

3. The processing system (120) according to claim 1 or 2, wherein: The machine learning algorithm has been trained using a training algorithm, the training algorithm being configured to receive a set of first training inputs and first known outputs, each first training input corresponding to a respective first known output; Each first training input includes a set of simulated two-dimensional multi-plane ultrasound images, wherein the set of simulated two-dimensional multi-plane ultrasound images is generated based on the three-dimensional ultrasound image; as well as Each first known output includes the three-dimensional ultrasound image of the set of simulated two-dimensional multi-plane ultrasound images used to generate the corresponding first training input.

4. The processing system (120) according to claim 1 or 2, wherein: The machine learning algorithm has been trained using a training algorithm, the training algorithm being configured to receive a set of second training inputs and second known outputs, each second training input corresponding to a respective second known output; Each second training input comprises a set of two-dimensional multi-planar ultrasound images of the anatomical structure of the subject: and Each second known output comprises a three-dimensional ultrasound image of the anatomical structure represented by the set of two-dimensional multi-plane ultrasound images of the corresponding second training input.

5. The processing system (120) according to any one of claims 1 to 4, wherein: The machine learning algorithm is a generative adversarial network.

6. The processing system (120) according to any one of claims 1 to 5, wherein: The processing system is also configured to provide the estimated three-dimensional ultrasound image (125) to an image analysis algorithm developed for processing and / or analyzing three-dimensional imaging data.

7. The processing system (120) of claim 6, wherein: The image analysis algorithm is a model-based segmentation algorithm.

8. The processing system (120) according to any one of claims 1 to 7, wherein: The processing system is also configured to provide a visual representation of the estimated three-dimensional ultrasound image (125) of the anatomical structure at a display device (130).

9. The processing system (120) according to any one of claims 1 to 8, wherein: The processing system is configured to: obtaining a second set of two-dimensional ultrasound images of the anatomical structure acquired by the multi-plane ultrasound scanning device (110) at a second time frame (f2), wherein in the second set, at least two of the two-dimensional ultrasound images have been acquired at planes of different orientations, and wherein the second time frame is different from the first time frame (f1); and The estimated three-dimensional ultrasound image (125) of the anatomical structure at the first time frame is generated by processing at least the first set of two-dimensional ultrasound images (115) and the second set of two-dimensional ultrasound images using the machine learning algorithm.

10. An ultrasound system comprising: Multi-plane ultrasound scanning equipment; as well as A processing system according to any one of claims 1 to 9.

11. A computer-implemented method (200) for processing multi-plane ultrasound imaging data, the computer-implemented method comprising: Obtaining a first set of two-dimensional ultrasound images (115) of an anatomical structure acquired by a multi-plane ultrasound scanning device (110) at a first time frame (f1), wherein at least two of the two-dimensional ultrasound images have been acquired at planes of different orientations; obtaining a three-dimensional ultrasound image of the anatomical structure acquired by a three-dimensional ultrasound scanning device in a third time frame (f3), wherein the third time frame is different from the first time frame (f1); and An estimated three-dimensional ultrasound image (125) of the anatomical structure at the first time frame is generated by processing at least the first set of two-dimensional ultrasound images and the three-dimensional ultrasound image of the anatomical structure using a machine learning algorithm.

12. The computer-implemented method (200) of claim 11, wherein: The computer-implemented method further comprises: obtaining a second set of two-dimensional ultrasound images of the anatomical structure acquired by the multi-plane ultrasound scanning device (110) at a second time frame (f2), wherein in the second set at least two of the two-dimensional ultrasound images have been acquired at planes of different orientations, and wherein the second time frame is different from the first time frame (f1); And wherein, the step of generating the estimated three-dimensional ultrasound image (125) of the anatomical structure in the first time frame includes processing at least the first set of two-dimensional ultrasound images (115) and the second set of two-dimensional ultrasound images using the machine learning algorithm.

13. A computer program product comprising computer program code means which, when executed on a computer device having a processing system, causes the processing system to perform all the steps of the method (200) according to claim 11 or 12.

Images