Methods and systems for analyzing ultrasound images

Abstract

A method for analyzing ultrasound images involves assessing the quality of the image based on which features of interest have been identified in the image, and assessing the segmentation quality related to the quality of segmentation of the image. These two quality assessments are combined to derive and output an overall quality assessment of biometric measurements obtained from the image.

Description

Technical Field

[0001] This invention relates to the field of ultrasound imaging, for example, to the field of fetal ultrasound imaging. Background Technology

[0002] Automated measurement of features from ultrasound images captured, for example, via fetal ultrasound scans, is common clinical practice. Measurements typically involve segmenting the image using a segmentation algorithm to identify features of interest, followed by performing biometric measurements. Examples of fetal measurements include, for example, abdominal circumference and nuchal translucency.

[0003] Deep learning-based algorithms are very effective in solving such segmentation and biometric measurement tasks.

[0004] The biometric measurements to be performed typically require adherence to strict image acquisition guidelines. Certain anatomical structures must be visible in the acquired images, and their size or proportions can be specified by the same guidelines.

[0005] In current clinical practice, users or software select frames deemed suitable for biometric measurements before applying segmentation and biometrics. Therefore, frame selection and biometrics are independent tasks.

[0006] This leads to inefficiencies in the process of conducting biometric measurements. Summary of the Invention

[0007] This invention is defined by the claims.

[0008] According to one aspect of the present invention, a method for analyzing ultrasound images is provided, comprising:

[0009] Receive ultrasound images of the imaging region containing features of interest;

[0010] The ultrasound image is segmented to identify features of interest;

[0011] Perform biometric measurements on the features of interest present in the image;

[0012] Generate a segmentation quality assessment related to the quality of the segmentation;

[0013] For the image generated, an image content quality assessment is performed, which is at least related to the features of interest already identified in the image;

[0014] The segmentation quality assessment and the image content quality assessment are combined to derive an overall quality assessment for the biometrics; and

[0015] Output the overall quality assessment.

[0016] This method performs segmentation to identify features of interest in ultrasound images. The quality of the segmentation is evaluated using known methods. For example, this can create a quality score, confidence level, or confidence map for the segmentation. Additionally, image content quality assessment is performed. This involves determining how well the image conforms to acquisition guidelines. It is based on an assessment of the presence of at least one set of specific anatomical features of interest, but optionally on other parameters of those features such as size, orientation, etc. Image content quality assessment is not independent of segmentation, and in fact, it depends on segmentation to assess image content. The overall quality indicator indicates whether (or to what extent) the biometric measurements can be relied upon. Thus, a simple indication of the quality of biometric measurements is provided to the user, taking into account whether the image meets standardized image acquisition requirements.

[0017] Image content quality assessment involves determining whether the image content is suitable for performing a specific biometric measurement. Segmentation quality assessment can be applied to individual pixels of an image and optionally generate an overall segmentation quality score for the entire image.

[0018] The method may include outputting an image content quality assessment and optionally also outputting information related to the reasons for a low image content quality assessment.

[0019] By outputting both image content quality and overall quality assessments, users can determine whether the low overall quality is due to the image content. A low image content quality assessment might be caused by the inability to identify specific features of interest within the image.

[0020] The quality assessment of generated image content may include identifying the presence of features of interest, and one or more of the following:

[0021] Identify alignments between features of interest;

[0022] Identify the dimensions of one or more features of interest;

[0023] Determine the size ratio between features of interest;

[0024] An orientation that identifies one or more features of interest;

[0025] Identify the shape of one or more features of interest;

[0026] Identify the proportion of an image occupied by one or more features of interest.

[0027] These characteristics of the features of interest determine whether the image is suitable for the biometric measurement. They are obtained using a segmentation algorithm.

[0028] The overall assessment may include the confidence level of the bioassay measurements.

[0029] Exporting a segmentation quality assessment may include generating a confidence map. An image can then be displayed, with the confidence map superimposed on it.

[0030] This allows users to visually identify where image segmentation may be unreliable, enabling them to also assess the reliability of what can be placed on biometric measurements.

[0031] This method can be used, for example, to analyze fetal ultrasound images.

[0032] Fetal imaging has strict image acquisition guidelines that need to be followed, which define images suitable for specific biometric measurements.

[0033] Features of interest, particularly fetal abdominal scans, preferably include one or more of the following:

[0034] spine;

[0035] Umbilical vein;

[0036] Stomach;

[0037] heart;

[0038] kidney;

[0039] Abdominal area.

[0040] Of course, other types of scans will also be interested in other features.

[0041] In fetal biometry, depending on the anatomy of the ultrasound image (whether it is the head or abdomen), measurements of interest may include the circumference of the head or abdomen, biparietal diameter, and / or occiputfrontal diameter. Abdominal circumference is often used in combination with head circumference and femur length to determine fetal weight and age. These measurements may, for example, allow for the determination of the head index and / or the ratio between femur length and abdominal circumference, as these are well-known measurements that can indicate fetal health.

[0042] In one particular example, the biometrics includes a measurement of the nuchal translucency, and in another example, the biometrics additionally or alternatively include a measurement of the abdominal circumference.

[0043] In all possible uses of this method, segmentation, generation of segmentation quality assessment, and generation of image content quality assessment can be performed using deep learning, such as deep neural networks (DNNs), such as one or more stochastic deep neural network networks.

[0044] Deep learning can perform segmentation and image content analysis in a fast and reliable manner.

[0045] The present invention also provides a computer program including computer program code, which, when run on a computer, is adapted to implement the method described above. A processor for analyzing fetal ultrasound images is also provided, the processor including memory storing the computer program.

[0046] The present invention also provides an ultrasound imaging system, comprising:

[0047] An ultrasonic probe, suitable for acquiring ultrasonic images of the imaging area;

[0048] Displays; and

[0049] The processor as described above. Attached Figure Description

[0050] To better understand the invention and to more clearly illustrate how it can be practiced, reference will now be made to the accompanying drawings by way of example only, wherein,

[0051] Figure 1 The method steps for analyzing an image are shown, executed by the processor.

[0052] Figure 2 This demonstrates how to evaluate an image to determine if it conforms to the guidelines;

[0053] Figure 3 This demonstrates how to evaluate an image to show segmentation quality;

[0054] Figure 4 The method for analyzing images is shown; and

[0055] Figure 5 An ultrasound system is shown. Detailed Implementation

[0056] The invention will be described with reference to the accompanying drawings.

[0057] It should be understood that while the detailed description and specific examples indicate exemplary embodiments of the devices, systems, and methods, they are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0058] This invention provides a method for analyzing ultrasound images, the method comprising: assessing the quality of the image in terms of identifying at least which features of interest are present in the image, and assessing the segmentation quality in relation to the quality of segmentation of the image. These two quality assessments are combined to derive and output an overall quality assessment of biometric measurements obtained from the image. This allows the user to be informed of the reliability of the biometric measurements.

[0059] Figure 1 The various functional units involved in this method are illustrated. All functions are executed by the processor of the ultrasound imaging system.

[0060] The purpose of this method is to perform bioassays and also to indicate the reliability of those measurements.

[0061] An ultrasound image 10 of an imaging region containing features of interest is received. For fetal imaging, the features of interest include, for example, one or more of the following:

[0062] spine;

[0063] Umbilical vein;

[0064] Stomach;

[0065] heart;

[0066] kidney;

[0067] Abdominal area.

[0068] The ultrasound image is provided to image processing algorithm 12. This performs image segmentation to identify and measure features of interest. The image processing algorithm then performs biometric measurements based on the image segmentation.

[0069] The image processing algorithm involves two quality assessment processes. The first process 14 involves determining the quality of the image content. This quality relates to whether the image is suitable for use in performing the specific biometric measurement to be performed.

[0070] For this purpose, Practice Guideline 16 was used. Biometric measurements in fetal ultrasound generally require adherence to strict image acquisition guidelines. Certain anatomical structures must be visible in the acquired images, and their size or proportions can be specified by the same guidelines. These conditions are stipulated in Practice Guideline 16.

[0071] In step 18, the guidelines are converted into image processing rules. Therefore, the image processing rules specify, for example, which features of interest are needed in the image, and the quality of the image content depends on the identified features.

[0072] In step 20, an image content quality assessment is determined. This can be considered an "image quality score".

[0073] Performing image content quality assessment includes identifying the presence of features of interest as described above, and it preferably includes one or more of the following additional steps:

[0074] Identify alignments between features of interest;

[0075] Identify the dimensions of one or more features of interest;

[0076] Determine the size ratio between features of interest;

[0077] An orientation that identifies one or more features of interest;

[0078] Identify the shape of one or more features of interest;

[0079] Identify the proportion of an image occupied by one or more features of interest.

[0080] These characteristics of the features of interest determine whether the image is suitable for the biometric measurement. They are obtained based on segmentation.

[0081] For example, below we explain some image-based guidelines and how to implement them using image processing.

[0082] For the presence of structures, image processing involves detection and classification.

[0083] To determine the alignment between structures, image processing performs localization and pose estimation.

[0084] To determine size and size ratio, image processing involves the segmentation and identification of regions of interest.

[0085] To determine orientation, image processing involves localization, segmentation, and pose estimation.

[0086] Additional metadata, such as that provided by the user or derived from other images, can also be used, involving one or more of the following: the object's (patient's) age range, the object's size range, and the object's weight range.

[0087] We will now introduce ways to translate certain guidelines into image processing rules, with a specific example involving fetal imaging.

[0088] The first example is the measurement of nuchal translucency (NT).

[0089] For example, the guidelines stipulate:

[0090] (i)NT should be measured between 11 weeks and 13 weeks + 6 days, corresponding to a crown-rump length (CRL) between 45 and 84 mm.

[0091] Image processing involves measuring the CRL of the corresponding image. This can be achieved through a pose estimation model or segmentation technique. (ii) The fetus should be in a neutral position (without excessive extension or bending of the neck).

[0092] Image processing is used to assess the absence of curvature in the posterior cranial and spinal regions by segmenting the ultrasound transmission space. The boundaries of this ultrasound transmission space should then have a curvature below a tolerance threshold.

[0093] (iii) The sagittal features of the fetal face in midline view are the presence of nasal tip echo, rectangular palate, translucent diencephalon in the middle and posterior nuchal membrane.

[0094] These requirements can be evaluated using object detection architectures such as "You Only See Once" object detection (YOLO) or object detection based on regions with convolutional neural networks (RCNN).

[0095] (iv) The image should be enlarged to the full screen, including the head and chest (≥75% of the image).

[0096] This can be achieved using head and chest segmentation (e.g., using U-Net).

[0097] The second example is waist circumference (AC) measurement.

[0098] For example, the guidelines stipulate:

[0099] (i) The presence of a stomach, a small umbilical vein aligned with the spine, and the absence of kidneys and heart.

[0100] The presence or absence of anatomical structures can be detected through object detection (YOLO, RCNN).

[0101] (ii) The abdominal portion should be as round as possible.

[0102] This can be determined through abdominal segmentation processing (U-Net).

[0103] The output given to the user can indicate whether the guidelines are met or not. For example, images can be annotated to show the presence of desired features and highlight missing features or conditions that are not met.

[0104] Figure 2 The top image shows an image that meets the AC measurement requirements—the spine, umbilical vein, and stomach are present, with the spine and UV aligned.

[0105] Figure 2 The bottom shows a graph that does not meet the requirements for AC measurements—UV is missing, and therefore alignment with the spine is not confirmed.

[0106] The second process 22 involves generating a segmentation quality assessment related to segmentation quality in step 24. This can be considered a "segmentation quality score". The segmentation quality assessment can be applied to individual pixels of the image, and optionally, an overall segmentation quality score can be generated for the entire image.

[0107] Image content quality assessment and segmentation quality assessment use the same image processing algorithm 12, therefore they are not independent.

[0108] For example, segmentation quality assessment involves establishing confidence (or certainty / uncertainty) scores and / or confidence plots.

[0109] To establish confidence scores, the method described in DeVries, T., & Taylor, GW (2018), "Leveraging Uncertainty Estimates for Predicting Segmentation Quality" (arXiv:18707.00502), can be used. This method employs Monte Carlo dropout (MC-dropout) during testing, whereas it is typically used during network training. MC-dropout involves randomly removing a portion of the connections from one layer to another. During training, using MC-dropout reduces the risk of the model overfitting the training data because it incorporates some randomness into the network's predictions. However, MC-dropout is usually removed during testing because users want a deterministic prediction and it's best to utilize all available connections in the network.

[0110] Therefore, using MC-dropout during testing will build a stochastic model. By using MC-dropout to make N inferences on the same model, a model with stochastic behavior can be obtained, which makes it possible to generate confidence maps.

[0111] The average prediction for each pixel is:

[0112] Where, ρ n (x) is the prediction of network n at pixel x. (1)

[0113] If the segmentation model has C classes (e.g., stomach, heart, umbilical vein), then the uncertainty for each pixel is:

[0114]

[0115] If only one category is divided, then equation (2) is written with two categories: foreground and background.

[0116] z(x)=-f(x)log(f(x))-(1-f(x))log(1-f(x))(3)

[0117] M=∫1 f(x)>0,5 dx is the segmentation region

[0118] ME=∫z(x)dx is the average entropy fraction

[0119] C = M / ME is the confidence score.

[0120] More detailed information can be found in the references above.

[0121] Please note that this is just one possible method for constructing confidence scores from the output of a neural network. Ensemble methods and probabilistic segmentation networks can also be used.

[0122] The results of the segmentation quality assessment for the neck transparency segmentation are as follows: Figure 3 As shown.

[0123] The left image shows the original image, the middle image shows the spatial segmentation of the acoustic waves, and the right image shows the confidence map, highlighting the uncertain regions. The confidence map information is overlaid on the images.

[0124] The top set of images shows a few highlighted pixels on the confidence map, resulting in a high confidence score and demonstrating that the DNN is confident in the quality of its biometric predictions.

[0125] The bottom image set displays a large number of highlighted pixels on the confidence map, resulting in a low confidence score and indicating that the DNN lacks confidence in the quality of its biometric predictions. Users should be warned and asked to perform a manual review. Alternatively, users can be provided with either numerical or graphical confidence scores.

[0126] The uncertainty map of the bottom image has a larger area of ​​high uncertainty than that of the top image.

[0127] Some guidelines are developed not only based on the presence of features of interest, but also on constraints on the shape of visible structures within the frame of interest, such as the circular portion of the abdomen or the neutral position of the fetal neck (not curved or overextended). Therefore, confidence plots associated with the corresponding segmentation can also be used to assess shape adherence to the guidelines. For example, if the abdomen is segmented as circular but the confidence score is low, the relevant score reflecting adherence to the guidelines will be correspondingly lower.

[0128] return Figure 1In step 26, the segmentation quality assessment and image content quality assessment are combined to obtain an overall quality assessment of the biometrics. This overall quality assessment is the output. Additionally, an image content quality assessment (image quality score) and optionally information about the reasons for a low image content quality assessment may also be output.

[0129] By outputting an image content quality assessment and (or as part thereof) an overall quality assessment, users understand the reasons for low overall quality in the image content. A low image content quality assessment might be due to the inability to identify a specific feature of interest in the image. In this way, two pieces of information are presented to the user. The frame quality assessment (or score) indicates the degree to which the current frame conforms to standard criteria and can provide the user with detailed information about which parts of the criteria were not followed. The overall quality assessment is an estimate of the confidence level of the biometric measurement output by the system.

[0130] It can output two images, one containing information about image quality (e.g., ... Figure 2 Another graph shows the confidence level (e.g.) Figure 3 These two images, when combined, can be considered as constituting an overall quality assessment. However, it is preferable to have an additional separate numerical or textual description of the overall confidence level of the bioassay measurement.

[0131] In summary, the processor of this invention implements a method for identifying features of interest in ultrasound images using segmentation. The quality of the segmentation is evaluated using known methods. For example, this may create a quality score, confidence level, or confidence map for the segmentation. Furthermore, image content quality assessment is performed. This involves determining how well the image conforms to acquisition guidelines. It is based on an assessment of the presence of at least one set of specific anatomical features of interest, but optionally on other parameters of those features such as size, orientation, etc. The overall assessment may include the confidence level measured by the biometrics.

[0132] Image processing algorithms include, for example, deep learning algorithms, such as stochastic deep neural networks. Therefore, the method is based on two observations related to automated measurement, such as in fetal ultrasound. Deep learning-based algorithms are very effective at solving segmentation tasks (on which many measurements rely), but the results can be difficult to interpret. This invention first provides a metric for scoring the confidence of the segmentation output. However, the method also ensures that the measurement output (segmentation based on the input frame) has sufficiently high confidence. Therefore, the overall quality score provides a combination of the confidence of the segmentation result (segmentation quality confidence score) and the confidence of matching practice guidelines (image quality score).

[0133] However, these two scores are related because they come from the same segmentation algorithm described above. In fact, in most cases, to check if an image conforms to the guidelines, the same structures must be segmented for biometric measurements and verification of conformity.

[0134] Figure 4 A method executed by a processor is illustrated. The method includes:

[0135] In step 30, an ultrasound image of the imaging region having features of interest is received;

[0136] In step 32, the ultrasound image is segmented to identify features of interest;

[0137] In step 34, biometric measurements are performed on the features of interest present in the image;

[0138] In step 36, a segmentation quality assessment related to segmentation quality is generated;

[0139] In step 38, an image content quality assessment is generated for the image, and the image content quality assessment is at least related to the features of interest that have been identified in the image;

[0140] In step 40, the segmentation quality assessment and the image content quality assessment are combined to derive an overall quality assessment for the biometrics; and

[0141] In step 42, the overall quality assessment is output.

[0142] Figure 5 An ultrasound system is shown, which includes a probe 50 adapted to acquire ultrasound images of an imaging area, a display 54, and a processor 52 programmed to perform the functions described above.

[0143] As described above, the system utilizes a processor to perform data processing. A processor can be implemented in various ways, using software and / or hardware, to perform a variety of required functions. A processor typically employs one or more microprocessors, which can be programmed using software (e.g., microcode) to perform the desired functions. A processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuitry for performing other functions.

[0144] Examples of circuits that may be used in the various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0145] In various implementations, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when run on one or more processors and / or controllers, perform the required functions. The various storage media may be fixed within the processor or controller, or they may be portable, allowing one or more programs stored thereon to be loaded into the processor.

[0146] Those skilled in the art, through studying the accompanying drawings, disclosure, and claims, will understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit can perform the functions of several items recited in the claims. Although specific measures are recited in dissimilar dependent claims, this does not imply that combinations of these measures cannot be advantageously used. Computer programs can be stored / distributed on suitable media such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems. No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A computer-implemented method for analyzing ultrasound images, comprising: Receive (30) an ultrasound image of the imaging region with features of interest; (32) Segmentation is performed on the ultrasound image to identify the feature of interest; (34) Biometric measurements are performed on the features of interest present in the image; Generate (36) a segmentation quality assessment related to the quality of the segmentation; For the image generation (38) image content quality assessment, the image content quality assessment is at least related to the features of interest that have been identified in the image; Combining (40) the segmentation quality assessment and the image content quality assessment to derive an overall quality assessment for the biometrics; and Output (42) the overall quality assessment, The segmentation quality assessment and the image content quality assessment are derived from the same segmentation algorithm.

2. The method according to claim 1, comprising: The image content quality assessment is output, and optionally, information related to the reasons for the low image content quality assessment is also output.

3. The method according to claim 1 or 2, wherein, Generating the image content quality assessment includes: identifying the presence of features of interest, and one or more of the following: Identify alignments between features of interest; Identify the dimensions of one or more features of interest; Determine the size ratio between features of interest; An orientation that identifies one or more features of interest; Identify the shape of one or more features of interest; Identify the proportion of an image occupied by one or more features of interest.

4. The method according to any one of claims 1 to 3, wherein, The overall quality assessment includes the confidence level of the biometrics.

5. The method according to any one of claims 1 to 4, wherein, Exporting segmentation quality assessments includes generating confidence plots.

6. The method according to claim 5, wherein, The method includes displaying the image, wherein the confidence map is superimposed on the image.

7. The method according to any one of claims 1 to 6, for analyzing fetal ultrasound images.

8. The method according to claim 7, wherein, The features of interest include one or more of the following: spine; Umbilical vein; Stomach; heart; kidney; Abdominal area.

9. The method according to claim 7 or 8, wherein, The biometric measurements include neck lamina transparency measurements.

10. The method according to claim 7 or 8, wherein, The biometric measurements include waist circumference measurement.

11. The method according to any one of claims 1 to 10, wherein, The segmentation, the generation of the segmentation quality assessment, and the generation of the image content quality assessment are performed using deep learning.

12. The method according to claim 11, wherein, The segmentation, the generation of the segmentation quality assessment, and the generation of the image content quality assessment are performed by a single stochastic deep learning neural network.

13. A computer program product comprising computer program code, wherein when the computer program is run on a computer, the computer program code is adapted to implement the method according to any one of claims 1 to 12.

14. A processor (52) for analyzing fetal ultrasound images, comprising a memory storing a computer program according to claim 13.

15. An ultrasound imaging system, comprising: An ultrasonic probe (50) is suitable for acquiring ultrasonic images of the imaging area; Display (54); as well as The processor (52) according to claim 14.

Images