System and method for adaptive measurement of medical images

Abstract

The present invention is entitled "System and method for adaptive measurement of medical images". The present disclosure provides methods and systems for improving the accuracy and / or applicability of automated measurement of medical images. In one embodiment, the present disclosure provides a method comprising: receiving one or more user-selected measurement points for automated measurement of anatomical features of a medical image from a user; predicting one or more additional measurement points for automated measurement based on the one or more user-selected measurement points and based on comparison of the anatomical features of the medical image with multiple images of the anatomical features via a trained network; and performing automated measurement of the anatomical features based at least on the one or more additional measurement points.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to medical imaging, and more particularly to systems and methods for adapting automated measurements to different user preferences. Background Art

[0002] Medical imaging is widely used for diagnostic and therapeutic purposes, where images of anatomical features within a patient's body are created. For example, clinical ultrasound is an imaging modality that uses ultrasound waves to detect the internal structures of a patient's body and produce corresponding images. An ultrasound probe, which includes multiple transducer elements, transmits ultrasound pulses, which are reflected or reflected back, refracted, or absorbed by structures in the body. The ultrasound probe then receives the reflected echoes, which are processed into an image. In echocardiography, ultrasound is used to generate an image of a patient's heart.

[0003] Medical imaging experts such as ultrasound clinicians often rely on measurements between features of medical images to assist in diagnosis, including 2D B-mode measurements, Doppler measurements, area measurements, and volume measurements. Such measurements are usually performed automatically based on automated measurement algorithms. However, inter-operator and intra-operator measurement variations may lead to inaccurate automated measurement algorithms or user dissatisfaction with automated measurement algorithms. Summary of the invention

[0004] The present disclosure at least partially solves one or more of the above-mentioned problems through a method, which includes: receiving one or more user-selected measurement points for automated measurement of anatomical features of a medical image from a user; predicting one or more additional measurement points for automated measurement based on the one or more user-selected measurement points and based on comparison of the anatomical features of the medical image with multiple images of the anatomical features via a trained network; and performing automated measurement of the anatomical features based at least on the one or more additional measurement points.

[0005] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when taken alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a selection of concepts further described in the detailed description in a simplified form. This is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to embodiments that address any disadvantages mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0007] Figure 1A first block diagram of an exemplary embodiment of a medical image processing system including an automated measurement system is shown.

[0008] Figure 2 A second block diagram of an exemplary embodiment of a medical image processing system including an automated measurement system is shown.

[0009] Figure 3A A first set of measurement points for measuring a medical image is shown.

[0010] Figure 3B A second set of measurement points for measuring a medical image is shown.

[0011] Figure 3C An alternative starting point for measuring a medical image is shown.

[0012] Figure 3D Alternative measurement points for measuring a medical image in a first format and a second format are shown.

[0013] Figure 4A The starting point of the segmentation algorithm is shown.

[0014] Figure 4B The first segment of the segmentation algorithm is shown.

[0015] Figure 5 A flow chart illustrating an exemplary high-level method for performing automated measurements of medical images is shown.

[0016] Figure 6 A flow chart illustrating an exemplary method for generating additional measurement points from initial measurement points is shown.

[0017] Figure 7 A flowchart illustrating an exemplary method for training a convolutional neural network is shown. DETAILED DESCRIPTION

[0018] For diagnostic purposes, a user of a medical image processing system may perform measurements on anatomical features of a medical image. A user (e.g., an ultrasound operator, a clinician, an ultrasound cardiologist, etc.) may measure the width, length, or thickness of an anatomical feature, or the space between two anatomical features in a two-dimensional (2D) medical image. For example, in a left ventricle (LV) study, an ultrasound cardiologist may measure the thickness of the septum, the thickness of the posterior wall of the LV, and / or the distance between the septum and the posterior wall, or the ultrasound cardiologist may measure the volume of the LV or the area of ​​a 2D feature (e.g., discoloration, swelling, etc.) on the surface of the LV. Additionally or alternatively, an ultrasound cardiologist may perform spectral Doppler measurements during an echocardiogram to analyze myocardial motion and blood flow.

[0019] To manually perform the measurement, the clinician may use a caliper, wherein the jaws of the caliper may be adjusted and placed at a caliper point on the medical image. When a first jaw of the caliper is placed at a first caliper point and a second jaw of the caliper is placed at a second caliper point, the caliper may indicate a measurement of the distance between the first caliper point and the second caliper point. When referring to a measurement, the first caliper point and the second caliper point may be referred to as measurement points of the measurement.

[0020] Many of these measurements can be accurately and automatically performed by an automated measurement system, thereby reducing inter-operator and intra-operator variability and improving diagnostic efficiency. The automated measurement system can take as input a medical image and the type of measurement to be performed on the image, can select measurement points based on the type of measurement to be performed, and measure the distance between the measurement points. For some types of measurements, the measurement points may include the start and end points of the measurement. Another type of measurement may include a set of component measurements (e.g., a measurement set), wherein the measurement points may include the start and end points of each component measurement of the measurement set. For example, one type of measurement may include a measurement set of three different measurements, including a first start and a first end point of a first measurement, a second start and a second end point of a second measurement, and a third start and a third end point of a third measurement. In some examples, the end point of one measurement may also be the start point of another measurement. For example, the first end point may be the second start point, the second end point may be the third start point, and so on. In one example, the automated measurement system may rely on one or more neural networks trained to predict measurement points based on the medical image and the measurement type (e.g., after training on a set of previously measured medical images).

[0021] However, for many types of measurements, the determined guidelines may leave room for interpretation in terms of the positioning of the starting point and / or other measurement points, and different clinicians may be trained to or may prefer to perform the type of measurement in different ways. For example, a first expert may have been trained to measure an anatomical feature from a first starting point to a first endpoint, while a second expert may have been trained to measure the anatomical feature from a second starting point to a second endpoint, where the first starting point and / or the first endpoint are different from the second starting point and / or the second endpoint. Further, when the anatomical feature is deformed, the first expert and the second expert may use different methods to perform the measurement of the anatomical feature, where the different methods may include selecting different starting points and / or endpoints for the measurement. If the neural network is trained using training data (e.g., medical images with measurements) that consistently measures the anatomical feature from the first starting point to the first endpoint, the first expert may be satisfied with the measurement, while the second expert may not be satisfied with the measurement. If the neural network is trained using training data that consistently measures the anatomical feature from the second starting point to the second endpoint, the second expert may be satisfied with the measurement, while the first expert may not be satisfied with the measurement. If the neural network is trained using training data that consistently measures the anatomical feature from the second starting point to the second endpoint, the second expert may be satisfied with the measurement, while the first expert may not be satisfied with the measurement. If the neural network is trained using training data that does not consistently measure the anatomical feature from the first starting point or the second starting point, the performance of the neural network may be unsatisfactory. If a measurement is obtained from an automated measurement that differs from what the clinician expected, the diagnosis of the patient based on that measurement may be less accurate.

[0022] The present disclosure provides a system and method for performing automated measurement of medical images in a semi-automatic mode, wherein a user may select one or more of the measurement points for performing the automated measurement, and the automated measurement system selects additional measurement points for performing the measurement. In this way, the automated measurement system "automatically completes" the selection of measurement points based on user input. In one example, the user selects a starting point for performing the automated measurement, and the automated measurement system predicts a corresponding end point based on the selected starting point. In another example, the automated measurement system predicts one or more starting points and one or more end points, and prompts the user to accept the measurement based on these measurement points, or manually adjusts one or more of these measurement points to repeat the automated measurement with the adjusted measurement points. In other examples, when performing segmentation of anatomical features, the user may be prompted to select one or more measurement points on the boundary of the anatomical feature, and the automated measurement system may predict additional segmentation measurement points to perform segmentation based on the measurement points selected by the user. If the user wishes to select a measurement point different from the predicted measurement point, the user may adjust the predicted measurement point on the image.

[0023] In one embodiment, a medical imaging system (such as Figure 1 The automated measurement system of the medical imaging system 100) is used to measure the anatomical features of the medical image. The automated measurement system can be used by a user of the medical imaging system according to Figure 2 Flowchart of an automated measurement system for use. The automated measurement system may include one or more neural networks, such as a convolutional neural network (CNN), which may be trained to generate one or more measurement points of an anatomical feature based on one or more user-selected measurement points of the anatomical feature. Figure 3A FIG. 4 shows a set of exemplary measurement points on an anatomical feature generated from a first user-selected starting point. Figure 3B In the case of the variant shown in FIG. 1 , an alternative user-selected starting point may be used. The alternative starting point may be based on a different starting point (such as Figure 3C The starting point shown) reflects Figure 3D Different measurement styles are shown in Figure 2. Measurement points along the boundaries of anatomical features can also be generated from the starting point during the segmentation task, such as Figure 4A and Figure 4B The automated measurement system can be based on Figure 6 The process described in the previous section predicts the measurement points according to Figure 5 The CNN can be trained to perform the measurements according to Figure 7 The process described in generates measurement points from a user-selected starting point.

[0024] refer to Figure 1 , an exemplary block diagram illustrates an exemplary medical image processing system 102 according to an embodiment. The medical image processing system 102 may operate within or as part of a larger medical imaging system (such as an ultrasound imaging system in which ultrasound images are acquired via an ultrasound probe), or the medical image processing system 102 may be a stand-alone system for viewing and / or processing medical images that have been previously acquired by a separate medical imaging system. For example, previously acquired medical images may be stored in a database on a server (e.g., a cloud-based server) accessible to the medical image processing system 102 via a wired or wireless service. The medical image processing system 102 may take as input one or more medical images from a medical imaging system or from a database.

[0025] In one example, a medical image is selected by a user 150 to be displayed on a display device 118 and / or processed via a user input device 116. The display device 118 may include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 118 may include a computer monitor. The user input device 116 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the medical image processing system 102. The user may use the user input device 116 to initiate and / or interact with one or more image processing routines of the medical image processing system 102. For example, the user may wish to adjust the properties of an ultrasound image (e.g., contrast, brightness, etc.), label or mark anatomical features of an ultrasound image, or annotate an ultrasound image. In addition and / or alternatively, the user may wish to perform one or more automated tasks such as an automated measurement task (e.g., to perform measurements of anatomical features) or an automated segmentation task (e.g., to define the boundaries of anatomical features), or another automated task on an ultrasound image and / or anatomical features of an ultrasound image.

[0026] The medical image processing system 102 includes a processor 104 configured to execute machine-readable instructions stored in a non-transitory memory 106. The processor 104 may generally include any number of microprocessors, ASICs, ICs, etc. The processor 104 may be single-core or multi-core, and the program executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 104 may optionally include separate components distributed in two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 104 may be virtualized and executed by a networked computing device capable of remote access configured in a cloud computing configuration.

[0027] In some embodiments, non-transitory memory 106 may include components disposed on two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 106 may include a remotely accessible networked storage device configured in a cloud computing configuration. Non-transitory memory 106 includes any non-transitory computer-readable medium having programming instructions stored therein. For the purposes of this disclosure, the term tangible computer-readable medium is explicitly defined as including any type of computer-readable storage device. Exemplary methods and systems may be implemented using coded instructions (e.g., computer-readable instructions) stored on a non-transitory computer-readable medium, such as flash memory, read-only memory (ROM), random access memory (RAM), cache, or any other storage medium in which information is stored for any duration (e.g., for an extended period of time, permanently, briefly, for temporary buffering and / or for caching of information). Computer memory, as referred to herein as computer-readable storage media, may include volatile and nonvolatile or removable and non-removable media for storing electronic format information (such as computer-readable program instructions or modules of computer-readable program instructions, data, etc., which may be independent or as part of a computing device). Examples of computer memory may include any other media that can be used to store information in a desired electronic format and can be accessed by one or more processors or at least a portion of a computing device.

[0028] The display device 118 may be combined with the processor 104, non-volatile memory 106 and / or user input device 116 in a shared housing, or may be a peripheral display device and may include a monitor, touch screen, projector, or other display device known in the art that enables a user to view medical images and / or interact with various data stored in the non-volatile memory 106.

[0029] The non-transitory memory 106 may include one or more systems for performing various image processing tasks. In one example, the non-transitory memory 106 includes an automated measurement system 108. In some embodiments, at least a portion of the automated measurement system 108 may be disposed at a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the medical imaging system and / or the medical image processing system 102 via a wired connection and / or a wireless connection. In some embodiments, at least a portion of the automated measurement system 108 is disposed at a separate device (e.g., a workstation), which may receive images from the medical imaging system and / or the medical image processing system 102 or from a storage device storing images / data generated by the medical imaging system (e.g., a database). The automated measurement system 108 may be operably / communicatively coupled to a user input device 116 and a display device 118. For example, a user 150 may view a medical image on the display device 118 and may use the user input device 116 to initiate an automated measurement task on the medical image. In one example, the automated measurement task may prompt the user to select a type of measurement to be performed and / or specify a starting point for the measurement (eg, a location on the medical image at which to begin the selected measurement).

[0030] The automated measurement system 108 may include a neural network model 110 that may be trained to predict one or more measurement criteria during the performance of a measurement to partially or fully automate the measurement process. By partially or fully automating the measurement process (e.g., and without requesting input from a user 150), the error rate of the measurement process may be reduced and / or the efficiency and / or speed of the measurement process may be improved. The neural network model 110 may include trained and / or untrained neural networks, and may also include various data or metadata related to one or more neural networks stored therein. In one example, the neural network model 110 includes a convolutional neural network (CNN) that may be trained to predict one or more measurement points for performing a measurement based on one or more user-selected measurement points according to instructions stored in the non-transitory memory 106, as described in more detail below.

[0031] The automated measurement system 108 may include a heat map generator 112 that can generate a heat map representing a user-selected starting point for the automated measurement for input into the neural network model 110. The automated measurement system 108 may also include an image preprocessor 114 that can perform one or more preprocessing tasks on the thermal map and / or ultrasound image before inputting the thermal map and / or ultrasound image into the neural network model 110. For example, the thermal map and ultrasound image can be scaled up or down, cropped, tilted, compressed, etc. to improve the efficiency and / or accuracy of the neural network model 110. Figure 2The image pre-processor 114 and the heat map generator 112 are explained in more detail.

[0032] It should be understood that Figure 1 The medical image processing system 102 shown is for illustration and not limitation. Another medical image processing system may include more, fewer, or different components. In addition, although ultrasound images are described herein, it should be understood that the medical images may include other types of medical images, such as X-ray images, CT images, MRI images, etc.

[0033] Figure 2 A block diagram of an exemplary embodiment of a medical image processing system 200 including an automated measurement system is shown, wherein the data flow between components of the medical image processing system is shown. The medical image processing system 200 is Figure 1 Non-limiting example of a medical image processing system 102 .

[0034] The medical image processing system 200 may include an automated measurement system 220 (e.g., Figure 1 202 is an automated measurement system 108 that can be operated by a user 250 to perform one or more automated measurements of a medical image 202. In one example, the medical image 202 is an ultrasound image generated by an ultrasound probe of an ultrasound system. In other examples, the medical image 202 can be an X-ray or CT scan or a different type of medical image. In some examples, the medical image 202 can be a single frame of a sequence of medical images acquired in real time. The medical image 202 may include one or more anatomical features acquired during an examination of a patient, such as a heart, liver, spleen, etc., wherein the user 250 can use the automated measurement system 220 to measure one or more elements of the one or more anatomical features to assist a specialist, for example, in performing a diagnosis of a patient. It should be understood that the examples provided herein are for illustrative purposes only and may include other types of medical images without departing from the scope of the present disclosure.

[0035] In some examples, medical image 202 may be obtained from a medical imaging system such as Figure 1 In some embodiments, the medical image 202 may be processed by an automated measurement system 220 within the medical imaging system 100 of the present invention, while in other examples, the medical image 202 may be processed by an automated measurement system 220 external to the medical imaging system. For example, the medical image 202 may be acquired by the medical imaging system and stored in a picture archiving and communication system (PACS), a radiology information system (RIS), or other suitable storage location. The medical image 202 may be displayed via a user interface (UI) / display device 204 accessed by a user 250. The UI / display device 204 may include a display device such as Figure 1The display device 118 (e.g., monitor, touch screen, etc.) and the UI / display device 204 may include a user input device such as Figure 1 Via the UI / display device 204 , the user 250 can initiate automated measurement of anatomical features of the medical image 202 .

[0036] The automated measurement system 220 may include a convolutional neural network (CNN) 210. In one example, the CNN 210 is trained to predict one or more measurement points of an automated measurement to be performed based on one or more user-selected measurement points. In one example, a user-selected starting point of an automated measurement is used to predict an end point of the automated measurement. In other examples, multiple measurement points of the automated measurement are used to predict one or more additional measurement points. In other words, the user 250 may select multiple measurement points for the automated measurement, and the automated measurement system may "fill in" or "autocomplete" multiple additional measurement points to perform the automated measurement.

[0037] Once a complete set of measurement points for an automated measurement is obtained, wherein the complete set of measurement points includes one or more user-selected measurement points and one or more predicted measurement points, the measurement algorithm 214 may perform an automated measurement based on the complete set of measurement points. In one example, the measurement algorithm 214 determines a path from a user-selected starting point to a predicted endpoint, and calculates the distance between the user-selected starting point and the predicted endpoint. In other examples, the automated measurement is a set of measurements, and the measurement algorithm 214 calculates the distance between the starting point and the endpoint of each component measurement of the measurement set, and sums these distances to obtain a total measurement value for the measurement set. The total measurement value may be displayed to the user on the UI / display device 204, and / or the distance between the connected starting point and endpoint may be displayed to the user on the UI / display device 204, so that the user 250 can see a visual display of the location where the automated measurement is performed and / or the complete set of measurement points used to perform the automated measurement. The total measurement value and / or distance between the connected starting point and endpoint may be recorded in one or more files of the user, and / or used in subsequent processes of the automated measurement system and / or the medical imaging system. As described below with reference to Figure 5 In more detail, user 250 may interact with UI / display device 204 to adjust one or more measurement points of the entire set of measurement points and repeat the automated measurement if user 250 is not satisfied with the results of the automated measurement.

[0038] When the user 250 initiates an automated measurement, one or more user-selected measurement points may be input into the heat map generator 206 of the automated measurement system 220. In one example, the heat map generator 206 outputs one or more heat maps 213, wherein each heat map 213 encodes a user-selected measurement point in the one or more user-selected measurement points via a 2D normal Gaussian probability density function. In other examples, the heat map generator 206 outputs a single heat map 213, wherein the heat map 213 includes an encoding of each user-selected measurement point in the one or more user-selected measurement points via a 2D normal Gaussian probability density function. As described below with respect to Figure 5 As described in further detail, heat map 213 may be used to input one or more user-selected measurement points into CNN 210 .

[0039] The heat map 213 generated by the heat map generator 206 can be paired with the medical image 202 to create a heat map / medical image input pair 212 to be input into the CNN 210. For example, if there are two user-selected measurement points on the medical image, the heat map / medical image input pair 212 can include two heat maps (e.g., one heat map for each user-selected measurement point) and the medical image, or the heat map / medical image input pair 212 can include a single heat map with encodings of the two user-selected measurement points and the medical image. Before input into the CNN 210, the heat map / medical image input pair 212 can be input into the image preprocessor 208, which can preprocess the heat map / medical image input pair 212 via one or more preprocessing routines. In one example, preprocessing the heat map / medical image input pair 212 includes preprocessing both images of the heat map / medical image input pair 212 according to a preprocessing routine in the one or more preprocessing routines. For example, if the pre-processing routine includes cropping, scaling, and / or skewing the image, both the medical image 202 and the thermal map 213 are cropped, scaled, and / or skewing according to the pre-processing routine. Figure 6 As described in more detail, preprocessing of the heat map / medical image input pair 212 may improve the efficiency of the CNN 210 .

[0040] Once the heat map / medical image input pair 212 has been preprocessed by the image preprocessor 208, the heat map / medical image input pair 212 may be input into the CNN 210, where the CNN 210 may predict one or more remaining measurement points in the entire set of measurement points used by the measurement algorithm 214 to perform automated measurement. In addition, the predicted one or more remaining measurement points used by the measurement algorithm 214 may be further paired with the heat map / medical image input pair 212 to create an input / target pair 216, which may be stored in the training database 218 for future training of the CNN 210. In other words, the input / target pair 216 may include an input element including the heat map / medical image input pair 212, and an output element including the predicted one or more remaining measurement points. In some examples, the result of the automated measurement may also be included in or associated with the input / target pair 216. For example, the result of the automated measurement may be a factor used to adjust the weighting of the CNN 210 based on the back propagation algorithm of the CNN 210.

[0041] One or more remaining measurement points predicted by CNN 210 may also be output to UI / display device 204, whereby user 250 may view the entire set of measurement points for performing automated measurements and / or adjust one or more measurement points in the entire set of measurement points to perform new automated measurements, as described above.

[0042] Reference now Figure 3A , shows an ultrasound image 300 of a patient's heart, including the septum 302, left ventricle 304, and posterior wall 306 of the heart. A method for use in an automated measurement system (e.g., Figure 2 The automated measurement system 220 of the embodiment of the present invention is used to perform automated measurement of the septum 302, the left ventricle 304, and the posterior wall 306 of the heart, including measurement point 308, measurement point 310, measurement point 312, and measurement point 314. The measurement point 308 indicates the outer boundary of the septum 302, and the measurement point 310 indicates the inner boundary of the septum 302, wherein the thickness of the septum 302 may be equal to the length of the line 309 between the measurement point 308 and the measurement point 310. The measurement point 314 indicates the outer boundary of the posterior wall 306 of the heart, and the measurement point 312 indicates the inner boundary of the posterior wall 306, wherein the thickness of the posterior wall 306 may be equal to the length of the line 311 between the measurement point 312 and the measurement point 314. The length of the line 313 between the measurement point 312 and the measurement point 310 may indicate the extent of the left ventricle 304 (e.g., the width of the left ventricle 304, or the distance between the septum 302 and the posterior wall 306).

[0043] In one example, a first set of measurement points including measurement point 308, measurement point 310, measurement point 312, and measurement point 314 comprises the entire set of measurement points for a single automated measurement. For example, a user of the automated measurement system may select a measurement type (e.g., a left ventricle measurement set) indicating that a series of related measurements of a cross-section of the left ventricle 304 is desired, including measurements of the septum 302, the posterior wall 306, and the distance between the septum 302 and the posterior wall 306. For the measurement set corresponding to the cross-section of the left ventricle 304, measurement point 308 may be considered a first starting point for a first component measurement of the left ventricle measurement set, and measurement point 310 may be considered a first endpoint for the first component measurement of the left ventricle measurement set. Measurement point 310 may be considered a second starting point for a second component measurement of the left ventricle measurement set, and measurement point 312 may be considered a second endpoint for the second component measurement of the left ventricle measurement set. Measurement point 312 may be considered a third starting point for a third component measurement of the left ventricle measurement set, and measurement point 314 may be considered a third endpoint for the third component measurement of the left ventricle measurement set.

[0044] In one example, the first set of measurement points is suggested to the user by the automated measurement system, whereby upon initiating the automated measurement and selection of the measurement type, the automated measurement system automatically generates the first set of measurement points (e.g., via a process such as described below). Figure 6 In the display device of the automated measuring system (e.g. Figure 2 After the first set of measurement points is displayed on the UI / display device 204 of the display device 204, the user may be prompted to accept the suggested first set of measurement points (e.g., the default selection). In one example, if the user accepts the suggested first set of measurement points, the automated measurement system performs automated measurement. If the user does not accept the suggested first set of measurement points, the user may adjust the position of one or more of measurement points 308, measurement points 310, measurement points 312, and measurement points 314, and command the automated measurement system to perform automated measurement based on the adjusted first set of measurement points. In some examples, adjustment of a measurement point (e.g., measurement point 308) may cause one or more of the suggested measurement points (e.g., measurement point 314) to be automatically adjusted.

[0045] In another example, Figure 3AThe first set of measurement points may include one or more user-selected measurement points and one or more measurement points predicted by the automated measurement system. For example, measurement point 308 may be selected by the user, while measurement point 310, measurement point 312, and measurement point 314 may be predicted by the automated measurement system; or measurement point 308 and measurement point 310 may be selected by the user, while measurement point 312 and measurement point 314 may be predicted by the automated measurement system; or measurement point 308 and measurement point 314 may be selected by the user, while measurement point 310 and measurement point 312 may be predicted by the automated measurement system. In addition, in some examples, one or more intermediate measurement points may be predicted between the measurement point that is the starting point of the automated measurement and the end point of the automated measurement. For example, during a segmentation task, multiple intermediate measurement points may be predicted to define the boundaries of a region of interest, as described below in Figure 4A and Figure 4B Described in more detail in .

[0046] Figure 3B A line graph 320 showing an ultrasound image of a heart includes Figure 3A The ultrasound image 300 of the septum 302, the left ventricle 304 and the posterior wall 306, wherein the septum 302 has a protrusion 322. The protrusion 322 is located on the septum 302. Figure 3A A dashed line 332 connects a first set of measurement points that may be suggested by an automated measurement system (e.g., from Figure 3A ). For example, a first set of measurement points may be suggested by a neural network model of the automated measurement system based on training data, wherein the training data includes ultrasound images of a heart without a bulging septum, but does not include ultrasound images of a heart with a bulging septum. Since the neural network model is not trained on images of a bulging septum, the first set of measurement points suggested by the automated measurement system may be poorly positioned, wherein the results of the automated measurement may not satisfy the user (e.g., the results may not provide an accurate measurement of the left ventricle 304 because the automated measurement is performed at the location of the bulge 322).

[0047] Since the user visually inspects the ultrasound image and detects the bulge 322 of the septum 302, the user may select the measurement point 324 as the starting point of the automated measurement instead of the measurement point 308. When the measurement point 324 is selected as the starting point of the automated measurement, the automated measurement system may predict the measurement point 326 on the inner boundary of the septum 302 as a substitute measurement point for the measurement point 310 for measuring the thickness of the septum, and may predict the measurement point 328 as a substitute measurement point on the inner boundary of the septum 302 and the measurement point 330 as a substitute measurement point on the inner boundary of the posterior wall 306 for measuring the extent of the left ventricle 304. The automated measurement system may predict the measurement point 312 and the measurement point 314 for measuring the thickness of the posterior wall 306 because the bulge 322 does not affect the thickness of the posterior wall 306.

[0048] As another example, the user may select measurement point 324 as the first starting point of the measurement set for the automated measurement, and may additionally select measurement point 328 as the second starting point of the automated measurement. For example, the user may wish to indicate to the automated measurement system a preference to measure the extent of the left ventricle 304 starting from measurement point 328 rather than measurement point 326. In response to the user selecting measurement point 324 as the first starting point of the measurement set and measurement point 328 as the second starting point of the measurement set, the automated measurement system may predict measurement point 326 (e.g., the first endpoint), measurement point 330 (e.g., the second endpoint), measurement point 312 as the third starting point, and measurement point 314 as the third endpoint based on the user-selected measurement point 324 and the user-selected measurement point 328.

[0049] Steering Figure 3C , showing an ultrasound image 350 (e.g., Figure 3A Ultrasound image 350 of FIG. 300 includes the septum 302, left ventricle 304, and posterior wall 306 of the heart. Ultrasound image 350 shows two alternative starting points for automated measurements performed by the automated measurement system. In one example, the automated measurements are similar to those described above with reference to FIG. Figure 3A The automated measurements described (eg, the measurement of the cross section of the left ventricle 304) are the same. In one example, the measurement point 308 is the starting point of the automatically generated measurement (eg, predicted by the automated measurement system).

[0050] An alternative starting point 352 is shown on the ultrasound image 350, where the alternative starting point 352 is selected by a user. For example, the measurement point 308 automatically selected by the automated measurement system can be a starting point based on a first measurement style, which the automated measurement system has learned because its machine learning model (e.g., CNN) is trained on multiple measurements performed in the first measurement style. The user-selected alternative starting point 352 can be based on a second measurement style, where the second measurement style is different from the first measurement style and reflects the user's training and / or preferences. In other examples, the measurement point 308 is a user-selected starting point selected by a first user, and the alternative starting point 352 is selected by a second user, where the first user performs measurements in the first measurement style, and the second user performs measurements in the second measurement style.

[0051] exist Figure 3D , ultrasound image 370 shows the measurement system used by the automated Figure 3C Alternative additional measuring points generated by two alternative starting points. Figure 3A A first set of measurement points is depicted in ultrasound image 370, including measurement point 308, measurement point 310, measurement point 312, and measurement point 314. In addition, a second set of measurement points is shown, including Figure 3C372, 374, and 376. The alternative starting point 352 is located on the outer boundary of the septum 302, the measurement point 372 is located on the inner boundary of the septum 302, the measurement point 376 is located on the outer boundary of the posterior wall 306, and the measurement point 374 is located on the inner boundary of the posterior wall 306. Therefore, the thickness of the septum 302 can be measured as the distance between the alternative starting point 352 and the measurement point 372 of the second set of measurement points, or as the distance between the measurement point 308 and the measurement point 310 of the first set of measurement points. Similarly, the thickness of the posterior wall 306 can be measured as the distance between the measurement point 376 and the measurement point 374 of the second set of measurement points, or as the distance between the measurement point 314 and the measurement point 312 of the first set of measurement points. The extent of the left ventricle 304 can be measured as the distance between the measurement point 372 and the measurement point 374 of the second set of measurement points, or as the distance between the measurement point 312 and the measurement point 310 of the first set of measurement points.

[0052] Thus, measurement points 310, 312, and 314 of the first set of measurement points are predicted by the automated measurement system based on measurement point 308, and measurement points 372, 374, and 376 of the second set of measurement points are predicted by the automated measurement system based on alternative starting point 352. By selecting different starting points for measuring the cross-section of the left ventricle 304 (including the septum 302 and the posterior wall 306), the first user can obtain the results of the first automated measurement and the second user can obtain the results of the second automated measurement. In other examples, a single user can select a preferred measurement of the two alternative measurements based on a preference for the first style or the second style.

[0053] Reference now Figure 4A , an ultrasound display 400 of an ultrasound image 402 is shown, wherein an automated segmentation task is performed on a region of interest (ROI) 404 of the ultrasound image 402. In one example, the automated segmentation task is performed by or in part by an automated measurement system (such as Figure 1 Automated measurement system 108 and / or Figure 2 The ultrasound display 400 may be displayed on a display device such as Figure 2 The user of the automated measurement system (e.g., Figure 2 of users 250) are displayed.

[0054] Performing the automated segmentation task may include estimating a boundary 406 of the ROI 404, wherein all of the ROIs 404 are within the boundary 406 and none of the ROIs 404 are outside the boundary 406. For example, segmentation may be used to identify anatomical features of interest (e.g., for subsequent measurements), to distinguish between volumes of abnormal tissue and volumes of normal tissue in a patient during an ultrasound examination, or other suitable tasks. In one example, estimating the boundary 406 includes selecting a plurality of segmentation measurement points 410 (in Figure 4A 404) and connects the segmented measurement points 410 to form a plurality of segments 412 that collectively approximate the shape of the ROI 404. The boundary 406 can be displayed to a user of the automated measurement system, where the segments 412 that approximate the shape of the ROI 404 can be superimposed on the ultrasound image 402, thereby allowing the user to more clearly see the shape of the ROI 404. The segments 412 of the boundary 406 can be used in subsequent processing routines, for example, to measure the area of ​​the ROI 404, to compare the shape of the ROI 404 with one or more shapes of anatomical features in other ultrasound images, and so on.

[0055] In some examples, the segmentation task is automatically performed by an automated measurement system, wherein the automated measurement system automatically selects the segmentation measurement point 410 and generates the segment without input from the user. The segmentation measurement point 410 may be selected by a machine learning segmentation model trained using one or more segmentation techniques (e.g., automatic ejection fraction (AutoEF), AFI ROI, etc.). However, in order to perform the segmentation task, one or more parameters of the segmentation task may be estimated by the automated measurement system. In one example, the automated measurement system may estimate the placement depth of the segmentation measurement point 410 and the corresponding segmentation into the tissue of the ROI 404. For example, the segmentation measurement point 410 may be placed at a first depth (e.g., on an estimated surface of the ROI 404), or the segmentation measurement point 410 may be placed at a second deeper depth (e.g., in tissue below the estimated surface of the ROI 404).

[0056] In one example, the segmentation task is performed semi-automatically, where the user selects one or more of the segmentation measurement points 410 to perform the segmentation task, and the segmentation model predicts one or more remaining segmentation measurement points 410 for performing the segmentation. For example, the user may select a starting point 408 for the segmentation task, and the automated measurement system may predict all of the one or more remaining segmentation measurement points 410 for performing the segmentation. By selecting the starting point 408, the user may provide a desired tissue depth to place the segmentation measurement points 410 to the automated measurement system. Due to receiving the starting point 408 of the segmentation task, the automated measurement system may set the depth of the tissue parameter based on the user input instead of estimating the depth of the tissue parameter, which may result in a more accurate segmentation and / or a more desired segmentation for the user.

[0057] Figure 4B An ultrasound display 450 of an ultrasound image 402 is shown, wherein a ROI 404 of the ultrasound image 402 is performed Figure 4A In ultrasound display 450, the user has selected a starting point 408 for the segmentation task and has additionally selected a second boundary point 452 for the segmentation measurement points 410. A first segment 454 is placed on boundary 406 by the automated measurement system, the first segment comprising a line connecting the starting point 408 and the second boundary point 452. One or more remaining segmentation measurement points 410 for performing the segmentation task may be predicted by the segmentation model of the automated segmentation system based on the starting point 408 and the second boundary point 452. By selecting the second boundary point 452, the user indicates that the segmentation measurement points 410 to be segmented are to be segmented. Figure 4A A preference is given to segmenting the ROI 404 by extending the left side of the boundary 406 slightly deeper into the ROI 404 , where the boundary point on the left side of the boundary 406 is predicted based on the starting point rather than the second boundary point 452 .

[0058] exist Figure 4B 4, the start point 408 and the second boundary point 452 are continuous points along the boundary 406. In other examples, the second boundary point 452 may not be a continuous boundary point to the start point 408, and the second boundary point 452 may be a different boundary point 410, or multiple boundary points of the segmentation measurement points 410 may be selected by the user. For example, the ROI 404 may include a portion of the boundary 406 that is clearly visible in the ultrasound image 402, and a portion of the boundary 406 that is not clearly visible. In addition to the start point 408, the user may select one or more segmentation measurement points 410 along the clearly visible portion of the boundary 406, and allow the automated measurement system to predict one or more segmentation measurement points 410 along the portion that is not clearly visible.

[0059] refer to Figure 5, a flow chart of a method 500 is shown for performing automated measurement of anatomical features of a medical image. The method 500 may be performed by an automated measurement system (such as Figure 1 Automated measurement system 108 and / or Figure 2 The automated measurement system 220 may also include a medical image processing system (such as Figure 1 In one embodiment, the operations of method 500 may be stored in a non-volatile memory and executed by a processor (e.g., respectively, by Figure 1 The medical image processing system 102 stores and executes the medical image processing system 102 in a non-volatile memory 106 and a processor 104).

[0060] At 502, method 500 includes, at a display device (eg, Figure 1 The medical image is displayed on a display device 118 of the medical image processing system 102. In one example, the medical image may be an ultrasound image. The ultrasound image may be a 2D ultrasound image or a 3D volume. In some examples, the ultrasound image may be a single frame of a sequence of medical images acquired from a patient in real time, for example, during an ultrasound examination performed by a clinician. In other examples, the medical image may be an X-ray image, a CT image, an MRI image, a visible light image, or another type of medical image that includes anatomical features of interest.

[0061] At 503, method 500 includes receiving a measurement type of an automated measurement to be performed. In one example, the measurement type is received from a user. In other examples, the automated measurement system may follow a scanning / measurement scheme, and the scheme may indicate the measurement to be performed. The measurement type may include a linear measurement in which the distance along a straight line between two points on an ultrasound image is measured, an area measurement in which a two-dimensional area of ​​an ultrasound image is measured, a volume measurement of a 3D image / volume, a Doppler measurement of blood flow, and / or another suitable measurement. For example, in a spectral Doppler measurement, one or more velocities and / or directions of blood flow captured by a Doppler ultrasound device are graphically displayed as a 2D curve. In some examples, the measurement type includes an anatomical feature in a medical image to be measured. For example, for an ultrasound cardiologist performing an ultrasound examination of a patient's heart, specifying a measurement type may include specifying that it is desired to measure the thickness of the patient's septum, or specifying that it is desired to measure the distance between the patient's septum and the left ventricular wall, or specifying that it is desired to measure a cross-section of the left ventricle including the septum and the left ventricular wall. In some examples, specifying the measurement type may include prompting a user via a display device to select a measurement type from a list of candidate measurement types, where the candidate measurement types are based on a type of ultrasound image, a specialty of the medical service in which the automated measurement is being performed, or another similar criteria.

[0062] At 504, method 500 includes providing, via a user input device such as Figure 2 The user interface 204 of the ultrasound system receives one or more user-selected measurement points of the automated measurement from the user. In one example, the starting point of the automated measurement is received from the user via a user input device. For example, the user can select the starting point by clicking on the location within the medical image with a user input device (e.g., a mouse or a touch screen), thereby transmitting the Cartesian coordinates of the location within the ultrasound image to the automated measurement system. In other examples, the one or more user-selected measurement points include the starting point of the automated measurement and one or more additional measurement points (e.g., one or more additional starting points and / or end points of the measurement set).

[0063] Alternatively, at 505, receiving the one or more user-selected measurement points of the automated measurement may include displaying one or more suggested measurement points of the automated measurement (e.g., a suggested start point, a suggested end point, and / or one or more suggested start points and end points of a measurement set), and receiving adjustments to one or more of the suggested measurement points from the user. The suggested measurement points may be selectable elements superimposed on the medical image, wherein the user may select and reposition the suggested measurement points via a user input device. Additionally, the suggested measurement points may be connected by one or more lines to indicate one or more segments of the automated measurement. For example, the user may select a suggested measurement point via user input, and adjust the position of the suggested measurement point by sliding the measurement point until the suggested measurement point reaches a desired new position. When the suggested measurement point is repositioned by the user, one or more lines (e.g., one or more segments) connecting the suggested measurement point to other suggested measurement points may be updated to the desired new position.

[0064] Thus, the user may select a starting point without guidance from the automated measurement system, or the user may select a starting point with guidance from the automated measurement system, for example, the automated measurement system may display one or more candidate starting points for the user to select. Candidate starting points may be generated based on the measurement type. For example, if the user specifies a measurement of the patient's diaphragm thickness in the above example, the automated measurement system may display one or more candidate starting points on the medical image based on historical data collected by the automated measurement system (e.g., from previous measurements, from manually selected starting points, etc.). Based on one or more algorithms of the automated measurement system, a single candidate starting point may be selected as a default starting point (e.g., if the default starting point is indicated by determined guidelines and / or historical data). In addition, the user may perform an automated measurement, and based on the results of the automated measurement, the user may repeat the automated measurement. If the user repeats the automated measurement, the starting point of the previous automated measurement may be displayed as a candidate starting point, where the user may adjust the position of the candidate starting point to specify a new starting point for the new automated measurement.

[0065] In other examples, the endpoint of a previous automated measurement may also be displayed, or one or more starting and ending points of a measurement set of a previous automated measurement may be displayed. Figure 3A The automated measurement may include measuring one or more distances between one or more starting points and endpoints of the automated measurement and summing the distances. If the one or more starting points and endpoints are displayed to the user, receiving the starting point of the automated measurement may also include receiving an adjustment to one or more of the one or more starting points and endpoints.

[0066] At 506, method 500 includes predicting additional measurement points for the automated measurement based on the user-selected measurement points. For example, the user may select a starting point, and the automated measurement system may predict an end point based on the starting point. As another example, the user may select one or more measurement points for the measurement set, and the automated measurement system may predict one or more additional measurement points for the measurement set based on the user-selected measurement points. Figure 6 Describes in more detail the prediction of additional measurement points for automated measurement based on user selected measurement points.

[0067] At 508, method 500 includes performing an automated measurement using the user-selected measurement point and the predicted additional measurement points. In one example, performing the automated measurement includes measuring a distance between a start point and an end point, or a distance between one or more start points and end points of a measurement set. For example, a user may select a start point for the automated measurement. Based on the start point for the automated measurement, the automated measurement system may predict a first end point, a second start point, a second end point, a third start point, and a third end point of the measurement set (e.g., as described above with reference to FIG. 1 ). Figure 3A As described above), a segmented measurement path can be created that begins at a starting point and continues through each of a first endpoint, a second starting point and endpoint, and a third starting point and endpoint of a measurement set, thereby forming three segments. A first length of the first segment can be calculated as the distance between the starting point and the first intermediate endpoint. A second length of the second segment can be calculated as the distance between the second starting point and the second endpoint. A third length of the third segment can be calculated as the distance between the third starting point and the third endpoint. The first length of the first segment, the second length of the second segment, and the third length of the third segment can be summed to produce a total measurement of the anatomical feature along the path. It should be understood that the terms "starting point" and "end point" refer to specific measurements, and a measurement point can be both the starting point of one measurement and the endpoint of another measurement.

[0068] At 510, method 500 includes displaying the entire set of measurement points including the user-selected measurement points and the predicted additional measurement points and the results of the automated measurement on a display device for acceptance by the user. As described above, the entire set of measurement points may be connected by one or more lines indicating one or more segments of the automated measurement. The results of the automated measurement may be displayed as annotations on the ultrasound image and / or may be displayed on a user interface of the automated measurement system (e.g., Figure 2 In some examples, separate measurements for one or more segments of the automated measurement may be displayed on a display device.

[0069] At 512, method 500 includes prompting the user to accept the automated measurement. In one example, the automated measurement system displays a pop-up element where the user can select a control (e.g., an accept button) to indicate that the user accepts the automated measurement, or select a different control (e.g., a cancel button) to indicate that the user does not accept the automated measurement. At 514, method 500 includes determining whether the user accepts the automated measurement. If it is determined at 514 that the user does not accept the automated measurement, method 500 returns to 505, where method 500 includes displaying the entire set of measurement points and receiving one or more adjustments to the entire set of measurement points as described above.

[0070] If it is determined at 514 that the user accepts the automated measurement, then method 500 proceeds to 516. At 516, method 500 includes recording measurement data of the automated measurement in a database. In one example, the measurement data includes user-selected measurement points and predicted additional measurement points, as well as results of the automated measurement, wherein the measurement data is stored in the database for the purpose of refining a neural network model for predicting predicted additional measurement points of the automated measurement based on the user-selected measurement points of the automated measurement, such as with respect to Figure 6 Furthermore, the automated measurements may be stored in memory along with the medical images, for example as part of the patient's electronic medical record and / or as part of a study / examination of the patient. Method 500 then ends.

[0071] Reference now Figure 6 , a flow chart of an exemplary method 600 is shown for predicting one or more additional measurement points based on one or more user-selected measurement points when measuring anatomical features of a medical image. In one example, one or more convolutional neural networks (CNNs) are used to predict the one or more additional measurement points based on the one or more user-selected measurement points. The method 600 may be implemented by an automated measurement system such as Figure 1 Automated measurement system 108 and / or Figure 2 In one example, the method 600 is as described above with respect to Figure 5 A portion of method 500 is performed.

[0072] At 601, method 600 includes receiving one or more user-selected measurement points (also referred to herein as user-selected measurement points) from a user of an automated measurement system. In one example, the user-selected measurement points are received by the automated measurement system as a plurality of sets of x / y coordinate points, wherein each set of x / y coordinate points corresponds to a user inputted via a user input device (e.g., Figure 2 The method of claim 10 is to map a plurality of x / y coordinate points in a medical image to a position on a medical image selected by the user of the UI / display device 204. However, machine learning algorithms (e.g., machine learning algorithms of CNNs) have been shown to perform poorly when mapping coordinates in a Cartesian coordinate space to a pixel space (e.g., a medical image). Therefore, the accuracy and / or efficiency of the machine learning algorithm may be improved by encoding multiple sets of x / y coordinate points into one or more heat maps, wherein user-selected measurement points input into a CNN of one or more CNNs are represented in pixel space.

[0073] At 602, method 600 includes encoding one or more user-selected measurement points as a heat map. In one example, each set of x / y coordinate points is encoded as a heat map by generating a 2D normal Gaussian probability density function with a standard deviation (e.g., 10 pixels) centered on the set of x / y coordinate points. The heat map can be generated at a first resolution (e.g., 512×256 pixels), where the first resolution matches the resolution of the medical image (e.g., raw data of the medical image). Once the heat map is generated, the heat map can be attached to the medical image, where the heat map and the medical image represent inputs into different channels of the CNN. As described above with reference to Figure 2 As described above, a single heat map may be used to represent multiple sets of x / y coordinate points, or multiple heat maps may be used to represent the multiple sets of x / y coordinate points.

[0074] For example, the medical image may be an ultrasound image of a portion of a patient's heart, and the user may select a starting point for performing automated measurements of the heart by clicking on the medical image with a mouse (e.g., as described above with respect to Figure 5 A set of x / y coordinates of a starting point may be received from a user by an automated measurement system. A first heat map having a resolution of 512×256 may be created (e.g., Figure 2 A heat map 213 of the medical image is provided, wherein the parameters (e.g., brightness, color, etc.) of each pixel of the first heat map are adjusted as a function of the set of x / y coordinates of the starting point according to a 2D normal Gaussian probability density function, so that the starting point is represented as a visual Gaussian pattern (also referred to herein as a Gaussian) in pixel space, wherein the starting point is at the center of the Gaussian. Thus, the overlay of the first heat map on the medical image (e.g., with the same size / resolution) indicates the location of the starting point on the medical image, which corresponds to the set of x / y coordinates of the starting point.

[0075] In other examples, a user may select a plurality of user-selected measurement points for performing automated measurements of the heart, wherein each of the plurality of user-selected measurement points is received as a set of x / y coordinates. A second heat map (e.g., also having a 512×256 resolution) may be created, wherein a parameter (e.g., brightness, color, etc.) of each pixel of the second heat map is adjusted as a function of each user-selected measurement point according to a 2D normal Gaussian probability density function, such that each user-selected measurement point is represented as a different Gaussian in pixel space. Thus, the overlay of the second heat map on the medical image indicates the location of each user-selected measurement point on the medical image, which corresponds to the set of x / y coordinates of each user-selected measurement point.

[0076] At 604, method 600 includes preprocessing the heat map and the medical image (also referred to herein as the input image) prior to input into the CNN. For example, the input image can be cropped and / or rescaled to a different resolution (e.g., to more closely align the expected spatial distribution of the Gaussian with one or more physical dimensions of the medical image), and / or the input image can be preprocessed in another similar manner.

[0077] At 606, method 600 includes inputting the input image to a selected CNN among the one or more CNNs. In one example, the one or more CNNs are trained CNNs, wherein each of the one or more CNNs is trained for multiple medical images of a particular anatomical feature of interest and / or different measurements of the anatomical feature, and the selected CNN is selected based on the anatomical feature of interest of the medical image and / or the type of measurement being performed. For example, the one or more CNNs may include a first CNN trained for multiple medical images of a first anatomical feature (e.g., heart), a second CNN trained for multiple medical images of a second anatomical feature (e.g., lungs), a third CNN trained for multiple medical images of a third anatomical feature (e.g., liver), and so on. Thus, if the input image includes a medical image of the heart, the automated measurement system may select the first CNN; if the input image includes a medical image of the lungs, the automated measurement system may select the second CNN; and so on. In one example, the selected CNN is selected based on the measurement type selected by the user, as described above with reference to Figure 5 The method 500 is described below. Figure 7 The training of one or more CNNs is described in more detail.

[0078] At 608, method 600 includes receiving one or more predicted additional measurement points output by the selected CNN based on the input image. In one example, the one or more predicted additional measurement points are output by the selected CNN in the form of an output heat map, wherein each of the one or more predicted additional measurement points is visually represented on the output heat map. In one example, each predicted additional measurement point is visually represented as a discrete point (e.g., a point), while in other examples, each predicted additional measurement point is visually represented by a function of the predicted additional measurement points (e.g., a probability density function), which function may be a Gaussian. For example, if three additional measurement points are predicted from a user-selected measurement point encoded in a heat map of the input image, the output heat map may include three Gaussians positioned on the output heat map.

[0079] In one example, an output heat map output by the selected CNN including one or more predicted additional measurement points is used by an automated measurement system to display the one or more predicted additional measurement points on a display device, as described above with reference to Figure 5 In other examples, one or more predicted additional measurement points output by the selected CNN in the output heat map are converted into multiple sets of x / y coordinates for display on a display device.

[0080] At 610, method 600 includes storing one or more user-selected measurement points (e.g., input points input into the selected CNN) and one or more predicted additional measurement points in a database (e.g., output points output by the selected CNN). In one example, the input points and output points are stored for training purposes of the selected CNN. For example, a training set including the input points and output points may be periodically used to further train the selected CNN to improve the accuracy of the CNN. Then, method 600 ends.

[0081] Reference now Figure 7 , a flow chart of an exemplary method 700 for training a convolutional neural network (CNN) to predict one or more additional measurement points based on a user-selected starting point during automated measurement of anatomical features of a medical image. The automated measurement may be performed by an automated measurement system, such as Figure 1 Automated measurement system 108 and / or Figure 2 220. CNN can be used with Figure 2 CNN 210 and / or references above Figure 6 The CNN described in method 600 is the same or similar.

[0082] At 702, method 700 includes generating a training set of medical images. In one example, medical images are collected from medical images stored in one or more databases, wherein the stored medical images are acquired during an examination of a patient. The medical images may be classified based on imaged anatomical features. For example, a first category of medical images may include medical images of the heart, a second category of medical images may include medical images of the lungs, and so on. Each category of medical images may be used to create a corresponding training set. For example, a heart training set may be created using images from the first category, a lung training set may be created using images from the second category, and so on.

[0083] At 704, method 700 includes: for each medical image of the training set, selecting a plurality of measurement points that can be used to measure anatomical features of the medical image. The plurality of measurement points may include a start point for the measurement and an end point for the measurement, as well as one or more additional start points and end points (e.g., one or more additional start points and end points for the measurement set). In one example, the plurality of measurement points are manually selected by one or more experts (e.g., expert users of the medical imaging system). In other examples, the plurality of measurement points may be obtained from automated measurements, or the plurality of measurement points may be adjustments to automated measurements made by one or more experts. For example, a user of an automated measurement system may perform automated measurements and select (or adjust and select) one or more measurement points generated by the automated measurement system for the training set.

[0084] At 706, method 700 includes: for each medical image of the training set, assigning one of the plurality of measurement points as a starting point for measurement, wherein the starting point will be input into the CNN, and the CNN will predict one or more remaining measurement points based on the starting point. The starting point can be any measurement point as long as the starting point is consistent in each image. In some examples, the starting point can be the topmost point, the bottommost point, the leftmost point, the rightmost point, a corner point, a point located at a specific anatomical feature or region, or another suitable point. An expert may indicate the starting point, or the starting point may be identified in an automatic manner (e.g., via a segmentation algorithm). In some examples, as described above, one or more additional measurement points may also be assigned for input into the CNN. At 708, method 700 includes: for each medical image of the training set, encoding the starting point into one or more input heat maps (e.g., as described above with reference to Figure 6 ), and encoding one or more remaining measurement points of the plurality of measurement points into one or more target heat maps (e.g., used as reference truth measurement points during training of the CNN).

[0085] At 710, method 700 includes: for each medical image in the training set, creating and preprocessing an input image pair for training a CNN, wherein each input image pair includes at least one input heat map and a medical image, and the input image pair is as follows: Figure 6The method 700 is preprocessed as described in the method 600. At 712, the method 700 includes: for each medical image in the training set, creating an input / target training pair for training a CNN in one or more CNNs, wherein each input / target training pair includes an input image pair and a target heat map.

[0086] At 714, method 700 includes training the CNN for the input / target training pairs of the training set. Training the CNN for the input / target training pairs may include learning a mapping from the input image pairs to the target heat map for each input / target training pair, wherein a gradient descent algorithm is used to iteratively adjust one or more weights of the CNN (e.g., via backpropagation) to minimize a difference between one or more output heat maps of the CNN and the target heat map. After training the CNN for the input / target training pairs of the training set, the automated measurement system may use the trained CNN to predict one or more additional measurement points based on the one or more user-selected training points in a semi-automatic mode, as described above with reference to Figure 5 described.

[0087] By following method 700, a plurality of trained CNNs may be created, wherein each of the plurality of trained CNNs corresponds to a training set in the one or more training sets described above (e.g., a CNN for the heart, a CNN for the lungs, etc.). In other examples, more than one measurement point may be input into the CNN, and training the CNN for the input / target training pair may include learning a mapping from the more than one measurement point to a target heat map for each input / target training pair, as described above with reference to Figure 3A and Figure 5 described.

[0088] Therefore, the present disclosure provides a system and method for an automated measurement system for performing automated measurement of anatomical features of a medical image in a semi-automatic mode. In the semi-automatic mode, a user selects one or more measurement points for performing automated measurement, and the automated measurement system predicts one or more additional measurement points for performing measurement based on the one or more user-selected measurement points. In this way, the automated measurement system "automatically completes" the selection of measurement points indicated by the user to perform measurement. The one or more user-selected measurement points and the predicted additional measurement points can be superimposed on the medical image and displayed to the user on a display device, wherein if the user is not satisfied with the automated measurement, the user can adjust one or more of the one or more user-selected measurement points and / or the predicted additional measurement points, and repeat the automated measurement using the adjusted one or more user-selected measurement points. The one or more user-selected measurement points can also be used to predict measurement points along the boundaries of the anatomical feature, wherein the accuracy of the segmentation task performed on the anatomical feature can be improved due to the initiation of the segmentation task based on the one or more user-selected measurement points. The technical effect of initiating automated measurement based on the one or more user-selected measurement points is that the user can ensure that the automated measurement is performed at a location preferred or desired by the user, thereby improving the accuracy of the measurement and / or subsequent diagnosis or treatment based on the measurement. Another advantage of the present disclosure is that the accuracy of the automated measurement system can be improved when the anatomical features are deformed. Another advantage of the methods provided herein is that they can be applied to various different types of measurements, including 2D mode measurements, segmentation measurements, Doppler measurements, area measurements, and volume measurements. In spatial measurements, providing a starting point for automated measurements allows the user to select the location where the measurement is performed. In segmentation measurements, providing a starting point for automated measurements can help the algorithm of the automated measurement system determine how deep in the tissue to perform the segmentation. In Doppler measurements, providing a starting point for automated measurements can help the algorithm determine sensitivity parameters. In general, by allowing the user to perform measurements according to a preferred method, the desired simplicity and speed of automated measurements are retained while improving the user's satisfaction with automated measurements.

[0089] The present disclosure also provides support for a method, the method comprising: receiving one or more user-selected measurement points for automated measurement of anatomical features of a medical image from a user; predicting one or more additional measurement points for automated measurement based on the one or more user-selected measurement points and based on a comparison of the anatomical features of the medical image with a plurality of images of the anatomical features via a trained network; and performing automated measurement of the anatomical features based at least on the one or more additional measurement points. In a first embodiment of the method, the trained network is a convolutional neural network (CNN). In a second embodiment of the method, the second embodiment optionally includes the first embodiment, predicting one or more additional measurement points for automated measurement based on the one or more user-selected measurement points and based on a comparison of the anatomical features of the medical image with a plurality of images of the anatomical features via a trained network comprises: encoding the one or more user-selected measurement points as a heat map and inputting the heat map into the CNN. In a third embodiment of the method, the third embodiment optionally includes one or both of the first and second embodiments, encoding one or more user-selected measurement points into a heat map includes: receiving the one or more user-selected measurement points from the user as multiple sets of x / y coordinates, and representing each of the one or more user-selected measurement points on the heat map as a function of a set of x / y coordinates of the user-selected measurement point. In a fourth embodiment of the method, the fourth embodiment optionally includes one or more or each of the first to third embodiments, wherein the function is a 2D normal Gaussian probability density function. In a fifth embodiment of the method, the fifth embodiment optionally includes one or more or each of the first to fourth embodiments, the method further includes: rescaling the heat map and the medical image used as input to the CNN to align the expected spatial distribution of the one or more user-selected measurement points with one or more physical dimensions of the medical image. In a sixth embodiment of the method, the sixth embodiment optionally includes one or more or each of the first to fifth embodiments, the CNN is configured to output an output heat map indicating the location of each of the one or more additional measurement points. In a seventh embodiment of the method, the seventh embodiment optionally includes one or more or each of the first to sixth embodiments, receiving one or more user-selected measurement points for automated measurement from a user further includes prompting the user to specify the one or more user-selected measurement points for automated measurement on a medical image via a user input device. In an eighth embodiment of the method, the eighth embodiment optionally includes one or more or each of the first to seventh embodiments, performing automated measurement includes further performing the automated measurement based on a type of measurement to be performed received from the user via a user input device, and further includes displaying a result of the automated measurement on a display device and / or storing the result of the automated measurement in a memory.In a ninth embodiment of the method, the ninth embodiment optionally includes one or more or each of the first to eighth embodiments, and the method further includes: after performing the automated measurement, displaying a whole set of measurement points and the result of the automated measurement on a display device, the whole set of measurement points including the one or more user-selected measurement points and the one or more additional measurement points. In a tenth embodiment of the method, the tenth embodiment optionally includes one or more or each of the first to ninth embodiments, receiving one or more user-selected measurement points of the automated measurement from the user further includes: performing an initial automated measurement without user input based on the measurement type; displaying one or more recommended measurement points selected from the initial automated measurement on the medical image; and receiving adjustments to the one or more recommended measurement points from the user, and setting the adjusted one or more recommended measurement points as the one or more user-selected measurement points. In an eleventh embodiment of the method, the eleventh embodiment optionally includes one or more or each of the first to tenth embodiments, and the method further includes: after the automated measurement, prompting the user to accept the automated measurement; in response to the user accepting the automated measurement, saving the automated measurement to a database; in response to the user not accepting the automated measurement, prompting the user to adjust one or more measurement points in the entire set of measurement points; and repeating the automated measurement using the adjusted one or more measurement points in the entire set of measurement points. In a twelfth embodiment of the method, the twelfth embodiment optionally includes one or more or each of the first to eleventh embodiments, and the anatomical feature includes part or all of the heart.

[0090] The present disclosure also provides support for a system, the system including a controller having computer-readable instructions stored on a non-transitory memory, the computer-readable instructions, when executed, causing the controller to: in response to a user initiating automated measurement of an anatomical feature of a medical image, prompt the user to select one or more measurement points on the medical image, and in response to the user selecting the one or more measurement points and initiating the automated measurement: predicting one or more measurement points of the automated measurement based on the one or more user-selected measurement points, and performing the automated measurement by measuring the distance along a path between the one or more user-selected measurement points and the one or more predicted measurement points. In a first embodiment of the system, predicting the one or more measurement points includes: encoding the one or more user-selected measurement points as a heat map, inputting the heat map and the medical image into a model, and receiving the one or more predicted measurement points as output of the model. In a second embodiment of the system, the second embodiment optionally includes the first embodiment, encoding the one or more user-selected measurement points as a heat map also includes: encoding the one or more user-selected measurement points as a function of one or more groups of x / y coordinates on the heat map, wherein each group of x / y coordinates corresponds to a user-selected measurement point in the one or more user-selected measurement points, and the function is a 2D normal Gaussian probability density function. In a third embodiment of the system, the third embodiment optionally includes one or both of the first and second embodiments, and the model is a convolutional neural network (CNN). In a fourth embodiment of the system, the fourth embodiment optionally includes one or more or each of the first to third embodiments, and the automated measurement is one of a Doppler measurement, a 2D B-mode measurement, an area measurement, and a volume measurement.

[0091] The present disclosure also provides support for a method, the method comprising receiving one or more user-selected segmentation measurement points on a boundary of an anatomical feature of a medical image from a user; predicting one or more segmentation measurement points on a boundary of the anatomical feature based on the one or more user-selected segmentation measurement points; and performing a segmentation task of the anatomical feature based on a whole set of segmentation measurement points, the whole set of segmentation measurement points including the one or more user-selected segmentation measurement points and the one or more segmentation measurement points. In a first embodiment of the method, the method further comprises: after performing the segmentation task, receiving one or more adjustments to the whole set of segmentation measurement points from the user; and repeating the segmentation task based on the one or more adjustments.

[0092] When introducing the elements of various embodiments of the present disclosure, the words "one", "a kind of" and "the" are intended to mean one or more elements present in these elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "include", "comprises", and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "coupled to", etc., an object (e.g., material, element, structure, member, etc.) can be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that reference to "an embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features.

[0093] In addition to any previously indicated modifications, numerous other variations and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present description, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described above in detail in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications may be made, including but not limited to form, function, mode of operation, and use, without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are intended to be illustrative in all respects only and should not be construed as limiting in any way.

Claims

1. A method comprising: receiving from a user one or more user-selected measurement points of automated measurement of anatomical features of a medical image; predicting one or more additional measurement points for the automated measurement based on the one or more user-selected measurement points and based on a comparison of the anatomical feature of the medical image with a plurality of images of the anatomical feature via a trained network; performing said automated measurement of said anatomical feature based at least on said one or more additional measurement points; After the automated measurement, prompting the user to accept the automated measurement; In response to the user accepting the automated measurement, saving the automated measurement to a database; In response to the user not accepting the automated measurement, prompting the user to adjust one or more measurement points in the entire set of measurement points; as well as The automated measurement is repeated using the adjusted one or more measurement points in the entire set of measurement points.

2. The method of claim 1, wherein the trained network is a convolutional neural network.

3. The method of claim 2, wherein predicting one or more additional measurement points for the automated measurement based on the one or more user-selected measurement points and based on a comparison of the anatomical feature of the medical image with a plurality of images of the anatomical feature via the trained network comprises: The one or more user-selected measurement points are encoded as a heat map and the heat map is input into the convolutional neural network.

4. The method of claim 3, wherein encoding the one or more user-selected measurement points into the heat map comprises: The one or more user-selected measurement points are received from the user as a plurality of sets of x / y coordinates, and each of the one or more user-selected measurement points on the heat map is represented as a function of the set of x / y coordinates of the user-selected measurement point. The method according to claim 4 , wherein the function is a 2D normal Gaussian probability density function.

6. The method according to claim 3, further comprising: The heat map and the medical image used as input into the convolutional neural network are rescaled to align an expected spatial distribution of the one or more user-selected measurement points with one or more physical dimensions of the medical image.

7. The method of claim 3, wherein the convolutional neural network is configured to output an output heat map indicating the location of each of the one or more additional measurement points.

8. The method of claim 1, wherein receiving the one or more user-selected measurement points for the automated measurement from the user further comprises prompting the user to specify the one or more user-selected measurement points for the automated measurement on the medical image via a user input device.

9. A method according to claim 1, wherein performing the automated measurement includes further performing the automated measurement based on a type of measurement to be performed received from the user via a user input device, and also includes displaying the result of the automated measurement on a display device and / or storing the result of the automated measurement in a memory.

10. The method according to claim 9, further comprising: After performing the automated measurement, a whole set of measurement points and a result of the automated measurement are displayed on a display device, the whole set of measurement points including the one or more user-selected measurement points and the one or more additional measurement points.

11. The method of claim 10, wherein receiving the one or more user-selected measurement points of the automated measurement from the user further comprises: performing an initial automated measurement based on the measurement type without user input; displaying on the medical image one or more suggested measurement points selected from the initial automated measurement; as well as Receive an adjustment to the one or more suggested measurement points from the user, and set the adjusted one or more suggested measurement points as the one or more user-selected measurement points.

12. The method of claim 1, wherein the anatomical feature comprises part or all of a heart.

13. A system comprising: A controller having computer readable instructions stored on a non-transitory memory, the computer readable instructions, when executed, causing the controller to: In response to a user initiating automated measurement of an anatomical feature of a medical image, prompting the user to select one or more measurement points on the medical image; as well as In response to the user selecting the one or more measurement points and initiating the automated measurement: predicting one or more measurement points of the automated measurement based on the one or more user-selected measurement points; as well as performing the automated measurement by measuring the distance along the path between the one or more user selected measurement points and the one or more predicted measurement points, Wherein predicting the one or more measurement points comprises: encoding the one or more user-selected measurement points as a heat map; inputting the thermal map and the medical image into a model; and The one or more predicted measurement points are received as an output of the model.

14. The system of claim 13, wherein encoding the one or more user-selected measurement points as a heat map further comprises: The one or more user-selected measurement points are encoded as a function of one or more sets of x / y coordinates on the heat map, wherein each set of x / y coordinates corresponds to a user-selected measurement point in the one or more user-selected measurement points, and the function is a 2D normal Gaussian probability density function.

15. The system of claim 13, wherein the model is a convolutional neural network.

16. The system of claim 13, wherein the automated measurement is one of a Doppler measurement, a 2D B-mode measurement, an area measurement, and a volume measurement.

17. A non-transitory computer-readable storage medium for storing a computer program, wherein when executed by a computer, the computer program causes the computer to perform the method according to any one of claims 1 to 12.

Images