Computer-implemented method for the automated analysis of bias in measurements performed on medical images of anatomical structures - Patents.com

Abstract

The present invention relates to a computer-implemented method for automated analysis of bias in measurements performed on medical images of anatomical structures, the method comprising the steps of receiving a plurality of medical images, the plurality of medical images including at least a first plurality of medical images and a second plurality of medical images, each of the plurality of medical images including the same anatomical structure of the same patient or different patients; receiving, for the first plurality of medical images, first measurement data R1 of a feature of the anatomical structure, the first measurement data R1 being acquired by a first user; identifying, for the first plurality of medical images, first baseline measurement data C1 of the feature of the anatomical structure by an automated measurement tool C1; and determining, by the first user, first baseline measurement data C1 of the feature of the anatomical structure based on the first measurement data R1 and the first baseline measurement data C1. receiving second measurement data R2 of the anatomical structure feature for a second plurality of medical images, the second measurement data R2 being acquired by a second user, the second user being different from the first user; identifying second baseline measurement data C2 of the anatomical structure feature for the second plurality of medical images by the automated measurement tool 1; identifying a second measurement bias B2 for the second user based on the second measurement data R2 and the second baseline measurement data C2; and comparing the first measurement bias B1 with the second measurement bias B2.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for the automated analysis of user-specific biases in measurements performed on medical images of anatomical structures, a computer program, and an automated measurement tool configured to implement the method. [Background technology]

[0002] Medical ultrasound measurements are used in a wide range of medical procedure areas. For example, echocardiographic measurements are typically used as an indirect method to assess the dimensions and function of human or animal anatomical structures, especially the human heart. However, measurement results are affected by inter- and intra-observer discrepancies. Bland-Altman analysis is a common method used in the prior art to identify both variability and bias between two different observers (e.g. users or physicians). For accreditation purposes, laboratories performing measurements (especially echocardiography laboratories) must undergo a standardization step to comply with quality guidelines. Furthermore, this accreditation includes an interpretive quality review (physician interpretation variability). The institution must evaluate the quality and accuracy of the interpretation based on the acquired images. A minimum of two examples per modality per department must be evaluated for the quality and accuracy of the interpretation of the acquired images. The examples must be representative of as many physicians as possible. To achieve a uniform observation interpretation, the interpretation differences must be adjusted. In a daily clinical routine, this results in an additional manual systematic workload. This is even worse in the case of long-term clinical studies, where the measurement observer often changes during the conduct of the study. Thus, problems of inter-observer variability and consensus training exist.

[0003] It is therefore desirable to have an independent detection and measurement system with standardized performance and uniform bias that simplifies compliance with quality guidelines, and that can be used to calibrate and / or normalize measurement results of different users.

[0004] Christian Knackstedt, in "Fully Automated Versus Standard Tracking of Left Ventricular Ejection Fraction", JOURNAL OF THE AMERICAN COLLEGE OF CARDIOLOGY; VOL. 66, NO. 13, 2015, discloses a comparison of human and machine obtained measurements, where left ventricular volume and longitudinal strain measurements were evaluated.

[0005] US Patent Application Publication No. 2018150598(A1) discloses using machine learning to compare acquired anatomical measurements with predefined thresholds and generate advisory information for adjusting the measurements.

[0006] Vikram Chalana, in "A Methodology for Evaluation of Boundary Detection Algorithms on Medical Images", IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL.16, NO.5, October 1997, focuses on comparing the difference in contour acquisition, i.e., comparing computer-generated boundaries with those generated by multiple observers and calculating the Cohen Kappa coefficient of agreement between the two observers.

[0007] US2014341449A1 discloses reducing inter- and intra-observer variability in image contour acquisition by using a consensus contour generated by machine learning to verify stability. Further, US2014341449A1 evaluates the degree of deviation of a user-generated contour from the consensus contour.

[0008] Harold L. Kunde, in "Measurement of Observer Agreement" in Radiology 2003;228:303-308, discloses a statistical measure for measuring agreement between two readers using Cohen's Kappa, which is used to measure categorical outcomes (positive vs. negative).

[0009] The anonymous paper "Man against machine: AI is better than dermatologists at diagnosing skin cancer" describes a study in which the performance of dermatologists and artificial intelligence is reviewed. A convolutional neural network is implemented as the artificial intelligence. Thus, both the dermatologist and the AI ​​evaluate a portion of human skin to determine whether the lesion is malignant or non-malignant. The results of the dermatologist and the AI ​​are then compared with the correct results. It was found that the AI ​​missed fewer melanomas and misdiagnosed benign moles as malignant less frequently than the dermatologists.

[0010] As outlined above, echocardiographic measurements are an example of an indirect method to assess cardiac dimensions and function in humans. Results will vary between measurements from two observers (interobserver), or even between two measurements from the same observer (intraobserver). A common method for the analysis of both variability (i.e., margin of error for the difference between the two observers, or 95% confidence interval) and bias (i.e., systematic error between the two observers, or mean difference) is the Bland-Altman analysis. The Bland-Altman method can be used to evaluate the agreement between two series of measurements. In particular, the Bland-Altman method provides information on the influence of the height of the measurements on the size of the deviation / difference. For this purpose, the difference (measurement difference) is represented as a function of the mean (measurement height) in a scatter plot (Bland-Altman plot).

[0011] In addition to the regression line with confidence intervals, the zero line (difference=0) and the mean difference (also with confidence intervals) and possibly also the limits of agreement are usually displayed. This information can be used to assess the degree of redundant sizing. Summary of the Invention [Problem to be solved by the invention]

[0012] It is therefore an object of the present invention to provide a computer-implemented method for the automatic analysis of bias in measurements performed on medical images of anatomical structures, whereby quality guidelines are met and the manual systematic workload is reduced. It is also an object of the present invention to provide a respective computer program, and an automatic measurement tool for carrying out the method. [Means for solving the problem]

[0013] In order to better address one or more of the concerns pointed out above, in a first aspect of the present invention, a computer-implemented method for the automated analysis of bias in measurements performed on medical images of anatomical structures is presented in claim 1. Useful embodiments are presented in the dependent claims.

[0014] According to this first aspect, a computer-implemented method for automated analysis of bias in measurements performed on medical images of anatomical structures comprises the steps of: receiving a plurality of medical images, the plurality of medical images including at least a first plurality of medical images and a second plurality of medical images, each of the plurality of medical images including the same anatomical structure of the same patient or different patients; receiving first measurement data of a feature of an anatomical structure for a first plurality of medical images, the first measurement data being acquired by a first user; identifying a first baseline measurement of an anatomical feature by an automated measurement tool for a first plurality of medical images; determining a first measurement bias for the first user based on the first measurement data and the first baseline measurement data; receiving second measurement data of the feature of the anatomical structure for a second plurality of medical images, the second measurement data being acquired by a second user, the second user being different from the first user; identifying a second baseline measurement of the anatomical feature by an automated measurement tool for a second plurality of medical images; determining a second measurement bias for a second user based on the second measurement data and the second baseline measurement data; comparing the first measurement bias to a second measurement bias; has.

[0015] Therefore, in one embodiment, the present invention provides a method in which the same medical image is analyzed by both a human user and an automated measurement tool, and the results are compared with each other to obtain a measurement bias for the human user. Thus, two different measurement instances or observations (human and machine) are used on the same medical image, and a bias is identified between them. This bias is then used to estimate the bias between different humans, e.g. a first user and a second user.

[0016] In general, bias is a measure of the degree to which measurement data differ from the estimated true underlying quantitative parameter. For example, a measurement is biased if it is determined in such a way that it systematically differs from the estimated population parameter. Moreover, bias is a discrepancy measure (i.e., deviation measure) or margin of error, in other words, the systematic error or average difference between two users (i.e., observers).

[0017] In particular, systematic errors change (i.e. drift) during an experiment; that is, the measurements show trends over time rather than varying randomly around a mean value. Drift is evident when a measurement of a certain quantity is repeated multiple times and the measurements drift in one direction during the experiment. If the next measurement is higher than the previous one, as occurs when an instrument gets warm during an experiment, for example the measured quantity fluctuates and it is possible to detect drift by checking for zero readings during and at the beginning of the experiment. On the other hand, if there is no pattern in the series of repeated measurements, the presence of a fixed systematic error can only be observed if the measurement is checked by measuring a known quantity or by comparing the readings (i.e. the measurement results) with readings obtained using a different device known to be more stable (i.e. providing measurements without drift). In particular, the present invention is particularly useful in the latter case. That is, according to the present invention, baseline measurement data (described below) is obtained by an automatic measurement tool (considered to be a different device). Furthermore, the automatic measurement tool is also called the baseline machine. In other words, the present invention provides a calibration / normalization tool that relates a given user's measurement bias to the systematic (and subjective) differences that the user produces when making a given measurement on an anatomical structure contained in multiple medical images.

[0018] Generally, the first and second measurement biases are absolute or relative measurement biases. More specifically, the first and second absolute measurement biases for the first and second user are preferably determined by calculating the difference between the first / second measurement data and the first / second baseline measurement data, respectively, i.e., by subtracting one from the other. On the other hand, the relative measurement bias is defined by normalizing the measurement data (i.e., for the type of measurement). In other words, the relative measurement bias is determined based on the relative difference between the measurement data and the baseline measurement data instead of the difference between the measurement data and the baseline measurement data (i.e., compared to the absolute measurement bias). In particular, the relative measurement bias is determined by the following formula:

number

[0019] Absolute bias is therefore functionally related to the physical properties and / or data of the underlying measurement data. In particular, absolute bias is functionally related to the properties of the modality used in generating the medical image, the properties of the measurement tool used in generating the measurement data, the conditions present during the measurement and / or during the generation of the medical image, and / or the absolute value of the measurement data. Absolute measurement bias is therefore not easily compared to other absolute measurement biases obtained based on different measurement data (e.g. measurement data including measurements of different features of an anatomical structure). Relative bias is independent of such functional relationships. As a result, relative measurement biases can be compared to other relative measurement biases based on different measurement data.

[0020] Therefore, according to another embodiment of the present invention, a relative difference between the measurement data and the baseline measurement data is determined for the measurement bias. That is, by determining the relative difference between the measurement data and the baseline measurement data, a relative bias is determined (for example as outlined above). The relative bias is particularly useful when a relatively small dimension is measured (for example the thickness of the septum of the heart), a deviation of 10% from the baseline measurement data results in a relatively small absolute bias. On the other hand, when a relatively large dimension is measured (for example the diameter of the left ventricle of the heart), a deviation of 10% results in a relatively high absolute bias. As a result, both measurements are accompanied by the same percentage deviation, but the absolute bias for each measurement is different. Therefore, in order to correct this deviation of the absolute bias, a relative bias is determined. As a result, the relative measurement biases are compared with each other independently of the measurement data on which the measurement bias is based. In other words, for example, independent of the characteristics of the anatomical structure being measured.

[0021] In the following discussion, for convenience, absolute bias and relative bias will be referred to simply as bias, but the term bias always refers to both absolute bias and relative bias.

[0022] Additionally, a per-user bias is identified, which is represented by the average of all measured biases for a particular user.

[0023] A medical image is generated by any medical imaging modality, for example ultrasound, especially echocardiography, or MRI (magnetic resonance imaging), X-ray imaging, CT (computed tomography), PET (positron emission tomography). When using ultrasound, the image may be obtained by color Doppler, tissue Doppler, continuous wave Doppler, pulsed wave Doppler, or M-mode, i.e. there are various acquisition modes on which manual or automatic measurements are based. A medical image is a 2D or 3D image. A medical image (or a set of medical images) is one or more or all of a sequence of frames of 2D or 3D images that make up a 4D image, where time is the fourth dimension.

[0024] The anatomical structure is any anatomical structure of the human or animal body, such as a limb, an organ or a blood vessel, such as the heart, the liver or a fetus. The same structure means that if a first medical image of the first plurality includes the heart as an anatomical structure, then the second medical image of the first plurality also includes the heart as an anatomical structure.

[0025] The medical images may originate from the same patient or from different patients, i.e. the medical images may originate from the same patient but at different time points. Furthermore, the method or modality used to acquire the medical images may be different for each image. For example, the first plurality of medical images is acquired using an echocardiography system and the second plurality of medical images is acquired, for example, using an MR system. Furthermore, the first plurality of medical images and the second plurality of medical images do not necessarily have to be received at the same location. That is, the method of the invention is partially performed at different locations, i.e. distributed across multiple locations. Preferably, only the step of comparing the first measurement bias with the second measurement bias is centralized (i.e. performed at one location), whereas other previous method steps are performed at other times. In particular, the method steps of receiving first measurement data of a feature of an anatomical structure for a first plurality of medical images, the first measurement data being acquired by a first user, determining first baseline measurement data of the feature of an anatomical structure by an automated measurement tool for the first plurality of medical images, and determining a first measurement bias for the first user based on the first measurement data and the first baseline measurement data are performed at one location. Similarly, the corresponding method steps related to the second plurality of medical images are performed at a different location. As a result, only the first measurement bias and the second measurement bias need to be transmitted to be compared with each other. In other words, the first medical images are received and processed (i.e. receiving the first measurement data and determining the first baseline measurement data) at a central laboratory located in Europe, whereas the second medical images are received and processed (i.e. receiving the second measurement data and determining the second baseline measurement data) at a central laboratory located in the United States. Furthermore, the comparison of the first and second measurement biases may be performed at different locations or at any of the locations mentioned above, i.e., only the first and second measurement biases need to be transmitted in order to compare them.The transmission is carried out, for example, via the Internet.

[0026] Naturally, the present invention extends to a third plurality of medical images and a further plurality of medical images, and associated third measurement data and baseline measurement data and further measurement data and baseline measurement data, to identify a third measurement bias and a further measurement bias of a third user and a further user.

[0027] The measurement data represents a physical parameter, such as a size, in particular a length, an area or a volume, of the anatomical structure being analyzed. The measurement data may be a flow rate, a duration or other physical parameter that can be measured from a medical image. More particularly, the measurement data includes a diameter and / or a circumference of a heart valve, such as the mitral valve and / or the aortic valve. The measurement data is obtained manually by at least one user. More particularly, the measurement data is obtained by a user, for example, by tracing a structure in a medical image. Furthermore, the measurement data is additionally or alternatively generated by a user by drawing a 3D freehand line in a 3D data set. Furthermore, the measurement data includes a contour of the anatomical structure.

[0028] The acquisition of the medical images is performed by the same user, who further performs (manual) measurements to obtain the measurement data, and the observation is further performed by other clinical staff, as is usually done in MR observations.

[0029] In one embodiment, each of the plurality of medical images includes one or more medical image sets, each medical image set being acquired during one observation session from one patient. Preferably, one measurement data of the anatomical structure feature and one baseline measurement data of the anatomical structure feature are acquired from each medical image set, and preferably each of the plurality of medical images includes multiple sets. Thus, typically, each of the plurality of medical images includes medical image sets from multiple patients. However, the plurality of medical images may include two or more medical image sets acquired in different observation sessions from the same patient, as in the case of longitudinal observation. For example, one medical image set includes all ultrasound images acquired during one observation session from a patient's heart, and thus includes many 2D and / or 3D images from various viewing angles and in different fields of view.

[0030] The user may be any person working in the field of medical image analysis, such as a doctor, a nurse, an employee in a measurement laboratory, etc. The first user and the second user are usually different people. In some cases, the first user and the second user are the same person who generates measurement data at different times or using different measurement techniques.

[0031] In one embodiment, the first (second) measurement data of the feature of the anatomical structure is generated by a first (second) user performing manual measurements on one or more of the first (second) plurality of images, for example by measuring a physical parameter such as length or area. Such measurements are usually performed manually, i.e. requiring user intervention, but may also be computer-assisted, for example by the user clicking on certain markers in a medical image displayed on a computer screen, and the distance between the markers being calculated automatically by the computer. Generally, the measurement data is obtained using computer-implemented measurement techniques that require user intervention, for example such that the user is responsible for the accuracy of the measurement data.

[0032] By receiving it is meant that the measurement data is received in real time, i.e. immediately after the measurement data is acquired. Alternatively, the measurement data is received from storage, i.e. previously generated and stored measurement data is later analyzed. As a result, older investigation results carried out in the past are newly analyzed and thus further processed using the method of the present invention. Furthermore, the measurement data is received by an automatic measurement tool. In addition, the first measurement data is received by a first automatic measurement tool and the second measurement data is received by a second automatic measurement tool, the first automatic measurement tool and the second automatic measurement tool being separate tools.

[0033] The first and second baseline measurement data of the anatomical structure features are obtained automatically, i.e. without user intervention, by an automated measurement tool. Optionally, the user selects a particular medical image from which the baseline measurements are obtained, for example one medical image from a set of medical images obtained during one observation session. Preferably, the baseline measurement data is obtained by a computer-implemented algorithm. Such algorithms are conventional algorithms, for example algorithms using techniques such as feature tracking, segmentation techniques (e.g. thresholding, region growing, pixel histogram analysis, etc.). Alternatively, the algorithms are machine learning algorithms, in particular algorithms based on deep learning techniques and including one or more artificial neural networks. Such artificial neural networks are trained on images from a reference database and on measurement data obtained by an expert user, optionally verified by a second user to minimize any errors.

[0034] The baseline measurement data is determined by the automated measurement tool, which includes all standard measurements that are usually performed on anatomical structures. Furthermore, the automated measurement tool determines the baseline measurement data by an algorithm (included in the automated measurement tool) that includes rules based on a reference database. More specifically, the reference database is generated by at least one expert (human), and the algorithm is configured to measure bias in a free state and / or with a certain bias relative to the reference database. In addition, the results of the automated measurement tool are reproducible, i.e., when the same measurement data is available, the same results (i.e., baseline measurement data) are obtained by the automated measurement tool. In connection with the step of receiving measurement data as described above, the first baseline measurement data and the second baseline measurement data are determined by the first automated measurement tool and the second automated measurement tool.

[0035] The automatic measurement tool is run in the background of currently running applications to determine baseline measurement data hidden from the user. That is, the user should not necessarily be aware that baseline measurement data is currently being determined. The automatic measurement tool is preferably stable in its results, that is, the automatic measurement tool produces the same results (output) for the same image or sequence of images (input). In particular, the standard measurements include all important measurements performed on a particular anatomical structure. If the anatomical structure is the heart, the important measurements include, for example, measurements of the diameter of the heart valves, the contours of the heart chambers. The automatic measurement tool is connected to, for example, a PACS (Picture Archiving and Communication System) of a hospital or a research institute. Thus, all medical images introduced into such a PACS are taken into account by the automatic measurement tool. As a result, a number of measurement biases are automatically identified by the automatic measurement tool. Furthermore, the automatic measurement tool is included in the modalities for generating and / or performing measurements on the medical images. More specifically, the measurement biases are identified directly (i.e. in situ) within the respective modalities. Therefore, only the measurement bias needs to be transmitted in order to compare it with a further measurement bias. Moreover, even a step of comparing two measurement biases with each other may be performed within the modality, and the result of such a comparison may be output.

[0036] When several measurement data of an anatomical structure measured by a first user (e.g. the diameter of the left ventricle) are received, baseline measurement data is identified by the automated measurement tool for each of the measurement data. Then, the difference between each measurement data and the baseline measurement data is identified, so that for each measurement data there is baseline measurement data. Then, a first measurement bias is identified as the mean or average value of each of the baseline measurement data for the first user. The first measurement bias is thus an expression of the systematic difference between the measurements performed by the first user compared to the acquired baseline measurement data. The same applies for the second measurement bias.

[0037] Finally, the first measurement bias is compared to the second measurement bias, e.g., by subtracting one from the other, or simply by reviewing the measurement bias for each user. The method of the present invention is extended to additional users, e.g., by repeating the steps for a third user, a fourth user, and additional users.

[0038] In both longitudinal and multi-center studies, the bias comparison results can be used to normalize / calibrate measurements between users.

[0039] According to the above-mentioned embodiment of the invention, an automatic analysis of two or more observers (even on different patients) is performed. Through a bias comparison of each user (more precisely, their measurement data) using the automatic measurement tool, an average bias between the two users can be calculated for all patients. This result is used to give feedback to all involved physicians (users) for routine clinical procedures and / or clinical longitudinal studies, without the need to measure the same study. For this function, the medical images measured by the users are used to perform the analysis using the automatic measurement tool, in order to minimize error sources between the users and the automatic measurement tool. In case of clinical longitudinal studies (both longitudinal and multi-center studies), the bias comparison result can be used to normalize the measurements between observers in case of clinical study scenarios. Through applying the measurements to all available views, more measurements are generated than in the manual case, thus resulting in a more stable system with less statistical noise in the analysis. Thus, a reduction in the manual systematic workload to ensure high quality can be ensured. Using the above-mentioned method, the user measurements are normalized and / or calibrated. The method is globally applicable, i.e., the method is not limited to local. Moreover, different users do not need to be at the same location, therefore the method offers greater flexibility in its application.

[0040] According to a further embodiment of the invention, the method further comprises the steps of identifying a global bias represented by an average value of all available measurement biases and comparing the global bias with one of the measurement biases.

[0041] The global bias is obtained by transmitting the locally determined measurement biases to a centralized facility. At the centralized facility, the global bias is determined by calculating the average or mean value of all locally determined measurement biases. For example, the measurement biases are transmitted from the PACS and / or modalities including the automated measurement tool to the centralized facility. Furthermore, to determine the global bias, a bias that is already an average value of multiple biases is used. Furthermore, the automated measurement tool may include a systematic bias for all human observers (i.e. all users). By determining the global bias, this systematic bias is determined. In addition, each user compares his / her own bias with the global bias to evaluate his / her own bias. Furthermore, a measurement bias against a standard value for a specific measurement result of an anatomical structure (e.g. published in some guidelines) is determined and compared with the global bias. Furthermore, the user compares his / her own bias with the global bias. Finally, the user compares both comparison results against the standard value to evaluate by himself / herself. Thus, each user validates by himself / herself.

[0042] According to a further preferred embodiment of the present invention, the first measurement data is determined based on one particular medical image from the first plurality of medical images and the first baseline measurement data is determined based on the same particular medical image. Further, the second measurement data is determined based on one particular medical image from the second plurality of medical images and the second baseline measurement data is determined based on the same particular medical image.

[0043] Generally, a user performing measurements on an anatomical structure to obtain measurement data selects a specific image (i.e., a specific view of the anatomical structure) in a medical image set that ensures the most accurate measurement results. The selected image is then used to perform the measurement. To minimize error sources between humans (i.e., the user) and machines (i.e., the automatic measurement tool), the image selected by the user is further used to perform the analysis by the automatic measurement tool. This makes the process very efficient, requiring less computing resources and less time. However, multiple medical image sets or all available images of each medical image set are analyzed by the automatic measurement tool. As a result, more measurements are generated than in the manual case, thus resulting in a more stable system with less statistical noise in the analysis.

[0044] In one embodiment, each measurement data and the baseline measurement data are identified based on the same particular medical images from each medical image set acquired during one observation session of the patient.

[0045] According to a preferred embodiment of the present invention, the measurement bias is continuously updated after new measurement data becomes available, either all available measurement data is used to update the measurement bias, or all available measurement data acquired during a predefined past period is used to update the measurement bias.

[0046] In other words, updated means that the bias is newly determined based on new available measurement data. Furthermore, continuously means that the measurement bias is frequently updated. Furthermore, the update means that the old measurement bias is replaced by the newly determined measurement bias. This update process is performed automatically without any specific user action. Furthermore, the update is performed so that the user is not aware that the update is being performed. The update is performed by calculating a new average incorporating the new difference between the new baseline measurement data and the measurement data. After new measurement data becomes available, the update is performed as frequently as possible. In some examples, if the method steps are performed at different locations, there is a delay in updating the measurement bias because no connection for the transmission of the newly determined measurement bias or measurement data goes out. Then, when the newly determined measurement bias or measurement data can be transmitted, the update is performed.

[0047] In one embodiment, the measurement bias for each user is based on the measurement data acquired during a predefined past period, such as the past one to three months. Thus, older measurement data no longer contributes. Rather, at any point in time, the measurement bias of a user represents the average bias of that person in all measurements performed during the predefined past period. That is, the measurement bias is updated based on the measurement data generated during a predefined period to update the measurement bias using a moving average of the measurement bias during the period. Alternatively, all available measurement data is taken into account by updating the measurement bias. That is, the newly acquired measurement data (and thus the newly identified measurement bias) is taken into account in addition to all old measurement bias / measurement data.

[0048] According to the above described embodiment, the measurement bias is automatically adapted to new situations (eg after the user has participated in training), i.e. higher / lower measurement accuracy of the user.

[0049] According to a further preferred embodiment of the invention, the method further comprises a step of determining a facility-wide bias represented by an average value of all measurement biases of the first user, the second user and the further user belonging to one given group or facility.

[0050] In other words, the bias across a site is the average of all the measurement biases of users belonging to a site, where the site may be a central laboratory, a hospital, a research team, etc. However, the bias across a site is the average bias of a group of users consisting of a given number of users. The average value is the sum of all the values ​​divided by the number of values.

[0051] As a result, it is easy to objectively assess the biases of a working group without having to assess each of the individual biases of each user. Moreover, different groups or facilities are easily compared to each other.

[0052] According to a preferred embodiment of the invention, the method further comprises the step of comparing the facility-wide bias with one of the measurement biases to obtain a facility-wide bias versus a per-user bias.

[0053] In other words, the bias per user is the individual bias of the user. Furthermore, the bias across the entire facility is the same as outlined above. Thus, each user is evaluated against the average of the measured biases of all users belonging to the facility. Furthermore, the evaluation is smoothed in that it can easily drive which users improve the overall facility bias and which users reduce the overall facility bias.

[0054] According to a preferred embodiment of the present invention, the step of comparing the facility-wide bias with one of the measurement biases comprises: determining a difference by subtracting each of the measurement biases by the facility-wide bias to determine a difference for each user belonging to the same group or facility; and determining one or more users with the largest difference among all users.

[0055] In other words, a further value (i.e., difference) is calculated that represents the difference between the average bias of the entire facility and the measurement bias per individual user. Thus, it is facilitated to identify which users need to improve their measurement accuracy. Furthermore, the above-described embodiment allows to generate an agenda regarding which users need to increase their accuracy in order to improve the bias of the entire facility to a predetermined level.

[0056] According to a further embodiment of the present invention, the determination of at least the first baseline measurement data and the second baseline measurement data is performed at different locations. In other words, the measurement data is further processed (including the determination of the baseline measurement data and the determination of the measurement bias) at the same location where the measurement data is acquired or received, for example. That is, an automatic measurement tool is present at the respective location where the measurement data is acquired or received in order to determine the baseline measurement data locally. In addition, the measurement bias is further determined locally at the same location where the measurement data is acquired or received. As a result, only the baseline measurement data and / or the measurement bias need to be transmitted for further processing.

[0057] According to a further preferred embodiment, the automated measurement tool uses artificial intelligence to identify the baseline measurements, i.e., the automated measurement tool operates according to a rules-based approach. For example, the automated measurement tool automatically performs a grey value analysis of the medical images to identify boundaries between tissue and blood flow. Furthermore, the automated measurement tool performs an image analysis of the medical images to find specific geometric objects. Based on those analyses, the measurement tool identifies the features of the anatomical structures to be measured. Later, the automated measurement tool performs measurements to obtain baseline measurement data.

[0058] According to a further preferred embodiment, the automated measurement tool uses machine learning to identify baseline measurements, and the automated measurement tool is trained with a reference database comprising reference images of anatomical structures and measurement data obtained by at least one expert user on the reference medical images.

[0059] The reference database includes examples of standard measurement results (i.e., exemplary measurement dates) generated by experts. In particular, the standard measurements include all important measurements performed on a particular anatomical structure. As a result, the automated measurement tool measures any anatomical structure based on the exemplary dates included in the reference database. Furthermore, the automated measurement tool identifies baseline measurement data such that bias relative to the reference database is minimized and / or is constant. Furthermore, the identification of baseline measurement data is reproducible, i.e., when the same input data (i.e., medical images) are input, the automated measurement tool outputs the same output (i.e., baseline measurement data). In other words, the results identified by the automated measurement tool are reproducible. Therefore, the automated measurement tool is suitable for reliable baseline measurements.

[0060] According to a further preferred embodiment of the present invention, the automated measurement tool uses an artificial neural network to identify the baseline measurement data.

[0061] More specifically, artificial neural networks (also called connectionist systems) are computing systems that are generally not programmed with task-specific rules, but rather "learn" to perform a task by considering examples. For example, in image recognition, artificial neural networks learn to identify images that contain specific content by analyzing example images that have been manually labeled as containing that content, and using the results to identify the specific content in other images. Artificial neural networks do this without using any prior knowledge of the specific content. Alternatively, they automatically generate discriminatory features from the examples they process. In the same manner, the artificial neural network according to the present embodiment analyzes a reference database. In particular, the reference database contains examples of standard measurements performed by experienced experts with very low interobserver bias. As a result, the artificial neural network performs measurements on unknowns (i.e. newly acquired medical images) with substantially the same bias as experienced experts. Furthermore, the artificial neural network analyzes all images of a plurality of medical images to generate an average bias that is less susceptible to measurement errors that occur in a particular image. Moreover, artificial neural networks are based on a collection of connected units or nodes called artificial neurons. Each connection can transmit a signal to other neurons, similar to a synapse in a biological brain. An artificial neuron that receives a signal can then process it and inform the neurons connected to it.

[0062] Therefore, the average value (i.e., baseline measurement data) output by analyzing multiple medical images of the same anatomical structure is less susceptible to measurement errors occurring in a particular image, i.e., a single outlier is harmless, whereas the baseline measurement data only needs to be accurate in the average value.

[0063] According to a preferred embodiment of the present invention, the measurement data includes at least one of the following: a size, e.g., a length, an area or a volume, or a volume flow rate, a blood velocity, and a tissue velocity. The present invention applies to a wide range of measurement data obtained from medical images. Further examples for the measurement data are given in "The American Society of Echocardiography Recommendations for cardiac chamber Quantification in Adults: A Quick Reference Guide from the Ase Workflow and Lab Management Task Force", "Recommendations for Cardiac Chamber Quantification by Echocardiography in Adults: An Update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging" published by Lang et al. in the Journal of the American Society of Echocardiography Volume 28, and "Recommendations for the Evaluation of Left Ventricular Diastolic Function by Echocardiography: An Update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging" published by Nagueh et al. in the Journal of the American Society of Echocardiography in April 2016. All of the above citations are incorporated herein by reference. In addition, the measurement data may include fetal dimensions such as at least one of crown-rump length, apex transparency, biparietal diameter, femur length, humerus length, head circumference, and abdominal circumference.Additionally, the measurement data may include Doppler flow profiles, heart valve diameters, and / or ventricular dimensions.

[0064] According to a further preferred embodiment of the present invention, for a measurement bias, a new measurement bias and an old measurement bias are identified, the new measurement bias being identified when a predetermined period of time has elapsed since the old measurement bias was identified, or the new measurement bias being identified when a predetermined amount of new measurement data is available after the old measurement bias is identified, and the new measurement bias is compared with the old measurement bias to obtain a drift in the measurement bias.

[0065] In other words, the old measurement bias is the measurement bias identified before the new measurement bias. Furthermore, the two measurement biases are compared with each other, each of which is separated from the other by a predetermined period (e.g., 1 to 3 months) or a predetermined amount of measurement data (e.g., 10 to 100, preferably 20 to 50 measurement data) obtained since the previous measurement bias was identified. As a result, a trend analysis is performed to identify in which way the measurement bias changes from the old measurement bias to the new measurement bias. Thus, the drift related to the previously defined consensus is recognized. That is, the consensus is defined at the same time when the old measurement bias is identified (e.g., at the beginning of the collection of data). This knowledge information is particularly interesting for central laboratories and for long-term studies.

[0066] According to another aspect, the invention provides a computer program comprising program code instructions which, when executed by a processor, enable the processor to carry out the method of the invention. The computer program is any code, in particular code suitable for computer vision applications, in particular for the purpose of measuring features of anatomical structures in medical images. In particular, computer vision (also called machine vision) is a technique used to provide imaging-based automated inspection and analysis. This technique is therefore particularly useful for use in automated measurement tools.

[0067] In a further aspect, the present invention relates to a computer readable medium comprising a computer program as defined above. The computer readable medium is any digital data storage device, for example a USB stick, a hard disk, a CD-ROM, an SD card or an SSD card. Of course, the computer program does not have to be stored on such a computer readable medium to be supplied to the customer, but may also be downloaded via the internet.

[0068] According to a further aspect, the invention relates to an automatic measurement tool configured to perform the method according to the one outlined above, the automatic measurement tool comprising an interface configured to receive medical images generated by an imaging system and to output results determined by performing the method of the invention. Any feature or useful embodiment described in connection with the method of the invention further applies to the automatic measurement tool. Furthermore, the automatic measurement tool is included in an imaging modality for acquiring medical images and / or in a measurement device for generating measurement data based on the medical images. In particular, the measurement data is transmitted to the interface of the automatic measurement tool as input to the automatic measurement tool. The transmission is established via a cable, a near-field connection, Wi-Fi, W-Lan or other suitable connection. After processing the above-mentioned method steps, at least one of the results is output via the interface of the automatic measurement tool. Thus, the automatic measurement tool is applied locally at the location where the measurement data was acquired or received. Alternatively or additionally, the measurement tool is connected to a database (e.g. PACS) storing a plurality of measurement data. Furthermore, the measurement tools may be provided separately as well as a server connected to several clients (e.g. imaging modalities, measurement modalities, data storage, separate users, facilities, groups, etc.). Furthermore, the method steps may be performed by more than one automatic measurement tool, i.e. one automatic measurement tool determines the measurement bias and another automatic measurement tool updates the measurement bias or determines, for example, a global bias. Furthermore, the automatic measurement tools may be located at different locations.

[0069] When the automatic measurement tool is implemented in an imaging system, the medical images are internally transmitted from the imaging system to the automatic measurement tool, and the measurement bias is calculated in situ by the automatic measurement tool within the system itself. As a result, only the measurement bias needs to be transmitted (output from the automatic measurement tool) for further processing. Therefore, the amount of data transmitted is significantly reduced, and there is no need to establish a large capacity transmission line for the purpose of evaluating the measurement bias. Furthermore, the automatic measurement tool is applied in workstations and in PACS (Picture Archiving and Communication System) systems.

[0070] In a preferred embodiment, the imaging system is an ultrasound system.

[0071] Useful embodiments of the present invention will now be described with reference to the accompanying drawings, in which like elements or features are designated by the same reference numerals, in which: [Brief description of the drawings]

[0072] [Figure 1] FIG. 1 is a schematic cross-section through the human heart (four-chamber view). [Diagram 2] FIG. 1 illustrates a schematic diagram of an automated metrology tool according to an embodiment of the present invention. [Diagram 3] FIG. 2 is another schematic diagram illustrating user measurement results relative to baseline measurement data obtained by an automated measurement tool according to an embodiment of the present invention. [Figure 4] 3 is a flow chart illustrating steps of a method according to one embodiment of the present invention. [Diagram 5] FIG. 1 illustrates a system including an automated metrology tool according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0073] In order to better visualize the preferred application of the method of the invention for the automated analysis of biases in measurements performed on medical images of anatomical structures, FIG. 1 shows the structure of the human heart as an example of an anatomical structure. Blood coming from the lungs flows into the left atrium 2 and from there through the mitral valve 3 into the left ventricle 4. From the left ventricle 4, blood is pumped into the aorta 6 through the aortic valve 5. This part is also called the left ventricular outflow tract (LVOT). Blood coming from the body flows into the right atrium 7 and is pumped into the right ventricle 9 through the tricuspid valve 8. From the right ventricle 9, blood is pumped into the pulmonary artery 11 through the pulmonary valve. The heart wall 12 is made of muscle tissue that surrounds the heart chambers 2, 4, 7 and 9. The left and right ventricles are separated by a septum 1. From FIG. 1, it can be seen that the heart has a complex shape and in addition moves continuously with a heartbeat, i.e., the heart is a dynamic anatomical structure. Such anatomical structures, in particular the human heart, are recorded using medical imaging systems, such as, for example, echocardiography systems, ultrasound systems, MR, CT, etc., to obtain medical images. The medical images are later used to measure features of the anatomical structures, for example, during routine clinical procedures and during clinical longitudinal studies. Due to the complexity of the anatomical structures, measurements of the shape, contour, distances, and / or area of, for example, the mitral valve 3, to analyze features of the anatomical structures are difficult and prone to individual errors. Generally, designated institutions carry out such measurements to generate measurement data, i.e., the measurement data are generated in hospitals or other medical facilities. For accreditation purposes, these facilities (e.g. echocardiography laboratories) need to carry out a standardization process (as already outlined in the introduction).

[0074] The method of the invention allows to evaluate and minimize intra-institutional bias between readers (i.e. users) in the same institution, e.g. a hospital or a central laboratory, without additional workload. In this embodiment shown in Fig. 2, two users from the same central laboratory independently read the study objects, i.e. independently measure the same anatomical structures of a first patient 15 and a second patient 16. The first user obtains first measurement data R1 and the second user obtains second measurement data R2. The central laboratory now wants to improve the quality in terms of inter-user variability and needs to document it as well. However, the laboratory needs stability with previous studies, therefore a global drift of the measurement results cannot be tolerated.

[0075] Therefore, the automatic measurement tool 1 (baseline machine) reads all the surveys in parallel. That is, for each of the manual measurements of the first user and the second user, the automated pendant (i.e. the first baseline measurement data C1 and the second baseline measurement data C2) is performed and stored. In this embodiment, the baseline measurement data C1, C2 are not displayed to the respective users. Later, for the first user, a first measurement bias B1 is determined based on the first measurement data R1 and the first baseline measurement data C1, and for the second user, a second measurement bias B2 is determined based on the second measurement data R2 and the second baseline measurement data C2. In other words, as shown in FIG. 3, the first measurement bias B1 and the second measurement bias B2 are determined based on the difference between the first measurement data R1 and the second measurement data R2 and the first baseline measurement data C1 and the second baseline measurement data C2, respectively. In an alternative embodiment, relative biases B1, B2 are determined based on the relative differences between the first and second measurement data R1, R2, and the first and second measurement baseline data C1, C2. Additionally, a per-user bias Bx is determined and continuously updated. In this embodiment, the measurement bias B1 for the first user and the measurement bias B2 for the second user.

[0076] By comparing the first measurement bias B1 with the second measurement bias B2, an average bias between the two users can be calculated. This result can be used to give feedback to all involved users for both routine clinical echocardiography scenarios and longitudinal clinical trial scenarios without the need to measure the same study. Thus, the use of the automatic measurement tool 1 to identify the measurement bias for each user provides a location-independent measure (i.e., used anywhere on the planet) similar to the "metre des Archives" related to length measurements. For the above mentioned functions, the medical images selected by the user can be used to perform the analysis using the automatic measurement tool 1 in order to minimize the error sources between the user and the automatic measurement tool 1.

[0077] According to a further embodiment of the invention, by applying the above mentioned method, a site-wide bias Bc between all users and the automated measurement tool 1 is identified and continuously updated. Furthermore, a difference dx=(Bx-Bc) can be associated to each user. The user with the largest difference in dx of the users can be selected for consensus training. Minimizing dx for all users reduces the inter-observer variability without introducing an overall drift in the central laboratory measurements. Monitoring the site-wide bias Bc over time ensures a stable reading behavior across the central laboratory. Monitoring the bias Bx per user over time ensures a stable reading for each user x or an intended change due to consensus training. Documentation of all these values ​​can be used to document the reading quality, the training plan and the evaluation training effect.

[0078] According to a further embodiment, the characterization of the automatic measuring tool 1 is performed substantially in a covert manner, i.e. none of the users are aware of the characterization being performed in the background.

[0079] Moreover, the automated measurement tool 1 can be further used to minimize inter-laboratory biases, since the automated measurement tool 1 does not necessarily match the average reading technique of the central laboratory study, i.e. the inter-laboratory bias Bc is large, in other words, the bias is large without compromising its usefulness as a benchmark between users and / or between user groups.

[0080] The TOMTEC® Arena could be one possible application of such an invention, but the image and information management system IntelliSpace Cardiovascular (ISCV) represents a good option as well. Moreover, such a system could even be integrated into the ultrasound system, since communication of the value of the bias Bx per user is sufficient to make it work, i.e., no transmission of image data is required.

[0081] With reference to FIG. 4, the steps of the method for automatic analysis of bias in measurements performed on medical images of anatomical structures are explained in more detail.

[0082] First, in step 90, a plurality of medical images are received. The plurality of medical images includes at least a first plurality of medical images and a second plurality of medical images, each of the plurality of medical images including the same anatomical structure of the same or different patient. In this embodiment, the plurality of medical images are not received by the same automated measurement tool 1, but by two automated measurement tools 1. That is, the first plurality of medical images are received by the first automated measurement tool 1, and the second plurality of medical images are received by the second automated measurement tool 1. Furthermore, the automated measurement tools 1 are located at different locations.

[0083] Then, in step 92, first measurement data R1 of the feature of the anatomical structure is received for a first plurality of medical images. The first measurement data R1 is acquired by a first user. Then, in step 94, first baseline measurement data C1 of the feature of the anatomical structure is determined by the automated measurement tool 1 for the first plurality of medical images. Then, in step 96, a first measurement bias B1 for the first user is determined based on the first measurement data R1 and the first baseline measurement data C1.

[0084] Simultaneously or subsequently, in step 98, second measurement data R2 of the anatomical structure feature is received for a second plurality of medical images. The second measurement data R2 is acquired by a second user, which is different from the first user, i.e. the first user and the second user are different people from each other. According to an alternative embodiment, the first user and the second user are the same person. Then, in step 100, second baseline measurement data C2 of the anatomical structure feature is determined by the other automatic measurement tool 1 for the second plurality of medical images. Simultaneously, in step 102, a second measurement bias B2 for the second user is determined based on the second measurement data R2 and the second baseline measurement data C2.

[0085] Finally, in step 104, the first measured bias B1 is compared to the second measured bias B2.

[0086] According to a further embodiment, the method further comprises, after step 104, a step 106 of determining a facility-wide bias Bc represented by the average value of all measurement biases B1, B2, ..., Bx of the first user, the second user and the further user belonging to one given group or facility.

[0087] According to yet another embodiment, the method further comprises, after step 106, a step 108 of comparing the facility-wide bias Bc with one of the measured biases B1, B2, ..., Bx to obtain a per-user bias relative to the facility-wide bias Bc.

[0088] According to a further embodiment, the step 108 of comparing the facility-wide bias Bc with one of the measurement biases B1, B2, ..., Bx further comprises: identifying differences dx1, dx2, ..., dxx by subtracting each of the measurement biases B1, B2, ..., Bx by the facility-wide bias Bc to identify differences dx1, dx2, ..., dxx for each user; and identifying one or more users with the largest differences dx1, dx2, ..., dxx among all users.

[0089] In a further embodiment, a global bias represented by the average value of all available measurement biases Bx is identified after step 102. Later, the global bias is compared with one of the measurement biases Bx.

[0090] If an update of the measurement biases B1, B2, ..., Bx of the first user, the second user and the further users or of the bias Bc of the entire facility is performed, the above steps are at least partly performed repeatedly.

[0091] FIG. 5 shows a schematic representation of a hypothetical system, in this case part of an ultrasound imaging machine or ultrasound scanner. The hypothetical system includes an automatic measurement tool 1. Furthermore, the hypothetical system includes a computer screen 20 including a touch-sensitive display 50. An ultrasound image 40 is currently displayed on the display 50. The hypothetical system can control any image or element displayed on the display 50. The automatic measurement tool 1 includes a processor 60, for example a CPU, configured to execute an embodiment of the method of the present invention. The hypothetical system further includes a data storage medium 80, for example a hard disk, connected to the automatic measurement tool 1. Furthermore, the automatic measurement tool 1 includes a storage in which the necessary computer programs for executing the present invention in the automatic measurement tool 1 or its processor 60 are stored. Furthermore, there is an insertable computer readable medium 65, for example a USB stick, used to load the necessary computer programs into the automatic measurement tool 1. Finally, the hypothetical system includes an ultrasound control unit 200 that controls the acquisition of ultrasound images by the ultrasound probe 120. A patient bed 14 is provided for the subject, for example a patient, to recline while the ultrasound images 40 are acquired by the user 17. The user 17 is anyone wishing to perform an accurate image evaluation, who in the case of medical images is often a radiologist or radiographer, but may also be an expert in any other field, such as, for example, a gynecologist, cardiologist, etc.

[0092] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered as illustrative or exemplary and not restrictive, and the invention is not limited to the disclosed embodiments.

Claims

1. 1. A computer-implemented method for automated analysis of bias in measurements performed on medical images of anatomical structures, the method comprising: receiving a plurality of medical images, the plurality of medical images including at least a first plurality of medical images and a second plurality of medical images, each of the plurality of medical images including the same anatomical structure of the same patient or different patients; receiving, for the first plurality of medical images, first measurement data of a first feature of the anatomical structure, the first measurement data being acquired by a first user, the first measurement data representing a physical parameter; identifying, for the first plurality of medical images, first baseline measurement data of the first feature of the anatomical structure using an automated measurement tool; determining a first measurement bias for the first user based on the first measurement data and the first baseline measurement data; receiving, for the second plurality of medical images, second measurement data of a second feature of the anatomical structure, the second measurement data being acquired by a second user, the second measurement data representing a physical parameter, the second user being different from the first user; identifying, for the second plurality of medical images, second baseline measurement data of the second feature of the anatomical structure using an automated measurement tool; determining a second measurement bias for the second user based on the second measurement data and the second baseline measurement data; comparing the first measured bias to the second measured bias; The method comprising:

2. determining a global bias represented by the average of all available measurement biases; comparing the global bias to one of the measured biases; The method of claim 1 further comprising:

3. the first measurement data is determined based on a particular medical image from the first plurality of medical images, and the first baseline measurement data is determined based on the same particular medical image; the second measurement data is determined based on a particular medical image from the second plurality of medical images, and the second baseline measurement data is determined based on the same particular medical image. The method according to claim 1 or claim 2.

4. the measurement bias is continuously updated as new measurement data becomes available; all available measurement data is used to update the measurement bias, or all available measurement data acquired during a predefined historical period is used to update the measurement bias; The method according to any one of claims 1 to 3.

5. determining a facility-wide bias represented by an average value of all measurement biases of the first user, the second user and further users belonging to one given group or facility, 5. The method according to any one of claims 1 to 4.

6. comparing the facility-wide bias to one of the measurement biases to obtain a per-user bias for the facility-wide bias. The method according to claim 5.

7. comparing the facility-wide bias to one of the measurement biases; determining the difference by subtracting each of the measurement biases by the facility-wide bias to determine the difference for each user belonging to the same group or facility; identifying one or more users with the largest difference among all users; having The method according to claim 6.

8. determining at least the first baseline measurement data and the second baseline measurement data at different locations; 8. The method according to any one of claims 5 to 7.

9. the automated measurement tool uses artificial intelligence to identify the baseline measurement data, the artificial intelligence being trained with a reference database including reference medical images of the anatomical structures and measurement data obtained by at least one expert user on the reference medical images.

9. The method according to any one of claims 1 to 8.

10. the automated measurement tool uses machine learning to identify the baseline measurement data, the automated measurement tool being trained with a reference database including reference medical images of the anatomical structure and measurement data obtained by at least one expert user on the reference medical images.

10. The method according to any one of claims 1 to 9.

11. the automated measurement tool uses an artificial neural network to identify the baseline measurement data, the artificial neural network being trained with a reference database including reference medical images of the anatomical structures and measurement data obtained by at least one expert user on the reference medical images.

11. The method according to any one of claims 1 to 10.

12. the measurement data includes at least one of the following dimensions: length, volume, area, volume flow, blood velocity, and tissue motion velocity; 12. The method according to any one of claims 1 to 11.

13. A new measurement bias and an old measurement bias are identified for the measurement bias; the new measurement bias is identified when a predetermined period of time has elapsed since the old measurement bias was identified, or the new measurement bias is identified when a predetermined amount of new measurement data is available after the old measurement bias is identified; the new measurement bias is compared to the old measurement bias to obtain a drift in the measurement bias; 13. The method according to any one of claims 1 to 12.

14. When executed by a processor, it enables the processor to carry out the method of any one of claims 1 to 13. A computer program comprising program code instructions.

15. 14. An automated metrology tool for carrying out the method of any one of claims 1 to 13, said automated metrology tool comprising an interface for receiving medical images produced by an imaging system and outputting results determined by carrying out said method. Automated measurement tool.

Images