Quality control method for echocardiograms processed by software

Abstract

To provide a quality control method for software processed echocardiograms.SOLUTION: The present invention relates to a method for evaluating a video image analyzed by software. The method comprises: receiving an input image and an image analyzed by corresponding software; generating a difference parameter that is predicted by at least one difference model; generating a shape parameter; and generating an evaluation result that is predicted based on the predicted difference parameter and the shape parameter by an evaluation model. The present invention also relates to a method for training a model implementing the above-mentioned difference parameter generation, and to a method for training a model implementing the evaluation result generation.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a method for quality control of images processed by software, particularly echocardiogram images.

Background Art

[0002] Medical image analysis is an important part of the medical field. In recent years, artificial intelligence (AI) has been applied to a number of medical devices and systems to analyze acquired medical images for further uses such as disease diagnosis. AI can perform various tasks including image analysis to assist physicians in image interpretation, and thus can be trained to reduce the workload of physicians.

[0003] Echocardiogram examination is an ultrasonic diagnosis of the heart, a kind of medical imaging of heart images, and is routinely used in the diagnosis, management, and continuous management of patients suspected of having heart disease or known to have heart disease. In recent years, some AI products have been commercially available for echocardiogram image analysis. Typically, when a user uploads an ultrasonic video of the heart (e.g., a video of the apical four-chamber view), the AI software automatically fits the edges of the inner wall in the video, uses the changes in the edges in the video, and calculates several parameters. However, the edges fitted by the software using statistical data models often suffer from large errors and cannot be directly used. Therefore, many products also provide a function for the user to adjust the edges on a specific frame after automatically fitting the edges. In this way, experts can instruct the software to calculate more accurate parameters by adjusting the edges.

[0004] The above workflow is extremely cumbersome because, for better accuracy, experts need to check each input image to confirm whether the automatically generated results require further adjustment. The manual inspection required for quality control conflicts with the purpose of automatic fitting by AI software.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Given the above background, there is a need to develop an automated method for evaluating the quality of results generated by software. [Means for solving the problem]

[0006] To address the problem, the present invention provides an automated method for evaluating the accuracy of software-generated results for medical images. If the evaluation determines that the generated results are of good quality, the user can use the results directly without further examination. Alternatively, if the evaluation indicates poor quality, the user can decide to make manual adjustments or re-acquire the medical images based on the evaluation summary. Thus, the method facilitates the physician's diagnostic workflow.

[0007] One aspect of the present invention provides a method for training a difference model to generate difference parameters related to the difference between a tracked contour and an adjusted contour by software, the method comprising training a first machine learning model with a plurality of first training datasets, each of the plurality of first training datasets comprising a first training image set as a training input and difference parameters as a training target. The first training image set and difference parameter set are generated by (a) obtaining a first training image set by selecting at least one image, (b) generating a tracked contour by software based on the first training video or the first training image set by analysis software, (c) obtaining an adjusted contour, and (d) obtaining a difference parameter set by software based on the tracked contour and the adjusted contour. In one embodiment, the first training image is an echocardiogram image.

[0008] In one embodiment, each image in the first training image set is processed by the software according to a traced contour before being used as training input.

[0009] In one embodiment, the first machine learning model is a regression model based on a convolutional neural network. Specifically, the first machine learning model may be a residual neural network (ResNet) model.

[0010] Another aspect of the present invention provides a method for training an evaluation model to generate a predictive evaluation error related to the difference between software-generated analysis results and adjusted analysis results, the method comprising training a second machine learning model with a plurality of second training datasets, each of which comprises at least one difference parameter as a training input, at least one shape parameter as a training input, and an evaluation result as a training target. Each of the at least one difference parameter represents the difference between a software-tracked contour and an adjusted contour, each of the at least one shape parameter is calculated based on the software-tracked contour generated by the analysis software, and the evaluation result is determined based on the difference between the software-generated analysis results and the adjusted analysis results. In one embodiment, the second training image is an echocardiogram image.

[0011] In one embodiment, the second machine learning model is a tree-based model. More specifically, the second machine learning model may be a regression model, and the evaluation result may be an error value indicating the difference between the software-generated analysis result and the adjusted analysis result. Alternatively, the second machine learning model may be a classification model, and the evaluation result may be a class indicating good or bad quality of the software-generated analysis result.

[0012] In one embodiment, each of at least one shape parameter set is generated by (a) analysis software generating software-tracked contours from at least one image, and (b) software calculating one of the at least one shape parameter set based on the software-tracked contours.

[0013] Each of at least one difference parameter can be generated by software by direct calculation from the tracked and adjusted contours, or by model prediction using a difference model. In one embodiment, each of at least one set of difference parameters is generated by (a) analysis software generating tracked contours from at least one image, (b) obtaining adjusted contours, and (c) software calculating one of at least one set of difference parameters based on the tracked and adjusted contours. In another embodiment, each of at least one set of difference parameters is generated by (a) obtaining a second set of training images by selecting at least one image, and (b) a difference model generating one of at least one set of difference parameters based on the second set of training images.

[0014] In one embodiment, at least one shape parameter set includes an ED (end-extension) shape parameter set and an ES (end-contraction) shape parameter set. Also in one embodiment, at least one difference parameter set includes an ED (end-extension) difference parameter set and an ES (end-contraction) difference parameter set.

[0015] In one embodiment, the evaluation result is generated by (a) calculating analysis results generated by the software based on the tracked ED contour and the tracked ES contour; (b) obtaining the adjusted ED contour and the adjusted ES contour; (c) calculating the adjusted analysis results based on the adjusted ED contour and the adjusted ES contour; and (d) determining the evaluation result based on the analysis results generated by the software and the adjusted analysis result.

[0016] In another aspect, the present invention provides a software-analyzed image quality control method, the method comprising: (a) receiving at least one input image and at least one image analyzed by corresponding software, wherein the image analyzed by at least one corresponding software is generated by the analysis of at least one input image by the analysis software; (b) generating at least one set of difference parameters predicted based on at least one input image by at least one difference model; (c) generating at least one set of shape parameters from the image analyzed by at least one corresponding software; and (d) generating an evaluation result predicted based on at least one set of predicted difference parameters and at least one set of shape parameters by an evaluation model.

[0017] In one embodiment of the quality control method, the evaluation result is an error value indicating the difference between the analysis result generated by the software and the adjusted analysis result. In another embodiment, the evaluation result is a class indicating whether the analysis result generated by the software is of good or poor quality.

[0018] The present invention also provides a non-temporary computer-readable medium in which an instruction set is stored, and the instruction set is executable by a processor of a computer system to perform a method comprising: (a) receiving at least one input image and an image analyzed by at least one corresponding software, the image analyzed by at least one corresponding software being generated by the analysis of at least one input image by the analysis software; (b) generating at least one set of difference parameters predicted based on at least one input image by at least one difference model; (c) generating at least one set of shape parameters from the image analyzed by at least one corresponding software; and (d) generating an evaluation result predicted based on at least one set of predicted difference parameters and at least one set of shape parameters by an evaluation model.

[0019] Other objects, advantages, and novel features of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0020] [Figure 1] This is the workflow for training the first model (the difference model). [Figure 2] This is the workflow for training the second model (evaluation model). [Figure 3] This is a workflow for quality control of images processed by software. [Figure 4A] This is an echocardiogram image of an end-diastolic (ED) frame, and the endocardial contour is tracked by analysis software. [Figure 4B] This is an echocardiogram image of the end-systolic (ES) frame, and the endocardial contour is tracked by analysis software. [Figure 4C] This is an echocardiographic image of an end-diastolic (ED) frame, and the endocardial contour is adjusted by a specialist. [Figure 4D] An echocardiogram video for the end-systolic (ES) frame, where the endocardial myocardial contour is adjusted by an expert. [Figure 5A] A diagram of the contour extracted from the end-diastolic (ED) frame of Figure 4A. [Figure 5B] A diagram of the contour area. [Figure 6] A diagram of some points assigned to the contour and / or area. Points A, M, L, R, and C are defined in this diagram. [Figure 7A] A diagram of the frame of the original echocardiogram video to which the mask should be applied. [Figure 7B] A diagram of the mask obtained by linear distance transformation and attenuation from the edge of the extracted contour (Figure 5A) or area (Figure 5B). [Figure 7C] A diagram of the mask area of Figure 7A. [Figure 7D] An image cropped to a smaller size. [Figure 8A] A graph of the test results predicted by the ES frame difference model. Figure 8A shows the Dice score of the endocardial myocardium of the left ventricle. [Figure 8B] A graph of the test results predicted by the ES frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 8B shows the area difference between the automatic result and the predicted manual result. [Figure 8C] A graph of the test results predicted by the ES frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 8C shows the difference in the line LRy intercept. [Figure 8D] A graph of the test results predicted by the ES frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 8D shows the dynamic time warping (DTW) distance between the automatic endocardial myocardial contour and the manual endocardial myocardial contour. [Figure 9A]This graph shows the test results predicted by the ED frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 9A shows the Dice score for the endocardial myocardium of the left ventricle. [Figure 9B] This graph shows the test results predicted by the ED frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 9B shows the area difference between the automated result and the predicted manual result. [Figure 9C] This graph shows the test results predicted by the ED frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 9C shows the difference in line L-ry intercepts. [Figure 9D] This graph shows the test results predicted by the ED frame difference model. The horizontal axis represents the ground truth value, and the vertical axis represents the predicted value. Figure 9D shows the dynamic time stretching (DTW) distance between the automated and manual endocardial contours. [Figure 10A] This graph shows the test results predicted by the evaluation model. Figure 10A shows the ground truth value and predicted value of the GLS difference. The R-squared value in Figure 10A is 0.4499, and the mean absolute error is 1.5586. [Figure 10B] This graph shows the test results predicted by the evaluation model. Figure 10B shows the ROC curve for QC pass / fail for the model. The QC pass / fail accuracy is 74.18% (with a cutoff at GLS error value = 2). [Figure 11A] This image is an example of a poor tracking result caused by poor image quality. Figure 11A shows an ED frame automatically tracked from an echocardiogram image. [Figure 11B] This is an example image of a failure tracking result caused by poor image quality. Figure 11B is an automatically tracked ES frame. [Figure 11C] This is an example image of a failure tracking result caused by poor image quality. Figure 11C is a manually adjusted ED frame. [Figure 11D]This is an example image of a failure tracking result caused by poor image quality. Figure 11D is a manually adjusted ES frame. [Figure 12A] This is another example image of poor tracking results caused by poor image quality. Figure 12A shows an ED frame automatically tracked from an echocardiogram image. [Figure 12B] This is another example image of a poor tracking result caused by poor image quality. Figure 12B is an automatically tracked ES frame. [Figure 12C] This is another example image of a bad tracking result caused by poor image quality. Figure 12C is a manually adjusted ED frame. [Figure 12D] This is another example image of a failure tracking result caused by poor image quality. Figure 12D is a manually adjusted ES frame. [Figure 13A] This is an image of an example of a fault tracking result obtained through faulty software analysis. Figure 13A shows an ED frame automatically tracked from an echocardiogram image. [Figure 13B] This is an image of an example of a defect tracking result obtained through defective software analysis. Figure 13B shows an automatically tracked ES frame. [Figure 13C] This is an image of an example of a failure tracking result obtained through faulty software analysis. Figure 13C is a manually adjusted ED frame. [Figure 13D] This is an image of an example of a failure tracking result obtained through faulty software analysis. Figure 13D is a manually adjusted ES frame. [Figure 14A] This graph shows the test results predicted by a linear regression model. Figure 14A shows the ground truth value and predicted value of the GLS difference. The R-squared value in Figure 14A is 0.03, and the mean absolute error is 1.7731. [Figure 14B] This graph shows the test results predicted by the linear regression model. Figure 14B shows the ROC curve for QC pass / fail for the model. The QC pass / fail accuracy is 58.38% (with a cutoff at GLS error value = 2). [Figure 15]This graph shows the test results predicted by a model trained directly on the entire image (without shape parameters). The R-squared value is 0.1438, and the mean absolute error is 2.1109. [Modes for carrying out the invention]

[0021] The terms used in the following description, although used in conjunction with detailed descriptions of several specific embodiments of the Art, are intended to be interpreted in their broadest and most appropriate form. Some terms may even be emphasized below. However, any term for which a restrictive interpretation is intended will be specifically defined as such in this "Modes for Carrying Out the Invention" section.

[0022] The embodiments described below can be implemented by programmable circuits. Programmable circuits are programmed or configured by software and / or firmware, or entirely by dedicated circuits, or in a combination thereof. Such dedicated circuits (if any) may take the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), etc.

[0023] The method for establishing a quality evaluation system includes four steps, including data labeling, image preprocessing, model training, and an inference pipeline. In this invention, two types of machine learning models are constructed to perform quality control (QC) work on images processed / analyzed by the software. After training, the first model can predict several difference parameters using input (unprocessed) images, and the second model can evaluate the quality of processing and / or analysis by the software using difference parameters and shape parameters obtained from images processed / analyzed by the software.

[0024] Figure 1 shows the training steps of the first model, which is a difference model. Step S11 is data acquisition, where multiple images (e.g., a series of echocardiogram images) are collected as a training dataset. In method S121, the training data is input into automated analysis software, and the analysis calculation results are recorded along with the image tracking results. An example of automated analysis is for the software to discover specific image frames from the input images, for example, to discover end-diastolic and end-systolic frames from echocardiogram images. Next, this tracking can be applied to these selected frames(s), and the tracking results can be made into automatically tracked contours by the analysis software. Next, in step S122, an expert adjusts the automated image tracking results to more accurate results, for example, redrawing the corrected contours. In step S13, the automated tracked results and adjusted results are recorded, and a difference parameter representing the difference between the automated tracked results and the adjusted results is calculated.

[0025] In step S14, before model training, the input image, which is a selected video frame, may be preprocessed to create a model that focuses on important image regions and incorporate the software's automatic tracking results. For example, the automatic tracking results may be used to determine masking and cropping regions. Then, the input image may be masked and cropped, leaving the important tracking regions within the image.

[0026] As described above, the model training of the present invention is divided into two parts. The first part is a difference model that uses a neural network to predict difference parameters, which represent the difference between software-generated results and expert-adjusted results (e.g., software-tracked contours and adjusted contours). The first step is to calculate parameters that represent the difference between the image tracking results before and after expert adjustment, with several significant parameters specifically selected by the expert. As shown in step S15 of Figure 1, the regression neural network model is trained using the difference parameters obtained in step S13 and used as training targets, and the pre-processed images obtained in step S14 and used as inputs, and the values ​​of each parameter are output.

[0027] The second part of model training involves training a tree-based evaluation model to predict the difference between the final analysis values ​​of the automated analysis software (e.g., global strain in the longitudinal direction) and the expert-adjusted values. The inputs are (1) the difference parameters predicted in the first part by the difference model, and (2) the shape parameters directly measured and quantified within the software's automated tracking results. The output of the second model predicts the error between the results initially generated by the analysis software and the expert-adjusted results. A threshold can be set to evaluate the quality of the automated tracking analysis results. If the predicted difference is below the threshold, the user (e.g., a physician) can trust the automated analysis results. Figure 2 shows the training steps of the second model.

[0028] In step S21, multiple images (e.g., a series of echocardiogram images) are collected as a training dataset. The dataset may be the same as or different from the dataset used for training the first model (described in step S11). In steps S221 and S222, two image sets, namely the ED set and the ES set, are individually tracked by automated analysis software. The procedure is the same as in S121. In steps S231 and S232, the tracked ED and ES contours are individually adjusted. This is the same as in step 122. In step S241, the automated tracked ED and ES contours obtained in steps S221 and S222 are integrated, and the automated analysis results are evaluated, for example, calculating the longitudinal global strain (GLS) value based on the ED and ES contours tracked by the software. The analysis results are the same as those output by the automated analysis software, except for the contours adjusted by an expert. In step S242, the manually adjusted ED and ES contours obtained in steps S231 and S232 are integrated, and the expert-adjusted results are evaluated, for example, the longitudinal global strain (GLS) value is calculated based on the adjusted ED and ES contours. The analysis results are the same as those output by the automated analysis software, along with the expert-adjusted contours. In step S25, the analysis results generated in steps S241 and S242 are combined, and the analysis error is calculated. The analysis error represents the error produced by the automated analysis software (compared to the expert-adjusted results).

[0029] In steps S261 and S262, the two image sets obtained in steps S221 and S222 can be preprocessed independently. This preprocessing is the same as in step S14. Next, as shown in steps S271 and S272, the independently preprocessed images are used to generate the difference parameters for ED and ES. The difference parameters for ED and ES are parameters that represent the difference between the software-generated contours and the adjusted contours for the ED and ES frames. The difference parameters can be generated (predicted) by an established difference model using the preprocessed images as input. Alternatively, the difference parameters can also be generated (calculated) by obtaining the software-tracked and adjusted ED (and, in the case of ES frames, obtaining the software-tracked and adjusted ES) and calculating these parameters. In steps S281 and S282, the shape parameters for ED and ES are calculated based on the automatically tracked ED and ES contours obtained in steps S221 and S222, respectively. The shape parameters represent the shape characteristics of the software-tracked contours.

[0030] In step S29, the data obtained in steps S25, S271, S272, S281, and S282 is used to train a second model, which is an evaluation model. The difference parameters obtained in steps S271 and S272, and the shape parameters obtained in S281 and S282 are used as training inputs. The evaluation error value obtained in step S25 is used as the learning target for training.

[0031] As shown in Figure 3, in the inference pipeline step, the training model, preprocessed images, and analysis software are placed in the pipeline. In step S31, the input video (e.g., echocardiogram video) is processed and analyzed by the analysis software to perform automated analysis. The output, which may include important frames in the video, tracked contours, and analysis results, is used in QC evaluation. In step S32, the extracted frames (e.g., frames near the ED and ES frames) are optionally preprocessed by the software based on tracked contours to minimize background interference. One example of image preprocessing is cropping the background from the image and retaining only the contour region. Next, in step S33, the processed image frames are analyzed by a difference model to predict the difference parameters between the automated tracked contours and the expert-adjusted contours. In a preferred embodiment, two difference models are used to independently generate predicted difference parameters for the ED and ES contours. In step S34, the software calculates several shape parameters based on the tracked contour(s). The parameters are several measurements of the contour. In step S35, the difference parameters (predicted by the difference model in step S33) and the shape parameters (calculated in step S34) are used as inputs to the evaluation model, i.e., the second training model. The evaluation model then generates an evaluation indicating the quality of the analysis results performed by the automated software. Therefore, the quality of the results generated by the software can be predicted from input images such as echocardiograms.

[0032] The following provides more details for each step.

[0033] Data display Echocardiographic images in the database are collected for data display. For data display, each input echocardiographic image is sent to software (e.g., Tomtec AutoStrain) for analysis, collecting automated longitudinal global strain (GLS) analysis results and left ventricular myocardial contour (e.g., tracked endocardial contour) tracking results. As shown in Figures 4A and 4B, the GLS analysis results are numerical, and the myocardial contour tracking results can be captured as screenshot images or directly exported as key point coordinates for end-diastolic (ED) and end-systolic (ES) frames. Next, experts can adjust the myocardial tracking results in the software, as shown in Figures 4C and 4D, to obtain more accurate GLS analysis results. The adjusted GLS analysis results and corresponding adjusted myocardial contours are also collected for future use. The data obtained through the above echocardiographic display steps are: (1) Automated myocardial contour tracking results for ED frames, (2) Automated myocardial contour tracking results for ES frames, (3) Manually adjusted myocardial contour relative to the ED frame (4) Manually adjusted myocardial contour relative to the ES frame, (5) Automated longitudinal global strain, and (6) Manually adjusted longitudinal global strain It may include.

[0034] Calculation of shape parameters and difference parameters In calculating shape parameters, the tracked endocardial contour and left ventricular region are extracted from the tracked images. Figure 5A is an example of a contour extracted from the end-diastolic (ED) frame of Figure 4A, and Figure 5B is the contour region. As shown in Figure 6, several points can be assigned to the contour and / or region. Based on the contour, surrounding region, and assigned points, several shape metrics of the automated tracked results are measured. The shape metrics include: (1) Contour region, (2) The gradient and y-intercept of line LR, (3) The gradient and y-intercept of line AM, (4) Width of the base (length of the LR section), (5) Height (length of AM section), and (6) Height-to-width ratio It may include.

[0035] Another contour and left ventricular region can be extracted from the expertly adjusted endocardial myocardial contour. Then, the difference in contour and region between the auto-adjusted contour and the manually adjusted contour can be calculated as further parameters. These further parameters are: (1) Difference in LR gradient, (2) Difference in AM gradient, (3) Difference between the Amy intercepts, (4) The distance between point A of the automatic contour and point A of the manual contour, (5) The distance between the L point of the automatic contour and the L point of the manual contour, (6) The distance between the R point of the automatic contour and the R point of the manual contour, (7) The distance between the M point of the automatic contour and the M point of the manual contour, (8) The distance between the C point of the automatic contour and the C point of the manual contour, (9) Distance between automated endocardial right-side contour segment and manual endocardial right-side contour segment, (10) Distance between automated endocardial apex contour division and manual endocardial apex contour division, (11) Distance between automated left endocardial contour segment and manual left endocardial contour segment, (12) The 2D distance between the automated right-sided endocardial contour segment and the manual right-sided endocardial contour segment, (13) The 2D distance between the automated endocardial apex contour division and the manual endocardial apex contour division, (14) The 2D distance between the automated left endocardial contour segment and the manual left endocardial contour segment, (15) Dice score between the automatic and manual areas (16) Area difference, (17) Dynamic time stretch distance between automatic contour and manual contour, (18) The difference in gradient and y-intercept between the automatic line LR and the manual line LR, (19) The difference in gradient and y-intercept between the automatic line AM and the manual line AM, and (20) Point distance between automatic L, R, C, A points and manual L, R, C, A points It may include.

[0036] To express the quality of the endocardial contour tracking results, some or all of the parameters can be selected. The selected parameters are calculated for the ED frame and ES frame tracking results and trained individually in a later step.

[0037] Image preprocessing This step is optional, but it can accelerate the AI ​​training speed and the execution speed of the trained model. Although the input echocardiogram is an image, model training usually does not require the entire image as input. As a result, video frames near the end-diastolic (ED) and end-systolic (ES) frames can be selected as input images, because the parameters are measured and calculated from the ED and ES frames.

[0038] Based on the left ventricular region extracted from screenshots of automatically tracking contours, a mask can be generated, and a deep neural network model can be facilitated to focus on the relevant image region. Mask generation, application, and image cropping are shown in Figures 7B to 7D.

[0039] Model training This step involves training the model and evaluating the quality of the automatically generated analysis results. This model can be divided into two parts: the first part is the difference model, and the second part is the evaluation model.

[0040] (1) Training of the difference model In the first part, two regression neural networks are constructed using preprocessed image frames and difference parameters as training data, enabling the prediction of difference parameters that represent the difference between automated tracking results and manual adjustment results (e.g., contours tracked and adjusted by software). One neural network takes one or more preprocessed image frames around the ED as input and outputs difference parameters calculated from the ED contour tracking results. The other neural network is similar, but uses images and parameters of the ES instead of the ED. The model can use one preprocessed image frame to produce excellent contour tracking results, or the model can use multiple preprocessed image frames as input to reduce noise within a single frame.

[0041] (2) Evaluation Model Training By combining the predicted difference parameters (predicted by the difference model) and measured shape parameters (calculated from the automatically traced contour), an evaluation model can be trained to predict the final target: the difference between the automated GLS and the expert-adjusted GLS (and therefore the quality of the automated trace analysis results). Shape parameters can be automatically measured from the automated traced contour when new data is inferred.

[0042] The evaluation model may be a classifier (classification model) that determines whether the difference is large (implying poor tracking and analysis results from the automated software) or small (implying good tracking and analysis results from the automated software), or it may be a regression model that accurately tells us how much of a GLS difference there is between the analysis results generated by the software and the analysis results adjusted by an expert.

[0043] example The following example is provided to further illustrate the details of a training model for quality control of echocardiogram images processed / analyzed by software.

[0044] 1. Data Collection and Labeling Approximately 1000 echocardiographic images of the apical four-chamber view are used as raw data. The raw data is divided into training / validation / examination sets. Data with good and poor follow-up results, secured by the division strategy, are evenly distributed across the datasets. Two sets of measurement / marking, namely automated GLS and manual GLS, are generated using these datasets. Automated GLS is generated using Tomtec (TOMTEC Imaging Systems GmbH) AutoStrain software, which includes automated endocardial contour tracking and GLS calculation. Manual GLS marking is generated from contours drawn by a physician.

[0045] 2. Calculation of shape parameters After data display, shape parameters measured from the automatically tracked endocardial contour are generated for each echocardiogram image. Referring to the points defined in Figure 6, the following parameters are used: (1) Area of ​​the endocardial myocardium of the left ventricle (A) (2) Gradient of line LR (3) Y-intercept of line LR (4) Gradient of line AM (5) Y-intercept of line AM (6) Height of the left ventricular endocardial myocardium (H) (7) Width of the base of the left ventricular endocardial myocardium (B) (8)H / B ratio (9)A(H×B) ratio Calculate.

[0046] Based on the above parameters, the difference parameters can be calculated by the software for the traced contour and the adjusted contour, and the difference parameters are: (1) Difference in LR gradient, (2) Difference in AM gradient, (3) Difference between the Amy intercepts, (4) The distance between point A of the automatic contour and point A of the manual contour, (5) The distance between the L point of the automatic contour and the L point of the manual contour, (6) The distance between the R point of the automatic contour and the R point of the manual contour, (7) The distance between the M point of the automatic contour and the M point of the manual contour, (8) The distance between the C point of the automatic contour and the C point of the manual contour, (9) Distance between automated endocardial right-side contour segment and manual endocardial right-side contour segment, (10) Distance between automated endocardial apex contour division and manual endocardial apex contour division, (11) Distance between automated left endocardial contour segment and manual left endocardial contour segment, (12) The 2D distance between the automated right-sided endocardial contour segment and the manual right-sided endocardial contour segment, (13) The 2D distance between the automated endocardial apex contour division and the manual endocardial apex contour division, (14) The 2D distance between the automated left endocardial contour segment and the manual left endocardial contour segment, (15) Dice score between the automatic and manual areas (16) Area difference, (17) Dynamic time stretch distance between automatic contour and manual contour, (18) The difference in gradient and y-intercept between the automatic line LR and the manual line LR, (19) The difference in gradient and y-intercept between the automatic line AM and the manual line AM, (20) Difference between points L, R, C, A automatically and points L, R, C, A manually Includes.

[0047] 3. Image preprocessing With respect to the input echocardiogram image (Figure 7A), the software tracks the left ventricular endocardial contour and corresponding region, as shown in Figures 5A and 5B. Next, a distance transformation is performed to obtain a mask, as shown in Figure 7B. Then, the mask is applied to the frame of the input echocardiogram image (Figure 7A), retaining only the region surrounded by the left ventricular endocardial myocardium, as shown in Figure 7C. Next, the masked image is cropped to remove the extraneous background region, as shown in Figure 7D.

[0048] 4. Difference Model Training Two regression neural networks are trained to predict parameters representing the difference between automatically tracked results and manually adjusted results. One neural network takes preprocessed image frames near the ED frame as input and outputs the difference parameter for the ED frame. The other neural network does the same, but uses ES images and parameters instead of ED. Frames near the ED and ES are extracted as training images. For ED model training, four echocardiogram images near the ED are extracted. For ED model training, eight echocardiogram images near the ES are extracted.

[0049] The next step after image extraction is data enhancement. Each image is displaced, scaled, and random brightness / contrast adjustments are applied.

[0050] We train a difference model using the deep residual learning model, ResnetRS3D-50 (arXiv:2103.07579, model code:https: / / github.com / tensorflow / models).

[0051] The inputs to the ED and ES difference models are 4 and 8 frames, respectively. Difference parameters calculated from the training set data are used as the learning target (ground truth). Before training the model, the parameter values ​​are standardized. The output layer is a high-density layer that outputs continuous values ​​of the difference parameters listed above in the shape parameter calculation paragraph. The model is trained to predict the difference parameters of echocardiographic images associated with the endocardial myocardial contour tracked by the software.

[0052] The model is trained over 100 epochs using an Nvidia RTX A6000 GPU in a Tensorflow 2.9.1 environment.

[0053] 5. Training results of the difference model The training results of the difference model are tested using the test dataset. The test results for the ES frame difference model are shown in Figures 8A to 8E. The total mean error is 0.5931 (variance). The test results for the ED frame difference model are shown in Figures 9A to 9E. The total mean error is 0.7253 (variance).

[0054] 6. Evaluation Model Training The evaluation model is trained using the results predicted by the ES frame difference model and the ED frame difference model. A tree-based model, the XGBoost (https: / / github.com / dmlc / xgboost) algorithm, is used for training. The GLS difference calculated from the marked data (including software-tracked and adjusted contours) is used as the training ground truth. The input includes the shape parameters and difference parameters described above in the shape parameter calculation paragraph, and the output is the error value, which is the difference between manual GLS and automated GLS.

[0055] 7. Training results of the evaluation model As shown in Figures 10A and 10B, the trained evaluation model is tested with the test dataset. The mean absolute error is 1.5586, and the QC pass / fail accuracy is 74.18% (with a cutoff at GLS error value = 2, as experts suggest considering a GLS error greater than 2 as a poor auto-tracking result).

[0056] 8. Poor Image Quality vs. Poor Image Analysis The failure prediction results generated by the software may be (1) poor image quality (e.g., low resolution or incorrect shooting angle) or (2) good image quality, but may result from failure analysis results predicted by automated analysis software. The present invention can address both cases, as shown in Figures 11 to 13.

[0057] Figures 11A to 11D show that, for a portion of the left ventricular region, input with poor image quality is not adequately obtained within the input image. This causes an error in the automatically tracked GLS value. The automatically tracked GLS value is -11.2%, and the manually adjusted GLS is -18.7%. The error value (manual minus automatic) is -7.5, which is considerably smaller than -2 and is classified as a poor prediction result. The model of the present invention similarly predicts an error value of -10.38, which is also considerably smaller than -2, and is therefore classified as a poor prediction result.

[0058] Figures 12A to 12D show another example of input with poor image quality. The image brightness is too high, causing an error in the automatically tracked GLS value. The automatically tracked GLS value is -12.3%, and the manually adjusted GLS is -20.9%. The error value (manual minus automatic) is -8.6, which is considerably smaller than -2, and is classified as a poor prediction result. The model of the present invention similarly predicts an error value of -7.42, which is also considerably smaller than -2, and is therefore classified as a poor prediction result.

[0059] Figures 13A to 13D show inputs with excellent image quality but with automated failure analysis. The automated tracking GLS value is -12.3%, and the manually adjusted GLS is -22.2%. The error value (manual minus automated) is -9.9, which is considerably smaller than -2, and is classified as a failure prediction result. The model of the present invention similarly predicts an error value of -12.21, which is also considerably smaller than -2, and is therefore classified as a failure prediction result.

[0060] 9. Comparison with conventional methods The model's performance is compared to previously available models. This model is an image classifier with confidence scores, which correlate with the GLS error value between automated and expert-adjusted results. A linear simple regression model is used (since the only input feature is the confidence score of the image classifier), and the model is trained using the same training dataset. The trained model is then tested with the test dataset. The results (Figures 14A and 14B) show a mean absolute error of 1.7731, and the QC pass / fail accuracy is 58.92% (with a cutoff at GLS error value = 2).

[0061] Numerous previous studies have shown that higher confidence scores in image classifiers correspond to better image quality and a closer relationship between automated GLS and manually adjusted GLS. However, the correlation is considerably weaker. Using the same test dataset, the difference between manual GLS and automated GLS predicted by our model has an R-squared value of 0.4499 to the ground truth GLS error value. On the other hand, the confidence score of the image classifier has an R-squared value of only 0.03 to the ground truth GLS error value.

[0062] 10. Comparison of model training using shape parameters versus model training using the entire image. Finally, we compare the performance of a model trained with shape parameters to a model directly trained with the entire image (without introducing shape parameters). For comparison, we use ResnetRS3D-50 to construct a model that directly predicts the error value between manual GLS and automatic GLS (without predicting the initial difference parameter or calculating the shape parameter). We train the model using the same training dataset. The training input consists of 16 frames sampled from one cardiac cycle in a DICOM image. The model is trained for 100 epochs using an Nvidia RTX A6000 GPU in a Tensorflow 2.9.1 environment. The results (Figure 15) show a mean absolute error of 2.1109, with a QC pass / fail accuracy of 58.38% (with a cutoff at GLS error value = 2).

[0063] The results above indicate that directly training a GLS error evaluation model from DICOM images without introducing shape parameters results in higher prediction errors and lower QC pass / fail accuracy. Furthermore, this target training model is more of a black box, as it cannot teach the pass / fail status of the input image for shape features as cues. The present invention's method of predicting quality control via shape parameters is more accurate and more meaningful for cardiac professionals.

[0064] The above description of embodiments is provided for those skilled in the art to perform and make the subject matter usable. Various modifications to these embodiments will be readily apparent to those skilled in the art. Novel principles and subject matter disclosed herein can be applied to other embodiments without the use of innovative skills. The claimed subject matter set forth in the claims is not intended to limit itself to the embodiments shown herein, but rather to provide the broadest scope that corresponds to the principles and novel features disclosed herein. Further embodiments are intended to be within the spirit and true scope of the disclosed subject matter. Accordingly, the present invention is intended to include modifications, variations, and equivalents that fall within the scope of the appended claims.

Claims

1. A method for training a difference model to generate difference parameters relating to the difference between a software-tracked contour and a manually adjusted contour, the method comprising training a first machine learning model with a plurality of first training datasets, each of the plurality of first training datasets comprising a first training image set as a training input and a difference parameter set as a training target, the first training image set and the difference parameter set are (a) A step of obtaining the first training image set by selecting at least one image, (b) The step of generating traced contours by the software based on the first training image set using analysis software, (c) The step of obtaining the manually adjusted contour which shows a more accurate contour than the contour traced by the software, and (d) Steps to obtain the difference parameter set based on the contour traced by the software and the contour adjusted by the manual adjustment. A method characterized by being generated by

2. The method according to claim 1, characterized in that each image in the first training image set is an echocardiogram image.

3. The method according to claim 2, characterized in that each image in the first training image set is preprocessed by masking and cropping based on contours traced by the software before being used as the training input, thereby minimizing background caused by interference between echo signals.

4. The method according to claim 1, characterized in that the first machine learning model is a regression model based on a convolutional neural network.

5. The method according to claim 4, characterized in that the first machine learning model is a residual neural network (ResNet) model.

6. A method for training an evaluation model to generate a predictive evaluation error related to the difference between software-generated analysis results and adjusted analysis results, the method comprising training a second machine learning model with a plurality of second training datasets, each of which comprises at least one difference parameter as a training input, at least one shape parameter as a training input, and an evaluation result as a training target. Each of the at least one set of difference parameters represents the difference between the software-tracked contour and the manually adjusted contour, generated by the difference model trained by the method described in claim 1. Each of the at least one shape parameter is calculated based on the contour traced by the analysis software, which is generated by the software. The method is characterized in that the evaluation result is determined based on the difference between the analysis result generated by the software and the adjusted analysis result.

7. The method according to claim 6, characterized in that the second machine learning model is a tree-based model.

8. The method according to claim 7, characterized in that the second machine learning model is a regression model, and the evaluation result is an error value indicating the difference between the analysis result generated by the software and the adjusted analysis result.

9. The method according to claim 7, characterized in that the second machine learning model is a classification model, and the evaluation result is a class indicating good quality or poor quality of the analysis result generated by the software.

10. Each of the above at least one set of shape parameters is: (a) The analysis software generates software-tracked contours from at least one image, and (b) A step of calculating one of the at least one set of shape parameters based on the contour traced by the software. The method according to claim 6, characterized in that it is generated by

11. The method according to claim 10, characterized in that the at least one image is an echocardiogram image.

12. Each of the above at least one set of difference parameters is: (a) the step of obtaining a second training image set by selecting at least one image, and (b) The step of generating one of the at least one set of difference parameters based on the second training image set using the difference model. The method according to claim 6, characterized in that it is generated by

13. The at least one shape parameter set includes an ED (end-extension) shape parameter set and an ES (end-contraction) shape parameter set, and the ED shape parameter set and the ES shape parameter set are (a) A step of obtaining an ED training image set by selecting at least one ED image, (b) A step of obtaining an ES training image set by selecting at least one ES image, (c) The analysis software generates a tracked ED contour based on the selected ED image, (d) The analysis software generates a tracked ES contour based on the selected at least one ES image. (e) the step of calculating the set of ED shape parameters based on the traced ED contour, and (f) A step of calculating the ES shape parameter set based on the traced ES contour. The method according to claim 6, characterized in that it is generated by

14. The at least one set of difference parameters includes an ED (end of diastolic) difference parameter set and an ES (end of systolic) difference parameter set, and the ED difference parameter set and the ES difference parameter set are (a) The steps of generating the ED difference parameter set based on the ED training image set using the ED difference model, and (b) The step of generating the ES difference parameter set based on the ES training image set using the ES difference model. The method according to claim 13, characterized in that it is produced by

15. The evaluation results mentioned above are: (a) A step of calculating analysis results generated by software based on the tracked ED contour and the tracked ES contour, (b) Steps to obtain the adjusted ED contour and the adjusted ES contour, (c) A step of calculating the adjusted analysis result based on the adjusted ED contour and the adjusted ES contour, (d) A step of determining the evaluation result based on the analysis results generated by the software and the adjusted analysis results. The method according to claim 13, characterized in that it is produced by

16. A method for quality control of images analyzed by software, wherein the method is: (a) Receiving at least one input image and at least one image analyzed by corresponding software, wherein the image analyzed by the at least one corresponding software is generated by the analysis of the at least one input image by the analysis software. (b) generating at least one set of difference parameters predicted based on the at least one input image by at least one difference model trained by the method of claim 1, (c) Generating at least one set of shape parameters from the image analyzed by the at least one corresponding software, (d) The evaluation model generates predicted evaluation results based on at least one set of predicted difference parameters and at least one set of shape parameters. A method characterized by including

17. Before generating at least one set of the first difference parameters, Processing the at least one input image based on the input image analyzed by the at least one software. The method according to claim 16, further comprising:

18. The method according to claim 16, characterized in that the input image is an echocardiogram image.

19. The method according to claim 16, characterized in that the evaluation result is an error value indicating the difference between the analysis result generated by the software and the adjusted analysis result.

20. The method according to claim 16, characterized in that the evaluation result is a class indicating good quality or poor quality of the analysis result generated by the software.

21. The method according to claim 16, characterized in that the image analyzed by the at least one corresponding software is a contour tracked by the software.

22. The aforementioned at least one input image includes at least one ED input image and at least one ES input image. The images analyzed by the aforementioned at least one corresponding software include the corresponding ED image and the corresponding ES image, The at least one difference model includes a difference model of ED and a difference model of ES, The at least one set of predicted difference parameters includes one set of predicted difference parameters for ED and one set of predicted difference parameters for ES. The method according to claim 16, characterized in that at least one set of shape parameters includes one set of shape parameters for ED and one set of shape parameters for ES.

Images