A deep learning-based echocardiographic image recognition system for patent ductus arteriosus in preterm infants

Abstract

The present invention provides a cardiac ultrasound image recognition system based on deep learning model to assist medical personnel in evaluating the hemodynamic status of premature infants. By inputting hemodynamic parameters obtained from cardiac ultrasound images, the system utilizes an object detection model to derive actual cardiac physical quantities, enabling the assessment of the physiological state of the premature infant's heart and assisting medical personnel in evaluating cardiac function. This invention effectively standardizes parameter extraction, allowing for a more precise understanding of the patient's cardiovascular condition. By reducing the impact of human factors, it helps improve diagnostic accuracy, thereby enhancing the quality of medical care for premature infants.

Description

Technical Field

[0001] A cardiac ultrasound image recognition system, particularly a system and its application that utilizes deep learning to achieve cardiac ultrasound image recognition. Prior Technology

[0002] Premature infants, especially those with extremely low birth weight (weighing less than 1000 grams or with a gestational age of less than 28 weeks), often experience heart problems after birth due to incomplete development. The sudden cessation of placental circulation and the rapid transition to pulmonary circulation after birth places enormous stress on the cardiovascular system of premature infants. The abrupt interruption of placental circulation leads to a sudden increase in systemic arterial resistance, thereby exacerbating the workload on the left ventricle of the heart.

[0003] Approximately one-third of preterm infants develop patent ductus arteriosus (PDA), a congenital heart defect, by day 7 after birth. Due to decreased pulmonary vascular resistance in preterm infants, PDA blood flow significantly shifts from the left atrium to the right atrium, leading to increased pulmonary blood flow. This increased pulmonary blood flow can trigger respiratory complications, including pulmonary edema, congestion, and in severe cases, respiratory failure. Therefore, a variety of clinical symptoms may arise from the increased pulmonary blood flow and decreased systemic blood flow. Documented complications from hemodynamically significant PDA include intraventricular hemorrhage (IVH), necrotizing enterocolitis (NEC), bronchopulmonary dysplasia (BPD), and increased mortality.

[0004] Ultrasound is a non-invasive medical technique that acquires images of the heart, allowing for the assessment of cardiac structure and function. Current advancements in deep learning and neural networks mean that most research utilizes specialized neural network models for image segmentation or classification to identify specific features within two-dimensional ultrasound images. In other words, deep learning-based image recognition technology is widely applied in medical imaging, encompassing not only ultrasound but also X-rays, computed tomography (CT), and magnetic resonance imaging (MRI).

[0005] Currently, the use of deep learning and neural network technologies to analyze medical ultrasound images typically focuses on image segmentation. However, applications for numerical measurement of M-mode (Motion Mode) and Pulsed-wave Doppler Mode ultrasound images are still unseen and require further development by researchers. Summary of the Invention

[0006] To provide a deep learning analysis technique specifically for M-mode (Motion Mode) ultrasound images and Pulsed-wave Doppler Mode ultrasound images, this invention provides a deep learning-based cardiac ultrasound image recognition system for preterm infants with open ductus arteriosus, comprising: A cardiovascular ultrasound imaging system is electrically or signalally connected to an ultrasound probe to acquire a cardiac ultrasound image; the cardiac ultrasound image includes an M-mode cardiac ultrasound image and a pulsed-wave Doppler cardiac ultrasound image; A physical or cloud-based host is used to receive the cardiac ultrasound image acquired by the cardiovascular ultrasound imager, which includes an object detection model and performs the following steps: The object detection model detected a rectangular bounding box representing various cardiac parameters in the echocardiogram image; and The pixel information in the cardiac ultrasound image with the rectangular bounding box is converted into a physical quantity.

[0007] The cardiac parameters include left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, ventricular rhythm cycle, or one or more combinations thereof.

[0008] Preferably, the training steps of the object detection model include: Step 1: Data Set Construction: The original dataset, composed of several echocardiogram images, is randomly divided into a training dataset and a validation dataset according to a set ratio. The original dataset then undergoes a preprocessing step and a data amplification step. Step 2 Model Training: Based on the rectangular bounding box of the cardiac ultrasound image in the training dataset, estimate the aspect ratio and area, select an appropriate anchor box, and use the object detection model to perform transfer learning according to the preset hyperparameters. Observe the training results and optimize the hyperparameters to obtain the model with the best accuracy.

[0009] The cardiac ultrasound image contains at least one annotation within the rectangular bounding box.

[0010] The annotation includes left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, ventricular rhythm cycle, or one or more combinations thereof.

[0011] The preprocessing step includes cropping the region of interest from the original cardiac ultrasound image data, converting it into a grayscale image, and standardizing and resizing it.

[0012] Standardization includes unifying the differences in brightness and contrast between different images; size conversion adjusts the cardiac ultrasound image to a fixed resolution to meet the model input requirements.

[0013] The object detection model includes YOLO.

[0014] The training steps for the object detection model, following step 2, further include: Step 3 Model Testing: The object detection model is used to perform performance analysis and verification using one of the test datasets according to Equation (1): …Equation (1); where: Vavg is the actual value of the heart-related parameters, Vavg' is the result of the model detecting the heart-related parameters, and the absolute difference between the two divided by Vavg is the error rate of the model detection. The error rates of p ultrasound images in the test dataset are calculated and their average values ​​are taken. This is the average percentage error of the object detection model.

[0015] Preferably, the physical quantity converted by the deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in preterm infants includes the pressure-velocity relationship between the mean circumferential fiber shortening velocity and the end-systolic wall stress of the left ventricle.

[0016] As can be seen from the above description, the present invention has the following beneficial effects and advantages:

[0017] This invention utilizes an object detection neural network model to provide a highly accurate neural network system specifically for M-mode and pulsed-wave Doppler echocardiogram data. The system provided by this invention is applicable to object detection technology on both M-mode and pulsed-wave Doppler echocardiograms. Combined with hemodynamic assessment, it provides an intelligent ultrasound identification system for effectively predicting and monitoring cardiovascular complications in premature infants. In clinical applications, this invention can effectively save medical staff time spent manually labeling images on premature infant echocardiograms, maximizing the use of limited medical resources.

[0018] Furthermore, the numerical labeling of M-mode and pulsed-wave Doppler echocardiograms exhibits significant operator-dependent characteristics. Because healthcare professionals differ in their training methods and judgment criteria, the manually labeled values ​​for cardiovascular parameters often lack clear standards and consistency. Therefore, this invention can effectively standardize the parameter extraction results, leading to a more accurate understanding of the patient's cardiovascular status. Reducing the influence of human factors will help improve diagnostic accuracy, thereby enhancing the quality of healthcare for premature infants. Simple Explanation of the Diagram

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are merely some examples or embodiments of the present invention and are not intended to absolutely limit the technical scope of the present invention. Unless obvious from the context or otherwise stated, the same reference numerals in the drawings represent the same structures or operations. Wherein: Figures 1 and 2 are schematic diagrams of the system architecture of the deep learning-based cardiac ultrasound image recognition system of the present invention. Figure 3 is a flowchart of the steps to convert the object detection model into physical quantities after training. Figure 4 is a schematic diagram of the data standard of the preferred embodiment of physical quantities. Figure 1 is a schematic diagram of a cardiac ultrasound image. Implementation

[0020] The present invention will be described in detail below with reference to several preferred embodiments. The accompanying drawings are merely some exemplary representations or embodiments of the present invention. For those skilled in the art, the present invention can be applied to other similar situations based on these drawings without any further effort.

[0021] The terms “system,” “apparatus,” “unit,” and / or “module” used in this invention are to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions. As illustrated in this invention, unless the context clearly indicates otherwise, words such as “a,” “an,” “an,” and / or “the” do not specifically refer to the singular and may also include the plural. Generally, the terms “comprising” and “including” only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list; the method or apparatus may also include other steps or elements.

[0022] This invention uses flowcharts to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0023] Please refer to Figures 1 and 2, which are schematic diagrams of the deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in premature infants according to the present invention, which includes:

[0024] A cardiovascular ultrasound imaging device 10, electrically or signalally connected to an ultrasound probe 11, is used to acquire an echocardiographic image I of a patient P; and

[0025] A physical or cloud-based host 20 is used to receive the cardiac ultrasound image I and perform calculations and analysis.

[0026] In detail, the cardiac ultrasound image I in a preferred embodiment of the present invention includes an M-mode (Motion Mode) cardiac ultrasound image and a Pulsed-wave Doppler Mode cardiac ultrasound image. The cardiac ultrasound image I contains a plurality of waves.

[0027] Preferably, the cardiac ultrasound image I includes at least one annotation within a rectangular bounding box, the annotation including left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, ventricular rhythm cycle (RR interval), or one or more combinations thereof.

[0028] Next, as shown in Figure 2, the physical or cloud host 20 includes an object detection model 21, which performs the following steps on the received cardiac ultrasound image I. Preferably, the object detection model 21 is a YOLO (You Only Look Once) object detection model, which can be any of the YOLO first generation (v1) to the seventh generation (v7) object detection models. The following description will use YOLO v4 as an example.

[0029] Step S1-1 Data set construction: The original data set formed by several cardiac ultrasound images I is randomly divided into a training data set, a validation data set, and (optionally) a test data set according to a certain ratio. The original data set is then preprocessed and amplified. In a preferred embodiment, the aforementioned random division ratio is 60% for the training data set, 10% for the validation data set, and 30% for the test data set.

[0030] The preprocessing includes cropping the original cardiac ultrasound image (Image I) into Region of Interest (ROI) blocks to remove irrelevant background information and focus on regions containing important cardiac parameters, thereby improving the efficiency and accuracy of subsequent model learning. Next, the color image of Cardiac Ultrasound Image I is converted to grayscale to simplify data complexity and allow the model to more effectively extract key features. To ensure that all Cardiac Ultrasound Image I data is processed at a uniform scale and within a consistent range, the images are standardized and resized. Standardization aims to unify the differences in brightness and contrast between different images, while resizing adjusts the images to a fixed resolution to meet the model's input requirements.

[0031] Data augmentation utilizes digital image processing technology to adjust the brightness and contrast of the cardiac ultrasound image I, simulating the cardiac ultrasound image I obtained under different signal intensity conditions, thereby enhancing the model's adaptability to various situations. This data augmentation strategy helps prevent model overfitting and improves its generalization ability to unknown data.

[0032] Step S1-2 Model Training: Based on the characteristics of the rectangular bounding box in the training dataset, the aspect ratio and area are estimated. An appropriate anchor box is selected, and the object detection model 21 is used for transfer learning according to preset hyperparameters. The training results are observed, and hyperparameters are optimized to obtain the model with the best accuracy. Preferably, the object detection model 21 used in this step is a model pre-trained on the MS-COCO image dataset (e.g., YOLO v4). Preferably, after training, the object detection model 21 includes one or more YOLO object detection models for left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, and ventricular rhythm cycle (RR interval).

[0033] The object detection model 21 of this invention includes a backbone network that adopts CSPDarknet53, which is a Darknet53 structure with cross-stage partial connections, solving the computational efficiency problem and enhancing the richness of gradient combinations. The neck network integrates the SPP module to fuse feature information at different scales, thereby enhancing the object detection capability. The head network consists of three detection architectures, enabling object detection at three different scales.

[0034] Step S1-3 (Optional) Model Testing: The proposed model is then used to perform performance analysis and verification on the test dataset. The method for evaluating the model accuracy is shown in Equation (1) below:

[0035] …Equation (1); where: Vavg is the actual value of the heart-related parameters, Vavg' is the result of the model detecting the heart-related parameters, and the absolute difference between the two divided by Vavg is the error rate of the model detection. The error rates of p ultrasound images in the test dataset are calculated and their average values ​​are taken. This is the average percentage error (MPE) of the object detection model 21.

[0036] The entire system of this invention consists of several key components. The first is the object detection model 21 (YOLO v4) trained based on various cardiac-related parameters. The object detection model 21 utilizes the labeled cardiac ultrasound image I, and this embodiment is specifically optimized for cardiac ultrasound images of premature infants.

[0037] Next, referring to Figure 3, the object detection model 21, after training, will perform the following steps:

[0038] Step S2-1: Input the cardiac ultrasound image I obtained from patient P;

[0039] Step S2-2: The object detection model 21 can detect the rectangular bounding box representing various heart-related parameters with high accuracy, such as left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, and RR interval.

[0040] Step S2-3: After obtaining these rectangular bounding boxes, the system of the present invention will convert the pixel information in the cardiac ultrasound image I into actual physical quantities; the physical quantities include one or more of the following: left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, and ventricular rhythm cycle (RR interval).

[0041] Preferably, the conversion process can be optionally combined with factors such as the resolution, image scale, and scan settings of the cardiovascular ultrasound imager 10 used to ensure accurate correspondence between pixel distance and actual size. After conversion, the present invention can successfully calculate accurate values ​​of various cardiac-related parameters based on any input cardiac ultrasound image I, providing a reliable basis for subsequent cardiac function assessment. This actual physical quantity is the cardiac physiological information of patient P.

[0042] Next, based on the obtained cardiac parameter values, the pressure-velocity relationship between the mean circumferential fiber shortening velocity (mVcfc) and the left ventricular end-systolic wall stress (ESWS) was calculated as a standard for assessing the cardiac status of patient P.

[0043] <Application Examples>

[0044] The pressure-velocity relationship between the average circumferential fiber shortening velocity (mVcfc) and the left ventricular end-systolic wall stress (ESWS) calculated by the system of the present invention is an important indicator for assessing myocardial function and cardiac load, especially for cardiovascular load in premature infants.

[0045] Please refer to Figure 4, which shows the pressure-velocity relationship between mVcfc and ESWS. This relationship can be used to assess the cardiac function of preterm infants. Figure 4 is a rectangular coordinate system, with ESWS on the horizontal axis and mVcfc on the vertical axis. Region I, where mVcfc exceeds 0.8 circ / s and ESWS is below 40 g / cm², indicates normal cardiac function. Conversely, if the pressure-velocity relationship of a preterm infant shows mVcfc below 0.8 circ / s and ESWS less than 40 g / cm² in Region II, relevant medical interventions may be necessary. If mVcfc is below 0.8 circ / s and ESWS exceeds 40 g / cm² in Region III, further treatment may be required. This stress-velocity framework, based on set mVcfc and ESWS thresholds, can accurately assess the cardiac function of preterm infants and determine the appropriate treatment strategy.

[0046] This invention provides a solution for automatically extracting key parameters from echocardiogram images and further analyzing cardiac function in premature infants. This not only improves diagnostic efficiency and accuracy but also reduces the workload of medical staff, providing strong support for clinical decision-making. The System I provided by this invention will help improve the quality of medical care for premature infants and promote more precise treatment and management.

[0047] The aforementioned PDA is only one application of this invention; other heart-related diseases can also be treated using the system provided by this invention, and are not limited thereto.

[0048] This invention uses flowcharts to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0049] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of the invention are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0050] Finally, it should be understood that the embodiments described in this invention are merely illustrative of the principles of the invention. Other modifications may also fall within the scope of this invention. Therefore, alternative configurations of the embodiments of this invention are considered as examples and not limitations, and are regarded as consistent with the teachings of this invention. Accordingly, the embodiments of this invention are not limited to those explicitly described and illustrated herein.

[0051] 10: Cardiovascular Ultrasound Imaging System 11: Ultrasonic probe 20: Physical or cloud-based hosting 21: Object Detection Model I: Echocardiogram P: Patient S1-1~S1-3, S2-1~S2-2: Steps

Claims

1. A deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in premature infants, comprising: a cardiovascular ultrasound imager electrically or signalally connected to an ultrasound probe to acquire a cardiac ultrasound image; the cardiac ultrasound image includes an M-mode (Motion Mode) cardiac ultrasound image and a pulsed-wave Doppler cardiac ultrasound image; a physical or cloud-based host for receiving the cardiac ultrasound image acquired by the cardiovascular ultrasound imager, comprising an object detection model and performing the following steps: the object detection model detects a rectangular bounding box representing various cardiac parameters in the cardiac ultrasound image; and converts pixel information in the cardiac ultrasound image from which the rectangular bounding box is acquired into a physical quantity.

2. The deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in preterm infants as described in claim 1, wherein: These cardiac parameters include left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, ventricular rhythm cycle, or one or more combinations thereof.

3. The deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in preterm infants as described in claim 1 or 2, wherein the training steps of the object detection model include: Step 1: Data set construction: A raw dataset formed by several cardiac ultrasound images is randomly divided into a training dataset and a validation dataset according to a ratio, and the raw dataset undergoes a preprocessing step and a data augmentation step; and Step 2: Model training: The aspect ratio and area of ​​the rectangular bounding box of the cardiac ultrasound image in the training dataset are estimated, an appropriate anchor box is selected, and the object detection model is used to perform transfer learning according to preset hyperparameters. The training results are observed, and the hyperparameters are optimized to obtain the model with the best accuracy.

4. The deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in preterm infants as described in claim 3, wherein: The cardiac ultrasound image contains at least one annotation within the rectangular bounding box.

5. The deep learning-based cardiac ultrasound image recognition system for patent ductus arteriosus in preterm infants as described in claim 4, wherein: This annotation includes left ventricular ejection time, left ventricular end-diastolic diameter, left ventricular end-systolic diameter, end-systolic wall thickness, ventricular rhythm cycle, or one or more combinations thereof.

6. The deep learning-based cardiac ultrasound image recognition system for patent ductus arteriosus in preterm infants as described in claim 3, wherein: The preprocessing steps include cropping the region of interest from the original cardiac ultrasound image data, converting it to grayscale, and standardizing and resizing it.

7. A deep learning-based cardiac ultrasound image recognition system for patent ductus arteriosus in preterm infants as described in claim 6, wherein: Standardization involves unifying the differences in brightness and contrast between different images; size conversion adjusts the cardiac ultrasound image to a fixed resolution to meet the model input requirements.

8. The deep learning-based cardiac ultrasound image recognition system for open ductus arteriosus in preterm infants as described in claim 3, wherein the training step of the object detection model further includes, after step 2: Step 3 Model testing: The object detection model is used to perform performance analysis and verification using one of the test datasets according to equation (1): …Equation (1); wherein: Vavg is the actual value of the heart-related parameters, and Vavg' is the result of the model detecting the heart-related parameters. The absolute difference between the two divided by Vavg gives the error rate of the model detection. The error rates of p ultrasound images in the test dataset are calculated separately and their average values ​​are taken. This is the average percentage error of the object detection model.

9. A deep learning-based cardiac ultrasound image recognition system for patent ductus arteriosus in preterm infants as described in claim 1, wherein: This physical quantity contains the pressure-velocity relationship between the average circumferential fiber shortening rate and the end-systolic wall stress of the left ventricle.

Images