Automatic acquisition method of qualified four-chamber heart section images in fetal heart ultrasound video

Abstract

The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound videos includes: performing frame processing on fetal heart ultrasound videos; manually screening out qualified four-chamber heart section images, and marking the four-chamber heart and descending aorta regions; building and training a YOLOv5 target detection network; building and training a Darknet53 classification network; performing frame processing and using the YOLOv5 target detection network to detect the four-chamber heart and descending aorta in the fetal heart ultrasound video of the section to be extracted frame by frame; recording whether the descending aorta exists in the video and extracting the detected four-chamber heart region image; using the Darknet53 classification network to classify the extracted quasi-four-chamber heart region, and at the same time calculating the scores of the four-chamber heart section images classified as qualified, sorting these section images according to the scores, and automatically giving the best four-chamber heart section image in the video. The method of the present invention can effectively obtain qualified four-chamber heart section images in fetal heart ultrasound videos, and can also give the best section image.

Description

Technical Field

[0001] The present invention relates to the technical field of fetal heart ultrasound imaging, and in particular to a method for automatically acquiring a qualified four-chamber heart section image in a fetal heart ultrasound video. Background Art

[0002] Ultrasound imaging technology has been widely used in the screening and diagnosis of fetal congenital heart disease before birth due to its advantages of being painless, non-invasive, non-ionizing radiation, and capable of real-time imaging. Compared with imaging methods such as CT and MRI, the quality of ultrasound images contains more noise and is also affected by fetal activity. In addition, there are subjective differences in ultrasound doctors' operating techniques and image reading. Many factors have led to a relatively low detection rate of fetal congenital heart disease. Every year, about 300,000 fetuses die from congenital heart disease during pregnancy or delivery.

[0003] As documented in the literature [1]: Vullings R, Fetal Electrocardiography and Deep Learning for Prenatal Detection of Congenital Heart Disease, 2019 Computing in Cardiology (CinC), 2019, pp. Page 1-Page 4, doi:10.23919 / CinC49843.2019.9005870.

[0004] In order to effectively improve the detection rate and screening efficiency of fetal congenital heart disease, it is crucial to obtain qualified cross-sectional images in fetal heart ultrasound videos. At present, prenatal screening based on key ultrasound cross-sectional images has become an indispensable part of prenatal examination. Ultrasound cross-sectional images are currently mainly obtained manually by professional doctors through ultrasound equipment. However, manual acquisition depends largely on the doctor's experience, is easily affected by the doctor's subjective factors, and requires a lot of time and effort.

[0005] Among many key ultrasound section images, the four-chamber heart section is the most important and basic section in fetal heart ultrasound images, and it is also the most used section in the screening process of congenital heart disease. If it can be automatically acquired using computer vision and artificial intelligence technology, it will not only reduce the workload of evaluation experts, but also have important application value in expanding the automatic acquisition of other ultrasound sections.

[0006] In recent years, deep learning technology has made breakthrough progress in the field of medical image processing and analysis, and more and more scientific researchers have begun to pay attention to its application in the quality evaluation of fetal heart and head ultrasound sections.

[0007] Reference [2]: Abdi A, Luong C, Tsang T, et al. Automatic Quality Assessment of Echocardiograms Using Convolutional Neural Networks: Feasibility on the Apical Four-Chamber View [J]. IEEE Trans Med Imaging, 2017: 1-1. A deep convolutional neural network was used to classify apical four-chamber echocardiogram images on a 0-5 scale. However, the network used can only classify the entire image because it cannot locate the specified anatomical structure. Therefore, it cannot provide stable, reliable and accurate classification results based on the clarity of the anatomical structure. On this basis, the literature [3]: Baumgartner CF, Kamnitsas K, et al. SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound. [J]. IEEE transactions on medical imaging, 2017. proposed SonoNet which can automatically detect 13 standard sections in two-dimensional fetal ultrasound images and locate various anatomical structures, such as the four-chamber ventricle of the fetal heart, in a weakly supervised manner through bounding boxes. However, the positioning accuracy and classification performance of this method are insufficient. With the continuous development of target detection networks, their detection accuracy and speed have been greatly improved.

[0008] Reference [8]: The Chinese patent "Ultrasound Image Evaluation and Screening Method and Device" (CN109191442B) uses the five anatomical structures in the fetal head circumference ultrasound image recorded in reference [4] Ren S, He K, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks [J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39 (6): 1137-1149. to detect the five anatomical structures in the fetal head circumference ultrasound image, and obtains the evaluation score corresponding to the image according to whether the corresponding anatomical structure is detected. However, the scoring effect of this method is relatively dependent on the manually set empirical threshold. When this method is used to detect fetal heart ultrasound video, when the threshold is set too low, some other structures will be misdetected, as shown in Figure 1 (a), Figure 1 (b), and Figure 1 (c).

[0009] The document [9]: Chinese patent "A method for quality control of ultrasound section images of fetuses in the middle and late pregnancy" (CN110464380A) uses the document [6] Redmon J and Farhadi A. 2018. YOLOv3: An Incremental Improvement [EB / OL]. [2021-11-15]. https: / / arxiv.org / pdf / 1804.02767.pdf to detect the anatomical structure of multiple fetal ultrasound section images, calculate the overlap of the key anatomical structures according to their corresponding position coordinates, and then filter out the misdetected anatomical structures according to the overlap association table. Finally, the tissue-score mapping table is used to obtain the quality score of the fetal ultrasound section image after the key structure is filtered. This method reduces the false detection of the target detection model to a certain extent, but it is more dependent on clinical prior knowledge.

[0010] Reference

[10] : Chinese patent "Ultrasonic section image quality control method, device and computer equipment" (CN112070119A) uses reference [5] T.Lin, P.Goyal, R.Girshick, K.He and P.Dollár, FocalLoss for Dense Object Detection, in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.42, no.2, pp.318-327, 1Feb.2020, doi:10.1109 / TPAMI.2018.2858826. The anatomical structures of multiple fetal ultrasound section images are detected. According to the rules of whether the structure in each section appears and whether it is clear, the evaluation results such as whether the fetal ultrasound section image is a standard section image are obtained, and the quality score of each image is generated. The quality score is calculated by the weighted sum of the weight of the relevant anatomical structure and the confidence of the detection box. However, the confidence of the detection box of the actual anatomical structure may not completely match its clarity, as shown in Figure 2(a) and Figure 2(b). The valves in the four-chamber heart area in Figure 2(a) are almost invisible, but the confidence is higher than that in Figure 2(b), indicating that it is unreliable to automatically obtain the qualified four-chamber heart section in the fetal heart ultrasound video by relying solely on the confidence of the detection box.

[0011] Based on the work of predecessors, the Chinese patent "Automatic quality control method for standard sections of early pregnancy fetus based on multi-task" (CN113393456A)

[11] proposed a multi-task deep convolutional neural network to automatically control the quality of standard sections of early pregnancy fetus. The network obtains the section category and its confidence corresponding to the section image, the category, coordinates and confidence of all anatomical structures in the section image, and inputs the section category corresponding to each ultrasound section image and the category of all anatomical structures in the section image into the trained SVM classification model. The section image is judged to be standard according to the pre-established matching rules. The quality control result of this method is easily affected by the accuracy of the classification module, especially when the section is misclassified. Even if the detection results of all anatomical structures on the section meet the requirements of the original standard section, the classification result is still a non-standard section.

[0012] In summary, it is unstable and unreliable to automatically acquire fetal ultrasound section images using only target detection or classification models. On the one hand, the classification model cannot locate the anatomical structure and cannot focus all its attention on the designated anatomical structure. On the other hand, the target detection model's detection frame confidence for the anatomical structure does not fully match its clarity, which can easily cause misdetection of the anatomical structure of the ultrasound image. It is necessary to combine the target detection and classification methods and adopt a strategy of automatic acquisition of section images that first locates and then classifies, so as to improve the reliability of automatic acquisition of qualified four-chamber heart section images in fetal ultrasound videos. Summary of the invention

[0013] The classification model cannot locate the specified anatomical structure, so it cannot focus all the model's attention on the four-chamber heart area; and the target detection model cannot accurately reflect the clarity of the four-chamber heart area, resulting in multiple missed detections at high thresholds and multiple false detections at low thresholds. The present invention proposes a method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound videos, first using a target detection network to locate the heart area, and then using an image classification network to implement a strategy for classifying qualified four-chamber heart sections. The resulting sorting results are consistent with the human eye sorting results, and can effectively acquire qualified four-chamber heart section images in fetal heart ultrasound videos, while also providing the best section images.

[0014] The technical solution adopted by the present invention is:

[0015] A method for automatically acquiring a qualified four-chamber heart section image in a fetal heart ultrasound video comprises the following steps:

[0016] Step 1: Frame processing of fetal heart ultrasound video;

[0017] Step 2: Manually screen qualified four-chamber heart section images and mark the four-chamber heart and descending aorta areas;

[0018] Step 3: Build and train the YOLOv5 target detection network;

[0019] Step 4: Build and train the Darknet53 classification network;

[0020] Step 5: Frame processing, and use the YOLOv5 target detection network to detect the four-chamber heart and descending aorta in the fetal heart ultrasound video of the section to be extracted frame by frame;

[0021] Step 6: Record whether the descending aorta exists in the video and extract the detected four-chamber heart area image;

[0022] Step 7: Use the Darknet53 classification network to classify the extracted quasi-four-chamber heart region, and calculate the scores of the four-chamber heart section images classified as qualified. These section images are sorted according to the scores, and the best four-chamber heart section image in the video is automatically given.

[0023] The invention discloses a method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound videos, and the beneficial effects are as follows:

[0024] 1) The image classification model cannot locate the anatomical structure in the ultrasound section image, so it cannot focus all the model's attention on the specified anatomical structure; and although the target detection model can give the location coordinates of the anatomical structure, it is easy to misdetect the anatomical structure because its detection box confidence does not completely match its clarity.

[0025] In view of the above problems, the method of the present invention uses the YOLOv5 model to roughly locate the key anatomical structures, extracts them and then uses Darknet53 to classify the key anatomical structures, which can reduce the situation of multiple missed detections under high thresholds and multiple false detections under low thresholds of YOLOv5.

[0026] 2) A qualified four-chamber heart section image generally includes the four-chamber heart and the descending aorta area. Since the fetal heart ultrasound video is continuous and dynamic, the recording process will be affected by factors such as the fetus and the doctor's technical level. Not every frame of the four-chamber heart section image has a clearly visible descending aorta area, which makes it difficult to judge whether the four-chamber heart section image is qualified, and it is difficult to obtain the best four-chamber heart section image in the video.

[0027] In view of the above problems, the method of the present invention automatically obtains qualified four-chamber heart section images by integrating the time series relationship. The method determines whether the four-chamber heart section in the video is a quasi-qualified four-chamber heart section image by whether there is a descending aorta in the video. Then, the four-chamber heart area in the quasi-qualified section image is extracted and handed over to Darknet53 to determine whether it is truly qualified. According to the output of the Darknet53 network before softmax processing and the detection confidence of the descending aorta, the score of the qualified four-chamber heart section image is calculated, and the qualified four-chamber heart section images in the video are sorted according to the score size. In this way, not only can the qualified four-chamber heart section image in the video be automatically obtained, but also the best four-chamber heart section image can be given at the same time.

[0028] 3) To solve the problem of stable, reliable and accurate automatic acquisition of qualified four-chamber heart section images in fetal heart ultrasound videos in the prior art, the method of the present invention first uses the target detection network to realize the positioning of the heart area, and then uses the classification network to realize the strategy of classifying qualified four-chamber heart sections, which effectively reduces the situation of multiple missed detections under high thresholds and multiple false detections under low thresholds by the target detection model.

[0029] 4) To solve the problem in 3) above that it is difficult to judge the eligibility of a single frame of a four-chamber heart section image that does not contain the descending aorta due to failure to consider the context information of the video frame, and it is difficult to obtain the best four-chamber heart section image in the video. On the basis of 3) above, the method of the present invention obtains a four-chamber heart section image sorting result that is basically consistent with human eye perception by integrating the temporal relationship and constructing a scoring formula for a qualified four-chamber heart section image, which can effectively extract qualified four-chamber heart section images in fetal heart ultrasound videos and provide the best section image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1(a) is a diagram of the misdetected four-chamber heart region;

[0031] Figure 1(b) is a second image of the misdetected four-chamber heart region;

[0032] Figure 1(c) is a third diagram of the misdetected four-chamber heart region.

[0033] Figure 2(a) is a non-standard regional diagram of a four-chamber heart;

[0034] FIG2( b ) is a non-standard regional diagram of a four-chamber heart.

[0035] FIG3( a ) is a flowchart of automatically acquiring qualified four-chamber heart view images (model training stage);

[0036] FIG3( b ) is a flowchart for automatically acquiring qualified four-chamber heart view images (model use stage).

[0037] Figure 4 (1) is a sample image of an unqualified four-chamber heart region;

[0038] Figure 4 (2) is a sample image of an unqualified four-chamber heart region;

[0039] Figure 4 (3) is a sample image of an unqualified four-chamber heart region;

[0040] Figure 4 (4) is a sample image of an unqualified four-chamber heart region;

[0041] Figure 4 (5) is a sample image of an unqualified four-chamber heart region;

[0042] Figure 4(6) is a sample image of an unqualified four-chamber heart region. Figure 6 ;

[0043] Figure 4 (7) is a sample image of an unqualified four-chamber heart region, Figure 7;

[0044] Figure 4(8) is a sample eight of an unqualified four-chamber heart region image.

[0045] Figure 5 (1) is a sample image of a qualified four-chamber heart region;

[0046] Figure 5 (2) is a sample image of a qualified four-chamber heart region;

[0047] Figure 5 (3) is a sample image of a qualified four-chamber heart region;

[0048] Figure 5 (4) is a sample image of a qualified four-chamber heart region;

[0049] Figure 5 (5) is a sample image of a qualified four-chamber heart region;

[0050] Figure 5(6) is a sample image of a qualified four-chamber heart region. Figure 6 ;

[0051] Figure 5 (7) is a sample image of a qualified four-chamber heart region, Figure 7;

[0052] Figure 5(8) is a sample eight of a qualified four-chamber heart region image.

[0053] Figure 6 Schematic diagram of the divided Darknet53 classification dataset.

[0054] FIG7( a ) is a diagram showing a situation where a four-chamber heart and a descending aorta are detected in a fetal heart ultrasound video (only the descending aorta is detected);

[0055] FIG7( b ) is a second diagram showing a situation where a four-chamber heart and descending aorta are detected in a fetal heart ultrasound video (only a four-chamber heart is detected);

[0056] FIG7( c ) is a third picture showing the situation where a four-chamber heart and a descending aorta are detected in a fetal heart ultrasound video (the descending aorta and the four-chamber heart are detected at the same time).

[0057] FIG8 (1) is a sample image 1 of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0058] FIG8(2) is a second sample of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0059] FIG8 (3) is a sample image 3 of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0060] FIG8 (4) is a sample image 4 of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0061] FIG8 (5) is a sample image 5 of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0062] Figure 8(6) shows a sample of the four-chamber heart region in a quasi-qualified four-chamber heart section image. Figure 6 ;

[0063] FIG8 (7) is a sample image 7 of the four-chamber heart region in a quasi-qualified four-chamber heart section image;

[0064] FIG8 (8) is a sample eight of the four-chamber heart region in a quasi-qualified four-chamber heart section image. DETAILED DESCRIPTION

[0065] A method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound videos, comprising:

[0066] The model training stage includes the following steps, as shown in FIG3( a):

[0067] Step 1: Frame processing of fetal heart ultrasound video;

[0068] Step 2: Manually screen qualified four-chamber heart section images and mark the four-chamber heart and descending aorta areas;

[0069] Step 3: Build and train the YOLOv5 target detection network;

[0070] Step 4: Build and train the Darknet53 classification network;

[0071] The model use phase includes the following steps, as shown in FIG3( b):

[0072] Step 5: Frame processing, and use the YOLOv5 target detection network to detect the four-chamber heart and descending aorta in the fetal heart ultrasound video of the section to be extracted frame by frame;

[0073] Step 6: Record whether the descending aorta exists in the video and extract the detected four-chamber heart area image;

[0074] Step 7: Use the Darknet53 classification network to classify the extracted quasi-four-chamber heart region, and calculate the scores of the four-chamber heart section images classified as qualified. These section images are sorted according to the scores, and the best four-chamber heart section image in the video is automatically given.

[0075] The step 1 comprises the following steps:

[0076] Step 1.1: Name the fetal heart ultrasound videos in the order of 1, 2, 3, ..., such as 1.avi, 2.avi, etc.;

[0077] Step 1.2: Divide each fetal heart ultrasound video into frames and name them frame by frame in the format of video sequence number_frame number, such as 1_1.png, 1_2.png, etc.

[0078] The step 2 is specifically as follows:

[0079] The fetal heart video frame dataset obtained in step 1.2 was manually screened to select qualified four-chamber heart section images, and the four-chamber heart and descending aorta regions were annotated and used as the training dataset, validation dataset, and test dataset of the YOLOv5 target detection network model.

[0080] The step 3 comprises the following steps:

[0081] Step 3.1: Divide the data set obtained in step 1.2 into training set, validation set, and test set in a ratio of 6:2:2, and use the divided data set to train the YOLOv5 target detection model;

[0082] Step 3.2: The detection performance of the YOLOv5 target detection network model is evaluated through the test set, as shown in Table 1; the test set here is only the qualified four-chamber heart section images manually screened in step 1.2.

[0083] Table 1 Performance indicators of the YOLOv5 model on the target detection task test set

[0084] Detection target Accuracy Recall Average accuracy Four-chamber heart 0.998 0.999 0.995 Descending aorta 0.904 0.871 0.865

[0085] Step 3.3: Use the trained YOLOv5 target detection network model to perform frame-by-frame detection on the remaining fetal heart ultrasound video frames after the qualified cross-sectional image frames are manually screened out in step 1.2. That is, use the trained YOLOv5 target detection network model to extract samples that are prone to misdetection, and extract the areas that are misdetected as four-chamber hearts as unqualified four-chamber heart area images, such as Figure 4(1) to Figure 4(8) shown.

[0086] Step 3.4: Using the manually marked position coordinates, extract the four-chamber heart region from the qualified four-chamber heart section image manually screened in step 2 as a qualified four-chamber heart region image, such as Figure 5(1) to Figure 5(8) shown.

[0087] Step 3.5: Use the unqualified four-chamber heart region images and qualified four-chamber heart region images in steps 3.3 and 3.4 to construct a training data set, a validation data set, and a test data set for the Darknet53 classification network model.

[0088] The step 4 comprises the following steps:

[0089] Step 4.1: In order to ensure that the number of qualified and unqualified four-chamber heart region images in the randomly divided training set, validation set and test set is consistent, the qualified and unqualified four-chamber heart region images obtained in step 3.5 are first divided into training set, validation set and test set in a ratio of 5:2:3, and then the divided data sets are merged, as shown in the following example: Figure 6 shown.

[0090] Step 4.2: Use the classification data set divided in step 4.1 to train the Darknet53 classification network, and use the test set to evaluate the classification performance of the Darknet53 classification network.

[0091] Experiments show that the average TOP-1 accuracy of the trained Darknet53 classification network model on the test set reached 98.82%. In order to evaluate the performance of the classification model in real scenarios, the fetal heart ultrasound video set used in the model use stage in the present invention is different from the fetal heart ultrasound video set used in the model training stage.

[0092] The step 5 comprises the following steps:

[0093] Step 5.1: Perform the same frame processing as steps 1.1 and 1.2 on the fetal heart ultrasound video of the section to be extracted

[0094] Step 5.2: Using the YOLOv5 target detection network, a threshold of 0.5 is used to detect the four-chamber heart and descending aorta region in the fetal heart ultrasound video of the section to be extracted frame by frame, as shown in Figure 7(a), Figure 7(b), and Figure 7(c).

[0095] The step 6 comprises the following steps:

[0096] Step 6.1: Name the detection results output by YOLOv5 in the format of video sequence number_frame number, and check whether the descending aorta exists in the detection result file output by YOLOv5. If the descending aorta exists, record the video sequence number, frame number and descending aorta detection confidence in the file name, and regard the sections in which the four-chamber heart region is detected in the video as quasi-qualified four-chamber heart section images; if the algorithm fails to detect a reliable descending aorta region in the video, even if the four-chamber heart region is detected in the video, the section image is regarded as an unqualified four-chamber heart section image and is not automatically acquired;

[0097] Step 6.2: Extract the four-chamber heart region from the quasi-qualified four-chamber heart section image in step 6.1, such as Figure 8(1) to Figure 8(8) shown.

[0098] In order to facilitate the search for the corresponding four-chamber heart section image, the quasi-qualified four-chamber heart section image is named in the format of video sequence number_frame number. Under different thresholds, YOLOv5 has different false detection rates and missed detection rates for the four-chamber heart region, as shown in Table 2. The calculation formulas for the missed detection rate and false detection rate are shown in (1) and (2):

[0099]

[0100]

[0101] Among them: X represents the number of frames in which YOLOv5 detects a quasi-four-chamber heart region, Y represents the number of frames in which YOLOv5 detects a truly qualified four-chamber heart region, and Z represents the number of frames in which a truly qualified four-chamber heart region is detected in the video to be detected.

[0102] Table 2 False detection rate and missed detection rate of YOLOv5 for the four-chamber heart area under different thresholds

[0103] Threshold 0.5 0.6 0.7 0.8 0.85 0.9 False positive rate 82.8％ 80.3 76.2％ 68.2％ 59.2％ 39.7％ Missed detection rate 0％ 0％ 1.0％ 8.3％ 20.1％ 75.8％

[0104] The step 7 comprises the following steps:

[0105] Step 7.1: When the YOLOv5 threshold in step 6.2 is 0.5, the extracted four-chamber heart region is input into the trained Darknet53 classification model, and the prediction result is output. The classification performance is shown in Table 3;

[0106]

[0107]

[0108]

[0109] Darknet53's false positive rate = 1 - accuracy (6)

[0110] Darknet53’s missed detection rate = 1-recall rate (7)

[0111] Among them: TP means the number of quasi-four-chamber heart regions where the actual result is qualified and the predicted result is qualified; TN means the number of quasi-four-chamber heart regions where the actual result is unqualified and the predicted result is unqualified; FP means the number of quasi-four-chamber heart regions where the actual result is unqualified and the predicted result is qualified; FN means the number of quasi-four-chamber heart regions where the actual result is qualified and the predicted result is unqualified

[0112] Table 3 Actual classification performance of Darknet53 classification model

[0113] Performance Indicators Accuracy Accuracy Recall False positive rate Missed detection rate Numeric 97.02％ 98.09％ 84.39％ 2.98％ 15.61％

[0114] Step 7.2: Obtain the four-chamber heart section image corresponding to the qualified four-chamber heart region, and calculate the score of the qualified four-chamber heart section image. If the four-chamber heart region in the section image is qualified, the output of the Darknet53 classification network before softmax processing is normalized and used as the score of the section image; if the four-chamber heart region in the section image is qualified and the descending aorta exists in the section image, the output of the Darknet53 classification network before softmax processing is normalized and added to the descending aorta detection confidence as the score of the corresponding section image;

[0115] The score of qualified four-chamber heart view image is calculated using formula (8), where D i represents the score of the qualified four-chamber heart section image of the i-th frame in the video, T max Indicates the highest score of the qualified four-chamber heart area in the video, T min Indicates the minimum score of the qualified four-chamber heart area in the video, T i Indicates the score of the qualified four-chamber heart region in the i-th frame of the video, i represents the frame number in the video, DAO_conf i The descending aorta score of the qualified four-chamber heart view in the i-th frame of the video is set to 0 if there is no descending aorta.

[0116]

[0117] Step 7.3: Sort the section images according to the scores, and automatically obtain the best four-chamber heart section image in the video, as shown in Table 4;

[0118] Table 4 Ranking of scores for qualified four-chamber heart view images

[0119]

[0120]

[0121]

[0122] Although the method of the present invention has a small amount of false detection and missed detection, it has been greatly improved compared with YOLOv5, and the ranking results of the four-chamber heart section images are similar to the subjective ranking results of the human eye, indicating that the method for automatically acquiring qualified four-chamber heart section images has good application value.

[0123] The present invention proposes a new method for automatically acquiring four-chamber ultrasound section images of the fetal heart, and its main process is divided into a model training stage and a use stage. In the model training stage, the fetal heart ultrasound video is firstly framed; then, qualified four-chamber section images are manually screened out and the four-chamber heart and descending aorta regions are annotated; then, the annotated four-chamber section image dataset is used to train the YOLOv5 network in the document [7]: Ultralytics.YOLOv5[EB / OL].[2021.11.15].https: / / github.com / ultralytics / YOLOv5; the trained YOLOv5 model is used to detect previously unannotated video frames (these are all non-four-chamber sections) using appropriate thresholds, and the misdetected section images are automatically extracted; finally, according to the position coordinates of the four-chamber heart region manually annotated (qualified) and misdetected by YOLOv5 (unqualified), the region is extracted and input into the document [6]: Redmon J and Farhadi A.2018.YOLOv3:AnIncremental Improvement[EB / OL].[2021-11-15].https: / / arxiv.org / pdf / 1804.02767.pdf for training.

[0124] In the model use stage, the trained YOLOv5 model is first used to detect the four-chamber heart and descending aorta regions in the fetal heart ultrasound video frame by frame using appropriate thresholds. If reliable descending aorta regions are detected in several frames of the video, even if the algorithm fails to find the descending aorta region in some sections containing reliable four-chamber heart regions, these section images can still be regarded as quasi-qualified four-chamber heart section images; then the detected four-chamber heart region is extracted and input into the trained Darknet53 model for classification to further determine whether the region is a qualified four-chamber heart region. If the four-chamber heart region is qualified, the output before softmax processing in the Darknet53 network is normalized as the score of the corresponding four-chamber heart section image; if the four-chamber heart region is qualified and the corresponding four-chamber heart section image has a descending aorta region, the output before softmax processing in the Darknet53 network is normalized and added to the descending aorta detection confidence as the score of the corresponding four-chamber heart section image; finally, the qualified four-chamber heart section images in the video are sorted according to the score. Through the image comparison experiment in Table 3 and Table 4, it is found that the strategy adopted by the present invention is to first use the target detection network to realize the positioning of the heart area, and then use the image classification network to realize the classification of qualified four-chamber heart sections. The obtained sorting result is consistent with the sorting result of the human eye, and can effectively obtain qualified four-chamber heart section images in fetal heart ultrasound videos, and at the same time can also provide the best section image.

Claims

1. A method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video, characterized in that The following steps are involved: Step 1: Frame processing of fetal heart ultrasound video; Step 2: Screen out qualified four-chamber heart section images and mark the four-chamber heart and descending aorta areas; Step 3: Build and train the YOLOv5 target detection network; Step 4: Build and train the Darknet53 classification network; Step 5: Frame processing, and use the YOLOv5 target detection network to detect the four-chamber heart and descending aorta in the fetal heart ultrasound video of the section to be extracted frame by frame; Step 6: Record whether the descending aorta exists in the video, and extract the detected four-chamber heart area image; Step 7: Use the Darknet53 classification network to classify the extracted quasi-four-chamber heart region, and calculate the scores of the four-chamber heart section images classified as qualified. These section images are sorted according to the scores, and the best four-chamber heart section image in the video is automatically given; Step 7 includes the following steps: Step 7.1: Input the four-chamber heart region extracted by YOLOv5 into the trained Darknet53 classification network model and output the prediction result; Darknet53's false positive rate = 1 - accuracy (6) Darknet53's missed positive rate = 1 - recall (7) Where: TP indicates the number of quasi-four-chamber heart regions where the actual result is qualified and the predicted result is qualified; TN indicates the number of quasi-four-chamber heart regions where the actual result is unqualified and the predicted result is unqualified; FP indicates the number of quasi-four-chamber heart regions where the actual result is unqualified and the predicted result is qualified; FN indicates the number of quasi-four-chamber heart regions where the actual result is qualified and the predicted result is unqualified; Step 7.2: Obtain the four-chamber heart section image corresponding to the qualified four-chamber heart region, and calculate the score of the qualified four-chamber heart section image. If the four-chamber heart region in the section image is qualified, the output of the Darknet53 classification network before softmax processing is normalized and used as the score of the section image; if the four-chamber heart region in the section image is qualified and the descending aorta exists in the section image, the output of the Darknet53 classification network before softmax processing is normalized and added to the descending aorta detection confidence as the score of the corresponding section image; The score of the qualified four-chamber heart view image is calculated using formula (8): Where: D i represents the score of the qualified four-chamber heart section image of the i-th frame in the video, T max Indicates the highest score of the qualified four-chamber heart area in the video, T min Indicates the minimum score of the qualified four-chamber heart area in the video, T i Indicates the score of the qualified four-chamber heart region in the i-th frame of the video, i represents the frame number in the video, DAO_conf i The descending aorta score of the qualified four-chamber heart view in the i-th frame of the video is set to 0 if there is no descending aorta. Step 7.3: Sort the section images according to the scores, and automatically obtain the best four-chamber heart section image in the video.

2. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 1, characterized in that: Step 2 is as follows: The fetal heart video frame dataset obtained in step 1 is manually screened to select qualified four-chamber heart section images, and the four-chamber heart and descending aorta regions are annotated and used as the training dataset, verification dataset, and test dataset of the YOLOv5 target detection network model.

3. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 2, characterized in that: Step 3 includes the following steps: Step 3.1: Divide the data set obtained in step 1 into training set, validation set, and test set according to a certain ratio, and use the divided data set to train the YOLOv5 target detection network model; Step 3.2: Evaluate the detection performance of the YOLOv5 target detection network model through the test set. The test set here refers to the qualified four-chamber heart section images manually screened from step 1. Step 3.3: Use the trained YOLOv5 target detection network to extract samples that are prone to misdetection, and extract the area that is misdetected as a four-chamber heart as an unqualified four-chamber heart area image; Step 3.4: Using the manually marked position coordinates, extract the four-chamber heart region from the qualified four-chamber heart section image manually screened in step 2 as a qualified four-chamber heart region image; Step 3.5: Use the unqualified four-chamber heart region images and qualified four-chamber heart region images in steps 3.3 and 3.4 to construct a training data set, a validation data set, and a test data set for the Darknet53 classification network model.

4. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 3, characterized in that: Step 4 includes the following steps: Step 4.1: Divide the qualified and unqualified four-chamber heart region images obtained in step 3.5 into a training set, a validation set, and a test set according to a certain ratio, and then merge the divided data sets; Step 4.2: Use the classification data set divided in step 4.1 to train the Darknet53 classification network, and use the test set to evaluate the classification performance of the Darknet53 classification network.

5. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 4, characterized in that: Step 5 includes the following steps: Step 5.1: Frame the fetal heart ultrasound video of the section to be extracted; Step 5.2: Use YOLOv5 to detect the four-chamber heart and descending aorta region in the fetal heart ultrasound video of the section to be extracted frame by frame using a certain threshold.

6. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 5, characterized in that: Step 6 includes the following steps: Step 6.1: Name the detection results output by YOLOv5 in the format of video sequence number_frame number, and check whether the descending aorta exists in the detection result file output by YOLOv5. If the descending aorta exists, record the video sequence number, frame number and descending aorta detection confidence in the file name, and regard the sections in which the four-chamber heart region is detected in the video as quasi-qualified four-chamber heart section images; if the algorithm fails to detect a reliable descending aorta region in the video, even if the four-chamber heart region is detected in the video, the section image is regarded as an unqualified four-chamber heart section image and is not automatically acquired; Step 6.2: Extract the four-chamber heart region from the quasi-qualified four-chamber heart section image obtained in step 6.

1.

7. The method for automatically acquiring qualified four-chamber heart section images in fetal heart ultrasound video according to claim 6, characterized in that: The quasi-qualified four-chamber heart section images are named in the format of video sequence number_frame number. Under different thresholds, YOLOv5 has different false detection rates and missed detection rates for the four-chamber heart region. The calculation formulas for the missed detection rate and false detection rate are shown in (1) and (2): Among them: X represents the number of frames in which YOLOv5 detects a quasi-four-chamber heart region, Y represents the number of frames in which YOLOv5 detects a truly qualified four-chamber heart region, and Z represents the number of frames in which a truly qualified four-chamber heart region is detected in the video to be detected.

Images