A multi-modal congenital heart disease ultrasound detection method and system

Abstract

The present application belongs to the technical field of medical image intelligent processing, and provides a multi-modal congenital heart disease ultrasonic detection method and system. The method comprises: obtaining a defect sample for preprocessing operation, mixing normal samples according to a preset proportion, and dividing a training set, a test set and a validation set; the original image of the training set is cropped according to a preset first cropping rate and a second cropping rate, the YOLO11s network is input according to the order of the first cropping rate, the second cropping rate and the original image, and the YOLO11s network is trained in stages and progressively; the to-be-detected image including a two-dimensional gray ultrasonic image and a color Doppler blood flow image is subjected to the preprocessing operation, and then input into the trained YOLO11s network to output the final defect detection result. The present application can improve the model specificity by mixing normal samples, and solves the problems of difficult data acquisition, missed detection and false detection in primary hospitals through multi-stage learning.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image processing technology, specifically to a multimodal ultrasound detection method and system for congenital heart disease. Background Technology

[0002] Congenital heart disease (CHD) refers to abnormalities in the anatomical structure of the heart caused by impaired formation or developmental abnormalities of the heart and great vessels during embryonic development, or the failure of pathways that should close automatically after birth to close. CHD is the leading cause of birth defects in my country and a major cause of neonatal and infant mortality, with an incidence rate of approximately 9 per 1,000 live births. Meanwhile, the number of adult CHD patients and new cases continues to rise, creating a clinical demand for testing that covers both children and adults. Among all subtypes of CHD, ventricular septal defect (VSD) and atrial septal defect (ASSD) are the two most common clinical types, accounting for over 75% of all CHD cases.

[0003] Transthoracic echocardiography is considered the preferred method for detecting congenital heart disease (CHD) and is the primary basis for developing surgical or interventional treatment plans and evaluating their effectiveness. However, due to a lack of experienced echocardiologists, a large number of CHD cases (especially occult mild cases in adults and small defects in children) are delayed in detection, missing the optimal surgical window. In recent years, artificial intelligence has been widely used in echocardiography-assisted detection, but current technology lacks clinical scenarios specifically for CHD and small sample sizes.

[0004] Therefore, developing an intelligent ultrasound detection system for congenital heart disease that is suitable for small sample scenarios is of great significance for promoting patient health. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal ultrasound detection method and system for congenital heart disease, which solves the problems of overfitting and poor generalization in training with small-sample medical images of congenital heart disease.

[0006] In a first aspect, the present invention provides a multimodal ultrasound detection method for congenital heart disease, comprising: Defective samples were obtained and preprocessed, and normal samples were mixed in according to a preset ratio to divide the sample into training, testing and validation sets. Both defective and normal samples included two-dimensional grayscale ultrasound images and color Doppler flow maps of the subxiphoid biatrial ventricle standard and non-standard sections. The original images in the training set are cropped according to a preset first cropping ratio and a second cropping ratio. The images are then input into the YOLO11s network in the order of the first cropping ratio, the second cropping ratio, and the original image. The YOLO11s network is trained in stages. The optimal weights obtained in the previous stage are used as the pre-training weights for the next stage. The size of the original image, the image at the second cropping ratio, and the image at the first cropping ratio are gradually reduced. The images to be detected, including two-dimensional grayscale ultrasound images and color Doppler blood flow images, are subjected to the preprocessing operation, and then input into the trained YOLO11s network to output the final defect detection results.

[0007] Optionally, the preprocessing operation includes: The image is normalized while preserving its aspect ratio, and boundary padding is performed. Offline enhancement is performed on the image according to a given trigger probability to transform the image pixel features; wherein, the offline enhancement is: brightness and contrast fine adjustment, Gamma correction or low-intensity Gaussian noise injection.

[0008] Optionally, the step of normalizing the aspect ratio of the defective samples and performing boundary padding includes: Two-dimensional grayscale ultrasound images are non-destructively converted into three-channel RGB images, while the original three-channel information of color Doppler flow maps is preserved. Scale the image proportionally, using the longest side of the target as the reference; the scaling ratio is... , Let h be the pixel value of the longest side of the target image, and h and w be the height and width of the original image, respectively. Apply symmetrical black borders to the scaled image so that both the width and height are integer multiples of 32.

[0009] Optionally, cropping the original images of the training set according to a preset first cropping ratio and a second cropping ratio includes: Determine the size of the annotation box, and crop the width and height of the defect sample image to the first and second cropping ratios of the annotation box; Based on a unified coordinate library of defective samples, normal samples are cropped to the same region and size as defective samples.

[0010] Optionally, the total loss function of the YOLO11s base network is: , in, For CIoU bounding box regression loss, For classifying losses, For distribution focus loss; , , These are the weighting coefficients for each loss term.

[0011] Optionally, the classification loss uses a labeled, smoothed binary cross-entropy loss, specifically: , in, The categories predicted by the YOLO11s base network. The original hard labels for the samples. To predict the probability that this sample is defective for the YOLO11s base network, is the label smoothing coefficient, and N is the total number of samples.

[0012] Optionally, after performing phased progressive training on the YOLO11s network, the process also includes evaluating and optimizing the trained YOLO11s network, including: The original images and TTA-processed images of each group in the test set were input into the trained YOLO11s network to obtain two independent prediction results; each group of images included a two-dimensional grayscale ultrasound image and a color Doppler blood flow image of the same defect; The two sets of prediction results were normalized. The WBF weighted fusion algorithm is used to fuse the two sets of normalized prediction results and obtain the fusion confidence score. The coordinates of the fused bounding boxes are inversely normalized to convert them into absolute pixel coordinates of the original image. After filtering by a pre-set confidence threshold, the final prediction result is output. The final prediction result is then used to tune the parameters of the trained YOLO11s network.

[0013] In a second aspect, the present invention provides a multimodal congenital heart disease ultrasound detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multimodal congenital heart disease ultrasound detection method provided in the first aspect.

[0014] By adopting the above technical solution, this application has the following beneficial effects: (1) Coverage of all age groups: This method performs a preprocessing procedure on ultrasound images of congenital heart disease in children and adults, enabling the model to learn age-independent common defect features, while adapting to the detection needs of ultrasound images of congenital heart disease in children and adults, thus having a wider range of applications; (2) Flexible and controllable data threshold: The single-category detection paradigm supports the inclusion of normal samples with a configurable ratio of 0.1 to 0.5. It can complete basic training with only labeled defective samples, or improve the model specificity with a small number of normal samples, thus solving the problems of difficult data acquisition, missed detection and false detection in primary hospitals. (3) Excellent generalization in small sample scenarios: Through ultrasound-specific offline enhancement, three-stage course learning, synchronous feature alignment of normal and defective samples, and customized regularization and loss function, the problem of overfitting in small sample medical image training is solved, and high-precision detection can be achieved on small datasets, which greatly reduces the data threshold for intelligent detection of congenital heart disease. (4) Strong clinical real-time performance: Based on the YOLO11s single-stage end-to-end architecture, there is no need for a serial process of classification before detection. The single-graph inference speed is ≤0.2s, which meets the needs of clinical real-time detection. (5) High detection accuracy: By adapting the mixed training of normal and defective samples with customized loss function and optimizing the fusion of TTA and WBF inference, it takes into account both high recall (low false negatives) and high precision (low false positives), and meets the requirements of high sensitivity, high specificity and real-time performance in clinical testing on small sample datasets. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0016] Figure 1 A flowchart of a multimodal ultrasound detection method for congenital heart disease provided by an embodiment of the present invention is shown; Figure 2 Echocardiograms of patients with common congenital heart defects are shown; Figure 3 The diagram shows a prediction result of a multimodal ultrasound detection method for congenital heart disease provided by an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention. It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0018] In one embodiment, such as Figure 1 As shown, a multimodal ultrasound detection method for congenital heart disease is provided, including: S1. Obtain defective samples for preprocessing, mix them with normal samples according to a preset ratio, and divide them into training set, test set and validation set; both defective samples and normal samples include two-dimensional grayscale ultrasound images and color Doppler blood flow images of standard and non-standard sections of the subxiphoid dual atria.

[0019] This dataset contains 798 echocardiograms of patients with congenital heart disease of various ages. Each case includes independent images from two perspectives: a standard subxiphoid biventricular ventricular view and a non-standard view. Each perspective includes a corresponding two-dimensional grayscale ultrasound image and a color Doppler flow map. Both types of images were included as independent training samples in the dataset. The ratio of standard to non-standard view samples is approximately 1:1, matching the frequency of different views in actual clinical examinations. This eliminates the need to force uniformity of view angles, further improving the model's generalization ability in real-world clinical scenarios. All defect annotations were completed by a senior ultrasound physician with over 10 years of clinical experience and validated using intraoperative diagnostic criteria. 266 normal control echocardiograms were also included to confound defect samples. All clinical samples in this study were approved by the hospital's ethics committee.

[0020] The dataset employs a patient-level stratified random partitioning strategy to strictly avoid data leakage: using patient ID as the smallest partitioning unit, all patients are randomly divided into training, validation, and test sets in a ratio of 7.5:1.5:1. This ensures that all ultrasound images of the same patient (including 2D grayscale images and color Doppler flow maps of the same defect, and images from different cross-sections) are grouped into the same set and will not appear simultaneously in the training and validation / test sets. During partitioning, the data is first stratified and sorted by the sample size of a single patient and then randomly shuffled to ensure that the sample distribution of each set is consistent with clinical reality. The partitioning results are reproducible by using a fixed random seed SEED=42.

[0021] Optionally, the preprocessing operations in S1 include: S110. Perform aspect ratio-preserving normalization on the image and fill the boundaries. Specifically, S11 includes: S111. The two-dimensional grayscale ultrasound image is converted into a three-channel RGB image without loss, and the original three-channel information of the color Doppler blood flow image is preserved. When saving, the PNG format uses lossless compression parameters and the JPG format uses 100% quality parameters to avoid loss of image information.

[0022] S112. Scale the image proportionally, using the longest side of the target as the reference; the scaling ratio is... , Let h be the pixel value of the longest side of the target image, and h and w be the height and width of the original image, respectively.

[0023] S113. Perform symmetrical black border padding on the scaled image to ensure that the image's width and height are both integer multiples of 32. Specifically, perform symmetrical padding on the top, bottom, left, and right sides of the scaled image to ensure that the image's width and height are both integer multiples of 32, with a padding value of (0,0,0) to adapt to the 5-fold downsampling input requirement of the YOLO11s framework convolutional neural network.

[0024] S114. Dataset Integrity Verification: Simultaneously perform dataset integrity verification to ensure that the images in the training set, validation set, and test set correspond one-to-one with the annotations, remove empty and corrupted files, and complete the standardization preprocessing of all images.

[0025] S120. Perform offline enhancement on the image according to the given trigger probability to transform the image pixel features. The offline enhancement only transforms the image pixel features, without changing the coordinates and size of defects, thus avoiding annotation offset. This includes: brightness and contrast fine-tuning, Gamma correction, or low-intensity Gaussian noise injection.

[0026] Specifically, pixel-level offline enhancement is performed on defect samples in the training set, transforming only the image pixel features without changing the coordinates and size of the defects. This complements the online geometric enhancement of the YOLO framework, avoiding homogenization of enhancement strategies. The offline enhancement strategies include: brightness / contrast fine-tuning to adapt to the uneven brightness characteristics of ultrasound images, with a random trigger probability of 35%; and low-intensity Gaussian noise injection to simulate ultrasound speckle noise and improve model robustness, with a random trigger probability of 40%. In this embodiment, the total number of enhanced images is set to 200 (including two types of samples: 2D grayscale ultrasound images and color Doppler blood flow images). The maximum number of enhancements for a single original sample is one, avoiding overfitting caused by repeated enhancements. The enhanced samples and their corresponding annotations are simultaneously saved to the training set, forming a complete training dataset together with all the original unenhanced defect samples and subsequently mixed normal samples, for use in subsequent network model training.

[0027] The formula for calculating Gamma correction is: , in, Let (x, y) be the pixel value of the input image at (x, y). These are the corrected pixel values. The value is a correction coefficient, ranging from 0.92 to 1.08, which is adapted to the uneven brightness characteristics of ultrasound images.

[0028] This method performs a preprocessing procedure on ultrasound images of congenital heart disease from a dataset of samples across all age groups, enabling the model to learn age-independent universal defect features. It also adapts to the detection needs of ultrasound images of congenital heart disease in both children and adults, thus having a wider range of applications.

[0029] Optionally, in S1, normal samples are mixed in according to a preset ratio, including: S130. Configure the normal sample mixing ratio; the mixing ratio is the ratio of the number of normal samples to the number of defective samples, and is set to 0.5. The number of normal samples is 50% of the number of defective samples; the normal samples simultaneously contain two types of multimodal images: corresponding two-dimensional grayscale ultrasound images and color Doppler blood flow images.

[0030] S140. Read all normal control samples and perform channel unification and aspect ratio normalization preprocessing that is completely consistent with the defective samples to ensure the consistency of image distribution between normal and defective samples.

[0031] S150. Based on the original number of defective samples in the training set and validation set, calculate the required number of normal samples for each set according to a 50% mixing ratio. Select the corresponding number of normal samples using a random shuffling strategy with a fixed random seed SEED=42 to ensure the reproducibility of the experiment.

[0032] S160. Generate corresponding empty label files for the screened normal samples, adapt to the training format of YOLO11s single-class detection, and mark them as background areas without defect annotations.

[0033] S2. The original images of the training set are cropped according to the preset first cropping ratio and second cropping ratio. The images are then input into the YOLO11s network in the order of the first cropping ratio, the second cropping ratio, and the original images. The YOLO11s network is trained in stages. The optimal weights obtained in the previous stage are used as the pre-training weights for the next stage. The size of the original image, the image at the second cropping ratio, and the image at the first cropping ratio are gradually reduced.

[0034] It should be noted that this step uses the YOLO11s basic network structure without modifying its core network structure such as convolutional layers and feature fusion layers.

[0035] Optionally, step S2 involves cropping the original images of the training set according to a preset first cropping ratio and a second cropping ratio, including: S211. Determine the size of the annotation box, and crop the width and height of the defect sample image to the first cropping ratio and the second cropping ratio of the annotation box.

[0036] S212. Based on the unified coordinate library of defective samples, normal samples are cropped to the same area and size as defective samples.

[0037] In step S2, the images are input into the YOLO11s network in the order of the first cropping ratio, the second cropping ratio, and the original image. Specifically: The model is trained in stages according to the order of first cropping ratio → second cropping ratio → original image. The optimal weights completed in the previous stage are used as the pre-training weights of the YOLO11s model in the next stage, gradually improving the model's receptive field, defect recognition ability and normal background discrimination ability. Each training stage is equipped with an early stopping mechanism, which terminates training when the validation set loss does not decrease for several consecutive rounds to avoid overfitting of the YOLO11s model with small sample sizes.

[0038] In one specific implementation, the first cropping factor is 3, and the second cropping factor is 5. The first cropping factor image, the second cropping factor image, and the original image are represented by zoom_3, zoom_5, and zoom_full, respectively. In each stage of the dataset, defective samples and normal samples undergo synchronous cropping processing: in the zoom_3 and zoom_5 stages, normal samples are cropped based on the unified coordinate library of defective samples, performing cropping in the same region and at the same scale as the defective samples, ensuring that the receptive field and feature space of normal samples and defective samples are consistent. Each stage of the dataset generates a corresponding YOLO11s training YAML configuration file, configuring the single-class detection parameters. Phased training: Training is conducted in phases in the order of zoom_3→zoom_5→zoom_full, with the optimal weights of the previous phase used as the pre-training weights for the next phase. The training epochs for the three phases are set to 30, 25, and 15, respectively, with initial learning rates of 0.00015, 0.0001, and 0.0001. The AdamW optimizer is used, with a weight decay coefficient of 0.01, a cosine learning rate decay strategy, a dropout rate of 0.3, a label smoothing coefficient of 0.08, and an early stopping mechanism (patience=10) configured to avoid overfitting. The first phase of training freezes the weights of the first 10 layers of the YOLO11s backbone to prevent the destruction of pre-trained general features under small sample sizes and to accelerate model convergence.

[0039] In one embodiment, customized optimization is performed for a scenario involving single-class detection of congenital heart disease and mixed training of normal and defective samples. The total loss function of the YOLO11s base network is: , in, CIoU bounding box regression loss is used to constrain the defect localization accuracy; For classifying losses, This is a distributed focus loss used to improve the robustness of bounding box regression; , , These are the weighting coefficients for each loss term, which can be set to 7.0, 0.5, and 1.0.

[0040] Specifically, the classification loss adopts labeled smoothed binary cross-entropy loss, replacing the native multi-class cross-entropy loss of YOLO11s, to adapt to the binary classification scenario of single-class detection and the balanced learning requirements of normal and defective samples. , in, The categories predicted by the YOLO11s base network. The original hard labels for the samples. To predict the probability that this sample is defective for the YOLO11s base network, The label smoothing coefficient is set to 0.08 in this embodiment, and N is the total number of samples.

[0041] Optionally, after performing phased progressive training on the YOLO11s network, the following steps are also included: S22. Evaluate and optimize the trained YOLO11s network.

[0042] Specifically, step S22 includes: S221. Input the original images and TTA-processed images of each group in the test set into the trained YOLO11s network to obtain two independent prediction results. The input for each case simultaneously includes a two-dimensional grayscale ultrasound image and a color Doppler blood flow image corresponding to the same defect. The two types of images are input into the model as independent samples for detection. The model automatically learns the complementary features of the two modalities to improve detection accuracy. Specifically, the TTA processing can be horizontal flipping, that is, the two independent prediction results are obtained based on the original image and the horizontally flipped image, respectively.

[0043] S222. Normalize the two sets of prediction results, converting the pixel coordinates into relative coordinates in the range of 0 to 1.

[0044] S223. The WBF weighted fusion algorithm is used to fuse the two sets of normalized prediction results and obtain the fusion confidence.

[0045] For K predicted bounding boxes of the same target, the coordinates of the fused bounding box are: , , , ; The confidence level after fusion is: ,in, Let be the confidence score of the i-th predicted bounding box. Let be the bounding box coordinates of the i-th predicted box.

[0046] S224. Perform inverse normalization on the coordinates of the fused bounding box to convert it into absolute pixel coordinates of the original image. After filtering with a pre-set confidence threshold, output the final prediction result. Use the final prediction result to tune the parameters of the trained YOLO11s network.

[0047] If the input image includes gold standard annotations, the fused inference results can be simultaneously matched with the annotations to calculate core performance metrics such as recall, precision, and mAP, thus completing quantitative validation of the model's performance. After inference and validation are complete, the detection results can be visualized. Figure 2 This is a schematic diagram of echocardiography in patients with common types of congenital heart defects; Figure 3 A visualization example of the final prediction results.

[0048] In this embodiment, the defect detection performance of the trained model on the test set is shown in Table 1.

[0049] Table 1

[0050] S3. Perform the preprocessing operation on the image to be detected, including the two-dimensional grayscale ultrasound image and the color Doppler blood flow image, and then input it into the trained YOLO11s network to output the final defect detection result.

[0051] In one embodiment, a multimodal congenital heart disease ultrasound detection system is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multimodal congenital heart disease ultrasound detection method described above.

[0052] The multimodal congenital heart disease ultrasound detection system provided in this application adopts the same inventive concept as the above-mentioned multimodal congenital heart disease ultrasound detection method and can achieve the same beneficial effects, which will not be repeated here.

[0053] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the descriptions of the above embodiments are only for the purpose of helping to understand the methods of the embodiments of the present invention and should not be construed as limiting the embodiments of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of the present invention.

Claims

1. A multimodal ultrasound detection method for congenital heart disease, characterized in that, include: Defective samples are obtained and preprocessed, and normal samples are mixed in according to a preset ratio to divide the set into training set, test set and validation set. Both defective and normal samples included two-dimensional grayscale ultrasound images and color Doppler flow maps of the standard and non-standard sections of the subxiphoid biatrial heart. The original images in the training set are cropped according to a preset first cropping ratio and a second cropping ratio. The images are then input into the YOLO11s network in the order of the first cropping ratio, the second cropping ratio, and the original image. The YOLO11s network is trained in stages. The optimal weights obtained in the previous stage are used as the pre-training weights for the next stage. The size of the original image, the image at the second cropping ratio, and the image at the first cropping ratio are gradually reduced. The images to be detected, including two-dimensional grayscale ultrasound images and color Doppler blood flow images, are subjected to the preprocessing operation, and then input into the trained YOLO11s network to output the final defect detection results.

2. The method according to claim 1, characterized in that, The preprocessing operation includes: The image is normalized while preserving its aspect ratio, and boundary padding is performed. Offline enhancement is performed on the image according to a given trigger probability to transform the image pixel features; wherein, the offline enhancement is: brightness and contrast fine adjustment, Gamma correction or low-intensity Gaussian noise injection.

3. The method according to claim 2, characterized in that, The process of normalizing the image while preserving its aspect ratio and filling the boundaries includes: Two-dimensional grayscale ultrasound images are non-destructively converted into three-channel RGB images, while the original three-channel information of color Doppler flow maps is preserved. Scale the image proportionally, using the longest side of the target as the reference; the scaling ratio is... , Let h be the pixel value of the longest side of the target image, and h and w be the height and width of the original image, respectively. Apply symmetrical black borders to the scaled image so that both the width and height are integer multiples of 32.

4. The method according to claim 1, characterized in that, The step of cropping the original images of the training set according to a preset first cropping ratio and a second cropping ratio includes: Determine the size of the annotation box, and crop the width and height of the defect sample image to the first and second cropping ratios of the annotation box; Based on a unified coordinate library of defective samples, normal samples are cropped to the same region and size as defective samples.

5. The method according to claim 1, characterized in that, The total loss function of the YOLO11s basic network is: , in, For CIoU bounding box regression loss, For classifying losses, For distribution focus loss; , , These are the weighting coefficients for each loss term.

6. The method according to claim 5, characterized in that, The classification loss uses labeled, smoothed binary cross-entropy loss, specifically: ， in, The categories predicted by the YOLO11s base network. The original hard labels for the samples. To predict the probability that this sample is defective for the YOLO11s base network, is the label smoothing coefficient, and N is the total number of samples.

7. The method according to claim 4, characterized in that, After performing phased progressive training on the YOLO11s network, the process also includes evaluating and optimizing the trained YOLO11s network, including: The original images and TTA-processed images of each group in the test set were input into the trained YOLO11s network to obtain two independent prediction results; each group of original images included a two-dimensional grayscale ultrasound image and a color Doppler blood flow image of the same defect; The two sets of prediction results were normalized. The WBF weighted fusion algorithm is used to fuse the two sets of normalized prediction results and obtain the fusion confidence score. The coordinates of the fused bounding boxes are inversely normalized to convert them into absolute pixel coordinates of the original image. After filtering by a pre-set confidence threshold, the final prediction result is output. The final prediction result is then used to tune the parameters of the trained YOLO11s network.

8. A multimodal ultrasound detection system for congenital heart disease, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of any of the methods described in claims 1-7.

Images