Method and device for identifying left heart development characteristics of fetal heart ultrasound

Abstract

The present invention relates to the field of image recognition technology, and particularly to a method and device for identifying left heart development features in fetal heart ultrasound. The method comprises acquiring historical ultrasound images and manually annotating them, dividing the historical images into salient images and non-salient images; constructing a deep neural network, constructing 4D ultrasound images as input to the deep neural network, and training the deep neural network using the historical images to obtain a binary classification deep neural network model for identifying whether an image is a salient image or a non-salient image; constructing image recognizers for salient images and non-salient images respectively, obtaining a focus area of the image through weakly supervised positioning in the image detector, and obtaining multiple detection frames through image detection; and using the detection frame closest to the focus area as the left heart development feature position of the image. The present invention can realize low-cost, fast, simple, and accurate identification and positioning of the left heart lesion area specified by fetal heart ultrasound.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for recognizing fetal cardiac ultrasound left heart development characteristics. Background Art

[0002] Currently, fetal cardiac ultrasound examinations are primarily performed manually, and the method of identifying specific left heart lesions, including areas of left heart hypoplasia, relies heavily on physicians.

[0003] However, manual operation obviously cannot meet the needs of quickly, simply and widely identifying and locating the fetal heart ultrasound designated left heart lesion area, including the left heart hypoplasia area.

[0004] Therefore, providing a system and method for quickly, simply and widely identifying and locating the fetal heart ultrasound left heart development characteristics is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] In order to quickly, simply, and accurately identify and locate a designated left heart lesion area on fetal cardiac ultrasound, the present invention proposes a method and device for identifying left heart developmental characteristics on fetal cardiac ultrasound. The method specifically comprises the following steps:

[0006] Obtain historical ultrasound images and manually annotate them, dividing them into significant images and non-significant images;

[0007] A deep neural network, ResNet-50, was constructed. A 4D ultrasound image was used as input for ResNet-50. The ResNet-50 was trained using historical images to obtain a binary classification deep neural network model for identifying whether an image is salient or non-salient.

[0008] Image recognizers are constructed for salient images and non-salient images respectively. In the image detector, the focus area of the image is obtained through weak supervision positioning, and multiple detection boxes are obtained through image detection.

[0009] The detection box closest to the region of interest is used as the left heart development feature position of the image.

[0010] Furthermore, the 4D ultrasound image is represented as:

[0011]

[0012] Where V represents a 4D ultrasound image, v t,s represents the s-th slice of the ultrasound image of size H×W at time t; It represents a set of T moments and S slices; H represents the height of the ultrasound image; and W represents the width of the ultrasound image.

[0013] Furthermore, the prediction result obtained by the image recognizer is expressed as:

[0014]

[0015] Among them, B out W total is the total importance weight of fetal heart ultrasound image, B pre is the best predicted bounding box; t represents the number of output predicted bounding boxes from the refined detection network.

[0016] Furthermore, the total importance weight W of the fetal heart ultrasound image is total The calculation includes:

[0017]

[0018]

[0019] Among them, m k represents the importance weight of the kth feature map, Y represents the number of classifications, represents the kth feature map, λ is the normalization parameter, W represents the width of the ultrasound image, and H represents the height of the ultrasound image.

[0020] Furthermore, the number of categories Y is expressed as:

[0021]

[0022] Among them, δ k represents the feature weight of the kth feature map, λ is the normalization parameter, represents the pixel value of the pixel in the i-th row and j-th column in the k-th feature map, W represents the width of the ultrasound image, and H represents the height of the ultrasound image.

[0023] Furthermore, the best predicted bounding box B pre Expressed as:

[0024]

[0025] Where N represents the number of feature maps; represents the kth feature map, B k represents the k-th feature map prediction box, Θ(,) represents the best prediction box calculation function.

[0026] Furthermore, the kth feature map express:

[0027]

[0028] Among them, v t,srepresents the s-th slice of the ultrasound image of size H×W at time t, Indicates v t,s The classification result of , Γ(,) represents the classification network function.

[0029] The present invention also proposes a device for identifying fetal heart ultrasound left heart development characteristics, which is used to implement a method for identifying fetal heart ultrasound left heart development characteristics, including an image coarse classification module and an image precise identification and positioning module, wherein:

[0030] The image coarse classification module is used to coarsely classify the fetal heart ultrasound images to be identified and located, and obtain ultrasound images with significant locations to be identified and located and ultrasound images with non-significant locations to be identified;

[0031] The image precise recognition and positioning module includes an image detection module and an image weak supervision positioning optimization module; the image detection module is used to identify and locate the fetal heart development image area in the ultrasound image that is significantly to be identified and positioned and the ultrasound image that is not significantly to be identified and positioned, and output multiple detection frames; the image weak supervision positioning optimization module is used to optimize the multiple detection frames output by the image detection module and output a unique detection frame.

[0032] Compared with the existing technology, the present invention discloses a system and method for identifying and locating the left heart development characteristics of fetal heart ultrasound based on dual stages (coarse classification and fine detection) and weakly supervised positioning, which can achieve the beneficial technical effect of low cost, fast, simple and accurate identification and positioning of the left heart lesion area specified by fetal heart ultrasound, including the left heart hypoplastic area. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of an embodiment of the present invention;

[0034] Figure 2 3. The three types of ultrasound image characterization features in one embodiment of the present invention are shown, wherein Figure (a) is a significantly normal heart image, Figure (b) is an image significantly containing a lesion, and Figure (c) is an image not significantly containing a lesion.

[0035] Figure 3 This is a schematic diagram of a two-stage detection embodiment of the present invention;

[0036] Figure 4 Schematic diagram of a fine detection module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] The present invention provides a method for identifying fetal left heart development characteristics by ultrasound, which specifically comprises the following steps:

[0039] Obtain historical ultrasound images and manually annotate them, dividing them into significant images and non-significant images;

[0040] A deep neural network, ResNet-50, was constructed. A 4D ultrasound image was used as input for ResNet-50. The ResNet-50 was trained using historical images to obtain a binary classification deep neural network model for identifying whether an image is salient or non-salient.

[0041] Image recognizers are constructed for salient images and non-salient images respectively. In the image detector, the focus area of the image is obtained through weak supervision positioning, and multiple detection boxes are obtained through image detection.

[0042] The detection box closest to the region of interest is used as the left heart development feature position of the image.

[0043] like Figure 1 In this embodiment, the images to be detected are classified into ultrasound images with significant locations to be identified and ultrasound images with non-significant locations to be identified, and corresponding detection models are used for detection to determine whether they contain areas of left heart hypoplasia. The detection models corresponding to the ultrasound images with significant locations to be identified and the ultrasound images with non-significant locations to be identified are trained using manually labeled significant ultrasound images and non-significant ultrasound images, respectively.

[0044] like Figure 3 The present invention mainly includes two detection stages. The first stage detects whether the image is a salient image or a non-salient image ( Figure 3 In this example, class1 and class2 represent salient images and non-salient images, respectively. In this example, ResNet-50 is used for classification and recognition, and manually labeled images are used for training. After the image is identified as a salient image or a non-salient image, the image is input into the corresponding recognition and positioning model, such as Figure 3 In this embodiment, the YOLO detection model is used for positioning and recognition. Three images of different sizes are input into the YOLO detection model to obtain three detection results. The final detection result is determined by weighted summation based on the three detection results and marked with a detection box. Figure 4 A specific detection process is given. After the image is input into the YOLO detection model, the output image has a probability of 0.95 that the image is a significantly normal heart image (Nomal), a probability of 0.98 that the image is an image with a significant lesion (HLHS-1), a probability of 0.81 that the image is an image with no significant lesion (HLHS-2), and a probability of not belonging to any of the above (Others).

[0045] In this embodiment, a method for identifying and locating fetal left heart development characteristics using ultrasound includes the following steps:

[0046] Step 1: Classify and label historical fetal heart ultrasound images

[0047] During labeling, only the bounding box label is used to cover the fetal heart ROI area.

[0048] In this embodiment, since the present invention focuses on classification and detection tasks, the direction is not considered when labeling images. Faced with a large amount of historical fetal heart ultrasound image training data, the simple image labeling method of the present invention can significantly reduce labor costs and improve the efficiency of classification and labeling of training data.

[0049] Step 2: Use the annotated historical fetal heart ultrasound images to train a binary classification deep neural network

[0050] A large number of 4D fetal heart ultrasound historical images from different times and sections were manually classified into two categories: significant historical ultrasound images and non-significant historical ultrasound images. The deep neural network ResNet-50 was trained using the manually classified significant and non-significant historical ultrasound images to ultimately obtain a binary classification deep neural network model.

[0051] The significant historical ultrasound images are input into a deep neural network YOLO for training to obtain a significant image detection model, and the non-significant historical ultrasound images are input into another deep neural network YOLO for training to obtain a non-significant image detection model. The present invention ultimately obtains two independent YOLO detection models.

[0052] In this embodiment, the image detection model includes a salient image detection model and a non-salient image detection model; the salient image detection model is used to detect the salient ultrasound image to be identified and located; the non-salient image detection model is used to detect the non-salient ultrasound image to be identified and located.

[0053] Step 3: Input the fetal heart ultrasound image to be identified and located into the two-class deep neural network model for coarse classification to obtain ultrasound images with significant locations to be identified and ultrasound images with non-significant locations to be identified.

[0054] The binary classification deep neural network model designed by the present invention helps to reduce the intra-class distance, expand the inter-class distance, and reduce the false detection rate of subsequent detection.

[0055] Step 4: Input the ultrasound image with significant location to be identified and the ultrasound image with non-significant location to be identified into the image detection model for identification and positioning, and output multiple detection frames.

[0056] The salient ultrasound images to be identified and located after rough classification are input into the salient image detection model, and the non-salient ultrasound images to be identified and located are input into the non-salient image detection model.

[0057] Step 5: Perform weakly supervised positioning optimization on the multiple detection frames output by the image detection model and output a unique detection frame

[0058] Weakly supervised positioning optimization includes correcting false positives on multiple detection frames output by the salient image detection model and the non-salient image detection model and outputting a unique detection frame;

[0059] Alternatively, the weakly supervised positioning optimization includes performing gradient calculation on the multi-scale feature maps generated by the salient image detection model and the non-salient image detection model and obtaining an importance distribution matrix, and outputting a unique detection box according to the importance distribution matrix.

[0060] Specifically, the weakly supervised positioning optimization method involves performing gradient calculations on the multi-scale feature maps generated by the salient image detection model and the non-salient image detection model to obtain an importance weight matrix on the feature maps, automatically capturing the image's focus area, and performing a closest distance calculation between the focus area and multiple detection areas output by the salient image detection model and the non-salient image detection model to ultimately obtain a unique detection positioning frame. This invention adds a weakly supervised positioning optimization function to the image detection model, effectively reducing the output of false positives caused by the maximum suppression threshold setting, ensuring that the final output frame is correct and unique.

[0061] Compared with the existing technology, the present invention adopts a deep learning method to detect fetuses with the potential to develop hypoplastic left heart (HLHS); at the same time, the present invention aims at the needs of rapid detection and low-cost labeling. This paper adopts YOLO as the backbone detection network. However, directly using the YOLO network has problems of missed detection and multiple detection in ultrasound detection. We found that this is mainly caused by the fact that the NMS in YOLO only sets a fixed threshold. Therefore, we optimize the NMS in the YOLO network and reduce the missed detection and multiple detection by adding CAM (class activation map). In addition, in model training, the present invention divides the three types of images into two categories of images for independent training. This training method can also improve the detection rate in complex ultrasound images with high noise.

[0062] In this embodiment, the binary classification network directly uses ResNet-50. Based on the adopted network, the present invention processes the image and defines a 4D ultrasound image, which is expressed as:

[0063]

[0064] Where V represents a 4D ultrasound image, v t,s represents the s-th slice of the ultrasound image of size H×W at time t; It represents a set of T moments and S slices; H represents the height of the ultrasound image; and W represents the width of the ultrasound image.

[0065] The binary classification network is defined as follows:

[0066]

[0067] Among them, Φ Binary represents the classification function, which refers to the classification function of resnet-50; C1 indicates that the significant images include normal heart images (e.g. Figure 2 (a)) and HLHS-1 type (i.e., images that significantly contain lesions, e.g. Figure 2 (b)), C2 represents the HLHS-2 type (i.e., images that do not contain lesions, e.g. Figure 2 (c)).

[0068] This embodiment optimizes the NMS mechanism of the YOLO network in the detection network, enabling it to achieve good detection and positioning even in high-noise, single-color images like ultrasound. The refined detection network uses YOLO as its backbone network, outputting detection results using pre-distorted labels and bounding boxes. This embodiment uses a basic focal loss function as the loss function. To achieve high-precision positioning of cardiac lesions, this embodiment introduces weakly supervised positioning based on CAM, which optimizes the pre-distorted bounding boxes.

[0069] The fine detection network can output feature maps of three scales. In this embodiment, the output feature maps are defined as follows:

[0070]

[0071] in, Indicates Category v t,s The kth feature map of Γ(,) represents the classification network function in the YOLO network, which is based on the input v t,s and its classification results determine the output feature map of the resnet-50 network.

[0072] Each feature map will generate a predicted bounding box based on the NMS threshold, and the network's best predicted bounding box can be given by the following formula:

[0073]

[0074] Wherein, N represents the number of feature maps, and in this embodiment, bit 3 is set; B k The prediction box of the kth feature map; Θ(,) represents the optimal prediction box calculation function, that is, the NMS prediction box selection function, and the existing NMS prediction box selection function can be used.

[0075] Because NMS is a simple bounding box selection strategy using a threshold, this can lead to misjudged bounding boxes. Therefore, this embodiment introduces weakly supervised localization to enhance the selection strategy. The method of this embodiment calculates the importance weights of three feature maps at different scales and obtains the final output predistorted bounding box based on the importance weight of each feature map.

[0076] To calculate the importance weight, this example first gives the gradient of the class score Y:

[0077]

[0078] Among them, δ k represents the feature weight of the k-th feature map (k = {1, 2, 3}, representing the three feature maps output by the YOLO network respectively), and λ represents the proportional constant normalized during the visualization process.

[0079] The importance weight on the k-th feature map can be given by:

[0080]

[0081] The total importance weight of the fetal heart ultrasound image can be given by:

[0082]

[0083] Among them, m k The weight representing the importance weight of the k-th feature map. In the method of this embodiment, m3=1, m1=0, and m2=0 are set.

[0084] Finally, we can get the following final predicted bounding box:

[0085]

[0086] Where t represents the number of output predicted bounding boxes from the refined detection network.

[0087] The present invention also provides a device for identifying fetal heart ultrasound left heart development characteristics, which is used to implement a method for identifying fetal heart ultrasound left heart development characteristics, including an image coarse classification module and an image precise identification and positioning module, wherein:

[0088] The image coarse classification module is used to coarsely classify the fetal heart ultrasound images to be identified and located, and obtain ultrasound images with significant locations to be identified and located and ultrasound images with non-significant locations to be identified;

[0089] The image precise recognition and positioning module includes an image detection module and an image weak supervision positioning optimization module; the image detection module is used to identify and locate the fetal heart development image area in the ultrasound image that is significantly to be identified and positioned and the ultrasound image that is not significantly to be identified and positioned, and output multiple detection frames; the image weak supervision positioning optimization module is used to optimize the multiple detection frames output by the image detection module and output a unique detection frame.

[0090] In this embodiment, the ultrasound images with significant locations to be identified include ultrasound images with significant normal locations to be identified and ultrasound images with significant left heart hypoplasia to be identified; and the ultrasound images with non-significant locations to be identified include ultrasound images with non-significant left heart hypoplasia to be identified.

[0091] Specifically, a large number of 4-dimensional fetal heart ultrasound historical images from different times and sections were manually classified into two categories: significant historical ultrasound images and insignificant historical ultrasound images. The manually classified significant and insignificant historical ultrasound images were then used to train a deep neural network ResNet-50, ultimately obtaining a binary classification deep neural network model. This binary classification deep neural network model was used to perform a coarse classification of fetal heart ultrasound images to be identified and located, ultimately classifying the images into two categories: significant ultrasound images to be identified and located, and insignificant ultrasound images to be identified and located. The binary classification deep learning neural network model designed by the present invention helps to reduce the intra-class distance, expand the inter-class distance, and reduce the false detection rate of subsequent detection.

[0092] The present invention sets the fetal heart development image region as the image region of left heart hypoplasia. The fetal heart of left heart hypoplasia usually presents symptoms of aortic stenosis, which causes the left ventricle (LV) to be spherical and the left ventricular wall thickness to increase.

[0093] Specifically, in the image precision recognition and positioning module, a large number of 4D fetal heart ultrasound historical images from different times and sections are manually classified into two categories: significant historical ultrasound images and non-significant historical ultrasound images. The significant historical ultrasound images are then input into a deep neural network YOLO for training to obtain a significant image detection model, while the non-significant historical ultrasound images are input into another deep neural network YOLO for training to obtain a non-significant image detection model. The present invention ultimately obtains two independent YOLO detection models. The coarsely classified significant ultrasound images to be identified and located are input into the significant image detection model, and the non-significant ultrasound images to be identified and located are input into the non-significant image detection model. The image weakly supervised localization module performs gradient calculations on the multi-scale feature maps generated by the significant image detection model and the non-significant image detection model to obtain the importance weight matrix on the feature map, automatically capturing the image's focus area. The focus area is then used to perform the nearest distance calculation with multiple detection areas output by the significant image detection model and the non-significant image detection model to ultimately obtain a unique detection and positioning frame. The present invention adds a weakly supervised localization optimization function to the image detection model, which can effectively reduce the output of false detection frames caused by the maximum suppression threshold setting, ensuring that the final output frame is correct and unique.

[0094] In this embodiment, images are annotated using only bounding box labels covering the fetal heart ROI. Because this invention focuses on classification and detection tasks, orientation is not considered when labeling images; only the bounding box labels covering the fetal heart ROI are sufficient. Given a large amount of historical fetal heart ultrasound image training data, this simple image labeling method can significantly reduce labor costs and improve the efficiency of training data classification and annotation.

[0095] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying fetal left heart development characteristics by ultrasound, characterized in that: The specific steps include: Obtain historical ultrasound images and manually annotate them, dividing them into significant images and non-significant images; A deep neural network ResNet-50 was constructed, and a 4D ultrasound image was constructed as the input of ResNet-50. ResNet-50 was trained using historical images to obtain a binary classification deep neural network model for identifying whether an image is a salient image or a non-salient image. The 4D ultrasound image is represented as: Where V represents a 4D ultrasound image, v t,s represents the s-th slice of the ultrasound image of size H×W at time t; represents a set of T time moments and S slices; H represents the height of the ultrasound image; W represents the width of the ultrasound image; Image recognizers are constructed for salient images and non-salient images respectively. The image detector obtains the focus area of the image through weak supervision positioning, and obtains multiple detection boxes through image detection. The prediction results obtained by the image recognizer are expressed as: Among them, B pre is the best predicted bounding box; t represents the number of output predicted bounding boxes from the fine detection network; W total is the total importance weight of the fetal heart ultrasound image, and its calculation process includes: Among them, m k represents the importance weight of the k-th feature map, represents the kth feature map, λ is the normalization parameter, W represents the width of the ultrasound image, H represents the height of the ultrasound image; Y represents the number of classifications, which is expressed as: Among them, δ k represents the feature weight of the kth feature map, λ is the normalization parameter, represents the pixel value of the pixel in the i-th row and j-th column of the k-th feature map, W represents the width of the ultrasound image, and H represents the height of the ultrasound image; The detection box closest to the region of interest is used as the left heart development feature position of the image.

2. The method for identifying fetal left heart development characteristics by ultrasound according to claim 1, characterized in that: The best predicted bounding box B pre Expressed as: Where N represents the number of feature maps; represents the kth feature map, B k represents the k-th feature map prediction box, Θ(,) represents the best prediction box calculation function.

3. The method for identifying fetal left heart development characteristics by ultrasound according to claim 1, characterized in that: kth feature map express: Among them, v t,s represents the s-th slice of the ultrasound image of size H×W at time t, Indicates v t,s The classification result of , Γ(,) represents the classification network function.

4. A device for identifying fetal left heart development characteristics using ultrasound, characterized in that: The method for realizing the recognition of fetal heart ultrasound left heart development characteristics as claimed in claim 1 comprises an image coarse classification module and an image precise recognition and positioning module, wherein: The image coarse classification module is used to coarsely classify the fetal heart ultrasound images to be identified and located, and obtain ultrasound images with significant locations to be identified and located and ultrasound images with non-significant locations to be identified; The image precise recognition and positioning module includes an image detection module and an image weak supervision positioning optimization module; the image detection module is used to identify and locate the fetal heart development image area in the ultrasound image that is significantly to be identified and positioned and the ultrasound image that is not significantly to be identified and positioned, and output multiple detection frames; the image weak supervision positioning optimization module is used to optimize the multiple detection frames output by the image detection module and output a unique detection frame.

Images