Ultrasound echocardiogram standard section identification method based on acoustic plane, medium and equipment

Abstract

The application provides an echocardiogram standard section recognition device based on a sound plane, a medium and equipment; wherein the method is: obtaining an echocardiogram image sequence to be recognized; performing image frame extraction on the echocardiogram image sequence and performing image preprocessing; inputting the preprocessed image into a double-branch structure model based on the sound plane angle embedding of the echocardiogram frame by frame; the double-branch structure model performs global feature and local feature extraction processing on the image, and performs global feature and local feature fusion to obtain fusion features; the fusion features are input into a classifier to obtain prediction probability values of various categories; and the recognition result is obtained according to the prediction probability values of various categories. The method integrates the sound plane angle into the deep neural network, so that the model can pay attention to the sound plane angle information of the local area; and the extracted angle features are used to optimize the discrimination of the neural network classifier, thereby improving the performance of the standard section recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, more particularly to an echocardiogram standard section recognition device based on acoustic planes, medium and equipment. BACKGROUND

[0002] Cardiac ultrasound technology can intuitively reflect the anatomical structure and functional state of the heart by generating, emitting ultrasound waves and receiving echoes from the internal structure of the body to form dynamic echocardiograms, thereby providing real-time monitoring of dynamic changes in the heart for doctors. In recent years, it has gradually become one of the most widely used medical imaging methods in clinical diagnosis and intraoperative monitoring of heart disease. Cardiac ultrasound technology has become a core technology that doctors in the fields of cardiology, anesthesiology, and intensive care need to master. Determining the heart section through echocardiogram is the basis for mastering the echocardiogram. Only by correctly identifying the type of section can the physiological structure of the heart be distinguished, and disease diagnosis can be performed. According to clinical experience, medical experts have summarized echocardiogram sections corresponding to specific heart function structure positions, which are called standard sections.

[0003] With the gradual progress of computer technology, many works based on deep learning algorithms have studied the recognition of echocardiogram standard sections. However, most of the current research schemes for recognizing echocardiogram standard sections are relatively simple, and only serve as a common classification task, using deep neural networks to build classification algorithms for recognition.

[0004] The acoustic plane angle is the physical angle of the ultrasound probe during ultrasound monitoring. By facing the probe wafer at different angles, different echocardiogram sections of the heart function structure position can be imaged accordingly. Therefore, the echocardiogram standard section has a certain relationship with the acoustic plane angle. However, past research has not utilized this information. SUMMARY

[0005] To overcome the shortcomings and deficiencies in the prior art, the purpose of the present application is to provide an echocardiogram standard section recognition device based on acoustic planes, medium and equipment. This method integrates the acoustic plane angle into the deep neural network, so that the model can focus on the acoustic plane angle information in the local area. The extracted angle features are used to optimize the discrimination of the neural network classifier, thereby improving the performance of standard section recognition.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: an echocardiogram standard section recognition method based on acoustic planes, comprising the following steps:

[0007] S1, an echocardiogram image sequence to be recognized is obtained; image frame extraction is performed on the echocardiogram image sequence, and image preprocessing is performed;

[0008] S2, input the preprocessed image X G frame by frame into a double-branch structure model based on echocardiogram sound plane angle embedding; the double-branch structure model performs global feature and local feature extraction processing on the image X G , and performs global feature and local feature fusion to obtain fusion feature f T ; the fusion feature f T is input into a classifier to obtain prediction probability values of each category.

[0009] S3, according to the prediction probability values of each category, an identification result is obtained.

[0010] The double-branch structure model comprises a global branch and a local branch.

[0011] The global branch comprises modules B1, B2, B3, B4 and GAP1 connected in sequence; the structures of modules B1, B2, B3 and B4 are all multi-layer convolution-batch normalization-nonlinear activation units; module GAP1 is a global average pooling layer one for global feature vector extraction.

[0012] The local branch comprises modules S1, S2 and GAP2 connected in sequence; the structure of module S1 is two-layer convolution-batch normalization-nonlinear activation unit; the structure of module S2 is one-layer convolution-batch normalization-nonlinear activation unit; module GAP2 is a global average pooling layer two for local feature vector extraction.

[0013] S2 refers to: input the preprocessed image X G frame by frame into the global branch; the preprocessed image X G is sequentially processed by modules B1, B2, B3, B4 and GAP1 to obtain feature maps l1, l2, l3, l4 and a global feature vector f G .

[0014] The feature map l2 is cropped to obtain a sound plane angle region image X L ; the sound plane angle region image X L is sequentially processed by modules S1, S2 and GAP2 to finally obtain a local feature vector f L .

[0015] The global feature vector f G and the local feature vector f L are spliced to obtain fusion feature f T ; the fusion feature f T is input into a classifier to obtain prediction probability values of each category.

[0016] Preferably, the feature map l2 is cropped from the sound plane angle region image X L means that, from the top left corner of the feature map l2, the right and downward regions of the feature map l2 are obtained width and height, as the sound plane angle region image X L .

[0017] Preferably, in S1, the image preprocessing refers to unifying the image size of the echocardiogram image sequence.

[0018] Preferably, the global feature vector f G and the local feature vector f L are respectively:

[0019] f G =GAP1(B4(B3(B2(B1(X G )))

[0020] f L =GAP2(S2(S1(X L ))

[0021] The fusion feature f T is:

[0022] f T =Concat(f G , f L ).

[0023] Preferably, in S3, the identification result obtaining method is: taking the class with the maximum prediction probability value as the final prediction standard section class.

[0024] Preferably, the double-branch structure model refers to a double-branch structure model that has been trained and tested; the training and testing method of the double-branch structure model comprises the following steps:

[0025] Y1, collect samples of echocardiogram image sequences to form a data set, and divide the data set into a training set and a validation set;

[0026] Y2, image preprocessing is performed on the training set, and the training set is augmented by using random flipping, translation, cropping, brightness and contrast variation;

[0027] Y3, input the preprocessed images in the training set into the double-branch structure model to obtain prediction probability values of each class;

[0028] Y4, use the cross-entropy loss function CE to calculate the loss between the classification label y and the prediction probability value p obtained by the double-branch structure model, to obtain the loss value L:

[0029] L = CE(y, p)

[0030] Using the loss value L, the double-branch structure model is back-propagated to calculate the gradient of each parameter in the double-branch structure model, and the new value of each parameter is calculated by the optimizer and updated;

[0031] Y5, Y3 and Y4 are repeatedly executed until the double-branch structure model converges;

[0032] Y6, after the double-branch structure model converges, the performance of the double-branch structure model is evaluated on the validation set, Y2 to Y5 are repeatedly executed by modifying the hyperparameters, and the performance optimization of the double-branch structure model is performed.

[0033] A readable storage medium, wherein the storage medium stores a computer program, the computer program causes the processor to execute the above-mentioned echocardiogram standard section recognition method based on the sound plane.

[0034] A computer device, comprising a processor and a memory for storing a program executable by the processor, when the processor executes the program stored in the memory, the above-mentioned echocardiogram standard section recognition method based on the sound plane is realized.

[0035] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0036] The present application integrates the sound plane angle into the deep neural network, uses the powerful feature extraction capability of the deep neural network for feature extraction, so that the model can pay attention to the sound plane angle information of the local area, and the extracted angle feature is used to optimize the discrimination of the neural network classifier, thereby improving the performance of the standard section recognition. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a structural schematic diagram of the double-branch structure model in the present application;

[0038] Figure 2 is a module structure schematic diagram of the double-branch structure model in the present application;

[0039] Figure 3 is a processing flow schematic diagram of the double-branch structure model in the present application;

[0040] Figure 4 is the attention heat map result without using the echocardiogram standard section recognition method of the present application and the attention heat map result using the echocardiogram standard section recognition method of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0042] Example one

[0043] The embodiment of the application is a standard section recognition method for echocardiogram based on acoustic plane, comprising the following steps:

[0044] S1, obtaining an echocardiogram image sequence to be recognized; performing image frame extraction on the echocardiogram image sequence and performing image preprocessing; the image preprocessing refers to unifying the image size of the echocardiogram image sequence.

[0045] S2, inputting the preprocessed image X G frame by frame into a double-branch structure model based on the acoustic plane angle embedding of the echocardiogram; the double-branch structure model performs global feature and local feature extraction processing on the image X G , and performs global feature and local feature fusion to obtain fusion features f T . T The fusion features f G are input into a classifier to obtain prediction probability values of each category.

[0046] The double-branch structure model comprises a global branch and a local branch, as shown in Figure 1 .

[0047] The global branch comprises modules B1, B2, B3, B4 and GAP1 connected in sequence; the structures of modules B1, B2, B3 and B4 are all multi-layer convolution-batch normalization-nonlinear activation units; module GAP1 is a global average pooling layer one for global feature vector extraction, as shown in Figure 2 .

[0048] The local branch comprises modules S1, S2 and GAP2 connected in sequence; the structure of module S1 is two-layer convolution-batch normalization-nonlinear activation units; the structure of module S2 is one-layer convolution-batch normalization-nonlinear activation units; module GAP2 is a global average pooling layer two for local feature vector extraction.

[0049] The S2 refers to inputting the preprocessed image X G frame by frame into the global branch; the preprocessed image X G is sequentially processed by modules B1, B2, B3, B4 and GAP1 to obtain feature maps l1, l2, l3, l4 and a global feature vector f G , respectively.

[0050] Modules B1-B4 are feature extraction modules. By stacking these modules, the model can extract rich feature representations at a deep level. These feature representations contain abstract information of the input image. The function of module GAP1 is to average the feature maps of each feature channel and reduce the dimensionality of the high-dimensional feature map to a scalar value in each feature channel, thereby reducing the dimensionality of the feature map.

[0051] f G =GAP1(B4(B3(B2(B1(X) G )))))

[0052] The acoustic plane angle region image X is cropped from feature map l2. L Specifically, starting from the upper left vertex of feature map l2, the features map l2 is obtained by moving to the right and downwards. Width and The region of height, as the angular region of the acoustic plane, is shown in the image X. L .

[0053] Acoustic plane angular region image X L After being processed sequentially by modules S1, S2, and GAP2, the local feature vector f is finally obtained. L ;

[0054] f L =GAP2(S2(S1(X) L ))).

[0055] The design concept of the dual-branch structure model is as follows: the global branch is a complete echocardiogram standard section recognition network; stacking a certain number of convolutional-batch normalization-nonlinear activation units helps improve the network's feature extraction capability, thereby improving classification performance. The local branches serve only as auxiliary branches to the global branch, and setting a shallow depth minimizes the amount of additional parameters and computational complexity. The input size of the local branches is small; stacking too many convolutional-batch normalization-nonlinear activation units would cause the feature map at the branch's end to become too small, failing to effectively capture important features in the input image. The input image of the local branches only contains information about the acoustic plane angle region, with relatively simple semantics, requiring only a small number of feature extraction modules, making excessive stacking unnecessary.

[0056] The global branch output feature map l2 of the middle layer is used as the input of the local branch, and the global branch and the local branch share part of the parameters, which can help the model learn more general features from the two tasks. In a convolutional neural network, the shallow feature extraction module is mainly responsible for extracting low-level features of the image, such as texture, contour, and other information. The deep feature extraction module extracts more abstract and high-level features. In echocardiogram recognition, texture information is more important. Therefore, it is more appropriate to use the output feature map of the shallow layer as the input of the local branch. Experiments have verified that using feature map l2 is the optimal choice.

[0057] The global feature vector f G and the local feature vector f L are spliced to obtain the fusion feature f T .

[0058] f T = Concat(f G , f L )

[0059] The fusion feature f T is input into the classifier to obtain the prediction probability value of each category.

[0060] S3, according to the prediction probability value of each category, obtain the recognition result: the category with the maximum prediction probability value is taken as the final prediction standard cross-section category.

[0061] The double-branch structure model refers to the trained and tested double-branch structure model; the training and testing method of the double-branch structure model comprises the following steps:

[0062] Y1, collect samples of echocardiogram image sequences to form a data set, and divide it into a training set and a validation set;

[0063] Y2, image preprocessing is performed on the training set, and the training set is expanded by using random flipping, translation, cropping, brightness and contrast change;

[0064] Y3, input the images in the preprocessed training set into the double-branch structure model to obtain the prediction probability value of each category;

[0065] Y4, use the cross-entropy loss function CE to calculate the loss of the classification label y and the prediction probability value p obtained by the double-branch structure model to obtain the loss value L:

[0066] L = CE(y, p)

[0067] Using the loss value L, the double-branch structure model is back propagated to calculate the gradient of each parameter in the double-branch structure model, and the new value of each parameter is calculated by the optimizer and updated;

[0068] Y5, repeatedly performing Y3 and Y4 until the double-branch structure model converges;

[0069] Y6, after the double-branch structure model converges, evaluating the performance of the double-branch structure model on the validation set, and repeatedly performing Y2 to Y5 to optimize the performance of the double-branch structure model by modifying the hyperparameters.

[0070] From the test indicators, the accuracy, recall rate, and F1 score of the deep neural network for recognizing the standard section of the echocardiogram are improved.

[0071] The feature activation maps before and after introducing the local branch to the double-branch structure model are visualized, as shown in FIG. 6. Figure 4 Figure 4 In FIG. 6, the left column is the original image, the middle column is the attention heat map without using the local branch and only using the global branch, and the right column is the attention heat map using the double-branch structure model. After comparison, it can be found that the attention weight of the acoustic plane angle region in the upper left corner is significantly improved when using the double-branch structure model of the application, which indicates that the model pays more attention to the acoustic plane angle region in the upper left corner of the image, which verifies the effectiveness of the introduction of the local branch.

[0072] Discriminating the section of the echocardiogram is a necessary step for clinical diagnosis and a basis for mastering the echocardiogram. Only by correctly discriminating the section type can the physiological structure of the heart, such as the left and right atria, left and right ventricles, mitral valve, and tricuspid valve, be distinguished, so as to further complete the evaluation of the function of the heart and diagnose whether there are abnormal conditions such as valvular disease, intracardiac tumor, and intracardiac thrombus.

[0073] The automatic classification of the standard section of the echocardiogram is realized, and after the algorithm model is deployed, it can be used as a clinical auxiliary tool for clinical ultrasonic examination, improving the accuracy of diagnosis and reducing the burden of clinicians. It can also be used in the teaching work of echocardiogram to help improve the teaching effect, shorten the training period, and alleviate the shortage of medical imaging talents.

[0074] Embodiment Two

[0075] The readable storage medium of the embodiment stores a computer program, and when the computer program is executed by a processor, the processor executes the method for recognizing the standard section of the echocardiogram based on the acoustic plane as described in embodiment one.

[0076] Embodiment Three

[0077] The computer device of the embodiment includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the method for recognizing the standard section of the echocardiogram based on the acoustic plane as described in embodiment one is implemented.

[0078] ​The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. An echocardiogram standard plane identification method based on acoustic planes, characterized in that: It comprises the following steps: S1, obtaining an echocardiogram image sequence to be identified; image frame extraction is performed on the echocardiogram image sequence, and image preprocessing is performed; S2, input the preprocessed image X G frame by frame into a double-branch structure model based on echocardiogram sound plane angle embedding; the double-branch structure model performs global feature and local feature extraction processing on the image X G , and performs global feature and local feature fusion to obtain fusion features f T , the fusion features f T are input into a classifier to obtain prediction probability values of each category; S3, obtaining an identification result according to the prediction probability values of each category; The double-branch structure model comprises a global branch and a local branch; The global branch comprises modules B1, B2, B3, B4 and GAP1 connected in sequence; the structures of modules B1, B2, B3 and B4 are all multi-layer convolution-batch normalization-nonlinear activation units; module GAP1 is a first global average pooling layer for global feature vector extraction; The local branch comprises modules S1, S2 and GAP2 connected in sequence; the structure of module S1 is two-layer convolution-batch normalization-nonlinear activation units; the structure of module S2 is one-layer convolution-batch normalization-nonlinear activation units; module GAP2 is a second global average pooling layer for local feature vector extraction; S2, refers to: the pre-processed image X G frame by frame input to the global branch; the pre-processed image X G Through module B1, module B2, module B3, module B4 and module GAP1 in turn, respectively get feature map l1, feature map l2, feature map l3, feature map l4 and global feature vector f G ; Crop the feature map l2 to the sound plane angle region image X l , the sound plane angle region image X l After sequentially processing through the module S1, the module S2 and the module GAP2, the local feature vector f L is finally obtained. The global feature vector f G and the local feature vector f L are spliced to obtain a fusion feature f T ; the fusion feature f T is input into a classifier to obtain a prediction probability value of each category; The feature map l2 is cropped to the sound plane angle region image X L means that, from the top left vertex of the feature map l2, the right and downward regions of the feature map l2 are obtained width and height, as the sound plane angle region image X L .

2. The acoustic plane based echocardiogram standard view identification method of claim 1, wherein: In S1, the image preprocessing refers to uniforming the image size of the echocardiogram image sequence.

3. The acoustic plane based echocardiogram standard view identification method of claim 1, wherein: The global feature vector f G and the local feature vector f L are respectively: f G = GAP1(B4(B3(B2(B1(X G )))))) f L = GAP2(S2(S1(X L ))) The fusion feature f T is: f T = Concat(f G ,f L ).

4. The acoustic plane based echocardiogram standard view identification method of claim 1, wherein: In S3, the identification result is obtained by taking the category with the maximum prediction probability value as the final prediction standard section category.

5. The acoustic plane based echocardiogram standard view identification method of claim 1, wherein: The double-branch structure model refers to a double-branch structure model that has been trained and tested; the training and testing method of the double-branch structure model comprises the following steps: Y1, collecting samples of echocardiogram image sequences to form a data set, and dividing the data set into a training set and a validation set; Y2, performing image preprocessing on the training set, and expanding the training set by using random flipping, translation, cropping and brightness contrast changes; Y3, inputting the images in the preprocessed training set into the double-branch structure model to obtain prediction probability values of each category; Y4, using a cross-entropy loss function CE to calculate the loss of the classification label y and the prediction probability value p obtained by the double-branch structure model to obtain a loss value L: L=CE(y,p) Using the loss value L, the double-branch structure model is back-propagated to calculate the gradient of each parameter in the double-branch structure model, and the new value of each parameter is calculated by an optimizer and updated; Y5, repeating Y3 and Y4 until the double-branch structure model converges; Y6, after the double-branch structure model converges, evaluating the performance of the double-branch structure model on the validation set, modifying the hyperparameters, and repeating Y2 to Y5 to optimize the performance of the double-branch structure model.

6. A readable storage medium characterized by, The storage medium stores a computer program, and the computer program causes the processor to execute the echocardiogram standard section identification method based on the acoustic plane according to any one of claims 1-5 when executed by the processor.

7. A computer device comprising a processor and a memory for storing a processor executable program, characterized in that, The processor executes the program stored in the memory to implement the echocardiogram standard section identification method based on the acoustic plane according to any one of claims 1-5.

Images