A lightweight echocardiographic standard section recognition method, device and medium

Abstract

The present invention relates to a lightweight echocardiogram standard section recognition method, device and medium, and the method comprises the following steps: S1: collecting data of two-dimensional echocardiogram standard sections to form a data set; S2: classifying the data set and dividing the data set into a training set, a test set and a validation set; S3: preprocessing and data enhancement of the data set; S4: building a deep learning network model as a training model, and training the training model with the training set; S5: verifying the training model with the validation set, adjusting the hyperparameters of the training model according to the evaluation results, and testing the generalization ability of the training model with the test set to obtain an echocardiogram classification model; S6: inputting the echocardiogram into the trained echocardiogram classification model to obtain an echocardiogram classification recognition result. By using a Mobilenet‑Echo network structure, based on a lightweight network design, the model parameters are reduced. It belongs to the field of detection technology.

Description

Technical Field

[0001] The invention relates to the field of detection technology, and in particular to a lightweight echocardiogram standard section recognition method, device and medium. Background Art

[0002] At present, heart disease not only has a significant impact on the quality of human life, but also threatens human health. Therefore, early diagnosis and treatment of heart disease is crucial. In recent years, with the continuous advancement of computer technology and artificial intelligence medical imaging technology, artificial intelligence medical imaging has gradually developed from an auxiliary examination method to the most important clinical diagnosis and differential diagnosis method in modern medicine.

[0003] Echocardiography can intuitively display and quantitatively measure important information about the heart's anatomical structure, function, and hemodynamics, thereby reflecting the cardiac configuration characteristics under physiological or pathological conditions. It is the main medical imaging method for diagnosing heart disease, and accurate parameter measurement is an important part of echocardiographic diagnosis.

[0004] In clinical examinations, doctors first need to find the area of ​​interest in the heart to measure parameters, but the heart structure is complex, and each patient's echocardiogram has many standard sections, such as apical two-chamber (A2C), apical three-chamber (A3C), apical four-chamber (A4C), short axis of the great arteries (PSA), parasternal long axis (PLA) and other standard echocardiogram section images. How to accurately identify echocardiogram images has become a time-consuming and labor-intensive task, which not only increases the doctor's workload, but also may cause inaccurate diagnosis results due to the doctor's lack of experience.

[0005] Patent name: A method and system for echocardiographic view recognition based on deep learning, application number: CN202111049989.1. The method disclosed in this patent includes obtaining an echocardiographic view; inputting the echocardiographic view into a trained echocardiographic view recognition model to obtain an echocardiographic view category recognition result; wherein the echocardiographic view recognition model is composed of StyleGAN and DenseNet101 networks; the StyleGAN is only used in the training process to supplement the echocardiographic view training data; the DenseNet101 is used to extract echocardiographic view features and identify echocardiographic view categories.

[0006] Patent name: Heart view recognition and left ventricle detection device and system based on deep learning, application number: CN202110553831.1. The invention discloses a heart view recognition and left ventricle detection device and system based on deep learning, including: a memory, a processor, and a computer program stored on the memory and run on the processor. When the processor executes the program, the following steps are implemented: receiving an echocardiogram; acquiring medical images of the apical two-chamber, apical three-chamber, and apical four-chamber in the echocardiogram; marking the contour of the left ventricle in different views and making them into original image data sets; preprocessing the original image data sets to obtain an experimental data set; inputting the experimental data set into a deep learning network to obtain heart view recognition results and left ventricle detection results.

[0007] The above-mentioned existing image recognition technologies are not developed for embedded devices and cannot solve the challenges brought by the large number of model parameters and long recognition time in this scenario. The first patent uses DenseNet101 to classify standard sections of echocardiograms. The network has a large amount of calculation and has high requirements for device performance. The second patent uses FPN (image pyramid) to extract features of each layer of the image, but there are obvious limitations on feature extraction at each layer, the inference time will increase sharply, and the memory usage of end-to-end training on FPN is large.

[0008] Echocardiographic standard section recognition based on deep learning encounters the following challenges: the existing classification model has too many parameters, resulting in slow operation in embedded devices and cannot meet real-time requirements well. Summary of the invention

[0009] In view of the technical problems existing in the prior art, the purpose of the present invention is to provide a lightweight echocardiogram standard section recognition method, device and medium that can quickly and accurately identify the category of the standard section of the echocardiogram.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A lightweight echocardiographic standard section recognition method comprises the following steps:

[0012] S1: Collect data of two-dimensional echocardiographic standard sections to form a data set; wherein the data of two-dimensional echocardiographic standard sections include echocardiographic standard section images and echocardiographic section types that identify the images, and the echocardiographic section types include apical two-chamber, apical three-chamber, apical four-chamber, aortic short axis, and parasternal long axis;

[0013] S2: Classify the data set and divide it into training set, test set and validation set;

[0014] S3: Preprocess and enhance the data set in step S2;

[0015] S4: Build a deep learning network model as a training model, and train the training model using the training set in step S2;

[0016] S5: Use the validation set to verify the training model, and further adjust the hyperparameters of the training model based on the evaluation results; use the test set to test the final generalization ability of the training model, and then obtain the echocardiography classification model;

[0017] S6: Input the echocardiogram into the trained echocardiogram classification model to obtain the echocardiogram classification and recognition result.

[0018] As a preferred embodiment, in the echocardiogram classification model, the deep learning network model includes an initial part, a middle part and a final part;

[0019] Among them, in the initial part: the ultrasound image of size 224x224x3 is input to a two-dimensional convolution layer, the standard section features of the echocardiogram are extracted through a 3x3 convolution kernel, and the feature layer is output;

[0020] In the middle part: it is composed of 9 bneck modules, which are connected in sequence; the feature layer output from the starting part is used as input, and after the convolution operation of each bneck module, the feature layer is output and used as the input of the next bneck module;

[0021] In the last part: the feature layer output from the middle part is used as input, and first passes through a 2D convolutional layer for a 1x1 convolution operation, and then the feature layer obtained after the convolution operation is input into the pooling layer. After the 7x7 convolution kernel, two 1x1 convolutional layers are used to replace the fully connected layer to output the category index k; the category index k matches the echocardiographic section type.

[0022] As a preferred embodiment, each bneck module includes an input unit, a first unit, a second unit, a third unit, a fourth unit and an output unit, and the input unit, the first unit, the second unit, the third unit, the fourth unit and the output unit are connected in sequence;

[0023] The feature layer of the input bneck module is first dimensionally upgraded using 1x1 convolution in the input unit, and then input to the next level after passing through the Relu activation function;

[0024] Then, the 3x3 depth separation convolution is performed in the second unit, the third unit, and the fourth unit through the Relu function, and input to the next level through the Relu function;

[0025] Finally, the dimensionality is reduced in the output unit through a 1x1 convolution kernel and output after a linear function.

[0026] As a preferred embodiment, in the middle part, the 9 bneck modules are bneck module 1, bneck module 2, bneck module 3, bneck module 4, bneck module 5, bneck module 6, bneck module 7, bneck module 8 and bneck module 9;

[0027] Introduce convolutional attention mechanism modules into bneck module 4, bneck module 6 and bneck module 7 respectively;

[0028] Each convolutional attention mechanism module includes a channel attention module and a spatial attention module, wherein the output of the second unit is connected to the input of the channel attention module for pooling and dimensionality reduction, the output of the channel attention module is connected to the input of the third unit, the output of the third unit is connected to the input of the spatial attention module for pooling and dimensionality reduction, and the output of the spatial attention module is connected to the input of the fourth unit; wherein the output of the channel attention module is the first weighted result, and the output of the spatial attention module is the second weighted result, and then a 3x3 depth separation convolution is performed through the fourth unit to obtain the final weighted result.

[0029] As a preferred embodiment, in step S1, the data set is formed by first acquiring a data mhd file, the mhd file is an echocardiogram data set, and the echocardiogram data set includes annotations of standard sections of the left ventricle of the heart, that is, annotations of the types of echocardiogram sections;

[0030] In step S2, the echocardiogram data set is converted into images in JPG format;

[0031] Classify the images, and divide the classified echocardiogram data set into a training set, a test set, and a validation set. The ratio of the training set, the test set, and the validation set is 6:2:2.

[0032] In step S3, the training set, the test set, and the validation set are preprocessed; data enhancement is performed on the training set, the test set, and the validation set using an online enhancement mode.

[0033] As a preferred embodiment, preprocessing includes desensitizing, cropping and masking the data to remove text, electrocardiogram and other information outside the scan sector; other information includes frame rate and hospital information; and finally the image is downsampled to a 224x224 size picture.

[0034] As a preferred embodiment, the enhancement methods include rotation, translation and folding.

[0035] A lightweight echocardiogram standard section recognition device comprises a memory, a processor and a program execution file. The program execution file uses a weakly supervised learning method to learn the features of the labeled image, so as to recognize the unlabeled echocardiogram standard sections of other hearts in the data set. When the processor executes the program, a lightweight echocardiogram standard section recognition method is implemented.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs a lightweight echocardiography standard section recognition method.

[0037] In general, the present invention has the following advantages:

[0038] 1. The present invention uses 1x1 convolution, 3x3 convolution and other technologies to reduce the amount of network calculation while ensuring the accuracy of the network as much as possible; while ensuring the accuracy of the model, the number of model parameters is reduced and the running speed is faster; the present invention can provide doctors with standard section category prompts during cardiac ultrasound diagnosis, reduce the workload of doctors, and avoid inaccurate diagnosis results caused by insufficient experience of doctors.

[0039] 2. The present invention uses a lighter model, which is easier to run in embedded devices and reduces deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of a lightweight echocardiographic standard section recognition method.

[0041] Figure 2 Flowchart for echocardiographic identification.

[0042] Figure 3 This is the Mobilenet-Echo network structure diagram.

[0043] Figure 4 This is the Bneck structure diagram.

[0044] Figure 5 Flowchart for forming the dataset.

[0045] Among them, k is the output category index, conv2d is the two-dimensional convolutional layer, bneck (bottleneck) is the bottleneck layer, pool is the pooling layer, NBN means not using batch normalization, Relu is the activation function, and CBAM is the convolutional attention mechanism module. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with specific implementation methods.

[0047] Embodiment 1

[0048] A lightweight echocardiographic standard section recognition method of this embodiment includes the following steps:

[0049] S1: Collect data of two-dimensional echocardiographic standard sections to form a data set; wherein the data of two-dimensional echocardiographic standard sections include echocardiographic standard section images and echocardiographic section types that identify the images, and the echocardiographic section types include apical two-chamber, apical three-chamber, apical four-chamber, aortic short axis, and parasternal long axis;

[0050] S2: Classify the data set and divide it into training set, test set and validation set;

[0051] S3: Preprocess and enhance the data set in step S2;

[0052] S4: Build a deep learning network model as a training model, and train the training model using the training set in step S2;

[0053] S5: Use the validation set to verify the training model, and further adjust the hyperparameters of the training model based on the evaluation results; use the test set to test the final generalization ability of the training model, and then obtain the echocardiography classification model;

[0054] S6: Input the echocardiogram into the trained echocardiogram classification model to obtain the echocardiogram classification and recognition result.

[0055] In the echocardiogram classification model, the deep learning network model includes the initial part, the middle part, and the final part;

[0056] Among them, in the initial part: the ultrasound image of size 224x224x3 is input to a two-dimensional convolution layer, the standard section features of the echocardiogram are extracted through a 3x3 convolution kernel, and the feature layer is output;

[0057] In the middle part: it is composed of 9 bneck modules, which are connected in sequence; the feature layer output from the starting part is used as input, and after the convolution operation of each bneck module, the feature layer is output and used as the input of the next bneck module;

[0058] In the last part: the feature layer output from the middle part is used as input, and first passes through a 2D convolutional layer for a 1x1 convolution operation, and then the feature layer obtained after the convolution operation is input into the pooling layer. After the 7x7 convolution kernel, two 1x1 convolutional layers are used to replace the fully connected layer to output the category index k; the category index k matches the echocardiographic section type.

[0059] The deep learning network of this embodiment adopts the network structure of Mobilnet-Echo, wherein conv2d is a two-dimensional convolutional layer; bneck (bottleneck) is a bottleneck layer; pool is a pooling layer; NBN means not using batch normalization; and k represents a category index.

[0060] Each bneck module includes an input unit, a first unit, a second unit, a third unit, a fourth unit and an output unit, wherein the input unit, the first unit, the second unit, the third unit, the fourth unit and the output unit are connected in sequence;

[0061] The feature layer of the input bneck module is first dimensionally upgraded using 1x1 convolution in the input unit, and then input to the next level after passing through the Relu activation function;

[0062] Then, the 3x3 depth separation convolution is performed in the second unit, the third unit, and the fourth unit through the Relu function, and input to the next level through the Relu function;

[0063] Finally, the dimensionality is reduced in the output unit through a 1x1 convolution kernel and output after a linear function. In the middle part, the 9 bneck modules are bneck module 1, bneck module 2, bneck module 3, bneck module 4, bneck module 5, bneck module 6, bneck module 7, bneck module 8 and bneck module 9;

[0064] Introduce convolutional attention mechanism modules into bneck module 4, bneck module 6 and bneck module 7 respectively;

[0065] Each convolutional attention mechanism module includes a channel attention module and a spatial attention module, wherein the output of the second unit is connected to the input of the channel attention module for pooling and dimensionality reduction, the output of the channel attention module is connected to the input of the third unit, the output of the third unit is connected to the input of the spatial attention module for pooling and dimensionality reduction, and the output of the spatial attention module is connected to the input of the fourth unit; wherein the output of the channel attention module is the first weighted result, and the output of the spatial attention module is the second weighted result, and then a 3x3 depth separation convolution is performed through the fourth unit to obtain the final weighted result.

[0066] In step S1, the data set is formed by first acquiring a data mhd file, where the mhd file is an echocardiogram data set, and the echocardiogram data set includes annotations of standard sections of the left ventricle of the heart, that is, annotations of the types of echocardiogram sections;

[0067] In step S2, the echocardiogram data set is converted into images in JPG format;

[0068] Classify the images, and divide the classified echocardiogram data set into a training set, a test set, and a validation set. The ratio of the training set, the test set, and the validation set is 6:2:2.

[0069] In step S3, the training set, the test set, and the validation set are preprocessed; data enhancement is performed on the training set, the test set, and the validation set using an online enhancement mode.

[0070] Preprocessing includes desensitizing, cropping and masking the data to remove text, electrocardiogram and other information outside the scan sector; other information includes frame rate and hospital information; finally, the image is downsampled to a 224x224 size image.

[0071] Enhancement methods include rotation, translation, and folding.

[0072] In step S4, the training method of the training model includes: in the training stage, the Tensorflow deep learning framework is used to learn the model parameters of the training set, first the image is input into the residual network structure of Mobilnet-Echo, and multiple small convolutions are used instead of large convolution kernels to reduce the model parameters and increase the number of nonlinear activation functions, and finally the ReLU activation function is used for activation. In this embodiment, TensorFlow is an open source, Python-based deep learning framework. The present invention uses TensorFlow to build the Mobilnet-Echo network model for training. After the image is input into the starting part of Mobilnet-Echo for convolution operation, it is further convolved through the 9 bneck modules in the middle part, and finally the category index k is output through the two 1x1 convolution layers in the last part.

[0073] In this embodiment,

[0074] Echocardiogram recognition method and technical solution:

[0075] Step 1: Construct an echocardiogram dataset;

[0076] Step 2: Input the data set formed in step 1 into the Mobilnet-Echo network model for training;

[0077] Step 3: Deploy the model trained in step 2;

[0078] Step 4: Input the echocardiographic data into step 3 to obtain the classification results;

[0079] Step 5: Input the classification results of step 4 into step 1 to form a data set.

[0080] Embodiment 2

[0081] A lightweight echocardiographic standard section recognition device based on deep learning in this embodiment includes a memory, a processor and a program execution file. The program execution file uses a weakly supervised learning method to learn the features of the labeled image, so as to recognize the unlabeled echocardiographic standard sections of other hearts in the data set. When the processor executes the program, a lightweight echocardiographic standard section recognition method based on deep learning is implemented.

[0082] The process of the program execution file when the processor is running is as follows:

[0083] S101: receiving echocardiogram;

[0084] S102: Obtain standard echocardiographic sections;

[0085] In this embodiment, the categories of standard echocardiographic sections include all standard echocardiographic section images such as apical two-chamber (A2C), apical three-chamber (A3C), apical four-chamber (A4C), short axis of the great arteries (PSA), and parasternal long axis (PLA).

[0086] S103: Inputting the echocardiographic view into the trained echocardiographic view recognition model to obtain an echocardiographic view category recognition result.

[0087] The echocardiogram view recognition model is composed of a Mobilenet-Echo network; the Mobilenet-Echo is used to extract echocardiogram view features and identify echocardiogram view categories.

[0088] In this embodiment, the image is first input into the residual network structure of Mobilenet-Echo, and multiple small convolutions are used instead of large convolution kernels to reduce model parameters and increase the number of nonlinear activation functions, making the amount of calculation smaller; finally, the ReLU activation function is used for activation. The ReLU function can make the network sparse and weaken the interdependence of parameters in the network, thereby suppressing overfitting.

[0089] In a specific implementation, image preprocessing is also included before the echocardiographic view is input into the echocardiographic view recognition model.

[0090] The image preprocessing includes random flipping, random cropping and size adjustment.

[0091] include:

[0092] (1) Random flip: flip the input image with a probability of 0.4;

[0093] (2) Random cropping: Cropping a given image to a random size and aspect ratio;

[0094] (3) Resizing: Resize the input image to a given size, such as upsampling or downsampling.

[0095] This embodiment uses the trained echocardiogram view recognition model to automatically identify the classification results of the echocardiogram views, wherein the echocardiogram view recognition model is composed of a Mobilenet-Echo network, and the Mobilenet-Echo is used to extract echocardiogram view features and identify echocardiogram view categories, thereby reducing the workload of doctors and avoiding inaccurate diagnosis results caused by insufficient experience of doctors, thereby improving the accuracy of echocardiogram view recognition.

[0096] The parts not mentioned in this embodiment are the same as those in the first embodiment.

[0097] Embodiment 3

[0098] A computer-readable storage medium of this embodiment stores a computer program, which, when executed by a processor, performs a lightweight echocardiography standard section recognition method based on deep learning.

[0099] The parts not mentioned in this embodiment are the same as those in the first embodiment.

[0100] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A lightweight echocardiographic standard section recognition method, characterized in that: The following steps are included: S1: Collect data of two-dimensional echocardiographic standard sections to form a data set; wherein the data of two-dimensional echocardiographic standard sections include echocardiographic standard section images and echocardiographic section types that identify the images, and the echocardiographic section types include apical two-chamber, apical three-chamber, apical four-chamber, aortic short axis, and parasternal long axis; S2: Classify the data set and divide it into training set, test set and validation set; S3: Preprocess and enhance the data set in step S2; S4: Build a deep learning network model as a training model, and train the training model using the training set in step S2; S5: Use the validation set to verify the training model, and further adjust the hyperparameters of the training model based on the evaluation results; use the test set to test the final generalization ability of the training model, and then obtain the echocardiography classification model; S6: inputting the echocardiogram into the trained echocardiogram classification model to obtain the echocardiogram classification and recognition result; In the echocardiogram classification model, the deep learning network model includes the initial part, the middle part, and the final part; Among them, in the initial part: the ultrasound image of size 224x224x3 is input to a two-dimensional convolution layer, the standard section features of the echocardiogram are extracted through a 3x3 convolution kernel, and the feature layer is output; In the middle part: it is composed of 9 bneck modules, which are connected in sequence; the feature layer output from the starting part is used as input, and after the convolution operation of each bneck module, the feature layer is output and used as the input of the next bneck module; In the last part: the feature layer output from the middle part is used as input, and first passes through a 2D convolutional layer for a 1x1 convolution operation, and then the feature layer obtained after the convolution operation is input into the pooling layer. After the 7x7 convolution kernel, two 1x1 convolutional layers are used to replace the fully connected layer to output the category index k; the category index k matches the echocardiographic section type.

2. A lightweight echocardiographic standard section recognition method according to claim 1, characterized in that: Each bneck module includes an input unit, a first unit, a second unit, a third unit, a fourth unit and an output unit, wherein the input unit, the first unit, the second unit, the third unit, the fourth unit and the output unit are connected in sequence; The feature layer of the input bneck module is first dimensionally upgraded using 1x1 convolution in the input unit, and then input to the next level after passing through the Relu activation function; Then, the 3x3 depth separation convolution is performed in the second unit, the third unit, and the fourth unit through the Relu function, and input to the next level through the Relu function; Finally, the dimensionality is reduced in the output unit through a 1x1 convolution kernel and output after a linear function.

3. A lightweight echocardiographic standard section recognition method according to claim 2, characterized in that: In the middle part, the nine bneck modules are bneck module 1, bneck module 2, bneck module 3, bneck module 4, bneck module 5, bneck module 6, bneck module 7, bneck module 8 and bneck module 9; Introduce convolutional attention mechanism modules into bneck module 4, bneck module 6 and bneck module 7 respectively; Each convolutional attention mechanism module includes a channel attention module and a spatial attention module, wherein the output of the second unit is connected to the input of the channel attention module for pooling and dimensionality reduction, the output of the channel attention module is connected to the input of the third unit, the output of the third unit is connected to the input of the spatial attention module for pooling and dimensionality reduction, and the output of the spatial attention module is connected to the input of the fourth unit; wherein the output of the channel attention module is the first weighted result, and the output of the spatial attention module is the second weighted result, and then a 3x3 depth separation convolution is performed through the fourth unit to obtain the final weighted result.

4. A lightweight echocardiographic standard section recognition method according to claim 1, characterized in that: In step S1, the data set is formed by first acquiring a data mhd file, where the mhd file is an echocardiogram data set, and the echocardiogram data set includes annotations of standard sections of the left ventricle of the heart, that is, annotations of the types of echocardiogram sections; In step S2, the echocardiogram data set is converted into images in JPG format; the images are classified, and the classified echocardiogram data set is divided into a training set, a test set, and a validation set, and the ratio of the obtained training set, test set, and validation set is 6:2:2; In step S3, the training set, the test set, and the validation set are preprocessed; data enhancement is performed on the training set, the test set, and the validation set using an online enhancement mode.

5. A lightweight echocardiographic standard section recognition method according to claim 4, characterized in that: Preprocessing includes desensitizing, cropping and masking the data to remove text, electrocardiogram and other information outside the scan sector; other information includes frame rate and hospital information; finally, the image is downsampled to a 224x224 size image.

6. A lightweight echocardiographic standard section recognition method according to claim 4, characterized in that: Enhancement methods include rotation, translation, and folding.

7. A lightweight echocardiogram standard section recognition device, characterized in that: It includes a memory, a processor and a program execution file. The program execution file uses a weakly supervised learning method to learn the features of the labeled image, so as to identify the unlabeled echocardiographic standard sections of other hearts in the data set. When the processor executes the program, a lightweight echocardiographic standard section recognition method as described in any one of claims 1-6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a lightweight echocardiographic standard section recognition method as described in any one of claims 1 to 6 is performed.

Images