Real-time detection and auxiliary analysis method of breast diseases based on breast ultrasound images

Abstract

The present invention relates to a kind of real-time detection and breast disease auxiliary analysis method based on breast ultrasound imaging, including: collecting video data during breast ultrasound examination, pre-processing the video data, inputting it into the yolov5 network for training, and obtaining a real-time detection model; according to the obtained real-time detection model, screening out breast lesion images for each case, and pre-processing, training a multi-view breast disease classification model based on the EfficientNetV2-L network; forming a real-time video with the echo of real-time detection of breast disease, inputting it into the real-time detection model, and outputting a lesion map with a confidence greater than a threshold; for the existence of a suspicious lesion map, reminding the size of the lesion, and outputting a prediction report if there is no lesion; inputting the lesion picture into the breast disease classification model, predicting the type and confidence of breast disease, and outputting relevant information. The present invention can assist doctors in the screening of breast lesions, and improves diagnostic efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of breast ultrasound screening, and in particular to a real-time detection and breast disease auxiliary analysis method based on breast ultrasound images. Background Art

[0002] Breast disease refers to diseases originating from breast-related tissues such as breast glands, fat, lymph, blood vessels, and nipples. Breast diseases include inflammatory breast diseases, benign breast lesions, malignant breast tumors, congenital developmental abnormalities, and male breast development. Among them, breast cancer occurs in about 270,000 cases per year in China, ranking first in the list of malignant tumors among Chinese women, and there is a trend of younger age. More than 10 years ago, the Chinese government established a "two cancer screening" project including breast cancer, providing free screening benefits to all women of appropriate age every year. However, due to the shortage of medical imaging physicians, the number of women who benefit each year is only about 2 million, which is a huge gap. With the gradual increase in life pressure, breast diseases are frequent and early-onset, and timely detection and treatment are helpful for patients' recovery. Regular breast examinations are the most effective way to prevent breast cancer. In fact, the detection rate of stage I in early breast cancer in Chinese women is only 20% to 25%, and the 5-year survival rate of patients is only about 80%. If breast cancer is discovered in the early stages and receives standardized treatment early, the patient's 5-year disease-free survival rate can reach 95%, and the 5-year survival rate of stage II patients can reach more than 80%.

[0003] "Real-time identification of lesions" is currently the biggest requirement of doctors for ultrasound screening of breast diseases. The difficulty of ultrasound detection lies in the fact that image acquisition and film reading need to be completed at the same time. In addition, compared with the results of magnetic resonance imaging, CT and electrocardiogram, ultrasound images are mostly diagnosed by doctors based on dynamic images of different sections collected by doctors, which requires a relatively high level of personal operation skills of ultrasound doctors, and there is currently a lack of corresponding auxiliary analysis technology. Summary of the invention

[0004] The object of the present invention is to provide a real-time detection and auxiliary analysis method of breast diseases based on breast ultrasound images, so as to solve the problem of lack of auxiliary diagnosis technology in the above-mentioned prior art.

[0005] The present invention discloses a real-time detection and auxiliary analysis method for breast diseases based on breast ultrasound images, which comprises: collecting video data during breast ultrasound examination, preprocessing the video data, and inputting the video data into a yolov5 network for training to obtain a real-time detection model; according to the obtained real-time detection model, screening out breast lesion images for each case, preprocessing the images, and training a multi-view breast disease classification model based on an EfficientNetV2-L network; forming a real-time video from echoes of real-time breast disease detection, inputting the real-time detection model, and outputting a lesion map with a confidence greater than a threshold; for an existing suspicious lesion map, reminding the size of the lesion, and outputting a prediction report if there is no lesion; inputting the lesion image into a breast disease classification model, predicting the type and confidence of the breast disease, and outputting relevant information.

[0006] According to an embodiment of the above method, video data from breast ultrasound examination is collected, the data is preprocessed, and then input into the yolov5 network for training to obtain a real-time detection model, including: Step 1: Collect a certain amount of video data from breast ultrasound examination, the ratio of benign lesion and malignant lesion images in the video data is roughly 1:1; Step 2: Extract a frame of image from the video data every second and scale it to form a detected image data set, and determine the label of the image data set by framing the area with lesions and marking the category of the corresponding lesions. The category of the lesion is based on the pathological section as the standard, including benign and malignant; Step 3: Preprocess the image data set; Step 4: Input the preprocessed image data set into the yolov5 network for training to obtain a real-time detection model.

[0007] According to an embodiment of the above method, in step 1, video data of 1,100 breast ultrasound examinations are collected, and the video data includes breast disease cases and no breast disease cases in a ratio of 10 to 1.

[0008] According to an embodiment of the above method, in step 2, a frame is extracted from the video every second, and the image size is adjusted to 512*512, and the label of each image is saved in the format of [category, x min ,y min ,x max ,y max ], where (x min ,y min ) is the coordinate of the upper left corner of the image, (x max ,y max ) is the coordinate of the lower right corner.

[0009] According to an embodiment of the above method, when the yolov5 network is trained, four images are used for training and one image is used for verification each time.

[0010] According to one embodiment of the above method, data enhancement includes: cropping, rotation, translation, scaling, color space adjustment, mosaic augmentation, self-adversarial training augmentation, multi-image combination, mixed enhancement MixUp, random occlusion Cutout and image occlusion.

[0011] According to an embodiment of the above method, based on the obtained real-time detection model, screening out breast lesion images for each case includes: screening the data of the detection model, screening out 4 breast lesion images at different angles for each case, and there are certain angle differences between the four images, the four images are grouped into a group to form a four-channel image, and then scaling the image to obtain the breast lesion image.

[0012] According to one embodiment of the above method, data preprocessing includes: performing data enhancement on the four-channel labeled images including: using random erasing, random occlusion Cutout, random hidden patches Hide and Seek, grid occlusion Grid Mask, mixed enhancement MixUp, mixed occlusion CutMix, mosaic enhancement, geometric distortion and illumination distortion; randomly dividing the data enhanced image into five parts, and using four image data as training and one image data as verification each time when training the EfficientNetV2-L network weights.

[0013] According to one embodiment of the above method, training a multi-view breast disease classification model based on the EfficientNetV2-L network includes: inputting the preprocessed data into the EfficientNetV2-L network for training, the training parameters are epochs=420, learning rate=0.0001, category=3, batchsize=32, and the loss function is cross entropy. After 420 cycles of iterations and five-fold cross validation training, a breast disease classification model is obtained.

[0014] According to an embodiment of the above method, outputting relevant information includes: outputting a lesion image and an auxiliary diagnosis report of the type and confidence level of the breast disease.

[0015] The present invention adopts the above-mentioned real-time detection and breast disease analysis method based on breast ultrasound imaging, which can identify whether there are suspected breast disease areas in breast ultrasound examination images, and accurately mark and display the size and type of lesions, and score suspected lesions, thereby assisting doctors in screening breast lesions, improving diagnostic efficiency and accuracy, and effectively balancing the problem of uneven distribution of medical resources in my country. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a training diagram of the real-time detection model of breast diseases based on ultrasound images of the present invention;

[0017] Figure 2 This is a training diagram of the ultrasound image-based breast disease classification model of the present invention;

[0018] Figure 3 The present invention is a flowchart of the real-time detection of breast ultrasound images and breast disease analysis;

[0019] Figure 4 This is a rendering of a real-time detection and breast disease analysis system based on breast ultrasound imaging according to an embodiment of the present invention.

[0020] Figure 4 On the left is a benign breast lesion, and on the right is a malignant breast lesion, shown in the boxes in the marked box diagram. DETAILED DESCRIPTION

[0021] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0022] like Figure 1 As shown, the present invention uses a real-time breast lesion detection network based on yolov5, which includes data preparation, data preprocessing, data training and data verification.

[0023] Step 1: First, we need to collect 1,100 video data of breast ultrasound examinations, including 1,000 cases with breast diseases and 100 cases without breast diseases. We need to ensure that the ratio between benign lesions and malignant lesions is 1:1.

[0024] Step 2: Then extract a frame from the video every second and scale it to 512*512 as the detection data set. The obtained image data set labels are obtained by two experienced ultrasound doctors to frame the area with lesions and mark the corresponding lesion categories. When the two doctors’ labeling results are inconsistent, a third doctor is used for review. The disease category is based on the pathological section as the standard, such as Figure 4 As shown, the label of each image is saved in the format of [category, x min ,y min ,x max ,y max ], where (x min ,y min ) is the coordinate of the upper left corner, (x max ,y max) is the lower right corner coordinate, and the categories include benign and malignant. When training yolov5, the categories of benign and malignant lesions in the present invention are all 1, and the mark without lesions is [none, 0, 0, 1, 1]. Since a large number of images in the data are without lesions, in order to ensure the balance of the data and thus make the model effect optimal, the present invention retains all the images with lesions, and the data ratio is images with lesions: images without lesions = 1:1.

[0025] Step 3: Then perform data preprocessing. The data preprocessing part includes data augmentation and five-fold cross-validation. First, the image data is randomly divided into five parts. When training yolov5, 4 parts of the data are used for training and 1 part of the data is used for validation each time. Data augmentation includes: cropping, rotation, translation, scaling, color space adjustment, mosaic augmentation, self-adversarial training augmentation, multi-image combination, mixed enhancement MixUp, random occlusion Cutout, and image occlusion.

[0026] Step 4: Input the preprocessed data into the yolov5 network for training. The training parameters are: epochs (the number of times all samples in the training set are trained) = 60, num_workers (the number of subprocesses) = 4, learning rate = 0.0001, category = 1, batchsize (the number of samples used in one iteration) = 32, loss function = Giou_loss. After 60 cycles of iteration and five-fold cross-validation training, the real-time detection model 1 is obtained. The network parameters are adjusted to ensure that the detection rate reaches more than 95%. The process is as follows: Figure 1 shown.

[0027] exist Figure 2 In the present invention, a multi-view breast disease classification model based on the EfficientNetV2-L network is trained, and the training process includes data preparation, data preprocessing, data training and data verification.

[0028] Step 1: First, we need to filter the data from the real-time detection model 1. Since the benign and malignant classification has been done, we only need to filter out 4 breast lesion images at different angles for each case. We need to try to ensure that there are large angle differences between the four images in order to achieve better resolution. Therefore, we can obtain 500*4 malignant lesion images and 500*4 benign lesion images through the above means. For better classification, we also find 500*4 normal images from the images. Data balancing can effectively increase the robustness of the algorithm. The 4-angle data is a group to form a 4-channel image, and then the image is scaled to 1024*1024*4.

[0029] Step 2: Data preprocessing: perform data enhancement on the labeled data, i.e., 500 benign four-channel images, 500 malignant four-channel images, and 500 normal four-channel images, using random erasing, random occlusion Cutout, random hidden patch Hide and Seek, grid occlusion Grid Mask, mixed enhancement MixUp, mixed occlusion CutMix, mosaic enhancement, geometric distortion and illumination distortion. Then, the enhanced data is randomly divided into 5 parts. When training the EfficientNetV2-L network weights, 4 parts of the data are used for training and 1 part of the data is used for verification each time.

[0030] Step 3: Data training: input the preprocessed data into the EfficientNetV2-L network for training. The training parameters are epochs = 420, learning rate = 0.0001, categories = 3, batchsize = 32. The classification model 2 is obtained through training. The loss function is cross entropy. After 420 cycles of iterations and five-fold cross validation training, the breast disease classification model 2 is obtained. The process is as follows: Figure 2 shown.

[0031] exist Figure 3 In the figure, a workflow diagram of the method for real-time detection of breast ultrasound images and analysis of breast diseases of the present invention is provided.

[0032] Step 1: Real-time detection of breast diseases. When the ultrasound doctor begins to examine the breast, the ultrasonic echo will form a real-time video. In order to achieve real-time detection, each frame of the video is first scaled into a 512*512 image; then it is input into the real-time detection model 1, and the lesions with a confidence level greater than 0.3 are displayed on the screen in real time. The detection frame can provide real-time prompts to the doctor, and the doctor can better diagnose breast diseases based on the real-time reminders on the screen.

[0033] Step 2: Manually select multi-angle lesion images. If there are suspicious lesions, the doctor needs to be reminded of the size of the lesion. The doctor then saves 4 breast lesion images from different angles based on experience. If there are N lesions, 4*N breast lesion images are saved, where N is a positive integer < 5. The images with lesions are scaled to 1024*1024*4. If there are no lesions, a diagnosis report is output.

[0034] Step 3: Classification of lesion areas. After the breast examination is completed, the lesion image is input into the breast disease classification model 2 to predict the type and confidence of the breast disease. If there are only malignant lesions or only benign lesions, the maximum lesion image, precise identification, lesion size, type and confidence are output; if there are both malignant and benign lesions, the maximum malignant lesion image, precise identification, lesion size, type and confidence are output.

[0035] Step 4: Output the image and measurement results as an auxiliary diagnosis report

[0036] The present invention discloses a real-time detection and breast disease analysis method based on breast ultrasound imaging, including real-time detection model training, disease classification model training and model verification. In order to detect the area of ​​lesions in real time, the present invention uses a yolov5 (You only look once: yolo, v5 is the fifth version) network as the real-time detection model of the present invention. The model trained using the yolov5 network can reach 15 frames per second on a common cpu machine and 100 frames per second on a machine with gpuGTX1080ti, so that real-time detection of breast lesions can be achieved on most machines.

[0037] The model trained by Yolov5 is a network that can detect and classify. There are three reasons why it is not used as a classification network: 1. Adding multiple classifications in the Yolov5 model will increase the difficulty of model learning, resulting in a reduction in the effect of lesion detection; 2. When screening lesions, ultrasound doctors often check the shapes of multiple angles for a lesion to finally confirm the type of breast lesions; 3. The Yolov5 model is relatively small and has limited classification capabilities. Therefore, the present invention uses EfficientNetV2-L (the second version of EfficientNet, with large network model parameters: Large) as a disease classification model, which can effectively solve the above-mentioned existing problems. EfficientNetV2-L is a high accuracy and fast classification network. As a disease classification model of the present invention, the speed does not need to meet the real-time requirements, and more spatial information of different angles of breast lesions needs to be considered, which can more accurately classify the type of breast lesions than using a single image alone.

[0038] The last step of the present invention is to verify the model of the trained real-time detection model and disease classification model, that is, the model assists the ultrasound doctor in diagnosis in actual clinical practice. When the ultrasound doctor examines the breast, the real-time breast lesion detection model can remind the doctor of the location and area of ​​the suspicious lesions on the screen in real time, which can effectively prevent missed diagnosis. At the same time, the ultrasound doctor needs to save the pictures of suspicious lesions at different angles, and then use the breast lesion classification model to obtain the type and confidence of the lesion, and output an auxiliary diagnosis report. The report content includes: lesion size, depth, type and confidence. It can effectively remind doctors and reduce missed diagnosis and misdiagnosis.

[0039] The present invention can identify whether there are suspected breast lesions in breast ultrasound examination images, accurately mark and display the size and type of the lesions, and score the suspected lesions, thereby assisting doctors in screening breast lesions, improving diagnostic efficiency and accuracy, and balancing the problem of uneven distribution of medical resources in my country.

[0040] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time detection and auxiliary analysis method for breast diseases based on breast ultrasound imaging, characterized in that: include: The video data of breast ultrasound examination was collected, pre-processed, and then input into the yolov5 network for training to obtain a real-time detection model; Based on the obtained real-time detection model, breast lesion images were screened for each case, preprocessed, and a multi-view breast disease classification model based on the EfficientNetV2-L network was trained; The echoes of real-time detection of breast diseases are transformed into real-time video, which is input into the real-time detection model, and the lesion images with a confidence level greater than a threshold are output; For suspicious lesion images, the size of the lesion is reminded, and if there is no lesion, a prediction report is output; Input the lesion image into the breast disease classification model, predict the type and confidence of the breast disease, and output relevant information; According to the obtained real-time detection model, the breast lesion images screened for each case include: The data of the detection model is screened, and four breast lesion images at different angles are screened for each case. There are certain angle differences between the four images. The four images are grouped into a four-channel image, and then the images are scaled to obtain the breast lesion image. Data preprocessing includes: data enhancement of four-channel labeled images, including: random erasing, random occlusion Cutout, random hidden patch Hide and Seek, grid occlusion Grid Mask, mixed enhancement MixUp, mixed occlusion CutMix, mosaic enhancement, geometric distortion and illumination distortion; randomly divide the data-enhanced images into five parts, and use four image data for training and one image data for verification each time when training the EfficientNetV2-L network weights; Training a multi-view breast disease classification model based on the EfficientNetV2-L network includes: inputting the preprocessed data into the EfficientNetV2-L network for training, with training parameters of epochs=420, learning rate=0.0001, category=3, batchsize=32, and loss function being cross entropy, and obtaining a breast disease classification model after 420 cycles of iterations and five-fold cross validation training.

2. The method according to claim 1, characterized in that The video data of breast ultrasound examination is collected, pre-processed and input into the yolov5 network for training. The real-time detection model includes: Step 1: Collect a certain amount of breast ultrasound examination video data, in which the ratio of benign lesion images to malignant lesion images is roughly 1:1; Step 2: Extract one frame of image from the video data every second and scale it to form a detection image dataset. Determine the label of the image dataset by framing the area with lesions and marking the category of the corresponding lesions. The category of lesions is based on the pathological section as the standard, including benign and malignant. Step 3: Preprocess the image dataset; Step 4: Input the preprocessed image dataset into the yolov5 network for training to obtain a real-time detection model.

3. The method according to claim 2, characterized in that In step 1, video data of 1,100 breast ultrasound examinations are collected, and the video data includes cases of breast disease and cases without breast disease in a ratio of 10 to 1.

4. The method according to claim 2, characterized in that In step 2, a frame is extracted from the video every second and the image size is adjusted to 512*512. The label of each image is saved in the format of [category, x min ,y min ,x max ,y max ], where (x min ,y min ) is the coordinate of the upper left corner of the image, (x max ,y max ) are the coordinates of the lower right corner.

5. The method according to claim 2, characterized in that When the yolov5 network is trained, four images are used for training and one image is used for verification each time.

6. The method according to claim 5, characterized in that Data augmentation includes: cropping, rotation, translation, scaling, color space adjustment, mosaic augmentation, self-adversarial training augmentation, multi-image combination, mixed enhancement MixUp, random occlusion Cutout and image occlusion.

7. The method according to claim 1, characterized in that Outputting relevant information includes: outputting lesion images and auxiliary diagnosis reports of breast disease types and confidence levels.

Images