Digital breast tomosynthesis imaging lesion localization device

Abstract

The present invention discloses a device for localizing lesions in digital breast tomographic images. The present invention includes a data input module for acquiring labeled digital breast tomographic images; an image preprocessing module for preprocessing the digital breast tomographic images; a data sorting module for performing data division on the labeled digital breast tomographic images; a model adjustment module including a model training submodule and a model testing submodule; wherein the model training submodule uses training set data to train the EfficientDet network; and the model testing submodule uses test set data to test the trained EfficientDet network; and a regional localization module uses a lesion fusion algorithm to fuse detection frames of lesions at the same position on different slices, and filters out detection results with a detection probability lower than a set threshold, while integrating diagnostic result labels for lesions at the same position. The present invention can efficiently and accurately locate the position of lesions in digital breast tomographic images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a digital breast tomographic imaging lesion positioning device. Background Art

[0002] Digital mammography, currently commonly used in clinical practice, can cause tissue overlap when imaging dense breasts. However, when using Digital Breast Tomosynthesis (DBT) to capture images, the X-ray tube rotates around the breast within a certain angle range, delivering a low-dose exposure each time the tube rotates. As the X-ray tube completes its rotation, the digital detector captures perspective images of the breast from different angles. Computer-generated three-dimensional tomographic images using simultaneous algebraic reconstruction techniques reduce tissue overlap, preventing missed or misdiagnosed lesions. Furthermore, DBT allows for the visualization of lesions in dense breasts, encompassing different locations and morphologies, leading to more accurate clinical diagnoses of breast cancer.

[0003] However, compared to traditional X-ray technology, digital breast tomosynthesis produces dozens of times more images. Doctors spend a significant amount of time reviewing these images, which can lead to missed or false positives due to visual fatigue. Therefore, locating lesions within this massive amount of image data is crucial for doctors. Therefore, research on lesion localization technology using digital breast tomosynthesis has significant practical value and significance, reducing the burden on doctors and providing accurate auxiliary results for clinical diagnosis. Summary of the Invention

[0004] The present invention addresses these challenges by providing an efficient and accurate device for locating lesions in digital breast tomosynthesis. This device, based on deep learning and combined with transfer learning strategies, utilizes a convolutional neural network (CNN) within a deep learning model to extract image features. A region generation network is used to generate candidate detection boxes, and a classification network is used to predict benign or malignant tumors. This device can address the current lack of experience among physicians in interpreting DBT images, effectively reducing workload and improving efficiency. This device will further promote the clinical application and technological development of digital breast tomosynthesis in breast cancer imaging.

[0005] To achieve this purpose, the present invention adopts the following technical solutions:

[0006] The digital breast tomographic imaging lesion positioning device of the present invention comprises a data input module, an image preprocessing module, a data sorting module, a model adjustment module and a region positioning module;

[0007] The data input module is used to obtain digital breast tomographic images with labels;

[0008] The image preprocessing module preprocesses the digital breast tomographic image transmitted by the data input module;

[0009] Preferably, the preprocessing method for the digital breast tomographic image includes invalid background removal and image denoising. Invalid background removal specifically uses edge cropping technology to remove invalid background areas of the digital breast tomographic image. Image denoising uses a masking algorithm to remove the pectoral muscle portion of the breast in the image after invalid background removal.

[0010] The data sorting module divides the labeled digital breast tomographic images transmitted by the image preprocessing module to obtain a training set and a test set;

[0011] The model adjustment module includes a model training submodule and a model testing submodule; wherein the model training submodule uses the training set data to train the EfficientDet network; the model testing submodule uses the test set data to test the trained EfficientDet network;

[0012] The regional positioning module uses a lesion fusion algorithm to fuse the detection frames of lesions at the same position on different slices, and screens out detection results with detection probabilities lower than a set threshold, while integrating the diagnostic result labels of lesions at the same position.

[0013] Preferably, the preprocessing method for digital breast tomographic images in the image preprocessing module includes invalid background removal and image denoising.

[0014] The EfficientDet network includes a backbone network, a feature pyramid module, a benign and malignant classification network, and a detection box prediction network;

[0015] 1) The backbone network uses EfficientNet-B0, which mainly includes input layer, convolution layer, MB convolution (Mobile Inverted Bottleneck Convolution) module, pooling layer, and fully connected layer;

[0016] 2) The feature pyramid module uses the BiFPN (Bi-directional Feature Pyramid Network) structure. Based on the PAFPN (Path Aggregation Feature Parymid Network), it removes some convolutional layers and adds a shortcut structure.

[0017] 3) The benign and malignant classification network fuses the features output by the backbone network to obtain a fused feature map, and outputs the corresponding benign and malignant prediction results and their probabilities;

[0018] 4) The detection box prediction network outputs the lesion location of the detection target based on the feature map extracted by the feature pyramid network.

[0019] Preferably, the backbone network is:

[0020] The input image is passed through the backbone network to obtain a feature map, and the feature maps obtained from the backbone network are placed in the same set according to the corresponding breast. In this way, each set contains the feature maps obtained after the same breast serial tomographic image passes through the backbone network;

[0021] In the same breast tomographic image, the image in the middle part can express more information. Using the normal distribution, in the process of feature fusion, the corresponding probability density is obtained according to the position of the image corresponding to the feature map in the tomographic layer. The probability density is multiplied by the corresponding feature map. Finally, the feature maps in the same set are added together to obtain the fused feature map. The calculation formula is shown in formula (1):

[0022] Formula (1)

[0023] in is the mean of the image sequence, is the standard deviation of the image sequence, is the serial number of the image, is the feature map corresponding to the serial number, and F is the fused feature map.

[0024] Preferably, the feature pyramid module includes five input nodes, three intermediate nodes, and five output nodes; wherein the intermediate node X1 is obtained by fusion of the input node P7 and the input node P6; the intermediate node X2 is obtained by fusion of the input node P5 and the intermediate node X1; the intermediate node X3 is obtained by fusion of the input node P4 and the intermediate node X2; the output node Y1 is obtained by fusion of the input node P7 and the output node Y2; the output node Y2 is obtained by fusion of the input node P6, the intermediate node X1 and the output node Y3; the output node Y3 is obtained by fusion of the input node P5, the intermediate node X2 and the output node Y4; the output node Y4 is obtained by fusion of the input node P4, the intermediate node X3 and the output node Y5; the output node Y5 is obtained by fusion of the input node P3 and the intermediate node X3; thus, the planar structure formed by the stacking of BiFPN is top to bottom, then bottom to top, and then repeated continuously.

[0025] Preferably, the lesion fusion algorithm of the regional positioning module is specifically:

[0026] 5-1 First, determine whether there is more than one lesion location in a single tomographic image. If so, calculate the intersection-over-union ratio between each lesion location and continue to determine whether the intersection-over-union ratio is greater than a threshold. If so, filter out the lesion locations with lower confidence, otherwise retain all of them.

[0027] 5-2 Arrange the remaining tomographic images processed in step 5-1 according to the order in which the digital breast tomographic images were taken, project them onto the same plane in sequence, calculate the intersection-over-union ratios (IORs) between the lesion locations in each of the remaining tomographic images, and continue to determine whether the IORs are greater than a threshold. If so, the two lesion locations are grouped into the same set; otherwise, the two lesion locations are grouped into different sets.

[0028] 5-3 Calculate the average confidence level of the lesion location in each set and determine whether the average is lower than a threshold. If so, remove the set; otherwise, retain it.

[0029] 5-4 Since the lesion locations in the same set are the same, it is necessary to determine whether the number of lesion locations in the same set is greater than a threshold. If so, they are retained; otherwise, they are deleted; and finally, the lesion locations of the remaining sets are output.

[0030] The present invention uses the DNN (Deep Neural Network) structure of a deep learning model, combined with a transfer learning strategy, to automatically learn the characteristics of digital breast tomography through model training, thereby localizing lesions, significantly reducing the workload of doctors and improving the diagnosis rate, further promoting the practical value of digital breast tomography in clinical practice, and thus forming a more efficient and standardized diagnostic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural block diagram of the present invention;

[0032] Figure 2 Schematic diagram of the location of lesions in digital breast tomography images of the present invention; (a) benign, (b) malignant;

[0033] Figure 3 This is a flow chart of the training and classification recognition of the model adjustment module in the present invention;

[0034] Figure 4 This is a flow chart of positioning determination of the regional positioning module 5 in the present invention;

[0035] Figure 5 This is the EfficientDet structure diagram used in the model adjustment module of the present invention;

[0036] Figure 6 Figure 2 shows the structure of the feature pyramid module in the present invention; (a) PAFPN, (b) BiFPN;

[0037] Among them, 1-data input module, 2-image preprocessing module, 3-data sorting module, 4-model adjustment module, and 5-region positioning module. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation examples:

[0039] like Figure 1 As shown, the digital breast tomographic imaging lesion localization device includes a data input module 1, an image preprocessing module 2, a data sorting module 3, a model adjustment module 4, and a regional positioning module 5; the output end of the data input module 1 is connected to the input end of the image preprocessing module 2, the output end of the image preprocessing module 2 is connected to the input end of the data sorting module 3, the output end of the data sorting module 3 is connected to the input end of the model adjustment module 4, and the output end of the model adjustment module 4 is connected to the input end of the positioning diagnosis module 5.

[0040] The data input module 1 is used to obtain the pixel information of the patient's digital breast tomographic image and convert it into a general image format; the image preprocessing module 2 preprocesses the digital breast tomographic image to improve the quality of the digital breast tomographic image and remove invalid information; the data sorting module 3 divides the labeled digital breast tomographic image transmitted by the image preprocessing module to obtain a training set and a test set; the model adjustment module 4 includes a model training submodule and a model testing submodule; the model training submodule uses the training set data to train the EfficientDet network; the model testing submodule uses the test set data to test the trained EfficientDet network. The model training module can be fed back by evaluating the model testing module, and the algorithm model can be corrected by using model optimization, parameter adjustment and data set adjustment; the regional positioning module 5 uses the lesion fusion algorithm to fuse the detection frames of lesions at the same position on different slices, and screen out the detection results with a detection probability lower than the set threshold, and at the same time integrate the diagnosis result labels of lesions at the same position.

[0041] The digital breast tomosynthesis imaging lesion detection and diagnosis method includes the following steps:

[0042] Step 1: Data input module 1 acquisition Figure 2 A digital breast tomographic image data DICOM file with labels is converted into a general image format; the labels are the location of the breast lesion and whether it is benign or malignant; the digital breast tomographic image is a plurality of tomographic images of the entire breast in a certain space.

[0043] Step 2: The image preprocessing module preprocesses the digital breast tomographic image obtained by the data input module;

[0044] The pre-processing specifically comprises using edge clipping technology to remove the black invalid background area at the edge, and using a mask algorithm to remove the pectoral muscle part of the oblique lateral image in the digital breast tomography image;

[0045] Step 3: The data sorting module uses the labeled preprocessed digital breast tomographic images as a data set, one part of which is a training set and the other part is a test set;

[0046] Step 4: Figure 3 The model training submodule in the model adjustment module uses the training set data to train the EfficientDet network; the model testing submodule uses the test set data to test the trained EfficientDet network.

[0047] like Figure 5 The EfficientDet network shown includes the backbone network, feature pyramid module, benign and malignant classification network, and detection box prediction network;

[0048] 1) The backbone network used is EfficientNet-B0, which primarily consists of an input layer, convolutional layers, MB convolutional modules, pooling layers, fully connected layers, and a Softmax classifier, ultimately outputting image features. The specific parameters of this network are shown in Table 1. Compared to other models, EfficientNet's greatest innovation lies in its MB convolutional module structure, which was derived through Neural Architecture Search (NAS). Current design approaches for neural network expansion primarily focus on increasing the number of network layers (depth), increasing the network width (i.e., the number of channels in the network (width), and increasing the resolution of the input image. However, expanding the network in a single dimension rarely yields significant improvements in accuracy given limited resources. EfficientDet addresses this issue by expanding the network model in three dimensions and, through neural architecture search, identifying the MB convolutional module.

[0049] Specifically:

[0050] The input image passes through the backbone network to obtain a feature map, and the feature maps obtained from the backbone network are placed in the same set according to the corresponding breast. In this way, each set contains the feature maps obtained after the same breast sequence (1-N) tomographic images pass through the backbone network.

[0051] In the same breast tomographic image, the image in the middle part can express more information. Using the normal distribution, in the process of feature fusion, the corresponding probability density is obtained according to the position of the image corresponding to the feature map in the tomographic layer. The probability density is multiplied by the corresponding feature map, and finally the feature maps in the same set are added together to obtain the fused feature map. The calculation formula is shown in formula (1):

[0052] Formula (1)

[0053] in is the mean of the image sequence, is the standard deviation of the image sequence, is the serial number of the image, is the feature map corresponding to the serial number, and F is the fused feature map.

[0054] 3) Feature Pyramid Networks (FPN) uses the BiFPN structure. Based on the PAFPN, some convolutional layers are removed and a shortcut structure is added. The comparison is shown in the figure below. Figure 6 It is used to further refine the features of the fused feature map output by the backbone network; compared with the ordinary FPN structure, it has fewer parameters and higher accuracy.

[0055] The feature pyramid module consists of five input nodes, three intermediate nodes, and five output nodes. Intermediate node X1 is obtained by fusing input node P7 and input node P6. Intermediate node X2 is obtained by fusing input node P5 and intermediate node X1. Intermediate node X3 is obtained by fusing input node P4 and intermediate node X2. Output node Y1 is obtained by fusing input node P7 and output node Y2. Output node Y2 is obtained by fusing input node P6, intermediate node X1, and output node Y3. Output node Y3 is obtained by fusing input node P5, intermediate node X2, and output node Y4. Output node Y4 is obtained by fusing input node P4, intermediate node X3, and output node Y5. Output node Y5 is obtained by fusing input node P3 and intermediate node X3. Consequently, the planar structure formed by the stacked BiFPN is top-to-bottom, then bottom-to-top, and then repeats. Because top-to-bottom and bottom-to-top have a sequential relationship, this planar structure can lead to poor information transmission, thus affecting object detection performance.

[0056] 3) The benign and malignant classification network obtains a fusion feature map based on the feature fusion output by the backbone network, and outputs the corresponding benign and malignant prediction results and their probabilities.

[0057] 4) The detection box prediction network outputs the lesion location of the detection target based on the feature map extracted by the feature pyramid network.

[0058] The image dataset generated by the data organization module is applied to the EfficientDet network structure to obtain the detection results of lesions on different tomographic images. If the feedback is not ideal, the model can be continuously adjusted to optimize it.

[0059] Step 5: Figure 4 The regional localization module utilizes a lesion fusion algorithm to merge detection frames for lesions at the same location across different slices and filter out detection results with a detection probability below a set threshold, thereby reducing the false-positive lesion detection rate in slice images. The fusion module also integrates the diagnostic result labels for lesions at the same location to obtain the final diagnosis for that location, providing further analysis for the doctor.

[0060] 5-1 First, determine whether there is more than one lesion location in a single tomographic image. If so, calculate the intersection-over-union ratio between each lesion location and continue to determine whether the intersection-over-union ratio is greater than a threshold (which can be 0.5). If so, filter out the lesion locations with lower confidence; otherwise, retain all of them.

[0061] 5-2 Arrange the remaining tomographic images processed in step 5-1 according to the shooting order of the digital breast tomographic images, project them onto the same plane in sequence, calculate the intersection-over-union ratio between the lesion locations in the remaining tomographic images, and continue to determine whether the intersection-over-union ratio is greater than a threshold (which can be 0.5). If so, the two lesion locations are classified into the same set; otherwise, the two lesion locations are classified into different sets.

[0062] 5-3 Calculate the average confidence level of the lesion location in each set and determine whether the average is lower than a threshold (which can be 0.5). If so, remove the set; otherwise, retain it.

[0063] 5-4 Since the lesion locations in the same set are the same, it is necessary to determine whether the number of lesion locations in the same set is greater than a threshold (which can be 1). If so, they are retained; otherwise, they are deleted; and finally, the lesion locations of the remaining sets are output.

[0064] Furthermore, in step 4, the model adjustment module includes a model training module and a model testing module; the model training module adopts the EfficientDet structure in the DNN structure of the deep learning model, and combines it with the transfer learning strategy to realize the detection of breast cancer tomographic image lesions and the analog diagnosis of benign and malignant lesions. The model training is completed under supervision, using the stochastic gradient descent method to minimize the difference between the actual output and the expected output, and using the back propagation method to calculate the gradient coefficient. By continuously adjusting the parameters between the neural network models, the training error is reduced; the model testing module uses the model trained by the model training module to test, evaluate and provide feedback, thereby obtaining the location of the lesion and the benign and malignant labels.

[0065] Table 1 EfficientNet-B0 network structure configuration

[0066] stage Module Resolution Number of channels Number of layers 1 Convolutional layer 3×3 224×224 32 1 2 MB convolution module 112×112 16 1 3 MB convolution module 112×112 24 2 4 MB convolution module 56×56 40 2 5 MB convolution module 28×28 80 3 6 MB convolution module 14×14 112 3 7 MB convolution module 14×14 192 4 8 MB convolution module 7×7 320 1 9 Convolutional layer 1×1 pooling layer Fully connected layer 7×7 1280 1

[0067] Any modifications and changes made to the present invention within the spirit and protection scope of the claims of the present invention shall fall within the protection scope of the present invention.

Claims

1. Digital breast tomosynthesis imaging lesion localization device, characterized by It includes data input module, image preprocessing module, data sorting module, model adjustment module and area positioning module; The data input module is used to obtain digital breast tomographic images with labels; The image preprocessing module preprocesses the digital breast tomographic image transmitted by the data input module; The data sorting module divides the labeled digital breast tomographic images transmitted by the image preprocessing module to obtain a training set and a test set; The model adjustment module includes a model training submodule and a model testing submodule; The model training submodule uses the training set data to train the EfficientDet network; the model testing submodule uses the test set data to test the trained EfficientDet network; the EfficientDet network includes a backbone network, a feature pyramid module, a benign and malignant classification network, and a detection box prediction network; 1) The backbone network uses EfficientNet-B0, which mainly includes input layer, convolution layer, MB convolution module, pooling layer, and fully connected layer; the backbone network is specifically: The input image is passed through the backbone network to obtain a feature map, and the feature maps obtained from the backbone network are placed in the same set according to the corresponding breast. In this way, each set contains the feature maps obtained after the same breast serial tomographic image passes through the backbone network; In the same breast tomographic image, the middle part of the image can express more information. Using the normal distribution, during the feature fusion process, the corresponding probability density is calculated based on the position of the image corresponding to the feature map in the tomographic layer. The probability density is multiplied by the corresponding feature map, and finally the feature maps in the same set are added together to obtain the fused feature map. The calculation formula is shown in Formula 1: Formula (1) in is the mean of the image sequence, is the standard deviation of the image sequence, is the serial number of the image, is the feature map corresponding to the serial number, and F is the fused feature map; 2) The feature pyramid module uses the BiFPN structure. Based on the PAFPN, some convolutional layers are removed and a shortcut structure is added. 3) The benign and malignant classification network fuses the features output by the backbone network to obtain a fused feature map, and outputs the corresponding benign and malignant prediction results and their probabilities; 4) The detection box prediction network outputs the lesion location of the detection target based on the feature map extracted by the feature pyramid network; The regional positioning module uses a lesion fusion algorithm to fuse the detection frames of lesions at the same location on different slices, and screens out detection results with a detection probability lower than a set threshold, while integrating the diagnostic result labels of lesions at the same location; The lesion fusion algorithm of the regional positioning module is specifically: 5-1 First, determine whether there is more than one lesion location in a single tomographic image. If so, calculate the intersection-over-union ratio between each lesion location and continue to determine whether the intersection-over-union ratio is greater than a threshold. If so, filter out the lesion locations with lower confidence, otherwise retain all of them. 5-2 Arrange the remaining tomographic images processed in step 5-1 according to the order in which the digital breast tomographic images were taken, project them onto the same plane in sequence, calculate the intersection-over-union (IoU) between the lesion locations in each of the remaining tomographic images, and continue to determine whether the IoU is greater than a threshold. If so, the two lesion locations currently being calculated for IoU are grouped into the same set; otherwise, the two lesion locations are grouped into different sets. 5-3 Calculate the average confidence level of the lesion location in each set and determine whether the average is lower than a threshold. If so, remove the set; otherwise, retain it. 5-4 Since the lesion locations in the same set are the same, it is necessary to determine whether the number of lesion locations in the same set is greater than a threshold. If so, they are retained; otherwise, they are deleted; and finally, the lesion locations of the remaining sets are output.

2. The digital breast tomosynthesis imaging lesion localization device according to claim 1, characterized in that The preprocessing method for digital breast tomographic images in the image preprocessing module includes invalid background removal and image denoising.

3. The digital breast tomosynthesis imaging lesion localization device according to claim 1, characterized in that The feature pyramid module includes five input nodes, three intermediate nodes, and five output nodes; the intermediate node X1 is obtained by fusion of input node P7 and input node P6; the intermediate node X2 is obtained by fusion of input node P5 and intermediate node X1; the intermediate node X3 is obtained by fusion of input node P4 and intermediate node X2; the output node Y1 is obtained by fusion of input node P7 and output node Y2; the output node Y2 is obtained by fusion of input node P6, intermediate node X1 and output node Y3; the output node Y3 is obtained by fusion of input node P5, intermediate node X2 and output node Y4; the output node Y4 is obtained by fusion of input node P4, intermediate node X3 and output node Y5; the output node Y5 is obtained by fusion of input node P3 and intermediate node X3; thus, the planar structure formed by the stacking of BiFPN is top to bottom, then bottom to top, and then repeated continuously.

Images