A method for processing images of structurally distorted lesions in digital breast three-dimensional tomography

Abstract

The present invention discloses a digital breast three-dimensional tomography structural distortion lesion image processing method, comprising: obtaining a breast DBT image training set, preprocessing the breast DBT image training set; obtaining input data according to the breast DBT image training set; detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region; aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence; detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result. A large amount of medical imaging data is used for training to obtain an image processing and detection classification model with excellent performance, which is used for the detection of breast DBT structural distortion and the benign and malignant detection classification, and can quickly and accurately perform image processing and detection on structural distortion lesions in breast DBT images, improve the detection rate of typical and atypical structural distortions, and help doctors find structural distortion lesions more quickly and accurately.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an automatic lesion detection and image processing method, and in particular to an image processing method for digital breast three-dimensional tomography structural distortion lesion. Background Art

[0002] Screening and prevention are key to reducing the incidence of breast cancer.

[0003] Architectural distortion is a subtle mammographic finding. According to the fifth edition of the American College of Radiology Reporting and Data System Guidelines, architectural distortion refers to radiating shadows from one point, linear burrs originating from the same point and radiating, with no clear mass in the center, or focal contraction, distortion of the parenchymal margins, or postoperative changes, and may be accompanied by signs (which may be accompanied by masses, asymmetry, and calcification). Architectural distortion can be caused by a variety of malignant or non-malignant causes, and the positive predictive value of malignancy is about 75%. Clinically, a variety of breast diseases may show signs of architectural distortion, such as sclerosing adenosis, cystic hyperplasia of the breast, postoperative scars, breast inflammation, and malignant lesions (invasive ductal carcinoma, invasive lobular carcinoma, ductal carcinoma in situ, etc.). Architectural distortion may be the earliest manifestation of breast cancer and one of the most common abnormal signs missed in false-negative mammography.

[0004] Digital Breast Tomosynthesis (DBT) uses multi-angle imaging to reduce breast tissue overlap and reduce the impact of tissue structure noise. Based on multi-angle imaging, the ability to detect and locate lesions is improved. Compared with full-field digital mammography (FFDM), DBT is a newer imaging technology that has been shown to improve cancer detection rates and reduce screening recall rates. The use of DBT alone improves the detection of structurally distorted lesions, but it also increases false positives. The combined use of FFDM can reduce the false positive rate and improve the diagnostic accuracy of doctors. However, the addition of DBT steps to the screening process increases the risk of false positives, leading to more patients being recalled, which potentially reduces the screening efficiency due to too many false positive recalls. The risk of abnormal malignant tumors detected by DBT alone is more significant, reportedly close to 50%, but this also has its shortcomings. DBT has many imaging layers, which inevitably increases the daily workload of imaging doctors, increases the possibility of reading fatigue, and may cause missed detection of structurally distorted lesions, reducing the detection rate. It can be seen that there is currently a lack of image processing methods for structurally distorted lesions in digital breast three-dimensional tomography. Summary of the invention

[0005] The purpose of the present invention is to address the above-mentioned problems. The present invention proposes a digital breast three-dimensional tomography structural distortion lesion image processing method, which can effectively solve the problems of low detection rate and high screening recall rate of digital breast three-dimensional tomography structural distortion lesions.

[0006] The technical solution adopted by the present invention is:

[0007] In a first aspect, an embodiment of the present invention provides a method for processing a digital breast three-dimensional tomography structural distortion lesion image, the method comprising:

[0008] Obtaining a breast DBT image training set, and preprocessing the breast DBT image training set;

[0009] According to the breast DBT image training set, input data is obtained; the input data is the original grayscale DBT layer image, the Gabor amplitude map, the x component of the Gabor direction field, the y component of the Gabor direction field, and the convergence map;

[0010] Detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region;

[0011] Aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence;

[0012] The three-dimensional candidate sequence is detected to obtain a benign or malignant classification result.

[0013] Furthermore, the step of obtaining a breast DBT image training set and preprocessing the breast DBT image training set specifically includes:

[0014] The breast DBT image is cropped to remove the air area that does not contain breast tissue in the image to obtain a cropped image;

[0015] Annotating the cropped image to obtain an annotated image;

[0016] Marking the edge of the pectoral muscle of the annotated image to obtain a marked image;

[0017] Flipping the marked image;

[0018] The marked image is processed by Gabor filtering and convergence calculation.

[0019] Furthermore, the step of annotating the cropped image to obtain an annotated image specifically includes:

[0020] Each structural distortion lesion layer of the cropped image and the upper two layers and the lower two layers of the structural distortion lesion layer are labeled to obtain a labeled image.

[0021] Furthermore, the step of marking the edge of the pectoral muscle of the annotated image to obtain the marked image specifically includes:

[0022] For each medial-lateral oblique DBT layer in the annotated image, a straight line is used to mark the edge of the pectoralis muscle of the layer with the largest breast area in the tomographic image to obtain a marked image.

[0023] Furthermore, the step of flipping the marked image specifically includes:

[0024] All right breast DBT images are flipped so that the breast areas in all the DBT images are located on the left side.

[0025] Furthermore, the step of detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region specifically includes:

[0026] By performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model, an image with a two-dimensional candidate region is obtained; the Fast-R-CNN architecture includes a feature extractor, a region proposal network (RPN) and an R-CNN head; the R-CNN head includes three fully connected layers, an activation function with a ReLU, a classification branch and a regression branch; the two-dimensional candidate region is a bounding box with a probability score of a structural distortion lesion.

[0027] Furthermore, the step of performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model to obtain an image with a two-dimensional candidate region includes:

[0028] The original grayscale DBT level image and all preprocessing results are combined and input into the model;

[0029] Five input channels are designed for the Fast-R-CNN architecture model; the original grayscale DBT layer image is input into the first channel, and the first channel is normalized by the z-score method; the Gabor amplitude map is set to be input into the second channel, and the second channel is processed by using histogram equalization and z-score normalization; the x component of the Gabor direction field is input into the third channel; the y component of the Gabor direction field is input into the fourth channel; the convergence map is input into the fifth channel, and the value range is 0 to 1;

[0030] Set the preselected area in the DBT image;

[0031] Processing the preselected area in the DBT image by a non-maximum suppression method;

[0032] For each of the pre-selected regions, the regions of interest are aggregated into a 7×7 feature map;

[0033] The classification branch gives a structural distortion probability score for the preselected region, and the regression branch adjusts a bounding box for the preselected region; the bounding box is named using a probability and is set as a score for a possible structural distortion region.

[0034] Furthermore, the step of performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model to obtain an image with a two-dimensional candidate region further includes:

[0035] Setting a probability deviation degree for each continuous layer of the preselected region, and calculating the metric value by the standard deviation of the probability scores based on the value close to the maximum response;

[0036] The structural distortion lesions were located within the fibroglandular tissue range by three methods: fibroglandular ratio measurement, position quantile, and distance from the candidate area to the pectoralis major muscle, thus reducing the false positive rate of structural distortion detection.

[0037] The feature extractor includes a deformable convolution, which adjusts the size and shape of the convolution kernel according to a learnable offset;

[0038] Get the feature map through regular convolution;

[0039] Using deformable convolution instead of regular convolution in the feature extractor to obtain more representative structural distortion image lesion features;

[0040] The R-CNN head identifies whether the pre-selected region is a true structural distortion lesion or a negative sample;

[0041] An image with two-dimensional candidate regions is obtained.

[0042] Furthermore, the step of aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence specifically includes:

[0043] The image with the two-dimensional candidate regions is subjected to two-dimensional structural distortion candidate region aggregation by a spatial clustering method based on density features, and the layers of adjacent candidate regions in the upper and lower directions are superimposed to obtain a three-dimensional sequence;

[0044] Connecting blank layers between candidate regions through three-dimensional opening and closing operations, and removing three-dimensional candidate regions that only occupy one or two layers in the three-dimensional sequence;

[0045] The three-dimensional bounding box of each candidate region and the two-dimensional plane of the subsequent region are scored with maximum probability to obtain the three-dimensional candidate sequence in the three-dimensional sequence.

[0046] Furthermore, the step of detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result specifically includes:

[0047] Using the ResNet 101 network, the structural distortion lesions are classified into benign and malignant types to obtain a benign and malignant classification result;

[0048] The main branch of the ResNet 101 network residual structure includes two 3x 3 convolutional layers, and the convolutional layers used in the main branch include the first convolutional layer, the second convolutional layer, and the third convolutional layer;

[0049] The first convolution layer is a 1x 1 convolution layer, which is used to compress the channel dimension; the second convolution layer is a 3x 3 convolution layer; the third convolution layer is a 1x 1 convolution layer, which is used to restore the channel dimension;

[0050] The dotted residual structure corresponding to the residual structure has a 1x1 convolution layer on the shortcut branch, and the number of convolution kernels is the same as the number of convolution kernels of the third convolution layer on the main branch.

[0051] In the second aspect, an embodiment of the present invention also provides an implementation device for a method for processing a digital breast three-dimensional tomography structural distortion lesion image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the implementation method for a hardware Trojan based on the temperature fluctuation principle as described in the first aspect above is implemented.

[0052] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the digital breast three-dimensional tomography structural distortion lesion image processing method as described in the first aspect above when executed by the processor.

[0053] The beneficial effects of the present invention are as follows: the present invention uses a large amount of medical imaging data for training to obtain an image processing and detection classification model with excellent performance, which is used for the detection of breast DBT structural distortion and benign and malignant detection. In clinical practice, the present invention can quickly and accurately perform image processing and detection of structural distortion lesions in breast DBT images, and provide important reference value for clinicians, thereby improving the detection rate of typical and atypical structural distortions, and can help doctors find structural distortion lesions more quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a method for processing a digital breast three-dimensional tomography structural distortion lesion image provided by an embodiment of the present invention;

[0055] Figure 2 The original layer sample, Gabor amplitude, Gabor direction field, and convergence graph provided by the embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of probability deviation calculation provided by an embodiment of the present invention;

[0057] Figure 4(a)-Figure 4(c) This is a diagram showing the effect of reducing the false positive rate of structural distortion detection by using anatomical prior knowledge provided by an embodiment of the present invention;

[0058] FIG5( a ) is a schematic diagram of a conventional convolution provided by an embodiment of the present invention;

[0059] FIG5( b ) is a schematic diagram of a deformable convolution provided by an embodiment of the present invention;

[0060] FIG5( c ) is a schematic diagram of a feature extractor provided in an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of RPN workflow and anchor points provided by an embodiment of the present invention;

[0062] Figure 7(a)-Figure 7(c) 7 is a schematic diagram of FROC curves of deformable and conventional convolution in 10-fold cross validation provided by an embodiment of the present invention; FIG. 7(a) is all independent test cases; FIG. 7(b) is a typical structural distortion and negative sequence; FIG. 7(c) is an atypical structural distortion and negative sequence;

[0063] Figure 8 It is a schematic diagram of the ROC curve of ResNet provided by an embodiment of the present invention for distinguishing benign and malignant structural distortions on an independent test set. DETAILED DESCRIPTION

[0064] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0065] Considering the need to improve the efficiency and accuracy of image detection and classification of digital breast 3D tomography structural distortion lesions, the digital breast 3D tomography structural distortion lesion image processing method of the embodiment of the present invention can automatically identify and locate various structures and abnormalities in the image, can classify the image according to different features and patterns, and support convenient storage and management of the image. It can automatically extract and mark important information in the image and store it in a structured database, which is convenient for doctors and researchers to query and review.

[0066] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0067] In a first aspect, an embodiment of the present invention provides a method for processing a digital breast three-dimensional tomography structurally distorted lesion image.

[0068] This embodiment discloses a method for processing digital breast three-dimensional tomography structural distortion lesion images, which can improve the detection rate of the image processing model for structural distortion lesions. Specifically, the method includes but is not limited to all the following steps.

[0069] The embodiment of the present invention discloses a method for processing a digital breast three-dimensional tomography structural distortion lesion image. Specifically, referring to Figure 1 , the method comprising:

[0070] S100, obtaining a breast DBT image training set, and preprocessing the breast DBT image training set;

[0071] S200, obtaining input data according to a breast DBT image training set; the input data includes an original grayscale DBT layer image, a Gabor amplitude map, an x ​​component of a Gabor direction field, a y component of a Gabor direction field, and a convergence map;

[0072] S300, detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region;

[0073] S400, aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence;

[0074] S500, detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result.

[0075] It can be understood that the embodiment of the present invention proposes a new structural distortion detection and classification model based on a deep learning method, which can solve the problem of low detection and classification capabilities of existing image processing models.

[0076] It can be understood that the detection model of structural distortion mainly includes two modules: two-dimensional detection and three-dimensional aggregation.

[0077] This embodiment specifically discloses the step S100 of obtaining a breast DBT image training set and preprocessing the breast DBT image training set, which specifically includes:

[0078] S110, cropping the breast DBT image to remove air areas that do not contain breast tissue in the image, thereby obtaining a cropped image;

[0079] S120, annotating the cropped image to obtain an annotated image;

[0080] S130, marking the edge of the pectoral muscle of the annotated image to obtain a marked image;

[0081] S140, flipping the marked image;

[0082] S150 , processing the marked image through Gabor filtering and convergence calculation.

[0083] Understandably, reference Figure 2 The embodiment of the present invention needs to preprocess the DBT image, and completes the preliminary image preprocessing work through Gabor filtering and convergence calculation. The Gabor filter is a good line extractor, which represents the product of a two-dimensional Gaussian function and an oscillation. The mathematical formula describing the Gabor kernel is as follows:

[0084]

[0085] x′=xcosγ-ysinγ, y′=xsinγ+ycosγ

[0086] Where (x, y) is the pixel coordinate within the kernel; σx and σy are the standard deviations of the Gaussian function; σ is the wavelength of the oscillation; λ is the direction parameter that controls the direction of the pre-transfer; σx, σy, and λ simultaneously determine the width and length of the object line; the cosine function in formula (1) is converted to a sine function to obtain the imaginary part of the Gabor filter kernel; in each layer, the distribution of fibroglandular tissue along different directions must be considered; therefore, the parameter λ is obtained from 0°, 1°, 2°, ..., 179°, 180°. In this way, the filtering result can be obtained by using Gγ(u,v) to represent the filtering result with the coordinate (u,v) direction as λ, and the Gabor amplitude M can be obtained.

[0087] This embodiment specifically discloses the step S120 of annotating the cropped image to obtain an annotated image, which specifically includes:

[0088] S121 , annotate each structural distortion lesion layer of the cropped image and the upper two layers and the lower two layers of the structural distortion lesion layer to obtain an annotated image.

[0089] It can be understood that each structural distortion lesion is marked with a rectangular frame at the level where the structural distortion lesion is most obvious, as well as the two layers above and the two layers below it, that is, a total of five consecutive image levels are marked with the location information of the structural distortion.

[0090] This embodiment specifically discloses the step S130 of marking the edge of the pectoral muscle of the annotated image to obtain the marked image, which specifically includes:

[0091] S131. For each medial-lateral oblique DBT layer in the annotated image, use a straight line to mark the edge of the pectoralis muscle at the layer with the largest breast area in the tomographic image to obtain a labeled image.

[0092] It will be appreciated that for each medial-lateral oblique DBT slice, a straight line is used to mark the edge of the pectoralis muscle at the slice with the largest breast area in the tomographic image.

[0093] This embodiment specifically discloses the step S140 of flipping the marked image, which specifically includes:

[0094] S141. Flip all right breast DBT images so that the breast areas in all DBT images are located on the left side.

[0095] This embodiment specifically discloses the step S300 of detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region, which specifically includes:

[0096] S310. Perform two-dimensional detection on the marked image based on the Fast-R-CNN architecture model to obtain an image with two-dimensional candidate regions; the Fast-R-CNN architecture includes a feature extractor, a region proposal network (RPN) and an R-CNN head; the R-CNN head includes three fully connected layers with a ReLU activation function, a classification branch and a regression branch; the two-dimensional candidate region is a bounding box with a probability score of a structural distortion lesion.

[0097] It can be understood that 2D detection is based on a Fast-R-CNN architecture, which includes three submodules: feature extractor, region proposal network (RPN) and R-CNN head.

[0098] This embodiment specifically discloses the step of performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model in step S310 to obtain an image with two-dimensional candidate regions, including:

[0099] S311, merging the original grayscale DBT layer image and all preprocessing results into the model;

[0100] S312. Design five input channels for the Fast-R-CNN architecture model; input the original grayscale DBT layer image into the first channel, and normalize the first channel by the z-score method; input the Gabor amplitude map into the second channel, and process the second channel by using histogram equalization and z-score normalization; input the x component of the Gabor direction field into the third channel; input the y component of the Gabor direction field into the fourth channel; input the convergence map into the fifth channel, with a value range of 0 to 1;

[0101] S313, setting a pre-selected area in the DBT image;

[0102] S314, processing the preselected area in the DBT image by a non-maximum suppression method;

[0103] S315, for each pre-selected region, aggregate the regions of interest into a 7×7 feature map;

[0104] S316. The classification branch gives a structural distortion probability score for the pre-selected region, and the regression branch adjusts the bounding box of the pre-selected region; the bounding box is named using the probability and is set as the score of the possible structural distortion region.

[0105] Understandably, Figure 2 Examples of original layers, Gabor amplitude, Gabor orientation field, and convergence plot. Arrows show the layers of the location structure warp (structure warp). The Gabor filter parameters used in this example are A = 16.σy = 8.9948. σy = 17.9895. y = 45°. Convergence is calculated within a ring with an outer radius of 200 pixels and an inner radius of 60 pixels. Since the structure warp has a typical radial pattern, it can be seen in the convergence plot that the center of the structure warp has a higher response.

[0106] It can be understood that the original gray level and all preprocessing results are combined and input into the model; a five-channel input is designed for the model. The first channel is the original gray level DBT level image, which is normalized using the z-score method; the second channel is the Gabor amplitude map, which is processed using histogram equalization and z-score normalization; the third and fourth channels are the x-component and y-component of the Gabor direction field, respectively; the last channel is the convergence map, with a value range of 0 to 1;

[0107] Since the pre-selected regions overlap each other in the image, the non-maximum suppression method is used, with 0.7 as the intersection exceeding the joint threshold;

[0108] The R-CNN head consists of three fully connected layers with a ReLU activation function and two branches: one for classification and the other for regression.

[0109] For each pre-selected region, a region of interest is aggregated into a 7×7 feature map;

[0110] The classification branch gives a probability score of structural distortion of the pre-selected region, and the regression branch adjusts the bounding box of the pre-selected region; the probability is used to name such a bounding box as the score of the "possible region of structural distortion".

[0111] This embodiment specifically discloses the step S310 of performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model to obtain an image with two-dimensional candidate regions, and further includes:

[0112] S317, setting a probability deviation degree for the continuous level of each pre-selected region, and calculating the metric value by the standard deviation of the probability scores based on the close maximum response value;

[0113] S318, by using three methods, namely, fibroglandular ratio measurement, position quantile, and distance from the candidate area to the pectoralis major muscle, the structural distortion lesion is located within the fibroglandular tissue range, thus reducing the false positive rate of structural distortion detection;

[0114] S319, the feature extractor includes a deformable convolution, which adjusts the size and shape of the convolution kernel according to the learnable offset;

[0115] S31A, obtain feature map through regular convolution;

[0116] S31B, using deformable convolution instead of regular convolution in the feature extractor to obtain more representative features of lesions in structurally distorted images;

[0117] S31C, R-CNN head identifies whether the pre-selected region is a true structural distortion lesion or a negative sample;

[0118] S31D, obtain an image with a two-dimensional candidate region.

[0119] It is understandable that, referring to Figure 3 , a probability deviation measure is defined for each continuous layer of candidate object regions, which is calculated based on the standard deviation of the probability scores close to the maximum response value.

[0120] A threshold (0.1) was set, and some candidate object regions that only obtained high probability scores in one or two layers were determined to be false positive sequences;

[0121] It is understandable that, referring to Figure 4(a)-Figure 4(c), which is an effect diagram of reducing the false positive rate of structural distortion detection through anatomical prior knowledge; among them, Figure 4(a) is a schematic diagram of breast anatomy, with the pectoralis muscle line as the rectangular coordinate system, establishing a vertical line from the pectoralis major to the nipple, and the pectoralis major as the coordinate axis; in Figure 4(b), a schematic diagram of the distance calculation from the candidate building or building deformation is the center point of the pectoralis muscle; in Figure 4(c), the position quantile calculation in the up and down directions, the candidate quantile digital breast tomography (DBT) sequence contains n layers, and the position quantile of layer i is (iin).

[0122] Understandably, reference Figure 4(a)-Figure 4(c) , through three methods, namely, the fibroglandular ratio measurement method, the position quantile, and the distance from the candidate area to the pectoralis major muscle, the structural distortion lesion is located within the fibroglandular tissue, rather than tissues such as fat, skin, and pectoralis major muscle; assuming that a straight line is used to represent the edge of the pectoralis major muscle, then a rectangular coordinate system is composed of vertical lines from the nipple to the pectoralis major muscle. The axes of the coordinate system are named muscle axis and nipple axis respectively; that is, the false positive rate of structural distortion detection is reduced through anatomical prior knowledge.

[0123] It is understandable that, referring to Figure 5(a)-Figure 5(c) , respectively, are schematic diagrams of conventional convolution, deformable convolution, and the feature extractor used in the present invention, wherein ReLU: rectified linear unit activation function; H: height of input image; W: width of input image.

[0124] It is understandable that, referring to Figure 5(a)-Figure 5(c) , the embodiment of the present invention designs and adds a more flexible feature extractor with adaptive receptive field, the deformable convolution contained therein can automatically adjust the size and shape of the kernel according to the learnable offset;

[0125] It is understandable that, referring to Figure 5(a)-Figure 5(c) , a 3×3-convolution kernel is proposed to reflect the deformable convolution mechanism; relative to the center of the convolution kernel, the coordinates of the convolution kernel elements can be expressed as the set (1, 1)}; For pixel p0 in feature map I, the next feature map y(p0) is obtained by regular convolution; the calculation process of y(p0) is:

[0126]

[0127] The core idea of ​​deformable convolution is to use a branch to calculate the offset and modify the size and shape of the convolution kernel. First, the input feature map I is convolved through a regular 3×3 convolution layer to generate an offset map with the same height and width as the input map and containing 18 channels. These 18 values ​​represent the horizontal and vertical offsets of the 9 elements at each position in the kernel. The offset at pixel p0 is represented by the set Roff. The Rof of the set f can be written as the set {(Δx1, Δy1), (Δx2, Δy2), …, (Δx9, Δy9)}. Subsequently, the convolution result of the deformable kernel at pixel p0 is:

[0128]

[0129] Where pn∈R, Δpn∈Rof f set; deformable convolution relies on the regular convolution branch to predict the offset of each element in the convolution kernel; this offset branch is part of the network, and its weight value is learned synchronously with other weight values ​​of the network during training; in the deformable convolution design, there is no explicit constraint on the offset value; however, since the offset is output by a regular convolution branch, the receptive field of this branch is limited; therefore, the value of the offset may be subject to implicit constraints; because the offset can be a continuous value, the coordinate p0+pn+Δpn may not be an integer; so bilinear interpolation is performed to obtain I(p0+pn+Δpn); in addition, different features lead to different offsets at different positions, and deformable convolution is located at a specific position; the input and output sizes of deformable convolution are the same as regular convolution; therefore, deformable convolution can be used instead of regular convolution in the feature extractor to obtain more representative structural distortion image lesion features;

[0130] The 3×3-regular convolution kernel in the last layer (layer 13) of the original feature extractor is replaced by a 3×3-deformed convolution kernel.

[0131] It is understandable that, referring to Figure 6 , a region proposal network (RPN) is added to automatically generate ROIs for the detection network, including sliding windows, intermediate layers, classification layers, regression layers, and anchors;

[0132] The feature extractor generates a 512-channel feature map in the previous step;

[0133] A sliding window of size 3×3 slides on the feature map;

[0134] The initial ROI is called an "anchor" in RPN. At each sliding position, nine anchors are determined with different aspect ratios and sizes. For each anchor, an aspect ratio of [0.5, 1, 2] and a size of [8, 16, 32] are selected in the feature map.

[0135] The middle layer is a convolutional layer with a kernel size of 3×3, followed by a ReLU activation function. This layer uses the features in the current sliding window as input and outputs a new feature map of size 1×1×512;

[0136] The R-CNN head is used to identify whether the pre-selected region is a real structural distortion lesion or a negative sample;

[0137] First, perform ROI pooling on each pre-selected region to generate a fixed-size feature map of 7×7×512;

[0138] Two fully connected layers reduce the feature dimension from 7×7×512 to 4096;

[0139] Two parallel fully connected layers classify whether the bounding box constitutes a real structural distortion lesion, so as to adjust the bounding box of potential structural distortion if necessary;

[0140] Finally, this bounding box with the probability score of the structure-distorted lesion is a 2D region proposal.

[0141] This embodiment specifically discloses the step S400 of aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence, which specifically includes:

[0142] S410, performing two-dimensional structural distortion candidate region aggregation on an image having two-dimensional candidate regions by using a spatial clustering method based on density features, and superimposing layers of adjacent candidate regions in upper and lower directions to obtain a three-dimensional sequence;

[0143] S420, connecting the blank layers between the candidate regions through a three-dimensional opening and closing operation, and removing the three-dimensional candidate regions that only occupy one or two layers in the three-dimensional sequence;

[0144] S430 , performing maximum probability scoring on the three-dimensional bounding box of each candidate region and the two-dimensional plane of the subsequent region to obtain a three-dimensional candidate sequence in the three-dimensional sequence.

[0145] It can be understood that the above steps S300-S31D generate a large number of independent pre-selected structural distortion lesion areas in the two-dimensional layers of the DBT sequence images; however, since DBT is a three-dimensional imaging technology, it is necessary to identify the pre-selected structural distortion areas on the three-dimensional continuous layers;

[0146] Using a density-based spatial clustering method, all candidate regions of 2D structural distortion in the same DBT sequence image are aggregated based on the center point of the 2D image. Subsequently, the layers of adjacent candidate regions in the up-down direction are superimposed to obtain multiple 3D sequences.

[0147] Perform 3D opening and closing operations to connect blank layers between candidate regions and remove 3D candidate regions that only occupy one or two layers in the DBT sequence;

[0148] The 3D bounding box of each candidate region and the maximum probability score on the 2D plane of the subsequent region are calculated, which form a 3D candidate sequence in the DBT sequence.

[0149] This embodiment specifically discloses the step S500 of detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result, which specifically includes:

[0150] S510, classifying the structurally distorted lesions into benign and malignant ones through the ResNet 101 network to obtain a benign and malignant classification result;

[0151] The main branch of the ResNet 101 network residual structure includes two 3x 3 convolutional layers. The convolutional layers used in the main branch include the first convolutional layer, the second convolutional layer, and the third convolutional layer;

[0152] The first convolution layer is a 1x 1 convolution layer, which is used to compress the channel dimension; the second convolution layer is a 3x 3 convolution layer; the third convolution layer is a 1x 1 convolution layer, which is used to restore the channel dimension;

[0153] The dotted residual structure corresponding to the residual structure has a 1x1 convolution layer on the shortcut branch, and the number of convolution kernels is the same as the number of convolution kernels of the third convolution layer on the main branch.

[0154] It is understandable that, referring to Figure 7(a)-Figure 7(c) , which is a schematic diagram of the FROC curves of deformable and conventional convolution in the 10-fold cross validation provided by an embodiment of the present invention, wherein FIG7(a) shows all independent test cases; FIG7(b) shows typical structural distortions and negative sequences; and FIG7(c) shows atypical structural distortions and negative sequences. The embodiment of the present invention uses a structural distortion detection model to detect typical and atypical structural distortions.

[0155] It can be understood that the present invention embeds a trained ResNet101 network after the structural distortion detection model to classify the detected structural distortion lesions into benign and malignant ones;

[0156] The residual structure main branch of the network is composed of two 3x 3 convolutional layers. The main branch uses three convolutional layers. The first is a 1x 1 convolutional layer to compress the channel dimension, the second is a 3x 3 convolutional layer, and the third is a 1x 1 convolutional layer to restore the channel dimension. The dotted residual structure corresponding to this residual structure also has a 1x1 convolutional layer on the shortcut branch. Its number of convolutional kernels is the same as that of the third convolutional layer on the main branch. Note the stride of each convolutional layer.

[0157] It is understandable that, referring to Figure 8 , the network can finally output the benign or malignant classification results of the structurally distorted lesions in the image.

[0158] It can be understood that the system includes a detection model and a diagnosis model for breast DBT structural distortion; the detection model is used to accurately detect typical structural distortion and atypical structural distortion in breast DBT images; the diagnosis model is used to analyze the structural distortion in the detection results and output the benign or malignant diagnosis results of the lesions.

[0159] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0160] The present invention uses a large amount of medical imaging data for training to obtain a detection and classification model with excellent performance, which is used for the detection of breast DBT structural distortion and the classification of benign and malignant lesions. It can quickly and accurately detect and classify structural distortion lesions in breast DBT images, and provide important reference value for clinicians, improve the detection rate of typical and atypical structural distortions, and detect structural distortions and classify them more quickly and accurately.

[0161] In a second aspect, an embodiment of the present invention further provides a device for implementing a method for processing a digital breast three-dimensional tomography structurally distorted lesion image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for processing a digital breast three-dimensional tomography structurally distorted lesion image as described in the first aspect above is implemented.

[0162] The processor and the memory can be connected via a bus or other means. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The non-transitory software program and instructions required to implement the digital breast three-dimensional tomography structure distortion lesion image processing method of the above embodiment are stored in the memory, and when executed by the processor, the digital breast three-dimensional tomography structure distortion lesion image processing method of the above embodiment is executed.

[0164] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor, which, when executed by the processor, is used to execute the digital breast three-dimensional tomography structural distortion lesion image processing method as described in the first aspect above.

[0165] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0166] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for processing digital breast three-dimensional tomography structural distortion lesion images, characterized in that: The digital breast three-dimensional tomography structural distortion lesion image processing method comprises: Obtaining a breast DBT image training set, and preprocessing the breast DBT image training set; According to the breast DBT image training set, input data is obtained; the input data is the original grayscale DBT layer image, the Gabor amplitude map, the x component of the Gabor direction field, the y component of the Gabor direction field, and the convergence map; Detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region; Aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence; Detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result; The step of obtaining a breast DBT image training set and preprocessing the breast DBT image training set specifically includes: The breast DBT image is cropped to remove the air area that does not contain breast tissue in the image to obtain a cropped image; Annotating the cropped image to obtain an annotated image; Marking the edge of the pectoral muscle of the annotated image to obtain a marked image; Flipping the marked image; Processing the marked image by Gabor filtering and convergence calculation; The step of annotating the cropped image to obtain an annotated image specifically includes: Annotating each structural distortion lesion layer of the cropped image and two upper layers and two lower layers of the structural distortion lesion layer to obtain an annotated image; The step of marking the edge of the pectoral muscle of the annotated image to obtain the marked image specifically includes: For each DBT layer in the annotated image at the inner and outer oblique positions, a straight line is used to mark the edge of the pectoralis muscle at the layer with the largest breast area in the tomographic image to obtain a marked image; The step of detecting the three-dimensional candidate sequence to obtain a benign or malignant classification result of the structurally distorted lesion specifically includes: The structural distortion lesions are classified into benign and malignant types by using a ResNet101 network to obtain a benign and malignant classification result; The main branch of the ResNet101 network residual structure includes two 3x3 convolutional layers, and the convolutional layers used in the main branch include the first convolutional layer, the second convolutional layer, and the third convolutional layer; the first convolutional layer is a 1x1 convolutional layer, which is used to compress the channel dimension; the second convolutional layer is a 3x3 convolutional layer; the third convolutional layer is a 1x1 convolutional layer, which is used to restore the channel dimension; the dotted line residual structure corresponding to the residual structure has a 1x1 convolutional layer on the shortcut branch, and the number of convolutional kernels is the same as the number of convolutional kernels of the third convolutional layer on the main branch; The step of performing two-dimensional detection on the labeled image based on the Fast-R-CNN architecture model to obtain an image with two-dimensional candidate regions includes: setting a pre-selected region in the DBT image, setting a probability deviation degree for each continuous layer of the pre-selected region, and calculating a metric value by a standard deviation of probability scores close to a maximum response value; The structural distortion lesions were located within the fibroglandular tissue by three methods: fibroglandular ratio measurement, position quantile, and distance from the candidate area to the pectoralis major muscle, thereby reducing the false positive rate of structural distortion detection.

2. The method for processing digital breast three-dimensional tomography structural distortion lesion images according to claim 1, characterized in that: The step of flipping the marked image specifically includes: All right breast DBT images are flipped so that the breast areas in all the DBT images are located on the left side.

3. The method for processing digital breast three-dimensional tomography structural distortion lesion images according to claim 1, characterized in that: The step of detecting the probability of structural distortion lesions on the input data to obtain a two-dimensional candidate region specifically includes: By performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model, an image with a two-dimensional candidate region is obtained; the Fast-R-CNN architecture includes a feature extractor, a region proposal network (RPN) and an R-CNNhead; the R-CNNhead includes three fully connected layers, an activation function of ReLU, a classification branch and a regression branch; the two-dimensional candidate region is a bounding box with a probability score of a structural distortion lesion.

4. The method for processing digital breast three-dimensional tomography structural distortion lesion images according to claim 3, characterized in that: The step of performing two-dimensional detection on the marked image based on the Fast-R-CNN architecture model to obtain an image with a two-dimensional candidate region includes: The original grayscale DBT level image and all preprocessing results are combined and input into the model; Five input channels are designed for the Fast-R-CNN architecture model; the original grayscale DBT layer image is input into the first channel, and the first channel is normalized by the z-score method; the Gabor amplitude map is set to be input into the second channel, and the second channel is processed by using histogram equalization and z-score normalization; the x component of the Gabor direction field is input into the third channel; the y component of the Gabor direction field is input into the fourth channel; the convergence map is input into the fifth channel, and the value range is 0 to 1; Processing the preselected area in the DBT image by a non-maximum suppression method; For each of the pre-selected regions, the regions of interest are aggregated into a 7×7 feature map; The classification branch gives a structural distortion probability score for the preselected region, and the regression branch adjusts a bounding box for the preselected region; the bounding box is named using a probability and is set as a score for a possible structural distortion region.

5. The method for processing digital breast three-dimensional tomography structural distortion lesion images according to claim 4, characterized in that: Also includes: Setting a probability deviation degree for each continuous layer of the preselected region, and calculating the metric value by the standard deviation of the probability scores based on the value close to the maximum response; The structural distortion lesion is located within the range of the fibroglandular tissue by three methods: fibroglandular ratio measurement, position quantile, and distance from the candidate area to the pectoralis major muscle, thereby reducing the false positive rate of structural distortion detection; the feature extractor includes a deformable convolution, which adjusts the size and shape of the convolution kernel according to the learnable offset; Get the feature map through regular convolution; Using deformable convolution instead of regular convolution in the feature extractor to obtain more representative structural distortion image lesion features; The R-CNNhead identifies whether the pre-selected region is a true structural distortion lesion or a negative sample; An image with two-dimensional candidate regions is obtained.

6. The method for processing digital breast three-dimensional tomography structural distortion lesion images according to claim 1, characterized in that: The step of aggregating the two-dimensional candidate regions to obtain a three-dimensional candidate sequence specifically includes: The image with the two-dimensional candidate regions is subjected to two-dimensional structural distortion candidate region aggregation by a spatial clustering method based on density features, and the layers of adjacent candidate regions in the upper and lower directions are superimposed to obtain a three-dimensional sequence; Connecting blank layers between candidate regions through three-dimensional opening and closing operations, and removing three-dimensional candidate regions that only occupy one or two layers in the three-dimensional sequence; The three-dimensional bounding box of each candidate region and the two-dimensional plane of the candidate region are scored with maximum probability to obtain the three-dimensional candidate sequence in the three-dimensional sequence.

Images