X-ray image detection system based on shortcut connection pyramid structure

Abstract

An X-ray image detection system based on shortcut connection pyramid structure, comprising: an adaptive image adjustment module for receiving a multi-center molybdenum target image, a high-resolution feature extraction module, a multi-scale shortcut connection and weighted fusion module and a region recommendation and detection module, wherein: the adaptive adjustment module normalizes the histogram of the multi-center molybdenum target image, and adaptively clips the image into a uniform size; the high-resolution feature extraction module extracts i different scale feature maps C i of the image; the multi-scale shortcut connection and weighted fusion module transmits and weightedly fuses the features conducive to calcification detection to generate i multi-scale detection feature maps P i ; the region recommendation and detection module recommends candidate regions based on the above multi-scale feature maps P i , and further regresses the bounding box and classifies the target for the candidate regions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a calcification point detection system in X-ray photography image based on shortcut connection pyramid structure. BACKGROUND

[0002] Micro-calcification is the most important diagnostic indicator of breast cancer in mammography X-ray photography image (hereinafter referred to as "mammogram image"). The computer-aided detection system assists doctors in clinical diagnosis of patients by detecting benign calcification and malignant calcification in the mammogram image. However, the multi-center mammogram image gray distribution has deviation, the pre-processed image has detail loss, and the multi-type benign and malignant calcification morphology and size have great differences. The above problems make it difficult for the existing computer-aided detection system to accurately distinguish and detect multiple types of calcification.

[0003] The existing X-ray image detection technology based on pyramid structure does not optimize the multi-center data, and the detection technology has poor generalization; the image preprocessing adopts slice cutting based on the region of interest, and cannot realize the end-to-end full image detection function and the full image detection function; the network structure has the defects of feature transmission and fusion, and cannot realize the detection of multiple sizes and multiple target lesions. SUMMARY

[0004] The present application proposes an X-ray image detection system based on shortcut connection pyramid structure to solve the above problems in the prior art. The system eliminates multi-center deviation and increases effective pixels through an adaptive image adjustment function, extracts image features through a deep learning network, and realizes accurate target bounding box regression and target classification according to the obtained multi-scale fusion feature map.

[0005] The present application is realized by the following technical solutions:

[0006] The present application relates to an X-ray image detection system based on shortcut connection pyramid structure, comprising: an adaptive image adjustment module for receiving multi-center mammogram images, a high-resolution feature extraction module, a multi-scale shortcut connection and weighted fusion module, and a region recommendation and detection module. The adaptive adjustment module normalizes the histogram of the multi-center mammogram image and adaptively crops the image into a uniform size; the high-resolution feature extraction module extracts i different scale feature maps C i ; the multi-scale shortcut connection and weighted fusion module transmits and weightedly fuses the features conducive to calcification detection to generate i multi-scale detection feature maps P i ; and the region recommendation and detection module is based on the above multi-scale feature maps P iA candidate region is recommended, and further bounding box regression and object classification are performed for the candidate region.

[0007] Technical effects

[0008] Compared with the prior art, the adaptive image adjustment module of the present application maximizes the number of effective pixels while ensuring full image detection, reduces the consumption of computing resources, and can optimize the generalization ability of the system for multi-center data. The multi-scale shortcut connection and weighted fusion module improves the detection capability of the image and the precision of the detection frame. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The system schematic diagram of the present application is shown in the figure.

[0010] Figure 2 The workflow diagram of adaptive image adjustment is shown in the figure.

[0011] Figure 3 The residual unit schematic diagram is shown in the figure.

[0012] Figure 4 The region recommendation network schematic diagram is shown in the figure.

[0013] Figure 5 The shortcut connection and weighted fusion module schematic diagram is shown in the figure.

[0014] Figure 6 The visualization results of feature maps in each stage of feature extraction are shown in the figure.

[0015] Figure 7 The visualization results of different methods for detecting calcification in molybdenum target images in the embodiment are shown in the figure.

[0016] Figure 8 The visualization results of the performance of the shortcut connection structure in the embodiment are shown in the figure.

[0017] Figure 9 The performance comparison curve graph of the embodiment and other methods is shown in the figure. DETAILED DESCRIPTION

[0018] As shown in the figure, it is a specific application example of an X-ray image detection system based on a shortcut connection pyramid structure in the field of mammography X-ray photography images involved in the present embodiment. The system of the present application is not limited to be applied in this scene. Figure 1

[0019] ​The system includes: an adaptive image adjustment module for receiving multi-center molybdenum target images, a high-resolution feature extraction module, a multi-scale shortcut connection and weighted fusion module, and a region recommendation and detection module. The adaptive adjustment module performs histogram normalization on the multi-center molybdenum target images and adaptively crops the images to a uniform size. The high-resolution feature extraction module, including a residual module layer, extracts feature maps C at different scales of the images. i The multi-scale shortcut connection and weighted fusion module will transfer and weightedly fuse features that are beneficial for calcification detection, generating a multi-scale detection feature map P. i The region recommendation and detection module is based on the aforementioned multi-scale feature map P. i Candidate regions are recommended, and the recommended candidate regions are further regressed in bounding box and classified into targets using a region detection network.

[0020] like Figure 2 As shown, the Adaptive Image Adjustment (AIA) module includes an image normalization unit and an adaptive cropping unit. The image normalization unit automatically adjusts the grayscale histograms of other source images S based on the grayscale histogram distribution of the target image T set in the multi-center mammogram image. The adaptive cropping unit adaptively crops the mammogram image to a uniform size while ensuring that the breast structure remains unchanged.

[0021] The automatic adjustment refers to: when using multi-center data, designating one center as the target image T and the other centers as the source images S; calculating the cumulative distribution function (CDF) of the source image S and the target image T, respectively, F S (·) and F T (·); By finding the condition that satisfies F s (G s ) = F T (G T Grayscale G T And define the matching equation as S(G) s ) = G T For discrete CDF values, find the nearest F. s (G s The equation S(·) is applied to each pixel of the other source images S in the multicenter data to achieve the adjustment.

[0022] The adaptive clipping considers that in most mammography images, the breast always occupies one side of the image, so the clipping starts from the middle: when the breast area is greater than half of the image, the clipping window is moved; when the breast exceeds the center line and the main area of the breast is on the right side, the clipping window is moved to the left to ensure that the left edge of the clipping window is aligned with the edge of the breast and the size of the clipped image remains unchanged. The two images of the same size after adaptive clipping are trained or detected in the system.

[0023] The high-resolution feature extraction module comprises an initial convolution layer Conv1, a residual module layer Conv2(256, 3), and second to fourth convolution layers Conv3(512, 4), Conv4(1024, 23), and Conv5(2048, 3) arranged in sequence, wherein the initial convolution layer Conv1 is a convolution layer with an output channel of 64, and each residual module comprises the number of output channels and the number of residual units in the bracket.

[0024] As shown in Figure 3 , the residual module layer comprises a cascade of one 1x1 convolution layer, one 3x3 convolution layer, one 1x1 convolution layer, and a residual link channel, and the output channel ratio of the three convolution layers is 1:1:4, and the activation function is Relu (Rectified Linear Unit).

[0025] The different scale feature maps C i include feature maps C2, C3, C4, and C5 output by the residual module layer Conv2 and the second to fourth convolution layers Conv3-Conv5 in the high-resolution feature extraction module, respectively, wherein the size of the feature map C i is twice that of the feature map C i+1 . For example, the output feature map size of C2 is 8 times that of C5.

[0026] As shown in Figure 4 , the multi-scale shortcut connection and weighted fusion module comprises a multi-scale shortcut connection structure (MSC) and a trainable weighted fusion unit (TWF), wherein the multi-scale shortcut connection structure is connected to the high-resolution feature extraction module and performs up-sampling and fusion on the received different scale feature maps C i to effectively pass the edge information of the shallow layer or the spatial semantic information of the deep layer; and the trainable weighted fusion unit autonomously learns the weight w i of the feature map to be fused according to different network depths during the training process, and generates a multi-scale detection feature map P by weighted fusion with a nonlinear activation function according to the weight.i .

[0027] The trainable weighted fusion unit fuses the two-layer feature maps, specifically: Wherein: w is a trainable parameter during network training, and the max() operation is realized by the ReLU activation function to ensure that the contribution of the feature map is not negative during fusion.

[0028] The feature maps C' of different scales i and the detection feature map P i The weights of the detection feature map P are optimized based on the gradient descent method based on the loss function.

[0029] In this embodiment, the weights are quickly normalized, and a parameter ∈ = 0.001 is added to prevent potential instability during model training.

[0030] The trainable weighted fusion unit allows different levels of information to have different weights during fusion, which represents the tendency of detection feature maps of different depths to two types of calcification detection tasks, so that the generated multi-scale detection feature map can provide more accurate feature information for different types of calcification.

[0031] The upsampling and fusion include:

[0032] ①Uniform channel operation is performed on the different scale feature maps C2, C3, C4 and C5 generated by the high-resolution feature extraction module of each layer to generate corresponding feature maps C'2, C'3, C'4 and P5 for subsequent feature fusion or detection, wherein:

[0033] In this embodiment, a 1x1 convolution layer is used to unify the channel number to 512, which is the same as the output channel number of C3.

[0034] ②Up-sampling (Upsample) is performed on the lowest layer detection feature map P5 to make it have the same size as the C'4 of the shallower layer of the previous layer, and the trainable weighted fusion (Fusion) is performed with C'4.

[0035] ③Convolution operation (Conv) is used after fusion to reduce the aliasing effect after fusion to obtain shallower detection feature maps P2 = Conv(Fusion(C'2, Upsample(P3))) and P4 = Conv(Fusion(C'4, Upsample(P5))), wherein: The multi-scale shortcut connection structure includes a shortcut connection that skips P4 in the up-sampling process, i.e. the detection feature map P3 = Conv(Fusion(C'3, Upsample(P5))), and Fusion() is the fusion process, i.e. AsFigure 4 The shortcut connection breaks the direct connection between C4 and P4 to P3, aiming to reduce the deep feature information of C4 and only keep the spatial information from the deeper feature map P5.

[0036] As Figure 5 shown, for the tiny benign calcification, the detection needs accurate edge and position information, which are abundant in the shallow high-resolution feature maps, such as C2 and C3. For the clustered calcification, the detection relies on the spatial semantic information to ensure the complete detection of the whole clustered calcification area, which is abundant in the deep low-resolution feature maps, such as C4 and C5. Due to the superposition of deep features, the shallow high-resolution detection feature maps superimpose a lot of deep low-resolution spatial information, such as Figure 5 The changes in (a) to (g) make it difficult to detect tiny benign calcification. The shortcut connection effectively promotes the preservation of shallow features and the transmission of deep features in the fusion process, providing accurate positioning information for both benign and malignant calcification points.

[0037] The up-sampling operation in the embodiment adopts a bilinear interpolation method.

[0038] In the embodiment, a 3x3 convolution layer (Conv 3×3 ) is used to reduce the aliasing effect after fusion.

[0039] The region recommendation and detection module includes a region recommendation network for generating candidate regions and a region detection network for realizing target classification and boundary box regression based on the recommended regions.

[0040] The recommendation of the candidate regions refers to recommending 1000 candidate regions by sorting the detection feature maps based on multiple scales according to the probability of including the target calculated from the recommendation network, and pooling the candidate regions to a size of 7x7.

[0041] The region recommendation network in the embodiment includes a 3x3 convolution layer with Relu, two 1x1 convolution layers and a Softmax layer, as Figure 6 shown.

[0042] The region detection network comprises a 7*7 convolution layer, a classification task full connection layer and a regression task full connection layer, wherein the convolution layer changes the recommended region into a vector with a size of 1024; the classification task full connection layer changes the vector size into 3, and after a Softmax operation, respectively represents the probability of the candidate region being background, benign calcification and malignant calcification; the regression task full connection layer changes the vector size into 12, and is readjusted into 3 groups of vectors with a size of 4, and each group of vectors corresponds to 4 parameters of the boundary box of each class (background, benign calcification and malignant calcification). Finally, the class and position of the detection target are determined according to the two tasks.

[0043] In the training stage of the embodiment, the region recommendation network and the region detection network both use cross-entropy to train the classification task, and use a Smooth L1 loss function to train the boundary box regression task.

[0044] Specifically, the actual experiment is based on a multi-center dataset composed of internal data of Shanghai Jiaotong University Affiliated Renji Hospital and public data INbreast, and a total of 225 images, including 91 malignant calcifications and 226 benign calcifications. The clinical sub-dataset has a total of 598 images, including 1672 benign calcifications. The multi-center dataset is used to train the model, and the training set, the validation set and the test set are 150, 25 and 50 respectively. The clinical sub-dataset is used for migration test of the model. The original format of the image is DICOM, and the computer platform comprises an NVIDIA Tesla V100 SXM2 image processing unit (GPU). In the training stage, the network is optimized by using a momentum gradient descent method with a learning rate of 0.001 and a momentum parameter of 0.9.

[0045] To prove the improvement effect of the adaptive image adjustment module (AIA) on system resource consumption and multi-center data deviation, the performance of the Faster-RCNN network with FPN with / without the adaptive image adjustment module is compared on the multi-center dataset including the clinical dataset and INbreast, and the experimental results are shown in Table 1.

[0046] Table 1 Comparison of the embodiment with the baseline method and the improvement effect of the system on the overall performance.

[0047]

[0048] In the above table, IM represents benign calcification, and CM represents malignant calcification. mAP (mean average precision) in the embodiment refers to the average value of the average precision of benign calcification detection and the average precision of malignant calcification detection, and the size of this value reflects the difference between the system detection boundary box and the manually labeled boundary box to a certain extent.

[0049] Under the same computing resource limit (single GPU resource maximum memory 32GB), the effective image (including breast part) resolution of the adaptive image adjustment module with / without is 1200*2954 / 882×2048 respectively, the former occupies GPU memory 17G during training, and the latter cannot be trained and inferred on a single card and needs to use multiple GPUs.

[0050] In addition, the model migration of the above-mentioned embodiments with (2) / without (3) adaptive image adjustment module to the multi-center data including only clinical sub-data set of benign calcification (not participating in the training of the system model) is tested for generalization performance comparison. The experimental results are shown in Table 1. For the two embodiments, the performance decay degree of model migration test is reduced from 3.5% without adaptive image adjustment to 0.9% with adaptive image adjustment module, and the performance decay is reduced by about 77%. Through this experiment, it can be further illustrated that the adaptive image adjustment module has stronger generalization performance for multi-center data.

[0051] To prove the improvement effect of the multi-scale shortcut connection structure (MSC) and the trainable weighted fusion unit (TWF) on the detection ability and boundary box regression accuracy of multi-scale and multi-type targets, the comparison of (3) / (4) multi-scale shortcut connection structure and (4) / (5) trainable weighted fusion unit is made in this embodiment. The addition of the module makes the recall ability of the target reach 100%, and the average value of the boundary box IOU is improved by 23%.

[0052] As shown in Figure 7 , from top to bottom in row, there are 4 examples from the multi-center data set. (a) to (e) represent the detection results of models ①, ②, ③, ④, ⑤ in Table 1 respectively. (f) is the true value labeled by a doctor. Through Figure 7 the detection result (c) of ③ on example 4 is shown, such a larger area of malignant calcification detection is divided into two parts by (3), and the upper part of the detection boundary box is considered as a false positive example because the IOU with the true value is less than 0.5. In addition, for the detection result of ③ on example 1, there are many benign calcifications that are too small in size, causing the position of the detection boundary box to be inaccurate, and the IOU is less than 0.5, which is also considered as a false positive example. Both factors affect the average precision mAP of the network. ④ Because MSC is added, the target in example 4 is correctly detected as a complete malignant calcification, and the boundary box of the extremely small benign calcification in example 1 is more accurate. ⑤ Because TWF is added, more extremely small benign calcifications are provided with accurate boundary boxes.

[0053] As shown in Figure 8As shown, (a) and (b) are the visualization results of P3 detection feature maps in the existing pyramid structure and the multi-scale shortcut connection structure, respectively. (b) is more focused on the area with micro-calcification than (a), providing more accurate boundary information for subsequent detection. Figure 8 The visualization results of (b) prove that the multi-scale shortcut connection structure effectively preserves the shallow edge features in the fusion process, improving the regression accuracy of the target bounding box.

[0054] In the comparative experiment with (IV) / without (V) the trainable weighted fusion unit, by observing the trainable weights, the network tends to increase the weights of feature maps C2 and C4 in the upsampling process in stages 2 and 4, and increase the weight of P3 in the downsampling process in stage 3. This embodiment illustrates that the trainable weighted fusion unit serves the designed purpose.

[0055] This embodiment carries out further experiments to compare the performance of the system with the prior art methods. The experiment is carried out on the public dataset INbreast. The experimental results are shown in Table 2.

[0056] Table 2 Comparison of overall performance of different methods for benign and malignant calcification point detection on the public dataset INbreast. Where CM represents malignant calcification and IM represents benign calcification.

[0057] Method INbreast (recall @ false positives per image) Clustering 0.40@1, 0.80@10 for CM PFCM 0.80@0.5, 0.95@4.6 for CM CGDL 0.89@20 for IM Faster RCNN+VGG-16 0.85@1.5 for Both Types The system 0.95@2 for CM, 0.93@10 for IM

[0058] As shown in the above table, the system significantly improves the recall ability and accuracy.

[0059] As Figure 9 shown, in the test of the INbreast dataset, the free-response receiver operating characteristic (Free-Response receiver operating characteristic) of this embodiment is significantly higher than that of other methods. Through adaptive image adjustment, multi-scale shortcut connection structure and multi-scale fusion of trainable weights, the system realizes fully automatic detection of benign and malignant calcification in mammography images, significantly improves the target detection ability and the regression accuracy of the bounding box, and provides excellent reference and readability.

[0060] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application. The protection scope of the present application is subject to the claims and is not limited by the above specific embodiments. Each implementation within the scope is subject to the constraints of the present application.

Claims

1. An X-ray image detection system based on shortcut connection pyramid structure, characterized by, The adaptive clipping considers that in most mammography images, the breast always occupies one side of the image, so the middle is clipped: when the breast area is greater than half of the image, the clipping window is moved; when the breast exceeds the center line and the main area of the breast is on the right side, the clipping window is moved to the left to ensure that the left edge of the clipping window is aligned with the edge of the breast and the size of the clipped image is unchanged. The two images of the same size obtained after adaptive clipping are trained or detected in the system. The adaptive image adjustment module, the high-resolution feature extraction module, the multi-scale shortcut connection and weighted fusion module and the region recommendation and detection module are used for receiving multi-center molybdenum target images, wherein the adaptive adjustment module performs histogram normalization on the multi-center molybdenum target images, and adaptively clips the images into a uniform size; the high-resolution feature extraction module extracts feature maps of the images of different scales ; the multi-scale shortcut connection and weighted fusion module transmits and weightedly fuses features conducive to calcification detection to generate a plurality of multi-scale detection feature maps ; and the region recommendation and detection module recommends candidate regions based on the multi-scale detection feature maps , and further performs boundary box regression and target classification on the candidate regions. The adaptive image adjustment module includes an image normalization unit and an adaptive cropping unit, wherein the image normalization unit adjusts the target image set in the multi-center molybdenum target image. The grayscale histogram distribution automatically adjusts other source images. The grayscale histogram; the adaptive cropping unit adaptively crops the mammogram to a uniform size while ensuring that the breast structure remains unchanged; The multi-scale shortcut connection and weighted fusion module comprises a multi-scale shortcut connection structure and a trainable weighted fusion unit, wherein the multi-scale shortcut connection structure is connected with the high-resolution feature extraction module and receives feature maps of different scales The up-sampling and fusion are respectively performed to effectively transfer edge information of a shallow layer or spatial semantic information of a deep layer; the trainable weighted fusion unit autonomously learns weights of feature maps to be fused in a training process according to different network depths , and multi-scale detection feature maps are generated through weighted fusion with a nonlinear activation function according to the weights .

2. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1, characterized in that, The automatic adjustment refers to: when using multi-center data, specifying certain center data as target image , other center data as source image ; calculating cumulative distribution function (CDF) of source image and target image , respectively and ; finding gray scale that satisfies and defining matching equation as ; for discrete distribution CDF value, finding nearest ; applying equation to each pixel of other source image of multi-center data to realize adjustment; The high-resolution feature extraction module comprises an initial convolution layer, a residual module layer and second to fourth convolution layers arranged in sequence, wherein the initial convolution layer is a 7*7 convolution layer with 64 output channels, and each residual module comprises the number of output channels and the number of residual units in the bracket.

3. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1, characterized in that, The trainable weighted fusion unit enables different levels of information to have different weights when fused, and the weights represent the tendency of detection feature maps of different depths to the two types of calcification detection tasks, so that the generated multi-scale detection feature maps can provide more accurate feature information for different types of calcification.

4. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 3, characterized in that, The feature maps of different scales The feature maps output by the residual module layer and the second to fourth convolutional layers in the high-resolution feature extraction module, respectively , , and The size of the feature map is twice that of the feature map .

5. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1, characterized in that, The trainable weighting fusion unit fuses two feature maps, specifically: Wherein: is a parameter trainable during network training, The operation is implemented by a ReLU activation function to ensure that the contribution of the feature map is not negative during the fusion process. The upsampling and fusion comprises:

6. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1, characterized in that, The upsampling adopts a bilinear interpolation method. ① For each layer of feature maps generated by the high-resolution feature extraction module at different scales , , and Perform unified channel operations to generate corresponding feature maps. , , and Used for subsequent feature fusion or detection, where: ② from the lowest layer detection feature map Up-sampling is performed to make it have the same size as the more shallow layer of the previous layer with the same size, and trainable weighted fusion is performed; ③ After fusion, use convolution operation ( ) to reduce the aliasing effect after fusion to obtain a shallower detection feature map and , wherein the multi-scale shortcut connection structure includes a shortcut connection across in the upsampling process, that is, the detection feature map , () is a fusion process, that is .

7. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1 or 6, characterized in that, The region recommendation and detection module comprises a region recommendation network for generating candidate regions and a region detection network for realizing target classification and boundary box regression based on the recommended regions, wherein:

8. The X-ray image detection system based on the shortcut connection pyramid structure according to claim 1, characterized in that, The region recommendation network comprises a 3*3 convolution layer with Relu, two 1*1 convolution layers and a Softmax layer; The region detection network comprises a 7*7 convolution layer, a classification task full connection layer and a regression task full connection layer. ​

Images