Super-resolution reconstruction device for Micro-CT images of rat ankle fractures

Abstract

The present invention relates to a super-resolution reconstruction device for Micro-CT images of rat ankle fractures, comprising a rat ankle image preprocessing module, which is used for performing high-resolution and low-resolution scans on rat tibia to ankle after ankle fracture modeling, respectively, to obtain a high-resolution image HR and a corresponding low-resolution image LR;) configuring an HR-LR image pair module, which is used for obtaining training data of a residual channel attention model based on Res2Net; a deep model module of the residual channel attention based on Res2Net, comprising a shallow feature extraction layer, a deep feature extraction layer and a feature upsampling layer; and an image super-resolution reconstruction module.

Description

Technical Field

[0001] The present invention discloses a super-resolution reconstruction of a rat ankle fracture Micro-CT image, and specifically relates to a deep learning model based on a self-attention mechanism to reconstruct a low-resolution rat Micro-CT image into a high-resolution image. Background Art

[0002] Fracture is a common disease in today's society, and research on fracture has never stopped. Among them, research on various fracture treatment plans or drug-dependent animal experiments, due to the small size and skeleton of animal experiments (rats, etc.), Micro-CT with high spatial resolution is a necessary equipment for animal experiments related to bone research. Studies have shown that high-dose x-rays hinder fracture healing, and low-dose x-rays accelerate the recovery of cartilage and intramembranous ossification. For the sake of treatment effect and ethics, in order to protect small animals, the implementation of Micro-CT programs should be adjusted. Therefore, it is particularly important to ensure sufficiently clear high-resolution CT images while reducing scanning time and ionizing radiation. This can not only improve the efficiency of animal experimental research and clinical application, but also improve the safety of animal experiments and even future clinical treatment. Super-resolution technology has the possibility of converting low-resolution images into high-resolution images.

[0003] References

[0004] [1] P.Alcantarilla, J.Nuevo, and A.Bartoli, "Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces," in Procedings of the BritishMachine Vision Conference 2013, Bristol, 2013, p.13.1-13.11.

[0005] [2] R.Laganiere, OpenCV 2 Computer Vision Application Programming Cookbook.OpenCV 2 computer vision application programming cookbook:, 2011.

[0006] [3]Y.Zhang,K.Li,K.Li,L.Wang,B.Zhong,and Y.Fu,“Image Super-ResolutionUsing Very Deep Residual Channel Attention Networks,”in Computer Vision–ECCV2018,vol.11211,V.Ferrari,M.Hebert,C.Sminchisescu,and Y.Weiss,Eds.Cham:Springer International Publishing,2018,pp.294–310.

[0007] [4]S.-H.Gao,M.-M.Cheng,K.Zhao,X.-Y.Zhang,M.-H.Yang,and P.Torr,“Res2Net:A New Multi-scale Backbone Architecture,”IEEE Trans.PatternAnal.Mach.Intell.,vol.43,no.2,pp.652–662,Feb.2021.

[0008] [5]H.Jie,S.Li,S.Gang,and S.Albanie,“Squeeze-and-Excitation Networks,”IEEE Transactions on Pattern Analysis and Machine Intelligence,vol.PP,no.99,2017.

[0009] [6]R.Keys,“Cubic convolution interpolation for digital imageprocessing,”IEEE Trans.Acoust.,Speech,Signal Process.,vol.29,no.6,pp.1153–1160,Dec.1981.

[0010] [7]C.Dong,CCLoy,K.He,and InternationalPublishing,2014,pp.184–199.

[0011] [8] B.Lim, S.Son, H.Kim, S.Nah, and KMLee, "Enhanced Deep ResidualNetworks for Single Image Super-Resolution," arXiv:1707.02921[cs], Jul.2017.

[0012] [9] Summary of the invention

[0013] The present invention proposes a super-resolution reconstruction device for Micro-CT images of rat ankle fractures. By adding Res2Net, the model has the ability to extract features at multiple scales and has a better reconstruction effect. The technical solution is as follows:

[0014] A super-resolution reconstruction device for Micro-CT images of rat ankle fractures, characterized by comprising:

[0015] (1) The rat ankle image preprocessing module is used to perform high-resolution and low-resolution scans on the tibia to ankle of rats that have undergone ankle fracture modeling, respectively, to obtain a high-resolution image HR and a corresponding low-resolution image LR, the resolution of which differs by 8 times.

[0016] (2) Configuring the HR-LR image pair module to obtain training data for the residual channel attention model based on Res2Net, the method is as follows: applying a feature detection algorithm to detect feature points, and matching feature points of the LR and HR images at the same scanning position; rotating the LR image by two sets of feature points so that the LR image and the HR image are in the same direction, and cropping the sub-images of the HR and LR images respectively with the feature points as the center to produce HR-LR image pairs;

[0017] (3) A deep model module based on the residual channel attention of Res2Net, including a shallow feature extraction layer, a deep feature extraction layer, and a feature upsampling layer:

[0018] The shallow feature extraction layer is a 3×3 convolution layer. The LR image is input into the shallow feature extraction layer to obtain the shallow feature map F0;

[0019] The deep feature extraction layer consists of two parts, namely, RCAB-group based on channel attention and Res2-group based on Res2Net; the RCAB-group consists of 10 RCABs with short skip connections; Res2-group is based on Res2Net, and 5 Res2blocks are connected in series with short skip connections to prevent overfitting in model training; the shallow feature map F0 passes through the first RCAB-group to obtain the feature map F1; the feature map F1 passes through the first Res2-group to obtain the feature map F2, and the feature map F2 passes through the second RCAB-group to obtain the feature map F2; the feature map F2 passes through the second Res2-group to obtain the feature map F3, and so on. After five such alternating series with long skip connections, the final feature map F 10 ;

[0020] The method of obtaining the feature map F1 through the first RCAB-group of the shallow feature map F0 is as follows: the shallow feature map F0 enters 10 RCABs in sequence. When the shallow feature map F0 enters the first RCAB, it undergoes a 3×3 convolution, a ReLU activation function and a 3×3 convolution to obtain the feature map X 0,1 , and then the feature map X 0,1 In the input channel attention mechanism layer, the feature map X is pooled by the global average pooling layer. 0,1 Compress it into a 1×1 vector, then pass the vector through a 1×1 convolution layer and ReLU activation function to reduce the number of channels, and then pass it through a 1×1 convolution layer and Sigmoid activation function to generate the attention weights to multiply the input feature map X 0,1 Get the new feature map F 0,1 ; Feature map F 0,1Then, the second to tenth RCAB processes are performed in sequence to obtain the feature map F1 that passes the first RCAB-group;

[0021] The method of obtaining the feature map F2 by passing the feature map F1 through the first Res2-group is as follows: the feature map F1 enters 5 Res2blocks in sequence. After entering the first Res2block, the feature map F1 passes through a 1×1 convolution layer and is divided into 4 sub-feature maps X i , where i is 1 to 4; the number of channels of each sub-feature map is 1 / 4 of the feature map F1; the sub-feature map X1 is directly obtained as the sub-feature map Y1 without convolution processing; the sub-feature map X2 is subjected to a 3×3 convolution to obtain the sub-feature map Y2; the sub-feature map X3 is added to the result of the previous sub-feature map Y2 and then subjected to a 3×3 convolution to obtain the sub-feature map Y3; the sub-feature map X4 is added to the result of the previous sub-feature map Y3 and then subjected to a 3×3 convolution to obtain the sub-feature map Y4; through the above operations, the receptive field is continuously increased to obtain four multi-scale sub-feature maps: Y1, Y2, Y3, and Y4; the four multi-scale sub-feature maps Y1, Y2, Y3, and Y4 are fused to obtain the output result after the first Res2-group; and then passed through 4 identical Res2blocks in sequence to obtain the feature map F2;

[0022] Feature upsampling layer, a sub-pixel convolution layer is used to obtain the final feature map F 10 Upsampling is performed to expand the resolution of the original input image to 8 times the original resolution. A 3×3 convolutional layer is then used to compress the number of channels of the feature map into a final image with 3 channels.

[0023] (4) An image super-resolution reconstruction module is used to reconstruct the LR image into a HR image, thereby obtaining a super-resolution reconstructed image of the rat ankle fracture Micro-CT image.

[0024] Furthermore, the AKAZA feature detection algorithm is applied to detect feature points, and the Brute-Force algorithm is used to match feature points between the LR and HR images at the same scanning position. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 : Flow chart of the present invention

[0026] Figure 2 : Make HR-LR image pairs

[0027] Figure 3 :R2-RCAN super-resolution model structure

[0028] Figure 4 :Channel Attention Mechanism

[0029] Figure 5 :RCAB structure

[0030] Figure 6 : Res2block structure

[0031] Figure 7 : Comparison of visual effects of reconstructed images of various methods, the bold ones are the best DETAILED DESCRIPTION

[0032] In order to explain the present invention in detail, the specific implementation process of the present invention is further described in conjunction with examples and drawings.

[0033] The present invention proposes a super-resolution reconstruction device for rat ankle fracture Micro-CT images, including a rat ankle image preprocessing module, a HR-LR image pair configuration module, an R2-RCAN deep model module and an image super-resolution reconstruction module. First, a professional operator performs a fracture model on the ankle of a living rat, and fixes the living rat in a Micro-CT of the model SkyScan 1276 of Bruker Company, and continuously scans its fracture site with high resolution and low resolution to obtain HR images and LR images, wherein the resolution of the HR image is 8 times that of the LR image. Then, we select one from every eight HR images to correspond to the LR image. According to the image processing technology of image feature point detection and feature point matching, HR-LR image pairs are produced as training data for the super-resolution model, and image cropping is completed at the same time to facilitate deep learning training. The present invention proposes an R2-RCAN improved based on RCAN, which is a super-resolution reconstruction model based on the self-attention mechanism. By adding Res2Net, the model has the ability to extract features at multiple scales. Compared with other classic super-resolution models, the R2-RCAN proposed in the present invention has achieved the best effect.

[0034] The overall flow chart is as follows Figure 1 As shown, the specific steps include:

[0035] Step 1: Rat ankle image preprocessing module, firstly, a professional operator performs an ankle fracture model on the rat and scans the rat tibia to ankle. Micro-CT is SkyScan 1276 of Bruker Company, and its configuration includes: maintenance-free 20-100kV micro-focus X-ray source, automatic 6-position filter converter; 11Mp cooled X-ray camera; continuously variable magnification, minimum pixel size of 2.8 microns; can resolve 5-6 micron object details, with a contrast of more than 10%. The present invention is an 8-fold super-resolution reconstruction, strict high-resolution shooting and low-resolution shooting strictly fix the position of the mouse. A high-resolution (10 microns) and a low-resolution (80 microns) scan are performed on the rat tibia to ankle, and 4000 HR and 500 LR images are obtained. Among them, the high-resolution 10-micron image scanning sequence is to take one HR out of 8 images, corresponding to the LR.

[0036] Step 2: Configure the HR-LR image pair module. Since Micro-CT has a higher spatial distribution rate than ordinary CT, it is an important tool for small animal imaging research. However, its disadvantage is that the time distribution rate is much lower. For the imaging of living small animals, one cycle of the Micro-CT frame is often accompanied by several breathing cycles of the small animal. Since the present invention performs super-resolution reconstruction of the fracture site on living rats, even if the position of the rat is strictly fixed, the HR and LR images cannot be completely matched one by one due to image offset caused by too long time and breathing. To solve the above problem, the present invention uses Opencv's AKAZA feature detection algorithm [1] to detect feature points, and uses the Brute-Force algorithm [2] to match feature points, and performs feature point matching on LR and HR images at the same scanning position. Take any two sets of feature points aA and bB (where a and b are feature points on LR, and A and B are feature points on HR), and calculate the angle between ab and the horizontal direction by formula (1):

[0037]

[0038] Where (x a ,y a ) and (x b ,y b ) are the coordinates of point a and b respectively. Similarly, the angle between AB and the horizontal direction is calculated. By rotating LR, LR and HR are rotated to the same direction, and LR and HR are cropped with the feature point pair as the center to obtain a one-to-one LR-HR image pair, as shown in Figure 2As shown in the figure, a 40×40 pixel sub-image is cropped from the LR image, and a 320×320 pixel sub-image is cropped from the HR image. A total of 960 image pairs with better structures are selected, of which 800 pairs are for the training set, 80 pairs are for the validation set, and 80 pairs are for the test set. The above operations not only solve the image offset problem, but also crop the image to facilitate subsequent deep learning training.

[0039] Step 3: The overall structure of R2-RCAN is as follows Figure 3 As shown in the figure, while the model has a certain depth, the model width is increased, so that the model has the ability to extract features at multiple scales. It includes a shallow feature extraction layer, a deep feature extraction layer with several RCAB-groups and Res2-groups stacked, and a feature upsampling layer based on sub-pixel convolution.

[0040] The shallow feature extraction layer is used to extract coarse-grained features of the image, which is conducive to the extraction of deep features. It consists of a 3×3 convolution layer, and the image is input into the shallow feature layer to obtain a shallow feature map.

[0041] Deep feature extraction was mainly composed of two parts: RCAB-group based on channel attention and Res2-group based on Res2Net.

[0042] The RCAB-group consists of 10 RCABs with short skip connections. RCAB is a residual structure based on the channel attention mechanism [3]. The channel attention mechanism is to allow the network to pay attention to more useful information of the image, and to use the dependencies between feature channels to give higher weights to feature channels with more informative low-frequency and valuable high-frequency information. This achieves better super-resolution learning results. The channel attention mechanism is as follows: Figure 4 As shown in the figure, the input feature map is first compressed into a 1×1 vector through a global average pooling layer, and then the vector is passed through a 1×1 convolution layer and a ReLU activation function to reduce the number of channels. Then, a 1×1 convolution layer and a Sigmoid activation function are used to generate the attention weights and multiply the input feature map to obtain a new feature map. The attention mechanism is added to the residual module to obtain RCAB, and the structure is as follows Figure 5 As shown. It contains two layers of 3×3 convolution and one CA layer, and equation (2) represents the convolution after two layers of stacking:

[0043]

[0044] where i and j represent the jth RCAB in the i-th RCAB-group, F i-1 represents input, X i-1,j Indicates the output, and Represents two stacked convolutional layers, δ represents the ReLU activation function, and the residual information X is extracted through convolution i,j , and then pass it through the CA layer, as shown in formula (3):

[0045] F i-1,j =F i,j-1 +R i,j (X i-1,j )·X i-1,j (3)

[0046] where F i-1,j Represents the output of this layer, R i,j Indicates the CA layer. Figure 3 As shown in the figure, the RCAB-group contains several RCAB layers, a convolutional layer and short skip connections. This stacked module with skip connections helps to deepen the number of network layers and obtain better super-resolution results.

[0047] Res2-group is mainly composed of 5 stacked Res2blocks. The Res2block used in the present invention is composed of Figure 6 As shown in Figure 1, it is the most basic module of Res2Net. As a new variant of ResNet, Res2Net is a multi-scale residual unit structure [4]. Res2Net represents multi-scale features in a finer granularity and increases the receptive field of each network layer. After the feature map enters Res2block, it passes through the first 1×1 convolution and divides the input into 4 blocks, each of which is X. i (i=1,2,3,4), their number of channels is 1 / 4 of the original. Except for X1, each block will perform a 3×3 convolution, using Y i Represents the convolution result of each block. X i+1 Convolution result Y with the previous block i The feature maps are added and then convolved to get Y by 3×3. i+1 , so that we can get outputs of different numbers and different receptive field sizes. Finally, by adding a SE block to assign a weight to each channel [5], the feature response of each channel can be increased. This idea of ​​splitting first and then fusing can obtain multi-scale features and better perform feature fusion, increase the network's receptive field, represent multi-scale features, and expand the network width.

[0048] Step 4: Image super-resolution reconstruction module. During the training of R2-RCAN, the training data is rotated and inverted. For each training batch, 32 LR images are extracted as input. Our model is trained by the ADAM optimizer with β1=0.9, β2=0.999. The initial learning rate is set to 10-4, and then every 2×10 5The number of back propagation iterations is reduced by half. The loss function used is L1 loss, which is shown in equation (4):

[0049]

[0050] Where H and W are the height and width of the image, respectively. SR and I HR are reconstructed images and HR images respectively. The R2-RCAN structure is set to 5 RCAB-groups and 5 Res2-groups, each group is alternately connected in series, where each RCAB-group contains 10 RCABs and each Res2-group contains 5 res2blocks. Except for the convolutional layers in channel downscaling and channel upscaling, the kernel size of the other convolutional layers is set to 3×3. We use PSNR and SSIM as the evaluation criteria for super-resolution, as shown in equations (5) and (6):

[0051]

[0052] Where MSE represents the root mean square error between the reconstructed image and the HR image.

[0053]

[0054] where u X and u Y represents the mean of images X and Y, σ X and σ Y Represents the standard deviation of images X and Y, σ XY represents the covariance of images X and Y, and C1, C2, and C3 are all constants. We compare the same data with classic models such as Bicubic[6], SRCNN[7], EDSR[8], RRDBnet[9], ESRGAN[9], and RCAN[3]. We use the PyTorch framework to train the model on a 2080Ti GPU.

[0055] Table 1: Quantitative comparison of the reconstruction results of rat ankle images by various models, where the ones marked in bold are the best results.

[0056] method Reconstruction multiple PSNR SSIM Bicubic[6] ×8 20.36 0.62 SRCNN[7] ×8 20.60 0.64 RRDBnet[9] ×8 21.18 0.66 ESRGAN[9] ×8 19.93 0.57 EDSR[8] ×8 20.98 0.65 RCAN[3] ×8 21.39 0.63 R2-RCAN ×8 21.46 0.68

[0057] Table 1 is a quantitative comparison of the reconstruction results of rat ankle Micro-CT by various models (for PSNR and SSIM). The references cited in the comparison method in Table 1 can be found in the background art.

[0058] The R2-RCAN proposed in the present invention performs best in 8-fold rat Micro-CT image reconstruction, with an average PSNR of 21.46 and a SSIM of 0.68 in the test set. Figure 7 A visual comparison of the results of each method is shown.

Claims

1. A super-resolution reconstruction device for Micro-CT images of rat ankle fractures, characterized in that: include: (1) A rat ankle image preprocessing module is used to perform high-resolution and low-resolution scans on the tibia to ankle of rats that have undergone ankle fracture modeling, respectively, to obtain a high-resolution image HR and a corresponding low-resolution image LR, the resolution of which differs by 8 times; (2) Configuring the HR-LR image pair module to obtain training data for the residual channel attention model based on Res2Net, the method is as follows: applying a feature detection algorithm to detect feature points, and matching feature points of the LR and HR images at the same scanning position; rotating the LR image by two sets of feature points so that the LR image and the HR image are in the same direction, and cropping the sub-images of the HR and LR images respectively with the feature points as the center to produce HR-LR image pairs; (3) A deep model module based on the residual channel attention of Res2Net, including a shallow feature extraction layer, a deep feature extraction layer, and a feature upsampling layer: The shallow feature extraction layer is a 3×3 convolution layer. The LR image is input into the shallow feature extraction layer to obtain the shallow feature map F0; The deep feature extraction layer consists of two parts, namely RCAB-group based on channel attention and Res2-group based on Res2Net; The RCAB-group is composed of 10 RCABs with short skip connections; the Res2-group is based on Res2Net, and 5 Res2blocks are connected in series with short skip connections to prevent overfitting in model training; the shallow feature map F0 passes through the first RCAB-group to obtain the feature map F1; the feature map F1 passes through the first Res2-group to obtain the feature map F2, and the feature map F2 passes through the second RCAB-group to obtain the feature map F2; the feature map F2 passes through the second Res2-group to obtain the feature map F3, and so on. After five such alternating series with long skip connections, the final feature map F 10 ; The method of obtaining the feature map F1 through the first RCAB-group of the shallow feature map F0 is as follows: the shallow feature map F0 enters 10 RCABs in sequence. When the shallow feature map F0 enters the first RCAB, it undergoes a 3×3 convolution, a ReLU activation function and a 3×3 convolution to obtain the feature map X 0,1 , and then the feature map X 0,1 In the input channel attention mechanism layer, the feature map X is pooled by the global average pooling layer. 0,1 Compress it into a 1×1 vector, then pass the vector through a 1×1 convolution layer and ReLU activation function to reduce the number of channels, and then pass it through a 1×1 convolution layer and Sigmoid activation function to generate the attention weights to multiply the input feature map X 0,1 Get the new feature map F 0,1 ; Feature map F 0,1 Then, the second to tenth RCAB processes are performed in sequence to obtain the feature map F1 that passes the first RCAB-group; The method of obtaining the feature map F2 by passing the feature map F1 through the first Res2-group is as follows: the feature map F1 enters 5 Res2blocks in sequence. After entering the first Res2block, the feature map F1 passes through a 1×1 convolution layer and is divided into 4 sub-feature maps X i , where i is 1 to 4; the number of channels of each sub-feature map is 1 / 4 of the feature map F1; the sub-feature map X1 is directly obtained as the sub-feature map Y1 without convolution processing; the sub-feature map X2 is subjected to a 3×3 convolution to obtain the sub-feature map Y2; the sub-feature map X3 is added to the result of the previous sub-feature map Y2 and then subjected to a 3×3 convolution to obtain the sub-feature map Y3; the sub-feature map X4 is added to the result of the previous sub-feature map Y3 and then subjected to a 3×3 convolution to obtain the sub-feature map Y4; through the above operations, the receptive field is continuously increased to obtain four multi-scale sub-feature maps: Y1, Y2, Y3, and Y4; the four multi-scale sub-feature maps Y1, Y2, Y3, and Y4 are fused to obtain the output result after the first Res2-group; and then passed through 4 identical Res2blocks in sequence to obtain the feature map F2; Feature upsampling layer, a sub-pixel convolution layer is used to obtain the final feature map F 10 Upsampling is performed to expand the resolution of the original input image to 8 times the original resolution. A 3×3 convolutional layer is then used to compress the number of channels of the feature map into a final image with 3 channels. (4) An image super-resolution reconstruction module is used to reconstruct the LR image into a HR image, thereby obtaining a super-resolution reconstructed image of the rat ankle fracture Micro-CT image.

2. The super-resolution reconstruction device according to claim 1, characterized in that: The AKAZA feature detection algorithm is used to detect feature points, and the Brute-Force algorithm is used to match feature points between LR and HR images at the same scanning position.

Images