A deep learning-based local micro-feature structure segmentation method

Abstract

The application provides a local micro-feature structure segmentation method based on deep learning, first, an input sequence sample image is segmented into multiple images for data enhancement, the enhanced data is down-sampled multiple times to obtain multiple scale images, the processed data is input into an image segmentation module containing an attention mechanism, the obtained final prediction image and the hidden value output from the image segmentation module are jointly input into an image reconstruction module to synthesize the original image to complete pre-training, then a discriminator is added to alternately train the image segmentation image reconstruction module and the discriminator, and the discriminator is used for image discrimination; the image segmentation module with the added attention mechanism can better focus on local micro-structures, can be applied to industrial CT and medical CT identification, can better identify fine and connected regions in the CT image, and has higher accuracy.

Description

Technical Field

[0001] This invention belongs to the field of CT image segmentation technology, specifically relating to a method for segmenting local small feature structures based on deep learning. Background Technology

[0002] In the fields of medical CT inspection and industrial CT scanning, people often focus on the different grayscale values ​​produced by varying densities in CT images. CT scans are widely used to obtain the internal morphology of the human body or workpieces. Accurately distinguishing target areas in images based on CT scan results allows for better analysis of the causes of problems in the human body or workpiece. However, the observation and recognition of CT images often requires professionals to analyze them based on years of relevant experience, which is time-consuming and labor-intensive. Therefore, fast and accurate image segmentation and recognition methods are becoming increasingly important.

[0003] CT images often contain numerous artifacts and noise due to factors such as dark current in the acquisition equipment and the scanning method of the target object. Traditional techniques for defect extraction via CT often employ thresholding for image segmentation. Thresholding relies on the uniformity of grayscale values ​​in CT images, thus its segmentation performance is poor when dealing with uneven grayscale distributions of defects in CT images. Addressing this issue requires significant manual adjustments to remove holes, jagged edges, and adhesions generated during thresholding, which is time-consuming and labor-intensive. Deep learning-based image segmentation methods can solve the problems associated with thresholding. One existing technique proposes an Attention U-NET network architecture, combining an attention mechanism module with U-NET for segmentation, but its performance is only moderate for some fine regional structures. Another existing technique proposes a three-branch image segmentation framework for skin blood vessels, and utilizes a consistency decoder and generative adversarial network from the paper to enable training on unlabeled data, proposing a semi-supervised learning model framework. However, this network is not integrated with an attention mechanism module and cannot focus on the target region. Another existing technology uses deep learning methods to segment and reconstruct industrial CT images, but this method is generally ineffective for small and fine connected regions.

[0004] Therefore, a fast and accurate method for segmenting local small feature structures is needed at this stage to segment target regions in CT images. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a local small feature structure segmentation method based on deep learning, which combines the attention mechanism in deep learning with the three-branch image segmentation method, and can solve the problems of long segmentation time and low segmentation accuracy of small regions and connected regions when traditional methods are used for defect segmentation of CT images.

[0006] A deep learning-based method for segmenting local small feature structures, characterized by:

[0007] The deep learning network model used includes a preprocessing module, an image segmentation module, an image reconstruction module, and a discriminator;

[0008] The preprocessing module segments the input CT image into multiple parts to complete data enhancement, and then downsamples it at different scales to obtain two or more sets of images at different scales.

[0009] The image segmentation module includes multiple scale-based image segmentation modules to process images of different scales. Each image segmentation module includes an encoder, a ConvLSTM module, and a decoder connected in sequence. The encoder includes four sequentially connected encoding processes. Each encoding process of the encoder includes a sequentially connected convolutional layer and three residual blocks, where each residual block contains two convolutional layers and two batch normalization (BN) layers, and ReLU is used as the activation function. The decoder includes one residual block and four sequentially connected decoding processing channels and a regular convolutional layer. Each decoding processing channel includes a sequentially connected deconvolutional layer and three residual blocks.

[0010] The image reconstruction module sequentially includes two fully connected layers, a size correction layer, three upsampling layers, an addition module, and three convolutional layers;

[0011] The discriminator comprises four sequentially connected convolutional layers, an average pooling layer, and a fully connected layer.

[0012] In the image segmentation module excluding the largest scale, the image is first convolved once to obtain a feature map. This feature map is then processed through four encoding steps to obtain feature maps fm1, fm2, fm3, and fm4. The final output feature map fm4 is then input into the ConvLSTM module, which outputs a hidden value H. l The ConvLSTM module serves as the next-scale image segmentation module to achieve information fusion.

[0013] The other output value of the ConvLSTM module is fed into the subsequent decoder. After being processed by the four decoding channels of the decoder, feature maps fd1, fd2, fd3 and fd4 are obtained respectively. Feature map fd4 is the mesoscale prediction map finally output by the image segmentation module.

[0014] The image segmentation modules at each scale also include an attention module, which consists of sequentially connected 1×1×1 convolutional layers, an addition module, a ReLU activation function, another 1×1×1 convolutional layer, a sigmoid activation function, a Resamaler function, and a multiplication module. For the current i-th decoding channel, the attention module adds the feature map fd(i-1) output from the previous decoding channel in the decoder through a 1×1×1 convolutional layer and the feature map Wg output from the encoder channel symmetrical to this decoding channel through a 1×1×1 convolutional layer. The result is then passed through a ReLU activation function, then through another 1×1×1 convolutional block, and finally through a sigmoid activation function and a Resamaler function to obtain the weight matrix of the feature map of the previous decoding channel. Multiplying this weight matrix by the feature map of the previous decoding channel allows the network to continuously focus on key regions.

[0015] In the maximum scale image segmentation module, the feature maps fd1, fd2, fd3 and fd4 obtained from each decoding processing channel are added together on the channel, and then the same convolution operation is performed to obtain the prediction map of the maximum scale image.

[0016] Then, the large-scale, i.e., the original-scale image segmentation prediction map is input into the channel adder of the image reconstruction module, and the feature map output by the encoder of the large-scale segmentation module is input into the first fully connected layer of the image reconstruction module. After processing by the image reconstruction module, the original image is synthesized to obtain a prediction original image.

[0017] The discriminator is input to the label image and the ground truth image, as well as the original scale image segmentation prediction image and the prediction image. After passing through the discriminator's final fully connected layer, it outputs 0 and 1, where 0 represents false and 1 represents true.

[0018] When training the deep learning network model, the image segmentation module and the image reconstruction module are initially trained. The loss function of the segmentation module during training is Diceloss, and the loss function of the reconstruction module is MSEloss. The differences between the three predicted images generated after the images of three scales enter the model and the real labels, as well as the differences between the synthesized image and the actual original image, are calculated. After the initial training of the image segmentation module is completed, the entire network model is trained. At this time, the loss of the entire network model is the sum of the Diceloss of the image segmentation module, the MSEloss of the image reconstruction module, and the MSEloss of the discriminator.

[0019] After training the deep learning network model, the input image is fed into the maximum-scale image segmentation module to obtain the predicted map of the maximum-scale image, which is the image segmentation result.

[0020] Preferably, the two fully connected layer outputs of the image reconstruction module are set to 32 and 65536 respectively; the size correction module corrects the image to a size of 64×64; and the upsampling factor for each of the three upsampling operations is set to 2.

[0021] Preferably, the number of channels used in the encoding processing channels of the image segmentation module at each scale increases sequentially, namely 8, 16, 36, and 64, respectively. A BN layer is added in each layer to normalize the data, the convolution kernel size is set to 5×5, and the image is padded with two rows and two columns of 0s around the perimeter.

[0022] The present invention has the following beneficial effects:

[0023] This invention provides a deep learning-based method for segmenting local minute feature structures. First, the input sequence sample image is segmented into multiple images for data augmentation. The augmented data is then downsampled multiple times to obtain images at various scales. The processed data is input into an image segmentation module incorporating an attention mechanism. The final predicted image and the hidden values ​​output from the image segmentation module are then input into an image reconstruction module to synthesize the original image for pre-training. Subsequently, a discriminator is added, and the image segmentation and reconstruction modules are trained alternately with the discriminator. The discriminator is then used for image discrimination. This invention utilizes an image segmentation module with an attention mechanism to better focus on local minute structures, making it applicable to industrial and medical CT recognition. It can better identify subtle and connected regions in CT images, achieving higher accuracy. By combining the attention mechanism with a multi-scale image segmentation method, this invention is more time-saving and labor-saving compared to traditional CT image target region extraction methods, and it also shows better performance in segmenting complex shapes of defects in industrial CT images. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a local small feature structure segmentation method based on deep learning according to the present invention.

[0025] Figure 2 This invention provides a segmentation method for sample data augmentation.

[0026] Figure 3 This is a diagram of the overall network framework of the present invention;

[0027] Figure 4 This is a structural diagram of the attention mechanism in the image segmentation module of the present invention. Detailed Implementation

[0028] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Addressing the urgent need for accurate and rapid image segmentation of defective regions in current industrial CT images, this invention proposes a deep learning-based method for segmenting local small feature structures. This method utilizes an attention mechanism to focus on local regions, thereby better segmenting small, connected regions in defective images. The main steps are as follows: Figure 1 As shown:

[0030] The deep learning network model used in this invention includes a preprocessing module, an image segmentation module, an image reconstruction module, and a discriminator;

[0031] The image segmentation module is illustrated using three scales: small-scale segmentation, medium-scale segmentation, and large-scale segmentation. Each segmentation module includes a sequentially connected encoder, a ConvLSTM module, and a decoder. The encoder comprises four sequentially connected encoding processes. Each encoding process includes a sequentially connected convolutional layer and three residual blocks, where each residual block contains two convolutional layers and two batch normalization (BN) layers, using ReLU as the activation function. The decoder includes one residual block, followed by four sequentially connected decoding channels and a regular convolutional layer (without residual blocks, only convolutional layers). Each decoding channel includes a sequentially connected deconvolutional layer and three residual blocks.

[0032] The image reconstruction module consists of two fully connected layers, a size correction layer, three upsampling layers, an addition module, and three convolutional layers. The fully connected layers output two values ​​set to 32 and 65536 respectively; the size correction module resizes the image to 64×64; and the upsampling factor for each of the three upsampling operations is set to 2.

[0033] The discriminator consists of four sequentially connected convolutional layers, an average pooling layer, and a fully connected layer.

[0034] This embodiment demonstrates the defect image segmentation process for a batch of composite materials, such as... Figure 2 As shown, in the preprocessing module, data enhancement is completed by cutting the sequence CT images into n parts. In this example, it is selected to cut into 4 parts. At the same time, each image after cutting is downsampled by n1 to obtain images of multiple scales as input to the image segmentation module. In this example, the images are downsampled twice and the original scale images, a total of three images, are used as input to the image segmentation module.

[0035] like Figure 3 As shown, images at three scales are input into three image segmentation modules. Since the processing of small-scale and medium-scale images is the same, only the input process of medium-scale image data is described, as follows:

[0036] The mesoscale image is first convolved once to obtain a feature map. This feature map is then processed through four encoding steps to obtain feature maps fm1, fm2, fm3, and fm4. The final output feature map fm4 is then input into a ConvLSTM module, which outputs a hidden value H. l The ConvLSTM module serves as the next-level image segmentation module to achieve information fusion. The input and output of the ConvLSTM module can be represented as follows:

[0037] C l H l =g(C l-1 H l-1 ,T l )

[0038] In the formula, C l For the current scale layer long-term memory state, H l Current scale layer short-term memory state, C l-1 For the long-term memory state of the next higher scale layer, H l-1 The short-term memory state of the next higher scale, T l is the output of the current scale layer encoder, and g is the ConvLSTM module.

[0039] The other output value of the ConvLSTM module is fed into the subsequent decoder. After being processed by the four decoding channels of the decoder, feature maps fd1, fd2, fd3 and fd4 are obtained respectively. Feature map fd4 is the mesoscale prediction map output by the mesoscale segmentation module.

[0040] Attention mechanisms are also used in each image segmentation module, such as Figure 4 As shown, the attention module includes a sequentially connected 1×1×1 convolutional layer, an addition module, a ReLU activation function, another 1×1×1 convolutional layer, a sigmoid activation function, a Resamaler function, and a multiplication module. For the current i-th decoding channel, the feature map fd(i-1) output from the previous decoding channel in the decoder is passed through a 1×1×1 convolutional layer, and the feature map Wg output from the encoder channel symmetrical to this decoding channel is passed through a 1×1×1 convolutional layer. The result is then passed through a ReLU activation function, then through another 1×1×1 convolutional block, and finally through a sigmoid activation function and a Resamaler function to obtain the weight matrix of the previous layer's feature map. Multiplying this weight matrix by the feature map of the previous decoding channel allows the network to continuously focus on key regions.

[0041] In this process, the number of channels used in the encoding processing channels of each scale image input layer increases sequentially, namely 8, 16, 36, and 64. A BN layer is added in each layer to normalize the data. The convolution kernel size is set to 5×5, and the image is padded with two rows and two columns of 0s around the perimeter.

[0042] The large-scale segmentation module processes large-scale images in a similar manner to small-scale and medium-scale images. However, in the decoder, the feature maps fd1, fd2, fd3, and fd4 obtained from each decoding channel are added together on the channel, and then the same convolution operation is performed to obtain the prediction map of the large-scale image. This prediction map will also be used as an input to the image reconstruction module.

[0043] Subsequently, the large-scale, i.e., original-scale image segmentation prediction map is input into the channel adder of the image reconstruction module, and the feature map output by the encoder of the large-scale segmentation module is input into the first fully connected layer of the image reconstruction module. After processing by the image reconstruction module, the original image is synthesized to obtain a prediction original image.

[0044] The discriminator takes the label image and the ground truth image as input, along with the original scale image segmentation prediction image and the prediction image. After passing through the discriminator's final fully connected layer, it outputs 0 and 1, where 0 represents false and 1 represents true.

[0045] When training the aforementioned deep learning framework, the image segmentation module and the image reconstruction module are initially trained. The loss function for the segmentation module is Diceloss, and the loss function for the reconstruction module is MSEloss. The differences between the three predicted images generated after the images of three scales enter the model and the real labels, as well as the difference between the synthesized image and the actual original image, are calculated. After the initial training of the image segmentation module is completed, the entire network model is trained. At this time, the loss of the entire network model is the sum of the Diceloss of the image segmentation module, the MSEloss of the image reconstruction module, and the MSEloss of the discriminator.

[0046] After completing the total number of training generations, the image segmentation results can be obtained. The best-performing generation of parameters is stored in the validation set. Using these parameters, fast and accurate image segmentation for CT image defects can be achieved.

[0047] When segmenting the input image, the input image is fed into the large-scale segmentation module to obtain the predicted map of the large-scale image, which is the segmentation result.

[0048] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for segmenting local small feature structures based on deep learning, characterized in that: The deep learning network model used includes a preprocessing module, an image segmentation module, an image reconstruction module, and a discriminator; The preprocessing module segments the input CT image into multiple parts to complete data enhancement, and then downsamples it at different scales to obtain two or more sets of images at different scales. The image segmentation module includes multiple scale-based image segmentation modules to process images of different scales. Each image segmentation module includes an encoder, a ConvLSTM module, and a decoder connected in sequence. The encoder includes four sequentially connected encoding processes. Each encoding process of the encoder includes a sequentially connected convolutional layer and three residual blocks, where each residual block contains two convolutional layers and two batch normalization (BN) layers, and ReLU is used as the activation function. The decoder includes one residual block and four sequentially connected decoding processing channels and a regular convolutional layer. Each decoding processing channel includes a sequentially connected deconvolutional layer and three residual blocks. The image reconstruction module sequentially includes two fully connected layers, a size correction layer, three upsampling layers, an addition module, and three convolutional layers; The discriminator comprises four sequentially connected convolutional layers, an average pooling layer, and a fully connected layer. In the image segmentation module excluding the largest scale, the image is first convolved once to obtain a feature map. This feature map is then processed through four encoding steps to obtain feature maps fm1, fm2, fm3, and fm4. The final output feature map fm4 is then input into the ConvLSTM module, which outputs a hidden value H. l The ConvLSTM module serves as the next-scale image segmentation module to achieve information fusion. The other output value of the ConvLSTM module is fed into the subsequent decoder. After being processed by the four decoding channels of the decoder, feature maps fd1, fd2, fd3 and fd4 are obtained respectively. Feature map fd4 is the mesoscale prediction map finally output by the image segmentation module. The image segmentation modules at each scale also include an attention module, which consists of sequentially connected 1×1×1 convolutional layers, an addition module, a ReLU activation function, another 1×1×1 convolutional layer, a sigmoid activation function, a Resamaler function, and a multiplication module. For the current i-th decoding channel, the attention module adds the feature map fd(i-1) output from the previous decoding channel in the decoder through a 1×1×1 convolutional layer and the feature map Wg output from the encoder channel symmetrical to this decoding channel through a 1×1×1 convolutional layer. The result is then passed through a ReLU activation function, then through another 1×1×1 convolutional block, and finally through a sigmoid activation function and a Resamaler function to obtain the weight matrix of the feature map of the previous decoding channel. Multiplying this weight matrix by the feature map of the previous decoding channel allows the network to continuously focus on key regions. In the maximum scale image segmentation module, the feature maps fd1, fd2, fd3 and fd4 obtained from each decoding processing channel are added together on the channel, and then the same convolution operation is performed to obtain the prediction map of the maximum scale image. Then, the large-scale, i.e., the original-scale image segmentation prediction map is input into the channel adder of the image reconstruction module, and the feature map output by the encoder of the large-scale segmentation module is input into the first fully connected layer of the image reconstruction module. After processing by the image reconstruction module, the original image is synthesized to obtain a prediction original image. The discriminator is input to the label image and the ground truth image, as well as the original scale image segmentation prediction image and the prediction image. After passing through the discriminator's final fully connected layer, it outputs 0 and 1, where 0 represents false and 1 represents true. When training the deep learning network model, the image segmentation module and the image reconstruction module are initially trained. The loss function of the segmentation module during training is Diceloss, and the loss function of the reconstruction module is MSEloss. The differences between the three predicted images generated after the images of three scales enter the model and the real labels, as well as the differences between the synthesized image and the actual original image, are calculated. After the initial training of the image segmentation module is completed, the entire network model is trained. At this time, the loss of the entire network model is the sum of the Diceloss of the image segmentation module, the MSEloss of the image reconstruction module, and the MSEloss of the discriminator. After training the deep learning network model, the input image is fed into the maximum-scale image segmentation module to obtain the predicted map of the maximum-scale image, which is the image segmentation result.

2. The method for segmenting local small feature structures based on deep learning as described in claim 1, characterized in that, The fully connected layer of the image reconstruction module outputs two values ​​of 32 and 65536 respectively; the size correction module corrects the image to 64×64; the upsampling factor for each of the three iterations is set to 2.

3. The method for segmenting local small feature structures based on deep learning as described in claim 1, characterized in that, The number of channels used in the encoding processing channels in the image segmentation module at each scale increases sequentially, namely 8, 16, 36, and 64. A BN layer is added in each layer to normalize the data. The convolution kernel size is set to 5×5, and the image is padded with two rows and two columns of 0s around the perimeter.

Images