Cellular lung segmentation method based on class imbalance and multi-level down-sampling feature fusion

Abstract

The application discloses a honeycomb lung segmentation method based on class imbalance and multi-stage down-sampling feature fusion, and relates to the field of medical image processing. The method comprises the following steps: S1, acquiring multiple honeycomb lung CT images, and dividing a training set and a test set; S2, constructing a honeycomb lung segmentation network model; S3, training the honeycomb lung segmentation network model in S2 by using the training set to obtain a trained honeycomb lung segmentation network model; and S4, inputting the test set into the trained honeycomb lung segmentation network model in S3 to obtain segmented honeycomb lung CT images. The method is helpful to improve the segmentation precision of the model on the basis of increasing a small amount of model parameters.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and particularly relates to a honeycomb lung segmentation algorithm based on class imbalance and multi-level down-sampling feature fusion. BACKGROUND

[0002] Honeycomb lung is a high mortality, strong concealment and destructive diffuse lung disease, which is characterized by part of the lung interstitium honeycomb and high density shadow in CT images. The lesion of the disease involves alveolar cavity and lung interstitium, and persistent alveolar epithelial injury causes pulmonary fibrosis, leading to irreversible damage to the lung. Due to its rapid progression, it can cause continuous decline in lung function and respiratory failure in patients, with an incidence and prevalence of 0.09-1.30 and 0.33-4.51 per 10,000 people, respectively. The five-year survival rate of patients is less than 30%, and the median survival time after acute exacerbation is only about 2 months. Therefore, accurate diagnosis of early pathology is crucial for patient prognosis and treatment options. Organ tissue images obtained by medical imaging methods such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound imaging can objectively reflect the lesion.

[0003] Currently, the diagnosis of honeycomb lung disease in the clinic mostly relies on professional radiologists to analyze whether the lung has lesions by analyzing the patient's lung CT images. With the large increase in clinical imaging data, there is a serious shortage of radiology professionals compared to the large population in China. In the process of disease diagnosis, professional doctors manually identify CT images based on their existing knowledge and experience, but a large amount of mental work and long hours of work can easily cause visual fatigue and visual errors, resulting in a high degree of subjectivity in the diagnosis results, and even misdiagnosis and missed diagnosis, thereby increasing the difficulty of treatment for patients in the later stage. Therefore, using image segmentation methods to automatically segment lung lesions can assist doctors in accurately diagnosing the severity of the patient's condition and improve the accuracy and efficiency of doctors in diagnosis and treatment, thereby providing appropriate solutions for guiding clinical decision-making and prognosis treatment, which has very important clinical value.

[0004] In recent years, due to the powerful feature expression ability and the ability to model complex tasks of deep learning, CNN-based methods have been widely used in the field of medical image processing. In particular, the U-shaped convolutional neural network with encoder-decoder structure has achieved remarkable performance in medical image segmentation. Ronneberger et al. first proposed a U-shaped encoder-decoder network model, UNet, for medical image segmentation using the idea of "full convolution". The high-level features of the up-sampling stage are fused with the low-level features of the down-sampling stage using a skip connection, and good segmentation results are obtained on three medical datasets. In order to obtain more accurate edge information, ALOM et al. proposed a R2U-Net model based on recurrent neural network, which uses recurrent residual convolutional layers for feature accumulation. This network can perform better in retinal segmentation tasks while keeping the number of network parameters unchanged. Chen et al. proposed a DeepLabv3+ model with an encoder-decoder structure, which uses atrous convolution and spatial pyramid pooling to increase the image receptive field and extract more rich context information. Chen et al. proposed a TransUNet network for multi-organ segmentation, which uses ViT instead of ordinary convolutional blocks as the encoder base module of the segmentation network, and uses the excellent global information modeling ability of Transformer to realize accurate positioning of the lesion site. Zhao et al. input the texture and color feature maps of the face and hand into the Transformer model to learn the deep features of the region of interest, which helps doctors to complete the detection of stroke disease. These networks have achieved good segmentation results by improving feature extraction, boundary segmentation, etc. However, compared with the UNet network, the parameter quantity of the model has increased by tens of millions. The explosive growth of model parameters can easily lead to network feature redundancy and waste of excessive computing resources, which is not conducive to the lightweight deployment of the model in hospitals.

[0005] In the field of medical image processing, UNet-based medical image segmentation networks have achieved satisfactory segmentation results, but the UNet network model ignores the problem of context feature gap in the skip connection structure. To solve the problem of different scales of medical images, IBTEHAZ et al. proposed a MultiResUNet network for the segmentation of skin diseases, which introduced a residual path block that integrates the residual idea into the skip connection of the UNet network, learned more detailed features using overlapping convolution blocks, and improved the training effect. Similarly, Sharp U-Net applies a sharpened convolution kernel to generate an intermediate feature map instead of a skip connection in the U-shaped network, solving the over-segmentation problem caused by semantic gap. The HDA-ResUNet network uses a channel attention mechanism to process low-level semantic information from the encoder, strengthening the representation of important information in the feature map from the channel perspective, and solving the problem of missing multi-scale information in the network. Based on UNet, UNet++ uses an improved dense skip connection to fuse feature information at different levels, reducing the semantic gap between the encoder and the decoder feature map. Similarly, CA-Net based on the encoder-decoder structure combines spatial attention and scale attention modules to recalibrate channel feature responses and enhance the expression of relevant feature channels in the skip connection stage. However, existing work mainly uses convolution module stacking, channel and spatial-based attention mechanisms, and other methods to improve feature extraction capability to optimize the skip connection, without considering the relevance of scale direction and target features.

[0006] In solving a specific image task, the quality of the data set will not only directly affect the segmentation result, but also depend on a certain specific strategy with a certain direction. If the specificity of the data is not considered, the segmentation model is blindly selected to segment the honeycomb lung lesions, which is easy to cause the segmentation effect to not meet the expectation. In the honeycomb lung CT image data collected in the hospital, different patients are included, and the lesion area in different directions of the lung cross section is also different. The size of the lesion area in each image and the proportion of the image background are quite different, so the data set has the problem of class imbalance.

[0007] Therefore, a honeycomb lung segmentation algorithm based on class imbalance and multi-level down-sampling feature fusion is proposed, which can improve the segmentation accuracy of the model on the basis of increasing a small amount of model parameter quantity, which is a problem that technicians in the field urgently need to solve. SUMMARY

[0008] Therefore, the present application provides a honeycomb lung segmentation algorithm based on class imbalance and multi-level down-sampling feature fusion, which can achieve the effect of improving the segmentation accuracy of the model on the basis of increasing a small amount of model parameter quantity.

[0009] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0010] A honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion, comprising the following steps:

[0011] S1, a plurality of honeycomb lung CT images are acquired, and a training set and a test set are divided;

[0012] S2, a honeycomb lung segmentation network model is constructed;

[0013] S3, the honeycomb lung segmentation network model in S2 is trained using the training set, and a trained honeycomb lung segmentation network model is obtained;

[0014] S4, the test set is input into the trained honeycomb lung segmentation network model in S3, and the segmented honeycomb lung CT image is obtained.

[0015] The system described above, optionally, in S1, the first 80% of the acquired plurality of honeycomb lung CT images is divided into a training set, and the last 20% is divided into a test set.

[0016] The system described above, optionally, the specific content of S2 is:

[0017] A model containing 5 encoders and 4 decoders is constructed, the first CA module is added to the first path of the jump connection part, the second CA module is added to the second path of the jump connection part; the first down-sampling feature fusion module is added to the third path of the jump connection, and the second down-sampling feature fusion module is added to the fourth path of the jump connection.

[0018] The first CA module and the second CA module in S2 respectively enhance the spatial and positional information of the features from the high and wide directions of the honeycomb lung CT image;

[0019] Specifically, the following steps are included:

[0020] 1) Encode along the vertical and horizontal directions of the low-level feature map to generate two separate direction position encoding feature maps, complete the conversion of input features to one-dimensional feature mapping, and perform global average pooling operation on them;

[0021] 2) The first CA module and the second CA module encode spatial information while saving position information from another dimension;

[0022] 3) Use the splicing operation to convert the pooling matrix containing position information and spatial features into a feature mapping, perform channel reduction convolution and non-linear calculation on it to obtain a feature tensor;

[0023] 4) Split the obtained feature tensor into two features, respectively using two two-dimensional convolution kernels to pass, each kernel increases the number of channels to a preset value, and finally uses the Sigmoid function to activate;

[0024] 5) The attention map is multiplied with the original input element by element in order, and finally a high-level feature map containing spatial and position information is obtained.

[0025] The system described above, optionally, the output formula of the xth channel at height g is as follows:

[0026]

[0027] Similarly, the output formula of the kth channel at width l is as follows:

[0028]

[0029] In the formula, H represents the height of the image; W represents the width of the image; g represents a specific value, indicating a certain height; l represents a specific value, indicating a certain width; Xc represents a specific global average pooling operation, and x and k are specific values representing the xth channel and the kth channel.

[0030] The system described above, optionally, the first down-sampling feature fusion module in the third path in S2 is an SK(1, 2, 3)-DFFM1 module, which internally contains a DFFM1 module and a 1x1 convolution, and the DFFM1 module contains two 5x5 convolutions.

[0031] The second down-sampling feature fusion module in the fourth path in S2 is an SK(1, 2, 3, 4)-DFFM2 module; it internally contains a DFFM2 and a 1x1 convolution, and the DFFM2 module contains three large kernel convolutions, which are 9x9, 7x7, and 5x5 convolutions, respectively.

[0032] The system described above, optionally, the DFFM1 module receives features from the first path and the second path, which contains two paths, one side path uses a 5x5 convolution layer with a step size of 4 and a padding size of 2, and the other side path uses a 5x5 convolution layer with a step size of 2 and a padding size of 2.

[0033] DFFM2 is composed of three paths, the first branch of the encoder feature in the first path uses a 9x9 convolution layer with a step size of 8 and a padding size of 2; the second branch uses a 7x7 convolution layer with a step size of 4 and a padding size of 3; and the third branch uses the same 3x3 convolution layer as the right branch in the DFFM1 module.

[0034] The system described above, optionally, in S3, the F-DLoss loss function is used to guide the learning process of the honeycomb lung segmentation network model, and through optimization of the loss direction, a trained honeycomb lung segmentation network model is obtained.

[0035] Compared with the prior art, the application provides a honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion, which has the following beneficial effects: 1) The application is aimed at the semantic gap problem existing in the U-shaped network jump connection stage, and different feature enhancement modules are taken for multiple paths, thereby reducing the information difference between the encoder and the decoder features. 2) In the low-level feature path of the network jump connection, the coordinated attention mechanism is used to re-encode the features from the encoder, and the feature information is captured from three angles of channel, position and direction, helping the model to more accurately locate and identify the lesion area. 3) The application considers the correlation between feature scale direction and target features, and in the high-level feature path of the network jump connection, a multi-level down-sampling feature fusion module is constructed, multiple large kernel convolutions are used for deep feature extraction, the expression of deep implicit features in the image is enhanced, and more rich and multi-scale lesion context information is extracted. 4) The application constructs a new F-DLoss loss function, reduces the influence of simple and difficult samples on the model loss value, and optimizes the network training direction. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0037] Figure 1 A honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion is disclosed in the present application.

[0038] Figure 2 A honeycomb lung segmentation model based on class imbalance and multi-level down-sampling feature fusion is disclosed in the present application.

[0039] Figure 3 A CA module structure diagram is disclosed in the present application.

[0040] Figure 4 A DFFM1 and SK(1, 2, 3)-DFFM1 module structure diagram is disclosed in the present application.

[0041] Figure 5 A DFFM2 and SK(1, 2, 3, 4)-DFFM2 module structure diagram is disclosed in the present application.

[0042] Figure 6 A region-based Dice loss schematic diagram is disclosed in the present application.

[0043] Figure 7The training set and test set loss and IoU value statistical chart of the training network disclosed in the present application;

[0044] Figure 8 The segmentation effect diagram of adding CA module and CFB module to the hop connection stage of the network disclosed in the present application;

[0045] Figure 9 The influence statistical chart of the F-DLoss function with different alpha values on the model evaluation index IoU disclosed in the present embodiment;

[0046] Figure 10 The segmentation effect diagram of the plurality of segmentation networks disclosed in the present embodiment. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0049] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0050] Referring to Figure 1 The present application discloses a honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion, comprising the following steps:

[0051] S1 obtains a plurality of honeycomb lung CT images, and divides the training set and the test set;

[0052] S2 constructs a honeycomb lung segmentation network model;

[0053] S3 trains the honeycomb lung segmentation network model in S2 using the training set to obtain a trained honeycomb lung segmentation network model;

[0054] S4 inputs the test set into the honeycomb lung segmentation network model trained in S3 to obtain a segmented honeycomb lung CT image.

[0055] Optionally, in S1, the first 80% of the acquired plurality of honeycomb lung CT images is divided into a training set, and the last 20% is divided into a test set.

[0056] Further, referring to FIG. 2, the specific content of S2 is as follows: Figure 2

[0057] A model containing 5 encoders and 4 decoders is constructed, the first CA module is added to the first path of the skip connection part, the second CA module is added to the second path of the skip connection part, the first down-sampling feature fusion module is added to the third path of the skip connection, and the second down-sampling feature fusion module is added to the fourth path of the skip connection.

[0058] Further, the first CA module and the second CA module in S2 respectively enhance the spatial and positional information of the features from the high and wide directions of the honeycomb lung CT image.

[0059] Specifically, the following steps are included:

[0060] 1) Two separate direction position encoding feature maps are respectively generated along the vertical and horizontal directions of the low-level feature map, completing the conversion of the input features to one-dimensional feature mapping and performing global average pooling operation on them;

[0061] 2) The first CA module and the second CA module encode spatial information while saving position information from another dimension;

[0062] 3) The pooling matrix containing position information and spatial features is converted into a feature map using a concatenation operation, and a channel reduction convolution and a nonlinear calculation are performed to obtain a feature tensor;

[0063] 4) The obtained feature tensor is split into two features, which are respectively transmitted by two two-dimensional convolution kernels, each kernel increases the number of channels to a preset value, and finally uses a Sigmoid function to activate;

[0064] 5) The attention map is multiplied by the original input element by element in order, and finally a high-level feature map containing spatial and positional information is obtained.

[0065] Further, the output formula of the xth channel at height g is as follows:

[0066]

[0067] Similarly, the output formula of the kth channel at width l is as follows: ​

[0068]

[0069] In the formula, H represents the height of the image; W represents the width of the image; g represents a specific value, indicating a certain height; l represents a specific value, indicating a certain width; Xc represents a specific global average pooling operation, x and k are specific values, representing the xth channel and the kth channel.

[0070] Further, the first down-sampling feature fusion module in the third path in S2 is an SK(1, 2, 3)-DFFM1 module, which internally contains a DFFM1 module and a 1x1 convolution, and the DFFM1 module contains two 5x5 convolutions.

[0071] The second down-sampling feature fusion module in the fourth path in S2 is an SK(1, 2, 3, 4)-DFFM2 module; it internally contains a DFFM2 and a 1x1 convolution, and the DFFM2 module contains three large kernel convolutions, which are 9x9, 7x7 and 5x5 convolutions, respectively.

[0072] Further, the DFFM1 module receives features from the first path and the second path, which contains two paths, one side path uses a 5x5 convolution layer with a step size of 4 and a padding size of 2; the other side path uses a 5x5 convolution layer with a step size of 2 and a padding size of 2.

[0073] DFFM2 is composed of three paths, the first branch of the encoder feature in the first path uses a 9x9 convolution layer with a step size of 8 and a padding size of 2; the second branch uses a 7x7 convolution layer with a step size of 4 and a padding size of 3; the third branch uses the same 3x3 convolution layer as the right branch in the DFFM1 module.

[0074] Further, in S3, the F-DLoss loss function is used to guide the learning process of the honeycomb lung segmentation network model, and through the optimization of the loss direction, a trained honeycomb lung segmentation network model is obtained.

[0075] Reference Figure 2As shown, the application discloses a honeycomb lung segmentation model based on class imbalance and multi-level down-sampling feature fusion: taking UNet as the backbone network, a total of 5 encoders and 4 decoders are contained; wherein, each encoder and decoder in the network contains two 3*3 ordinary convolution modules, BN and Relu activation function, a total of 18 ordinary convolution layers and 4 transpose convolution layers are contained; the CA module is added to the 1, 2 paths of the jump connection part, receiving the low-level semantic information from the encoder, using channel attention calculation and fully activating the position and spatial information in the features; the newly constructed SK(i)-DFFM module is added to the 3, 4 paths of the jump connection respectively, the SK(1, 2, 3)-DFFM1 module inside the 3 path contains a DFFM1 module and a 1*1 convolution, the DFFM1 module contains two 5*5 convolutions, and the 1*1 convolution is used to modify the dimension of the feature map; the SK(1, 2, 3, 4)-DFFM2 module inside contains DFFM2 and a 1*1 convolution, the DFFM2 module contains three large kernel convolutions, which are 9*9, 7*7 and 5*5 convolutions respectively, the multi-scale feature fusion from the encoder in this module can improve the acquisition ability of the foreground in the image, and the large step and large size convolution kernel in the module is beneficial to extract the global features in the feature map. Finally, F-DLoss is used to guide the learning process of the network in the training stage, and through the optimization of the loss direction, the segmentation of the honeycomb lung CT image is completed.

[0076] Because the splicing operation in the jump connection stage of the U-shaped network will cause the loss of part of the feature information, and the deep-level features of the low-level domain information are not activated. Therefore, a novel and efficient attention mechanism is used in the jump connection part of the U-shaped network, the Coordinate Attention (CA) module is used to perform feature calculation and selection on the low-level features obtained in the U-shaped network, and the scale sparsity features are spliced and covered to a larger area, while avoiding a large amount of calculation overhead.

[0077] Reference Figure 3 As shown, the CA module preserves the original feature vector by using the residual idea, and performs deformation operation on the original features in the new path, respectively enhances the spatial and position information of the features from the height and width directions of the image. Specifically, first, two separate direction position coding feature maps are generated by separately coding along the vertical and horizontal directions of the feature map, the conversion of the input features to one-dimensional feature mapping is completed, and the global average pooling operation is performed on them, that is, the feature map uses two spatial range pooling kernels (H, 1) and (1, W) to encode the feature channels along the horizontal and vertical coordinate directions. While encoding the spatial information, the module saves the position information from another dimension, ensuring accurate capture of the spatial structure of the image by the visual task. At this time, the output formula of the xth channel at height g is as follows.

[0078]

[0079] Similarly, the output formula of the kth channel at the wide l is as follows.

[0080]

[0081] In the formula, H represents the height of the image; W represents the width of the image; g represents a specific value indicating a certain height; l represents a specific value indicating a certain width; Xc represents a specific global average pooling operation, x and k are specific values representing the xth channel and the kth channel.

[0082] After the pooling method of restoring the horizontal and vertical boundary detection, the pooling matrix containing position information and spatial features is converted into a feature map using the splicing operation, and the channel reduction convolution and nonlinear calculation are performed on it; the Swish activation function is used after the channel reduction in the CA module, but the channel reduction rate and the selection of the nonlinear activation function in the module depend on the position and the type of the task of the module, so the Relu function is used in the module. In order to restore the channel number of the feature map, the obtained feature tensor is split into two features, which are transmitted by two two-dimensional convolution kernels respectively, and the channel number is increased to a preset value by each kernel, and finally the Sigmoid function is activated. The activated two feature weight vectors belong to attention mapping; the attention map is multiplied with the original input element by element in order, and finally the feature map containing spatial and position information is obtained, which significantly improves the segmentation performance of the network.

[0083] In the U-shaped network, the encoder features have 4 paths output to the decoder end, but each jump connection path can only transmit low-level features with the same size as the current decoder features, and multiple convolution and pooling operations in the network will cause part of the information to be lost. The present application verifies each path in the jump connection stage, and compared with the 3rd and 4th jump connection paths, the low-level features transmitted by the 1st and 2nd paths in the network have a more promoting effect on the network and the decoder features.

[0084] Referring to Figure 4 and Figure 5 , the present application proposes a downsampling feature fusion module (DFFM), which constructs a SK(i)-DFFM module in the jump connection stage, aiming to improve the utilization rate of features in the encoder and make up for the lost features caused by the pooling operation at the decoder end.

[0085] Considering the characteristics of the jump connection structure, the present application proposes two different downsampling feature fusion modules to replace the jump connection paths in the network. For Figure 4 and Figure 5, the left side is the DFFM1, DFFM2 receiving the features of the upper layer encoder, the path in the module is determined by the features of the encoder, aiming to make up for the lost features in the network pooling operation process, so the module does not use a 3x3 dense convolution kernel, but uses a larger size convolution kernel; for the SK(1, 2, 3)-DFFM1 on the right side of the figure is the jump connection structure after adding the DFFM module. In order to prevent the step size of the convolution kernel from being too large and causing part of the feature map to be ignored, the step size of the convolution kernel used in the DFFM module is smaller than the size of the convolution kernel. Among them, the DFFM1 module receives features from the 1st and 2nd paths, a total of two paths, the left path uses a 5x5 convolution layer with a step size of 4 and a padding size of 2; the right path uses a 5x5 convolution layer with a step size of 2 and a padding size of 2. DFFM2 is composed of three paths, the left branch receiving the 1st encoder feature uses a 9x9 convolution layer with a step size of 8 and a padding size of 2; the middle branch uses a 7x7 convolution layer with a step size of 4 and a padding size of 3; the right branch uses the same 3x3 convolution layer as the right branch of the DFFM1 module. After each large kernel convolution in the module, Gelu is used as a nonlinear activation unit to increase the discriminability of nonlinear features.

[0086] In the SK(1, 2, 3)-DFFM1 module, in addition to receiving the features of the DFFM1 module, the identity mapping features of the current encoder stage are also added to ensure that the semantic information is transmitted and prevent the occurrence of gradient anomalies; the identity mapping path and the multi-scale sparse feature mapping obtained by the DFFM module are spliced, and finally a 1x1 convolution is used to compress the channel number to adapt the dimension of the feature map to the input of the subsequent module. Similar to the SK(1, 2, 3)-DFFM1 module, the SK(1, 2, 3, 4)-DFFM2 module fuses the received 3 encoder features of the upper layer with the features of the current layer, and finally uses 1x1 to modify the dimension of the feature map to increase the nonlinear fitting ability of the network.

[0087] Referring to Figure 6 As shown in the figure, the Dice loss is a region-based loss function, which focuses on the degree of overlap between the real region and the predicted region. The CT image of honeycomb lung in the data set used by the present application contains 163 patients, each patient contains several slices, which represent images of different direction cross sections. However, due to the small lesion area of some slices of the patient, the lesion area and the background ratio is small, thereby increasing the training difficulty, and the segmentation effect of small lesions is not ideal. Therefore, for class imbalance data in semantic segmentation tasks, MILLETARI et al. proposed a new loss function dice loss to solve the case where the ratio of background pixels to target pixels in the image is too large.

[0088] The calculation formula of the Dice loss is as follows:

[0089]

[0090] Where, p i It is the predicted pixel value, g i is the pixel value of the true label, and N is the total number of pixels in the image. The original Dice loss is calculated by adding all pixel values ​​of the predicted label and the true label as the denominator, and multiplying each pixel value of the predicted label and the true label and summing the results as the numerator. To ensure the stability of the calculation (the denominator is not zero), the square of each pixel value in the denominator is squared before summing. At this point, p i The pixel representing the predicted label, g i g represents the pixels representing the background or lesion area in the labeled image. i The values ​​are 0 or 1, where 0 represents a background area unrelated to the lesion in the celluloid lung image, 1 represents a celluloid lung lesion area, and N represents the total number of pixels in the image.

[0091] When calculating the loss using Dice loss, the loss value of a single pixel is not only related to the predicted pixel value but also correlated with the predicted pixel values ​​of other points. The calculation of Dice loss can be viewed as a masking operation; in images of different sizes, the same proportion of positive samples has the same effect on network training when calculating the loss. During training, Dice loss favors the foreground portion of the image. The imbalance between positive and negative samples is precisely because the foreground portion accounts for a smaller proportion; during the gradient update process for positive and negative samples in Dice loss, the gradient value is larger compared to the background region. However, if the positive sample is a target with a small area, the loss function will oscillate. This is because once a portion of the pixels of a small target is predicted, Dice's preference for foreground attributes will lead to highly unstable loss values, causing gradient saturation.

[0092] Reference Figure 7 The diagram shows the visualization of loss and IoU values. The left figure shows the loss between the training and test sets when training the UNet network with the CA module added using the Dice loss function; the right figure shows the IoU values ​​when using BCE and Dice losses in the UNet network with the CA module added.

[0093] Depend on Figure 7It can be seen that the loss value in the early stage of training still maintains a downward trend although it appears several obvious shocks; when the training times epoch is equal to 50, trainloss and testloss no longer decrease, and the IoU value also remains in an error range that is less than 85%, and the network reaches a convergent state. However, compared with using the BCE Loss loss function, the IoU value is at least 2% lower, proving that the update bias of the gradient of the positive sample in the Dice loss at this time is too large, and the learning intensity of the network for simple and difficult samples in the image is the same in the network training process, which makes the network deviate from the correct learning direction, and the accuracy is easy to fall into a local optimal solution. Although the Dice loss can enhance the learning of the foreground region in the network, it cannot distinguish between simple and difficult samples, and cannot better control the correct direction of the gradient update of the loss function. If a penalty term for punishing simple samples or a regularization term for promoting the learning of difficult samples is added to the loss function, the dominant loss function can better grasp the update trend of the gradient of the positive sample.

[0094] Similar to the Dice loss, the Focal loss was initially proposed to deal with class imbalance in the target detection task, and later extended to pixel-level tasks such as medical image segmentation. This loss function improves the cross-entropy by setting a balance factor a e [0, 1] to control. Due to the different sizes of target organs and tissues in medical imaging, there is an imbalance between the segmentation of large background areas and small target areas in the training process of the segmentation network, but this loss function can solve the problem of imbalance in training samples while also solving the problem of different training levels of difficult and simple samples in the samples. For the input matrix x and the prediction matrix The Focal loss is defined as follows:

[0095]

[0096] where a and g are hyperparameters, and a can be adjusted to adjust the imbalance problem in the sample category; the size of g adjusts the proportion of simple and difficult samples in the sample number, thereby reducing the influence of the large difference on the loss function. Usually, most of the samples in a class are easy to predict, but due to the dominant loss function and the gradient process, it is difficult to predict the class containing a small number of samples. However, the Focal loss function reduces the weight of the simple sample in the loss function by setting two hyperparameters, solving the class imbalance problem. However, unlike the Dice loss, the Focal loss and the cross-entropy are added to all losses to complete the calculation, and this calculation method is absolute, while the Dice loss can consider the proportion of the number of correctly predicted pixels in the true label, so even if the area of the target region is very small, the loss value will still remain in a large range, and the loss can be calculated from a global perspective.

[0097] Focal loss has the ability to solve sample imbalance and distinguish simple and difficult samples at the same time; Dice loss solves the sample imbalance problem by considering the area ratio to complete the calculation of the loss function. Therefore, the present application proposes a F-DLoss loss function from different angles to solve the data sample imbalance, which contains Focal loss and Dice loss with different weight coefficients.

[0098] L F-DLoss =μL focal +κL dice ,μ+κ=1 (5)

[0099] Wherein, mu and kappa are hyperparameters, by setting the size of the hyperparameters, the proportion of different classes in the training sample is completed, so that the model pays more attention to the segmentation foreground. Through experiments, it is found that the setting of the hyperparameters is closely related to the shape and size of the lesions in the data set, and by setting reasonable mu and kappa values, the honeycomb lung lesion segmentation accuracy can be improved.

[0100] To prove the effectiveness of the CA module in the U-shaped network jump connection part, the context-aware fusion module and the CA module are added to the jump connection part respectively, and the experimental results are shown in Table 1. Since the U-shaped network contains 4 jump connection paths, the second column in the table represents the jump connection path number of the module added.

[0101] Table 1 comparison experiment

[0102]

[0103] Comparing the paths in the jump connection stage of the segmentation network, the present embodiment experiment compares path 1 and path 2 with path 3 and path 4, and it can be seen from Table 1 that the effect of adding a module to path 1 and path 2 is better than that of path 3 and path 4, which shows that the features of path 1 and path 2 in the jump connection path contain more lesion information, and the original UNet network does not dig deeper into the feature information, and the model performance is limited.

[0104] As can be seen from Table 1, the network effect of the U-CFB(1,2) model is significantly improved, and each evaluation index reaches the best, proving that the module not only can promote the effective propagation of features in the skip connection, but also can extract low-level features in the encoder in a deep way. In the model with the added CA module, compared with the original UNet network model, the Jaccard coefficient of U-CA(1,2) is increased by 1.72%, the mIoU and DSC are increased by 0.88% and 1.14% respectively; but compared with U-CFB(1,2), the Jaccard coefficient and other three indicators are more than 0.1% lower, but the parameter amount of the module is only increased by 0.1M compared with the original UNet. By comparing and analyzing the two models of U-CFB(1,2) and U-CA(1,2), the segmentation effect of U-CFB(1,2) is the best, and the difference between the model segmentation indexes is only within 0.1%; compared with the original UNet, the parameter amount of U-CFB(1,2) model is increased by 1M, while the parameter amount of U-CA(1,2) is only increased by 0.01M. Therefore, compared with other models in the table, the U-CA(1,2) model can get a segmentation accuracy of 93.99% for the evaluation index DSC on the honeycomb lung image with only a small amount of parameter increase. Therefore, the U-CA(1,2) network model is better than other networks in segmenting honeycomb lung CT images.

[0105] Referring to Figure 8 The CA and CFB modules are added to the multiple paths of the skip connection of the UNet model for experiments, and the segmentation results of the honeycomb lung images are shown. The experiment contains three honeycomb lung example images with different lesion sizes, Figure 8 The first two columns in the table are the original honeycomb lung image and the gold standard annotation image, the third and fourth columns are the segmentation results of the CA and CFB modules added to the 1st and 2nd paths of the network's skip connection, and the fifth and sixth columns are the segmentation results of the CA and CFB modules added to the 3rd and 4th paths. From the table, Figure 8As can be seen from the enlarged part of the segmentation result of the third column in the first row (UNet+CA(1,2)), there is an incomplete segmentation in the upper right corner of the segmentation edge part, the segmentation result of the fourth column in the first row (UNet+CFB(1,2)) has a relatively smooth contour, and the segmentation effect is better, and there is an error in the right upper corner of the lesion details in the last two columns in the first row, which is identified as background; in the second row, there is under-segmentation of the moon-shaped lesion contour at the top of the image, and the fusion of the UNet network and the CA(1,2) and CFB(1,2) structure can still achieve better segmentation results; for the lesion edge shape pointed by the red arrow in the third row, the CA(1,2) and CFB(1,2) structure has a relatively clear detailed shape in the honeycomb lung segmentation image, while the CA(3,4) and CFB(3,4) structure has a relatively smooth lesion shape, but there is no more clear and accurate lesion contour. In summary, since the CFB module contains more network parameters, and the CA module structure with fewer parameters can achieve the same segmentation performance as the CFB module, therefore, the embodiment selects to add the CA module in the 1st and 2nd paths of the skip connection structure, and re-calculates the low-level features from the encoder in the horizontal and vertical directions to activate the hidden features and improve the segmentation effect of the network.

[0106] To verify the performance of the DFFM module in the U-shaped network, the embodiment adds the module to each path of the skip connection for experimental comparison and analysis: in Table 2, model 2 separately adds DFFM1 module from the features of two encoders in the U-shaped network, uses SK(1,2,3)-DFFM1 as the skip connection structure of the 3rd path in the network, and the Jaccard coefficient and mIoU are increased to nearly 87.98% and 94.35%, respectively, and the improvement value of DSC is nearly 1%, which indicates that the skip connection structure which simply splices the features will suppress the expression of part of the features, the sparse feature information extracted by the convolution layer with a larger step size and convolution kernel size has a larger receptive field, and multiple convolution layers contain multi-scale position detail information, which can enhance the expression of context features in the image; the Jaccard coefficient, mIoU and DSC of model 3 are 87.76%, 94.93% and 93.59%, respectively, which are better than the original UNet network; the Jaccard coefficient of the network adding SK(1,2,3)-DFFM1 and SK(1,2,3,4)-DFFM two modules is 88.16%, and the DSC and mIoU are improved, and the performance of the model reaches the best. The above experimental results prove that the multi-scale information from the encoder is transmitted to the decoder, which can deeply extract the semantic features ignored in the early stage of the network, and these features contain position encoding information and contour features of the honeycomb lung lesion, which improves the segmentation accuracy of the honeycomb lung to a certain extent, and proves the effectiveness of the module.

[0107] Table 2 comparative experiments

[0108]

[0109]

[0110] Focal loss can solve the problems of sample imbalance and the distinction between simple and difficult samples by simply calculating all losses, and Dice loss increases the weight of the predicted correct samples and the real correct sample coverage area from the perspective of region, and balances the positive and negative samples. Therefore, the embodiment analyzes the two in the U-shaped network through experiments, and the experimental results are shown in Table 3. First, the Focal loss and the Dice loss are added separately in the model for comparative experiments, and the network with the added Focal loss has an increase of nearly 0.4% in the Jaccard coefficient compared with before, and the corresponding mIoU and DSC indicators also increase by nearly 0.2%, which proves that the Focal loss can solve the class imbalance problem in the honeycomb lung CT image and improve the segmentation accuracy of the network.

[0111] Referring to Figure 9 Although the network with the Dice loss function performs worse than before, to explore the influence of loss calculated from different angles on the performance of the network model, the loss function proposed in the embodiment fuses the Focal loss and the Dice loss, and through multiple experiments with different weights set in the loss function (wherein, α and β are the weight coefficients of the Focal loss and the Dice loss, respectively), in the process of changing the value of α from large to small, the effect of the segmentation model presents a bimodal function, and the highest point of the function is 89.41%, which is almost the same as the segmentation effect of adding the Focal loss function alone. Although the loss value and the gradient are dominated by the Focal loss, the experimental results prove that when α is 0.9 and β is 0.1, the F-DLoss (α = 0.9, β = 0.1) can still obtain the best segmentation result. Therefore, when segmenting the honeycomb lung CT image, the F-DLoss function used in the embodiment is better than the BCELoss and the Dice loss, and can specifically reduce the sample difference in the training data, increase the weight of the image difficult to identify the lesion in the training process, and play a positive role in the segmentation network.

[0112] Table 3 F-DLoss ablation experiment analysis

[0113]

[0114]

[0115] The ablation experiment results of the embodiment are shown in Table 4. As can be seen from the table, the various segmentation indexes of the UNet network in the honeycomb lung data set reach 87.29%, 92.85%, and 93.5%. Although the segmentation result is the lowest in the table, by referring to the related literature and the segmentation result of the network in the honeycomb lung data set, it can still be seen that the UNet network has excellent generalization on medical images, and therefore the UNet network is used as the backbone network in the embodiment.

[0116] Compared with the original UNet network, the U-shaped network added with CA and F-DLoss improves the Jaccard coefficient by 1.72% and 1.63% respectively, but the UNet network added with the CA module improves the segmentation indexes mIoU and DSC by 1.53% and 0.49% respectively. The CA module extracts the position and spatial information in the encoder features by using channel attention, thereby reselecting and calculating the features and guiding the network to focus on the key regions in the image. The UNet network added with F-DLoss also has obvious improvement in the segmentation indexes mIoU and DSC, proving that F-DLoss can balance the attention degree of foreground and background at the same time, improve the attention of the network to difficult samples, and thereby improve the segmentation accuracy of the model. The Jaccard coefficient, DSC, and mIoU of the U-shaped network added with CA, F-DLoss, and DFFM reach 90.67%, 95.17%, and 94.81% respectively, and various segmentation indexes are greatly improved as a whole. Therefore, the embodiment model achieves the best segmentation result on the honeycomb lung data set, proving the excellent performance of the model.

[0117] Table 4 Ablation experiment

[0118]

[0119]

[0120] In order to quantitatively analyze the method of the embodiment, comparative experiments are performed, and the results are shown in Table 5. Compared with the UNet network, the Jaccard coefficient, mIoU, and DSC segmentation indexes of the method of the embodiment are improved by 3.38%, 1.67%, and 1.96% respectively. Compared with the HDA-ResUNet network, the Jaccard coefficient is 0.04% higher, but the mIoU and DSC are 0.03% and 0.18% lower respectively, reaching the same level of segmentation performance as the existing popular models. The parameter amount of the original UNet network is only 2.1M different from the parameter amount of the embodiment model, and the embodiment model can achieve better segmentation performance on the basis of increasing a small amount of parameter amount.

[0121] Reference Figure 10As shown, from the segmentation effects of all models on the first honeycomb lung lesion image, it can be seen that the lesions segmented by the SETR model (the part pointed to by the red arrow) are obviously different, and errors occur in segmentation. The segmentation method of the embodiment can well segment the shape of the lesion, which is almost consistent with the gold standard shape. For the second image, some network models mistakenly identify the irrelevant background in the middle part of the lung as a lesion. The method of the embodiment can still accurately identify and segment the background and foreground. In the third honeycomb lung image with a larger lesion area and rich edge shape, the method of the embodiment is more accurate in segmenting the boundary of the lesion part on the inner side of the lung, and there is no edge sawtooth phenomenon as in the Sharp U-Net and SETR. For the fifth image to be segmented, the method of the embodiment can still accurately outline the shape of the honeycomb lung in the boundary part. However, for the fourth image, the prediction of some pixels is wrong when the embodiment method segments the lesion in the upper left part, resulting in the appearance of "point cloud-like" pixels in the segmented binary image, and the segmentation result is not accurate. Therefore, by comprehensively analyzing all the experiments and the segmentation effect diagrams obtained by the experiments in the embodiment, the algorithm of the embodiment still has good segmentation performance.

[0122] Table 5 Comparative experiment

[0123]

[0124]

[0125] Due to the class imbalance problem of honeycomb lung CT images, the existing segmentation network has poor generalization for honeycomb lung dataset. To solve this problem, the application proposes a new loss function F-DLoss, which can not only balance the proportion of different classes of positive and negative samples, but also increase the proportion of difficult sample training, and improve the segmentation accuracy of foreground and background. In addition, the simple splicing operation in the U-shaped network jump connection leads to the scale asymmetry between features, which affects the segmentation result. Based on this, the embodiment first verifies the effective feature of each path in the jump connection through experimental analysis, adds CA modules in the first and second paths, strengthens the extraction of position and spatial information in the feature map, and more accurately captures the spatial structure of the image; in order to improve the effective utilization rate of the third and fourth path features, the image features are more completely recovered in the decoder upsampling process, the embodiment constructs a series of DFFM modules, receives features from multiple jump connection paths, fuses global information and abstract local features, effectively supplements the detail information of the lesion edge, and optimizes the segmentation result. The CA module, the DFFM module and the F-DLoss loss function are proved to be effective for honeycomb lung CT image segmentation through independent ablation experiments; then through ablation experiments and comparative experiments, and the display of model segmentation effect image, it is shown that the method of the embodiment is superior to most of the current mainstream segmentation algorithms, and the superiority of the method of the application is verified.

[0126] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related part can be referred to the part of the method embodiment. The system and system embodiment described above are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0127] In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been described in the above description in general terms according to functions. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical scheme. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0128] The foregoing description of the disclosed embodiments enables one skilled in the art to make or use the application. Numerous modifications of those embodiments can be apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the innovation falling outside the spirit and scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion, characterized in that, The method comprises the following steps: S1 obtaining multiple honeycomb lung CT images, and dividing a training set and a test set; S2 constructing a honeycomb lung segmentation network model; S3 training the honeycomb lung segmentation network model in S2 using the training set to obtain a trained honeycomb lung segmentation network model; S4 inputting the test set into the trained honeycomb lung segmentation network model in S3 to obtain segmented honeycomb lung CT images; The specific content of S2 is: A model containing 5 encoders and 4 decoders is constructed, taking UNet as the backbone network, a first CA module is added to the first path of the jump connection part, a second CA module is added to the second path of the jump connection part, a first down-sampling feature fusion module is added to the third path of the jump connection, and a second down-sampling feature fusion module is added to the fourth path of the jump connection; The first CA module and the second CA module in S2 respectively enhance the spatial and positional information of the features from the height and width directions of the honeycomb lung CT image; The specific content comprises the following steps: 1) separately encoding along the vertical and horizontal directions of the low-level feature map to generate two separate direction position encoding feature maps, completing the conversion of the input features to one-dimensional feature mapping and performing global average pooling operation on the one-dimensional feature mapping; 2) the first CA module and the second CA module encode spatial information while saving position information from another dimension; 3) using a splicing operation to convert the pooling matrix containing position information and spatial features into a feature mapping, performing channel reduction convolution and non-linear calculation on the feature mapping to obtain a feature tensor; 4) splitting the obtained feature tensor into two features, respectively using two two-dimensional convolution kernels to transmit, each kernel increases the number of channels to a preset value, and finally uses a Sigmoid function to activate; 5) multiplying the attention map element by element in order with the original input to finally obtain a high-level feature map containing spatial and positional information; The first down-sampling feature fusion module in the third path in S2 is an SK(1, 2, 3)-DFFM1 module, which contains a DFFM1 module and a 1x1 convolution inside, and the DFFM1 module contains two 5x5 convolutions; The second down-sampling feature fusion module in the fourth path in S2 is an SK(1, 2, 3, 4)-DFFM2 module; it contains a DFFM2 and a 1x1 convolution inside, and the DFFM2 module contains three large kernel convolutions, which are 9x9, 7x7 and 5x5 convolutions; The DFFM1 module receives features from the first path and the second path, which contains two paths, one side path uses a 5x5 convolution layer with a step size of 4 and a padding size of 2; the other side path uses a 5x5 convolution layer with a step size of 2 and a padding size of 2; The DFFM2 is composed of three paths, the first branch of the encoder feature in the first path uses a 9x9 convolution layer with a step size of 8 and a padding size of 2; the second branch uses a 7x7 convolution layer with a step size of 4 and a padding size of 3; the third branch uses the same 3x3 convolution layer as the right branch in the DFFM1 module.

2. The honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion according to claim 1, characterized in that, In S1, the first 80% of the acquired multiple honeycomb lung CT images are divided into a training set, and the last 20% are divided into a test set.

3. The honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion according to claim 1, characterized in that, The output formula of the xth channel at height g is as follows: （1） Similarly, the output formula of the kth channel at width l is as follows: （2） wherein, H represents a height of an image; W represents a width of an image; g represents a specific value, refers to a certain height; l represents a specific value, refers to a certain width; Xc represents a specific global average pooling operation, x , k is a specific value, represents the first x channel and the first k channel.

4. The honeycomb lung segmentation method based on class imbalance and multi-level down-sampling feature fusion according to any one of claims 1-3, characterized in that, In S3, the F-DLoss loss function is used to guide the learning process of the honeycomb lung segmentation network model, and through optimization of the loss direction, a trained honeycomb lung segmentation network model is obtained.

Images