A CT image segmentation method based on dual-branch CNN multi-scale fusion network

Abstract

The present invention relates to a CT image segmentation method based on a dual-branch CNN multi-scale fusion network. A low-level detail branch is used to extract spatial detail features from a lung CT image to retain high-resolution detail features and capture small lesions and complex boundaries. A high-level semantic branch is used to capture low-resolution global semantic information. A boundary information block is introduced to perform weighted fusion on a detail feature map and a global semantic information feature map to generate high-quality boundary features. The boundary features are integrated into the detail feature map to make the boundary of the lesion area clearer. An adaptive bidirectional weighted fusion block adaptively aggregates the features of the low-level detail branch and the high-level semantic peak value to maximize the retention of their respective advantages, effectively improve the overall lesion recognition ability of the dual-branch CNN multi-scale fusion network model, realize the accurate segmentation of small lesion areas and complex lesion areas of the lung CT image, and improve the segmentation effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a CT image segmentation method based on a double-branch CNN multi-scale fusion network. Background Art

[0002] CT images have gradually gained attention as an auxiliary diagnostic method for pneumonia. However, manual diagnosis of CT images is not only time-consuming and laborious, but also the efficiency and accuracy of manual diagnosis are easily limited, especially when faced with a large number of images, which brings new challenges to clinical diagnosis. With the rapid development of artificial intelligence technology, medical image segmentation technology based on deep learning has shown great potential in CT image segmentation. Convolutional neural network (CNN) architectures such as UNet are widely used in medical image segmentation due to their encoder-decoder structure and jump connections, and have produced a variety of improved variants, such as UNet++, DeepLab, and Y-Net. In recent years, the introduction of Transformer has further promoted the development of segmentation technology. Visual Transformer (ViT) has been applied to medical image segmentation as a backbone network. Typical models such as TransUNet, SegFormer, and SegVit have performed well in capturing global context information.

[0003] Although the above methods have made some progress, CT image segmentation still faces several challenges. First, pneumonia lesions have diverse morphologies, including complex pathological features such as ground-glass opacities and consolidation, which increases the difficulty of segmentation, especially in small lesion areas; second, the lesion boundaries are blurred and difficult to distinguish from normal lung tissue, making it difficult for the model to accurately segment the boundaries. In addition, noise and artifacts in CT images have a negative impact on segmentation accuracy and increase the computational burden of the model. To address the above problems, some studies have combined CNN and Transformer to form a dual-branch architecture. However, this type of architecture also has disadvantages such as high model complexity, feature conflicts, and information confusion, making it difficult to achieve efficient segmentation under limited resource conditions. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to solve the problems of blurred boundaries, noise interference and detail loss in CT images and improve the segmentation accuracy of CT images.

[0005] The present invention provides a CT image segmentation method based on a dual-branch CNN multi-scale fusion network, comprising:

[0006] Step 1, building a dual-branch CNN multi-scale fusion network model and training the dual-branch CNN multi-scale fusion network model; the dual-branch CNN multi-scale fusion network model includes a low-level spatial detail branch, a high-level semantic branch, a boundary information block, a bidirectional weighted fusion block and a segmentation head;

[0007] Step 2, obtaining a lung CT image to be segmented;

[0008] Step 3: Use the low-level spatial detail branch to extract detail features from the lung CT image to obtain detail feature maps of different sizes. ;

[0009] Step 4: Preprocess the lung CT images, use high-level semantic branches to extract global semantic information from the preprocessed lung CT images, and obtain global semantic information feature maps of different sizes. ;

[0010] Step 5: Use the boundary information block to transform the detail feature map and global semantic information feature map Perform weighted fusion to obtain boundary features ;

[0011] Step 6: Boundary features With detail feature map Fusion to obtain boundary detail features ;

[0012] Step 7: Use a bidirectional weighted fusion block to combine boundary detail features and global semantic information feature map Perform bidirectional weighted fusion to obtain feature maps ;

[0013] Step 8: Use the segmentation head based on the feature map Perform image segmentation and obtain the target segmentation map.

[0014] Compared with the prior art, the present invention has the following advantages: the present invention uses a low-level detail branch to extract spatial detail features from lung CT images to retain high-resolution detail features and capture small lesions and complex boundaries; uses a high-level semantic branch to capture low-resolution global semantic information and enhance the recognition ability of the overall lesion; and introduces boundary information to quickly identify the detail feature map. and global semantic information feature map Perform weighted fusion to generate high-quality boundary features and integrate boundary features into detail feature maps In the experiment, the boundary of the lesion area is made clearer; then, the adaptive bidirectional weighted fusion block is combined to adaptively aggregate the features of low-level detail branches and high-level semantic peaks, maximizing the retention of their respective advantages, effectively improving the overall lesion recognition ability of the dual-branch CNN multi-scale fusion network model, and realizing the accurate segmentation of small lesion areas and complex lesion areas in lung CT images, thus improving the segmentation effect.

[0015] In a possible implementation, the low-level spatial detail branch includes a first-order low-level spatial block, a second-order low-level spatial block, and a third-order low-level spatial block connected in sequence, wherein the first-order low-level spatial block extracts detail features of the lung CT image to obtain a detail feature map. The second-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map , the third-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map .

[0016] Compared with the existing technology, the first-order low-level spatial blocks, second-order low-level spatial blocks and third-order low-level spatial blocks connected in sequence are used to extract detail features, so as to retain high-resolution detail features, thereby capturing small lesions and complex boundaries and ensuring that the details of the lesions are accurately segmented.

[0017] In a possible implementation, the high-level semantic branch includes a Stem block, a first-order high-level semantic block, a second-order high-level semantic block, a third-order high-level semantic block, and a fourth-order high-level semantic block connected in sequence, wherein the Stem block is used to preprocess the lung CT image, and the first-order high-level semantic block extracts global semantic information from the lung CT image to obtain a global semantic information feature map. The second-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map The third-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map The fourth-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map .

[0018] Compared with the existing technology, the use of Stem blocks for preprocessing lung CT images can minimize the loss of detail information, enhance the feature expression of the lesion area, and suppress background noise; the use of sequentially connected first-order high-level semantic blocks, second-order high-level semantic blocks, third-order high-level semantic blocks, and fourth-order high-level semantic blocks to extract global semantic information in turn can capture low-resolution global semantic information and enhance the ability to recognize the entire lesion; high-level semantic branches and low-level detail branches are designed to take into account both global and local features, improve the dual-branch CNN multi-scale fusion network model's ability to recognize the entire lesion, achieve accurate segmentation of small lesion areas and complex lesion morphology, and improve the segmentation effect.

[0019] In a possible implementation, the input end of the boundary information block is connected to the output ends of the first-order low-level spatial block, the second-order low-level spatial block, the third-order low-level spatial block, the first-order high-level semantic block, the second-order high-level semantic block, and the third-order high-level semantic block, respectively; the output end of the boundary information block is added to the output end of the third-order spatial block and then connected to the input end of the bidirectional weighted fusion block; the fourth-order high-level semantic block is connected to the input end of the bidirectional weighted fusion block; and the output end of the bidirectional weighted fusion block is connected to the segmentation head.

[0020] The bidirectional weighted fusion block includes a MW block for performing a global maximum pooling operation, a sigmoid activation operation, and a normalization operation, and an AW block for performing a global average pooling operation, a sigmoid activation operation, and a normalization operation.

[0021] In a possible implementation, the preprocessing of the lung CT image in step 4 by using the Stem block specifically includes:

[0022] Step 401: The lung CT image is subjected to convolution operation to generate a feature map ;

[0023] Step 402, feature map Generate feature maps through SPD-Conv convolution operation , feature map After the secondary convolution operation and pooling operation, the feature map is generated , the expression is:

[0024] ;

[0025] ;

[0026] In the formula, Represents the convolution operation; Indicates passing Layers and non-strided convolutional layers Convolution operation; Represents a pooling operation;

[0027] Step 403: The feature map and feature map Splicing, the weight vector is obtained through adaptive global average pooling, sigmoid activation function, normalization and chunk function and ;

[0028] Step 404, in the feature map Add the weight vector , in the feature map Add the weight vector , after fusion, the output is the feature map , the expression is: .

[0029] Compared with the existing technology, the traditional convolution and pooling operation will cause the feature map to lose some important information during the downsampling operation; the SPD-Conv convolution operation can retain the fine-grained information of the features, and the SPD-Conv convolution operation and a set of convolution and pooling fusion methods are used to reduce the dimension of the output, which can minimize the loss of detail information.

[0030] In a possible implementation manner, the step 5 uses the boundary information block to map the detail feature map. and global semantic information feature map Perform weighted fusion to obtain boundary features , specifically including:

[0031] Step 501: Detail feature map and global semantic information feature map Through the convolution operation, we get the same dimension ;

[0032] Step 502: Concatenate along the channel dimension to get the feature map ,Will Concatenate along the channel dimension to get the feature map ;

[0033] Step 503: transform the feature map and feature map The global average pooling and global maximum pooling operations are performed respectively, and the attention weights of features of different scales are obtained after the sigmoid activation function, normalization and grouping operations. , the expression is:

[0034] ;

[0035] ;

[0036] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function; Indicates the use of chunk function for grouping operations;

[0037] Step 504: Detail feature map and global semantic information feature map With attention weight Multiply to get the detail branch boundary features and global branch boundary features , the expression is:

[0038] ;

[0039] ;

[0040] Step 505: Introducing adaptive fusion factor , the detail branch boundary features and global branch boundary features Fusion to obtain boundary features , the expression is:

[0041] ;

[0042] In the formula, .

[0043] Compared with the existing technology, the boundary information block is used to refine the detail feature map and global semantic information feature map Weighted fusion can enhance the extraction of boundary features and suppress noise information.

[0044] In a possible implementation, the step 7 uses a bidirectional weighted fusion block to combine the boundary detail features and global semantic information feature map Perform bidirectional weighted fusion to obtain feature maps , specifically including:

[0045] Step 701, using MW block to identify boundary detail features Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to refine boundary detail features. Perform global average pooling, sigmoid activation, and normalization operations, and add the outputs of the MW block and the AW block to get the weights , the expression is:

[0046] ;

[0047] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function;

[0048] Step 702: Use MW blocks to map the global semantic information feature map Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to map the global semantic information feature map. Perform global average pooling, sigmoid activation, and normalization operations, and add the outputs of the MW block and the AW block to get the weights , the expression is:

[0049] ;

[0050] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function;

[0051] Step 703: weight and weight Perform similarity evaluation and obtain relevance weights , the calculation formula is:

[0052] ;

[0053] In the formula, represents the cosine similarity function;

[0054] Step 704: Boundary detail features and global semantic information feature map Perform convolution and upsampling operations to obtain a feature map with uniform dimensions ;

[0055] Step 705, by correlation weight Weighted bidirectional weighted averaging is performed to obtain the feature map , the expression is:

[0056] .

[0057] Compared with the existing technology, the adaptive bidirectional weighted fusion block can perform bidirectional weighting in the feature fusion process of the two branches. By adaptively aggregating the features of the two branches, the advantages of each branch can be maximized and information conflicts can be avoided, thereby improving the integrity of feature expression and the consistency of segmentation results. The adaptive fusion mechanism adopted effectively solves the problems of feature conflict and information loss in the traditional dual-branch architecture, making the segmentation results more accurate and stable.

[0058] In a possible implementation, the dual-branch CNN multi-scale fusion network model further includes a separable convolution block for assisting in accelerating the training of the dual-branch CNN multi-scale fusion network model, wherein the depth-separable convolution block performs a detailed feature map output by the low-level spatial detail branch. After resizing, the detail feature map after resizing The input segmentation head is used as an auxiliary segmentation head for back propagation of dual-branch CNN multi-scale fusion network model training.

[0059] In a possible implementation, the method further includes step 9, training the dual-branch CNN multi-scale fusion network model in combination with a depth-separable convolution block.

[0060] In a possible implementation, the training of the dual-branch CNN multi-scale fusion network model specifically includes:

[0061] Step 901, obtaining a lung CT image dataset, and dividing the dataset into a test set and a training set, wherein the lung CT images in the test set are marked with category labels;

[0062] Step 902: Use a depth-separable convolutional block to perform a detailed feature map After resizing, input it into the split head as an auxiliary split head;

[0063] Step 903, design comprehensive loss function , including classification loss , Mask loss and Dice loss , the expression is:

[0064] ;

[0065] In the formula, , is the default value;

[0066] Classification Loss The expression is:

[0067] ;

[0068] In the formula, represents the pixel value, Represents the label value, represents the predicted value;

[0069] ;

[0070] In the formula, P represents the predicted value, represents the true value, , represents a hyperparameter;

[0071] ;

[0072] Step 904: train the dual-branch CNN multi-scale fusion network model based on the test set, and then test the trained dual-branch CNN multi-scale fusion network model based on the test set. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a network architecture diagram of the dual-branch CNN multi-scale fusion network model of the present invention;

[0074] Figure 2 This is a flow chart of the CT image segmentation method based on a dual-branch CNN multi-scale fusion network of the present invention;

[0075] Figure 3 It is a structural diagram of the low-level space block of the present invention;

[0076] Figure 4 This is a flow chart of the present invention using the Stem block to preprocess lung CT images;

[0077] Figure 5 is a structural diagram of a boundary information block of the present invention;

[0078] Figure 6 It is a structural diagram of the adaptive bidirectional weighted fusion block of the present invention;

[0079] Figure 7 This is an example of the image segmentation model comparison between the prior art and the present invention. Figure 1 Segmentation effect comparison chart;

[0080] Figure 8 This is an example of the image segmentation model of the prior art and the present invention. Figure 2 Segmentation effect comparison chart;

[0081] Fig. 9 This specific embodiment is based on the image segmentation model of the present invention. Figure 3 Visual segmentation effect diagram of ablation experiment;

[0082] Fig.10 This specific embodiment is based on the image segmentation model of the present invention. Figure 4 Visual segmentation effect diagram of ablation experiment. DETAILED DESCRIPTION

[0083] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0084] In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific circumstances.

[0085] In the embodiments of the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0086] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] See also Figure 1~2 As shown, the embodiment of the present application discloses a CT image segmentation method based on a dual-branch multi-scale fusion network, comprising:

[0088] Step 1: Build a dual-branch CNN multi-scale fusion network model, such as Figure 1 As shown, the dual-branch CNN multi-scale fusion network model includes parallel low-level spatial detail branches and high-level semantic branches, and also includes a boundary information block BIB, a bidirectional weighted fusion block BFB and a segmentation head;

[0089] The low-level spatial detail branch is used to capture small lesions and complex boundary features. The low-level spatial detail branch includes a first-order low-level spatial block, a second-order low-level spatial block, and a third-order low-level spatial block connected in sequence, such as Figure 3 As shown, each low-level spatial block includes an input layer, three groups of feature extraction layers and an output layer, and each group of feature extraction layers consists of a depth-separable convolution layer, a batch normalization layer and a RULE activation function; each low-level spatial block of this specific embodiment is represented by the character LSB; wherein DW stands for Depthwise, indicating depth, and DWConv stands for depth-separable convolution; BN stands for Batch Normalization, indicating batch normalization;

[0090] The high-level semantic branch includes a Stem block, a first-order high-level semantic block, a second-order high-level semantic block, a third-order high-level semantic block and a fourth-order high-level semantic block connected in sequence; each high-level semantic block of this specific embodiment is represented by the character HSB;

[0091] The input end of the boundary information block BIB is connected to the output ends of the first-order low-level spatial block, the second-order low-level spatial block, the third-order low-level spatial block, the first-order high-level semantic block, the second-order high-level semantic block, and the third-order high-level semantic block respectively; the output end of the boundary information block is added to the output end of the third-order spatial block and then connected to the input end of the bidirectional weighted fusion block; the fourth-order high-level semantic block is connected to the input end of the bidirectional weighted fusion block; and the output end of the bidirectional weighted fusion block is connected to the segmentation head.

[0092] The bidirectional weighted fusion block includes a MW block for performing a global maximum pooling operation, a sigmoid activation operation, and a normalization operation, and an AW block for performing a global average pooling operation, a sigmoid activation operation, and a normalization operation.

[0093] In order to speed up the training of the later dual-branch CNN multi-scale fusion network model, the dual-branch CNN multi-scale fusion network model described in this specific embodiment also includes a depth-separable convolution block for assisting in speeding up the training of the dual-branch CNN multi-scale fusion network model, and the depth-separable convolution block outputs the detail feature map of the low-level spatial detail branch. After resizing, the detail feature map after resizing The input segmentation head is used as an auxiliary segmentation head for back propagation of dual-branch CNN multi-scale fusion network model training.

[0094] Step 2: Obtain a lung CT image to be segmented. In this specific embodiment, the size of the lung CT image is ;

[0095] Step 3: Use the low-level spatial detail branch to extract detail features from the lung CT image and extract detail feature maps of different sizes. The first-order low-level spatial block extracts detail features of the lung CT image to obtain a detail feature map , the detailed characteristic diagram of this specific embodiment The size is The second-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map , the detailed characteristic diagram of this specific embodiment The size is , the third-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map , the detailed characteristic diagram of this specific embodiment The size is ; Capturing and retaining high-resolution detail features through low-level spatial detail branches is suitable for capturing small lesions and complex boundary features, thus ensuring that the details of the lesions can be accurately segmented;

[0096] Step 4: Preprocess the lung CT images, use high-level semantic branches to extract global semantic information from the preprocessed lung CT images, and obtain global semantic information feature maps of different sizes. ;

[0097] In order to minimize the loss of detail information, enhance the feature expression of the lesion area, and suppress background noise, the Stem block is used to preprocess the lung CT image, such as Figure 4 As shown, specifically including:

[0098] Step 401: The lung CT image is subjected to convolution operation to generate a feature map ;

[0099] Step 402, feature map Generate feature maps through SPD-Conv convolution operation , feature map After the secondary convolution operation and pooling operation, the feature map is generated , the expression is:

[0100] ;

[0101] ;

[0102] In the formula, Represents the convolution operation; Indicates passing Layers and non-strided convolutional layers Convolution operation; represents the pooling operation; where The full English name is Space-to-Depth, which means space to depth. The full English name is Space-to-Depth Convolution, which means space to depth convolution;

[0103] Step 403: The feature map and feature map Splicing, the weight vector is obtained through adaptive global average pooling, sigmoid activation function, normalization and chunk function and ;

[0104] Step 404, in the feature map Add the weight vector , in the feature map Add the weight vector , after fusion, the output is the feature map , the expression is: ;

[0105] Then, the first-order high-level semantic block extracts global semantic information from the preprocessed lung CT image to obtain a Global semantic information feature map The second-order high-level semantic block is based on the global semantic information feature map The global semantic information is extracted to obtain a The global semantic information feature map The third-order high-level semantic block is based on the global semantic information feature map Extract global semantic information Global semantic information feature map The fourth-order high-level semantic block is based on the global semantic information feature map The global semantic information is extracted to obtain a Global semantic information feature map ; This specific embodiment uses high-level semantic branches to capture low-resolution global semantic information, improving the recognition ability of the dual-branch CNN multi-scale fusion network model for the overall lesion;

[0106] Combining the low-level spatial detail branch and the high-level semantic branch can take into account both global and local features, achieve accurate segmentation of small lesion areas and complex lesions, and greatly improve the segmentation effect.

[0107] Step 5: Use the boundary information block to transform the detail feature map and global semantic information feature map Perform weighted fusion to obtain boundary features ,like Figure 5 As shown; specifically including:

[0108] Step 501: Detail feature map and global semantic information feature map Through the convolution operation, we get the same dimension ;

[0109] Step 502: Concatenate along the channel dimension to get the feature map ,Will Concatenate along the channel dimension to get the feature map ;

[0110] Step 503: transform the feature map and feature map The global average pooling and global maximum pooling operations are performed respectively, and the attention weights of features of different scales are obtained after the sigmoid activation function, normalization and grouping operations. , the expression is:

[0111] ;

[0112] ;

[0113] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function; Indicates the use of chunk function for grouping operations;

[0114] Step 504: Detail feature map and global semantic information feature map With attention weight Multiply to get the detail branch boundary features and global branch boundary features , the expression is:

[0115] ;

[0116] ;

[0117] Step 505: Introducing adaptive fusion factor , the detail branch boundary features and global branch boundary features Fusion to obtain boundary features , the expression is:

[0118] ;

[0119] In the formula, .

[0120] The features of the low-level spatial detail branch and the high-level semantic branch are weightedly fused through the boundary information block, and high-quality boundary features are generated and added to the output features of the low-level spatial detail branch, which effectively improves the recognition ability of the dual-branch CNN multi-scale fusion network model for lesion boundaries. It can also greatly reduce the loss of boundary information in subsequent model training, making the boundaries of the lesion area clearer, especially when the boundaries are blurred.

[0121] Step 6: Boundary features With detail feature map Fusion to obtain boundary detail features .

[0122] Step 7: Use a bidirectional weighted fusion block to combine boundary detail features and global semantic information feature map Perform bidirectional weighted fusion to obtain feature maps ;like Figure 6 As shown, specifically including:

[0123] Step 701, using MW block to identify boundary detail features Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to refine boundary detail features. Perform global average pooling, sigmoid activation, and normalization operations, and add the outputs of the MW block and the AW block to get the weights , the expression is:

[0124] ;

[0125] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function;

[0126] Step 702: Use MW blocks to map the global semantic information feature map Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to map the global semantic information feature map. Perform global average pooling, sigmoid activation, and normalization. MW stands for Maxpooling Weights, which indicates the maximum pooling weights, and AW stands for Avgpooling Weights, which indicates the average pooling weights. The weights are obtained by adding the outputs of the MW block and the AW block. , the expression is:

[0127] ;

[0128] In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function;

[0129] Step 703: weight and weight Perform similarity evaluation and obtain relevance weights , the calculation formula is:

[0130] ;

[0131] In the formula, represents the cosine similarity function;

[0132] Step 704: Boundary detail features and global semantic information feature map Perform convolution and upsampling operations to obtain a feature map with uniform dimensions ;

[0133] Step 705, by correlation weight Weighted bidirectional weighted averaging is performed to obtain the feature map , the expression is:

[0134] .

[0135] By adopting an adaptive bidirectional weighted fusion block to perform bidirectional weighting in the feature fusion process of the low-level spatial detail branch and the high-level semantic branch, the features of the two branches are adaptively aggregated to maximize the retention of their respective advantages and avoid information conflicts, thereby improving the integrity of feature expression and the consistency of segmentation results. It effectively solves the problems of feature conflict and information loss in the traditional dual-branch architecture, making the segmentation results more accurate and stable.

[0136] Step 8: Use the segmentation head based on the feature map Image segmentation is performed to obtain a target segmentation map. The size of the target segmentation map is consistent with the size of the input lung CT image, and the value of each pixel represents its classification probability.

[0137] Step 9, train the dual-branch CNN multi-scale fusion network model. The test environment of this specific embodiment is based on the Windows 10 64-bit operating system, the processor is Intel(R) Core(TM) i9-10885H, and the deep learning open source framework PyTorch1.7.1 is used and the training is accelerated by the NVIDIA RTX 3090 GPU; the optimizer uses AdamW, the initial learning rate LR is set to 1e-4, the momentum range is (0.9, 0.999), eps is set to 1e-8, and the weight decay is set to 0.05; the poly learning rate strategy is used, and it decays by a power of 0.9 after each iteration; the training process specifically includes:

[0138] Step 901, training is performed on three datasets: CT Segmentation Dataset, CT Lung and Infection Segmentation Dataset, and MosMedData. The CT Segmentation Dataset includes a data subset DatasetA and a data subset DatasetB. In order to ensure fairness in training and testing, the data subsets DatasetA, DatasetB, CT Lung and Infection Segmentation Dataset, and MosMedData are divided into training sets and test sets. The data subset DatasetA contains 100 labeled lung CT images, 80 of which are divided into training sets and 20 into test sets. The data subset DatasetB contains 373 lung CT images, 316 of which are divided into training sets and 57 of which are divided into test sets. The Dataset dataset contains 20 cases and a total of 1844 lung CT images, of which 12 cases with a total of 1513 lung CT images are divided into training sets, and 7 cases with a total of 331 lung CT images are divided into test sets; MosMedData contains 50 cases and a total of 785 lung CT images, of which 30 cases with a total of 626 lung CT images are divided into training sets, and 20 cases with a total of 159 lung CT images are divided into test sets; 50K iterations are performed on the above four datasets, and the lung CT images are uniformly adjusted to a resolution of 512×512;

[0139] Step 902: Use a depth-separable convolutional block to perform a detailed feature map After resizing, the segmentation head is input as an auxiliary segmentation head. The adaptive bidirectional weighted fusion block inputs the features of the segmentation head as the main segmentation head. The auxiliary segmentation head is used for back propagation in the training stage to enhance the training effect and speed up the training speed.

[0140] Step 903, in order to ensure accurate segmentation of the lesion area of ​​the lung CT image and maintain spatial consistency and feature appearance, this specific embodiment is a comprehensive loss function designed for training Including classification loss , Mask loss and Dice loss , the expression is:

[0141] ;

[0142] In the formula, , is the default value;

[0143] Classification Loss Binary cross entropy loss is used to quantify the difference between the predicted category label and the true label. The classification loss function The expression is:

[0144] ;

[0145] In the formula, represents the pixel value, Represents the label value, represents the predicted value;

[0146] Mask loss Tversky Loss is used to emphasize small areas of pneumonia, mask loss The expression is:

[0147] ;

[0148] In the formula, P represents the predicted value, Represents the label value, , represents a hyperparameter;

[0149] Dice loss Used to improve mid-overlap accuracy, Dice loss The expression is:

[0150] ;

[0151] Step 904: train the dual-branch CNN multi-scale fusion network model based on the test set, and then test the trained dual-branch CNN multi-scale fusion network model based on the test set.

[0152] The comprehensive loss function (including classification loss, mask loss and Dice loss) adopted in this specific embodiment enables the dual-branch CNN multi-scale fusion network model to better balance the accuracy and global consistency of the lesion area. Among them, the classification loss helps the model to distinguish the foreground and background more accurately, the mask loss improves the sensitivity to small areas, and the Dice loss further optimizes the overlap accuracy, thereby achieving a more comprehensive segmentation performance.

[0153] Segmentation effect comparison experiment

[0154] The lung CT images in the same data set were segmented using different algorithms together with the present invention. The results are as follows: Figure 7 and Figure 8 As shown, Figure 7 Part (a) of Figure 1 The original image, Figure 7Part (b) of is the true label map. Figure 7 Part (c) shows the segmentation effect of the DeepLabv3 model proposed by Liang-Chieh Chen et al. Figure 7 Part (d) shows the segmentation effect of the SegFormer model proposed by Enze Xie et al. Figure 7 Part (e) shows the segmentation effect of the AFFormer model proposed by Dong B et al. Figure 7 Part (f) is a segmentation effect diagram using the image segmentation model of the present application; Figure 8 Part (a) of the example is Figure 2 The original image, Figure 8 Part (b) of is the label map. Figure 8 Part (c) shows the segmentation effect of the DeepLabv3 model proposed by Liang-Chieh Chen et al. Figure 8 Part (d) shows the segmentation effect of the SegFormer model proposed by Enze Xie et al. Figure 8 Part (e) shows the segmentation effect of the AFFormer model proposed by Dong B et al. Figure 8 Part (f) is a diagram showing the segmentation effect of the image segmentation model of the present invention. It can be seen that compared with the actual label value, other algorithms have a higher proportion of missed detections, false detections, etc., while the segmentation effect of the algorithm of the present invention is very close to the label value, especially in some small, blurred areas or areas with large boundary noise, the algorithm of the present invention can still detect them.

[0155] Ablation experiment

[0156] In order to better evaluate the effectiveness of each module in the present invention, two images in CT SEGMENTATION Dataset_A are selected for visualization. Fig. 9 and Fig.10 As shown, Fig. 9 Part (a) of the present invention is an example of an image segmentation model Figure 3 Visual segmentation effect diagram; Fig. 9 Part (b) is an example of a BIB pair where the image segmentation model of the present invention lacks boundary information blocks Figure 3 Visual segmentation effect diagram; Fig. 9 Part (c) is an example of the image segmentation model of the present invention lacking a bidirectional weighted fusion block BFB pair Figure 3 Visual segmentation effect diagram; Fig. 9 Part (d) is an example of the image segmentation model of the present invention lacking high-level spatial detail branches Figure 3 Visual segmentation effect diagram; Fig. 9Part (e) is an example of the image segmentation model of the present invention lacking low-level spatial detail branches Figure 3 Visual segmentation effect diagram; Fig.10 Part (a) of the present invention is an example of an image segmentation model Figure 4 Visual segmentation effect diagram; Fig.10 Part (b) is an example of a BIB pair where the image segmentation model of the present invention lacks boundary information blocks Figure 4 Visual segmentation effect diagram; Fig.10 Part (c) is an example of the image segmentation model of the present invention lacking a bidirectional weighted fusion block BFB pair Figure 4 Visual segmentation effect diagram; Fig.10 Part (d) is an example of the image segmentation model of the present invention lacking high-level spatial detail branches Figure 4 Visual segmentation effect diagram; Fig.10 Part (e) is an example of the image segmentation model of the present invention lacking low-level spatial detail branches Figure 4 Visual segmentation effect diagram; from Fig. 9 and Fig.10 From part (b) of the graph, we can see that without the introduction of BIB, the boundary positions of some lesion areas are ignored, while after the introduction of BIB, the boundary positions of the predicted image and the real image fit well. Fig. 9 and Fig.10 As can be seen from part (c) of the figure, the segmentation prediction map without BFB has missed detection compared with that with BFB, which shows that BFB can enhance the feature perception ability of the network by fusing details and semantic features. Fig. 9 and Fig.10 From parts (d) and (e) of the figure, we can see that the segmentation prediction map of the single-branch network has a large area of ​​false detection and some areas are missed, indicating that the upper branch network with only spatial detail features lacks contextual semantic information, while the lower branch with only semantic features lacks detail information, thus affecting the segmentation effect; our dual-branch network structure can combine the two well, extracting contextual information of a large receptive field while taking into account high-resolution spatial information.

[0157] In general, the present invention uses a low-level detail branch to extract spatial detail features from lung CT images to retain high-resolution detail features and capture small lesions and complex boundaries. It uses a high-level semantic branch to capture low-resolution global semantic information and enhance the recognition ability of the overall lesion. It also introduces boundary information to quickly identify the detail feature map. and global semantic information feature map Perform weighted fusion to generate high-quality boundary features and integrate boundary features into detail feature maps In the experiment, the boundary of the lesion area is made clearer; then, the adaptive bidirectional weighted fusion block is combined to adaptively aggregate the features of low-level detail branches and high-level semantic peaks, maximizing the retention of their respective advantages, effectively improving the overall lesion recognition ability of the dual-branch CNN multi-scale fusion network model, and realizing the accurate segmentation of small lesion areas and complex lesion areas in lung CT images, thus improving the segmentation effect.

[0158] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0159] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0160] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A CT image segmentation method based on a dual-branch CNN multi-scale fusion network, characterized in that: include: Step 1, building a dual-branch CNN multi-scale fusion network model and training the dual-branch CNN multi-scale fusion network model; the dual-branch CNN multi-scale fusion network model includes a low-level spatial detail branch, a high-level semantic branch, a boundary information block, a bidirectional weighted fusion block and a segmentation head; Step 2, obtaining a lung CT image to be segmented; Step 3: Use the low-level spatial detail branch to extract detail features from the lung CT image to obtain detail feature maps of different sizes. ; Step 4: Preprocess the lung CT images, use high-level semantic branches to extract global semantic information from the preprocessed lung CT images, and obtain global semantic information feature maps of different sizes. ; Step 5: Use the boundary information block to transform the detail feature map and global semantic information feature map Perform weighted fusion to obtain boundary features ; Step 6: Boundary features With detail feature map Fusion to obtain boundary detail features ; Step 7: Use a bidirectional weighted fusion block to combine boundary detail features and global semantic information feature map Perform bidirectional weighted fusion to obtain feature maps ; Step 8: Use the segmentation head based on the feature map Perform image segmentation and obtain the target segmentation map.

2. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 1 is characterized in that: The low-level spatial detail branch includes a first-order low-level spatial block, a second-order low-level spatial block and a third-order low-level spatial block connected in sequence, wherein the first-order low-level spatial block extracts detail features of the lung CT image to obtain a detail feature map The second-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map , the third-order low-level spatial block is based on the detail feature map Detail feature extraction is performed to obtain a detail feature map .

3. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 2 is characterized in that: The high-level semantic branch includes a Stem block, a first-order high-level semantic block, a second-order high-level semantic block, a third-order high-level semantic block and a fourth-order high-level semantic block connected in sequence. The Stem block is used to preprocess the lung CT image. The first-order high-level semantic block extracts global semantic information from the lung CT image to obtain a global semantic information feature map. The second-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map The third-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map The fourth-order high-level semantic block is based on the global semantic information feature map Extract global semantic information to obtain a global semantic information feature map .

4. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 3 is characterized in that: The input end of the boundary information block is connected to the outputs of the first-order low-level spatial block, the second-order low-level spatial block, the third-order low-level spatial block, the first-order high-level semantic block, the second-order high-level semantic block, and the third-order high-level semantic block respectively; the output end of the boundary information block is connected to the input end of the bidirectional weighted fusion block after being added to the output end of the third-order spatial block; the fourth-order high-level semantic block is connected to the input end of the bidirectional weighted fusion block; and the output end of the bidirectional weighted fusion block is connected to the segmentation head; The bidirectional weighted fusion block includes a MW block for performing a global maximum pooling operation, a sigmoid activation operation, and a normalization operation, and an AW block for performing a global average pooling operation, a sigmoid activation operation, and a normalization operation.

5. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 4 is characterized in that: The step of using the Stem block to preprocess the lung CT image in step 4 specifically includes: Step 401: The lung CT image is subjected to convolution operation to generate a feature map ; Step 402, feature map Generate feature maps through SPD-Conv convolution operation , feature map After the secondary convolution operation and pooling operation, the feature map is generated , the expression is: ； ； In the formula, Represents the convolution operation; Indicates passing Layers and non-strided convolutional layers Convolution operation; Represents a pooling operation; Step 403: The feature map and feature map Splicing, the weight vector is obtained through adaptive global average pooling, sigmoid activation function, normalization and chunk function and ; Step 404, in the feature map Add the weight vector , in the feature map Add the weight vector , after fusion, the output is the feature map , the expression is: 。 6. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 4 is characterized in that: The step 5 uses the boundary information block to and global semantic information feature map Perform weighted fusion to obtain boundary features , including: Step 501: Detail feature map and global semantic information feature map Through the convolution operation, we get the same dimension ; Step 502: Concatenate along the channel dimension to get the feature map ,Will Concatenate along the channel dimension to get the feature map ; Step 503: transform the feature map and feature map The global average pooling and global maximum pooling operations are performed respectively, and the attention weights of features of different scales are obtained after the sigmoid activation function, normalization and grouping operations. , the expression is: ； ； In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function; Indicates the use of chunk function for grouping operations; Step 504: Detail feature map and global semantic information feature map With attention weight Multiply to get the detail branch boundary features and global branch boundary features , the expression is: ； ； Step 505: Introducing adaptive fusion factor , the detail branch boundary features and global branch boundary features Inlet fusion, get boundary features , the expression is: ； In the formula, .

7. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 4 is characterized in that: The step 7 uses a bidirectional weighted fusion block to combine the boundary detail features and global semantic information feature map Perform bidirectional weighted fusion to obtain feature maps , including: Step 701, using MW block to identify boundary detail features Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to refine boundary detail features. Perform global average pooling, sigmoid activation, and normalization operations, and add the outputs of the MW block and the AW block to get the weights , the expression is: ； In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function; Step 702: Use MW blocks to map the global semantic information feature map Perform global maximum pooling, sigmoid activation and normalization operations, and use AW blocks to map the global semantic information feature map. Perform global average pooling, sigmoid activation, and normalization operations, and add the outputs of the MW block and the AW block to get the weights , the expression is: ； In the formula, represents global maximum pooling; represents global average pooling; Represents the sigmoid activation function; Step 703: weight and weight Perform similarity evaluation and obtain relevance weights , the calculation formula is: ； In the formula, represents the cosine similarity function; Step 704: Boundary detail features and global semantic information feature map Perform convolution and upsampling operations to obtain a feature map with uniform dimensions ; Step 705, by correlation weight Weighted bidirectional weighted averaging is performed to obtain the feature map , the expression is: 。 8. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 1 is characterized in that: The dual-branch CNN multi-scale fusion network model also includes a depth-separable convolution block for assisting in accelerating the training speed of the dual-branch CNN multi-scale fusion network model, wherein the depth-separable convolution block outputs a detail feature map of the low-level spatial detail branch. After resizing, the detail feature map after resizing The input segmentation head is used as an auxiliary segmentation head for back propagation of dual-branch CNN multi-scale fusion network model training.

9. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 8 is characterized in that: The method also includes step 9, in which the dual-branch CNN multi-scale fusion network model is trained in combination with the depth-separable convolutional block.

10. The CT image segmentation method based on a dual-branch CNN multi-scale fusion network according to claim 9, characterized in that: The training of the dual-branch CNN multi-scale fusion network model in step 9 specifically includes: Step 901, obtaining a lung CT image dataset, and dividing the dataset into a test set and a training set, wherein the lung CT images in the test set are marked with category labels; Step 902: Use a depth-separable convolutional block to perform a detailed feature map After resizing, input it into the split head as an auxiliary split head; Step 903, design comprehensive loss function , including classification loss , Mask loss and Dice loss , the expression is: ； In the formula, , is the default value; Classification Loss The expression is: ； In the formula, represents the pixel value, Represents the label value, represents the predicted value; ； In the formula, P represents the predicted value, Represents the label value, , represents a hyperparameter; ； Step 904: train the dual-branch CNN multi-scale fusion network model based on the test set, and then test the trained dual-branch CNN multi-scale fusion network model based on the test set.

Images