Medical image segmentation method based on detail and semantic deep supervision, electronic device and storage medium

Abstract

The application discloses a medical image segmentation method based on details and semantic deep supervision, an electronic device and a storage medium, solves the problems in the prior art, and guarantees synchronous fusion of low-level features and high-level features to generate more reliable segmentation results. The application comprises the following specific steps: step 1, for the medical image input into the deep supervision neural network, the low-level features and the high-level features of the medical image are extracted and input into a detail enhancement module and a semantic enhancement module for decoding to generate a supervision signal; step 2, in the training stage, optimization of each supervision signal is performed; and step 3, in the inference process, all the supervision signals are accumulated to obtain the final segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep supervision neural networks applied to medical image segmentation, and particularly refers to a medical image segmentation method based on detail and semantic deep supervision, an electronic device and a storage medium. BACKGROUND

[0002] Deep supervision neural networks have shown significant effectiveness in the field of medical images. Existing technologies are mainly divided into two categories: detail importance-based methods (such as UNet++, UNet3+, EGEUNet, etc.) and semantic importance-based methods (such as PVTCASCADE, CaraNet, UACANet, etc.). Among them, the detail importance-based method enhances the features of each stage output side through the features of the shallow layer information. Typical technologies include multi-subnet aggregation in UNet++ and full-scale skip connection in UNet3+. This kind of method can effectively identify fine-grained pathological features, but often ignores high-level overall semantic attributes. On the contrary, the semantic importance-based method first obtains a global mapping of aggregated semantic information, and then performs progressive decoding. For example, UACANet gradually optimizes the uncertainty area of global features, while CaraNet uses flip attention with context awareness to further enhance the global mapping. This kind of method performs well in saliency lesion segmentation, but often loses segmentation targets when dealing with data that require high identification of fine granularity (high resolution, discrete lesions).

[0003] In addition, existing technologies usually allocate loss weights for each stage according to heuristic experience. Among them, the semantic importance-based method equally supervises all signals by assuming that they have the same importance, while the detail importance-based method allocates more punishment to the output closest to the input layer. However, manually setting the weight of each stage requires a lot of professional knowledge and may not necessarily produce the best supervision effect. At the same time, existing researches are often limited to single supervision of coarse-grained semantic features or fine-grained detail features, ignoring the important correlation between the two types of features in medical image analysis. SUMMARY

[0004] The purpose of the present application is to provide a medical image segmentation method based on detail and semantic deep supervision, an electronic device and a storage medium, which solves the problems existing in the prior art and ensures the synchronous fusion of low-level features and high-level features to produce more reliable segmentation results.

[0005] In order to achieve the above purpose, the solution of the present application is:

[0006] A medical image segmentation method based on detail and semantic deep supervision, using a deep supervised neural network including a detail enhancement module and a semantic enhancement module, includes the following specific steps:

[0007] Step 1: For medical images input into a deep supervised neural network, extract their low-level features. and advanced features The signal is then input into the detail enhancement module and the semantic enhancement module for decoding to generate a supervision signal;

[0008] Step 2: During the training phase, optimize each supervision signal;

[0009] Step 3: During the inference process, all the supervision signals are accumulated to obtain the final segmentation result;

[0010] The detail enhancement module receives low-level features. and advanced features As input, perform the following operations:

[0011] Low-level features After a max-pooling layer and two convolutional layers of the same size, the sigmoid function is applied to obtain the detail mask. Its calculation formula is expressed as:

[0012] ;

[0013] in, and These represent the sigmoid function and the max pooling operation, respectively. Indicates the kernel size as Convolution operations;

[0014] For advanced features Perform a 2x upsampling, then combine it with the low-level features. Perform vector concatenation, followed by several deployments implemented sequentially using ReLU functions. Convolutional layers and a deployment The convolutional layers ultimately yield fused features. Its calculation formula is expressed as:

[0015] ;

[0016] in, and These represent the kernel size as follows: , Convolution operations; Represents the ReLU function; This represents a vector concatenation operation along the channel dimension. denotes a two times up-sampling operation;

[0017] In the detail mask element-wise multiplication is performed between the fused feature and a residual connection containing low-level features to generate a detail-enhanced decoder output , whose calculation formula is expressed as:

[0018] ;

[0019] The semantic enhancement module receives low-level features and high-level features as input, and performs the following operations:

[0020] After a two times up-sampling of the high-level features , a semantic mask is generated through spatial attention , whose calculation formula is expressed as:

[0021] ;

[0022] wherein, denotes spatial attention;

[0023] The semantic mask is multiplied with the low-level features , and channel attention is used to model the correlation between channels, and a residual connection containing high-level features is sampled to generate the final output of the semantic enhancement module , whose calculation formula is expressed as:

[0024] ;

[0025] wherein, denotes channel attention.

[0026] Preferably, 3 detail enhancement modules are configured in the deep supervision neural network, and different filter sizes of convolution operations are equipped in the 3 detail enhancement modules.

[0027] Preferably, in step 2, adaptive supervision based on uncertainty is used to guide the optimization of each supervision signal, specifically:

[0028] All supervision signals are collectively referred to as , to denote the first to the supervision signal; and the supervision signal coefficient of uncertainty of The calculation formula is:

[0029] ;

[0030] Among them, Indicates the average function;

[0031] Normalized by the Sofamax function And Max Scaling is used to further emphasize the uncertainty difference between the supervision weights, and the deep supervision weight , The calculation formula is:

[0032] ;

[0033] Among them, Indicates the base of natural logarithm;

[0034] The calculation formula is:

[0035] ;

[0036] Among them, Indicates the maximum value operation;

[0037] The adaptive supervision loss based on uncertainty is calculated The calculation formula is as follows:

[0038] ;

[0039] Among them, Indicates the true value label, Indicates the total number of supervision signals; 、 Indicate the weighted IoU loss and the weighted binary cross entropy loss, respectively.

[0040] An electronic device comprising a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the medical image segmentation method by the processor.

[0041] A computer readable storage medium, the storage medium stores a computer program; the computer program executes in the computer, makes the computer execute the medical image segmentation method.

[0042] After adopting the above technical scheme, the present application has the following technical effects:

[0043] Through the above scheme, the application proposes an innovative learning paradigm, through the constructed detail enhancement module and semantic enhancement module, both of which run simultaneously in the decoding stage, can capture coarse-grained semantic attributes and fine-grained detail features at the same time, thereby ensuring the synchronous fusion of low-level features and high-level feature supervision, thereby being able to produce more reliable segmentation results.

[0044] In addition, it is also very important to consider the allocation of the size of the supervision loss at each stage in the deep supervision network, and the application proposes an innovative uncertainty-based adaptive deep supervision, which can allocate the loss weight of each stage according to the uncertainty estimation of the output feature. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Flowchart of the specific embodiment of the application;

[0046] Figure 2 Computational flowchart of the detail enhancement module and the semantic enhancement module of the specific embodiment of the application;

[0047] Figure 3 Qualitative analysis graph of the specific embodiment of the application and other prior art on the intestinal polyp data set;

[0048] Figure 4 Qualitative analysis graph of the specific embodiment of the application and other prior art on the ultrasound breast tumor and cell nucleus segmentation data set. DETAILED DESCRIPTION

[0049] In order to further explain the technical scheme of the application, the application will be described in detail below through specific embodiments.

[0050] REFERENCE Figures 1-2 As shown in the drawings, the application discloses a medical image segmentation method based on detail and semantic deep supervision, the deep supervision neural network used includes a detail enhancement module and a semantic enhancement module, and includes the following specific steps:

[0051] Step 1, for the medical image input into the deep supervision neural network, the low-level features and high-level features thereof are extracted by using a pyramid vision transformer And input into the detail enhancement module and the semantic enhancement module for decoding to generate a supervision signal;

[0052] Step 2, in the training stage, optimization of each supervision signal is performed;

[0053] Step 3, in the inference process, all supervision signals are accumulated to obtain the final segmentation result.

[0054] The detail enhancement module and the semantic enhancement module are specifically described as follows: ​

[0055] (a) Detail Enhancement Module

[0056] Low-level features are located in the shallow layers of the neural network, mainly focusing on fine-grained information such as color, texture, edge, and contrast, which provide valuable guidance for analyzing lesion boundaries and capturing subtle pathological changes. However, existing methods only rely on skip connections to pass low-level features to the decoding process for supervision, failing to fully exploit the great potential of low-level features. To alleviate this problem, the present invention constructs a detail enhancement module, aiming to generate a new detail mask . As shown in Figure 2 (a), the detail enhancement module receives low-level features and high-level features from different depths of the neural network as inputs, and performs the following operations:

[0057] After the low-level features go through a max-pooling layer and two convolutional layers with the same size, a sigmoid function is applied to obtain the detail mask , whose calculation formula is expressed as:

[0058] ;

[0059] where and represent the sigmoid function and the max-pooling operation, respectively, represents the convolution operation with a kernel size of ; where the goal of max-pooling is to retain the most relevant detail features while reducing the impact of noise in shallow features, and the two convolutions are used to expand the receptive field.

[0060] A two-fold upsampling is performed on the high-level features , and then a vector concatenation operation is performed with the low-level features . After that, a series of cascading convolutional layers (the initial layers deploy convolutional layers to reduce the channel count, and are realized successively through the ReLU function, and the final layers deploy convolutional layers) are further used to extract features to complete fine-grained feature fusion, and finally obtain the fused features , whose calculation formula is expressed as:

[0061] ;

[0062] where and represent the convolution operation with a kernel size of , respectively; represents the ReLU function; This represents a vector concatenation operation along the channel dimension. This indicates a double upsampling operation.

[0063] Finally, in detail masking With fusion features Element-wise multiplication between them effectively amplifies the detailed information and samples a sample containing low-level features. The residual connections generate the detail-enhanced decoder output. Its calculation formula is expressed as:

[0064] .

[0065] (b) Semantic Enhancement Module

[0066] Unlike low-level features, high-level features encapsulate a wealth of semantic information, including category, context, and relationships. Therefore, supervising these high-level features to enhance semantics is practical and helps in the precise localization of pathological objects in clinical diagnosis. However, existing methods simply employ a global mapping of high-level features to generate monitoring signals, failing to fully extract the semantic information embedded within these features. To further enhance this, this invention constructs a semantic enhancement module that includes a semantic mask guided by high-level features. To enhance semantic supervision. For example... Figure 2 As shown in (b), the semantic enhancement module receives low-level features from different depths of the neural network. and advanced features As input, perform the following operations:

[0067] For advanced features After performing a 2x upsampling, a semantic mask is generated using spatial attention (SA). Its calculation formula is expressed as:

[0068] ;

[0069] in, This represents spatial attention.

[0070] Subsequently, semantic mask With low-level features Multiplication yields semantically enhanced features. Considering that high-level semantics contain extensive contextual information, which is typically channel-guided, this invention uses channel attention (CA) to model the associations between channels. Finally, a feature containing high-level features is also sampled. The residual connections are used to generate the final output of the semantic enhancement module. Its calculation formula is expressed as:

[0071] ;

[0072] wherein, denotes channel attention.

[0073] Through the above scheme, the application proposes an innovative learning paradigm, through the constructed detail enhancement module and semantic enhancement module, both of which run simultaneously in the decoding stage, which can capture coarse-grained semantic attributes and fine-grained detail features at the same time, thereby ensuring the synchronous fusion of low-level features and high-level feature supervision, thereby being able to produce more reliable segmentation results.

[0074] In addition, it is also very important to consider the allocation size of the supervision loss of each stage in the deep supervision network, and the application proposes an innovative uncertainty-based adaptive deep supervision, which can allocate the loss weight of each stage according to the uncertainty estimation of the output feature.

[0075] The following shows a specific embodiment of the application.

[0076] In order to adapt to the case that shallow features of different scales have different resolutions, the application configures 3 detail enhancement modules in the deep supervision neural network, and different convolution operations with different filter sizes are equipped in the 3 detail enhancement modules. For example, a convolution kernel with a maximum resolution is used for feature extraction.

[0077] In the above step 2, the uncertainty-based adaptive supervision is used to guide the optimization of each supervision signal. In order to solve the non-optimal problem caused by heuristic deep supervision weight allocation, the application introduces an uncertainty-based adaptive supervision method, which is designed to automatically allocate loss weight to different supervision signals. The main motivation is that the uncertainty of pixels can be used as a reliable measure to evaluate the quality of the signal in medical image analysis. Therefore, we suggest using the uncertainty of the supervision signal as a measure to evaluate the corresponding loss weight distribution, which is:

[0078] All supervision signals are collectively referred to as , to denote the first to the supervision signal; and calculate the uncertainty coefficient of the first supervision signal , the calculation formula is:

[0079] ;

[0080] wherein, denotes the average function;

[0081] ​Since is generally a very small value and is not suitable for the deployment of supervision weights, the present application normalizes it by the Sofamax function to obtain , and further emphasizes the uncertainty difference between supervision weights by Max Scaling to obtain deep supervision weights , The calculation formula of is:

[0082] ;

[0083] wherein, denotes the base of the natural logarithm;

[0084] The calculation formula of is:

[0085] ;

[0086] wherein, denotes the maximum value operation;

[0087] In summary, the adaptive supervision loss based on uncertainty is calculated as , and the calculation formula is as follows:

[0088] ;

[0089] wherein, denotes the true value label, denotes the total number of supervision signals; , respectively denote the weighted IoU loss and the weighted binary cross-entropy loss.

[0090] The technical effects of the present application are verified by experimental data as follows.

[0091] Referring to Figure 3 , it is a qualitative analysis diagram of the present application and other prior art on the intestinal polyp data set, which shows three typical polyp lesion cases: large polyps (first row), small polyps (second row) and polyps similar to tissue height (third row). From Figure 3 It can be observed that the deep supervision scheme based on details and semantics shows the most similar results to the labeled area; in particular, in small polyps and polyps similar to tissue, the present application still achieves segmentation in those targets that are difficult to distinguish with the naked eye, while other methods show missed detection, false detection and even segmentation loss, etc.

[0092] Referring to Figure 4Figures for qualitative analysis of the present application and other prior art on the breast tumor and cell nucleus segmentation dataset, which shows the superiority of the present application in breast tumor segmentation and cell nucleus segmentation, for example, in malignant tumors (the first row, with jagged or lobulated boundaries), other methods show obvious false negative examples, while the present application can well solve this problem.

[0093] The above results show that the deep supervision method based on details and semantics can achieve better segmentation performance than the existing single-view deep supervision method in various datasets.

[0094] The application further discloses an electronic device, including a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above method by the processor.

[0095] The application further discloses a computer readable storage medium, the storage medium stores a computer program; the computer program is executed in the computer, and the computer program causes the computer to execute the above method. The storage medium includes various storage program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0096] The above embodiments and drawings are not limited to the product form and style of the application, and any appropriate changes or modifications made by those skilled in the art to the above embodiments and drawings shall be considered as not departing from the patent scope of the application.

Claims

1. A medical image segmentation method based on details and semantic deep supervision, characterized in that The deep supervision neural network used includes a detail enhancement module and a semantic enhancement module, including the following specific steps: Step 1, for the medical image inputted into the deep supervision neural network, extract its low-level features and high-level features , and input into the detail enhancement module and the semantic enhancement module for decoding to generate the supervision signal; Step 2, in the training phase, the optimization of each supervision signal is carried out; Step 3, in the inference process, all supervision signals are accumulated to obtain the final segmentation result; The detail enhancement module receives low-level features and high-level features as input and performs the following operations: low-level features After a max-pooling layer and two convolutional layers with the same size, sigmoid function is applied to obtain the detail mask The calculation formula is expressed as: ； wherein, and sigmoid and max-pooling operation, respectively, denotes a convolution operation with kernel size of on high-level features is performed, and then concatenated with low-level features After that, a number of convolutional layers with different configurations are implemented in succession by ReLU function, and a convolutional layer with a configuration is implemented, and finally the fusion features are obtained, and the calculation formula is expressed as: ； wherein, and represent convolution operations with kernel size , ; represents a ReLU function; represents a vector concatenation operation in the channel dimension; represents a two times up-sampling operation; In detail mask Element-wise multiplication is performed between the fusion features and the residual connection containing low-level features to generate a detail-enhanced decoder output , which is calculated as: ； The semantic enhancement module receives low-level features and high-level features as input and performs the following operations: on high-level features After a two times up-sampling, the semantic mask is generated by spatial attention The calculation formula is expressed as: ； wherein denotes spatial attention; semantic mask With low-level features Multiply, and use channel attention to model the correlation between channels, then sample a sample containing high-level features. The residual connections are used to generate the final output of the semantic enhancement module. Its calculation formula is expressed as: ； wherein, denotes channel attention.

2. The medical image segmentation method based on detail and semantic deep supervision according to claim 1, characterized in that: The deep supervision neural network is configured with three detail enhancement modules, and different filter sizes of convolution operations are equipped in the three detail enhancement modules.

3. The medical image segmentation method based on detail and semantic deep supervision according to claim 1, characterized in that: In step 2, the optimization of each supervision signal is guided by an uncertainty-based adaptive supervision, specifically: All the supervisory signals are collectively referred to as , to represent the first to the supervisory signal; and calculate the uncertainty coefficient of the first supervisory signal , the formula is: ； wherein denotes the average function; The normalization is performed by the Sofamax function to obtain and Max Scaling is used to further emphasize the uncertainty difference between the supervision weights to obtain the deep supervision weight , The calculation formula of the deep supervision weight is: ； wherein denotes the base of the natural logarithm; The calculation formula is: ； wherein denotes the operation of taking the maximum value; An uncertainty-based adaptive supervision loss is computed with the following formula: ； wherein, denotes a true value label, denotes a total number of supervision signals; , denote a weighted IoU loss, a weighted binary cross-entropy loss, respectively.

4. An electronic device, characterized in that... The application program is stored in the memory and is configured to be executed by the processor to perform the medical image segmentation method based on detail and semantic deep supervision according to any one of claims 1 to 3.

5. A computer readable storage medium characterized by The storage medium stores a computer program; when the computer program is executed in the computer, the computer program causes the computer to perform the medical image segmentation method based on detail and semantic deep supervision according to any one of claims 1 to 3.

Images