Medical image segmentation method based on detail and semantic deep supervision, electronic equipment and storage medium

Abstract

The invention discloses a medical image segmentation method based on detail and semantic deep supervision, electronic equipment and a storage medium, which solve the problems in the prior art and ensure the synchronous fusion of low-level feature supervision and high-level feature supervision so as to generate a more reliable segmentation result. The method comprises the following specific steps: step 1, extracting a low-level feature # imgabs0 # and a high-level feature # imgabs1 # of a medical image input into a deep supervision neural network, and inputting the low-level feature # imgabs0 # and the high-level feature # imgabs1 # into a detail enhancement module and a semantic enhancement module for decoding to generate a supervision signal; 2, in the training stage, optimizing each supervision signal; and step 3, in a reasoning process, accumulating all the supervision signals to obtain a final segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of deep supervised neural networks applied to medical image segmentation, and in particular to a medical image segmentation method, electronic equipment and storage medium based on detail and semantic deep supervision. Background Art

[0002] Deep supervised neural networks have shown remarkable effectiveness in the field of medical imaging. Existing technologies can be divided into two categories: methods based on detail importance (such as UNet++, UNet3+, EGEUNet, etc.) and methods based on semantic importance (such as PVTCASCADE, CaraNet, UACANet, etc.). Among them, methods based on detail importance use the features of shallow information to enhance the features of the output side of each stage. Typical technologies include multi-subnet aggregation in UNet++ and full-scale skip connections in UNet3+. Such methods can effectively identify fine-grained pathological features, but often ignore high-level overall semantic attributes. In contrast, methods based on semantic importance first obtain a global mapping of aggregated semantic information and then perform progressive decoding. For example, UACANet gradually optimizes the uncertainty region of global features, while CaraNet uses context-aware flipping attention to further enhance the global mapping. Such methods perform well in the segmentation of significant lesions, but often suffer from the problem of segmentation target loss when processing data with high requirements for fine-grained recognition (high resolution, discrete lesions).

[0003] In addition, existing techniques usually assign loss weights for each stage based on heuristic experience. Among them, methods based on semantic importance supervise all signals equally by assuming that they have the same importance, while methods based on detail importance assign more penalties to the outputs closest to the input layer. However, manually setting the weights for each stage requires a lot of human expertise and may not necessarily produce the best supervision effect. At the same time, existing research is often limited to supervising coarse-grained semantic features or fine-grained detail features alone, while ignoring the important correlation between these two types of features in medical image analysis. Summary of the invention

[0004] The purpose of the present invention is to provide a medical image segmentation method, electronic device and storage medium based on detail and semantic deep supervision to solve the problems existing in the prior art and ensure the synchronous fusion of low-level features and high-level feature supervision so as to produce more reliable segmentation results.

[0005] In order to achieve the above object, the solution of the present invention is: A medical image segmentation method based on detail and semantic deep supervision, the deep supervision neural network used includes a detail enhancement module and a semantic enhancement module, including the following specific steps: Step 1: Extract low-level features of medical images input into the deep supervised neural network and advanced features , and input into the detail enhancement module and semantic enhancement module for decoding to generate supervision signals; Step 2: During the training phase, optimize each supervisory signal; Step 3: During the inference process, all supervisory signals are accumulated to obtain the final segmentation result; The detail enhancement module receives low-level features and advanced features As input, and perform the following operations: The low-level features After a max pooling layer and two convolutional layers of the same size, a sigmoid function is applied to obtain the detail mask. , and its calculation formula is expressed as: ; in, and Represent the sigmoid function and the maximum pooling operation respectively, Indicates that the convolution kernel size is Convolution operation; For advanced features Perform a two-fold upsampling and then combine it with the low-level features Perform vector concatenation operations, followed by several deployments implemented continuously by the ReLU function The convolutional layer and a deployment The convolution layer finally obtains the fusion feature , and its calculation formula is expressed as: ; in, and They represent the convolution kernel sizes respectively. , Convolution operation; Represents the ReLU function; Represents the vector concatenation operation of the channel dimension; Indicates a two-fold upsampling operation; In detail mask With fusion features Perform element-wise multiplication between them and sample a low-level feature The residual connection of , generates the decoder output with enhanced details , and its calculation formula is expressed as: ; The semantic enhancement module receives low-level features and advanced features As input, and perform the following operations: For advanced features After a two-fold upsampling, a semantic mask is generated through spatial attention. , and its calculation formula is expressed as: ; in, Indicates spatial attention; Semantic mask With low-level features Multiply them together and use channel attention to model the association between channels, and then sample a channel containing high-level features The residual connection of , and its calculation formula is expressed as: ; in, represents channel attention.

[0006] Preferably, three detail enhancement modules are configured in the deep supervised neural network, and convolution operations with different filter sizes are provided in the three detail enhancement modules.

[0007] Preferably, in step 2, uncertainty-based adaptive supervision is used to guide the optimization of each supervisory signal, specifically: All supervisory signals are collectively referred to as , to Respectively represent to supervision signal; and calculate the Supervisory signal The uncertainty coefficient , and its calculation formula is: ; in, represents the average function; Normalized by Sofamax function , and Max Scaling is used to further emphasize the uncertainty differences between supervision weights to obtain deep supervision weights , The calculation formula is: ; in, represents the base of natural logarithms; The calculation formula is: ; in, Indicates the operation of taking the maximum value; Calculate the uncertainty-based adaptive supervision loss , and its calculation formula is as follows: ; in, represents the true value label, represents the total number of supervisory signals; , They represent weighted IoU loss and weighted binary cross entropy loss respectively.

[0008] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the medical image segmentation method.

[0009] A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the medical image segmentation method.

[0010] After adopting the above technical solution, the present invention has the following technical effects: Through the above scheme, the present invention proposes an innovative learning paradigm. Through the constructed detail enhancement module and semantic enhancement module, both run simultaneously in the decoding stage, which can simultaneously capture coarse-grained semantic attributes and fine-grained detail features, thereby ensuring the synchronous fusion of low-level features and high-level feature supervision, thereby producing more reliable segmentation results.

[0011] In addition, considering that the size of the supervision loss allocation at each stage in the deep supervision network is also very important, the present invention proposes an innovative uncertainty-based adaptive deep supervision, which can allocate the loss weight of each stage according to the uncertainty estimation of the output features. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of a specific embodiment of the present invention; Figure 2 is a calculation flow chart of a detail enhancement module and a semantic enhancement module in a specific embodiment of the present invention; Figure 3 It is a qualitative analysis diagram of the specific embodiment of the present invention and other prior arts on the intestinal polyp data set; Figure 4 It is a qualitative analysis diagram of the specific embodiment of the present invention and other prior arts on ultrasound breast tumor and cell nucleus segmentation datasets. DETAILED DESCRIPTION

[0013] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0014] refer to Figure 1-2 As shown, the present invention discloses a medical image segmentation method based on detail and semantic deep supervision, the deep supervision neural network used includes a detail enhancement module and a semantic enhancement module, and includes the following specific steps: Step 1: For medical images input into the deep supervised neural network, use the pyramid visual transformer to extract its low-level features and advanced features , and input into the detail enhancement module and semantic enhancement module for decoding to generate supervision signals; Step 2: During the training phase, optimize each supervisory signal; Step 3: During the inference process, all supervisory signals are accumulated to obtain the final segmentation result.

[0015] The following describes the detail enhancement module and the semantic enhancement module in detail: (a) Detail enhancement module Low-level features are located in the shallow layers of the neural network, mainly fine-grained information such as color, texture, edge and contrast, which provide valuable guidance for analyzing lesion boundaries and capturing subtle pathological changes. However, existing methods only rely on jump connections to pass low-level features to the decoding process for supervision, which fails to fully utilize the huge potential of low-level features. In order to alleviate this problem, the present invention constructs a detail enhancement module, which aims to generate a new detail mask .like Figure 2 As shown in (a), the detail enhancement module receives low-level features from different depths of the neural network. and advanced features As input, and perform the following operations: The low-level features After a max pooling layer and two convolutional layers of the same size, a sigmoid function is applied to obtain the detail mask. , and its calculation formula is expressed as: ; in, and Represent the sigmoid function and the maximum pooling operation respectively, Indicates that the convolution kernel size is The goal of the maximum pooling is to retain the most relevant detail features while reducing the impact of noise in shallow features, while the two convolutions are used to expand the receptive field.

[0016] For advanced features Perform a two-fold upsampling and then combine it with the low-level features Perform vector concatenation operations, and then pass through a series of cascaded convolutional layers (initial layer deployment The convolutional layer is used to reduce the channel count and is implemented continuously through the ReLU function. The final layer is deployed The convolutional layer of the CNN further extracts features to complete fine-grained feature fusion, and finally obtains the fused feature , and its calculation formula is expressed as: ; in, and They represent the convolution kernel sizes respectively. , Convolution operation; Represents the ReLU function; Represents the vector concatenation operation of the channel dimension; Indicates a two-fold upsampling operation.

[0017] Finally, in the detail mask With fusion features Element-wise multiplication is performed between them, effectively amplifying the details and sampling a low-level feature The residual connection of , generates the decoder output with enhanced details , and its calculation formula is expressed as: .

[0018] (b) Semantic Enhancement Module Unlike low-level features, high-level features encapsulate a lot of semantic information, including categories, contexts, and relations. Therefore, it is practical to supervise these high-level features to enhance semantics, which helps to accurately locate pathological objects in clinical diagnosis. However, existing methods simply adopt a global mapping of high-level features to generate monitoring signals, and to some extent fail to thoroughly mine the semantic information embedded in these high-level features. To further enhance this, the present invention constructs a semantic enhancement module, which contains a semantic mask guided from high-level features. To strengthen semantic supervision. Figure 2 As shown in (b), the semantic enhancement module receives low-level features from different depths of the neural network. and advanced features As input, and perform the following operations: For advanced features After a two-fold upsampling, a semantic mask is generated through spatial attention (SA) , and its calculation formula is expressed as: ; in, Represents spatial attention.

[0019] Then, the semantic mask With low-level features Considering that high-level semantics contains extensive contextual information, which is usually guided by channels, the present invention uses channel attention (CA) to model the association between channels, and finally samples a high-level feature The residual connection of , and its calculation formula is expressed as: ; in, represents channel attention.

[0020] Through the above scheme, the present invention proposes an innovative learning paradigm. Through the constructed detail enhancement module and semantic enhancement module, both run simultaneously in the decoding stage, which can simultaneously capture coarse-grained semantic attributes and fine-grained detail features, thereby ensuring the synchronous fusion of low-level features and high-level feature supervision, thereby producing more reliable segmentation results.

[0021] In addition, considering that the size of the supervision loss allocation at each stage in the deep supervision network is also very important, the present invention proposes an innovative uncertainty-based adaptive deep supervision, which can allocate the loss weight of each stage according to the uncertainty estimation of the output features.

[0022] Specific embodiments of the present invention are shown below.

[0023] In order to adapt to the situation where shallow features of different scales have different resolutions, the present invention configures three detail enhancement modules in the deep supervised neural network and equips the three detail enhancement modules with convolution operations of different filter sizes. For example, the feature map with the largest resolution uses a The convolution kernel is used for feature extraction.

[0024] In the above step 2, uncertainty-based adaptive supervision is used to guide the optimization of each supervisory signal. In order to solve the non-optimal problem caused by heuristic deep supervision weight allocation, the present invention introduces an uncertainty-based adaptive supervision method, which is designed to automatically assign loss weights to different supervisory signals. The main motivation is that the uncertainty of pixels can be used as a reliable measure for evaluating signal quality in medical image analysis. Therefore, we suggest using the uncertainty of the supervisory signal as a measure to evaluate the distribution of the corresponding loss weights, specifically: All supervisory signals are collectively referred to as , to Respectively represent to supervision signal; and calculate the Supervisory signal The uncertainty coefficient , and its calculation formula is: ; in, represents the average function; because It is generally a very small value and is not suitable for the deployment of supervisory weights. Therefore, the present invention uses the Sofamax function to normalize it to obtain , and Max Scaling is used to further emphasize the uncertainty differences between supervision weights to obtain deep supervision weights , The calculation formula is: ; in, represents the base of natural logarithms; The calculation formula is: ; in, Indicates the operation of taking the maximum value; In summary, the uncertainty-based adaptive supervision loss is calculated , and its calculation formula is as follows: ; in, represents the true value label, represents the total number of supervisory signals; , They represent weighted IoU loss and weighted binary cross entropy loss respectively.

[0025] The technical effect of the present invention is verified by experimental data as follows.

[0026] See also Figure 3 , which is a qualitative analysis diagram of the present invention and other existing technologies on the intestinal polyp dataset, shows three typical polyp lesion cases: large polyps (first row), tiny polyps (second row) and polyps that are highly similar to tissues (third row). Figure 3 It can be observed that the detail- and semantic-based deep supervision scheme shows the most similar results to the annotated areas; especially in small polyps and polyps of similar tissues, the present invention still achieves segmentation in those targets that are difficult to distinguish with the naked eye, while other methods show missed detections, false detections, and even segmentation losses.

[0027] See also Figure 4, which is a qualitative analysis diagram of the present invention and other existing technologies on ultrasound breast tumor and cell nucleus segmentation datasets, which demonstrates the excellence of the present invention in breast tumor segmentation and cell nucleus segmentation. For example, in malignant tumors (the first row, with jagged or lobed boundaries), other methods show obvious false negative instances, while the present invention can solve this problem well.

[0028] The above results show that the deep supervision method based on details and semantics can achieve better segmentation performance in various datasets compared with the existing single-view deep supervision method.

[0029] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0030] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0031] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A medical image segmentation method based on detail and semantic deep supervision, characterized in that The deep supervised neural network used includes a detail enhancement module and a semantic enhancement module, including the following specific steps: Step 1: Extract low-level features of medical images input into the deep supervised neural network and advanced features , and input into the detail enhancement module and semantic enhancement module for decoding to generate supervision signals; Step 2: During the training phase, optimize each supervisory signal; Step 3: During the inference process, all supervisory signals are accumulated to obtain the final segmentation result; The detail enhancement module receives low-level features and advanced features As input, and perform the following operations: The low-level features After a max pooling layer and two convolutional layers of the same size, a sigmoid function is applied to obtain the detail mask. , and its calculation formula is expressed as: ； in, and Represent the sigmoid function and the maximum pooling operation respectively, Indicates that the convolution kernel size is Convolution operation; For advanced features Perform a two-fold upsampling and then combine it with the low-level features Perform vector concatenation operations, and then pass through several deployments implemented continuously by the ReLU function The convolutional layer and a deployment The convolution layer finally obtains the fusion feature , and its calculation formula is expressed as: ； in, and They represent the convolution kernel sizes respectively. , Convolution operation; Represents the ReLU function; Represents the vector concatenation operation of the channel dimension; Indicates a two-fold upsampling operation; In detail mask With fusion features Perform element-wise multiplication between them and sample a low-level feature The residual connection of , generates the decoder output with enhanced details , and its calculation formula is expressed as: ； The semantic enhancement module receives low-level features and advanced features As input, and perform the following operations: For advanced features After a two-fold upsampling, a semantic mask is generated through spatial attention. , and its calculation formula is expressed as: ； in, Indicates spatial attention; Semantic mask With low-level features Multiply them together and use channel attention to model the association between channels, and then sample a channel containing high-level features The residual connection of , and its calculation formula is expressed as: ； in, represents channel attention.

2. A medical image segmentation method based on detail and semantic deep supervision as claimed in claim 1, characterized in that: Three detail enhancement modules are configured in the deep supervised neural network, and convolution operations with different filter sizes are provided in the three detail enhancement modules.

3. The medical image segmentation method based on detail and semantic deep supervision as claimed in claim 1, characterized in that: In step 2, uncertainty-based adaptive supervision is used to guide the optimization of each supervision signal, specifically: All supervisory signals are collectively referred to as , to Respectively represent to supervision signal; and calculate the Supervisory signal The uncertainty coefficient , and its calculation formula is: ； in, represents the average function; Normalized by Sofamax function , and Max Scaling is used to further emphasize the uncertainty differences between supervision weights to obtain deep supervision weights , The calculation formula is: ； in, represents the base of natural logarithms; The calculation formula is: ； in, Indicates the operation of taking the maximum value; Calculate the uncertainty-based adaptive supervision loss , and its calculation formula is as follows: ； in, represents the true value label, represents the total number of supervisory signals; , They represent weighted IoU loss and weighted binary cross entropy loss respectively.

4. An electronic device, characterized in that It comprises a processor, a memory and an application; the application is stored in the memory and is configured so that the processor executes a medical image segmentation method based on detail and semantic deep supervision as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer is caused to execute a medical image segmentation method based on detail and semantic deep supervision as described in any one of claims 1 to 3.

Images