An image segmentation method and system based on multi-level and multi-scale gradual fusion network

Abstract

The invention relates to the field of image processing, and proposes an image segmentation method and system based on a multi-level multi-scale stepwise fusion network. A multi-level multi-scale stepwise fusion network is designed to improve the accuracy of overall image segmentation. The multi-level multi-scale stepwise fusion network first extracts semantic features of multiple different levels of an image, performs multi-level feature fusion, so as to improve the overall understanding of the features and reduce the semantic gap between high and low levels. Then, multi-scale stepwise fusion is performed. While reducing the amount of calculation, input data is paid attention to from different scales, and the overall expression ability of the multi-level multi-scale stepwise fusion network is enhanced. Then, multi-level stepwise fusion is performed to further reduce the semantic gap between high and low level features, and the feature fusion process is smoothed. The invention greatly improves the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image segmentation method and system based on a multi-level and multi-scale step-by-step fusion network. Background Art

[0002] With the rapid development of computer vision technology, image segmentation technology has gradually been widely used in various scenarios, especially medical image scenarios. Since medical images have high complexity adaptability, high data volume adaptability and processing accuracy requirements when processing, image segmentation technology has naturally become an important part of medical image processing.

[0003] In the prior art, medical image segmentation processing is usually performed by combining a visual transformer model with a medical image segmentation model UNet network to meet the higher performance requirements of medical image segmentation. However, existing medical image segmentation models usually lack multi-scale feature information at the same level, and the semantic gap between the acquired features is large, which greatly affects the accuracy of segmentation.

[0004] Therefore, how to design an image segmentation method to avoid the influence of multi-scale missing and semantic gap to improve the accuracy of image segmentation is an urgent problem to be solved. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide an image segmentation method and system based on a multi-level multi-scale step-by-step fusion network. By designing a multi-level multi-scale step-by-step fusion network, the accuracy of the overall image segmentation is improved. The multi-level multi-scale step-by-step fusion network first extracts semantic features of multiple different levels of the image, performs multi-level feature fusion, so as to improve the comprehensive understanding of the features, while reducing the semantic gap between high and low levels, and then performs multi-scale step-by-step fusion. While reducing the amount of calculation, it pays attention to input data from different scales, enhances the overall expression ability of the multi-level multi-scale step-by-step fusion network, and then performs multi-level step-by-step fusion to further reduce the semantic gap between high and low level features, and smoothes the feature fusion process. The present invention greatly improves the accuracy of image segmentation.

[0006] The present invention proposes an image segmentation method based on a multi-level and multi-scale step-by-step fusion network, comprising:

[0007] Preprocess the image to be segmented, and input the preprocessed image into a multi-level and multi-scale gradual fusion network;

[0008] Acquire multiple semantic features at different levels according to the preprocessed image, each of the semantic features contains semantic information at a single level, and any two of the semantic features contain semantic information at different levels;

[0009] Dividing each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and splicing the semantic feature sub-segments of different semantic features into a multi-level semantic feature;

[0010] Extracting a multi-scale feature corresponding to each of the multi-level semantic features according to the feature scale, wherein the multi-scale feature includes a plurality of single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature queue;

[0011] Performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, wherein the multi-scale fusion features uniquely correspond to the multi-level semantic features;

[0012] Arranging the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, performing feature fusion on the multi-scale fusion feature queue according to a preset fusion rule to obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature;

[0013] According to the multi-level fusion features, a final segmented image is obtained.

[0014] In summary, according to the above-mentioned image segmentation method based on a multi-level multi-scale stepwise fusion network, a multi-level multi-scale stepwise fusion network is designed to improve the accuracy of overall image segmentation. The multi-level multi-scale stepwise fusion network first extracts semantic features of multiple different levels of the image, performs multi-level feature fusion, so as to improve the comprehensive understanding of the features, and at the same time reduce the semantic gap between high and low levels, and then performs multi-scale stepwise fusion. While reducing the amount of calculation, it pays attention to input data from different scales, enhances the overall expression ability of the multi-level multi-scale stepwise fusion network, and then performs multi-level stepwise fusion to further reduce the semantic gap between high and low level features, and smoothes the feature fusion process. The present invention greatly improves the accuracy of image segmentation. Specifically, the image to be segmented is preprocessed, the preprocessed image is input into a multi-level multi-scale step-by-step fusion network, and multiple semantic features at different levels are obtained according to the preprocessed image, each of the semantic features contains semantic information of a single level, and the levels of semantic information contained in any two of the semantic features are different. Each of the semantic features is divided into multiple semantic feature sub-segments according to the channel dimension, and the semantic feature sub-segments of different semantic features are spliced ​​into a multi-level semantic feature to improve the comprehensive understanding of the feature and reduce the semantic gap between high and low levels. The multi-scale feature corresponding to each of the multi-level semantic features is extracted according to the feature scale, the multi-scale feature contains multiple single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature. The invention discloses a method for obtaining a multi-scale fusion feature queue, wherein the adjacent single-scale features in the single-scale feature queue are subjected to multi-stage pairwise fusion to obtain a multi-scale fusion feature, wherein the multi-scale fusion feature uniquely corresponds to the multi-level semantic feature, and the input data is paid attention to from different scales while reducing the amount of calculation, so as to enhance the overall expression ability of the multi-level multi-scale step-by-step fusion network, and the multi-scale fusion features are arranged according to the order of feature levels to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to feature fusion according to a preset fusion rule to obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature, so as to further reduce the semantic gap between high-level and low-level features and smooth the feature fusion process, and obtain a final segmented image according to the multi-level fusion feature. The invention greatly improves the accuracy of image segmentation.

[0015] Furthermore, the step of obtaining semantic features at multiple different levels based on the preprocessed image specifically includes:

[0016] After obtaining the preprocessed image, the semantic features are extracted linearly according to the order of the semantic information level to obtain semantic features of multiple different levels;

[0017] Then, channel dimension processing is performed on the semantic features at the multiple different levels, so as to uniformly map the channel dimensions of the semantic features at the multiple different levels to the same value.

[0018] Furthermore, the step of dividing each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and splicing the semantic feature sub-segments of different semantic features into a multi-level semantic feature specifically includes:

[0019] After obtaining semantic features of multiple different levels, the semantic features of each different level are evenly split along the channel dimension to obtain multiple semantic feature sub-segments, and then evenly divided into different feature groups according to the number of feature levels, each of which includes semantic features of all feature levels;

[0020] Perform up- and down-sampling operations on the features in each feature group to achieve resolution consistency, and then splice the features in the feature group with consistent resolution along the channel dimension to obtain multiple spliced ​​features;

[0021] After performing a convolution operation on the concatenated features, a multi-level semantic feature is obtained, each of which includes semantic information at multiple levels;

[0022] The specific algorithm for obtaining multi-level semantic features is as follows:

[0023] ,

[0024] ,

[0025] ,

[0026] in, represents the features after splitting, Indicates the number of splits, Indicates the level number, Represents a split operation, Represents semantic features, Represents the splicing feature, Represents a splicing operation, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, Represents multi-level semantic features.

[0027] Furthermore, the step of performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features specifically includes:

[0028] After obtaining the multi-level semantic features, the multi-level semantic features are split along the channel dimension, so that the split multi-level semantic features are respectively input into convolution blocks of different scales to obtain single-scale features of multiple different scales. The specific algorithm for obtaining single-scale features of multiple different scales is as follows:

[0029] ,

[0030] ,

[0031] in, Represents multi-level semantic features, Represents the multi-level semantic features after splitting, Represents a split operation, A single-scale feature representing multiple different scales, represents the activation function, , , and Represents convolution operations of different scales;

[0032] The single-scale features of multiple different scales are gradually fused to obtain fusion features. The step-by-step fusion includes multiple fusion stages. In each fusion stage, the semantic features of all adjacent scales are fused in pairs in parallel. The specific algorithm of the step-by-step fusion is as follows:

[0033] , ,

[0034] , ,

[0035] in, represents the fusion feature, represents the feature ordinal number, Represents a splicing operation, represents the fusion stage, Represents a single scale feature;

[0036] The gradually fused fusion features are residually connected with the original multi-level semantic features to obtain multi-scale fusion features. The specific algorithm of the residual connection is as follows:

[0037] ,

[0038] in, represents multi-scale fusion features, Represents multi-level semantic features, Represents a batch normalization operation.

[0039] Further, the step of arranging the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, and fusing the multi-scale fusion feature queue according to a preset fusion rule to obtain a multi-level fusion feature specifically includes:

[0040] After obtaining the multi-scale fusion features, the multi-scale fusion features are arranged in order according to the feature levels to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to multi-level step-by-step fusion, wherein the multi-level step-by-step fusion includes a plurality of fusion stages, and each fusion stage fuses adjacent level features of the current highest level feature, so as to gradually fuse from high to low according to the feature level order, so as to obtain the multi-level fusion features;

[0041] The specific algorithm of the multi-level step-by-step fusion is as follows:

[0042] , =1,

[0043] , ,

[0044] in, represents the output features of the fusion of features at adjacent levels, represents the feature ordinal number, represents the fusion stage, represents the adjacent level feature fusion operation, Represents multi-scale fusion features.

[0045] Furthermore, the step of fusing features at adjacent levels specifically includes:

[0046] The specific algorithm for adjacent level feature fusion is as follows:

[0047] ,

[0048] ,

[0049] ,

[0050] ,

[0051] in, Indicates the current highest level feature, represents the current highest level feature after upsampling, represents the upsampling operation, represents low-level features, CAS represents channel spatial attention block, SA represents spatial attention block, represents a 3×3 convolution operation, Represents a splicing operation, Represents the Hadamard product.

[0052] Furthermore, the step of obtaining the final segmented image according to the multi-level fusion features further includes:

[0053] The loss is calculated according to the loss function, and the specific algorithm of the loss function is as follows:

[0054] ,

[0055] ,

[0056] ,

[0057] in, represents the total loss, and represents the hyperparameter, represents the cross entropy loss, represents the Dice loss, N Represents the total number of pixels, represents the number of categories, m Represents the pixel ordinal number, n represents the category ordinal number, g represents the true value label, p Indicates the probability of the pixel corresponding to the category.

[0058] The present invention proposes an image segmentation system based on a multi-level and multi-scale step-by-step fusion network, comprising:

[0059] A preprocessing module is used to preprocess the image to be segmented and input the preprocessed image into a multi-level and multi-scale gradual fusion network;

[0060] A basic feature extraction module, used for obtaining semantic features of multiple different levels according to the preprocessed image, each of the semantic features contains semantic information of a single level, and any two of the semantic features contain semantic information of different levels;

[0061] A multi-level feature fusion module, used to divide each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and to splice the semantic feature sub-segments of different semantic features into a multi-level semantic feature;

[0062] A multi-scale step-by-step fusion module, used for extracting multi-scale features corresponding to each of the multi-level semantic features according to the feature scale, wherein the multi-scale features include a plurality of single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale features, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature queue;

[0063] Performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, wherein the multi-scale fusion features uniquely correspond to the multi-level semantic features;

[0064] A multi-level step-by-step fusion module, used to arrange the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, perform feature fusion on the multi-scale fusion feature queue according to a preset fusion rule, and obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature;

[0065] According to the multi-level fusion features, a final segmented image is obtained.

[0066] The present invention also provides a storage medium, which stores one or more programs, and when the programs are executed by a processor, the image segmentation method based on the multi-level and multi-scale step-by-step fusion network as described above is implemented.

[0067] The present invention further provides a computer device, the computer device comprising a memory and a processor, wherein:

[0068] The memory is used to store computer programs;

[0069] When the processor is used to execute the computer program stored in the memory, the image segmentation method based on the multi-level and multi-scale step-by-step fusion network as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart of an image segmentation method based on a multi-level and multi-scale step-by-step fusion network proposed in the first embodiment of the present invention;

[0071] Figure 2 A flowchart of an image segmentation method based on a multi-level and multi-scale step-by-step fusion network proposed in the second embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the structure of an image segmentation system based on a multi-level and multi-scale step-by-step fusion network proposed in the third embodiment of the present invention.

[0073] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0074] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0075] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0077] See also Figure 1 , which is a flowchart of an image segmentation method based on a multi-level multi-scale step-by-step fusion network proposed in the first embodiment of the present invention, the image segmentation method based on a multi-level multi-scale step-by-step fusion network includes steps S01 to S04, wherein:

[0078] Step S01: preprocessing the image to be segmented, inputting the preprocessed image into a multi-level multi-scale step-by-step fusion network, and obtaining semantic features of multiple different levels according to the preprocessed image;

[0079] It should be noted that the multi-level and multi-scale step-by-step fusion network in this embodiment extracts semantic features at multiple different levels through multiple PVTV2 encoders. The encoders are connected together with a linear structure, and each encoder corresponds to a unique semantic information level. After obtaining the preprocessed image, semantic features are linearly extracted according to the order of the semantic information levels to obtain semantic features at multiple different levels, and then channel dimension processing is performed on the semantic features at the multiple different levels to uniformly map the channel dimensions of the semantic features at the multiple different levels to the same value. In this embodiment, the PVTV2 encoder encodes the image to generate 4 features with different semantic information levels, and the number of feature channels is {64, 128, 320, 512} respectively. In order to effectively reduce the parameters and computational complexity of the overall model, the channel dimension of the features at each semantic level is uniformly mapped to 64 through the convolution module.

[0080] Step S02: Divide each semantic feature into multiple semantic feature sub-segments according to the channel dimension, and splice the semantic feature sub-segments of different semantic features into a multi-level semantic feature;

[0081] It should be noted that, in this embodiment, after obtaining semantic features at multiple different levels, each semantic feature at a different level is evenly split along the channel dimension to obtain multiple semantic feature sub-segments, and then evenly divided into different feature groups according to the number of feature levels, and each of the feature groups includes semantic features of all feature levels;

[0082] Perform up- and down-sampling operations on the features in each feature group to achieve resolution consistency, and then splice the features in the feature group with consistent resolution along the channel dimension to obtain multiple spliced ​​features;

[0083] After performing a convolution operation on the concatenated features, a multi-level semantic feature is obtained, each of which includes semantic information at multiple levels;

[0084] The specific algorithm for obtaining multi-level semantic features is as follows:

[0085] ,

[0086] ,

[0087] ,

[0088] in, represents the features after splitting, Indicates the number of splits, Indicates the level ordinal number, Represents a split operation, Represents semantic features, Represents the splicing feature, Represents a splicing operation, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, Represents multi-level semantic features;

[0089] In this embodiment, the features of the encoder are evenly split into 4 groups along the channel dimension, and then the feature groups corresponding to the features are up-sampled and down-sampled to achieve consistent resolution. At the same time, they are spliced ​​along the channel, and then feature extraction is performed through the convolution module to obtain multi-scale feature expression. Since each feature integrates features of multiple levels, it helps the model to fully understand the features and reduce the semantic gap between high and low levels to a certain extent.

[0090] Step S03: extracting multi-scale features corresponding to each multi-level semantic feature according to the feature scale, arranging the single-scale features in the multi-scale features in the order of feature scales to form a single-scale feature queue, performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, and the multi-scale fusion features uniquely correspond to the multi-level semantic features;

[0091] It should be noted that, in this embodiment, after obtaining the multi-level semantic features, the multi-level semantic features are split along the channel dimension, so that the split multi-level semantic features are respectively input into convolution blocks of different scales to obtain single-scale features of multiple different scales. The specific algorithm for obtaining single-scale features of multiple different scales is as follows:

[0092] ,

[0093] ,

[0094] in, Represents multi-level semantic features, Represents the multi-level semantic features after splitting, Represents a split operation, A single-scale feature representing multiple different scales, represents the activation function, , , and Represents convolution operations of different scales;

[0095] The single-scale features of multiple different scales are gradually fused to obtain fusion features. The step-by-step fusion includes multiple fusion stages. In each fusion stage, the semantic features of all adjacent scales are fused in pairs in parallel. The specific algorithm of the step-by-step fusion is as follows:

[0096] , ,

[0097] , ,

[0098] in, represents the fusion feature, represents the feature ordinal number, Represents a splicing operation, represents the fusion stage, Represents a single scale feature;

[0099] The gradually fused fusion features are residually connected with the original multi-level semantic features to obtain multi-scale fusion features. The specific algorithm of the residual connection is as follows:

[0100] ,

[0101] in, represents multi-scale fusion features, Represents multi-level semantic features, Represents batch normalization operation;

[0102] In this embodiment, the fusion of multi-scale features is crucial for processing lesion areas and organs with variable sizes and shapes in medical images. Different from the method of using element-by-element summation or serial fusion of multi-scale features, the present invention effectively utilizes each scale feature by gradually fusing adjacent scale features in parallel. First, the input features are divided into 4 groups of features along the channel. This not only reduces the amount of calculation, but also forms different subspaces like multi-head self-attention, allowing attention to input data from different angles and enhancing the expression ability of the overall model. Then, the kernel size is used. The convolution module extracts the multi-scale information of the feature group and forms 4 groups of multi-scale features.

[0103] Step S04: Arrange the multi-scale fusion features in order of feature levels to form a multi-scale fusion feature queue, perform feature fusion on the multi-scale fusion feature queue according to a preset fusion rule, obtain multi-level fusion features, and obtain a final segmented image based on the multi-level fusion features;

[0104] It should be noted that the preset fusion rule in this embodiment is that after obtaining the multi-scale fusion features, the multi-scale fusion features are arranged in order according to the feature level to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to multi-level step-by-step fusion, and the multi-level step-by-step fusion includes multiple fusion stages, and each of the fusion stages fuses the adjacent level features of the current highest level feature, so as to gradually fuse from high to low according to the feature level order, so as to obtain the multi-level fusion feature;

[0105] The specific algorithm of the multi-level step-by-step fusion is as follows:

[0106] , =1,

[0107] , ,

[0108] in, represents the output features of the fusion of features at adjacent levels, represents the feature ordinal number, represents the fusion stage, represents the adjacent level feature fusion operation, Represents multi-scale fusion features;

[0109] In the existing technology, the improved UNet-based models generally adopt a bottom-up feature fusion method. This method will cause the semantic gap between the encoder features and the decoder features to become larger as the top layers are fused, which seriously affects the segmentation performance. To this end, we propose a multi-level step-by-step fusion method, which includes three stages. Each stage parallelly fuses adjacent level features with relatively small semantic gaps, and uses the adjacent level interactive fusion designed in this paper to fuse the final fused features;

[0110] In order to further reduce the semantic gap between high-level and low-level features and smooth the feature fusion process, we design an interactive fusion of adjacent features. Compared with high-level features, low-level features have more background noise, but also contain richer detail information. Compared with low-level features, high-level features are insensitive to detail information. Therefore, the designed interactive fusion of adjacent features can enable the upper and lower features to interact in the spatial dimension. The attention map generated by the high-level features is used to guide the low-level features to filter the background noise, and the attention map generated by the low-level features is used to guide the high-level features to pay attention to the detail information. Specifically, the high-level feature map is upsampled to the same resolution as the low-level feature map. The two adjacent level feature maps are respectively obtained under the action of the graph spatial attention module to obtain the spatial attention map. Then, each spatial attention map is multiplied with the adjacent level features and then added with their own original input features. The obtained features are spliced ​​along the channel. Finally, the feature channel is reduced by 2 times through the convolution module, and the channel spatial attention module is introduced to weight the feature map in the spatial and channel dimensions, so that the model pays more attention to important channel and regional features.

[0111] The specific algorithm for adjacent level feature fusion is as follows:

[0112] ,

[0113] ,

[0114] ,

[0115] in, Indicates the current highest level feature, represents the current highest level feature after upsampling, represents the upsampling operation, represents low-level features, CAS represents channel spatial attention block, SA represents spatial attention block, represents a 3×3 convolution operation, Represents a splicing operation, represents the Hadamard product;

[0116] Due to the problem of class imbalance in medical image segmentation, the present invention uses a combination of cross entropy loss and Dice loss to define the loss function of the model, and performs loss calculation according to the loss function. The specific algorithm of the loss function is as follows:

[0117] ,

[0118] ,

[0119] ,

[0120] in, represents the total loss, and represents the hyperparameter, represents the cross entropy loss, represents the Dice loss, N Represents the total number of pixels, represents the number of categories, m Represents the pixel ordinal number, n represents the category ordinal number, g represents the true value label, p Indicates the probability of the pixel corresponding to the category.

[0121] In summary, according to the above-mentioned image segmentation method based on a multi-level multi-scale stepwise fusion network, a multi-level multi-scale stepwise fusion network is designed to improve the accuracy of overall image segmentation. The multi-level multi-scale stepwise fusion network first extracts semantic features of multiple different levels of the image, performs multi-level feature fusion, so as to improve the comprehensive understanding of the features, and at the same time reduce the semantic gap between high and low levels, and then performs multi-scale stepwise fusion. While reducing the amount of calculation, it pays attention to input data from different scales, enhances the overall expression ability of the multi-level multi-scale stepwise fusion network, and then performs multi-level stepwise fusion to further reduce the semantic gap between high and low level features, and smoothes the feature fusion process. The present invention greatly improves the accuracy of image segmentation. Specifically, the image to be segmented is preprocessed, the preprocessed image is input into a multi-level multi-scale step-by-step fusion network, and multiple semantic features at different levels are obtained according to the preprocessed image, each of the semantic features contains semantic information of a single level, and the levels of semantic information contained in any two of the semantic features are different. Each of the semantic features is divided into multiple semantic feature sub-segments according to the channel dimension, and the semantic feature sub-segments of different semantic features are spliced ​​into a multi-level semantic feature to improve the comprehensive understanding of the feature and reduce the semantic gap between high and low levels. The multi-scale feature corresponding to each of the multi-level semantic features is extracted according to the feature scale, the multi-scale feature contains multiple single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature. The invention discloses a method for obtaining a multi-scale fusion feature queue, wherein the adjacent single-scale features in the single-scale feature queue are subjected to multi-stage pairwise fusion to obtain a multi-scale fusion feature, wherein the multi-scale fusion feature uniquely corresponds to the multi-level semantic feature, and the input data is focused on from different scales while reducing the amount of calculation, so as to enhance the overall expression ability of the multi-level multi-scale step-by-step fusion network, and the multi-scale fusion features are arranged in order of feature levels to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to feature fusion according to a preset fusion rule to obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature to further reduce the semantic gap between high-level and low-level features and smooth the feature fusion process, and the final segmented image is obtained according to the multi-level fusion feature. The invention greatly improves the accuracy of image segmentation.

[0122] See also Figure 2 , which is a flow chart of an image segmentation method based on a multi-level multi-scale step-by-step fusion network proposed in the second embodiment of the present invention, wherein the image segmentation method based on a multi-level multi-scale step-by-step fusion network comprises steps S11 to S15, wherein:

[0123] Step S11: after obtaining the preprocessed image, linearly extracting semantic features according to the order of the semantic information level to obtain semantic features of multiple different levels, and then performing channel dimension processing on the semantic features of multiple different levels to uniformly map the channel dimensions of the semantic features of multiple different levels to the same value;

[0124] It should be noted that the order of the semantic information levels in this embodiment is progressive from a low semantic information level to a high semantic information level, corresponding to the linear structure of the encoder, and adopts a linear order.

[0125] Step S12: After obtaining semantic features at multiple different levels, each semantic feature at a different level is evenly split along the channel dimension to obtain multiple semantic feature sub-segments, and then evenly divided into different feature groups according to the number of feature levels, and up- and down-sampling operations are performed on the features in each feature group to perform resolution consistency, and then the features in the feature group after resolution consistency are spliced ​​along the channel dimension to obtain multiple spliced ​​features, and the spliced ​​features are convolved to obtain multi-level semantic features;

[0126] It should be noted that the semantic feature sub-segments in this embodiment are obtained by splitting a single semantic feature and have semantic information of corresponding levels. One semantic feature sub-segment is selected from each level to form a feature group, and each feature group includes semantic information of all levels.

[0127] Step S13: after obtaining the multi-level semantic features, the multi-level semantic features are split along the channel dimension, and the split multi-level semantic features are respectively input into convolution blocks of different scales to obtain a plurality of single-scale features of different scales, and the plurality of single-scale features of different scales are gradually fused to obtain fusion features, wherein the gradual fusion includes a plurality of fusion stages, and each fusion stage fuses the semantic features of all adjacent scales in pairs in parallel, and the fusion features after the gradual fusion are residually connected with the original multi-level semantic features to obtain the multi-scale fusion features;

[0128] Step S14: after obtaining the multi-scale fusion features, the multi-scale fusion features are arranged in order according to the feature level to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to multi-level step-by-step fusion. The multi-level step-by-step fusion includes multiple fusion stages, and each fusion stage fuses the features of the adjacent levels of the current highest level feature, so as to gradually fuse from high to low according to the feature level order, so as to obtain the multi-level fusion features;

[0129] Step S15: Calculate the loss according to the loss function;

[0130] It should be noted that the image segmentation method based on the multi-level multi-scale step-by-step fusion network proposed in the present invention is compared with the existing model for medical image segmentation. The test results are shown in Table 1 below:

[0131] Table 1

[0132]

[0133] Among them, evaluation indicators 1, 2, 3, and 4 are Dice, RV, Myo, and LV, respectively. According to the data in Table 1 above, it can be seen that the image segmentation method based on the multi-level and multi-scale step-by-step fusion network of the present invention has significant progress compared with the existing models for medical image segmentation, and greatly improves the accuracy of image segmentation.

[0134] In summary, according to the above-mentioned image segmentation method based on a multi-level multi-scale stepwise fusion network, a multi-level multi-scale stepwise fusion network is designed to improve the accuracy of overall image segmentation. The multi-level multi-scale stepwise fusion network first extracts semantic features of multiple different levels of the image, performs multi-level feature fusion, so as to improve the comprehensive understanding of the features, and at the same time reduce the semantic gap between high and low levels, and then performs multi-scale stepwise fusion. While reducing the amount of calculation, it pays attention to input data from different scales, enhances the overall expression ability of the multi-level multi-scale stepwise fusion network, and then performs multi-level stepwise fusion to further reduce the semantic gap between high and low level features, and smoothes the feature fusion process. The present invention greatly improves the accuracy of image segmentation. Specifically, the image to be segmented is preprocessed, the preprocessed image is input into a multi-level multi-scale step-by-step fusion network, and multiple semantic features at different levels are obtained according to the preprocessed image, each of the semantic features contains semantic information of a single level, and the levels of semantic information contained in any two of the semantic features are different. Each of the semantic features is divided into multiple semantic feature sub-segments according to the channel dimension, and the semantic feature sub-segments of different semantic features are spliced ​​into a multi-level semantic feature to improve the comprehensive understanding of the feature and reduce the semantic gap between high and low levels. The multi-scale feature corresponding to each of the multi-level semantic features is extracted according to the feature scale, the multi-scale feature contains multiple single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature. The invention discloses a method for obtaining a multi-scale fusion feature queue, wherein the adjacent single-scale features in the single-scale feature queue are subjected to multi-stage pairwise fusion to obtain a multi-scale fusion feature, wherein the multi-scale fusion feature uniquely corresponds to the multi-level semantic feature, and the input data is focused on from different scales while reducing the amount of calculation, so as to enhance the overall expression ability of the multi-level multi-scale step-by-step fusion network, and the multi-scale fusion features are arranged in order of feature levels to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to feature fusion according to a preset fusion rule to obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature to further reduce the semantic gap between high-level and low-level features and smooth the feature fusion process, and the final segmented image is obtained according to the multi-level fusion feature. The invention greatly improves the accuracy of image segmentation.

[0135] See also Figure 3 , which is a schematic diagram of the structure of an image segmentation system based on a multi-level and multi-scale step-by-step fusion network proposed in the third embodiment of the present invention, and the system includes:

[0136] A preprocessing module 10 is used to preprocess the image to be segmented and input the preprocessed image into a multi-level multi-scale step-by-step fusion network;

[0137] A basic feature extraction module 20 is used to obtain multiple semantic features of different levels according to the preprocessed image, each of the semantic features contains a single level of semantic information, and any two of the semantic features contain semantic information of different levels;

[0138] A multi-level feature fusion module 30, for dividing each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and splicing the semantic feature sub-segments of different semantic features into a multi-level semantic feature;

[0139] A multi-scale step-by-step fusion module 40 is used to extract a multi-scale feature corresponding to each of the multi-level semantic features according to the feature scale, wherein the multi-scale feature includes a plurality of single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature queue;

[0140] Performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, wherein the multi-scale fusion features uniquely correspond to the multi-level semantic features;

[0141] A multi-level step-by-step fusion module 50 is used to arrange the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, and perform feature fusion on the multi-scale fusion feature queue according to a preset fusion rule to obtain a multi-level fusion feature. The preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature;

[0142] According to the multi-level fusion features, a final segmented image is obtained.

[0143] The present invention also proposes a computer storage medium on which one or more programs are stored, and when the program is executed by a processor, the above-mentioned image segmentation method based on a multi-level and multi-scale step-by-step fusion network is implemented.

[0144] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned image segmentation method based on a multi-level and multi-scale step-by-step fusion network.

[0145] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0146] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0147] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0148] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0149] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An image segmentation method based on a multi-level and multi-scale step-by-step fusion network, characterized in that: include: Preprocess the image to be segmented, and input the preprocessed image into a multi-level and multi-scale gradual fusion network; Acquire multiple semantic features at different levels according to the preprocessed image, each of the semantic features contains semantic information at a single level, and any two of the semantic features contain semantic information at different levels; Dividing each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and splicing the semantic feature sub-segments of different semantic features into a multi-level semantic feature; Extracting a multi-scale feature corresponding to each of the multi-level semantic features according to the feature scale, wherein the multi-scale feature includes a plurality of single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale feature, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature queue; Performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, wherein the multi-scale fusion features uniquely correspond to the multi-level semantic features; The step of performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features specifically includes: After obtaining the multi-level semantic features, the multi-level semantic features are split along the channel dimension, so that the split multi-level semantic features are respectively input into convolution blocks of different scales to obtain single-scale features of multiple different scales. The specific algorithm for obtaining single-scale features of multiple different scales is as follows: ， ， in, Represents multi-level semantic features, Represents the multi-level semantic features after splitting, Represents a split operation, A single-scale feature representing multiple different scales, represents the activation function, , , and Represents convolution operations of different scales; The single-scale features of multiple different scales are gradually fused to obtain fusion features. The step-by-step fusion includes multiple fusion stages. In each fusion stage, the semantic features of all adjacent scales are fused in pairs in parallel. The specific algorithm of the step-by-step fusion is as follows: ， ， ， ， in, represents the fusion feature, represents the feature ordinal number, Represents a splicing operation, represents the fusion stage, Represents a single scale feature; The gradually fused fusion features are residually connected with the original multi-level semantic features to obtain multi-scale fusion features. The specific algorithm of the residual connection is as follows: ， in, represents multi-scale fusion features, Represents multi-level semantic features, Represents batch normalization operation; Arranging the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, performing feature fusion on the multi-scale fusion feature queue according to a preset fusion rule to obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature; According to the multi-level fusion features, a final segmented image is obtained.

2. The image segmentation method based on a multi-level and multi-scale step-by-step fusion network according to claim 1, characterized in that: The step of obtaining semantic features at multiple different levels according to the preprocessed image specifically includes: After obtaining the preprocessed image, the semantic features are extracted linearly according to the order of the semantic information level to obtain semantic features of multiple different levels; Then, channel dimension processing is performed on the semantic features at the multiple different levels, so as to uniformly map the channel dimensions of the semantic features at the multiple different levels to the same value.

3. The image segmentation method based on a multi-level and multi-scale step-by-step fusion network according to claim 1, characterized in that: The step of dividing each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and splicing the semantic feature sub-segments of different semantic features into a multi-level semantic feature specifically includes: After obtaining semantic features of multiple different levels, the semantic features of each different level are evenly split along the channel dimension to obtain multiple semantic feature sub-segments, and then evenly divided into different feature groups according to the number of feature levels, each of which includes semantic features of all feature levels; Perform up- and down-sampling operations on the features in each feature group to achieve resolution consistency, and then splice the features in the feature group with consistent resolution along the channel dimension to obtain multiple spliced ​​features; After performing a convolution operation on the concatenated features, a multi-level semantic feature is obtained, each of which includes semantic information at multiple levels; The specific algorithm for obtaining multi-level semantic features is as follows: ， ， ， in, represents the features after splitting, Indicates the number of splits, Indicates the level ordinal number, Represents a split operation, Represents semantic features, Represents the splicing feature, Represents a splicing operation, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, Represents multi-level semantic features.

4. The image segmentation method based on a multi-level and multi-scale step-by-step fusion network according to claim 1, characterized in that: The step of arranging the multi-scale fusion features in order of feature levels to form a multi-scale fusion feature queue, and fusing the multi-scale fusion feature queue according to a preset fusion rule to obtain a multi-level fusion feature specifically includes: After obtaining the multi-scale fusion features, the multi-scale fusion features are arranged in order according to the feature levels to form a multi-scale fusion feature queue, and the multi-scale fusion feature queue is subjected to multi-level step-by-step fusion, wherein the multi-level step-by-step fusion includes a plurality of fusion stages, and each fusion stage fuses adjacent level features of the current highest level feature, so as to gradually fuse from high to low according to the feature level order, so as to obtain the multi-level fusion features; The specific algorithm of the multi-level gradual fusion is as follows: ， =1， ， ， in, represents the output features of the fusion of features at adjacent levels, represents the feature ordinal number, represents the fusion stage, represents the adjacent level feature fusion operation, Represents multi-scale fusion features.

5. The image segmentation method based on a multi-level and multi-scale step-by-step fusion network according to claim 4, characterized in that: The step of fusing features at adjacent levels specifically includes: The specific algorithm for adjacent level feature fusion is as follows: ， ， ， in, Indicates the current highest level feature, represents the current highest level feature after upsampling, represents the upsampling operation, represents low-level features, CAS represents channel spatial attention block, SA represents spatial attention block, represents a 3×3 convolution operation, Represents a splicing operation, Represents the Hadamard product.

6. The image segmentation method based on a multi-level and multi-scale step-by-step fusion network according to claim 1, characterized in that: The step of obtaining the final segmented image according to the multi-level fusion features further includes: The loss is calculated according to the loss function, and the specific algorithm of the loss function is as follows: ， ， ， in, represents the total loss, and represents the hyperparameter, represents the cross entropy loss, represents the Dice loss, N Represents the total number of pixels, represents the number of categories, m Represents the pixel ordinal number, n represents the category ordinal number, g represents the true value label, p Indicates the probability of the pixel corresponding to the category.

7. An image segmentation system based on a multi-level and multi-scale step-by-step fusion network, characterized in that: include: A preprocessing module is used to preprocess the image to be segmented and input the preprocessed image into a multi-level and multi-scale gradual fusion network; A basic feature extraction module, used for obtaining semantic features of multiple different levels according to the preprocessed image, each of the semantic features contains semantic information of a single level, and any two of the semantic features contain semantic information of different levels; A multi-level feature fusion module, used to divide each of the semantic features into a plurality of semantic feature sub-segments according to the channel dimension, and to splice the semantic feature sub-segments of different semantic features into a multi-level semantic feature; A multi-scale step-by-step fusion module, used for extracting multi-scale features corresponding to each of the multi-level semantic features according to the feature scale, wherein the multi-scale features include a plurality of single-scale features, and the feature scales of any two of the single-scale features are different. In the multi-scale features, the single-scale features are arranged in order according to the feature scale size to form a single-scale feature queue; Performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features, wherein the multi-scale fusion features uniquely correspond to the multi-level semantic features; The step of performing multi-stage pairwise fusion of adjacent single-scale features in the single-scale feature queue to obtain multi-scale fusion features specifically includes: After obtaining the multi-level semantic features, the multi-level semantic features are split along the channel dimension, so that the split multi-level semantic features are respectively input into convolution blocks of different scales to obtain single-scale features of multiple different scales. The specific algorithm for obtaining single-scale features of multiple different scales is as follows: ， ， in, Represents multi-level semantic features, Represents the multi-level semantic features after splitting, Represents a split operation, A single-scale feature representing multiple different scales, represents the activation function, , , and Represents convolution operations of different scales; The single-scale features of multiple different scales are gradually fused to obtain fusion features. The step-by-step fusion includes multiple fusion stages. In each fusion stage, the semantic features of all adjacent scales are fused in pairs in parallel. The specific algorithm of the step-by-step fusion is as follows: ， ， ， ， in, represents the fusion feature, represents the feature ordinal number, Represents a splicing operation, represents the fusion stage, Represents a single scale feature; The gradually fused fusion features are residually connected with the original multi-level semantic features to obtain multi-scale fusion features. The specific algorithm of the residual connection is as follows: ， in, represents multi-scale fusion features, Represents multi-level semantic features, Represents batch normalization operation; A multi-level step-by-step fusion module, used to arrange the multi-scale fusion features according to the order of feature levels to form a multi-scale fusion feature queue, perform feature fusion on the multi-scale fusion feature queue according to a preset fusion rule, and obtain a multi-level fusion feature, wherein the preset fusion rule is used to fuse the multi-scale fusion feature queue into a multi-level fusion feature; According to the multi-level fusion features, a final segmented image is obtained.

8. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the image segmentation method based on a multi-level and multi-scale step-by-step fusion network as described in any one of claims 1 to 6.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the image segmentation method based on the multi-level and multi-scale step-by-step fusion network described in any one of claims 1 to 6.

Images