Image segmentation method and device based on multi-path fusion convolution, equipment and medium

By introducing multi-path fusion convolution and dual-branch downsampling modules into the medical image segmentation model, the problems of high computational complexity and insufficient processing of detailed information are solved, efficient medical image segmentation is achieved, and lesion segmentation is improved Accuracy and model applicability.

CN119068187APending Publication Date: 2024-12-03HUNAN UNIV OF CHINESE MEDICINE
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411094417.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing medical image segmentation model has high computational complexity, large model parameters, insufficient processing of local detailed information when processing complex medical images, making it difficult to achieve efficient real-time segmentation, and insufficient accurate positioning of the details of the lesion edge.

Method used

Using the image segmentation method based on multipath fusion convolution, the shallow feature acquisition module with depth separation of convolution and residual structures is added to the encoder head, and the convolution module is replaced with the multipath fusion convolution module, and downsampling is performed. During the process, a dual-branch downsampling module is used to reduce the loss of feature information and improve the accuracy of lesion segmentation.

Benefits of technology

It reduces the calculation cost and parameter quantity, improves the accuracy of lesion segmentation and the applicability of the model, can better adapt to complex situations such as lesion morphology and blurred edges, meets the requirements of rapid segmentation, and improves the effect of medical image segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068187A_ABST
    Figure CN119068187A_ABST
Patent Text Reader

Abstract

The invention relates to an image segmentation method and device based on multi-path fusion convolution, equipment and a medium. The method comprises the following steps of: adding a shallow feature acquisition module consisting of deep convolution and point convolution with the convolution kernel size of 7 * 7 to the head of an encoder so as to reserve main features in an input image and reduce calculation cost and parameter quantity; the double-branch down-sampling module is provided with a convolution branch and a pooling branch, and the focus segmentation precision is improved by reducing a large amount of loss of feature information; each path in the multi-path fusion convolution module is composed of different numbers of convolutions, and the multi-path convolutions and information are fused, so that the network can obtain different scales of lesion features and global context information, the network can better adapt to complex conditions such as different lesion forms and fuzzy edges, and the applicability of the model is improved; according to the method, the rapid segmentation requirement of an actual application scene is met, effective features can be extracted from different depth feature layers, and the medical image segmentation effect is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation method, device, equipment and medium based on multi-path fusion convolution. Background Art

[0002] The accuracy and efficiency of medical image segmentation have an important impact on medical diagnosis and treatment. In recent years, deep learning methods based on convolutional neural networks (CNNs) have made significant breakthroughs in the field of medical image segmentation. The UNet model proposed by Ronneberger et al. first adopted an encoder-decoder structure. The encoder is used to extract the feature representation of the input image, and the decoder restores the resolution of the original image through upsampling operations, and fuses the features extracted in the encoder with the information in the decoder to achieve accurate target positioning. It has been widely used in medical image segmentation. Zhou et al. proposed the Attention-UNet model. By introducing the attention mechanism on the basis of the traditional UNet structure, the model can automatically learn and focus on important feature areas, solving the problem that traditional UNet may ignore detail information when processing large images, thereby improving the accuracy and efficiency of image segmentation. Jha et al. proposed the ResUNet++ model, which combines the advantages of Residual connection and UNet structure. By introducing dense connections and multi-scale feature fusion mechanisms, the ResUNet++ model effectively solves the problems of gradient vanishing and information loss in deep network training while improving the feature extraction capability, and improves the performance and stability of image segmentation. Xu et al. proposed the LeViT-UNet model, which uses the visual attention mechanism to collect long-range spatial information from feature maps of different scales, and achieved excellent performance in medical image segmentation tasks. Valanarasu et al. proposed the UNeXt network, which effectively labeled and projected convolutional features, achieving faster reasoning, lower complexity, and fewer parameters. Chen et al. proposed the TransUNet model, which uses the self-attention mechanism of the Transformer to capture global information in the image and effectively represents image features through sequence modeling, and achieved remarkable results in medical image segmentation tasks.

[0003] Although the above models have achieved remarkable results in medical image segmentation tasks, the existing models still have problems such as high computational complexity, large number of model parameters, and insufficient processing of local detail information when processing complex medical images. It is difficult to achieve efficient real-time segmentation, and there is still room for improvement in the accurate positioning of details such as lesion edges. Summary of the invention

[0004] Based on this, it is necessary to provide an image segmentation method, device, equipment and medium based on multi-path fusion convolution to address the above technical problems.

[0005] An image segmentation method based on multi-path fusion convolution, the method comprising:

[0006] A medical image is acquired, and preprocessed to obtain a preprocessing result.

[0007] The preprocessing results are input into the trained image segmentation model based on multi-path fusion convolution to obtain the medical image segmentation results.

[0008] Among them, the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder; the encoder replaces the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replaces the second to fourth convolution modules with multi-path fusion convolution modules, and replaces the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder replaces all convolution modules in the classic UNet network decoder with multi-path fusion convolution modules; the shallow feature acquisition module is used to extract shallow features in the input image using depthwise separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and salient features of the input features using convolution branches and pooling branches respectively, and fuse the features extracted by the two branches.

[0009] An image segmentation device based on multi-path fusion convolution, the device comprising:

[0010] The image preprocessing module is used to acquire medical images, preprocess the medical images, and obtain preprocessing results.

[0011] The image segmentation module is used to input the preprocessing results into the trained image segmentation model based on multi-path fusion convolution to obtain the medical image segmentation results; wherein, the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder; the encoder is obtained by replacing the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replacing the second to fourth convolution modules with multi-path fusion convolution modules, and replacing the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder is obtained by replacing all convolution modules in the classic UNet network decoder with multi-path fusion convolution modules; the shallow feature acquisition module is used to extract shallow features in the input image using depth-separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and significant features of input features using convolution branches and pooling branches respectively, and fuse the features extracted by the two branches.

[0012] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0013] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0014] The above-mentioned image segmentation method, device, equipment and medium based on multi-path fusion convolution, the method includes: adding a shallow feature acquisition module composed of deep convolution and point convolution with a convolution kernel size of 7×7 to the encoder head to retain the main features in the input image and reduce the calculation cost and parameter amount; the dual-branch downsampling module has a convolution branch and a pooling branch, which improves the lesion segmentation accuracy by reducing the large loss of feature information; each path in the multi-path fusion convolution module is composed of a different number of convolutions, and the multi-path convolution and information are fused, so that the network can obtain lesion features and global context information of different scales, better adapt to complex situations such as different lesion morphology and blurred edges, and improve the applicability of the model; this method meets the requirements of fast segmentation in actual application scenarios, and can extract effective features from different depth feature layers to improve the medical image segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of a process of an image segmentation method based on multi-path fusion convolution in one embodiment;

[0016] Figure 2 A structural diagram of an image segmentation model in another embodiment;

[0017] Figure 3 This is a structural diagram of a SFA-Block shallow feature acquisition module in another embodiment;

[0018] Figure 4 A CM-Block dual-branch downsampling module in another embodiment;

[0019] Figure 5 It is a structural diagram of the MFC-Block multi-scale fusion convolution module in another embodiment;

[0020] Figure 6 : This is a comparison of the segmentation results of different methods on the ISIC-2018 dataset in another embodiment, where (a) is the original image, (b) is the true label, (c) is the segmentation result of the UNet model, (d) is the segmentation result of the AttentionUNet model, (e) is the segmentation result of the ResUNet++ model, (f) is the segmentation result of the TransUNet model, (g) is the segmentation result of the LeViTUNet model, (h) is the segmentation result of the UNeXt model, and (i) is the segmentation result of the model of this method;

[0021] Figure 7 The figure is a comparison of the segmentation results of different methods on the Kvasir-SEG dataset in another embodiment, wherein (a) is the original image, (b) is the true label, (c) is the segmentation result of the UNet model, (d) is the segmentation result of the AttentionUNet model, (e) is the segmentation result of the ResUNet++ model, (f) is the segmentation result of the TransUNet model, (g) is the segmentation result of the LeViTUNet model, (h) is the segmentation result of the UNeXt model, and (i) is the segmentation result of the model of this method;

[0022] Figure 8 The results of the first 20 epochs of each model in two test data sets in another embodiment are shown in FIG. 1 , where (a) is the test result on the ISIC-2018 data set, and (b) is the test result on the Kvasir-SEG data set;

[0023] Fig. 9 is a structural block diagram of an image segmentation device based on multi-path fusion convolution in one embodiment;

[0024] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The image segmentation model based on multi-path fusion convolution proposed in this application is referred to as the MPFC-Net model.

[0027] In one embodiment, Figure 1 As shown, a method for image segmentation based on multi-path fusion convolution is provided, and the method comprises the following steps:

[0028] Step 100: Acquire a medical image, preprocess the medical image, and obtain a preprocessing result.

[0029] Specifically, the steps of preprocessing the medical images include: adjusting the size of each medical image to 512×512, and using image transformation techniques such as random cropping, random rotation, horizontal and vertical flipping to perform data augmentation on the training set.

[0030] Step 102: Input the preprocessing result into the trained image segmentation model based on multi-path fusion convolution to obtain the medical image segmentation result; wherein, the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder; wherein, the encoder is obtained by replacing the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replacing the second to fourth convolution modules with multi-path fusion convolution modules, and replacing the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder is obtained by replacing all convolution modules in the classic UNet network decoder with multi-path fusion convolution modules; the shallow feature acquisition module is used to extract shallow features in the input image by using depth-separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales by using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and salient features of input features by using convolution branches and pooling branches, respectively, and fuse the features extracted by the two branches.

[0031] Specifically, the structure of the image segmentation model based on multi-path fusion convolutional network (MPFC-Net) is as follows: Figure 2As shown in Figure 2, MPFC-Net includes 1 shallow feature acquisition module, 4 dual-branch downsampling modules and 8 multi-path fusion convolution modules. The shallow feature acquisition module is located in the model encoder head, the dual-branch downsampling module is located in each part of the downsampling process, and the multi-path fusion convolution module is located in the convolution operation in the entire network architecture.

[0032] The network architecture of the image segmentation model based on multi-path fusion convolution retains the original encoder-decoder architecture and skip connection module. In the encoder part, most of the effective feature information in the input image is first retained through a shallow feature acquisition module, in which the 7×7 convolution kernel can broaden the receptive field of the network without increasing the number of parameters. The multi-path fusion convolution module (Multi-path fusion convolution module, referred to as MFC module) adopts multi-path convolution, which fuses channel information in different perceptual ranges and highlights positive semantic features. Each module is followed by a dual-branch downsampling module (Convolution and Maxpooling modules, referred to as CM modules), which reduces feature loss in the downsampling operation by fusing the features extracted by the pooling branch and the convolution branch. The encoder part is used to gradually extract the features of the input image and encode it into a higher-level representation. The decoder uses transposed convolution to restore the low-resolution feature map obtained by the encoder downsampling to a feature map with the same resolution as the input image, and combines skip connections to fuse low-level features with high-level semantic information to produce more accurate segmentation boundaries and object masks. Finally, the number of channels of the feature is changed through a 1×1 point-wise convolution to obtain the final segmentation mask.

[0033] Introducing a shallow feature acquisition module in the encoder head can more fully acquire features from the input image.

[0034] The dual-branch downsampling module extracts more detailed feature information in the feature map.

[0035] The multi-path fusion convolution module adopts a multi-path structure, and each path consists of a different number of convolutions in order to obtain semantic information at different levels and improve the generalization ability of the model.

[0036] In the above-mentioned image segmentation method based on multi-path fusion convolution, the method comprises:

[0037] A shallow feature acquisition module consisting of deep convolution and point convolution with a convolution kernel size of 7×7 is added to the encoder head to retain the main features of the input image and reduce the computational cost and parameter amount; the dual-branch downsampling module has a convolution branch and a pooling branch, which improves the lesion segmentation accuracy by reducing the large loss of feature information; each path in the multi-path fusion convolution module is composed of a different number of convolutions, and the multi-path convolution and information are fused, so that the network can obtain lesion features and global context information of different scales, better adapt to complex situations such as different lesion morphology and blurred edges, and improve the applicability of the model; this method meets the requirements of fast segmentation in actual application scenarios, and can extract effective features from different depth feature layers to improve the medical image segmentation effect.

[0038] In one embodiment, step 102 includes the following steps:

[0039] Step 200: Input the preprocessing result into the encoder to obtain five encoding features of different scales.

[0040] Step 202: The fifth-scale coding feature is subjected to feature extraction by a multi-path fusion convolution module and then up-sampled, the up-sampling result is fused with the fourth-scale coding feature, and the fusion result is input into the first multi-path fusion convolution module of the decoder to obtain the fourth-scale decoding feature.

[0041] Step 204: upsample the fourth-scale decoding features and fuse them with the third-scale encoding features, and input the obtained fusion result into the second multi-path fusion convolution module of the decoder to obtain the third-scale decoding features.

[0042] Step 206: upsample the third-scale decoding features and fuse them with the second-scale coding features, and input the obtained fusion result into the third multi-path fusion convolution module of the decoder to obtain the second-scale decoding features.

[0043] Step 208: upsample the second-scale decoding features and fuse them with the first-scale encoding features, and input the obtained fusion result into the fourth multi-path fusion convolution module of the decoder to obtain the first-scale decoding features.

[0044] Step 210: performing point convolution on the first-scale decoded features to obtain a medical image segmentation result.

[0045] In one embodiment, step 200: includes: inputting the preprocessing result into a shallow feature acquisition module of the encoder to obtain a first-scale coding feature; downsampling the first-scale coding feature using a dual-branch downsampling module to obtain a first downsampling feature; inputting the first downsampling feature into a first multi-path fusion convolution module of the encoder to obtain a second-scale coding feature; downsampling the second-scale coding feature using a dual-branch downsampling module and inputting it into a second multi-path fusion convolution module of the encoder to obtain a third-scale coding feature; downsampling the third-scale coding feature using a dual-branch downsampling module and inputting it into a third multi-path fusion convolution module of the encoder to obtain a fourth-scale coding feature; downsampling the fourth-scale coding feature using a dual-branch downsampling module to obtain a fifth-scale coding feature.

[0046] In one embodiment, the shallow feature acquisition module includes: a 3×3 convolution layer, a 7×7 deep convolution layer and a point convolution layer; the preprocessing result is input into the shallow feature acquisition module of the encoder to obtain the first-scale coding feature, including: inputting the preprocessing result into the 3×3 convolution layer of the shallow feature acquisition module to obtain a shallow convolution feature map; inputting the shallow convolution feature map into and out of the 7×7 deep convolution layer of the shallow feature acquisition module, and inputting the obtained deep convolution result into the point convolution layer of the shallow feature acquisition module to obtain the point convolution feature; fusing the shallow convolution feature map and the point convolution feature to obtain the first-scale coding feature.

[0047] Specifically, for most medical image segmentation networks, when extracting input image features in the encoder head, stacked 3×3 convolutions are generally used to obtain low-level semantic information. For example, the U-Net architecture, which is widely used in different models, extracts and processes features by stacking two 3×3 convolution layers. The stacking of convolution layers helps the model capture features at different levels of abstraction in the image; ResUNet++ stacks three convolution layers and adds a residual structure to mitigate the potential impact of gradient vanishing. Stacking more convolution blocks can increase the receptive field of the model and construct more levels of feature representation. However, it will lead to increased computational complexity, limiting the usability of the model in resource-constrained environments. In addition, recent studies have shown that when the number of stacked 3×3 convolutions increases, the effective receptive field will be reduced to a certain extent.

[0048] In order to solve this problem, the present application proposes a shallow feature acquisition module (SFA) for retaining the main features in the input image. Figure 3As shown in Figure 1, the main improvement of the shallow feature acquisition module is the use of depthwise separable convolution, where 3×3 convolution is used to change the number of channels and extract feature information, 7×7 depthwise convolution is used to merge large-scale spatial information, and 1×1 point convolution is used to combine channel information. Depthwise separable convolution is an efficient neural network architecture. The structure consists of two parts: depthwise convolution and pointwise convolution. Depthwise convolution applies a convolution filter on each channel separately, while pointwise convolution applies 1x1 convolution on the output of depthwise convolution to combine information from different channels. Compared with standard convolution, depthwise separable convolution greatly reduces the amount of computation and parameters, but there is no obvious loss in accuracy. Therefore, this method is widely used in lightweight neural networks. Its efficient computation and parameter efficiency enable these models to be deployed in resource-constrained environments and provide real-time responsiveness. Although depthwise separable convolution improves computational efficiency, it may limit the representation ability of the model, thereby affecting the performance of the model. To solve this problem, this application introduces a new shallow feature acquisition method in the encoder head, adopting a residual deep separable architecture with a large kernel size to enrich the underlying semantic information in the initial downsampling block and generate a more accurate segmentation mask.

[0049] By adding 7×7 deep convolution and 1×1 point convolution in the shallow feature acquisition module of the encoder head, the main features in the input image are retained and the computational cost and parameter amount are reduced; dual-branch downsampling with convolution branches and pooling branches is used to reduce the large loss of feature information to improve the lesion segmentation accuracy; through multi-path convolution and information fusion, the network can obtain lesion features and global context information of different scales, thereby improving the applicability of the model.

[0050] In one embodiment, the dual-branch downsampling module includes: a convolution branch, a pooling branch and a fusion layer; the first-scale decoding feature is downsampled using the dual-branch downsampling module to obtain a first downsampled feature, including: inputting the first-scale decoding feature into the convolution branch to obtain a convolution feature map; inputting the first-scale decoding feature into the pooling branch to obtain a pooling feature map; splicing the convolution feature map and the pooling feature map and inputting them into the fusion layer for processing through a second 3×3 convolution layer to obtain the first downsampled feature.

[0051] In one embodiment, the convolution branch includes: a first point convolution layer and a third 3×3 convolution layer; and the pooling branch includes a maximum pooling layer.

[0052] Specifically, in UNet, downsampling is usually achieved through pooling operations. Common pooling operations include maximum pooling and average pooling, whose main functions are to compress features, reduce the number of parameters and the risk of overfitting. By gradually downsampling, some feature information can be filtered to reduce the computational burden, but a large amount of information will be lost in the process, resulting in reduced segmentation accuracy of the model. Therefore, this application proposes a dual-branch downsampling module (Convolution and Maxpooling modules, CM), which improves the expressiveness of features and reduces the loss of lesion features by expanding the perception range. The dual-branch downsampling module structure is as follows: Figure 4 As shown in the figure, there are two branches, the convolution branch and the pooling branch. The convolution branch extracts more detailed feature information from the feature map, and the pooling branch is used to retain the most significant features in the input feature map to improve the generalization ability of the model. Finally, the feature maps extracted by the two branches are fused in terms of the number of channels, and the target feature map is obtained through 3×3 convolution. The purpose of 3×3 convolution is to change the number of channels and further extract features from the fused feature map.

[0053] The convolution branch consists of two convolution blocks, where the 1×1 convolution makes the branch focus more on the semantic information on the channel and reduces the number of parameters. The purpose of the 3×3 convolution is to extract features and restore the number of channels. Batch Normalization and Relu activation are performed after each convolution to make the data distribution regular and speed up the network convergence. The original image passes through the convolution branch to obtain the convolution feature map F. c The pooling branch uses the maximum pooling with a size of 3×3 and a step size of 2. The original image passes through the pooling branch to obtain the pooling feature map F p Finally, the feature map F is obtained through the fusion layer m The calculation formula of the CM module is:

[0054] F m =δ(Conv(Cat(F c ,F p ))) (1)

[0055] Among them, F m In order to finally obtain the downsampled feature map, δ is the batch normalization layer (Batch Normalization) and the Relu function, Cat(.) represents feature fusion on the number of channels, and Conv(.) is the convolution operation in the fusion layer.

[0056] In one embodiment, the multi-path fusion convolution module includes: the multi-path fusion convolution module includes: a second point convolution layer, two feature paths, a feature fusion module, a sixth 3×3 convolution layer and a fifth point convolution layer; wherein the first feature path includes a fourth 3×3 convolution layer; the second feature path includes a fifth 3×3 convolution layer; the feature fusion module includes an average pooling layer, a third point convolution layer, a fourth point convolution layer and a SoftMax activation function; step 202 includes: extracting the fifth scale coding feature using the multi-path fusion convolution module and then upsampling it, fusing the upsampling result with the fourth scale coding feature to obtain a fused feature map; inputting the fused feature map into the first multi-path fusion convolution module of the decoder, and passing the fused feature map through the second After processing by the point convolution layer, the points are respectively input into the fourth 3×3 convolution layer of the first feature path and the fifth 3×3 convolution layer of the second feature path to obtain the first path feature map and the second path feature map; the first path feature map and the second path feature map are spliced ​​to obtain the spliced ​​feature; the spliced ​​feature is input into the sixth 3×3 convolution layer to obtain the first fused feature map; the spliced ​​feature is input into the feature fusion module, and is processed by the average pooling layer, the third point convolution layer, and the fourth point convolution layer in sequence, and activated by the SoftMax function to obtain the second fused feature map; the spliced ​​feature, the first fused feature map and the second fused feature map are multiplied and processed by the fifth point convolution layer to obtain the final fused feature; the final fused feature is spliced ​​with the fused feature map to obtain the fourth scale decoding feature.

[0057] Specifically, stacking convolutional layers can increase the receptive field to a certain extent, but using convolutional layers with the same structure cannot obtain multi-scale feature information of the input feature map. In the task of medical image segmentation, the size and shape of lesions are different, and repeated stacking of the same convolution will result in low segmentation accuracy. The ResNeSt model proposed by Zhang et al. uses the complementary advantages of feature map attention and multi-path representation to produce more diverse representations through cross-feature interactions. However, the model still uses the same convolution operation and accepts the same convolution size.

[0058] In order to solve this problem and make it more suitable for medical image segmentation, this application optimizes its model structure and proposes a multi-path fusion convolution module. Each path uses a different convolution structure to ensure the full representation of multi-scale features. The structure of the multi-path fusion convolution module is as follows: Figure 5As shown in the figure, this module uses two feature paths to obtain feature information of different scales. First, they will go through a 1×1 convolution and a 3×3 convolution. The 1×1 convolution is used to obtain feature information of the current scale. In order to extract multi-scale features, the output features of the second path will be combined with the results of the first path and receive semantic information through a 3×3 convolution. In fusing the two sets of features, average pooling is used to collect global spatial information, and by assigning weights, the key information in the feature map is captured. The formula for fusing the feature map is:

[0059]

[0060] Where H, W, and C are the scales of the output feature map. Global average pooling sums the spatial information to enhance the robustness of the input feature map to spatial variations. The H×W×C tensor is then converted to a 1×1×C tensor Y c , establish the relationship between features and channels, and obtain the global feature information of the image and the importance of each channel through self-learning. The specific formula is:

[0061]

[0062] The pixel-level features are then mapped to the class probability distribution to achieve the segmentation of skin lesions. i (.) represents the use of batch normalization layer and ReLU activation. Two 1×1 convolutions quantize the spatial information Y obtained by global average pooling, and the formula is:

[0063]

[0064] Finally, the entire module is designed with a classic residual structure, which enables it to capture more complex features and allows the network to learn and update weights faster.

[0065] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0066] In a validation example, the method was tested on two public medical image datasets. The two public medical image datasets are the ISIC-2018 dataset and the Kvasir-SEG dataset. The ISIC-2018 dataset covers different types of lesions, patients of various ages and skin colors, and images under different imaging conditions; the Kvasir-SEG dataset includes 1,000 gastrointestinal polyp images and their corresponding segmentation masks, all of which are personally annotated and verified by senior gastroenterologists.

[0067] The detailed information of the medical segmentation dataset is shown in Table 1.

[0068] Table 1 Detailed information of the medical segmentation dataset

[0069] Dataset Image Input size Train Valid Test ISIC-2018 2594 Variable 1868 467 259 Kvasir-SEG 1000 Variable 700 200 100

[0070] (1) Evaluation indicators

[0071] In order to more objectively evaluate the effectiveness of the proposed method, this embodiment uses mean intersection over union (mIoU), accuracy, recall, precision, and F1 value to measure the segmentation performance of the proposed method. Among them, mIoU is a common metric used to compare the performance between each model. Since skin lesions are extremely small and irregular, there is an extreme class imbalance. The model performance of this experiment mainly refers to the results of the two indicators mIoU and F1-score, and other indicators are used as supplements. It should be noted that the higher the above indicators, the better the segmentation effect. The evaluation indicators are directly calculated from the confusion matrix, as shown in Formulas 5 to 9.

[0072]

[0073] Among them, TP and FP are true positive variables and false positive variables, respectively, representing the number of skin lesion pixels correctly segmented by the model and the number of background pixels incorrectly classified as skin lesions; correspondingly, TN is the true negative variable, representing the correctly segmented background pixels; FN is the false negative variable, indicating that the skin lesion pixels are incorrectly marked as background pixels.

[0074] (2) Experimental details

[0075] All methods in this embodiment are implemented using the PyTorch framework. The experiment was completed on a single NVIDIA Tesla P40 GPU (24GB) and a 6-core CPU. The Adam optimizer was used in the training, the initial learning rate lr was 10e-4, the batch size (batchsize) was set to 4, the iteration (epoch) was 150, and ReduceLROnPlateau was used to optimize the learning rate. In addition, other models were trained with default parameters, and the pre-trained ViT model was loaded when training TransUNet and LeViT-UNet, and the remaining models were trained from scratch.

[0076] During training, the image sizes of the ISIC-2018 dataset and the Kvasir-SEG dataset were uniformly resized to 512 × 512. Image transformation techniques such as horizontal flipping, rotation, and cutting out were randomly applied to the training set of each dataset with a probability of 0.25 for data augmentation to expand sample diversity and enhance the generalization of the model.

[0077] (3) Experimental results

[0078] 1) Ablation experiment

[0079] In order to evaluate the effectiveness of the MPFC-Net model and the importance of each module, this embodiment conducted an ablation experiment on the ISIC-2018 dataset. UNet+SFA represents the addition of a shallow feature extraction module to the first convolutional layer of the encoder, UNet+CM represents the addition of a dual-branch downsampling module to UNet, UNet+MFC represents the addition of a multi-path fusion convolution module to each convolutional layer, UNet+CM+MFC represents the addition of a CM module and an MFC module to UNet at the same time, and MPFC-Net represents the introduction of the three modules of SFA, CM and MFC into the UNet network structure, that is, the method of this application. Table 2 shows the experimental results of the model.

[0080] Table 2 Detailed ablation results of MPFC-Net architecture (mean ± standard deviation)

[0081] Model Accuracy Accuracy Recall F1 score Average intersection over U (mIoU) UNet 0.939±0.118 0.854±0.218 0.862±0.225 0.832±0.221 0.757±0.239 UNet+SFA 0.940±0.121 0.871±0.191 0.846±0.248 0.833±0.224 0.759±0.237 UNet+CM 0.946±0.109 0.893±0.162 0.876±0.190 0.860±0.176 0.786±0.204 UNet+MFC 0.948±0.107 0.860±0.176 0.921±0.156 0.868±0.163 0.793±0.189 UNet+SFA+MFC 0.952±0.108 0.881±0.178 0.906±0.168 0.874±0.174 0.805±0.194 MPFC-Net 0.960±0.101 0.908±0.159 0.895±0.159 0.885±0.152 0.818±0.178

[0082] As shown in Table 2, compared with UNet, after adding only SFA, except for the recall rate which decreased by 1.6%, the other indicators all increased slightly, indicating that SFA can improve the network's ability to capture low-level semantic feature information; after adding only CM, all indicators increased significantly, indicating that CM can extract more lesion detail feature information and reduce feature loss; after adding only MFC, all indicators increased significantly, even reaching a maximum of 92.1% in the recall rate, an increase of 5.9%. Therefore, it can be considered that MFC can extract richer multi-scale feature information, enhance the model's ability to adapt to different visual scales, and has a more significant impact than SFA and CM modules. After adding CM and MFC at the same time, the network's performance in distinguishing noise and target areas is improved. MPFC-Net, that is, after adding SFA, CM and MFC at the same time, the model has higher mIoU and F1-score, which increased by 6.1% and 5.3% respectively, which shows that the segmentation performance is best when all modules are reasonably combined, verifying the rationality and effectiveness of the MPFC-Net model.

[0083] 2) Comparative experiment

[0084] In order to verify the lesion segmentation performance of the method of this application, the experiment was compared with the medical image segmentation algorithm that has been widely used recently. The experimental results of different networks on the ISIC-2018 dataset are shown in Table 3, where the bold part is the optimal value of this indicator. As can be seen from Table 3, the mIoU, F1-score and accuracy on the ISIC-2018 dataset are 0.818, 0.885 and 0.960 respectively. Except for the recall rate which is only 1.3% lower than the UNeXt model, other indicators are better than other advanced algorithms. In summary, MPFC-Net performs well in the mIoU and F1-score indicators, and is better than other advanced algorithms as a whole, and can better distinguish lesion pixels from background pixels.

[0085] Table 3 Experimental results of different networks on the ISIC2018 dataset (mean ± standard deviation)

[0086] Model Accuracy Accuracy Recall F1 score Average intersection over U (mIoU) UNet 0.939±0.118 0.854±0.218 0.862±0.225 0.832±0.221 0.757±0.239 Attention-UNet 0.941±0.112 0.865±0.192 0.867±0.212 0.839±0.201 0.762±0.223 ResUNet++ 0.954±0.128 0.889±0.156 0.869±0.168 0.864±0.162 0.780±0.181 TransUNet 0.953±0.111 0.876±0.173 0.877±0.178 0.859±0.193 0.779±0.248 LeViTUNet 0.955±0.114 0.887±0.169 0.881±0.193 0.862±0.183 0.782±0.186 UNeX 0.950±0.098 0.884±0.168 0.908±0.139 0.877±0.146 0.804±0.179 MPFC-Net 0.960±0.101 0.908±0.159 0.895±0.159 0.885±0.152 0.818±0.178

[0087] 3) Generalization experiment

[0088] In order to verify the generalization ability of the model, the Kvasir-SEG dataset is used to evaluate the performance of the MPFC-Net model and other networks, and the experimental results are shown in Table 4. The results show that the F1 score of the MPFC-Net model is 0.872, which is 1.3% higher than TransUNet; the mIoU score is 0.754, which is 3.1% higher than LeViTUNet. In general, the MPFC-Net model proposed in this application shows the highest score in all evaluation indicators.

[0089] Table 4 Experimental results of different networks on the Kvasir-SEG dataset (mean ± standard deviation)

[0090] Model Accuracy Accuracy Recall F1 score Average intersection over U (mIoU) UNet 0.929±0.091 0.833±0.233 0.803±0.250 0.791±0.228 0.700±0.243 Attention-UNet 0.937±0.073 0.851±0.225 0.814±0.256 0.795±0.231 0.708±0.251 ResUNet++ 0.932±0.085 0.855±0.198 0.826±0.254 0.802±0.226 0.714±0.244 TransUNet 0.918±0.093 0.806±0.246 0.775±0.265 0.859±0.193 0.632±0.262 LeViTUNet 0.930±0.109 0.854±0.218 0.832±0.252 0.804±0.234 0.723±0.261 UNeX 0.918±0.097 0.785±0.285 0.798±0.279 0.742±0.266 0.648±0.278 MPFC-Net 0.942±0.093 0.867±0.223 0.847±0.231 0.872±0.224 0.754±0.251

[0091] 4) Computational performance evaluation

[0092] In order to more objectively reflect the performance of the model, this application measures the time complexity of the method of this application by calculating the number of floating-point operations (FLOPS) required to input a single 256×256 RGB skin cancer image into the model. The number of parameters required for statistical model training is used to measure the space complexity of the method of this application. The comparison results of the MPFC-Net model of this application and other advanced models are shown in Table 5.

[0093] Table 5 Parameters and floating point operations of different networks

[0094] Model Parameter quantity / M FLOPS / G UNet 34.52 65.52 Attention-UNet 34.88 66.63 ResUNet++ 14.48 70.99 TransUNet 66.82 32.63 LeViTUNet 52.16 25.66 UNeX 1.47 0.58 MPFC-Net 10.61 13.52

[0095] As can be seen from Table 5, the UNeXt model performs better in both Params and FLOPS. This is because the model uses a labeled multi-layer perceptron (MLP) in the latent space, which reduces the number of parameters and computational complexity, but the method of this application has improved the segmentation accuracy within an acceptable range. Compared with other models, the method of this application reduces the spatial complexity by at least 26.73% and at most 84.12%; the time complexity is reduced by at least 47.31% and at most 80.96%; in terms of segmentation accuracy, the method of this application is relatively better than the other 6 segmentation networks. In summary, the comprehensive performance of the MPFC-Net model of this application is better, but there is still room for improvement to reduce resource requirements while maintaining segmentation accuracy.

[0096] 5) Experimental conclusion

[0097] To further demonstrate the significant improvement of the MPFC-Net model in the skin cancer lesion segmentation task, some segmentation results were visualized using all models for challenging images. The segmentation results of different methods on the ISIC-2018 dataset are shown in Figure 2. Figure 6As shown, (a) is the original image, (b) is the true label, (c) is the segmentation result of the UNet model, (d) is the segmentation result of the AttentionUNet model, (e) is the segmentation result of the ResUNet++ model, (f) is the segmentation result of the TransUNet model, (g) is the segmentation result of the LeViTUNet model, (h) is the segmentation result of the UNeXt model, and (i) is the segmentation result of the model of this method. From the qualitative results, compared with other methods, the segmentation results of the MPFC-Net model clearly distinguish the adhesions, holes and boundaries of the lesions, and are closer to the true labels, which verifies that the method of this application can improve the ability of the model to segment the edges of the target area. The edge contours of the UNet model segmentation results are not very precise. This is because although UNet can retain some position information, the retained position information is not completely accurate, which may lead to the situation where the size or position of the target area is significantly different from the original image. AttentionUNet is not very clear in the edge detail segmentation, but AttentionUNet pays more attention to the target area and has a certain inhibitory effect on the background, so the segmentation result is smoother. ResNet++ usually adopts different feature fusion strategies, such as cascading, parallel connection, etc., to improve the model's ability to extract and express image features. However, on the ISIC-2018 dataset, more sophisticated feature fusion strategies may be needed to capture subtle features and structural information in the image, and ResNet++'s feature fusion is not flexible or adaptive enough. TransUNet and LeViTUNet are Transformer-based neural network architectures that mainly rely on self-attention mechanisms to capture global features in images. However, skin lesion images have complex structures and textures, and both local and global features need to be captured simultaneously for effective segmentation. Traditional self-attention mechanisms may be more inclined to capture global information when capturing image features, while ignoring local detail information, resulting in the model's lack of sensitivity to subtle features of the image. Although the UNeXt network effectively processes convolutional features through labeling and projection mechanisms, there are limitations in feature representation when processing skin cancer images. The UNeXt network design may not be able to fully capture these complex feature information, resulting in performance degradation. For the skin lesion segmentation task with complex backgrounds, the scale and distribution of features require different network structures to better capture, so the MPFC-Net model has better segmentation results. Although the evaluation parameters of the model in this application are not all optimal, the model in this application performs best in terms of the comprehensive performance of correct segmentation, over-segmentation and under-segmentation, as well as the accurate positioning of lesions.

[0098] like Figure 7The figure shows the comparison of segmentation results of different methods on the Kvasir-SEG dataset (colon polyp dataset), where (a) is the original image, (b) is the true label, (c) is the segmentation result of the UNet model, (d) is the segmentation result of the AttentionUNet model, (e) is the segmentation result of the ResUNet++ model, (f) is the segmentation result of the TransUNet model, (g) is the segmentation result of the LeViTUNet model, (h) is the segmentation result of the UNeXt model, and (i) is the segmentation result of the model of this method. By observing the segmentation results, it can be clearly seen that the segmentation mask generated by our model is closer to the true label, and the number of mis-segmented areas is relatively small compared to other methods. This further verifies the generalization ability of the method of this application on the colon polyp dataset and shows a more significant segmentation effect. This high generalization performance is particularly important in the field of medical image segmentation, because it means that the MPFC-Net model not only performs well on specific datasets, but also achieves good performance in different datasets and medical applications. This capability makes the MPFC-Net model more widely applicable and can be effectively applied in various practical scenarios, thus providing more valuable tools and solutions for medical image analysis.

[0099] In addition, this example also compares the convergence speed of each model on the ISIC-2018 and Kvasir-SEG datasets. Figure 8 As shown. It can be observed that in the first 20 epochs, the convergence speed of the MPFC-Net model proposed in this application on the ISIC-2018 dataset is slightly slower than that of the UNeXt model with fewer parameters and lower computational complexity, but on the Kvasir-SEG dataset, the convergence speeds of the two are similar. This means that the MPFC-Net model has a faster convergence speed than other methods and can achieve a reliable performance level within the same training round. Therefore, the MPFC-Net model can save a lot of computing resources and time costs, making the medical image segmentation process more efficient.

[0100] In one embodiment, Fig. 9 As shown, an image segmentation device based on multi-path fusion convolution is provided, including: an image preprocessing module and an image segmentation module, wherein:

[0101] An image preprocessing module is used to acquire medical images, preprocess the medical images, and obtain preprocessing results;

[0102] The image segmentation module is used to input the preprocessing results into the trained image segmentation model based on multi-path fusion convolution to obtain the medical image segmentation results; wherein, the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder. Specifically, the encoder is obtained by replacing the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replacing the second to fourth convolution modules with multi-path fusion convolution modules, and replacing the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder is obtained by replacing all convolution modules in the classic UNet network decoder with multi-path fusion convolution modules; the shallow feature acquisition module is used to extract shallow features in the input image using deep separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and significant features of input features using convolution branches and pooling branches respectively, and fuse the features extracted by the two branches.

[0103] In one embodiment, the image segmentation module is further used to input the preprocessing result into the encoder to obtain five different scale coding features; the fifth scale coding feature is upsampled after feature extraction using a multi-path fusion convolution module, the up-sampling result is fused with the fourth scale coding feature, and the fusion result is input into the first multi-path fusion convolution module of the decoder to obtain the fourth scale decoding feature; the fourth scale decoding feature is upsampled and fused with the third scale coding feature, and the fusion result is input into the second multi-path fusion convolution module of the decoder to obtain the third scale decoding feature; the third scale decoding feature is upsampled and fused with the second scale coding feature, and the fusion result is input into the third multi-path fusion convolution module of the decoder to obtain the second scale decoding feature; the second scale decoding feature is upsampled and fused with the first scale coding feature, and the fusion result is input into the fourth multi-path fusion convolution module of the decoder to obtain the first scale decoding feature; the first scale decoding feature is point convolved to obtain the medical image segmentation result.

[0104] In one of the embodiments, the image segmentation module is further used to input the preprocessing result into the shallow feature acquisition module of the encoder to obtain a first-scale coding feature; downsample the first-scale coding feature using a double-branch downsampling module to obtain a first downsampled feature; input the first downsampled feature into a first multi-path fusion convolution module of the encoder to obtain a second-scale coding feature; downsample the second-scale coding feature using a double-branch downsampling module and input it into a second multi-path fusion convolution module of the encoder to obtain a third-scale coding feature; downsample the third-scale coding feature using a double-branch downsampling module and input it into a third multi-path fusion convolution module of the encoder to obtain a fourth-scale coding feature; downsample the fourth-scale coding feature using a double-branch downsampling module to obtain a fifth-scale coding feature.

[0105] In one of the embodiments, the shallow feature acquisition module includes: a 3×3 convolution layer, a 7×7 deep convolution layer and a point convolution layer; the image segmentation module is also used to input the preprocessing result into the 3×3 convolution layer of the shallow feature acquisition module to obtain a shallow convolution feature map; the shallow convolution feature map is input and output into the 7×7 deep convolution layer of the shallow feature acquisition module, and the obtained deep convolution result is input into the point convolution layer of the shallow feature acquisition module to obtain point convolution features; the shallow convolution feature map and the point convolution feature are fused to obtain the first scale coding feature.

[0106] In one embodiment, the dual-branch downsampling module includes: a convolution branch, a pooling branch and a fusion layer; an image segmentation module, which is also used to input the first-scale decoding feature into the convolution branch to obtain a convolution feature map; input the first-scale decoding feature into the pooling branch to obtain a pooling feature map; splice the convolution feature map and the pooling feature map and input them into the fusion layer for processing through a second 3×3 convolution layer to obtain a first downsampling feature.

[0107] In one of the embodiments, the convolution branch in the image segmentation module includes: a first point convolution layer and a third 3×3 convolution layer; the pooling branch includes a maximum pooling layer.

[0108] In one embodiment, the multi-path fusion convolution module includes: a second point convolution layer, two feature paths, a feature fusion module, a sixth 3×3 convolution layer and a fifth point convolution layer; the first feature path includes a fourth 3×3 convolution layer; the second feature path includes a fifth 3×3 convolution layer; the feature fusion module includes an average pooling layer, a third point convolution layer, a fourth point convolution layer and a SoftMax activation function; the image segmentation module is also used to extract the fifth scale coding feature using the multi-path fusion convolution module and then upsample it, and fuse the upsampled result with the fourth scale coding feature to obtain a fused feature map; the fused feature map is input into the first multi-path fusion convolution module of the decoder, and the fused feature map is processed by the second point convolution layer. Then they are respectively input into the fourth 3×3 convolution layer of the first feature path and the fifth 3×3 convolution layer of the second feature path to obtain the first path feature map and the second path feature map; the first path feature map and the second path feature map are spliced ​​to obtain the spliced ​​feature; the spliced ​​feature is input into the sixth 3×3 convolution layer to obtain the first fused feature map; the spliced ​​feature is input into the feature fusion module, and is processed by the average pooling layer, the third point convolution layer, and the fourth point convolution layer in sequence, and then activated by the SoftMax function to obtain the second fused feature map; the spliced ​​feature, the first fused feature map and the second fused feature map are multiplied and processed by the fifth point convolution layer to obtain the final fused feature; the final fused feature is spliced ​​with the fused feature map to obtain the fourth scale decoding feature.

[0109] For the specific definition of the image segmentation device based on multi-path fusion convolution, please refer to the definition of the image segmentation method based on multi-path fusion convolution above, which will not be repeated here. Each module in the above-mentioned image segmentation device based on multi-path fusion convolution can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0110] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image segmentation method based on multi-path fusion convolution is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0111] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image segmentation method based on multi-path fusion convolution, characterized in that: The method comprises: Acquire a medical image, and preprocess the medical image to obtain a preprocessing result; Inputting the preprocessing result into a trained image segmentation model based on multi-path fusion convolution to obtain a medical image segmentation result; Among them, the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder; the encoder is obtained by replacing the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replacing the second to fourth convolution modules with multi-path fusion convolution modules, and replacing the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder is obtained by replacing all convolution modules in the classic UNet network decoder with the multi-path fusion convolution module; the shallow feature acquisition module is used to extract shallow features in the input image using depth-separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and salient features of input features using convolution branches and pooling branches respectively, and fuse the features extracted by the two branches.

2. The method according to claim 1, characterized in that The preprocessing result is input into the trained image segmentation model based on multi-path fusion convolution to obtain the medical image segmentation result, including: Inputting the preprocessing result into the encoder to obtain five encoding features of different scales; The fifth scale coding feature is extracted by a multi-path fusion convolution module and then up-sampled, the up-sampling result is fused with the fourth scale coding feature, and the fusion result is input into the first multi-path fusion convolution module of the decoder to obtain a fourth scale decoding feature; Upsampling the fourth-scale decoding feature and fusing it with the third-scale encoding feature, and inputting the obtained fusion result into the second multi-path fusion convolution module of the decoder to obtain the third-scale decoding feature; Upsampling the third-scale decoding feature and fusing it with the second-scale encoding feature, and inputting the obtained fusion result into the third multi-path fusion convolution module of the decoder to obtain the second-scale decoding feature; Upsampling the second-scale decoding feature and fusing it with the first-scale encoding feature, and inputting the obtained fusion result into the fourth multi-path fusion convolution module of the decoder to obtain the first-scale decoding feature; The first-scale decoded features are point-convolved to obtain a medical image segmentation result.

3. The method according to claim 1, characterized in that The preprocessing result is input into the encoder to obtain five different scale coding features, including: Inputting the preprocessing result into the shallow feature acquisition module of the encoder to obtain a first-scale coding feature; Downsampling the first scale coding feature using a dual-branch downsampling module to obtain a first downsampling feature; Inputting the first down-sampled features into the first multi-path fusion convolution module of the encoder to obtain a second scale encoded feature; Downsampling the second-scale coding feature using a dual-branch downsampling module and inputting the downsampled feature into the second multi-path fusion convolution module of the encoder to obtain a third-scale coding feature; Downsampling the third scale coding feature by using a dual-branch downsampling module and inputting the downsampled feature into the third multi-path fusion convolution module of the encoder to obtain a fourth scale coding feature; The fourth-scale coding feature is downsampled by using a dual-branch downsampling module to obtain a fifth-scale coding feature.

4. The method according to claim 3, characterized in that The shallow feature acquisition module includes: a 3×3 convolution layer, a 7×7 depth convolution layer and a point convolution layer; Inputting the preprocessing result into the shallow feature acquisition module of the encoder to obtain a first-scale coding feature, including: Inputting the preprocessing result into the 3×3 convolution layer of the shallow feature acquisition module to obtain a shallow convolution feature map; Input the shallow convolution feature map into the 7×7 depth convolution layer of the shallow feature acquisition module, and input the obtained depth convolution result into the point convolution layer of the shallow feature acquisition module to obtain point convolution features; The shallow convolution feature map and the point convolution feature are fused to obtain a first-scale coding feature.

5. The method according to claim 3, characterized in that: The dual-branch downsampling module includes: a convolution branch, a pooling branch and a fusion layer; Downsampling the first scale decoding feature using a dual-branch downsampling module to obtain a first downsampling feature, including: Inputting the first-scale decoded features into the convolution branch to obtain a convolution feature map; Inputting the first-scale decoded features into the pooling branch to obtain a pooling feature map; The convolution feature map and the pooling feature map are concatenated and input into the fusion layer for processing through the second 3×3 convolution layer to obtain the first down-sampling feature.

6. The method according to claim 5, characterized in that The convolution branch includes: a first point convolution layer and a third 3×3 convolution layer; The pooling branch includes a maximum pooling layer.

7. The method according to claim 3, characterized in that The multi-path fusion convolution module includes: a second point convolution layer, two feature paths, a feature fusion module, a sixth 3×3 convolution layer and a fifth point convolution layer; wherein the first feature path includes a fourth 3×3 convolution layer; the second feature path includes a fifth 3×3 convolution layer; the feature fusion module includes an average pooling layer, a third point convolution layer, a fourth point convolution layer and a SoftMax activation function; The fifth scale coding feature is extracted by a multi-path fusion convolution module and then up-sampled, the up-sampling result is fused with the fourth scale coding feature, and the fusion result is input into the first multi-path fusion convolution module of the decoder to obtain the fourth scale decoding feature, including: The fifth scale coding feature is extracted by a multi-path fusion convolution module and then up-sampled, and the up-sampling result is fused with the fourth scale coding feature to obtain a fusion feature map; Input the fused feature map into the first multi-path fusion convolution module of the decoder, and input the fused feature map into the fourth 3×3 convolution layer of the first feature path and the fifth 3×3 convolution layer of the second feature path respectively after being processed by the second point convolution layer to obtain a first path feature map and a second path feature map; splice the first path feature map and the second path feature map to obtain a spliced ​​feature; input the spliced ​​feature into the sixth 3×3 convolution layer to obtain a first fused feature map; input the spliced ​​feature into the feature fusion module, and after being processed by the average pooling layer, the third point convolution layer, and the fourth point convolution layer in sequence, the SoftMax function is used for activation to obtain a second fused feature map; multiply the spliced ​​feature, the first fused feature map, and the second fused feature map, and then process them through the fifth point convolution layer to obtain a final fused feature; After the final fusion feature is concatenated with the fusion feature map, a fourth-scale decoding feature is obtained.

8. An image segmentation device based on multi-path fusion convolution, characterized in that: The device comprises: An image preprocessing module, used to acquire a medical image, preprocess the medical image, and obtain a preprocessing result; An image segmentation module is used to input the preprocessing result into a trained image segmentation model based on multi-path fusion convolution to obtain a medical image segmentation result; wherein the image segmentation model retains the original encoder-decoder architecture and jump connection module in the classic UNet network, and improves the encoder and decoder; the encoder is obtained by replacing the first convolution module in the classic UNet network encoder with a shallow feature acquisition module, replacing the second to fourth convolution modules with multi-path fusion convolution modules, and replacing the downsampling layer after each convolution module with a dual-branch downsampling module; the decoder is obtained by replacing all convolution modules in the classic UNet network decoder with the multi-path fusion convolution module; the shallow feature acquisition module is used to extract shallow features in the input image using depth-separable convolution and residual structure; the multi-path fusion convolution module is used to extract feature information of different scales using multiple feature extraction branches including convolution structure and residual structure, and fuse them; the dual-branch downsampling module is used to extract detail features and significant features of input features using convolution branches and pooling branches respectively, and fuse the features extracted by the two branches.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Deep interpretable skin disease identification method based on small sample learning

    CN119251228A

  • A deep interpretable skin disease recognition method based on small sample learning

    CN119251228B

  • Lesion grading method, device and equipment and storage medium

    CN119693375A

  • Lesion grading method, device, equipment and storage medium

    CN119693375B

  • Lightweight segmentation method and system for medical image

    CN120339306A