Image segmentation method of high-order interaction space U-shaped model based on mixed selectivity

Abstract

The invention provides an image segmentation method for a high-order interaction space U-shaped model based on mixed selectivity, and the method comprises the steps: collecting a plurality of related images as a training image set, and carrying out the preprocessing of the training image set, so as to set the labels of the images; respectively inputting a plurality of features of the image into a high-order interaction space U-shaped model with mixed selectivity, wherein the high-order interaction space U-shaped model comprises an encoder, a decoder and a jump connection part; the second-order, third-order, fourth-order and fifth-order extracted features are fused, and a full connection layer is used to obtain guidance features; the number of input channels of the guidance features is restored to the original number of channels, the input channels are divided into a plurality of full connection layers, then the guidance features and correlation functions are used at the corresponding positions of the full connection layers so that the sum of the weights of the corresponding positions can be 1, multiplication is carried out, and obtained feature maps are added to obtain a mixed high-order interaction feature map. According to the method provided by the invention, higher performance can be obtained by adopting interaction operation of different orders in the same layer of information.

Description

Technical Field

[0001] The present invention relates to a graphics processing method, and more specifically, to an image segmentation method based on a high-order interactive space U-shaped model of mixed selectivity. Background Art

[0002] The skin, the largest organ in the human body, is an important organ for protecting the body's health. The surface of the skin exposed to the environment is susceptible to infections such as ultraviolet rays, sunscreen, and itching. The formation of malignant skin lesions (such as malignant melanoma) is due to the rapid production of melanin in the skin under the irradiation of appropriate ultraviolet rays. If malignant melanoma is not taken seriously in the early stage, the mortality rate of malignant melanoma diagnosed in the late stage within five years is as high as 75%. At the same time, some benign skin lesions such as chloasma, nevus of Ota, Mongolian spots, etc., although they will not pose a fatal threat to human health, will destroy the original beauty of the patient's facial appearance. These benign skin lesions are currently receiving more and more attention, especially for young people who have high requirements for appearance. Therefore, in order to increase the probability of correct identification of skin lesions, the application of computer vision has gradually joined the team of assisting doctors in diagnosis.

[0003] Manual diagnosis, as a traditional diagnostic treatment method, is seriously affected by the professional knowledge level of dermatologists, and the manual diagnosis rate is only between 24% and 77%. Computer-assisted diagnosis can effectively solve this problem, which is particularly important for areas with scarce medical resources. A dermatologist only needs to use an ordinary computer to effectively improve the diagnostic efficiency. Automatic segmentation of medical images is one of the key elements of computer vision-assisted diagnosis. Through computer automatic segmentation, the region of interest (ROI), that is, the area of ​​skin lesions, can be quickly found. Accurate segmentation of lesion areas can provide important support for the accuracy of subsequent diagnosis and classification of lesions and treatment.

[0004] Image segmentation is to divide an image into several regions based on features such as color, grayscale, spatial texture, and geometric shape. These regions are usually consistent or similar in themselves, and show differences in different regions. Image segmentation is an important field of image understanding and processing, and is also a key issue in computer vision. Traditional image segmentation uses mathematical knowledge and digital image processing to segment images. Traditional image segmentation methods have fast segmentation speed and low computational complexity, but in the case of medical image segmentation, which requires high details, the segmentation accuracy cannot meet the requirements. At present, the use of deep learning methods for medical image segmentation has occupied half of the market, and the segmentation ability has even exceeded that of traditional image segmentation. The fully convolutional network (FCN) is a pioneer in image segmentation and the first deep learning network that successfully applied deep learning methods to image segmentation. With the introduction of the concept of fully convolutional networks, more and more research has been carried out based on fully convolutional networks. UNet, as a very outstanding U-shaped network architecture, was proposed in medical image segmentation based on a fully convolutional structure. The jump connections in UNet ensure that the recovered feature map is a fusion of high-level features and low-level features, which also makes the UNet network architecture occupy a dominant position in medical image segmentation.

[0005] Dermoscopy has always been a key technology for traditional manual skin diagnosis. With the latest developments in graphics card functions and deep learning, computer-aided diagnosis has become another key technology. One of the key technologies of computer-aided diagnosis is automatic segmentation. Automatic segmentation is also achieved by extracting and learning the features of the lesion area in the dermoscopic image through the computer, so as to achieve automatic segmentation by computer.

[0006] In recent years, more and more studies have made improvements to automatic segmentation algorithms to improve segmentation performance. People have applied the classic UNet model to skin lesion segmentation. The results show that the traditional UNet segmentation performance is basically better than the accuracy and speed of traditional manual segmentation. The performance of the UNet architecture in the field of medical segmentation has led more and more researchers to develop based on the UNet architecture. Att U-Ne adds an attention gate to the traditional UNet architecture to focus on the lesion area and improve the model's ability to extract lesion area features. TransUNet adds Transformers to the UNet model architecture. TransUNet combines the advantages of Transformers' strong ability to extract global information and convolution's strong ability to extract local features. FAT-Ne uses Transformers encoders in the encoder part of the UNet model architecture. The decoder part uses a feature adaptation module for multi-level feature fusion. ATTENTION SWIN U-NET is an optimization and improvement based on SwinU-Net. The attention mechanism is added to the cascade operation to suppress unimportant information. MALUNet is a lightweight UNet model architecture that uses multi-level and multi-scale information fusion to solve the problems of segmentation speed and large memory usage. Transnorm also uses a UNet-like architecture that uses both Transformer and CNN, and uses an attention mechanism in the jump connection part to adaptively calibrate the expressive power of features. The C2SDG model uses a feature contrast enhancement method. C2SDG extracts low-level features and style enhancement corresponding features for each input image and uses them for contrast training. Recently, a universal segmentation model has attracted widespread attention. The arbitrary segmentation model (SAM) has good results in both natural and medical scenes. The medical universal segmentation model (MSA) is based on SAM, and the Adapter module is added for fine-tuning to make it more suitable for the field of medical image segmentation.

[0007] Computer-assisted diagnosis of skin diseases by clinical dermatologists is an important auxiliary means. Computer-assisted technology mainly uses deep neural networks. Recently, the introduction of high-order spatial interaction operations in deep neural networks has attracted widespread attention. It has the advantages of both convolution and transformer, as well as high efficiency, scalability and translation variation. However, in high-order interaction operations, the selection of interaction order requires tedious manual selection of the appropriate interaction order.

[0008] At present, automatic segmentation mainly uses deep learning algorithms. In the past few years, the use of deep learning for medical image segmentation has become increasingly popular. Researchers have developed a variety of deep learning algorithms to adapt to different diseases and obtain higher accuracy or lower complexity. Many deep learning-based methods currently outperform traditional manual segmentation methods. For retinal blood image segmentation, a full residual autoencoding batch normalization network PixelBNN composed of UNet and PixelCNN is proposed, and skip connections are used in line with FCN. For automatic segmentation of the heart region, LU-Net is proposed. This is an improved UNet model architecture that mainly solves the problem of low accuracy of the traditional UNet model in ventricle segmentation.

[0009] Recently, a high-order interaction module gnconv was proposed that combines the advantages of both VIT and convolution. However, the proposed HorNet model simply connects the orders [2, 3, 4, 5] in sequence, such as Figure 1 Shown is the prior art HorNet model architecture. Summary of the invention

[0010] It is an object of at least the preferred embodiments of the present invention to address some of the above disadvantages.An additional or alternative object is to at least provide the public with a useful choice.

[0011] The present invention proposes an image segmentation method based on a high-order interactive space U-shaped model of mixed selectivity, and the image segmentation method comprises:

[0012] S1: Collect several related images as training image sets and preprocess them to set image labels;

[0013] S2: inputting the multiple features of the image into the hybrid selective high-order interactive space U-shaped model respectively, the high-order interactive space U-shaped model includes an encoder, a decoder and a jump connection part, the encoder includes a five-layer structure, the first layer structure adopts convolution, the second layer structure, the third layer structure, the fourth layer structure and the fifth layer structure all adopt a hybrid selective high-order interactive module, and the four different operation channels of the hybrid selective high-order interactive module respectively adopt 2nd order, 3rd order, 4th order and 5th order feature extraction operations;

[0014] S3: Fusing the features extracted from the 2nd, 3rd, 4th and 5th orders, using a fully connected layer to obtain guidance features, wherein the fully connected layer consists of a 1×1 standard convolution, a BatchNorm operation and a Relu activation function;

[0015] S4: The number of input channels of the guiding feature is restored to the four original channels and divided into four fully connected layers. Then, the guiding feature and related functions are used at the corresponding positions of the fully connected layers so that the sum of the weights of each corresponding position is 1. The weight values ​​of each order of each channel are multiplied by the interaction operation, and the obtained feature maps are added to obtain a mixed high-order interaction feature map, thereby obtaining the final image segmentation result.

[0016] On the one hand, the features are element-wise multiplied once with the depthwise convolution in 2nd and 3rd order interactions, while the features are element-wise multiplied once with the global-filter in 4th and 5th order interactions.

[0017] On the one hand, the high-level semantic information extracted in the fifth layer of the encoder and decoder is added to the ConvMixer module, and the channel attention module and spatial attention module with multi-level and multi-scale information fusion are adopted in the skip connection part.

[0018] In one aspect, before the extraction operation in S2, the input features are input to IEAB to map the input to a higher dimensional space, where

[0019] X'=IEAB(X) (1)

[0020] x 1 ,x 2 ,x 3 ,x 4 =Order 2,3,4,5 (X',X',X',X') (2)

[0021] Among them, X is the input feature, IEAB(X) is to input feature X into the IEAB module, Order 2,3,4,5 To perform 2nd, 3rd, 4th, and 5th order interactive operations on the input features respectively.

[0022] On the one hand, the fusion operation is represented as,

[0023] P = x 1 +x 2 +x 3 +x 4 (3)

[0024] P′=GAP(P) (4)

[0025] F = Relu(BN(Conv(P′))) (5)

[0026] Among them, GAP is global average pooling, Relu is the Relu activation function, BN is the BatchNorm operation, Conv is a 1×1 convolution operation, and the number of output channels after convolution is 32.

[0027] On the one hand, a softmax function is used at the corresponding position of the fully connected layer, and the weights represent weight values ​​of each order of each channel number obtained by the guidance feature.

[0028] On the one hand, in S4,

[0029] f = Reshape(Conv(F)) (6),

[0030] f′=softmax(f,dim=1) (7),

[0031] w 1 ,w 2 ,λ 3 ,w 4 =Chunk4(f′) (8),

[0032] Out=w 1 ·x 1 +w 2 ·x 2 +w 3 ·x 3 +w 4 ·x 4 (9)

[0033] Among them, Conv is a 1×1 convolution operation, and the number of output channels after convolution is 4 times the original number of channels. Reshape is to split the 4 times number of channels into 4 multiples of the original number of channels. Softmax is to calculate the weights channel by channel through the softmax function. Chunk4 is to divide the input f' into 4 weight matrices.

[0034] On the one hand, for each interaction in the mixed-selective high-order interaction module, the extracted informative features are subjected to an element-wise multiplication operation with a depth-wise convolution or global-filter.

[0035] On the one hand, the high-order interactive spatial U-shaped model based on mixed selectivity includes a ConvMixer block, which consists of depthwise convolution and 1×1 ordinary convolution, and each convolution is accompanied by a batchnormalization and GELU activation function.

[0036] In summary, the method provided by the present invention proposes a hybrid selective high-order interactive spatial U-shaped model (HSH-UNet), which can adaptively select different orders of spatial interactive operations in different channels of each layer of feature information. The HSH-UNet proposed by the present invention can better extract features. In particular, in terms of model generalization ability, using different orders of interactive operations in the same layer of information can achieve more powerful performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Preferred forms of embodiment according to the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0038] Figure 1 The prior art HorNet model architecture is shown, where only a single order is used for interaction at each layer, and the order of the HorNet interaction order is set to 2, 3, 4, and 5;

[0039] Figure 2 The HSH-UNet model architecture according to a preferred embodiment of the present invention is shown;

[0040] Figure 3 A mixed selective high-order interaction module (HSHB) in the HSH-UNet model according to a preferred embodiment of the present invention is shown;

[0041] Figure 4 The structures of g2conv, g3conv, g4conv and g5conv according to a preferred embodiment of the present invention are shown;

[0042] Figure 5 A single high-order interaction according to a preferred embodiment of the present invention is shown, g1conv is a first-order interaction, g3conv is a third-order interaction, wherein the key structure of Horblock is gnconv;

[0043] Figure 6 The structure of the IEAB according to a preferred embodiment of the present invention is shown;

[0044] Figure 7 The structure of the ConvMixer block according to a preferred embodiment of the present invention is shown;

[0045] Figure 8 is a visual representation of the segmentation results of the most popular models and the models in ISIC2017 and ISIC2018 according to a preferred embodiment of the present invention;

[0046] Fig. 9 For PH 2 A visual representation of the dataset and the segmentation results of external validation on the data according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0048] An embodiment of the present invention proposes an image processing method, which is particularly suitable for an image segmentation method of medical images, which is specifically a hybrid selective high-order interactive U-shaped module HSH-UNet to solve the problem of manually selecting the interaction order. Specifically, the present application proposes a hybrid selective high-order interactive module HSHB embedded in a U-shaped model. HSHB can adaptively select the appropriate order for the interactive operation of each channel according to the calculated guidance features. The hybrid selective high-order interactive spatial U-shaped model (HSH-UNet) proposed in this article is used to solve the problem of requiring cumbersome manual selection of orders and fixed interaction orders at each level. HSH-UNet adaptively selects gnconv of different orders for different channels at the same information level to extract information, and finally integrates the information obtained from different orders to form the final feature map. Multi-order spatial interaction can improve the model's ability to extract features and enhance the model's generalization ability. At the same time, combined with the UNet architecture, a channel attention module (CAB) and a spatial attention module (SAB) with multi-level and multi-scale information fusion are introduced. The introduction of CAB and SAB modules can improve the model's ability to segment targets of different sizes. Therefore, the present invention proposes a hybrid selective high-order interaction module (HSHB) to solve the key problem of manually selecting the interaction order by guiding the feature F to adaptively select different order spatial interaction operations in different channels at the same information level.

[0049] First, several related images are collected as a training image set, and preprocessed to set labels for the images and extract the region of interest containing the target area. The image can be a medical image, and several related images are collected as a training image set, and the region of interest is annotated by a professional clinician to obtain a labeled image, wherein the lesion area is selected, and the original image and the labeled image with the lesion area marked are used for algorithm model training. The image is implemented with a hybrid selective high-order interactive spatial U-shaped model (HSH-UNet) network architecture as proposed in one embodiment of the present invention, such as Figure 2As shown. HSH-UNet adopts a U-shaped architecture, which consists of at least an encoder, a decoder and a jump connection part. The encoder adopts a classic 5-layer structure with the number of channels [32, 64, 128, 256, 512]. The first layer adopts a 3×3 standard convolution, and the second to fifth layers adopt a mixed selective high-order interaction module (HSHB). HSHB adopts a mixed selection order of [2, 3, 4, 5]. HSHB will be introduced in detail later. The decoder is symmetrical with the encoder to maintain the best utilization efficiency of information. The high-level semantic information extracted in the fifth layer of the encoder and decoder can be added to the ConvMixer module to further improve the generalization ability of the model. The jump connection part adopts a channel attention module (CAB) and a spatial attention module (SAB) with multi-level and multi-scale information fusion. CAB and SAB can accelerate the convergence speed of the model and improve the segmentation ability of lesions of different sizes.

[0050] For HSHB, there are 4 different mixing orders, namely 2nd, 3rd, 4th, and 5th. HSHB can automatically select different order operations. Three operators (Split, Fuse, Select) are used to implement mixing selective operations. Figure 3 Before explaining the three operators in detail, a single high-order interaction module gnconv is introduced.

[0051] The HorNet model composed of high-order interaction modules gnconv performs very well in natural scenes. The Horblock module in the HorNet model adopts a similar Transformers module composition form, replacing the self-attention layer with a high-order interaction module gnconv. gnconv is mainly composed of deep convolution (DW) or global-filter (GLF), standard convolution, linear projection and element multiplication. gnconv can realize any order of interaction above the second order, such as Figure 4 g2conv, g3conv, g4conv, g5conv are shown. Figure 5 The figure shows the Horblock module and the high-order interaction module gnconv. The self-attention layer in traditional Transformers can only perform second-order interactions, and ordinary convolutions cannot perform multi-order interactions.

[0052] Split Operation

[0053] The Split operation is the first step of HSHB. Specifically, assuming that the input feature X∈R H×W×C, where X represents the input image, R represents the features of the input image, H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the input image. In a preferred embodiment of the present invention, the input features are respectively input into four different operation channels in HSHB, and 2nd-order, 3rd-order, 4th-order, and 5th-order feature extraction operations are performed separately. Of course, other suitable numbers of channels can also be used. It should be noted that before performing feature extraction operations of different orders, in order to improve the generalization ability of the model, the input features are input into the inverted external attention block (IEAB). IEAB can also map the input to a higher-dimensional space and can comprehensively memorize the overall feature information. The composition of IEAB is as follows Figure 6 The Split operation can be specifically expressed by the following formula:

[0054] X'=IEAB(X) (1)

[0055] x 1 ,x 2 ,x 3 ,x 4 =Order 2,3,4,5 (X′,X′,X′,X′) (2)

[0056] Where X is the input feature, IEAB is the feature input into the IEAB module, and Order 2,3,4,5 To perform 2nd, 3rd, 4th, and 5th order interactive operations on the input respectively.

[0057] Fuse Operation

[0058] In order to be able to adaptively select operations of different orders, it is necessary to fuse the features obtained by each independent operation of different orders. Then, global average pooling (GAP) is used in each channel to obtain global information features. In order to achieve accurate and adaptive selection of guidance for selecting operations of different orders, a fully connected layer is used. The fully connected layer consists of a 1×1 standard convolution, a BatchNorm operation, and a Relu activation function. It should be noted that in order to improve efficiency, a dimensionality reduction operation is performed. The number of channels after dimensionality reduction is 32. The Fuse operation can be specifically expressed by the following formula:

[0059] P = x 1 +x 2 +x 3 +x 4 (3)

[0060] P′=GAP(P) (4)

[0061] F = Relu(BN(Conv(P′))) (5)

[0062] In the formula, GAP is the global average pooling, Relu is the Relu activation function, BN is the BatchNorm operation, Conv is the 1×1 convolution operation, and the number of output channels after convolution is 32.

[0063] Select Operation

[0064] F is used as a guiding feature to guide the self-selection operation. The number of input channels of F is restored to 4 times the original number of channels, and divided into 4 fully connected layers. The softmax function is then used in the corresponding positions of the four fully connected layers to ensure that the sum of the weights of each corresponding position is 1. The weight is the weight value of each order of each channel number obtained under the prompt of the F guiding feature. Each weight is then multiplied by the second-order, third-order, fourth-order, and fifth-order interaction operations that were initially performed separately. The multiplication results in the result of each individual high-order interaction operation obtained through adaptation. Finally, the feature maps obtained by the individual high-order interaction operations are added to obtain a mixed high-order interaction feature map. The Select operation can be specifically expressed by the following formula:

[0065] f = Reshape(Conv(F)) (6)

[0066] f′=Softmax(f,dim=1) (7)

[0067] w 1 ,w 2 ,w 3 ,w 4 =Chunk4(f′) (8)

[0068] Out=w 1 ·x 1 +w 2 ·x 2 +w 3 ·x 3 +w 4 ·x 4 (9)

[0069] Where Conv is a 1×1 convolution operation, the number of output channels after convolution is 4 times the original number of channels, Reshape is to split the 4 times number of channels into 4 original multiples of the number of channels, Softmax is to calculate the weights channel by channel through the softmax function, and Chunk4 is to divide the input f' into 4 weight matrices.

[0070] As detailed above, for an input image feature X∈R H×W×C , after a 1X1 convolution and a reshaping operation, the feature size forms a two-dimensional X∈R C×HWAfter performing the integrated convolution operation on the two-dimensional X using the first storage unit, a set of X∈R is obtained. 4C×HW Then the second storage unit is used to adjust the characteristic shape to restore it to the two-dimensional X∈R C ×HW Finally, after a 1X1 convolution, the residual operation is used to add the initial information for output.

[0071] The hybrid selective high-order interaction module (HSHB) is the key module of HSH-UNet. The reasons why HSHB significantly improves the model performance can be summarized as follows. First, Figure 5 As shown in the figure, g1conv and g3conv are 1st order interactions and 3rd order interactions respectively. After each interaction, the extracted information features will be element-wise multiplied with the deep convolution (DW) or global filter (GLF). However, too many DW and GLF operations will cause the extracted feature information to gradually lose its original information, and the model fitting speed will be greatly reduced. However, too few DW and GLF operations will make the model have poor generalization. Therefore, adaptively selecting a suitable order for each input channel to interact through the F guidance feature can effectively improve the performance of the model. The formation of the guidance feature F is mainly composed of Figure 3 The Fuse operation of the HSHB module shown in Figure 3 Specifically, the feature information after the separate 2nd, 3rd, 4th, and 5th order interactive operations is added and fused to form a new feature information X∈R H×W×C This feature information is subjected to a global average pooling (GAP) to obtain a global information feature, which is used as the initial feature of the guiding feature F. Then it is sorted and output after a two-dimensional 1X1 convolution, batch normalization, Relu activation function and another two-dimensional 1X1 convolution. The above operation improves the fitting speed during model training. The final output is four one-dimensional guiding features F (such as Figure 3 The guiding feature F has weight information for each image channel number. Then, the gate selection is performed with the results of the 2nd, 3rd, 4th, and 5th order interactions performed separately, and the weight of the guiding feature F determines which order of interaction is selected for each channel number. In a preferred embodiment of the present invention, the selection is made based on the feature information of the image itself and the region of interest, that is, which order is more capable of extracting the region of interest during the extraction process, and the weight will be larger accordingly.

[0072] Furthermore, if Figure 3 As shown in the figure, the HSHB module can perform operations of different orders on different input channels. The 2nd and 3rd orders use DW, while the 4th and 5th orders use GLF. Finally, the results of the interaction of each order are fused at the output. Figure 2 As shown in the figure, there are 4 HSHB modules in the encoding part and 4 HSHB modules in the decoding part. The input of each module is different. The higher the stage, the more the extracted features are in line with the characteristics and abstraction of the region of interest. The lower the stage, the more background information the extracted features have. Compared with the traditional fixed order, the HSHB module also has the function of multi-order interactive fusion, and the extracted feature information has richer feature information.

[0073] The method of the present invention also adopts a channel attention module (CAB) and a spatial attention module (SAB), thereby improving the model convergence speed and the sensitivity of lesions of different sizes. The channel attention module (CAB) can connect the features of different stages in series and better fuse the information of each stage. CAB is mainly composed of global average pooling, cascade operation, one-dimensional convolution operation, fully connected layer and Sigmoid activation function. The spatial attention module (SAB) mainly generates attention maps at each stage and inputs them into the CAB module for fusion. SAB is mainly composed of average pooling, maximum pooling and shared extended convolution.

[0074] The existing ConvMixer block has a large kernel that can help the model improve its ability to extract global information. The ConvMixer block is mainly composed of deep convolution and 1×1 normal convolution. At the same time, each convolution is accompanied by a BatchNorm operation and a Gelu activation function. Figure 7 As shown in the figure, in the composition structure of the ConvMixer block, the input features are first input into the deep convolution, and then output through a 1×1 ordinary convolution operation. The module is simple in composition, but can effectively improve the model's ability to integrate overall information. In a preferred embodiment of the present invention, the ConvMixer block is combined with the last layer of the model proposed by the present invention, and the output of the HSHB module in stage 5 is then input into the ConvMixer module.

[0075] Experimental verification

[0076] To verify the performance of the proposed model, we used three public dermoscopy image datasets (e.g., ISIC2017, ISIC2018, and PH 2 The ISIC2017 and ISIC2018 datasets are two large-scale datasets. 2 The dataset and the dataset of the present invention are small datasets. 2 The dataset of the present invention is used as an external validation dataset to verify the effectiveness of various generation performances of the model proposed in the present invention.

[0077] Example 1

[0078] First, PH2 and the applicant's datasets are used as external validation datasets to perform diversified validation of the generalization ability of the method proposed in a preferred embodiment of the present invention. The external validation is to directly use the weight information obtained through ISIC2017 training to predict two small datasets and compare them with several currently optimal models.

[0079] 2000 images were obtained from the ISIC2017 dataset, as well as dermoscopic images with segmentation mask labels. The initial image size was 576×767 pixels. According to the TransNorm setting, 1250 images were used as training sets, 150 images as validation sets, and 600 images as test sets. All images were resized to 256×256 pixels.

[0080] 2594 images were obtained from the ISIC2018 dataset, as well as dermoscopic images with segmentation mask labels. The initial image size was 2016×3024 pixels. According to the TransNorm settings, 1815 images were used for training, 259 images were used for validation, and 520 images were used for testing. All images were resized to 256×256 pixels.

[0081] From PH 2 200 images were obtained in the dataset, as well as dermoscopic images with segmentation mask labels. The initial image size is 768×560 pixels. 2 The dataset is used for external validation, and all 200 images are used for testing. All images are resized to 256×256 pixels.

[0082] The inventor's own dataset is data from 39 skin lesions obtained from Ruijin Hospital affiliated to Shanghai Jiao Tong University School of Medicine. The skin color is mainly yellow skin of Asians. Under the same conditions, dermoscopic images were obtained by the PhotoMaxPro Analyzer system with a magnification of 30. The segmented mask labels were annotated simultaneously by 3 dermatologists with rich clinical knowledge. If there was an inconsistent annotation, the data was discarded. The inventor's own dataset was used as external validation, and all data were used as tests. The initial image size was 5184×3456 pixels. All images were resized to 256×256 pixels.

[0083] The experiments were all implemented based on Python 3.8 and Pytorch1.12.0. The experiments were all implemented on a single NVIDIA GeForce RTX 4080Laptop GPU with 12GB of memory. For the training data, data augmentation operations were used to improve data diversity, including horizontal flip, vertical flip, and random rotation operations. The input model training image size was unified to 256×256 pixels. The BceDice loss function was used. The training epoch was set to 200, the batch size was 8, and AdamW was used as the optimizer. The initial learning rate size was set to 0.001, the cosine annealing learning rate scheduler was used, and the minimum learning rate was set to 0.00001.

[0084] In the field of medical image segmentation, similarity coefficient (DSC), accuracy (ACC), sensitivity (SE), and specificity (SP) are the most commonly used evaluation criteria for evaluating segmentation performance. Similarity coefficient (DSC) is mainly used to measure the similarity between masks and the correct answer (ground truth), which is particularly important among the four evaluation indicators. Accuracy (ACC) is mainly used to measure the proportion of correct classification. Sensitivity (SE) is mainly used to measure the percentage of true positives (TP) in true positives (TP) and false negatives (FN). Specificity (Spe) is mainly used to measure the percentage of false positives (FP) in true negatives (TN) and false positives (FP).

[0085]

[0086]

[0087]

[0088]

[0089] Where TP represents true positive, FP represents false positive, FN represents false negative, TN represents true negative, specificity represents specificity, and sensitivity / Recall represents sensitivity / recall.

[0090]

[0091] Table 1: Performance comparison on the ISIC2017 dataset

[0092]

[0093] Table 2: Performance comparison on the ISIC2018 dataset

[0094]

[0095] Table 3: PH2 Comparison of the performance of several currently better models with the external validation results of the model of the present invention

[0096]

[0097] Table 4: Comparison of the performance of several current better models on the present invention's own dataset and the external validation results of the present invention's model

[0098] Comparison results

[0099] To prove the effectiveness of the model, the experimental results are compared with several most popular medical image segmentation models and general models, including U-Net, Att U-Net, TransUNet, FAT-Net, ATTENTION SWIN U-NET, MALUNet, TransNorm, HorNet, MSA, and C2SDG.

[0100] Table 1 and Table 2 are the comparative experimental results of the method proposed in this paper and the most popular methods on the ISIC2017 and ISIC2018 datasets, respectively. It can be seen from the table that the DSC indicators of the model proposed in this invention are better than the most popular models. In terms of DSC value, the model of the present invention is 9.49% and 5.14% higher than the UNet model. Compared with the recently popular medical general model MSA, the DSC values ​​are improved by 1.34% and 2.3%, respectively. Compared with the model HorNet that only uses one high-order interaction operation in each layer, the model of the present invention also has better performance. In other evaluation indicators, the model of the present invention is also better than the currently popular models in multiple indicators. Some experimental results are visualized. As shown Figure 8 As shown in FIG. 1 , it is a visualization result diagram of experiments on ISIC2017 and ISIC2018 datasets. It can be seen that the model of the present invention has a more accurate extraction of lesion features. Specifically, the model of the present invention has a more sensitive recognition of fuzzy boundaries.

[0101] In particular, the model of the present invention has an even more obvious advantage in generalization ability. 2 The data set of the present invention was externally verified. Tables 3 and 4 are the external verification experimental results of the method proposed in this paper and the models that performed well in the internal data set. Compared with the 6 models that performed well in the internal data set, the model of the present invention still has the best performance. As shown in Table 3, the DSC value of the model of a preferred embodiment of the present application is 1.34 PH. 2The DSC values ​​of the proposed method are improved by 3.61%, 3.76%, 2.74%, 1.3%, and 3.32% on the datasets. The DSC values ​​of the proposed method are improved by 3.18%, 4.54%, 2.76%, 1.69%, and 0.52% on the datasets. The DSC values ​​of the proposed method are improved by 4.07%, 4.24%, 3.06%, 1.43%, 3.73%, and 2.17% on the datasets. The proposed method is better than the five most popular models in multiple indicators. Fig. 9 A visual representation of the results of the external validation plot is shown. Figure 8 It can be seen intuitively that the performance of each model in the external verification is not as good as the performance results of the internal data set. However, the model proposed by the present invention has the best performance in recognition accuracy, and the recognition of contour shape and fuzzy boundary is most consistent with the true value.

[0102] Ablation experiment

[0103] In order to verify the influence of different numbers of mixing orders of the mixed selective high-order interaction module (HSHB), an ablation experiment was conducted. The specific operation of the experiment is that different mixing orders are used for HSHB. As shown in Table 5, the cases where HSHB is [2,3] order, [4,5] order, [2,3,4] order, [3,4,5] order, and [2,3,4,5] order are analyzed respectively. It can be seen that the DSC value of mixing [4,5] order is better than that of mixing [2,3] order and mixing [2,3,4] order. This shows that the performance of HSHB mixing multiple high-orders can be significantly improved. The DSC value of mixing [3,4,5] order is better than that of mixing [4,5] order, and the performance of mixing [2,3,4,5] order is better than that of mixing [3,4,5] order. This shows that when mixing the same high-order interactions, increasing the number of mixing can improve the performance. However, it should be noted that mixing more orders and higher orders will also increase the computational complexity. Therefore, the present invention selects the mixing order [2, 3, 4, 5] as the main mixing selection order of HSHB.

[0104]

[0105] Table 5: Ablation experiments performed on different mixing orders of HSHB modules

[0106]

[0107] Table 6: Impact of ablation experiments on the HSHB module on the UNet model architecture

[0108] In order to verify the influence of the high-order interaction module (HSHB) of hybrid selectivity on the UNet model architecture, an ablation experiment was conducted. The specific operation of the experiment is to replace the encoder, decoder and fixed-order operations in both the encoder and decoder of the baseline model with HSHB. In order to better highlight the effectiveness of the order of hybrid selectivity, the baseline model sets the encoder and decoder of UNet according to the settings of HorNet. Specifically, the interaction orders of the UNet stages 2-5 of the baseline model are set to 2, 3, 4, and 5 respectively. Each level of the UNet of the baseline model has only one high-order interaction operation. The jump connection adopts multi-level and multi-scale information fusion, and a ConvMixer block is added to the last layer. Table 6 shows the experimental results. It can be seen that the performance of replacing the encoder and decoder with HSHB has been significantly improved. The DSC values ​​increased by 1.13% and 1.5% respectively. When the encoder and decoder are replaced with HSHB respectively, the performance is further improved. The DSC value is 1.98% higher than the baseline model. This ablation experiment not only proves the impact of HSHB on the UNet model architecture, but also confirms that mixing different order performance results has better performance than a single high-order interaction.

[0109] Although high-order interactive operations have been proposed in the HorNet of the prior art and have achieved excellent performance in natural scenes. However, the proposed HorNet only uses a single high-order interactive operation to extract information at each information layer. In the field of medical image segmentation, it is necessary to extract fine detail features in a refined manner. The use of mixed selective high-order interactive operations is the key to solving this problem. The HSHB proposed in the present invention can adaptively select operations of different orders under the guidance feature F. For the selection of the order, the traditional situation requires manual adjustment of the interaction order experimental verification to approximate an optimal interaction order. The adaptive selection of key operations of different orders by HSHB can get rid of this cumbersome process. It can be seen from experiments that the mixed selective selection of operations of different orders at each information layer has a significant performance effect on improving the generalization ability. By using PH 2 The results of the data set and the inventor's own data set have confirmed this. This is due to the fact that HSHB can extract features from different levels, which is particularly important for medical image segmentation.

[0110] The hybrid selective high-order interaction module (HSHB) proposed in the present invention not only solves the key problem of manually selecting the interaction order, but also solves the problem of fixed interaction order at each level. It is achieved by using hybrid order at each level. In addition, the proposed hybrid selective high-order interaction model is combined with the UNet architecture to form the HSH-UNet model. The hybrid selective high-order interaction module successfully combines the channel attention module (CAB) and the spatial attention module (SAB) of multi-level and multi-scale information fusion.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The word "comprising" used in this specification means "at least partially comprising". When interpreting each statement in this specification containing the word "including", there may also be features other than or beginning with this word. Related terms such as "comprising" and "including" should be interpreted in the same way.

[0113] As used herein, the term "and / or" means "and" or "or" or both.

[0114] In the description of this specification, reference may be made to subject matter that is not within the scope of the appended claims. Such subject matter should be easily recognized by those skilled in the art and can assist in putting the invention as defined in the appended claims into practice.

[0115] Although the present invention is generally as defined above, it will be appreciated by those skilled in the art that the present invention is not limited thereto, and the present invention also includes the following examples to give exemplary embodiments. The foregoing description of the present invention includes its preferred forms. It can be modified without departing from the scope of the present invention. For those skilled in the art to which the present invention belongs, many changes in structure of the present invention and widely different embodiments and applications of the present invention will be apparent without departing from the scope of the present invention defined by the appended claims. The disclosure and description herein are purely illustrative and are not intended to be limiting in any sense. When referring to specific integers with known equivalents in the field related to the present invention in this article, these known equivalents are deemed to be combined herein, as if set forth separately.

Claims

1. An image segmentation method based on a high-order interactive space U-shaped model with mixed selectivity, characterized in that: The image segmentation method comprises: S1: Collect several related images as training image sets and preprocess them to set image labels; S2: inputting the multiple features of the image into the hybrid selective high-order interactive space U-shaped model respectively, the high-order interactive space U-shaped model includes an encoder, a decoder and a jump connection part, the encoder includes a five-layer structure, the first layer structure adopts convolution, the second layer structure, the third layer structure, the fourth layer structure and the fifth layer structure all adopt a hybrid selective high-order interactive module, and the four different operation channels of the hybrid selective high-order interactive module respectively adopt 2nd order, 3rd order, 4th order and 5th order feature extraction operations; S3: Fusing the features extracted from the 2nd, 3rd, 4th and 5th orders, using a fully connected layer to obtain guidance features, wherein the fully connected layer consists of a 1×1 standard convolution, a BatchNorm operation and a Relu activation function; S4: The number of input channels of the guiding feature is restored to the original number of channels and divided into multiple fully connected layers, and then the guiding feature and related functions are used at the corresponding positions of the fully connected layers so that the sum of the weights of each corresponding position is 1, and the weight values ​​of each order of each channel are multiplied by the interaction operation, and the obtained feature maps are added to obtain a mixed high-order interaction feature map, thereby obtaining the final image segmentation result.

2. The image segmentation method according to claim 1, characterized in that: In the 2nd and 3rd order interaction operations, the features are element-wise multiplied with the depth convolution once, while in the 4th and 5th order interaction operations, the features are element-wise multiplied with the global filter once.

3. The image segmentation method according to claim 1, characterized in that: The high-level semantic information extracted in the fifth layer of the encoder and decoder is added to the ConvMixer module, and the channel attention module and spatial attention module with multi-level and multi-scale information fusion are adopted in the skip connection part.

4. The image segmentation method according to claim 1, characterized in that: Before the extraction operation in S2, the input features are fed into IEAB to map the input to a higher dimensional space, where X′=IEAB(X) (1) x1,x2,x3,x4=Order 2，3，4，5 (X′,X’,X′,X′) (2) Among them, X is the input feature, IEAB(X) is to input feature X into the IEAB module, Order 2，3，4，5 To perform 2nd, 3rd, 4th, and 5th order interactive operations on the input features respectively.

5. The image segmentation method according to claim 4, characterized in that: The fusion operation is expressed as, P=x1+x2+x3+x4 (3) P′=GAP(P) (4) F = Relu(BN(Conv(P′))) (5) Among them, GAP is global average pooling, Relu is the Relu activation function, BN is the BatchNorm operation, Conv is a 1×1 convolution operation, and the number of output channels after convolution is 32.

6. The image segmentation method according to claim 1, characterized in that: The softmax function is used at the corresponding position of the fully connected layer, and the weight represents the weight value of each order of each channel number obtained by the guidance feature.

7. The image segmentation method according to claim 6, characterized in that: In S4, f = Reshape(Conv(F)) (6), f′=Softmax(f, dim=1) (7), w1, w2, w3, w4=Chunk4(f′) (8), Out=w1·x1+w2·x2+w3·x3+w4·x4 (9), Among them, Conv is a 1×1 convolution operation, and the number of output channels after convolution is 4 times the original number of channels. Reshape is to split the 4 times number of channels into 4 multiples of the original number of channels. Softmax is to calculate the weights channel by channel through the softmax function. Chunk4 is to divide the input f' into 4 weight matrices.

8. The image segmentation method according to claim 1, characterized in that: For each interaction in the mixed-selective high-order interaction module, the extracted informative features are subjected to an element-wise multiplication operation with a depth-wise convolution or global-filter.

9. The image segmentation method according to claim 1, characterized in that: The high-order interactive spatial U-shaped model based on mixed selectivity includes a ConvMixer block composed of depthwise convolution and 1×1 ordinary convolution, and each convolution is accompanied by a Batch normalization and GELU activation function.

Images