Image semantic segmentation method based on snakelike convolution

Abstract

The invention discloses an image semantic segmentation method based on snakelike convolution, and relates to the technical field of artificial intelligence image processing and computer vision. According to the invention, a new fast, efficient and accurate image segmentation model DSM-Net is obtained through the design of the method; the DSM-Net effectively performs feature extraction and up-sampling by using a symmetric coding-decoding architecture, and end-to-end pixel level prediction is realized; wherein the encoding stage is combined with a snakelike convolution (EncoderDS), and the position of a convolution kernel can be linearly adjusted to fully learn object features; fusion pooling is combined with convolution and pooling to efficiently transmit effective features extracted by an encoder; the mixed attention module is used for modeling a space and a channel, so that the network pays attention to different local structure features and realizes efficient feature fitting capability; reasonable construction of the overall architecture greatly reduces training cost, and reasonable network depth and internal module design solve the problem of low segmentation precision caused by too large difference between morphological distribution and size of segmentation targets.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing and computer vision technology, and in particular to an image semantic segmentation method based on snake convolution. Background Art

[0002] Image processing is a collection of techniques and methods for operating images to enhance their visibility or extract useful information. It has a wide range of applications, including medical imaging, remote sensing, computer vision, photography, entertainment, etc. Its process usually includes image acquisition, preprocessing, feature extraction, representation and description, recognition and classification, and post-processing. Then, a specific task is completed through the designed model structure.

[0003] Semantic segmentation is a technique in image processing and computer vision that aims to divide an image into several regions and assign a semantic label to each pixel to indicate the object category to which the pixel belongs. It has a wide range of applications, including autonomous driving, medical image analysis, remote sensing image processing, and intelligent monitoring. The process usually includes steps such as image preprocessing, feature extraction, classification, and labeling. Using deep learning to train the model can extract multi-scale features while maintaining high resolution, achieving fine pixel-level classification. Semantic segmentation technology plays an important role in applications such as road scene understanding, tumor detection, and object classification.

[0004] In recent years, the rapid development of deep learning has demonstrated powerful performance in the field of image segmentation. With automatic feature learning, convenient operation and high performance, deep learning models can achieve high-precision segmentation while taking into account time and space efficiency. The classic fully convolutional network (FCN) was proposed by Long et al. in 2015. Its main feature is to accept input images of any size and generate output images of the same size, assigning semantic labels to each pixel, thereby achieving end-to-end pixel-level semantic segmentation. However, due to multiple pooling operations, FCN loses its focus on image details. Subsequently, in order to solve the problem of precision loss, U-Net was proposed.

[0005] Because its symmetrical encoder-decoder structure provides the original feature connection path for each layer of decoder, the overall network fully retains the original detail information and obtains more accurate segmentation results. However, the U-Net network still has shortcomings. Due to its own structural limitations, it has poor performance for special scenes or special tasks. For this reason, TransUNet is designed by combining the global feature fitting ability of the Transformer structure with U-Net. The ability of convolutional neural network to efficiently extract local features is combined with the global attention perception ability of Transformer to achieve more accurate segmentation results. For different scenes, using different convolution kernels to adapt to the distribution patterns of different objects can also effectively improve the segmentation effect. For example, CE-Net combines hole convolution as a context feature extraction module, aiming to fully pay attention to the context feature information between objects, so that the model can learn deeper features. TP-Net uses hole convolution as the processing module for the final image output. The multi-scale features are integrated through hole convolution kernels of different sizes, thereby enhancing the robustness of segmentation.

[0006] In recent years, deep learning has made significant progress in the field of image segmentation. Deep learning models can automatically learn features, are easy to operate, and have high performance. They can obtain high-precision segmentation results while taking into account both time and space efficiency. The classic U-Net network was designed by Ronneberger et al. It adopts a symmetrical encoder-decoder structure and uses jump connections to transfer primitive features, thereby increasing the network's attention to subtle features. These features enable it to perform retinal vascular segmentation tasks well. However, due to the widespread distribution of thin vessels and small target vessels in fundus images, it is still difficult to accurately capture vascular features. In addition, the information loss in the pooling operation exacerbates the difficulty of thin vessel segmentation. To solve these problems, Gu et al. designed a multi-core pooling fusion multi-attribute convolution module to fully integrate contextual information for vascular feature extraction, thereby solving the problem of information loss caused by pooling. Yin et al. used multi-source image input to ensure the transmission of retinal vascular features, and used feature fusion of different scales to provide each layer of the network with the original feature information of the vessels, which to some extent supplemented the information lost by the pooling operation. In addition, they used the Hessian matrix to obtain weak information in the vessels, which, although it would generate some noise, also enhanced the attention to the vessels. Li et al. used dual CNN and RNN encoders to extract features and fuse fine blood vessel features, and used multiple context information fusion modules to fully extract blood vessel features and alleviate the problem of information loss. In summary, despite many improvements, image segmentation, especially in the segmentation of subtle features and small objects, still faces many challenges and technical bottlenecks.

[0007] Deficiencies of existing technology:

[0008] (1) Most segmentation algorithms are limited to traditional convolutional models. Due to spatial invariance constraints, they have limitations in segmenting different objects in different scenes. In particular, the segmentation accuracy of objects with different morphological distributions and objects with large size differences is low.

[0009] (2) Since there are quite a lot of small objects in the actual scene, most networks cannot effectively perceive them and tend to ignore them. However, small objects are often widely distributed in the accurate segmentation effect. For example, objects such as vehicles and trees are captured by remote sensing satellite photos, while the segmentation effect of leaves on trees captured at low altitude is poor and difficult to process.

[0010] (3) In order to achieve high-precision segmentation, feature transfer is a very important step. The existing pooling methods are prone to losing subtle information, making it impossible for deeper network structures to perceive small target features and thus unable to perform segmentation. Existing models tend to ignore this subtle feature problem, resulting in low accuracy in the segmentation of small targets.

[0011] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0012] The purpose of the present invention is to provide an image semantic segmentation method based on snake convolution to solve the technical problems raised in the background technology.

[0013] To achieve the above object, the present invention provides the following technical solution: an image semantic segmentation method based on snake convolution, comprising at least the following steps:

[0014] S1: Batch data preprocessing of the collected color fundus data;

[0015] S2: Divide the data into non-overlapping data sets;

[0016] S3: Design loss function for accurate segmentation;

[0017] S4: Design a segmentation network based on snake convolution;

[0018] S5: Train the segmentation model and perform segmentation verification. The segmentation model training is completed, and the image semantics is segmented through the trained segmentation model.

[0019] Furthermore, the S1 at least includes the following steps: processing the fundus image in JPG format, and normalizing the data of different channels to transfer them into the model.

[0020] Furthermore, the data set division in S2 at least includes the following steps:

[0021] Randomly shuffle and divide according to fundus data;

[0022] Divide into training, validation, and test sets in a ratio of 8:1:1;

[0023] The data of any fundus cannot overlap between these sets.

[0024] Furthermore, the loss function of S3 includes DiceLoss, BCELoss and mixed loss;

[0025] The DiceLoss is as follows:

[0026]

[0027] The BCELoss is as follows:

[0028]

[0029] The mixing loss is as follows:

[0030] L seg =α·L dice +β·L bce

[0031] Among them, α and β are the weight hyperparameters of the two losses, and their sum is 1;

[0032] In the formula, N represents the total number of pixels, and C represents the total number of categories; Indicates the cth class in the binary GroundTruth, the i-th pixel corresponds to the class value; Represents the predicted probability value of the corresponding category;

[0033] ε=1e-5, which represents the smoothing exponent, is used to prevent the denominator from being predicted to be 0, which can lead to some extreme situations. In addition, it can also play a role in smoothing the loss and gradient.

[0034] Furthermore, the design of the segmentation network based on snake convolution in S4 at least includes the following steps:

[0035] Train using the hybrid loss function specified in S3;

[0036] Use the gradient back propagation of the neural network during training to update the model weights, judge whether the model is better based on the segmentation effect on the validation set after model training, and update the saved model weights;

[0037] Finally, after the model training is completed, the segmentation effect is evaluated on the test set.

[0038] Further, the segmentation validation in S5 uses five commonly used evaluation indicators, including Acc (accuracy), Se (sensitivity), Sp (specificity), F1 (F1 score) and AUC (area under the curve);

[0039] The judgment rule of the segmentation verification in S5 includes that the threshold of the probability map generated by the network is 0.5, and the pixel points greater than 0.5 are predicted to be blood vessels, otherwise they are predicted to be background.

[0040] Furthermore, the segmentation model is a DSM-Net model, which includes a snake convolutional encoder, a hybrid attention module and a fusion pooling module.

[0041] Furthermore, the snake convolution encoder includes traditional convolution and snake convolution, the traditional convolution is used to capture conventional targets, and the snake convolution is used to further extract morphological features of different objects, thereby extracting deeper semantic features;

[0042] The snake convolution encoder tracks different structures by adjusting the shape of the convolution kernel;

[0043] The operation of the snake convolution encoder includes two 3×3 convolution operations and two-directional snake convolution operations;

[0044] The application of the snake convolution encoder comprises at least the following steps:

[0045] First, ton passes through a 3×3 kernel convolutional layer to generate a smooth feature map;

[0046] At the same time, the original feature map is linearly shifted and feature extracted by convolution in the x and y directions through snake convolution, and the resulting smooth feature map is fused with the snake convolution response map in the two directions;

[0047] The response map includes the original features of the object and the topological information extracted by the snake convolution layer;

[0048] Finally, the fused features are further fused and the number of channels is adjusted by another 3×3 kernel convolution layer;

[0049] The final output includes two basic components: sliding convolution information extracted by two smooth convolution kernels and fused adaptive transformation snake convolution, and the vascular structure features are obtained as shown in the following formula:

[0050] F out =Conv(DSC_x(F in )⊙DSC_y(F in )⊙Conv(F in ))

[0051] In the formula, Fin is the input feature map, Fout is the output feature map, ⊙ is the connection, Conv is the batch normalization and ReLU activation after the 3×3 convolution operation, and DSC x and DSC y represent the dynamic snake convolution on the x-axis and y-axis respectively.

[0052] Furthermore, the application of the hybrid attention module includes at least the following steps:

[0053] Create a hybrid block through a sliding window mechanism to build a local map of the input feature map;

[0054] Subsequently, a self-attention operation is applied to establish local-global co-dependencies to promote the association of mixed group information within the window, which not only preserves the spatial structure of different scales but also integrates the mixed channel information, see the following formula:

[0055] F out =Hardswish(BN(SeparConv(F in )))

[0056] In the formula, Fin is the input feature, Fout is the output feature, SeparConv is the depth-wise separable convolution with different kernel sizes, the number of channels remains unchanged during the convolution process, BN is batch normalization, Hardswish is the Hardswish activation function, and local mapping is performed through different sliding window methods.

[0057] Furthermore, the fusion pooling module includes at least one 2×2 convolution and pooling module;

[0058] The application of the fusion pooling module comprises at least the following steps:

[0059] First, the input feature map is passed through a 2×2 convolution to keep the number of channels unchanged to preserve the original subtle feature information;

[0060] The result of the convolution is then used as the maximum pooling and average pooling. The pooling operation effectively reduces the feature dimension while retaining valid information.

[0061] In view of the feature information caused by pooling, the downsampling operation is divided into two branches, one of which uses fusion pooling to combine maximum pooling and average pooling to ensure effective feature extraction, and the convolution operation is introduced in the other branch;

[0062] The convolution operation uses a kernel size of 2 and sets the stride to 2, which increases the learnability of the pooling operation. This convolution operation can be learned through parameter adjustment, significantly retains subtle feature information, and produces a feature map of the same size as the pooling operation.

[0063] Finally, the two feature maps are fused with equal weights, which greatly preserves the characteristic information of thin blood vessels.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention obtains a new fast, efficient and accurate image segmentation model DSM-Net through the design of the method; DSM-Net uses a symmetric encoding-decoding architecture to effectively perform feature extraction and upsampling, and realizes end-to-end pixel-level prediction; wherein, the encoding stage combines snake convolution (EncoderDS), which can linearly adjust the position of the convolution kernel to fully learn the object features; fusion pooling is used to combine convolution and pooling to efficiently transfer the effective features extracted by the encoder; the hybrid attention module is used to model the space and channel, so that the network focuses on different local structural features and realizes efficient feature fitting capabilities; the reasonable construction of the overall architecture greatly reduces the training cost, and the reasonable network depth and internal module design solve the problem of low segmentation accuracy caused by the morphological distribution and large size difference of the segmentation target; in order to reflect the excellent performance of the proposed segmentation model in different segmentation tasks, especially the tubular segmentation task, multiple data sets are used for training to expand the scope of application of the model.

[0066] 2. Compared with other segmentation algorithms, DSM-Net combines the snake convolution mode, which can adaptively adjust the relative position of the convolution kernel according to the scene and object, so as to learn features at a deeper level, grasp discrete morphological data, and achieve more accurate segmentation effect;

[0067] 3. The overall model architecture of DSM-Net of the present invention is implemented by a symmetric encoder-decoder, which has strong feature fitting ability and feature scale restoration ability. Training on this basis can effectively perform gradient backpropagation to reduce loss value. Combining the hybrid attention mechanism and fusion pooling module can effectively utilize encoder features for modeling on spatial scale, focusing on the spatial representation of objects, and can be effectively applied to various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0069] Figure 1 The overall image segmentation algorithm flow chart provided by the present invention;

[0070] Figure 2 A schematic diagram of the DSM-Net network model provided by the present invention;

[0071] Figure 3 A schematic diagram of the EncoderDS module provided by the present invention;

[0072] Figure 4 Schematic diagram of the hybrid attention framework provided by the present invention;

[0073] Figure 5 A schematic diagram of a local feature mapping framework provided by the present invention;

[0074] Figure 6 A schematic diagram of a global feature mapping framework provided by the present invention;

[0075] Figure 7 A schematic diagram of a fusion pooling module provided by the present invention;

[0076] Figure 8 This is a visualization comparison diagram of segmentation masks of different models provided by the present invention on dataset A;

[0077] Fig. 9 A visualization comparison diagram of segmentation masks of different models provided by the present invention on data set B;

[0078] Fig.10 This is a comparison chart of model parameters of different models provided by the present invention on data set C. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0080] See also Figure 1 , an image semantic segmentation method based on snake convolution, comprising at least the following steps:

[0081] S1: Batch data preprocessing of the collected color fundus data;

[0082] S2: Divide the data into non-overlapping data sets;

[0083] S3: Design loss function for accurate segmentation;

[0084] S4: Design a segmentation network based on snake convolution;

[0085] S5: Train the segmentation model and perform segmentation verification. The segmentation model training is completed, and the image semantics is segmented through the trained segmentation model.

[0086] S1 at least includes the following steps: processing the fundus image in JPG format, and normalizing the data of different channels to pass them into the model.

[0087] The data set partitioning in S2 includes at least the following steps:

[0088] Randomly shuffle and divide according to fundus data;

[0089] Divide into training, validation, and test sets in a ratio of 8:1:1;

[0090] The data of any fundus cannot overlap between these sets.

[0091] The loss functions of S3 include DiceLoss, BCELoss and mixed loss;

[0092] DiceLoss is as follows:

[0093]

[0094] BCELoss is as follows:

[0095]

[0096] The mixed loss is as follows:

[0097] L seg =α·L dice +β·L bce

[0098] α and β are weight hyperparameters of two losses, and their sum is 1. They are set to [0.5, 0.5] in the experiment to make the model training more balanced and stable.

[0099] In the formula, N represents the total number of pixels, and C represents the total number of categories; Indicates the cth class in the binary GroundTruth, the i-th pixel corresponds to the class value; Represents the predicted probability value of the corresponding category;

[0100] ε=1e-5, which represents the smoothing exponent, is used to prevent the denominator from being predicted to be 0, which can lead to some extreme situations. In addition, it can also play a role in smoothing the loss and gradient.

[0101] The design of the segmentation network based on snake convolution in S4 includes at least the following steps:

[0102] Train using the hybrid loss function specified in S3;

[0103] Use the gradient back propagation of the neural network during training to update the model weights, judge whether the model is better based on the segmentation effect on the validation set after model training, and update the saved model weights;

[0104] Finally, after the model training is completed, the segmentation effect is evaluated on the test set.

[0105] Segmentation validation in S5 uses five commonly used evaluation metrics, including Acc (accuracy), Se (sensitivity), Sp (specificity), F1 (F1 score) and AUC (area under the curve);

[0106] The judgment rules for segmentation verification in S5 include that the threshold of the probability map generated by the network is 0.5, and pixels greater than 0.5 are predicted as blood vessels, otherwise they are predicted as background.

[0107] For Acc, Se, Sp, F1, the definitions are as follows:

[0108]

[0109]

[0110]

[0111]

[0112] The decision rules for segmentation verification in S5 include taking the ground truth as the benchmark, correctly classified retinal vessels are classified as TP class, those that are actually retinal vessels but classified as background by the network are classified as FN class, correctly classified background is classified as TN class, and incorrectly classified background classified as retinal vessels is classified as FP class.

[0113] All the test results of the models trained on the dataset are constructed with distributions of mean and standard deviation, and all indicators exceed the 95% confidence level.

[0114] See also Figure 2 , the segmentation model is the DSM-Net model, and the segmentation model includes a snake convolution encoder, a hybrid attention module and a fusion pooling module.

[0115] participate Figure 3 The snake convolution encoder includes traditional convolution and snake convolution. Traditional convolution is used to capture conventional targets, and snake convolution is used to further extract the morphological features of different objects, and then extract deeper semantic features;

[0116] The snake convolution encoder tracks different structures by adjusting the shape of the convolution kernel;

[0117] Different structures are tracked by adjusting the shape of the convolution kernel. However, in the small object segmentation task, due to the complex topological structure, cluttered background information, and small morphological size, current methods are difficult to adapt to complex tubular segmentation scenarios. Therefore, segmenting small objects and ensuring their segmentation continuity remains a major research challenge.

[0118] The position transformation of the snake kernel is achieved by iteratively offsetting the horizontal and vertical directions, and then continuously linearly offsetting through bilinear interpolation. The offset of the convolution kernel is constrained by the position of the previous convolution kernel, which prevents the convolution kernel from being attracted by background noise, thereby ensuring more effective extraction of object features.

[0119] Due to the advantages of linear offset in dynamic snake convolution, it adaptively adapts to the narrow and elongated distribution pattern of the target and effectively captures the morphological characteristics of curved and tiny objects. This can extract richer vascular information and enable the network to detect the smallest blood vessels.

[0120]

[0121] The linear offset of the convolution kernel position is achieved by accumulating iterative offsets in two directions.

[0122] The operation of the snake convolution encoder includes two 3×3 convolution operations and snake convolution operations in two directions;

[0123] The application of the snake convolutional encoder includes at least the following steps:

[0124] First, ton passes through a 3×3 kernel convolutional layer to generate a smooth feature map;

[0125] At the same time, the original feature map is linearly shifted and feature extracted by convolution in the x and y directions through snake convolution, and the resulting smooth feature map is fused with the snake convolution response map in the two directions;

[0126] The response map includes the original features of the object and the topological information extracted by the snake convolution layer;

[0127] Finally, the fused features are further fused and the number of channels is adjusted by another 3×3 kernel convolution layer;

[0128] The final output includes two basic components: sliding convolution information extracted by two smooth convolution kernels and fused adaptive transformation snake convolution, and the vascular structure features are obtained as shown in the following formula:

[0129] F out =Conv(DSC_x(F in )⊙DSC_y(F in )⊙Conv(F in ))

[0130] In the formula, Fin is the input feature map, Fout is the output feature map, ⊙ is the connection, Conv is the batch normalization and ReLU activation after the 3×3 convolution operation, and DSC x and DSC y represent the dynamic snake convolution on the x-axis and y-axis respectively.

[0131] CSA is composed of different local feature maps and global feature maps, including depthwise separable convolutions with different kernel sizes. Although attention mechanisms have been shown to be effective in segmentation tasks, most models focus on high-dimensional semantic feature maps. However, ignoring attention operations in low-scale feature maps means missing critical spatial scale information. Different segmentation task scenarios are filled with a large number of tiny objects, each of which exhibits rich distribution patterns and curvatures, as well as other spatial features. For example, Mou et al. designed CS-Net to perform spatial and channel attention calculations at the highest feature layer. However, after multiple pooling operations, the high-dimensional feature map loses a lot of spatial scale information of blood vessels. This loss ultimately affects the effectiveness of attention calculations. In order to extract the various distribution patterns present in images in different scenes and establish robust spatial dependencies, a hybrid local-global response paradigm is applied.

[0132] The spatial morphological information about objects in a local region is the key to effective segmentation. Therefore, it becomes crucial to establish effective spatial relationships of local vessels.

[0133] See also Figure 4 , the application of the hybrid attention module includes at least the following steps:

[0134] Create a hybrid block through a sliding window mechanism to build a local map of the input feature map;

[0135] Subsequently, a self-attention operation is applied to establish local-global co-dependencies to promote the association of mixed group information within the window, which not only preserves the spatial structure of different scales but also integrates the mixed channel information, see the following formula:

[0136] F out =Hardswish(BN(SeparCon(F in )))

[0137] In the formula, Fin is the input feature, Fout is the output feature, SeparConv is the depth-wise separable convolution with different kernel sizes, the number of channels remains unchanged during the convolution process, BN is batch normalization, and Hardswish is the Hardswish activation function. Local mapping is performed through different sliding window methods.

[0138] Local Mapping Figure 5 , global mapping see Figure 6 .

[0139] See also Figure 7 , the fusion pooling module includes at least one 2×2 convolution (2x2 Conv) and pooling module;

[0140] The application of the fusion pooling module includes at least the following steps:

[0141] First, the input feature map is passed through a 2×2 convolution to keep the number of channels unchanged to preserve the original subtle feature information;

[0142] The convolution result is then used for maximum pooling (Max-Pool) and average pooling (Avg-Pool). The pooling operation effectively reduces the feature dimension while retaining valid information.

[0143] However, the loss of feature information caused by pooling cannot be ignored, especially in the task of segmenting a large number of relatively small objects. Therefore, the key to achieving accurate segmentation is to use the detailed information loss caused by pooling and fully retain effective features. For current mainstream technologies, such as Max-Pooling and Avg-Pool, they tend to discard a lot of detail information. Max-Pooling can actively retain overall image features and keep details, especially in tasks at the single-digit pixel level. Therefore, the present invention proposes a fusion pooling method specifically for accurate segmentation in different scenarios. It is different from the existing models that only fuse Max-Pooling and Avg-Pool together.

[0144] In view of the feature information caused by pooling, the downsampling operation is divided into two branches, one of which uses fusion pooling to combine maximum pooling and average pooling to ensure effective feature extraction, and the convolution operation is introduced in the other branch;

[0145] The convolution operation uses a kernel size of 2 and sets the stride to 2, which increases the learnability of the pooling operation. This convolution operation can be learned through parameter adjustment, significantly retains subtle feature information, and produces a feature map of the same size as the pooling operation.

[0146] Finally, the two feature maps are fused with equal weights, which greatly preserves the characteristic information of thin blood vessels.

[0147] Based on the above, we further propose:

[0148] The proposed DSM-Net network was trained and tested on three public datasets: DRIVE, STARE, and CHASE DB1. The performance of DSM-Net was compared with other existing models.

[0149] DRIVE: The DRIVE dataset contains 40 images, 7 of which are pathological images. We use the official dataset division, that is, the first 20 are used for training and the last 20 are used for testing. The entire dataset is screened by the Dutch DR screening collection, with a size of 584×565 and acquired by a Canon CR5 non-expanded 3CCD camera with a field of view of 45°. The first manually labeled image is used as the ground truth.

[0150] CHASE DB1: The CHASE DB1 dataset includes 28 fundus images of 14 pairs of left and right eyes of children's retinas, with a size of 999×960, taken by the London Cardiovascular Health Survey handheld NidekNM200D fundus camera with a field of view of 30°. We use the first 20 images for training and the last 8 images for testing, and we use the first manually labeled image as groundtruth.

[0151] STARE: The STARE dataset contains 20 images, 10 of which are pathological images with a size of 700×605, collected by a TopCon TRV-50 fundus camera with a field of view of 35°. We randomly select 20% of them as test images and the rest of the images are used for network training. The first manually labeled image is used as the ground truth.

[0152] Compared with the most advanced methods, including 8 specialized retinal vessel segmentation models and 2 general medical image segmentation models, the specific results of the three datasets are shown in Tables 1, 2 and 3. There are 5 evaluation indicators: Acc, AUC, Se, Sp, F1, the highest indicator is indicated in black bold, and "-" indicates that the experimental data is not provided in the original literature.

[0153] And further proposed the cross-validation results on the two datasets as shown in Table 4.

[0154] See also Figure 8-10 , Figure 8 From left to right in the figure are fundus image, CS-Net segmentation result, ResDO-UNet segmentation result, DSM-Net segmentation result, and annotated image.

[0155] Fig. 9 From left to right in the figure are fundus image, CS-Net segmentation result, ResDO-UNet segmentation result, DSM-Net segmentation result, and annotated image.

[0156] Fig.10 From left to right in the figure are fundus image, CS-Net segmentation result, ResDO-UNet segmentation result, DSM-Net segmentation result, and annotated image (detailed segmentation visualization result).

[0157] Table 1 Model comparison results on the DRIVE dataset

[0158]

[0159] Table 2 Model comparison results on the STARE dataset

[0160]

[0161] Table 3 Model comparison results on the CHASE_DB1 dataset

[0162]

[0163] Table 4 Cross validation results on two datasets

[0164]

[0165] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for image semantic segmentation based on snake convolution, characterized in that: At least the following steps are included: S1: Batch data preprocessing of the collected color fundus data; S2: Divide the data into non-overlapping data sets; S3: Design loss function for accurate segmentation; S4: Design a segmentation network based on snake convolution; S5: Train the segmentation model and perform segmentation verification. The segmentation model training is completed, and the image semantics is segmented through the trained segmentation model.

2. The method for image semantic segmentation based on snake convolution according to claim 1, characterized in that: The S1 at least includes the following steps: processing the fundus image in JPG format, and normalizing the data of different channels to pass them into the model.

3. The method for image semantic segmentation based on snake convolution according to claim 1, characterized in that: The data set division in S2 at least includes the following steps: Randomly shuffle and divide according to fundus data; Divide into training, validation, and test sets in a ratio of 8:1:1; The data of any fundus cannot overlap between these sets.

4. The image semantic segmentation method based on snake convolution according to claim 1, characterized in that: The loss functions of S3 include DiceLoss, BCELoss and mixed loss; The DiceLoss is as follows: The BCELoss is as follows: The mixing loss is as follows: L seg =α·L dice +β·L bce Among them, α and β are the weight hyperparameters of the two losses, and their sum is 1; In the formula, N represents the total number of pixels, and C represents the total number of categories; Indicates the cth class in the binary GroundTruth, the i-th pixel corresponds to the class value; Represents the predicted probability value of the corresponding category; ε=1e-5, which represents the smoothing exponent, is used to prevent the denominator from being predicted to be 0, which can lead to some extreme situations. In addition, it can also play a role in smoothing the loss and gradient.

5. The method for image semantic segmentation based on snake convolution according to claim 1, characterized in that: The design of the segmentation network based on snake convolution in S4 at least includes the following steps: Train using the hybrid loss function specified in S3; Use the gradient back propagation of the neural network during training to update the model weights, judge whether the model is better based on the segmentation effect on the validation set after model training, and update the saved model weights; Finally, after the model training is completed, the segmentation effect is evaluated on the test set.

6. The method for image semantic segmentation based on snake convolution according to claim 1, characterized in that: The segmentation validation in S5 uses five commonly used evaluation indicators, including Acc (accuracy), Se (sensitivity), Sp (specificity), F1 (F1 score) and AUC (area under the curve); The judgment rule of the segmentation verification in S5 includes that the threshold of the probability map generated by the network is 0.5, and the pixel points greater than 0.5 are predicted to be blood vessels, otherwise they are predicted to be background.

7. The image semantic segmentation method based on snake convolution according to claim 1, characterized in that: The segmentation model is a DSM-Net model, which includes a snake convolutional encoder, a hybrid attention module and a fusion pooling module.

8. The method for image semantic segmentation based on snake convolution according to claim 7, characterized in that: The snake convolution encoder includes traditional convolution and snake convolution, the traditional convolution is used to capture conventional targets, and the snake convolution is used to further extract morphological features of different objects, thereby extracting deeper semantic features; The snake convolution encoder tracks different structures by adjusting the shape of the convolution kernel; The operation of the snake convolution encoder includes two 3×3 convolution operations and two-directional snake convolution operations; The application of the snake convolutional encoder is at least The following steps are involved: First, ton passes through a 3×3 kernel convolutional layer to generate a smooth feature map; At the same time, the original feature map is linearly shifted and feature extracted by convolution operation in the x and y directions through snake convolution, and the obtained smooth feature map is fused with the snake convolution response map in two directions; The response map includes the original features of the object and the topological information extracted by the snake convolution layer; Finally, the fused features are further fused and the number of channels is adjusted by another 3×3 kernel convolution layer; The final output includes two basic components: sliding convolution information extracted by two smooth convolution kernels and fused adaptive transformation snake convolution, and the vascular structure features are obtained as shown in the following formula: F out =Conv(DSC_x(F in )⊙DSC_Y(F in )⊙Conv(F in )) In the formula, Fin is the input feature map, Fout is the output feature map, ⊙ is the connection, Conv is the batch normalization and ReLU activation after the 3×3 convolution operation, and DSC x and DSC y represent the dynamic snake convolution on the x-axis and y-axis respectively.

9. The method for image semantic segmentation based on snake convolution according to claim 7, characterized in that: The application of the hybrid attention module comprises at least the following steps: Create a hybrid block through a sliding window mechanism to build a local map of the input feature map; Subsequently, a self-attention operation is applied to establish local-global co-dependencies to promote the association of mixed group information within the window, which not only preserves the spatial structure of different scales but also integrates the mixed channel information, see the following formula: F out =Hardswish(BN(SeparCon(F in ))) In the formula, Fin is the input feature, Fout is the output feature, SeparConv is the depth-wise separable convolution with different kernel sizes, the number of channels remains unchanged during the convolution process, BN is batch normalization, Hardswish is the Hardswish activation function, and local mapping is performed through different sliding window methods.

10. The method for image semantic segmentation based on snake convolution according to claim 7, characterized in that: The fusion pooling module includes at least one 2×2 convolution and pooling module; The application of the fusion pooling module includes at least the following steps: First, the input feature map is passed through a 2×2 convolution to keep the number of channels unchanged to preserve the original subtle feature information; The result of the convolution is then used as the maximum pooling and average pooling. The pooling operation effectively reduces the feature dimension while retaining valid information. In view of the feature information caused by pooling, the downsampling operation is divided into two branches, one of which uses fusion pooling to combine maximum pooling and average pooling to ensure effective feature extraction, and the convolution operation is introduced in the other branch; The convolution operation uses a kernel size of 2 and sets the stride to 2, which increases the learnability of the pooling operation. This convolution operation can be learned through parameter adjustment, significantly retains subtle feature information, and produces a feature map of the same size as the pooling operation. Finally, the two feature maps are fused with equal weights, which greatly preserves the characteristic information of thin blood vessels.

Images