Tubular structure image segmentation method and related equipment

Abstract

The invention provides a tubular structure image segmentation method and related equipment, and the method comprises the steps: obtaining tubular structure image data, inputting the data into a segmentation model for training, and obtaining a tubular structure image segmentation model; inputting the target tubular structure image into a tubular structure image segmentation model for segmentation to obtain a segmentation result; the segmentation model comprises a feature extraction module, a feature enhancement module and an output module; compared with the prior art, the method has the advantages that feature extraction, fusion and enhancement modes are improved, the features of the tubular structure in different directions including the horizontal direction, the vertical direction and the global direction are extracted through the feature extraction module, the features from the different directions are fused through the feature fusion module, spatial information in the different directions is fully utilized, and the feature extraction efficiency is improved. And the tubular structure features are further enhanced through the feature enhancement module, so that the segmentation precision of the tubular structure image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a tubular structure image segmentation method and related equipment. Background Art

[0002] Segmentation of tubular structure images is a fundamental problem in the fields of medical imaging, remote sensing applications, and industrial quality control. Accurate segmentation of tubular structure images is a key factor in improving the accuracy and efficiency of downstream tasks such as organ localization and lesion diagnosis, satellite image road network segmentation, and wire defect diagnosis in the above fields.

[0003] Due to the special morphological characteristics and topological structure of tubular structures, the accuracy of traditional automatic segmentation methods is difficult to guarantee and the segmentation precision is not high. Traditional manual segmentation methods also have problems such as low efficiency and high labor costs. High-precision intelligent tubular structure image segmentation methods have become a rigid demand in many fields to reduce labor costs, improve production efficiency, and promote intelligence and automation.

[0004] With the development of artificial intelligence, deep learning technology has gradually replaced traditional methods to become the mainstream of automatic segmentation algorithms for tubular structures. The most widely used is the convolutional neural network (CNN), but the network still has the following defects: 1) Tubular structures often have complex morphology and distribution, and they may cross or overlap with other structures. In this case, successful segmentation of tubular structures is very dependent on their contextual information. However, the receptive field of CNN is limited and it is unable to extract sufficient contextual information, resulting in mis-segmentation caused by insufficient contextual information. 2) Tubular structures have different sizes and shapes, and exhibit different characteristics at different scales. Multi-scale information is required to observe and analyze these structures at different scales, so as to more comprehensively capture their characteristics and details. However, CNN does not have the ability to extract multi-scale information. The lack of multi-scale information will lose the ability to differentiate different scales, resulting in inaccurate segmentation. 3) Tubular structures usually have complex directions and branching patterns in images. These structures may extend in multiple directions, and even bend or cross in some areas. Multi-directional information is required to more comprehensively capture the spatial distribution and direction of tubular structures, so as to more accurately identify and segment these structures. However, CNN usually directly extracts all local information but cannot extract directional information alone, which limits its segmentation ability when facing multi-directional structures of tubular structures.

[0005] Therefore, faced with the high requirements of tubular structures for special feature information such as contextual information, multi-scale information, and multi-directional information, how to accurately segment tubular structures and provide stable and effective intelligent tubular structure segmentation technology for downstream tasks in various fields is a problem that needs to be solved urgently. Summary of the invention

[0006] The invention provides a tubular structure image segmentation method and related equipment, the purpose of which is to improve the segmentation accuracy of the tubular structure image.

[0007] In order to achieve the above object, the present invention provides a tubular structure image segmentation method, comprising:

[0008] Step 1, obtaining tubular structure image data;

[0009] Step 2, inputting the tubular structure image data into the segmentation model for training to obtain a tubular structure image segmentation model;

[0010] Step 3, inputting the target tubular structure image into the tubular structure image segmentation model for segmentation to obtain a segmentation result;

[0011] The segmentation model includes a feature extraction module for extracting horizontal features, vertical features, global features and performing encoding and decoding, a feature enhancement module for enhancing the slender features of the tubular structure, and an output module for performing image segmentation according to the extracted features. The first input end of the feature extraction module is the input end of the segmentation model, the first output end of the feature extraction module is connected to the input end of the feature enhancement module, the output end of the feature enhancement module is connected to the second input end of the feature extraction module, the second output end of the feature extraction module is connected to the input end of the output module, and the output end of the output module is the output end of the segmentation model.

[0012] Further, the feature extraction module includes an encoder and a decoder;

[0013] The input of the encoder is the input of the feature extraction module;

[0014] The output end of the encoder is connected to the input end of the feature enhancement module;

[0015] The input end of the decoder is connected to the output end of the feature enhancement module;

[0016] The output terminal of the decoder is connected to the input terminal of the output module.

[0017] More specifically, the encoder includes:

[0018] A first multi-view feature fusion unit, a first maximum pooling downsampling unit, a second multi-view feature fusion unit, a second maximum pooling downsampling unit, a third multi-view feature fusion unit, a third maximum pooling downsampling unit, and a fourth multi-view feature fusion unit connected in sequence;

[0019] The input end of the first multi-view feature fusion unit is the input end of the encoder;

[0020] An output end of the fourth multi-view feature fusion unit is connected to an input end of the feature enhancement module.

[0021] More specifically, the decoder includes:

[0022] A fifth multi-view feature fusion unit, a first bilinear interpolation upsampling unit, a sixth multi-view feature fusion unit, a second bilinear interpolation upsampling unit, a seventh multi-view feature fusion unit, a third bilinear interpolation upsampling unit, and an eighth multi-view feature fusion unit connected in sequence;

[0023] The input end of the fifth multi-view feature fusion unit is connected to the output end of the feature enhancement module;

[0024] The output end of the eighth multi-view feature fusion unit is connected to the input end of the output module.

[0025] Further, the first multi-view feature fusion unit, the second multi-view feature fusion unit, the third multi-view feature fusion unit, the fourth multi-view feature fusion unit, the fifth multi-view feature fusion unit, the sixth multi-view feature fusion unit, the seventh multi-view feature fusion unit, and the eighth multi-view feature fusion unit all include:

[0026] The first large core snake convolution layer, the second large core snake convolution layer, the multi-scale stereo attention layer, the first convolution layer, the splicing layer, the second convolution layer, the batch normalization layer, the Relu activation layer, and the multi-scale feature enhancement layer;

[0027] The input of the first large-core snake convolution layer, the input of the second large-core snake convolution layer, and the input of the multi-scale stereo attention layer are all connected to the output of the previous unit of each multi-view feature fusion unit;

[0028] The output end of the first large-core snake convolution layer is connected to the first input end of the concatenation layer;

[0029] The output end of the second large-core snake convolution layer is connected to the second input end of the concatenation layer;

[0030] The output of the multi-scale stereo attention layer is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the third input of the splicing layer, the output of the splicing layer is connected to the input of the second convolutional layer, the output of the second convolutional layer is connected to the input of the batch normalization layer, the output of the batch normalization layer is connected to the input of the Relu activation layer, the output of the Relu activation layer is connected to the input of the multi-scale feature enhancement layer, and the output of the multi-scale feature enhancement layer is connected to the input of the next level unit of each multi-view feature fusion unit.

[0031] Further, the feature enhancement module includes four branched multi-scale feature enhancement units connected sequentially;

[0032] An input end of a first branch multi-scale feature enhancement unit among the four branch multi-scale feature enhancement units is connected to an output end of the multi-scale feature enhancement layer in the fourth multi-view feature fusion unit;

[0033] The output end of the fourth branch multi-scale feature enhancement unit in the four branch multi-scale feature enhancement units is connected to the input end of the first large core snake convolution layer, the input end of the second large core snake convolution layer, and the input end of the multi-scale stereo attention layer in the fifth multi-view feature fusion unit.

[0034] Furthermore, the first large core snake convolution layer and the second large core snake convolution layer both include:

[0035] Dynamic snake convolution sublayer, depth-wise separable convolution sublayer, depth-wise separable dilated convolution sublayer, multiplier;

[0036] The input end of the dynamic snake convolution sublayer in the first large-core snake convolution layer is the input end of the first large-core snake convolution layer;

[0037] The input end of the dynamic snake convolution sublayer in the second large-core snake convolution layer is the input end of the second large-core snake convolution layer;

[0038] A first output terminal of the dynamic snake convolution sublayer is connected to an input terminal of the depthwise separable convolution sublayer, and a second output terminal of the dynamic snake convolution sublayer is connected to a first input terminal of the multiplier;

[0039] The output of the depthwise separable convolutional sublayer is connected to the input of the depthwise separable dilated convolutional sublayer;

[0040] An output of the depthwise separable dilated convolutional sublayer is connected to a second input of the multiplier;

[0041] The output end of the multiplier in the first large-core snake convolution layer is the output end of the first large-core snake convolution layer;

[0042] The output end of the multiplier in the second large-core snake convolution layer is the output end of the second large-core snake convolution layer.

[0043] Specifically, each branch multi-scale feature enhancement unit includes:

[0044] Low-resolution feature enhancement branch structure, high-resolution feature enhancement branch structure, residual branch structure, adder;

[0045] The input end of the low-resolution feature enhancement branch structure, the first input end of the high-resolution feature enhancement branch structure, and the input end of the residual branch structure are all input ends of each branch multi-scale feature enhancement unit;

[0046] The first output end of the low-resolution feature enhancement branch structure is connected to the second input end of the high-resolution feature enhancement branch structure;

[0047] The second output end of the low-resolution feature enhancement branch structure is connected to the third input end of the high-resolution feature enhancement branch structure;

[0048] The output end of the high-resolution feature enhancement branch structure is connected to the first input end of the adder;

[0049] The output end of the residual branch structure is connected to the second input end of the adder;

[0050] The output end of the adder is the output end of each branch multi-scale feature enhancement unit.

[0051] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for segmenting a tubular structure image is implemented.

[0052] The present invention also provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for segmenting a tubular structure image is implemented.

[0053] The above scheme of the present invention has the following beneficial effects:

[0054] The present invention obtains tubular structure image data and inputs it into a segmentation model for training to obtain a tubular structure image segmentation model; inputs a target tubular structure image into the tubular structure image segmentation model for segmentation to obtain a segmentation result; the segmentation model includes a feature extraction module for extracting horizontal features, vertical features, and global features and performing encoding and decoding, a feature enhancement module for enhancing the slender features of the tubular structure, and an output module for performing image segmentation according to the extracted features; compared with the prior art, the present invention improves feature extraction, fusion and enhancement methods, extracts different directional features of the tubular structure including horizontal, vertical, and global features through the feature extraction module, fuses features from different directions through the feature fusion module, fully utilizes spatial information in different directions, and further strengthens the tubular structure features through the feature enhancement module, thereby improving the segmentation accuracy of the tubular structure image.

[0055] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a segmentation model in an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of the structure of a multi-view feature fusion unit in an embodiment of the present invention;

[0059] Figure 4 Schematic diagram of the structure of a large-core snake-shaped convolutional layer in an embodiment of the present invention;

[0060] Figure 5 Schematic diagram of the structure of a multi-scale stereo attention layer in an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of the structure of a multi-scale feature enhancement layer in an embodiment of the present invention;

[0062] Figure 7 Schematic diagram of the structure of a branch multi-scale feature enhancement unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0065] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0067] In view of the existing problems, the present invention provides a tubular structure image segmentation method and related equipment.

[0068] like Figure 1 As shown, an embodiment of the present invention provides a method for segmenting a tubular structure image, comprising:

[0069] Step 1, obtaining tubular structure image data;

[0070] Step 2, inputting the tubular structure image data into the segmentation model for training to obtain a tubular structure image segmentation model;

[0071] Step 3, inputting the target tubular structure image into the tubular structure image segmentation model for segmentation to obtain a segmentation result;

[0072] The segmentation model includes a feature extraction module for extracting horizontal features, vertical features, global features and performing encoding and decoding, a feature enhancement module for enhancing the slender features of the tubular structure, and an output module for performing image segmentation according to the extracted features. The first input end of the feature extraction module is the input end of the segmentation model, the first output end of the feature extraction module is connected to the input end of the feature enhancement module, the output end of the feature enhancement module is connected to the second input end of the feature extraction module, the second output end of the feature extraction module is connected to the input end of the output module, and the output end of the output module is the output end of the segmentation model.

[0073] Specifically, step 1 includes:

[0074] Tubular structure image data such as arterial images, venous images, or intestinal images are collected, and all tubular structure image data are labeled one by one, and divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0075] Specifically, if Figure 2 As shown, the feature extraction module includes an encoder and a decoder;

[0076] The input of the encoder is the input of the feature extraction module;

[0077] The output end of the encoder is connected to the input end of the feature enhancement module;

[0078] The input end of the decoder is connected to the output end of the feature enhancement module;

[0079] The output terminal of the decoder is connected to the input terminal of the output module.

[0080] Specifically, if Figure 2 As shown, the encoder includes:

[0081] A first multi-view feature fusion unit, a first maximum pooling downsampling unit, a second multi-view feature fusion unit, a second maximum pooling downsampling unit, a third multi-view feature fusion unit, a third maximum pooling downsampling unit, and a fourth multi-view feature fusion unit connected in sequence;

[0082] The input end of the first multi-view feature fusion unit is the input end of the encoder;

[0083] An output end of the fourth multi-view feature fusion unit is connected to an input end of the feature enhancement module.

[0084] Specifically, if Figure 2 As shown, the decoder includes:

[0085] A fifth multi-view feature fusion unit, a first bilinear interpolation upsampling unit, a sixth multi-view feature fusion unit, a second bilinear interpolation upsampling unit, a seventh multi-view feature fusion unit, a third bilinear interpolation upsampling unit, and an eighth multi-view feature fusion unit connected in sequence;

[0086] The input end of the fifth multi-view feature fusion unit is connected to the output end of the feature enhancement module;

[0087] The output end of the eighth multi-view feature fusion unit is connected to the input end of the output module.

[0088] Specifically, if Figure 3As shown, the first multi-view feature fusion unit, the second multi-view feature fusion unit, the third multi-view feature fusion unit, the fourth multi-view feature fusion unit, the fifth multi-view feature fusion unit, the sixth multi-view feature fusion unit, the seventh multi-view feature fusion unit, and the eighth multi-view feature fusion unit all include:

[0089] The first large core snake convolution layer, the second large core snake convolution layer, the multi-scale stereo attention layer, the first convolution layer, the splicing layer, the second convolution layer, the batch normalization layer, the Relu activation layer, and the multi-scale feature enhancement layer;

[0090] The input of the first large-core snake convolution layer, the input of the second large-core snake convolution layer, and the input of the multi-scale stereo attention layer are all connected to the output of the previous unit of each multi-view feature fusion unit;

[0091] The output end of the first large-core snake convolution layer is connected to the first input end of the concatenation layer;

[0092] The output end of the second large-core snake convolution layer is connected to the second input end of the concatenation layer;

[0093] The output of the multi-scale stereo attention layer is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the third input of the splicing layer, the output of the splicing layer is connected to the input of the second convolutional layer, the output of the second convolutional layer is connected to the input of the batch normalization layer, the output of the batch normalization layer is connected to the input of the Relu activation layer, the output of the Relu activation layer is connected to the input of the multi-scale feature enhancement layer, and the output of the multi-scale feature enhancement layer is connected to the input of the next level unit of each multi-view feature fusion unit.

[0094] In the embodiment of the present invention, the multi-view feature fusion unit is referred to as MVFCB unit, which stands for Multi-view Fusion Convolution Block; each multi-view feature fusion unit includes three branches, the first branch is composed of the first large core snake convolution layer, which is used to extract the horizontal features of the tubular structure image data; the second branch is composed of the second large core snake convolution layer, which is used to extract the vertical features of the tubular structure image data; the third branch is composed of a multi-scale stereo attention layer and a first convolution layer, firstly, the multi-scale stereo attention layer is used to refine the features, then the global features are extracted through the first convolution layer, and then the horizontal features, vertical features and global features are input into the splicing layer for splicing operation and connected, and then the second convolution layer, batch normalization layer, Relu activation layer and multi-scale feature enhancement layer are serially connected, and the serial fused features are output. The calculation formula of MVFCB is as follows:

[0095] MVFCB(x i)=MFEU(σ(Conv(cat(LKSC x (x i ),LKSC y (x i ),Conv(MDSA(x i ))))))

[0096] Among them, MFEU(·) represents the multi-scale feature enhancement operation, σ(·) represents the continuous operation of batch normalization and Relu activation function, Conv(·) represents the convolution operation, cat(·) represents the concatenation operation, LKSC x (·) represents the large kernel snake convolution operation in the horizontal direction, LKSC y (·) represents the large kernel snake convolution operation in the vertical direction, and MDSA(·) represents the multi-scale stereo attention operation.

[0097] Specifically, due to the special deformation mode and sufficient receptive field of the large kernel snake convolution, it can adapt to the special topological structure of the tubular structure and provide sufficient contextual information, such as Figure 4 As shown, the first large core snake convolution layer and the second large core snake convolution layer both include:

[0098] Dynamic snake convolution sublayer, depth-wise separable convolution sublayer, depth-wise separable dilated convolution sublayer, multiplier;

[0099] The input end of the dynamic snake convolution sublayer in the first large-core snake convolution layer is the input end of the first large-core snake convolution layer;

[0100] The input end of the dynamic snake convolution sublayer in the second large-core snake convolution layer is the input end of the second large-core snake convolution layer;

[0101] A first output terminal of the dynamic snake convolution sublayer is connected to an input terminal of the depthwise separable convolution sublayer, and a second output terminal of the dynamic snake convolution sublayer is connected to a first input terminal of the multiplier;

[0102] The output of the depthwise separable convolutional sublayer is connected to the input of the depthwise separable dilated convolutional sublayer;

[0103] An output of the depthwise separable dilated convolutional sublayer is connected to a second input of the multiplier;

[0104] The output end of the multiplier in the first large-core snake convolution layer is the output end of the first large-core snake convolution layer;

[0105] The output end of the multiplier in the second large-core snake convolution layer is the output end of the second large-core snake convolution layer.

[0106] In the embodiment of the present invention, the large-core snake convolution layer is composed of a dynamic snake convolution and a large-core convolution with the convolution kernels located in the same direction; the input first passes through the dynamic snake convolution sublayer to form a preliminary feature F S , the preliminary feature F S While retaining, it is fed into a large kernel convolution consisting of a depth-wise separable convolution sublayer and a depth-wise separable dilated convolution sublayer in series to form the attention weight w attn , the attention weight w attn With the preliminary feature F S After multiplication by the multiplier, the batch normalization layer and the Relu activation layer are passed once, and the horizontal feature or vertical feature is finally output, and its expressions are:

[0107] LKSC x (xi) = DWDC x (DWC x (DSC(x i )))×DSC(x i )

[0108] LKSC y (x i )=DWDC y (DWC y (DSC(x i )))×DSC(x i )

[0109] Among them, DWDC x (·), DWDC y (·) represents the depthwise separable dilated convolution in the horizontal direction and the depthwise separable dilated convolution in the vertical direction, respectively. x (·), DWD y (·) represents the depthwise separable convolution in the horizontal direction and the depthwise separable convolution in the vertical direction, respectively, and DSC(·) represents the dynamic snake convolution.

[0110] In an embodiment of the present invention, the multi-scale stereo attention layer combines multi-scale feature extraction and fusion attention; Figure 5 As shown in Figure 1, the multi-scale stereo attention layer first extracts the shallow feature F0 of the tubular structure image data through a depth-separable convolution sublayer, which is expressed as:

[0111] F0=DWC(x i )

[0112] Where DWC(·) represents the depthwise separable convolution operation;

[0113] The shallow feature F0 is passed through three convolution sublayers with different convolution kernel sizes to generate multi-scale information F1, F2 and F3. The three convolution sublayers with different convolution kernel sizes are 3×3 convolution sublayer, 5×5 convolution sublayer and 7×7 convolution sublayer, respectively. The expression is:

[0114] F i =Conv i (F0),i=1,2,3

[0115] Conv i (·) represents the convolution operation to extract the i-th scale feature;

[0116] Then, each scale information F1, F2, F3 and the shallow feature F0 are added through an adder to obtain the fusion feature M, which is expressed as:

[0117]

[0118] The fusion feature M generates the channel attention weight w through the channel attention branch and the spatial attention branch respectively. c and the spatial attention weight w p , the expression is as follows:

[0119] w c =MLP2(MLP1(gap(M)))

[0120] w p =DWC 1×1 (Conv 3×3 (Conv 3×3 (Conv 1×1 (M))))

[0121] Among them, MLP i (·) represents the i-th MLP, gap(·) represents the global average pooling operation, Conv i×i (·) indicates the convolution operation with kernel i×i, DWC 1×1 (·) indicates a depthwise separable convolution operation with a convolution kernel of 1×1;

[0122] The channel attention branch consists of a global average pooling sublayer and two multi-layer perceptrons (MLPs) in sequence; the spatial attention branch consists of a 1×1 convolution sublayer, two 3×3 convolution sublayers, and a 1×1 depth-separable convolution sublayer. c and the spatial attention weight w p Multiply them together to get the stereo attention weight w s , the expression is:

[0123] w s =w c ×p

[0124] The stereo attention weight w s Decompose the matrix into the attention weight w corresponding to each scale information i , i=0,1,2,3, the expression is:

[0125] w i =unsqueeze(w s )

[0126] Among them, unsqueeze(·) represents the matrix decomposition operation;

[0127] The split attention weight w i Normalized by softmax function, the normalized weight w′ of each scale information is obtained i , the expression is:

[0128] w′ i =σ s (w i )

[0129] Among them, represents the softmax activation function;

[0130] The normalized weight w′ is added through multipliers and adders. i After multiplying with the corresponding scale information and then adding them in sequence, the weighted feature F is obtained. Finally, after passing through a 1×1 convolution sublayer and adding it to the input, the final output is obtained. The expression is:

[0131] F=∑w′ i ×F i

[0132] MSA(x i )=Conv 1×1 ((F)+x i

[0133] Among them, Conv 1×1 (·) represents a 1×1 normal convolution operation.

[0134] Specifically, if Figure 6 As shown in the figure, the multi-scale feature enhancement layer is referred to as MFEU (Multi-scale Feature Enhancement Unit), which includes multiple depth-separable dilated convolution sublayers with dilation rates of 1, 2, 4 and 8 respectively to generate multi-scale information, and then splice the information of different scales through the splicing sublayer, and then operate through the channel attention sublayer and the convolution sublayer in turn, and then add the input features through the adder to obtain the enhanced features. The calculation expression of the multi-scale feature enhancement layer is:

[0135] MFEU(x i )=Conv(CA(cat(DWDC1(x i ),DWDC2(x i ),DWDC4(x i ),DWDC8(x i ))))

[0136] Among them, Conv(·) represents the convolution operation, CA(·) represents the channel attention operation, cat(·) represents the concatenation operation, DWDC1(·), DWDC2(·), DWDC4(·), and DWDC8(·) represent the depthwise separable dilated convolution operations with dilation rates of 1, 2, 4, and 8, respectively.

[0137] Specifically, if Figure 2 As shown in FIG. 1 , the feature enhancement module includes four sequentially connected branching multi-scale feature enhancement blocks (Branching Multi-scale Feature Enhancement Blocks), referred to as BMFEBs, which can enhance the slender features of tubular structures and enable the network to better identify and segment tubular structures.

[0138] An input end of a first branch multi-scale feature enhancement unit among the four branch multi-scale feature enhancement units is connected to an output end of a multi-scale feature enhancement layer in a third multi-view feature fusion unit;

[0139] The output end of the fourth branch multi-scale feature enhancement unit in the four branch multi-scale feature enhancement units is connected to the input end of the first large core snake convolution layer, the input end of the second large core snake convolution layer, and the input end of the multi-scale stereo attention layer in the fifth multi-view feature fusion unit.

[0140] Specifically, if Figure 7 As shown, each branch multi-scale feature enhancement unit includes:

[0141] Low-resolution feature enhancement branch structure, high-resolution feature enhancement branch structure, residual branch structure, adder;

[0142] The input end of the low-resolution feature enhancement branch structure, the first input end of the high-resolution feature enhancement branch structure, and the input end of the residual branch structure are all input ends of each branch multi-scale feature enhancement unit;

[0143] The first output end of the low-resolution feature enhancement branch structure is connected to the second input end of the high-resolution feature enhancement branch structure;

[0144] The second output end of the low-resolution feature enhancement branch structure is connected to the third input end of the high-resolution feature enhancement branch structure;

[0145] The output end of the high-resolution feature enhancement branch structure is connected to the first input end of the adder;

[0146] The output end of the residual branch structure is connected to the second input end of the adder;

[0147] The output end of the adder is the output end of each branch multi-scale feature enhancement unit.

[0148] In the embodiment of the present invention, the input of the residual branch structure first passes through a common 1×1 convolution layer, and then passes through a sigmoid activation function to obtain a feature calibration coefficient F C , and then use the multiplier to multiply the characteristic calibration coefficient F C Multiply it with the original input to get the residual branch output F Res ;

[0149] The input of the high-resolution feature enhancement branch structure is fed into the first multi-scale feature enhancement unit for the first multi-scale feature enhancement and then into the first splicing layer. The input of the low-resolution feature enhancement branch structure is fed into the maximum pooling downsampling unit for the maximum pooling downsampling and then into the first multi-scale feature enhancement unit for the first multi-scale feature enhancement. Then, the input is fed into the first bilinear interpolation upsampling unit for the first interpolation upsampling and then into the first splicing layer in the high-resolution feature enhancement branch structure and the features processed by the first multi-scale feature enhancement unit in the high-resolution feature enhancement branch structure for the first splicing. The first splicing result is sequentially fed into the first convolution unit and the second multi-scale feature enhancement unit for the second multi-scale feature enhancement. After scale feature enhancement, the second splicing layer is input, and the first multi-scale feature enhancement result in the low-resolution feature enhancement branch result is input into the second multi-scale feature enhancement unit for the second multi-scale feature enhancement, and then input into the second bilinear interpolation upsampling unit for the second interpolation upsampling, and then input into the second splicing layer in the high-resolution feature enhancement branch result and the features processed by the second multi-scale feature enhancement unit in the high-resolution feature enhancement branch structure are spliced ​​for the second time to obtain the second splicing result, and the second splicing result is sequentially input into the second convolution unit and the third multi-scale feature enhancement unit for the third multi-scale feature enhancement, and then input into the adder, and the third multi-scale feature enhancement result is combined with the residual branch output F through the adder. Res Add them together to get the feature enhancement result, the expression is:

[0150] x′ i =MFEU(x i )

[0151] x′ low =MFEU(map(x i ))

[0152] x″low =MFEU(x′ low )

[0153] x″ i =Conv 1×1 (cat(MFEU(x′ i ), up(MFEU(x′ low )))

[0154] x″′ i =Conv 1×1 (cat(MFEU(x″ i )), up(MFEU(x″ low )))

[0155] BMFEB(x i ) = MFEU(x″′ i )+x i ×Sig(Conλ 1×1 (x i ))

[0156] Among them, MFEU(·) represents the multi-scale feature enhancement operation, map(·) represents the maximum pooling operation, up(·) represents the bilinear interpolation upsampling operation, and Sig(·) represents the Sigmoid function.

[0157] Specifically, the output module includes a convolution unit and a Sigmoid function connected in sequence. The input end of the convolution unit is connected to the output end of the eighth multi-view feature fusion module. Finally, the Sigmoid function outputs an image probability map, and compares the pixel points of each pixel coordinate in the probability map with the threshold to determine the source of the pixel point, that is, whether it belongs to a tubular structure, segment the tubular structure from the background, and extract the target tubular structure.

[0158] It should be noted that the convolution units and convolution sublayers mentioned in the embodiments of the present invention are conventional ordinary convolution layers; the splicing units and splicing sublayers are conventional splicing layers used for feature splicing.

[0159] In the embodiment of the present invention, the final segmentation result image and the binary label of the corresponding image are input into the loss function to calculate the loss of the response. The loss calculation formula is as follows:

[0160] L=βL CE +·(1-β)L clD ,β∈[0,1]

[0161] Among them, L CE represents the cross entropy loss, L clD represents clDice loss, β represents weight;

[0162] The calculation formula for cross entropy loss is:

[0163]

[0164] Among them, represents the number of samples, represents the actual label of the sample (0 or 1), Indicates the probability that the model predicts that the sample is a positive class. In the embodiment of the present invention, 0 and 1 represent that a pixel belongs to the background and belongs to the tubular structure respectively;

[0165] The calculation formula of clDice loss is as follows:

[0166]

[0167] where S(·) represents the skeletonization function used for prediction or labeling, and y i is the actual label of the sample (0 or 1), and ∈ is a small constant used to prevent the denominator from being zero.

[0168] In an embodiment of the present invention, the role of the skeletonization function in the clDice loss is to extract the skeleton of the output predicted by the model and the real label image, so that the loss function can focus on maintaining the integrity and continuity of slender structures. This method is particularly suitable for processing segmentation tasks that focus on maintaining the structural integrity of objects, such as the segmentation of tubular structures. By directly optimizing the skeleton, the clDice loss makes the segmented images generated by the deep learning model closer to the real image in structure, especially performing better on slender and continuous structures.

[0169] In the embodiment of the present invention, after calculating the loss, back propagation is required to further optimize the parameters. The gradient of the loss function to the parameters in the network is calculated by the chain rule. The Nadam optimizer is used to update each parameter using the gradient in the direction of loss reduction. The Nadam parameter update formula is:

[0170]

[0171] m t =β1m t-1 +(1-β1)g t

[0172]

[0173] Where t represents the time step, represents the gradient at time t, θ t-1 represents the parameter value at time t-1, α represents the learning rate, β1, β2, ε, w, ∈ are constants, respectively It is the t-th power of β1 and β2, and the model with the best performance on the validation set is finally saved.

[0174] The collected image data is input into the tubular structure image segmentation model, and segmented by the final tubular structure segmentation model. In the embodiment of the present invention, the segmentation effect of the segmentation method provided by the present application and other commonly used tubular structure segmentation methods are compared on the public blood vessel dataset DRIVE, the public road dataset ROADS and the collected wire dataset, as shown in the following table:

[0175] Table 1

[0176]

[0177]

[0178] The Dice in Table 1 above represents the Dice similarity coefficient, which is usually used to measure the similarity between two samples and is widely used in image segmentation to evaluate the segmentation quality predicted by the model. Its formula is as follows:

[0179]

[0180] Among them, X and Y represent the segmentation results predicted by the model and the actual labels respectively. The value range of the Dice coefficient is 0 to 1. The higher the value, the closer the predicted result is to the actual result.

[0181] RDice, or Robust Dice, is a variant of the Dice coefficient, which aims to enhance its robustness in dealing with noise and outliers. Its core idea is to reduce the impact of noise and outliers by introducing weights or adjusting the calculation method. Its formula is to add a weight on the basis of Dice.

[0182] clDice is a loss function specially designed for tubular object segmentation. It aims to optimize the segmentation performance of the model for slender structures and improve the segmentation accuracy by directly optimizing the consistency between skeletons.

[0183] ACC is short for Accuracy. Accuracy is one of the most intuitive performance indicators in classification tasks. It is used to measure the proportion of correctly classified samples to the total samples. Here, it is used to predict the proportion of correct labels to the total labels. The formula is:

[0184]

[0185] Among them, TP represents true positive examples, TN represents true negative examples, FP represents false positive examples, and FN represents false negative examples.

[0186] AUC is the abbreviation of the area under the receiver operating characteristic curve. The ROC curve is a graphical method to describe the performance of a classification model under all classification thresholds. AUC measures the area of ​​the entire two-dimensional region under the ROC curve. The value range of AUC is 0 to 1. The larger the value, the better the model performance. AUC can be regarded as the probability that the classifier randomly selects a positive example higher than the probability of randomly selecting a negative example.

[0187] The calculation of AUC is more complicated, and there is no simple formula that can be directly calculated like other indicators, but the basic idea is to evaluate the performance of the model under various threshold settings, which is achieved by plotting the true positive rate (TPR) against the false positive rate (FPR).

[0188] Hausdorff distance (HD) is a metric used to measure the distance between two point sets. It is often used in image processing and computational geometry, especially in evaluating the accuracy of image segmentation and object detection. It provides a method to measure the similarity of two point sets in the worst case. For image segmentation, these two point sets are usually the predicted segmentation boundary and the actual segmentation boundary. The definition of Hausdorff distance is as follows:

[0189] HD(A,B)=max{h(A,B),h(B,A)}

[0190] h(A,B)=max(a∈A)min(b∈B)||ab||

[0191] h(B,A)=max(b∈B)min(a∈A)||ba||

[0192] Among them, A and B are two point sets in space, ||ab|| is the distance between point a and point b, usually Euclidean distance.

[0193] It can be seen from Table 1 that the tubular structure segmentation method provided by the embodiment of the present invention exceeds other comparison methods in various indicators and has better performance than other methods. The experimental results prove that in the tubular structure segmentation task, the embodiment of the present invention can accurately segment the tubular structure.

[0194] The embodiment of the present invention obtains tubular structure image data and inputs it into a segmentation model for training to obtain a tubular structure image segmentation model; inputs a target tubular structure image into the tubular structure image segmentation model for segmentation to obtain a segmentation result; the segmentation model includes a feature extraction module for extracting horizontal features, vertical features, and global features and performing encoding and decoding, a feature enhancement module for enhancing the slender features of the tubular structure, and an output module for performing image segmentation according to the extracted features; compared with the prior art, the embodiment of the present invention improves the feature extraction, fusion and enhancement methods, extracts different directional features of the tubular structure including horizontal, vertical, and global features through the feature extraction module, fuses features from different directions through the feature fusion module, fully utilizes spatial information in different directions, and further enhances the tubular structure features through the feature enhancement module, thereby improving the segmentation accuracy of the tubular structure image.

[0195] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for segmenting a tubular structure image is implemented.

[0196] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above method embodiment, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0197] An embodiment of the present invention further provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for segmenting a tubular structure image when executing the computer program.

[0198] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0199] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0200] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0201] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0202] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0203] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for segmenting tubular structure images, It is characterized in that include: Step 1, obtaining tubular structure image data; Step 2, inputting the tubular structure image data into a segmentation model for training to obtain a tubular structure image segmentation model; Step 3, inputting the target tubular structure image into the tubular structure image segmentation model for segmentation to obtain a segmentation result; The segmentation model includes a feature extraction module for extracting horizontal features, vertical features, and global features and performing encoding and decoding, a feature enhancement module for enhancing the slender features of the tubular structure, and an output module for performing image segmentation according to the extracted features. The first input end of the feature extraction module is the input end of the segmentation model, the first output end of the feature extraction module is connected to the input end of the feature enhancement module, the output end of the feature enhancement module is connected to the second input end of the feature extraction module, the second output end of the feature extraction module is connected to the input end of the output module, and the output end of the output module is the output end of the segmentation model.

2. The tubular structure image segmentation method according to claim 1, It is characterized in that The feature extraction module includes an encoder and a decoder; The input end of the encoder is the input end of the feature extraction module; The output end of the encoder is connected to the input end of the feature enhancement module; The input end of the decoder is connected to the output end of the feature enhancement module; The output end of the decoder is connected to the input end of the output module.

3. The tubular structure image segmentation method according to claim 2, It is characterized in that The encoder comprises: A first multi-view feature fusion unit, a first maximum pooling downsampling unit, a second multi-view feature fusion unit, a second maximum pooling downsampling unit, a third multi-view feature fusion unit, a third maximum pooling downsampling unit, and a fourth multi-view feature fusion unit connected in sequence; The input end of the first multi-view feature fusion unit is the input end of the encoder; The output end of the fourth multi-view feature fusion unit is connected to the input end of the feature enhancement module.

4. The tubular structure image segmentation method according to claim 3, It is characterized in that The decoder comprises: A fifth multi-view feature fusion unit, a first bilinear interpolation upsampling unit, a sixth multi-view feature fusion unit, a second bilinear interpolation upsampling unit, a seventh multi-view feature fusion unit, a third bilinear interpolation upsampling unit, and an eighth multi-view feature fusion unit connected in sequence; The input end of the fifth multi-view feature fusion unit is connected to the output end of the feature enhancement module; The output end of the eighth multi-view feature fusion unit is connected to the input end of the output module.

5. The tubular structure image segmentation method according to claim 4, It is characterized in that The first multi-view feature fusion unit, the second multi-view feature fusion unit, the third multi-view feature fusion unit, the fourth multi-view feature fusion unit, the fifth multi-view feature fusion unit, the sixth multi-view feature fusion unit, the seventh multi-view feature fusion unit, and the eighth multi-view feature fusion unit all include: The first large core snake convolution layer, the second large core snake convolution layer, the multi-scale stereo attention layer, the first convolution layer, the splicing layer, the second convolution layer, the batch normalization layer, the Relu activation layer, and the multi-scale feature enhancement layer; The input of the first large-core snake convolution layer, the input of the second large-core snake convolution layer, and the input of the multi-scale stereo attention layer are all connected to the output of the previous unit of each multi-view feature fusion unit; The output end of the first large-core snake-shaped convolutional layer is connected to the first input end of the concatenation layer; The output end of the second large-core snake-shaped convolutional layer is connected to the second input end of the concatenation layer; The output end of the multi-scale stereo attention layer is connected to the input end of the first convolutional layer, the output end of the first convolutional layer is connected to the third input end of the splicing layer, the output end of the splicing layer is connected to the input end of the second convolutional layer, the output end of the second convolutional layer is connected to the input end of the batch normalization layer, the output end of the batch normalization layer is connected to the input end of the Relu activation layer, the output end of the Relu activation layer is connected to the input end of the multi-scale feature enhancement layer, and the output end of the multi-scale feature enhancement layer is connected to the input end of the next level unit of each multi-view feature fusion unit.

6. The tubular structure image segmentation method according to claim 5, It is characterized in that The feature enhancement module includes four branch multi-scale feature enhancement units connected in sequence; An input end of a first branch multi-scale feature enhancement unit among the four branch multi-scale feature enhancement units is connected to an output end of the multi-scale feature enhancement layer in the fourth multi-view feature fusion unit; The output end of the fourth branch multi-scale feature enhancement unit in the four branch multi-scale feature enhancement units is connected to the input end of the first large-core serpentine convolution layer, the input end of the second large-core serpentine convolution layer, and the input end of the multi-scale stereo attention layer in the fifth multi-view feature fusion unit.

7. The tubular structure image segmentation method according to claim 6, It is characterized in that The first large core snake-shaped convolution layer and the second large core snake-shaped convolution layer both include: Dynamic snake convolution sublayer, depth-wise separable convolution sublayer, depth-wise separable dilated convolution sublayer, multiplier; An input end of the dynamic snake convolution sublayer in the first large core snake convolution layer is an input end of the first large core snake convolution layer; An input end of the dynamic snake convolution sublayer in the second large core snake convolution layer is an input end of the second large core snake convolution layer; A first output terminal of the dynamic serpentine convolution sublayer is connected to an input terminal of the depthwise separable convolution sublayer, and a second output terminal of the dynamic serpentine convolution sublayer is connected to a first input terminal of the multiplier; An output end of the depthwise separable convolutional sublayer is connected to an input end of the depthwise separable dilated convolutional sublayer; An output terminal of the depthwise separable dilated convolutional sublayer is connected to a second input terminal of the multiplier; The output end of the multiplier in the first large-core serpentine convolution layer is the output end of the first large-core serpentine convolution layer; The output end of the multiplier in the second large-core serpentine convolution layer is the output end of the second large-core serpentine convolution layer.

8. The tubular structure image segmentation method according to claim 7, It is characterized in that Each of the branch multi-scale feature enhancement units comprises: Low-resolution feature enhancement branch structure, high-resolution feature enhancement branch structure, residual branch structure, adder; The input end of the low-resolution feature enhancement branch structure, the first input end of the high-resolution feature enhancement branch structure, and the input end of the residual branch structure are all input ends of each of the branch multi-scale feature enhancement units; The first output end of the low-resolution feature enhancement branch structure is connected to the second input end of the high-resolution feature enhancement branch structure; The second output end of the low-resolution feature enhancement branch structure is connected to the third input end of the high-resolution feature enhancement branch structure; The output end of the high-resolution feature enhancement branch structure is connected to the first input end of the adder; The output end of the residual branch structure is connected to the second input end of the adder; The output end of the adder is the output end of each branch multi-scale feature enhancement unit.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the tubular structure image segmentation method according to any one of claims 1 to 8 is implemented.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the tubular structure image segmentation method according to any one of claims 1 to 8 is implemented.

Images