Convolutional neural network based on affinity feature fusion for vessel segmentation

Abstract

The application discloses a kind of convolutional neural network blood vessel segmentation systems based on affinity feature fusion, including feature encoding module, unsupervised affinity feature enhancement decoding module, and supervised multiscale affinity feature enhancement decoding module, through supervised multiscale and unsupervised affinity feature enhancement decoding module, simultaneously explicit and implicit extraction affinity field, establish the semantic relationship between different scale features, encode local space geometry, carry out the enhancement of segmentation feature, uniformly enhance the pixel-level precision and topological integrity and robustness of blood vessel segmentation.The application provides an end-to-end training convolutional neural network blood vessel segmentation system based on affinity feature fusion.Compared with traditional blood vessel segmentation method and many existing blood vessel segmentation systems based on deep learning, the application simultaneously optimizes pixel feature, constrains topological structure and obtains better contrast robustness by establishing the semantic relationship between pixel feature and feature.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and more particularly relates to a convolutional neural network blood vessel segmentation system based on affinity feature fusion. BACKGROUND

[0002] Blood vessel segmentation aims to obtain accurate blood vessel boundaries and complete topological structures, and needs to be more robust when facing the contrast of images. This is the key to many medical blood vessel segmentation applications today. However, most existing research methods can only solve part of the above three problems, and cannot solve the above three problems at the same time.

[0003] Most blood vessel segmentation methods pay more attention to pixel-level accuracy, and few methods pay attention to improving topological performance at the same time. For example, the paper (Hu X, Li F, Samaras D, et al. Topology-preserving deep image segmentation [J]. Advances in neural information processing systems, 2019, 32.). They use local topology explicit constraints (based on local holes and connected domains) for explicit supervision learning. Better topological integrity has been obtained. However, local topology optimization is very sensitive to complex background changes.

[0004] Further, there are also methods that combine pixel-level accuracy and topological accuracy. For example, the paper (Cheng M, Zhao K, Guo X, et al. Joint topology-preserving and feature-refinement network for curvilinear structure segmentation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 7147-7156.). They use a double-flow network structure, use an edge detection branch to establish the relationship between topology and edge connectivity, and then use the information of edge connectivity to enhance the segmentation features and improve the pixel-level and topological-level accuracy of existing blood vessel segmentation methods. However, simply using edge information to enhance features cannot represent the semantic structure relationship between local pixels, so as to better improve the topological performance. Moreover, these edge information is more susceptible to image contrast and other domain differences, and has a certain impact on the generalization performance. SUMMARY

[0005] The problem to be solved by the present application is to obtain good pixel-level accuracy, complete topological structure and more robust segmentation results when facing contrast changes for the blood vessel segmentation task. The system utilizes the local semantic relationship between pixel features established by affinity field to simultaneously encode local structure and obtain better topological integrity. It is the first blood vessel segmentation system that simultaneously realizes accurate pixel segmentation, complete topological structure and more robust segmentation.

[0006] In order to achieve the above-mentioned purpose, the present application provides a convolutional neural network blood vessel segmentation system based on affinity feature fusion, which comprises a feature encoding module, an unsupervised affinity feature enhancement decoding module and a supervised multi-scale affinity feature enhancement decoding module. The unsupervised affinity feature enhancement decoding module and the supervised multi-scale affinity feature enhancement decoding module simultaneously extract affinity field explicitly and implicitly, establish semantic relationship between features of different scales, encode local spatial geometric structure, enhance segmentation features, and uniformly enhance pixel-level accuracy, topological integrity and robustness of blood vessel segmentation, wherein:

[0007] The feature encoding module is an auto-encoding fully convolutional network with skip connection. The skip connection is connected with the corresponding decoding module through a gating attention unit to enhance features. Through the gating attention unit, the saliency of existing blood vessel features is supplemented and enhanced to obtain better feature representation.

[0008] The unsupervised affinity feature enhancement decoding module learns single-scale affinity field unsupervisedly, implicitly represents affinity relationship between pixel features, and guides segmentation feature enhancement on different resolution features.

[0009] The supervised multi-scale affinity feature enhancement decoding module learns multi-scale affinity field supervisedly, explicitly represents affinity relationship between pixel features of different scales, and guides segmentation feature enhancement.

[0010] In an embodiment of the present application, the feature encoding module is composed of an initial feature extraction submodule and a feature convolution downsampling submodule. The initial feature extraction submodule is composed of two convolution layers, and then four feature convolution downsampling submodules are connected. The feature convolution downsampling submodule is composed of two convolution layers and one max-pooling layer. Each feature convolution downsampling submodule down-samples features. The output of each part of the initial feature extraction submodule and the feature convolution downsampling submodule is connected with the corresponding decoding module through skip connection.

[0011] In an embodiment of the present application, before the input feature encoding module, the image is randomly cropped to a 256x256 size image, and the image block generates a feature map through the feature encoding module; the original [1, 256, 256] or [3, 256, 256] image is input into the feature encoding module, and a feature map is extracted through multi-layer convolution, wherein the feature dimensions output by the initial feature extraction submodule and the four-layer feature convolution downsampling submodule are [64, 128, 256, 512, 512] respectively, and the output features of each feature extraction part are [64, 256, 256], [128, 128, 128], [256, 64, 64], [512, 32, 32], [512, 16, 16] respectively, wherein the output of the submodules of the feature encoding module is connected to the corresponding decoding module through a gating attention unit, and the gating attention unit is used for blood vessel feature enhancement.

[0012] In an embodiment of the present application, the output of the feature encoding module is connected to the corresponding decoding module through a gating attention unit, and the gating attention unit is composed of a feature supplement submodule and a feature enhancement submodule; the input features of the gating attention unit are the encoding features of the feature encoding module and the predicted segmentation result is the segmentation result of the predicted segmentation filtering and enhancing the feature encoding features obtaining the enhanced features through the enhancement module is the output of the feature supplement submodule, represents the output of the feature enhancement submodule, M is a mask of relevant attention, and the supplement module mainly uses residual features to supplement the encoding features which are not present or weak, and the specific supplement module and the acquisition of the relevant residual are as follows, wherein the spatial SA and channel attention CA and the residual features are used to obtain the relevant attention mask:

[0013]

[0014]

[0015]

[0016]

[0017] For the feature enhancement submodule, the existing features are enhanced by using the predicted and M:

[0018]

[0019]

[0020] In one embodiment of the present application, the unsupervised affinity feature enhancement decoding module is composed of an unsupervised semantic segmentation feature decoding submodule, an unsupervised affinity feature decoding submodule and an unsupervised segmentation affinity feature enhancement submodule; the unsupervised semantic segmentation feature decoding submodule first extracts semantic features through a convolution layer, and simultaneously cascades the feature of the feature encoding module of the skip connection to the unsupervised semantic segmentation feature decoding submodule part, and then outputs corresponding decoding semantic features through 2 convolution layers; the unsupervised affinity feature decoding submodule cascades the corresponding semantic features output by the unsupervised semantic segmentation feature decoding submodule and the output features of the unsupervised affinity feature decoding submodule, and then outputs an 8-channel affinity field by using a convolution layer; the unsupervised affinity feature decoding submodule expresses the semantic relationship of adjacent pixels by using the output affinity field; and the unsupervised segmentation affinity feature enhancement submodule performs feature enhancement on the semantic segmentation features output by the corresponding unsupervised semantic segmentation feature.

[0021] In one embodiment of the present application, the unsupervised affinity feature decoding submodule unsupervisedly learns an 8-channel affinity field corresponding to each pixel single-scale neighborhood position, implicitly represents the affinity relationship between pixel features, uses the affinity field to guide the enhancement of local segmentation features at different positions on different resolution features, establishes local semantic relationship and encodes local structure to obtain a more complete topological structure.

[0022] In one embodiment of the present application, the unsupervised affinity feature enhancement decoding module is used to solve the problem that the affinity field cannot be explicitly obtained after down-sampling features; the unsupervised feature enhancement decoding module predicts the affinity relationship between pixels at different layers in an implicit manner, enhances semantic features at different resolutions through the unsupervised segmentation affinity feature enhancement submodule, and establishes the relationship between adjacent pixel features; the segmentation feature enhancement manner of the unsupervised segmentation affinity feature enhancement submodule is similar to that of the multi-scale segmentation affinity feature enhancement submodule in the supervised multi-scale affinity feature enhancement decoding module, and the specific feature enhancement manner is as follows: l represents the position of one of the neighborhood of pixel x, f represents the enhanced feature, and f s represents the segmentation feature output by x position, is the selected affinity, and f seg represents the segmentation feature before enhancement:

[0023]

[0024] The unsupervised affinity feature enhancement decoding module enhances the segmentation features by using the adjacent pixel features and the predicted single-scale affinity field, which establishes the semantic relationship of the adjacent pixels, encodes more abundant context and geometric information, and thus better encodes the topological information in combination with the deeper semantic features and the affinity feature extraction network, and establishes more abundant relationships.

[0025] In an embodiment of the present application, the supervised multi-scale affinity feature enhancement decoding module comprises a supervised semantic segmentation feature decoding submodule, a supervised multi-scale affinity feature decoding submodule and a multi-scale segmentation affinity feature enhancement submodule, wherein the supervised semantic segmentation feature decoding submodule and the supervised multi-scale affinity feature decoding submodule are a double-flow branch structure; the supervised semantic segmentation feature decoding submodule and the unsupervised semantic segmentation feature decoding submodule correspond to the semantic segmentation feature part, have the same structure, and are used for extracting corresponding semantic and affinity features; the corresponding semantic features output by the supervised semantic segmentation feature decoding submodule and the output of the supervised multi-scale affinity feature decoding submodule are concatenated and then output 8-channel affinity fields of multiple scales; the supervised multi-scale affinity feature decoding submodule performs feature enhancement on the multi-scale semantic relationship of each pixel expressed by the output multi-scale affinity field through the multi-scale segmentation affinity feature enhancement part.

[0026] In an embodiment of the present application, the supervised multi-scale affinity feature decoding submodule outputs the relevant affinity relationship of 8 directions of the corresponding pixel position under different scales, describes the semantic relationship between the pixel features of the same category, and the multi-scale segmentation affinity feature enhancement submodule calculates the mean value of the affinity of the same neighborhood, selects the segmentation features of the position with the affinity greater than the mean value for feature enhancement, and the specific feature enhancement mode is as follows, f s represents the enhanced feature, represents the segmentation feature of the x position, is the selected affinity, and W M represents the multi-scale adaptive weight, f seg represents the segmentation feature before enhancement:

[0027]

[0028] In an embodiment of the present application, the specific functions of the multi-scale segmentation affinity feature enhancement submodule include: first, the predicted multi-directional affinity mean value as the reference for selecting the semantic feature affinity, the average affinity represents the overall semantic relationship between the pixel x and the adjacent pixel features, and is defined as follows:

[0029]

[0030] Where N is the total number of all neighboring pixels of x, in the average affinity Under the guidance of [the relevant authority], for any position Greater than The position l is considered to have more semantic information than the position x.

[0031] Then construct a similar category location affinity. Each x is compared with its affinity fields at different positions in its eight channels. and average affinity field Similar locations have been selected and defined as

[0032]

[0033] For a selected location l, the segmentation feature f seg (x) is enhanced using the following formula:

[0034]

[0035] Where f seg (x l ) is a segmentation feature relative to the l position, W M (x) represents the weights of multi-scale adaptive affinity, using W M (x) Semantic information is obtained from the affinity at different scales and the adjacent features at different scales. Different size weights have different attention to objects at different scales to adapt to blood vessels of different sizes. Given blood vessels of different sizes, multi-scale affinity relationships are used in combination to guide context and information combination.

[0036] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0037] (1) This invention provides an end-to-end trained convolutional neural network blood vessel segmentation system based on affinity feature fusion. Compared with traditional blood vessel segmentation methods and many existing deep learning-based blood vessel segmentation systems, the method of this invention simultaneously optimizes pixel features, constrains topological structure, and obtains better contrast robustness by establishing semantic relationships between pixel features;

[0038] (2) The application provides a 2D convolution-based double-flow affinity segmentation automatic blood vessel segmentation system. Compared with other deep learning-based blood vessel segmentation features, the method learns to extract an affinity field through a network, obtains the semantic relationship between pixels, better represents the semantic relationship and structural relationship between pixels, enhances the segmentation features, increases the semantic correlation between features, establishes the geometric relationship between adjacent pixels, and improves the topological performance; meanwhile, due to the establishment of more correlations between pixels, the model is less affected by single features, and therefore has better contrast robustness and generalization ability.

[0039] (3) The affinity feature fusion-based convolutional neural network blood vessel segmentation network provided by the application is composed of a feature encoding module, an unsupervised affinity feature enhancement decoding module and a supervised multi-scale affinity feature enhancement decoding module. Based on the feature enhancement idea, the affinity field is used to enhance multi-scale pixel features and establish local pixel structural relationships, thereby improving the pixel-level accuracy and topological performance. On the XCAD public dataset, the method improves the pixel-level accuracy by 3.72% and the topological-level accuracy by 7.03% compared with the current best method. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a structural schematic diagram of the affinity feature fusion-based convolutional neural network blood vessel segmentation network in the embodiment of the application;

[0041] Figure 2 is a process schematic diagram of the feature enhancement using the affinity feature in the embodiment of the application;

[0042] Figure 3 is a description diagram of the affinity field according to the method of the application in the embodiment of the application;

[0043] Figure 4 is a result of the contrast change of the method of the application and other methods in the XCAD and DRIVE public data in the embodiment of the application, which reflects the performance accuracy of the method of the application when facing different contrast changes; wherein Figure 4 (a) is the F1 index of different methods when facing different contrasts on the XCAD dataset; Figure 4 (b) is the Quality index of different methods when facing different contrasts on the XCAD dataset; Figure 4 (c) is the F1 index of different methods when facing different contrasts on the DRIVE dataset; Figure 4 (d) is the Quality index of different methods when facing different contrasts on the DRIVE dataset. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0045] In order to solve the problems existing in the prior art, the present application provides a convolutional neural network blood vessel segmentation system based on affinity feature fusion, which comprises a feature encoding module, an unsupervised affinity feature enhancement decoding module, and a supervised multi-scale affinity feature enhancement decoding module. The affinity field is extracted explicitly and implicitly by the supervised multi-scale and unsupervised affinity feature enhancement decoding modules, the semantic relationship between different scale features is established, the local spatial geometric structure is encoded, the segmentation feature is enhanced, and the pixel-level precision, topological integrity and robustness of the blood vessel segmentation are uniformly enhanced, wherein:

[0046] The feature encoding module is an auto-encoding fully convolutional network with skip connection, and the skip connection is connected with the corresponding decoding module through a gating attention unit for feature enhancement. Through the gating attention unit, the saliency of the existing blood vessel features is supplemented and enhanced to obtain better feature representation.

[0047] Specifically, the feature encoding module is composed of an initial feature extraction submodule and a feature convolution downsampling submodule. The initial feature extraction submodule is composed of two convolution layers, and then four feature convolution downsampling submodules are connected. The feature convolution downsampling submodule is composed of two convolution layers and one max-pooling layer. Each feature convolution downsampling submodule down-samples the features. For the initial feature extraction and the feature convolution downsampling submodule, the output of each part is connected to the corresponding decoding module using skip connection.

[0048] Further, before inputting the feature encoding module, the image is randomly cropped into an image with a size of 256x256. The image block generates a feature map through the feature encoding module.

[0049] Further, the original [1, 256, 256] or [3, 256, 256] image input feature encoding module is extracted by multi-layer convolution feature map, wherein the initial feature extraction sub-module and the four-layer feature convolution downsampling sub-module output feature dimensions are [64, 128, 256, 512, 512], respectively, and the output features of each feature extraction part are [64, 256, 256], [128, 128, 128], [256, 64, 64], [512, 32, 32], [512, 16, 16], respectively, wherein the output of the sub-module of the feature encoding module is connected with the corresponding decoding module by the gating attention unit, and the gating attention unit is used for blood vessel feature enhancement.

[0050] Further, the output of the feature encoding module is connected with the corresponding decoding module by the gating attention unit, and the gating attention unit is composed of a feature supplementing sub-module and a feature enhancing sub-module; the input feature of the gating attention unit is the encoding feature of the feature encoding module and the predicted segmentation result is the predicted segmentation result the filtered enhanced feature encoding feature the enhanced feature obtained by the enhancement module is the output of the feature supplementing sub-module, the output of the feature enhancing sub-module, M is a mask of relevant attention, and the supplementing module mainly uses residual features to supplement the encoding feature which is weak or non-existent in the feature. The specific supplementing module and the relevant residual are obtained as follows, wherein the spatial and channel attention and the residual feature are used to obtain the relevant attention mask:

[0051]

[0052]

[0053]

[0054]

[0055] For the feature enhancing sub-module, the existing features are enhanced by using the predicted and M:

[0056]

[0057]

[0058] The unsupervised affinity feature enhancement decoding module unsupervisedly learns a single-scale affinity field, implicitly represents the affinity relationship between pixel features, and guides segmentation feature enhancement on different resolution features.

[0059] Specifically, the unsupervised affinity feature enhancement decoding module is composed of an unsupervised semantic segmentation feature decoding submodule, an unsupervised affinity feature decoding submodule, and an unsupervised segmentation affinity feature enhancement submodule. The unsupervised semantic segmentation feature decoding submodule first extracts semantic features through a convolution layer, and then cascades the features of the feature encoding module connected by a skip connection to the unsupervised semantic segmentation feature decoding submodule part. Then, the corresponding decoded semantic features are output through two convolution layers. The unsupervised affinity feature decoding submodule cascades the corresponding semantic features output by the unsupervised semantic segmentation feature decoding submodule and the output features of the unsupervised affinity feature decoding submodule, and then outputs an 8-channel affinity field using a convolution layer. The unsupervised affinity feature decoding submodule expresses the semantic relationship between adjacent pixels using the output affinity field. Then, the unsupervised segmentation affinity feature enhancement submodule enhances the semantic segmentation features output by the unsupervised semantic segmentation feature decoding submodule.

[0060] Further, the unsupervised affinity feature decoding submodule unsupervisedly learns an 8-channel affinity field corresponding to each pixel single-scale neighborhood position, implicitly represents the affinity relationship between pixel features, and uses the affinity field to guide the enhancement of local segmentation features at different resolution features, establishes local semantic relationships, and encodes local structures to obtain more complete topological structures.

[0061] Further, the unsupervised affinity feature enhancement decoding module is used to solve the problem that the affinity field cannot be explicitly obtained after down-sampling the features. The unsupervised feature enhancement decoding module predicts the affinity relationship between pixels at different layers in an implicit manner, enhances the semantic features at different resolutions through the unsupervised segmentation affinity feature enhancement submodule, and establishes the relationship between adjacent pixel features. The segmentation feature enhancement manner of the unsupervised segmentation affinity feature enhancement submodule is similar to that of the multi-scale segmentation affinity feature enhancement submodule in the supervised multi-scale affinity feature enhancement decoding module. The specific feature enhancement manner is as follows, where l represents the position of the neighborhood of pixel x, f represents the enhanced feature, and f s represents the segmentation feature output by x position, is the selected affinity, and f seg represents the segmentation feature before enhancement:

[0062]

[0063] The unsupervised affinity feature enhancement decoding module enhances the segmentation features by using the adjacent pixel features and the predicted single-scale affinity field, which establishes the semantic relationship of adjacent pixels, encodes more abundant context and geometric information, and thus better encodes the topological information in combination with the deeper semantic features and the affinity feature extraction network, and establishes more abundant relationships.

[0064] The supervised multi-scale affinity feature enhancement decoding module learns the multi-scale affinity field in a supervised manner, explicitly represents the affinity relationship between pixel features of different scales, and guides the segmentation feature enhancement.

[0065] Further, the supervised multi-scale affinity feature enhancement decoding module comprises a supervised semantic segmentation feature decoding submodule, a supervised multi-scale affinity feature decoding submodule, and a multi-scale segmentation affinity feature enhancement submodule, wherein the supervised semantic segmentation feature decoding submodule and the supervised multi-scale affinity feature decoding submodule are a double-flow branch structure; the supervised semantic segmentation feature decoding submodule and the unsupervised semantic segmentation feature decoding submodule correspond to the semantic segmentation feature part, have the same structure, and are used to extract corresponding semantic and affinity features; the corresponding semantic features output by the supervised semantic segmentation feature decoding submodule and the output of the supervised multi-scale affinity feature decoding submodule are concatenated and then output 8-channel affinity fields of multiple scales; the supervised multi-scale affinity feature decoding submodule performs feature enhancement on the multi-scale semantic relationship of each pixel represented by the output multi-scale affinity field through the multi-scale segmentation affinity feature enhancement part.

[0066] Further, the supervised multi-scale affinity feature decoding submodule outputs the relevant affinity relationship of 8 directions (up, down, left, right, top left, bottom left, top right, bottom right) of the corresponding pixel position under different scales, describes the semantic relationship between the pixel features of the same category, and the multi-scale segmentation affinity feature enhancement submodule calculates the mean value of the affinity of the same neighborhood, selects the segmentation features of the position with an affinity greater than the mean value for feature enhancement, and the specific feature enhancement method is as follows, f s represents the enhanced feature, represents the segmentation feature of the x position, is the selected affinity, W M represents the multi-scale adaptive weight, f seg represents the segmentation feature before enhancement:

[0067]

[0068] Further, the supervised multi-scale affinity feature decoding submodule learns a multi-scale affinity field supervisedly, explicitly represents the affinity between pixel features of different scales, uses the affinity relationship learned by the existing semantic segmentation label to explicitly guide the enhancement of the local multi-scale segmentation feature, establishes the local semantic relationship under different scales of 3x3, 5x5 and 13x13 to adapt to the blood vessel structure of different scales, and thus obtains more accurate and robust blood vessel segmentation.

[0069] Further, the multi-scale affinity field output by the supervised multi-scale affinity feature decoding submodule considers the semantic relationship between pixels, adapts to different blood vessels and backgrounds by establishing an affinity field of different scales, and the affinity is defined as follows: for each semantic segmentation G s , is the set of adjacent 8-neighborhood pixels of x, and l represents one of the eight directions of the neighborhood, i.e. left, right, up, down, lower left, upper left, upper right and lower right of x. For specific cases, three different sizes of affinity with adaptive weights are set, for example, 3x3, 9x9 and 15x15, and x l is a pixel located at the relative position of x, and if the pixels x l belong to the same category, the affinity A l (x) is set to 1, and the affinity A l (x) of each position is as follows:

[0070]

[0071] Further, the predicted segmentation result and the affinity result are respectively constrained with the real label, and the overall objective function is as follows:

[0072]

[0073]

[0074]

[0075]

[0076] where X A and X are the multi-scale affinity field and the semantic segmentation graph, y A (x) and y s (x) are the predicted affinity field and the predicted segmentation result respectively, G A (x) is the label of the affinity field composed of the affinity A l (x) represented by each direction at x, |*| represents the number of pixels, ||*|| represents the calculation modulus, and λ bis a hyper-parameter for balancing the loss function, minimizing Help the model to extract better multi-scale semantic relationship of different sizes and backgrounds.

[0077] Further, the specific functions of the multi-scale segmentation affinity feature enhancer module are as follows: first, the predicted multi-directional affinity is calculated As a reference for the selection of semantic feature affinity, the average affinity represents the overall semantic relationship between pixel x and its adjacent pixel features, and is defined as follows:

[0078]

[0079] where N is the total number of all adjacent pixels of x, and the average affinity Under the guidance of the average affinity The position l where is greater than is considered to have more similar semantic information with the feature of the position x;

[0080] Then a similar class position affinity is constructed Each x is selected by comparing its eight-channel different position affinity field

[0081]

[0082] For the selected position l, the segmentation feature f seg (x) is enhanced by the following formula:

[0083]

[0084] where f seg (x l ) is the segmentation feature relative to the position l, W M (x) represents the weight of the multi-scale adaptive affinity, and W M (x) is used to obtain semantic information from adjacent features at different scales using different scale affinities, and different weights of different sizes have different attention degrees for different scale objects, so as to adapt to different size blood vessels, and given different size blood vessels, the multi-scale affinity relationship is used to guide the context and combined information.

[0085] The method of the present application is further described in detail in combination with specific embodiments, such as Figure 1As shown, the structure of the affinity feature fusion-based convolutional neural network blood vessel segmentation system provided by the application is shown in the schematic diagram, and the system mainly has three components: a feature encoding module, an unsupervised affinity feature enhancement decoding module, and a supervised multi-scale affinity feature enhancement decoding module. The three parts will be introduced below:

[0086] 1. Feature encoding module

[0087] The feature encoding module is composed of an initial feature extraction sub-module and a feature convolution downsampling sub-module. The initial feature extraction sub-module is composed of two convolution layers, and then four feature convolution downsampling sub-modules are composed. The feature convolution downsampling sub-module is composed of two convolution layers and one max-pooling layer. Each feature convolution downsampling sub-module down-samples the features. For the initial feature extraction and feature convolution downsampling sub-modules, the output of each sub-module is connected to the corresponding decoding module using a jump connection.

[0088] Before inputting the feature encoding module, the image is randomly cropped into a 256x256 size image, and the image block generates a feature map through the feature encoding module.

[0089] The original [1, 256, 256] or [3, 256, 256] image is input into the feature extraction network, and the feature map is extracted through multiple layers of convolution, wherein the feature dimensions output by the initial feature extraction sub-module and the four-layer feature convolution downsampling sub-module are [64, 128, 256, 512, 512]. The output features of each sub-module are [64, 256, 256], [128, 128, 128], [256, 64, 64], [512, 32, 32], [512, 16, 16], respectively, wherein the output of each sub-module is connected to the corresponding decoding module through a jump link with a gated attention unit, and the gated attention unit is used to filter the blood vessel features.

[0090] The output of the feature encoding module is connected to the corresponding decoding module through a jump link with a gated attention unit. The gated attention unit is composed of a feature supplement sub-module and a feature enhancement sub-module. The input features of the gated attention unit are the encoding features of each layer sub-module of the feature encoding module and the predicted segmentation result. The encoding features will obtain enhanced features through the enhancement sub-module. is the output of the feature supplement sub-module, indicates the output of the feature enhancement sub-module. M is a mask of relevant attention. The supplement sub-module mainly uses residual features to supplement the features that do not exist or are weak in the encoding features. The specific supplement sub-module and the relevant residual are obtained as follows, wherein the spatial and channel attention and the residual features are used to obtain the relevant attention mask:

[0091]

[0092]

[0093]

[0094]

[0095] For feature enhancement module to enhance the existing features, using the predicted and M.

[0096]

[0097]

[0098] 2、Unsupervised affinity feature enhancement decoding module

[0099] Unsupervised affinity feature enhancement decoding module, by unsupervised semantic segmentation feature decoding sub module, unsupervised affinity feature decoding sub module and unsupervised segmentation affinity feature enhancement sub module. Unsupervised semantic segmentation feature decoding sub module, first through the convolution layer further extracts semantic features, again through the jump connection cascade feature coding module corresponding features, and output corresponding semantic features through 2 layer convolution layer. Unsupervised semantic segmentation feature decoding sub module outputs corresponding semantic features. Unsupervised affinity feature decoding sub module cascades corresponding semantic features and outputs 8 channel affinity field. Unsupervised affinity feature decoding sub module module will express each pixel corresponding to the semantic relationship of the adjacent pixel of the output affinity field, and the corresponding semantic segmentation feature is enhanced by the unsupervised segmentation affinity feature enhancement part.

[0100] Unsupervised affinity feature enhancement decoding module is used to solve the problem that affinity field cannot be obtained explicitly after down-sampling feature. Unsupervised affinity feature decoding sub module predicts the affinity relationship between pixels at different layers in an implicit way. Through the unsupervised segmentation affinity feature enhancement sub module, the relationship between the features of adjacent pixels is established. Unsupervised segmentation affinity feature enhancement sub module is similar to supervised segmentation affinity feature enhancement sub module, which constructs a similar class position affinity field For different position segmentation feature enhancement:

[0101]

[0102] The unsupervised segmentation affinity feature enhancement submodule enhances the segmentation features by using the features of adjacent pixels and predicting a single-scale affinity field. This establishes the semantic relationship of adjacent pixels, which is used to encode more abundant context and geometric information. This can better encode the topological information by combining the deeper semantic features and the affinity feature extraction network, and establish more abundant relationships.

[0103] 3. The supervised multi-scale affinity feature enhancement decoding module

[0104] The supervised multi-scale affinity feature decoding submodule outputs the relevant affinity relationships of the corresponding pixel positions (up, down, left, right, top-left, bottom-left, top-right, bottom-right) in different scales. The semantic relationship of whether it is in the same category is also output. The multi-scale segmentation affinity feature enhancement submodule calculates the mean affinity of the same neighborhood and selects the segmentation features of the positions with greater affinity than the mean affinity for feature enhancement. The specific feature enhancement method is as follows, f s (x) represents the enhanced features, f seg (x) represents the segmentation features of the x position, W M (x) represents the adaptive weights of multiple scales.

[0105]

[0106] The supervised multi-scale affinity feature enhancement decoding module is composed of a supervised semantic segmentation feature decoding submodule, a supervised multi-scale affinity feature decoding submodule, and a multi-scale segmentation affinity feature enhancement submodule. The supervised semantic segmentation feature decoding submodule and the supervised multi-scale affinity feature decoding submodule have a double-flow branch structure. The supervised semantic segmentation feature decoding submodule and the unsupervised semantic segmentation feature decoding submodule have the same structure and extract corresponding semantic and affinity features. The corresponding semantic features output by the supervised semantic segmentation feature decoding submodule are concatenated with the output of the supervised multi-scale affinity feature decoding submodule to output a multi-scale 8-channel affinity field. The multi-scale semantic relationship of each pixel expressed by the multi-scale affinity field output by the supervised multi-scale affinity feature decoding submodule is enhanced by the multi-scale segmentation affinity feature enhancement submodule.

[0107] The multi-scale affinity field output by the supervised multi-scale affinity feature decoding submodule considers the semantic relationship between pixels and adapts to different blood vessels and backgrounds by establishing different scale affinity fields, as shown in Figure 3 The definition of affinity is as follows: for each semantic segmentation G s , is the set of adjacent 8-neighborhood pixels of x. l represents one of the eight directions of the neighborhood left, right, up, down, left-down, left-up, right-up, right-down. For specific cases, set 3 different sizes of affinity with adaptive weights, for example, 3x3, 9x9, 15x15. Define x l is the pixel located at the relative position of x, if the pixel x and x l belong to the same class, then the affinity A l (x) is set to 1, for each position of the affinity A l (x) is as follows:

[0108]

[0109] The predicted segmentation result and the affinity result are respectively constrained with the real label. The overall objective function is as follows:

[0110]

[0111]

[0112]

[0113]

[0114] where X A and X are the multi-scale affinity field and the semantic segmentation map. y A (x) and y s (x) are the predicted affinity field and the predicted segmentation result respectively. G A (x) is the label of the affinity field at x composed of the affinity A l (x) with each direction representation. |*| represents the number of pixels, ||*|| represents the calculation modulus, λ b is a hyperparameter for balancing the loss function. Minimizing helps the model to extract better multi-scale semantic relationships of different sizes and backgrounds.

[0115] Further, as shown in Figure 2 , it is a process schematic diagram for using affinity features to enhance features in the embodiment of the present application. The specific functions of the multi-scale segmentation affinity feature enhancement sub-module are as follows. First, the mean value of the predicted multi-direction affinity is calculated as the reference for selecting semantic feature affinity. The average affinity represents the overall semantic relationship between the pixel x and its adjacent pixel features, and is defined as follows:

[0116]

[0117] where N is the total number of all adjacent pixels of x, and in the average affinity under the guidance of the position l greater than The position l is considered to have more similar semantic information with the x position feature. Then, the feature enhancement selects more features that are likely to be the same category. To this end, a similar category position affinity field is constructed Each x is compared to its eight-channel position affinity field and the average affinity field The similar position category for x is defined as

[0118]

[0119]

[0120] For the selected position l, the segmentation feature f seg (x) is enhanced by the following formula:

[0121]

[0122] Where f seg (x l ) is the segmentation feature relative to the position l, and W M {x) represents the weight of the multi-scale adaptive affinity. Using W M (x) to obtain semantic information from adjacent features at different scales. Different sizes of weights have different attention to different sizes of objects to adapt to different sizes of blood vessels. Given different sizes of blood vessels, the multi-scale affinity relationship is used to guide the context and combined information.

[0123] As shown in Figure 4 , the results of the contrast change of the method of the present application and other methods in the XCAD and DRIVE data are compared, which reflect the performance and accuracy of the method of the present application in the face of different contrast changes; wherein Figure 4 (a) is the F1 index of different methods in the face of different contrast in the XCAD data set; Figure 4 (b) is the Quality index of different methods in the face of different contrast in the XCAD data set; Figure 4 (c) is the F1 index of different methods in the face of different contrast in the DRIVE data set; Figure 4 Figure 4(d) In the DRIVE dataset, the different methods are compared in different contrast. Compared with the current blood vessel segmentation method, the affinity feature fusion based convolutional neural network blood vessel segmentation system proposed in the application uses affinity to integrate local features, so that the feature learning fuses features of the same category at different positions, reduces the dependence on single position features, and on the XCAD and DRIVE public datasets, the application is compared with the most advanced method in the pixel level segmentation index F1 and the topological performance index Quality.

[0124] As shown in Table 1, the prediction accuracy results of the application method and other methods in the application embodiment under the XCAD and DRIVE two different blood vessels are shown in the table.

[0125] Table 1 Comparison of the affinity feature fusion based convolutional neural network blood vessel segmentation system proposed in the application with the most advanced method on the XCAD and DRIVE two public datasets.

[0126]

[0127] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the application, and is not intended to limit the application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An affinity feature fusion based convolutional neural network blood vessel segmentation system, characterized in that, The system comprises a feature encoding module, an unsupervised affinity feature enhancement decoding module, and a supervised multi-scale affinity feature enhancement decoding module, which simultaneously extracts affinity fields explicitly and implicitly through the supervised multi-scale and unsupervised affinity feature enhancement decoding modules, establishes semantic relationships between features of different scales, encodes local spatial geometry, enhances segmentation features, and uniformly enhances the pixel-level accuracy, topological integrity, and robustness of vascular segmentation. The feature encoding module is an auto-encoding fully convolutional network with skip connections, which are connected to the corresponding decoding modules through gating attention units for feature enhancement. The gating attention units supplement and enhance the saliency of existing vascular features to obtain better feature representation. The unsupervised affinity feature enhancement decoding module learns single-scale affinity fields unsupervisedly, implicitly represents affinity relationships between pixel features, and guides segmentation feature enhancement on different resolution features. The supervised multi-scale affinity feature enhancement decoding module learns multi-scale affinity fields supervisedly, explicitly represents affinity relationships between pixel features of different scales, and guides segmentation feature enhancement.

2. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 1, wherein, The feature encoding module is composed of an initial feature extraction submodule and feature convolution downsampling submodules. The initial feature extraction submodule is composed of two convolution layers, and then four feature convolution downsampling submodules are connected. Each feature convolution downsampling submodule is composed of two convolution layers and one max-pooling layer. Each feature convolution downsampling submodule down-samples the features. The output of each part of the initial feature extraction and feature convolution downsampling submodules is connected to the corresponding decoding module through skip connections.

3. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 1 or 2, wherein, Before inputting the feature encoding module, the image is randomly cropped to a size of 256x256. The image block generates a feature map through the feature encoding module. The original [1, 256, 256] or [3, 256, 256] image is input into the feature encoding module, and the feature map is extracted through multiple convolution layers. The feature dimensions output by the initial feature extraction submodule and the four feature convolution downsampling submodules are [64, 128, 256, 512, 512], respectively. The output features of each feature extraction part are [64, 256, 256], [128, 128, 128], [256, 64, 64], [512, 32, 32], and [512, 16, 16], respectively. The output of each submodule of the feature encoding module is connected to the corresponding decoding module through skip connections with gating attention units, which enhances the vascular features using the gating attention units.

4. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 1 or 2, wherein, The output of the feature encoding module is connected to the corresponding decoding module by a gating attention unit, which is composed of a feature supplementing submodule and a feature enhancing submodule; the input feature of the gating attention unit is the encoding feature of the feature encoding module and the predicted segmentation result The encoding feature The feature obtained by the enhancing module is enhanced is the output of the feature supplementing submodule, is the segmentation result of the predicted segmentation The enhanced feature is filtered, represents the output of the feature enhancing submodule, M is a mask of relevant attention, and the supplementing module mainly utilizes residual features The supplementing encoding feature The feature is weak or does not exist in the feature, and the specific supplementing module and the relevant residual are obtained as follows: the relevant attention mask is obtained by utilizing spatial SA and channel attention CA and residual features: For the feature enhancer module to enhance existing features, the predicted and M:

5. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 1 or 2, wherein, The unsupervised affinity feature enhancement decoding module is composed of an unsupervised semantic segmentation feature decoding submodule, an unsupervised affinity feature decoding submodule and an unsupervised segmentation affinity feature enhancement submodule; the unsupervised semantic segmentation feature decoding submodule firstly extracts semantic features through a convolution layer, and simultaneously cascades the feature of the feature encoding module of the jump connection to the unsupervised semantic segmentation feature decoding submodule part, and then outputs corresponding decoding semantic features through 2 convolution layers; the unsupervised affinity feature decoding submodule cascades the corresponding semantic features output by the unsupervised semantic segmentation feature decoding submodule and the output features of the unsupervised affinity feature decoding submodule, and then outputs an 8-channel affinity field by using a convolution layer; the unsupervised affinity feature decoding submodule expresses the semantic relationship of adjacent pixels by using the output affinity field; and the unsupervised segmentation affinity feature enhancement submodule performs feature enhancement on the semantic segmentation features output by the corresponding unsupervised semantic segmentation feature.

6. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 5, wherein, The unsupervised affinity feature decoding submodule unsupervisedly learns an 8-channel affinity field corresponding to each single-scale neighborhood position of each pixel, implicitly represents the affinity relationship between pixel features, uses the affinity field to guide the enhancement of local segmentation features at different positions on different resolution features, establishes local semantic relationships and encodes local structures to obtain more complete topological structures.

7. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 5, wherein, The unsupervised affinity feature enhancement decoding module is used to solve the problem that the affinity field cannot obtain the feature after down-sampling. The unsupervised feature enhancement decoding module predicts the affinity relationship between pixels in different layers in an implicit manner. The unsupervised segmentation affinity feature enhancement submodule enhances the semantic features of different resolutions, and establishes the relationship between the features of adjacent pixels. The segmentation feature enhancement manner of the unsupervised segmentation affinity feature enhancement submodule is similar to that of the multi-scale segmentation affinity feature enhancement submodule in the supervised multi-scale affinity feature enhancement decoding module. The specific feature enhancement manner is as follows: l represents the neighborhood of pixel x wherein one of the positions, f s represents the enhanced feature, and f is the selected affinity, f seg represents the segmentation feature before enhancement: The unsupervised affinity feature enhancement decoding module enhances the segmentation features by using adjacent pixel features and a predicted single-scale affinity field, which establishes the semantic relationship of adjacent pixels, encodes more abundant context and geometric information, and thus better encodes topological information in combination with deeper semantic features and an affinity feature extraction network, and establishes more abundant relationships.

8. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 1 or 2, wherein, The supervised multi-scale affinity feature enhancement decoding module is composed of a supervised semantic segmentation feature decoding submodule, a supervised multi-scale affinity feature decoding submodule and a multi-scale segmentation affinity feature enhancement submodule, wherein the supervised semantic segmentation feature decoding submodule and the supervised multi-scale affinity feature decoding submodule are a double-flow branch structure; the corresponding semantic segmentation feature parts of the supervised semantic segmentation feature decoding submodule and the unsupervised semantic segmentation feature decoding submodule have the same structure and are used for extracting corresponding semantic and affinity features; the corresponding semantic features output by the supervised semantic segmentation feature decoding submodule are cascaded with the output of the supervised multi-scale affinity feature decoding submodule, and then an 8-channel affinity field of multiple scales is output; the supervised multi-scale affinity feature decoding submodule expresses the multi-scale semantic relationship of each pixel by using the output multi-scale affinity field, and performs feature enhancement through the multi-scale segmentation affinity feature enhancement part.

9. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 8, wherein, The supervised multi-scale affinity feature decoding submodule outputs the relevant affinity relationships of the corresponding pixel positions in 8 directions under different scales, describes the semantic relationship between the pixel features of the same category, and the multi-scale segmentation affinity feature enhancement submodule calculates the mean value of the affinity of the same neighborhood, selects the segmentation features of the positions with the affinity greater than the mean value for feature enhancement, and the specific feature enhancement method is as follows, f s indicates the enhanced features, and indicates the segmentation features at the x position, is the selected affinity, and W M indicates the adaptive weight of the multi-scale, and f seg indicates the segmentation features before enhancement:

10. The affinity feature-fusion-based convolutional neural network blood vessel segmentation system of claim 8, wherein, The specific functions of the multi-scale segmentation affinity feature enhancement submodule include: First, the predicted multi-direction The mean of affinities As a reference for the selection of semantic feature affinities, the mean affinity represents the overall semantic relationship between a pixel x and its neighboring pixel features, defined as follows: where N is the total number of all neighboring pixels of x, and the average affinity Under the guidance of the average affinity greater than the position l is considered to have more similar semantic information with the position feature of x. A similarity class position affinity is then constructed Each x is compared to its eight channel different position affinity fields and the average affinity field The selected similar positions are defined as For a selected position / , the segmentation feature f seg (x) is enhanced by the following equation: where f seg (x l ) is the segmentation feature of the relative l position, W M (x) represents the weight of multi-scale adaptive affinity, using W M (x) to obtain semantic information of adjacent features at different scales, and different weights of different sizes have different attention degrees for different size objects, so as to adapt to blood vessels of different sizes, and given blood vessels of different sizes, the multi-scale affinity relationship is used to guide the context and combined information.

Images