A small-sample fundus blood vessel image segmentation method based on nested structure
By using a nested fundus blood vessel segmentation network model and employing a prototype extraction layer and a masked average pooling layer, the problem of insufficient accuracy and generalization ability in the segmentation of small sample fundus blood vessel images is solved, and high-precision automated segmentation is achieved.
Patent Information
- Application Number
- CN202410286686.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-03-13
AI Technical Summary
In the field of medical image segmentation, especially in the segmentation of fundus blood vessel images, existing technologies struggle to achieve high-precision automated segmentation with small sample sizes. Traditional methods rely on feature design and have poor generalization ability, while deep learning-based methods suffer from insufficient training due to the scarcity of samples, resulting in inadequate segmentation accuracy and generalization ability.
A nested retinal vessel segmentation network model is adopted, including a prototype extraction layer, a masked average pooling layer, and a segmentation head. Through multi-layer nested feature extraction and masked average pooling, the model's ability to extract prototype feature vectors is improved. The model is trained using an N-way K-shot classification task to adapt to small sample datasets.
It improves the segmentation accuracy and generalization ability of the model in the case of small samples, solves the problems of strong sample dependence and poor accuracy, and achieves more efficient segmentation of fundus blood vessel images.
Smart Images

Figure CN118229972B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning technology and image processing technology, specifically to a small-sample fundus blood vessel image segmentation method based on nested structures. Background Technology
[0002] With the increasing global population aging, diabetes, and lifestyle changes, glaucoma, diabetic retinopathy (DR), and age-related macular degeneration (AMD) have become the leading causes of blindness in the elderly. Studies on the correlation between these diseases and the morphological structure of the retinal vascular system have shown a close relationship between the retinal vascular system and various diseases such as hypertension, stroke, and cardiovascular disease. Early detection and diagnosis are of paramount importance for both doctors and patients. The dendritic structure of the retinal vessels is extremely complex. Uneven contrast and brightness distribution issues during fundus camera acquisition make vessel identification and extraction difficult. Manual segmentation of retinal and fundus vessels is a time-consuming, laborious, and tedious task for doctors. There is an urgent need for an automated segmentation method to assist in this process. The development of intelligent algorithms has enabled computers to quickly complete fundus vessel segmentation tasks, greatly reducing the annotation burden on ophthalmologists and improving their work efficiency.
[0003] Currently, automatic segmentation methods for fundus vessel images are mainly divided into two categories: traditional fundus vessel segmentation algorithms and deep learning-based segmentation algorithms. Traditional fundus vessel segmentation algorithms mainly rely on digital image processing techniques, most of which employ rule-based methods, including matched filtering, multi-scale analysis, vessel tracking, morphological processing, and region growing. Their advantages include no need for additional dataset annotation, relatively fast speed, and low computational complexity. However, traditional methods suffer from over-reliance on feature design and extensive manual parameter tuning, lacking the ability to automatically extract features, resulting in lower segmentation accuracy and poor generalization ability. Deep learning-based segmentation algorithms, with their superior feature extraction capabilities, have gradually replaced traditional segmentation methods. Among them, U-Net-based segmentation models have been widely used in medical image segmentation. However, multiple downsampling in U-Net can lead to distortion of detail information. Unet++ optimizes its structure through nesting and pruning; dense skip connections better preserve detail information at different network depths, resulting in stronger feature extraction capabilities.
[0004] However, deep learning-based segmentation models typically require a large number of samples for training. In the field of medical image segmentation, due to factors such as patient privacy and hospital policies, obtaining samples is difficult and the quantity is scarce, making it difficult to meet the training requirements of deep learning models. Currently, there are few methods for segmenting images containing vascular structures with small sample sizes, and existing small sample models are unable to perform fine segmentation of vascular structures.
[0005] Therefore, how to effectively segment fundus vascular structures in medical images with a small sample size is a problem that urgently needs to be solved by technicians in this field. Summary of the Invention
[0006] To address the shortcomings of the existing technologies, this invention provides a small-sample fundus blood vessel image segmentation method based on a nested structure, which improves the model's ability to extract prototype feature vectors and enhances the model's segmentation accuracy in small-sample situations, thereby solving the problems of strong model dependence on samples, poor accuracy, and weak generalization ability in the existing technologies.
[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0008] A nested structure-based few-sample fundus blood vessel image segmentation method is proposed, which acquires the fundus blood vessel image to be processed, inputs it into a pre-trained fundus blood vessel segmentation network model, and obtains the blood vessel region segmentation result of the fundus blood vessel image to be processed.
[0009] The fundus vessel segmentation network model includes a prototype extraction layer, a masked average pooling layer, and a segmentation head. The prototype extraction layer is used to perform multi-layer nested feature extraction on the input fundus vessel image to obtain a prototype feature map of the fundus vessel image. The masked average pooling layer is used to extract the prototype feature vector corresponding to the prototype feature map according to a set segmentation category mask. The segmentation head is used to perform segmentation processing on the fundus vessel image according to the prototype feature vector to obtain the fundus vessel image region segmentation result.
[0010] As a preferred embodiment, the prototype extraction layer includes n cascaded stages and n levels of feature recovery units, wherein any nth stage... The feature recovery unit of the stage includes a context feature refinement module and a sequentially connected feature recovery module. A nested detail recovery module, The fundus blood vessel image input to the prototype extraction layer is used as the input to the first-stage feature recovery unit. The output of the context feature refinement module in each stage feature recovery unit is processed by convolutional parallel pooling and then used as the input to the next stage feature recovery unit. The prototype feature map output from the last nth layer of each stage feature recovery unit is used as the prototype feature map extracted from the fundus blood vessel image, thus obtaining the prototype feature map corresponding to the fundus blood vessel image. ;
[0011] Any of the prototype extraction layers The prototype feature map output of any j-th layer in the feature recovery unit of the stage. This can be expressed by the following formula:
[0012] ;
[0013] ;
[0014] ;
[0015] in, This is the input to the feature recovery unit in the i-th stage of the prototype extraction layer. This indicates the processing of the context feature refining module. This indicates the processing of nested detail recovery modules at iteration k-1. This indicates convolutional parallel pooling processing;
[0016] For any output of layer j in stage i of the prototype extraction layer Represented as:
[0017] ;
[0018] Among them, satisfying , This represents the ReLU activation function operation.
[0019] As a preferred embodiment, the processing procedure of the context feature refinement module includes:
[0020] The input feature map is expanded by performing a 1×1 convolution to obtain an expanded feature map. Then, the expanded feature map is sequentially processed by a 3×3 convolution and a 1×1 convolution to extract and aggregate context features, resulting in an aggregated feature map. Next, the aggregated feature map is sequentially processed by a channel attention module and a spatial attention module to extract channel attention features and spatial attention features. Finally, the extracted feature map is added element-wise to the expanded feature map and activated to obtain the output feature map.
[0021] As a preferred embodiment, the convolutional parallel pooling process includes:
[0022] The input feature map is subjected to max pooling and combined convolution processing respectively. In the combined convolution processing, the input feature map is first expanded by 5×5 convolution, and then adjusted by 2×2 convolution. Then, the max pooling result map and the combined convolution result map are added element by element to obtain the output feature map.
[0023] As a preferred embodiment, for any nested detail recovery module at the j-th layer in the i-th stage of the prototype extraction layer, the processing procedure includes:
[0024] First, process the input feature map Dense upsampling convolution is performed to partition and map the input feature map into... The dimensions are: height, width, and number of channels. The feature submap is then compared with the feature submaps located at the j-th layer in previous stages. to After concatenation, information is extracted and fused through contextual feature refinement. The fused result is then compared with the feature sub-graph located at the j-th layer of the previous stage. After element-wise addition, the feature map is obtained as the output. ;
[0025] The dense upsampling convolution process involves first performing a 3×3 convolution on the input feature map, and then dividing the convolutionally processed feature map into... The dimensions are: height, width, and number of channels. A subset of feature maps, in which , where r is the upsampling factor. For the target number of channels, Given the number of channels in the input feature map, a subset of the feature map is finally mapped to... The dimensions are: height, width, and number of channels. The feature subgraph.
[0026] As a preferred approach, the final nth layer output of any i-th stage in the prototype extraction layer... Afterwards, it is further processed by the weak information refinement module before being passed to the nth layer of the (i+1)th stage; the processing of the weak information refinement module includes:
[0027] The input feature map is subjected to information activation processing and spatial attention extraction processing respectively. In the information activation processing, the input feature map is first dynamically linearly activated using a linear activation function, then channel information is activated through a 3×3 convolution and mapped by a sigmoid function to obtain a probability map. The probability map is then multiplied with the input feature map to obtain the corresponding information activation processing result map. Then, the information activation processing result map and the spatial attention extraction processing result map are added element-wise, and then multiplied with the input feature map to obtain the output feature map.
[0028] As a preferred embodiment, the processing procedure of the mask average pooling layer includes:
[0029] First, calculate the prototype feature map corresponding to the fundus blood vessel image. Their respective prototype features:
[0030] ;
[0031] in, This represents the i-th prototype feature in the prototype vector set. This indicates that the prototype feature map extracted from the j-th fundus blood vessel image in the i-th stage of the prototype extraction layer is in... Feature values of pixel coordinates For the j-th fundus blood vessel image, the prototype feature map corresponding to the i-th stage in the prototype extraction layer is... The classification mask corresponding to the pixel coordinates. , where n represents the number of stages contained in the prototype extraction layer; K represents the number of fundus blood vessel images; thus, the prototype vector set is obtained. And fuse them to obtain the prototype feature vector. The fusion method is as follows:
[0032] .
[0033] As a preferred embodiment, the segmentation head calculates the prototype feature map extracted from the fundus blood vessel image in the nth stage of the prototype extraction layer using cosine distance. With prototype feature vector The distance is used to determine the region segmentation probability of the fundus blood vessel image, and thus obtain the region segmentation result of the fundus blood vessel image.
[0034] As a preferred embodiment, the fundus blood vessel segmentation network model is trained in the following manner:
[0035] First, fundus blood vessel sample images with completed blood vessel image region classification and masking are used as training samples. The training samples are divided into support set and query set. The fundus blood vessel segmentation network model is trained using the N-way K-shot classification task training method. The training steps for each region segmentation category to be identified include:
[0036] In the first stage, the prototype feature map corresponding to the support set input prototype extraction layer is extracted. The vascular image region classification mask is used as the mask for the masked average pooling layer, and the prototype vector set of the support set is extracted using the masked average pooling layer. The fusion yields the prototype feature vector of the support set. ;
[0037] In the second stage, the query set is input into the prototype extraction layer to extract the corresponding prototype feature map. And calculate the prototype feature map of the query set through the segmentation head. With support set prototype feature vector distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results;
[0038] In the third stage, the region segmentation category probabilities output from the second stage are... The mask is used as the mask for the masked average pooling layer to extract the prototype vector set of the query set. The fusion yields the prototype feature vector of the query set. And calculate the prototype feature map of the support set through the segmentation head. With the prototype feature vector of the query set distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results;
[0039] Therefore, by repeating the above three stages of operation, the parameters of the fundus vessel segmentation network model are trained to optimize the loss function until the fundus vessel segmentation network model converges, thus obtaining the trained fundus vessel segmentation network model.
[0040] As a preferred approach, the binary cross-entropy loss is calculated using the following formula during the training of the fundus vessel segmentation network model:
[0041] ;
[0042] in, This represents the true region classification mask label for the i-th training sample. Let M represent the region segmentation class probability predicted for the i-th training sample, and M represent the number of training samples.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] This invention proposes a nested-structure fundus vessel segmentation network model for small-sample fundus vessel image segmentation by combining and improving the Unet++ network and the prototype network. The model performs multi-layered nested feature extraction on the input fundus vessel image through a prototype extraction layer to obtain a prototype feature map. A masked average pooling layer extracts the prototype feature vector corresponding to the prototype feature map based on a set segmentation category mask. Finally, a segmentation head calculates the distance between the prototype feature map and the prototype feature vector extracted in the last stage of the prototype extraction layer, thereby determining the region segmentation category probability of the fundus vessel image and obtaining the region segmentation result. This invention improves the model's ability to extract prototype feature vectors and enhances the segmentation accuracy in small-sample scenarios, addressing the problems of strong sample dependence, poor accuracy, and weak generalization ability in existing technologies. Attached Figure Description
[0045] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:
[0046] Figure 1 This is a schematic diagram of the prototype extraction layer structure in an embodiment of the present invention.
[0047] Figure 2 This is a schematic diagram of the processing procedure of the Context Feature Refinement (CFR) module in an embodiment of the present invention.
[0048] Figure 3 This is a schematic diagram of the processing procedure of the Convolutional Parallel Pooling (CPP) module in an embodiment of the present invention.
[0049] Figure 4 This is a schematic diagram of the processing procedure of the Nested Details Recovery (NDR) module in an embodiment of the present invention.
[0050] Figure 5 This is a schematic diagram illustrating the processing of the weak information activation function in an example of the present invention.
[0051] Figure 6 This is a schematic diagram of the processing procedure of the Weak Information Refinement (WIR) module in an example of the present invention.
[0052] Figure 7 This is a schematic diagram of the training process of the fundus blood vessel segmentation network model in an example of the present invention. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0054] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; these relative positional relationships may change accordingly when the absolute position of the described object changes.
[0055] This invention provides a small-sample fundus blood vessel image segmentation method based on a nested structure. It acquires a fundus blood vessel image to be processed, inputs it into a pre-trained fundus blood vessel segmentation network model, and obtains the blood vessel region segmentation result of the fundus blood vessel image. The fundus blood vessel segmentation network model includes a prototype extraction layer, a masked average pooling layer, and a segmentation head. The prototype extraction layer performs multi-layer nested feature extraction on the input fundus blood vessel image to obtain a prototype feature map of the fundus blood vessel image. The masked average pooling layer extracts the prototype feature vector corresponding to the prototype feature map according to a set segmentation category mask. The segmentation head performs segmentation processing on the fundus blood vessel image based on the prototype feature vector to obtain the fundus blood vessel image region segmentation result.
[0056] The present invention provides a detailed description of a small-sample fundus blood vessel image segmentation method and the fundus blood vessel segmentation network model architecture used therein, with reference to the accompanying drawings.
[0057] The fundus vessel segmentation network model used in the method of this invention includes a prototype extraction layer, a masked average pooling layer, and a segmentation head. The following sections will elaborate on each part of the model.
[0058] (a) Prototype Extraction Layer
[0059] U-Net++ is a nested deep convolutional model that can effectively process input images of arbitrary size and multiple channels. By nesting U-Net structures at different depths, it can retain feature information from different depths at multiple output ports. The feature extraction process of the U-Net structure is optimized by performing depth supervision on the outputs of these ports, and feature information from different depths is better fused through dense skip connections. This invention proposes an improved model based on U-Net++ as a prototype extraction layer, the structure of which is as follows: Figure 1As shown.
[0060] This invention improves the feature extraction module of the encoder by proposing a context feature refinement module (CFR), which enhances foreground vascular information and suppresses background noise by concatenating spatial attention and channel attention, and supplements information through chained residual structures and convolutional parallel pooling (CPP).
[0061] For the upsampling process of the decoder, this invention designs a nested detail recovery module (NDR) based on dense upsampling convolution (DUC) to restore image resolution.
[0062] like Figure 1 As shown, the prototype extraction layer includes n cascaded stages and n levels of feature recovery units (FRs), where any nth stage... The feature recovery unit FR-i in the stage includes a context feature refinement module and a series of sequentially connected modules. A nested detail recovery module, The fundus blood vessel image input to the prototype extraction layer is used as the input to the first-stage feature recovery unit. The output of the context feature refinement module (CFR) in each stage's feature recovery unit (FR) is processed by convolutional parallel pooling (CPP) and then used as the input to the next stage's feature recovery unit (FR). The prototype feature map output from the last nth layer of each stage's feature recovery unit (FR) is used as the prototype feature map extracted from the fundus blood vessel image, thus obtaining the prototype feature map corresponding to the fundus blood vessel image. .
[0063] Among them, any first in the prototype extraction layer The prototype feature map output of any j-th layer in the feature recovery unit of the stage. This can be expressed by the following formula:
[0064] ;
[0065] ;
[0066] ;
[0067] in, This is the input to the feature recovery unit in the i-th stage of the prototype extraction layer. This indicates the processing of the context feature refining module. This indicates the processing of nested detail recovery modules at iteration k-1. This indicates convolutional parallel pooling processing;
[0068] For any output of layer j in stage i of the prototype extraction layer Represented as:
[0069] ;
[0070] Among them, satisfying , This represents the ReLU activation function operation.
[0071] Furthermore, as an optimization design, in the prototype extraction layer, to better facilitate the nested layer propagation of extracted features, the output of the nth layer in any i-th stage is... Afterwards, it is processed by the Weak Information Refinement (WIR) module and then passed to the nth layer of the i+1th stage to refine the weak information at different depths and gradually pass it to the deeper layers of the network to supplement the detailed information of the shallow layers.
[0072] 1.1 Contextual Feature Refinement Module (CFR)
[0073] Unlike medical images of the kidneys and lungs, where foreground pixels are relatively concentrated, vascular structures in fundus images are more complex, with some vessels having fewer pixels, making them easily confused with background noise and leading to reduced segmentation accuracy. To better extract and fuse vascular feature information, this invention designs a contextual feature refinement (CFR) module based on an attention mechanism, the structure of which is as follows: Figure 2 As shown.
[0074] pass Figure 2 As can be seen, the processing steps of the Context Feature Refinement (CFR) module include: expanding the channel count of the input feature map through a 1×1 convolution to obtain an expanded feature map; then, extracting and aggregating context features from the expanded feature map through a 3×3 convolution and a 1×1 convolution to obtain an aggregated feature map; next, extracting channel attention features and spatial attention features from the aggregated feature map through a channel attention module (CA) and a spatial attention module (SA); finally, adding the extracted feature map element-wise with the expanded feature map and performing activation operations to obtain the output feature map.
[0075] 1.2 Convolutional Parallel Pooling (CPP)
[0076] Classical max pooling activates only the maximum value in a region while discarding other information. While average pooling retains this information, it weakens some features of interest. This invention designs a convolutional parallel pooling method by combining convolution and max pooling to supplement feature information over a wider field of view. Its structure is as follows: Figure 3 As shown.
[0077] The Convolutional Parallel Pooling (CPP) process includes: performing max pooling and combined convolution processing on the input feature map respectively; in the combined convolution processing, the input feature map is first expanded by a 5×5 convolution, and then adjusted by a 2×2 convolution; then, the max pooling result map and the combined convolution result map are added element-wise to obtain the output feature map.
[0078] 1.3 Nested Details Recovery (NDR) Module
[0079] Since upsampling using transposed convolution or bilinear interpolation can cause a certain degree of information distortion, this invention designs a nested detail recovery (NDR) module based on dense upsampling convolution (DUC), the structure of which is as follows: Figure 4 As shown.
[0080] For any nested detail recovery module (NDR) in the j-th layer of the i-th stage of the prototype extraction layer, the processing steps include: first, performing dense upsampling convolution (DUC) on the input feature map to divide and map the input feature map into... The dimensions are: height, width, and number of channels. The feature submaps; DUC can map the information stored in the channels to the spatial dimension, thereby restoring the image resolution and reducing the information loss caused by upsampling using transposed convolution and interpolation; then, it is compared with the feature submaps located in the j-th layer in previous stages. to After concatenation, information is extracted and fused through contextual feature refinement. The fused result is then compared with the feature sub-graph located at the j-th layer of the previous stage. After element-wise addition, the feature map is obtained as the output. .
[0081] The dense upsampling convolution process involves first performing a 3×3 convolution on the input feature map, and then dividing the convolutionally processed feature map into... The dimensions are: height, width, and number of channels. A subset of feature maps, in which , where r is the upsampling factor. For the target number of channels, Given the number of channels in the input feature map, a subset of the feature map is finally mapped to... The dimensions are: height, width, and number of channels. The feature subgraph.
[0082] 1.4 Weak Information Refinement Module (WIR)
[0083] One challenge in retinal vessel image segmentation is the low contrast between the vessels and the background. To address this, methods such as... Figure 5The weak information activation function shown is in different The activation values displayed are slightly greater than zero, and the weak information activation function can be represented by the following formula:
[0084]
[0085] in, The output of the last nth layer in the i-th stage of the prototype extraction layer. The output after the weak information activation function for The output after dynamic linear activation and Let be the learnable coefficients of the m-th linear activation function in the i-th stage. , For the Sigmoid function; finally, same To merge, and the fusion results Used to stitch into the feature map of the next stage.
[0086] Using the above design approach, the processing procedure for the Weak Information Refinement (WIR) module is as follows: Figure 6 As shown: Information activation processing and spatial attention extraction processing are performed on the input feature map respectively. In the information activation processing, a linear activation function (DyReLU dynamic linear activation function can be selected) is first used to dynamically linearly activate the input feature map. Then, channel information is activated by 3×3 convolution and mapped by sigmoid function to obtain a probability map. After erasing significant blood vessel information, it is multiplied with the original feature map to obtain the corresponding activation information. Then, the probability map is multiplied with the input feature map to obtain the corresponding information activation processing result map. Then, spatial attention is used to supplement significant blood vessel information and suppress background noise. The information activation processing result map and the spatial attention extraction processing result map are added element-wise and then multiplied with the input feature map to obtain the output feature map, which is passed to the next stage.
[0087] (ii) Masked average pooling layer and segmentation head
[0088] The masked average pooling layer is used to extract the prototype feature vector corresponding to the prototype feature map based on a set segmentation category mask. The processing steps of the masked average pooling layer include:
[0089] First, calculate the prototype feature map corresponding to the fundus blood vessel image. Their respective prototype features:
[0090] ;
[0091] in, This represents the i-th prototype feature in the prototype vector set. This indicates that the prototype feature map extracted from the j-th fundus blood vessel image in the i-th stage of the prototype extraction layer is in... Feature values of pixel coordinates For the j-th fundus blood vessel image, the prototype feature map corresponding to the i-th stage in the prototype extraction layer is... The classification mask corresponding to the pixel coordinates. , where n represents the number of stages contained in the prototype extraction layer; K represents the number of fundus blood vessel images; thus, the prototype vector set is obtained. And fuse them to obtain the prototype feature vector. The fusion method is as follows:
[0092] .
[0093] The segmentation head calculates the prototype feature map extracted from the fundus blood vessel image in the nth stage of the prototype extraction layer using cosine distance. With prototype feature vector The distance is used to determine the region segmentation probability of the fundus blood vessel image, and thus obtain the region segmentation result of the fundus blood vessel image.
[0094] (III) Training of the fundus blood vessel segmentation network model
[0095] The fundus vessel segmentation network model maps the support set to the embedding space through a prototype extraction layer, thereby extracting prototype feature vectors for each category, i.e., prototypes. Then, it classifies and segments the target by calculating the distance between the prototype feature vectors and the query set features. This strategy achieves classification and segmentation tasks on new datasets and categories by training the network to extract prototypes on other datasets, exhibiting good generalization ability.
[0096] Considering the scarcity of medical image datasets, this invention constructs a fundus vessel segmentation network model based on the aforementioned prototype extraction layer to meet the needs of small-sample segmentation tasks. First, the training dataset is instantiated as an N-way K-shot segmentation task, and a support set is partitioned for each segmentation category of the region to be identified. and query set Where N represents the N segmentation classes, and K is the number of samples in the support set, with each support set... With real labels One-to-one correspondence; then, the prototype feature map of the support set is extracted through the nested structure of the prototype extraction layer. , to use real labels As a mask, the prototype vector set of the support set is extracted through a masked average pooling (MAP) layer. Then the prototype vector set of the support set The fusion yields the prototype feature vector of the support set. Then, the query set Input prototype extraction layer And calculate using cosine distance and distance Finally, the split head is used to... Normalization yields the probability map of the output. .
[0097] To further improve the effectiveness of the prototype feature vector, the resulting probability map... As a mask for the query set, the prototype vector set of the query set is obtained through MAP. The same fusion method was used to obtain Calculate its relationship with cosine distance And obtain the prediction probability map through the segmentation head. The loss between the support set and query set probability maps and the ground truth labels is calculated, and the fundus vessel segmentation network model is trained. Finally, the fundus vessel image dataset is instantiated into an N-way K-shot segmentation task, which similarly divides the dataset into support set and query set. Prototype feature vectors are extracted using only the support set, and the query set is then segmented.
[0098] The training process of the fundus blood vessel segmentation network model is as follows: Figure 7 As shown, the overall processing flow can be explained through the following steps.
[0099] 1. Data Preparation Stage
[0100] Using a fundus vessel dataset (which can be a publicly available dataset) that does not intersect with the test set as the training set, fundus vessel sample images are obtained as training samples. Pre-processing is performed to classify and label the vessel image regions in the sample images to determine the true mask. Simultaneously, to better ensure the consistency of the training samples, sample images obtained through different methods can be preprocessed. Image grayscale is used to fuse the number of channels to 1 to adapt to the input of fundus vessel images. Adaptive histogram equalization and gamma transform are used for image enhancement and normalization. The images are divided into support and query sets and uniformly cropped to... The image blocks are then fed into the fundus blood vessel segmentation network model for training.
[0101] 2. Training Phase
[0102] First, fundus blood vessel sample images with completed blood vessel image region classification and masking are used as training samples. The training samples are divided into support set and query set. The fundus blood vessel segmentation network model is trained using the N-way K-shot classification task training method. The training steps for each region segmentation category to be identified include:
[0103] In the first stage, the prototype feature map corresponding to the support set input prototype extraction layer is extracted. The vascular image region classification mask is used as the mask for the masked average pooling layer, and the prototype vector set of the support set is extracted using the masked average pooling layer. The fusion yields the prototype feature vector of the support set. ;
[0104] In the second stage, the query set is input into the prototype extraction layer to extract the corresponding prototype feature map. And calculate the prototype feature map of the query set through the segmentation head. With support set prototype feature vector distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results;
[0105] In the third stage, the region segmentation category probabilities output from the second stage are... The mask is used as the mask for the masked average pooling layer to extract the prototype vector set of the query set. The fusion yields the prototype feature vector of the query set. And calculate the prototype feature map of the support set through the segmentation head. With the prototype feature vector of the query set distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results.
[0106] In training the fundus blood vessel segmentation network model, the binary cross-entropy loss is calculated using the following formula:
[0107] ;
[0108] in, This represents the true region classification mask label for the i-th training sample. Let M represent the region segmentation class probability predicted for the i-th training sample, and M represent the number of training samples.
[0109] Therefore, by repeating the above three stages of operation, the parameters of the fundus vessel segmentation network model are trained to optimize the loss function until the fundus vessel segmentation network model converges, thus obtaining the trained fundus vessel segmentation network model.
[0110] 3. Model Testing Phase
[0111] With the model parameters fixed, the fundus blood vessel images are divided into a support set and a query set. Prototype feature vectors are extracted only from the support set, and the query set is fed into the fundus blood vessel segmentation network model to calculate the segmentation results. This is used to test the model training effect and confirm the training completion of the fundus blood vessel segmentation network model.
[0112] In summary, this invention proposes a nested-structure fundus vessel segmentation network model for small-sample fundus vessel image segmentation by combining and improving the Unet++ network and the prototype network. The model uses a prototype extraction layer to perform multi-layered nested feature extraction on the input fundus vessel image, obtaining a prototype feature map. A masked average pooling layer extracts the prototype feature vector corresponding to the prototype feature map based on a set segmentation category mask. Finally, a segmentation head calculates the distance between the prototype feature map and the prototype feature vector extracted in the last stage of the prototype extraction layer, thereby determining the region segmentation category probability of the fundus vessel image and obtaining the region segmentation result. This invention improves the model's ability to extract prototype feature vectors, enhances the segmentation accuracy in small-sample scenarios, and effectively solves the problems of existing models such as strong sample dependence, poor accuracy, and weak generalization ability.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.
Claims
1. A small-sample fundus blood vessel image segmentation method based on nested structure, characterized in that, The fundus blood vessel image to be processed is obtained and input into a pre-trained fundus blood vessel segmentation network model to obtain the blood vessel region segmentation result of the fundus blood vessel image to be processed. The fundus vessel segmentation network model includes a prototype extraction layer, a masked average pooling layer, and a segmentation head. The prototype extraction layer is used to perform multi-layer nested feature extraction on the input fundus vessel image to obtain a prototype feature map of the fundus vessel image. The masked average pooling layer is used to extract the prototype feature vector corresponding to the prototype feature map according to a set segmentation category mask. The segmentation head is used to perform segmentation processing on the fundus vessel image based on the prototype feature vector to obtain the fundus vessel image region segmentation result. The prototype extraction layer comprises n cascaded stages and n-level feature recovery units, wherein any nth stage... The feature recovery unit of the stage includes a context feature refinement module and a sequentially connected feature recovery module. A nested detail recovery module, The fundus blood vessel image input to the prototype extraction layer is used as the input to the first-stage feature recovery unit. The output of the context feature refinement module in each stage feature recovery unit is processed by convolutional parallel pooling and then used as the input to the next stage feature recovery unit. The prototype feature map output from the last nth layer of each stage feature recovery unit is used as the prototype feature map extracted from the fundus blood vessel image, thus obtaining the prototype feature map corresponding to the fundus blood vessel image. ; Any of the prototype extraction layers The prototype feature map output of any j-th layer in the feature recovery unit of the stage. This can be expressed by the following formula: ; ; ; in, This is the input to the feature recovery unit in the i-th stage of the prototype extraction layer. This indicates the processing of the context feature refining module. This indicates the processing of nested detail recovery modules at iteration k-1. This indicates convolutional parallel pooling processing; For any output of layer j in stage i of the prototype extraction layer Represented as: ; Among them, satisfying , This represents the ReLU activation function operation.
2. The method for segmenting small-sample fundus blood vessel images based on nested structures according to claim 1, characterized in that, The processing steps of the context feature refinement module include: The input feature map is expanded by performing a 1×1 convolution to obtain an expanded feature map. Then, the expanded feature map is sequentially processed by a 3×3 convolution and a 1×1 convolution to extract and aggregate context features, resulting in an aggregated feature map. Next, the aggregated feature map is sequentially processed by a channel attention module and a spatial attention module to extract channel attention features and spatial attention features. Finally, the extracted feature map is added element-wise to the expanded feature map and activated to obtain the output feature map.
3. The small-sample fundus blood vessel image segmentation method based on nested structure according to claim 1, characterized in that, The convolutional parallel pooling process includes: The input feature map is subjected to max pooling and combined convolution processing respectively. In the combined convolution processing, the input feature map is first expanded by 5×5 convolution, and then adjusted by 2×2 convolution. Then, the max pooling result map and the combined convolution result map are added element by element to obtain the output feature map.
4. The small-sample fundus blood vessel image segmentation method based on nested structure according to claim 1, characterized in that, For any nested detail recovery module at level j in stage i of the prototype extraction layer, the processing procedure includes: First, process the input feature map Dense upsampling convolution is performed to partition and map the input feature map into... The dimensions are: height, width, and number of channels. The feature submap is then compared with the feature submaps located at the j-th layer in previous stages. to After concatenation, information is extracted and fused through contextual feature refinement. The fused result is then compared with the feature sub-graph located at the j-th layer of the previous stage. After element-wise addition, the feature map is obtained as the output. ; The dense upsampling convolution process involves first performing a 3×3 convolution on the input feature map, and then dividing the convolutionally processed feature map into... The dimensions are: height, width, and number of channels. A subset of feature maps, in which , where r is the upsampling factor. For the target number of channels, Given the number of channels in the input feature map, a subset of the feature map is finally mapped to... The dimensions are: height, width, and number of channels. The feature subgraph.
5. The method for segmenting small-sample fundus blood vessel images based on nested structures according to claim 1, characterized in that, The final nth layer output of any i-th stage in the prototype extraction layer. Afterwards, it is further processed by the weak information refinement module before being passed to the nth layer of the (i+1)th stage; the processing of the weak information refinement module includes: The input feature map is subjected to information activation processing and spatial attention extraction processing respectively. In the information activation processing, the input feature map is first dynamically linearly activated using a linear activation function, then channel information is activated through a 3×3 convolution and mapped by a sigmoid function to obtain a probability map. The probability map is then multiplied with the input feature map to obtain the corresponding information activation processing result map. Then, the information activation processing result map and the spatial attention extraction processing result map are added element-wise, and then multiplied with the input feature map to obtain the output feature map.
6. The method for segmenting small-sample fundus blood vessel images based on nested structures according to claim 1, characterized in that, The processing steps of the masked average pooling layer include: First, calculate the prototype feature map corresponding to the fundus blood vessel image. Their respective prototype features: ; in, This represents the i-th prototype feature in the prototype vector set. This indicates that the prototype feature map extracted from the j-th fundus blood vessel image in the i-th stage of the prototype extraction layer is in... Feature values of pixel coordinates For the j-th fundus blood vessel image, the prototype feature map corresponding to the i-th stage in the prototype extraction layer is... The classification mask corresponding to the pixel coordinates. , where n represents the number of stages contained in the prototype extraction layer; K represents the number of fundus blood vessel images; thus, the prototype vector set is obtained. And fuse them to obtain the prototype feature vector. The fusion method is as follows: 。 7. The method for segmenting small-sample fundus blood vessel images based on nested structures according to claim 6, characterized in that, The segmentation head calculates the prototype feature map extracted from the fundus blood vessel image in the nth stage of the prototype extraction layer using cosine distance. With prototype feature vector The distance is used to determine the region segmentation probability of the fundus blood vessel image, and thus obtain the region segmentation result of the fundus blood vessel image.
8. The method for segmenting small-sample fundus blood vessel images based on nested structures according to claim 1, characterized in that, The fundus blood vessel segmentation network model was trained in the following manner: First, fundus blood vessel sample images with completed blood vessel image region classification and masking are used as training samples. The training samples are divided into support set and query set. The fundus blood vessel segmentation network model is trained using the N-way K-shot classification task training method. The training steps for each region segmentation category to be identified include: In the first stage, the prototype feature map corresponding to the support set input prototype extraction layer is extracted. The vascular image region classification mask is used as the mask for the masked average pooling layer, and the prototype vector set of the support set is extracted using the masked average pooling layer. The fusion yields the prototype feature vector of the support set. ; In the second stage, the query set is input into the prototype extraction layer to extract the corresponding prototype feature map. And calculate the prototype feature map of the query set through the segmentation head. With support set prototype feature vector distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results; In the third stage, the region segmentation category probabilities output from the second stage are... The mask is used as the mask for the masked average pooling layer to extract the prototype vector set of the query set. The fusion yields the prototype feature vector of the query set. And calculate the prototype feature map of the support set through the segmentation head. With the prototype feature vector of the query set distance The corresponding region segmentation category probabilities are obtained. Calculate the binary cross-entropy loss of the classification results; Therefore, by repeating the above three stages of operation, the parameters of the fundus vessel segmentation network model are trained to optimize the loss function until the fundus vessel segmentation network model converges, thus obtaining the trained fundus vessel segmentation network model.
9. The small-sample fundus blood vessel image segmentation method based on nested structure according to claim 8, characterized in that, In training the fundus blood vessel segmentation network model, the binary cross-entropy loss is calculated using the following formula: ; in, This represents the true region classification mask label for the i-th training sample. Let M represent the region segmentation class probability predicted for the i-th training sample, and M represent the number of training samples.
Citation Information
Patent Citations
Single sample image segmentation method based on category prototype learning
CN113569865A
Multi-low-level feature fusion retinal vessel segmentation method
CN113781514A
