Retinal blood vessel segmentation model training method, training device, segmentation method, equipment and storage medium

Abstract

The present invention relates to the field of image processing, and to a retinal vascular segmentation model training method, including obtaining a fundus image as a training data set; the training data set includes a query set and a support set; constructing a network structure of the retinal vascular segmentation model; using the query set and the support set to train the network of the retinal vascular segmentation model, in each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to a cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, and obtains the final retinal vascular segmentation model. The present invention also proposes a model training device, a retinal vascular segmentation method, equipment and a storage medium. The present invention can improve the ability to extract fundus image features and improve the recognition effect of a computer-aided diagnosis system when there are fewer sample labels.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a retinal blood vessel segmentation model training method, a training device, a segmentation method, a device and a storage medium. Background Art

[0002] Changes in retinal blood vessels can often indicate the presence of various eye diseases. Fundus imaging is crucial in the diagnosis of retinal vascular structure and helps diagnose eye diseases.

[0003] Fundus images are acquired through ophthalmic imaging technology, which can display the retina, optic disc, retinal blood vessels, etc., and accurately depict the retinal vascular network. The current mainstream medical image diagnosis method is computer-aided diagnosis (CAD). The computer-aided diagnosis system first extracts retinal blood vessels from the fundus images acquired through the fundus camera, and then uses deep learning technology to segment the retinal blood vessels into specific areas, so that the image analysis model can more accurately analyze the characteristics of specific areas, including size, shape and structure, thereby improving the recognition of abnormal structural changes in retinal blood vessels, so that ophthalmologists can identify and diagnose various retinal diseases.

[0004] However, deep learning-based retinal vessel segmentation tools require large training datasets, and obtaining adequately annotated data comes at a huge cost. Manual annotation of vascular patterns in fundus photographs relies heavily on the expertise of ophthalmologists, making this process particularly laborious and time-consuming. In addition, since retinal fundus photographs are taken with dedicated fundus cameras, the characteristics of these images may vary from device manufacturer to device manufacturer. Therefore, images generated by different devices may form different data domains, leading to domain shift problems in retinal vessel segmentation-related tasks. The above factors significantly increase the complexity of retinal vessel segmentation algorithms in clinical development, and therefore improvements to existing retinal vessel segmentation methods are needed. Summary of the invention

[0005] In order to improve the ability to extract fundus image features and improve the recognition effect of a computer-aided diagnosis system when there are fewer sample labels and lower image resolution, the present application provides a retinal vessel segmentation method, system, device and storage medium.

[0006] In order to achieve the above-mentioned object of the present invention, according to a first aspect of the present invention, the present invention provides a retinal blood vessel segmentation model training method.

[0007] The present invention provides a retinal blood vessel segmentation model training method, comprising:

[0008] Obtain fundus images as a training data set; the training data set includes a query set and a support set; the support set includes a small number of labeled support image samples, and the query set includes a number of unlabeled query image samples;

[0009] Construct the network structure of the retinal blood vessel segmentation model;

[0010] The network of the retinal vascular segmentation model is trained using the query set and the support set. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model and obtain the final retinal vascular segmentation model.

[0011] Optionally, the steps of processing the query set and the support set by the retinal vessel segmentation model include:

[0012] Extracting image features of the support image to obtain support features, and extracting image features of the query image to obtain query features;

[0013] The support features and query features are processed at different scales according to the cross-window attention mechanism to obtain multi-scale image features;

[0014] Multiple multi-scale image features are fused to obtain retinal vascular features;

[0015] Generate a retinal vessel segmentation map based on retinal vessel features.

[0016] Optionally, before the step of processing the supporting features and the query features at different scales according to the cross-window attention mechanism to obtain a plurality of multi-scale image features, the training method further comprises:

[0017] Dividing the support feature into a plurality of support patches and dividing the query feature into a plurality of query patches, wherein the number of the support patches is the same as the number of the query patches;

[0018] Calculate the cosine similarity between the query patch and each supporting patch respectively, and determine the supporting patch that is most similar to each query patch according to a preset rule;

[0019] Reorder the support patches according to the cosine similarity between each support patch and the query patch, so that each query patch is aligned with the most similar support patch;

[0020] The aligned query patch and support patch are fused according to the cross-window attention mechanism to obtain multi-scale image features.

[0021] Optionally, a query patch and supporting patches aligned with the query patch are considered as a set of patches;

[0022] The aligned query patch and support patch are processed according to the cross-window attention mechanism to obtain multi-scale image features including:

[0023] Input the query patch into the query generator to obtain the query vector Q of the query patch;

[0024] Input the support patch into the KV generator to obtain the key vector K of the support patch and the value vector V of the support patch;

[0025] The intra-patch similarity score is obtained by element-wise multiplication of the query vector Q of the query patch and the key vector K of the supporting patch in the same group;

[0026] The similarity scores between patches are obtained by element-wise multiplication of the query vector Q of the query patch and the key vector K of the support patch of different groups;

[0027] Determine the comprehensive calibration scores of the support set and the query set according to the intra-patch similarity score and the inter-patch similarity score;

[0028] Generate a new query patch based on the comprehensive calibration score and the value vector V of the support patch;

[0029] The query features of the new query patch and the supporting features of the supporting patches are concatenated to obtain multi-scale image features.

[0030] Optionally, generating a retinal vessel segmentation map based on retinal vessel features includes:

[0031] The number of channels of the supporting features and the query features in the retinal vascular features is adjusted to obtain an adjusted retinal vascular feature map;

[0032] The adjusted retinal vessel feature map is upsampled using a pixel rearrangement operation to obtain a retinal vessel segmentation map.

[0033] Optionally, the network structure of the retinal blood vessel segmentation model includes a feature extraction module, a cross-window module, a feature fusion module and a generation module;

[0034] The feature extraction module is used to extract image features of the support image to obtain the support feature, and is used to extract image features of the query image to obtain the query feature;

[0035] The cross-window module is used to process the support features and the query features at different scales according to the cross-window attention mechanism to obtain multi-scale image features;

[0036] The feature fusion module is used to fuse multiple multi-scale image features to obtain retinal vascular features;

[0037] The generating module is used to generate a retinal vessel segmentation map based on retinal vessel features;

[0038] The cross-window modules are provided with a plurality of them, and the support features and the query features are sequentially passed through the plurality of cross-window modules to obtain multi-scale image features at different levels; the training method further comprises:

[0039] The multi-scale image features at different levels are fused through jump connections to obtain retinal vascular features.

[0040] In order to solve the above problems, the present invention further provides a retinal blood vessel segmentation model training device, the training device comprising:

[0041] An acquisition module, used to acquire fundus images as training data sets; the training data sets include a query set and a support set; the support set is a labeled support image sample, and the query set is an unlabeled query image sample;

[0042] An input module is used to construct the network structure of the initial retinal blood vessel segmentation model;

[0043] The training module is used to train the network of the retinal vascular segmentation model using the query set and the support set. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, and obtains the final retinal vascular segmentation model.

[0044] In order to solve the above problem, the present invention further provides a retinal blood vessel segmentation method, the retinal blood vessel segmentation method comprising:

[0045] Acquiring a fundus image to be processed;

[0046] The fundus image is input into the retinal blood vessel segmentation model trained by the above-mentioned retinal blood vessel segmentation model training method to obtain a fundus image segmentation result.

[0047] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned retinal vessel segmentation model training method or retinal vessel segmentation method.

[0051] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned retinal vessel segmentation model training method or retinal vessel segmentation method.

[0052] In summary, this application includes the following beneficial technical effects:

[0053] The retinal vascular segmentation model uses a cross-window attention mechanism in the feature extraction part, so that the retinal vascular segmentation model can obtain multi-scale image features of fundus images, can fully acquire the knowledge of the support set, improve the ability to extract fundus image features, and can extract a large number of features under limited sample conditions, thereby improving the recognition effect of the computer-aided diagnosis system when there are fewer sample labels and lower image resolution, and effectively reducing the manual annotation burden associated with new fundus image datasets;

[0054] By aligning each query patch with its most similar support patch, effective cross-attention is achieved, alleviating the problem of invalid support patches and misaligned support patches; the aligned query patches and support patches are processed according to the cross-window attention mechanism to obtain multi-scale image features, and multiple multi-scale image features are fused to obtain retinal vascular features; by creating a comprehensive calibration score, vascular features in fundus images can be more effectively distinguished;

[0055] During the decoding process, a pixel rearrangement operation is used for upsampling, and the feature map pixels are reorganized to make the generated retinal segmentation image smoother and reduce visual fragmentation. Through the feature fusion module and combined with jump connections, the features of the support set are fully utilized in the image restoration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of a retinal blood vessel segmentation model training method provided in one embodiment of the present invention;

[0057] Figure 2 A network structure diagram of a retinal blood vessel segmentation model provided in one embodiment of the present invention;

[0058] Figure 3 An architectural diagram of a cross-window module provided by an embodiment of the present invention;

[0059] Figure 4 An architectural diagram of a bottleneck submodule provided in one embodiment of the present invention;

[0060] Figure 5 An architectural diagram of a patch alignment submodule provided in one embodiment of the present invention;

[0061] Figure 6 An architecture diagram of a cross-window attention submodule provided by an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of a pixel reorganization module according to an embodiment of the present invention;

[0063] Figure 8 A schematic diagram of a pixel reorganization implementation process of a pixel reorganization module provided in an embodiment of the present invention;

[0064] Fig. 9 A schematic diagram of the implementation process of the QS fusion module provided in one embodiment of the present invention;

[0065] Fig.10 This is an example of comparing the predicted results of the retinal vessel segmentation model with the actual situation of the dataset after training the retinal vessel segmentation model on the CHASE-DB1 and DRIVE datasets;

[0066] Fig.11 A schematic diagram of the structure of an electronic device for implementing the retinal blood vessel segmentation method provided by an embodiment of the present invention.

[0067] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0068] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0069] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0070] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0071] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0072] Example 1

[0073] Reference Figure 1 FIG. 2 is a flow chart of a retinal blood vessel segmentation model training method provided by an embodiment of the present invention. In this embodiment, the retinal blood vessel segmentation model training method includes:

[0074] S1, obtaining fundus images as training data sets;

[0075] The training dataset includes several query sets and several support sets; the support set contains a small number of labeled support image samples, and the query set includes several unlabeled query image samples;

[0076] Fundus images contain rich information, such as blood vessel texture, diseased area and color distribution, etc., which are crucial for diagnosing eye diseases. Fundus images include foreground and background. The foreground of fundus images mainly refers to the key diagnostic information in the fundus images, such as blood vessels, diseased areas, etc., while the background of fundus images refers to the area outside the above key diagnostic information. In this embodiment, fundus images of different scenes are selected as training data sets. For example, fundus images collected under different lighting environments can be selected as training data, or fundus images collected under different physical conditions of the human body can be selected as training sample data, or images of the same environment but with different resolutions can be selected as training samples; the above scenes are the basic elements of the retinal vascular segmentation model based on scene training;

[0077] Each scene consists of a support set S = {S image ,S mask} and a query set Q = {Q image ,Q mask}, where S image Indicates support for images, S mask represents the binary mask of the support image annotation, Q image represents the query image, Q mask Indicates the mask corresponding to the query image. In the training phase, the neural network model to be trained operates within the scope of the training dataset and learns to segment the query image Q under the guidance of the support set S. image,Then in the testing phase, the knowledge learned from the support set is applied to the test data set to verify the learning results of the neural network model, so that the neural network model can obtain better recognition results even with fewer sample labels, reducing the manual annotation burden associated with the new fundus image data set.

[0078] S2. Construct the network structure of the retinal blood vessel segmentation model.

[0079] Reference Figure 2 ,In this embodiment, the network structure of the retinal vessel segmentation model includes a ,feature extraction module, a cross-window module, a feature fusion module and a generation module connected in ,sequence;

[0080] The feature extraction module can extract image features of the support image to obtain the support features, and can be used to extract image features of the query image to obtain the query features; the feature extraction module is composed of a convolution layer, a batch normalization layer (Batch Normalization), a ReLU (Rectified Linear Unit) activation function and a maximum pooling layer.

[0081] Reference Figure 2 In some examples of this embodiment, the feature extraction module includes a support feature extraction module (Support Feature Extraction Block) and a query feature extraction module (Query Feature Extraction Block), and the structures of the support feature extraction module and the query feature extraction module are the same; Figure 2 In the example, the feature extraction module is the module corresponding to Query Feature Extraction Block and Support Query Feature Extraction Block, and the cross-window module is Figure 2 The module corresponding to QS Cross-Window Block in the feature fusion module Figure 2 In the QSFusion Block corresponding to Figure 2 Channel Dimension Concatenation means connection in the channel dimension.

[0082] When training the model, for the query set, first the query image Q image Input query feature extraction module (Query Feature Extraction Block), after the query feature extraction module, the number of channels of the query image increases to 64; for the support set, the support image S image and the binary mask S corresponding to the support image maskThe images are concatenated in the channel dimension and then input into the Support Feature Extraction Block to obtain the primary features of the query image and the support image.

[0083] Reference Figure 2 ,The cross-window module can process the support features and the query features at different scales according to the cross-window attention mechanism to obtain the multi-scale image features; ,the cross-window module is provided with multiple ,support features and query features are sequentially passed through multiple cross-window modules to obtain multi-scale image features at different levels;

[0084] The cross-window module (QS Cross-Window Block) is a feature extraction module that integrates convolutional neural networks (CNN) and Swin-Transformer. It is used to extract features from the support set and query set and merge the support features with the query features to comprehensively extract fundus image features from different scales. "Swin" is the abbreviation of "Shifted Windows", which means "moving window". Swin-Transformer is an existing model based on the Transformer architecture, which is particularly suitable for computer vision tasks such as target detection and image classification.

[0085] Reference Figure 2 and Figure 3 In the preferred implementation of this embodiment, the cross-window module includes a bottleneck submodule (i.e. Figure 3 Bottleneck Blocks in the Figure 3 PAModule in ) and the cross-window attention submodule (i.e. Figure 3 The input end of the patch alignment submodule is connected to the output end of the feature extraction module, the output end of the patch alignment submodule is connected to the input end of the cross-window attention submodule, and the output end of the cross-window attention submodule is connected to the input end of the feature fusion module;

[0086] Reference Figure 3 and Figure 4 ,In the cross-window block, the features of the support set and the query set first pass through the ,bottleneck sub-module, which extracts more advanced features by deepening the ,channel, which can efficiently extract spatial features without increasing a lot of ,computational overhead.

[0087] Reference Figure 5, then, the patch alignment submodule reshapes the feature into a sequential form. Before the support features and / or query features are input into the patch alignment submodule, the support features are first divided into a number of support patches, and the query features are divided into a number of query patches, and the number of support patches and query patches are consistent; the patch alignment submodule can reorder the query patches and support patches so that the query patches are aligned with the support patches with the closest cosine similarity to them, calculate the cosine similarities between the query patches and each support patch, and confirm the support patches that are most similar to each query patch according to preset rules; the patch alignment submodule realizes effective cross attention by aligning each query patch with the support patch that is most similar to it, thereby alleviating the problem of invalid support patches and misaligned support patches.

[0088] In the preferred implementation of this embodiment, the processing rules for query patches and support patches are as follows: first, average pooling operations are performed on each query patch and support patch respectively, and then the similarity is evaluated by calculating the cosine similarity of the query features of each query patch and the support features of the support patch. In the last step, the most similar small blocks are identified by calculating the maximum value, and each support patch is reordered according to the cosine similarity between the query patch and the support patch.

[0089] The cross-window attention submodule can process the aligned query patch and support patch according to the cross-window attention mechanism to obtain multi-scale image features.

[0090] Reference Figure 3 and Figure 6 The feature fusion module can fuse multiple multi-scale image features to obtain retinal vascular features; after being processed by the patch alignment submodule, the aligned features are guided through the cross-window attention submodule (Cross-Window Attention Block). The cross-window attention submodule improves the window attention mechanism in Swin-Transformer to meet the needs of few-sample learning; in the preferred implementation of this embodiment, the processing mechanism of the cross-window attention submodule for query patches and support patches is:

[0091] Input the query patch into the query generator to obtain the query vector Q of the query patch; input the support patch into the KV generator to obtain the key vector K of the support patch and the value vector V of the support patch;

[0092] In the KV generator, K represents the key and V represents the value; both the query generator and the KV generator are composed of fully connected layers;

[0093] The intra-patch similarity score is obtained by element-wise multiplication of the query vector Q of the query patch in the same group and the key vector K of the support patch. The inter-patch similarity score is obtained by element-wise multiplication of the query vector Q of the query patch in different groups and the key vector K of the support patch. The intra-patch similarity score represents the similarity between the patches in the same group in the support set and the query set; the inter-patch similarity score represents the similarity between the query patches and the support patches in different groups in the support set and the query set.

[0094] The inter-patch similarity score is divided element-wise by the L2-normalized intra-patch similarity score to obtain the QS similarity score; then, the QS similarity score is concatenated with the intra-patch similarity score along the sample dimension to form a comprehensive calibration score for the support set and the query set; the comprehensive calibration score is multiplied element-wise with the value vector V generated by the support patch, and after passing through a projection layer composed of a fully connected layer, a new query patch is obtained; finally, the support patch and the query patch are reassembled, and the features of the same group of new query patches and support patches are residually connected with the initial features (the query features and support features extracted for the first time) to obtain multi-scale image features. The complete feature information is restored through the fully connected layer and input into the next Cross-WindowAttention block for feature extraction through the convolution layer.

[0095] The self-attention score and cross-window attention score obtained in the traditional window attention are combined in the cross-window attention module to create a comprehensive calibration score, which can more effectively distinguish the foreground and background in fundus images.

[0096] Reference Figure 7 and Figure 8 The generation module can generate a retinal vessel segmentation map based on the retinal vessel features. Specifically, the generation module includes a channel tuning block (i.e. Figure 7 Channel Tuning Block) and pixel rearrangement block (i.e. Figure 7 The Pixel Shuffle in the retinal vascular feature is used to adjust the number of channels of the supporting features and the query features in the retinal vascular feature to obtain the adjusted retinal vascular feature map; the pixel rearrangement block pre-adjusts the position of the retinal vascular feature by using the pixel rearrangement operation, and then upsamples the adjusted retinal vascular feature map to convert the number of channels into spatial resolution.

[0097] Reference Figure 2 and Figure 7During deconvolution, the kernel sliding over the input feature map may result in uneven coverage, where some pixel regions may be overlapped multiple times while other regions are less covered. This uneven overlap results in an uneven distribution of pixel values ​​in the output image, forming a checkerboard-like pattern. By choosing pixel rearrangement instead of deconvolution, we effectively alleviate the "checkerboard artifact" problem associated with deconvolution. As a result, the retinal segmentation images generated by this method have higher smoothness and reduced visual fragmentation.

[0098] Reference Figure 6 and Fig. 9 After the pixel shuffle process, the features of the support set and the query set are guided through the QS fusion block again (i.e. Figure 6 The QS Fusion Block in the figure is composed of three different convolutional layers: the information fusion block (i.e. Fig. 9 Information Fusion Block in the Fig. 9 Updated Support Block) and Update Query Block (i.e. Fig. 9 The QS fusion block enhances the interaction between the support set and query set features by concatenating them along the channel dimension and applying convolution. The multi-scale image features extracted from the cross-window blocks via skip connections are processed through the QS fusion block. These features are then concatenated with the support set and query set features along the channel dimension and finally passed through the convolutional block to obtain the final retinal vessel segmentation map.

[0099] S3. The network of the retinal vascular segmentation model is trained using the query set and the support set. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, and obtains the final retinal vascular segmentation model.

[0100] Specifically, firstly, the network parameters of the retinal blood vessel segmentation model are determined using the support image to obtain an initial retinal blood vessel segmentation model;

[0101] Then, the query image is segmented using the initial retinal blood vessel segmentation model to obtain the fundus image segmentation result;

[0102] Afterwards, the loss function of the retinal blood vessel segmentation model is constructed based on the fundus image segmentation results;

[0103] Finally, the network parameters of the initial retinal vessel segmentation model are optimized based on the loss function to obtain the final retinal vessel segmentation model.

[0104] In this embodiment, the mean square error (MSE) loss function is used to optimize the network, and the model is optimized by minimizing the square error between the actual value and the predicted value; the expression of the loss function is:

[0105]

[0106] Among them, n represents the number of samples, i is the sample index, and Y i represents the sample label, Represents the prediction result generated by the model; the smaller the MSE Loss value, the smaller the difference between the model's predicted value and the true value, and the better the model performance.

[0107] The steps of the retinal vessel segmentation model for processing the query set and support set include:

[0108] S31, extracting image features of the support image to obtain support features, and extracting image features of the query image to obtain query features;

[0109] S32, processing the support features and query features at different scales according to the cross-window attention mechanism to obtain multi-scale image features;

[0110] S33, fusing multiple multi-scale image features to obtain retinal vascular features;

[0111] The multi-scale image features at different levels are fused through jump connections to obtain retinal vascular features. The specific processing methods are described in detail when introducing the retinal vascular image segmentation model. For the sake of brevity, they are not introduced again in this embodiment.

[0112] S34. Generate a retinal blood vessel segmentation map based on the retinal blood vessel features.

[0113] Generating a retinal vascular segmentation map based on retinal vascular features includes:

[0114] S41, adjusting the number of channels of the supporting features and the query features in the retinal vascular features to obtain an adjusted retinal vascular feature map;

[0115] S42, up-sampling the adjusted retinal vessel feature map by using a pixel rearrangement operation to obtain a retinal vessel segmentation map.

[0116] The specific processing method was described in detail when introducing the retinal vascular image segmentation model. For the sake of brevity, it will not be introduced here again.

[0117] Before processing the support features and the query features at different scales according to the cross-window attention mechanism to obtain a plurality of multi-scale image features, the training method further includes:

[0118] S51, dividing the support feature into a number of support patches, and dividing the query feature into a number of query patches, wherein the number of the support patches and the number of the query patches are the same;

[0119] S52, respectively calculating the cosine similarity between the query patch and each supporting patch, and determining the supporting patch most similar to each query patch according to a preset rule;

[0120] S53, reordering the supporting patches according to the cosine similarity between each supporting patch and the query patch, so that each query patch is aligned with the most similar supporting patch;

[0121] S54, processing the aligned query patch and support patch according to the cross-window attention mechanism, and obtaining multi-scale image features including:

[0122] S61, inputting the query patch into the query generator to obtain a query vector Q of the query patch;

[0123] S62, input the support patch into the KV generator to obtain a key vector K of the support patch and a value vector V of the support patch;

[0124] S63, obtaining an intra-patch similarity score by element-by-element multiplication of the query vector Q of the query patch in the same group and the key vector K of the supporting patch, and obtaining an inter-patch similarity score by element-by-element multiplication of the query vector Q of the query patch in different groups and the key vector K of the supporting patch;

[0125] S64, determining a comprehensive calibration score of the support set and the query set according to the intra-patch similarity score and the inter-patch similarity score;

[0126] S65, generating a new query patch based on the comprehensive calibration score and the value vector V of the support patch;

[0127] S66: Connect the query features of the new query patch and the support features of the support patch to obtain multi-scale image features.

[0128] The retinal vessel segmentation model trained by the above training method can learn sufficient retinal vessel features from a very small number of labeled samples in most data sets. In addition, after completing the training on one data set, the retinal vessel segmentation model can perform well in the remaining data areas without fine-tuning. In order to verify the segmentation ability of the constructed retinal vessel segmentation model for fundus images, the retinal vessel segmentation model was trained on the CHASE-DB1 dataset and the DRIVE dataset, and its predicted results were compared with the actual situation on each dataset. The comparison results are shown in Figure 2. Fig.10 shown.

[0129] The CHASE-DB1 dataset is a retinal vessel segmentation dataset containing 28 color retinal images of size 999×960 pixels, collected from the left and right eyes of 14 schoolchildren. Each image was annotated by two independent human experts; the DRIVE dataset contains 40 fundus retinal color images, of which 7 images show signs of mild diabetic retinopathy.

[0130] Fig.10 The left half of the figure shows the comparison between the predicted results of the retinal blood vessel segmentation model for the fundus image and the actual situation after the retinal blood vessel segmentation model was trained using the CHASE-DB1 dataset. The first column is the fundus image, the second column is the retinal blood vessel segmentation result, and the third column is the actual distribution result of the retinal blood vessels in the fundus image.

[0131] Fig.10 The right half of the picture shows the comparison between the predicted results of the retinal blood vessel segmentation model for the fundus image and the actual situation after the retinal blood vessel segmentation model was trained using the DRIVE dataset. The first column is the fundus image, the second column is the retinal blood vessel segmentation result, and the third column is the actual distribution result of the retinal blood vessels in the fundus image.

[0132] Based on the same inventive concept, an embodiment of the present invention provides a retinal blood vessel segmentation model training device.

[0133] The retinal blood vessel segmentation model training device of the present invention can be loaded in an electronic device. According to the functions to be implemented, the retinal blood vessel segmentation model training device includes an acquisition module, an input module and a training module.

[0134] Among them, the acquisition module can acquire fundus images of different scenarios as training data sets; the training data sets include query sets and support sets; the support set contains a small number of labeled support image samples, and the query set includes a number of unlabeled query image samples;

[0135] The input module can construct the network structure of the initial retinal vascular segmentation model. The training module can use the query set and the support set to train the network of the retinal vascular segmentation model. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, and obtain the final retinal vascular segmentation model.

[0136] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0137] The various variations and specific examples of the retinal vascular segmentation model training method provided in the above embodiment are also applicable to the retinal vascular segmentation model training device of this embodiment. Through the above detailed description of the retinal vascular segmentation model training method, those skilled in the art can clearly know the implementation method of the retinal vascular segmentation model training device in this embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0138] The present application also discloses a retinal blood vessel segmentation method, comprising:

[0139] S71, obtaining a fundus image to be processed;

[0140] S72. Input the fundus image into the retinal vessel segmentation model trained by the retinal vessel segmentation model training method described in Example 1 to obtain a retinal vessel segmentation result.

[0141] The present application also discloses an electronic device, such as Fig.11 , which is a schematic diagram of the structure of an electronic device for a retinal vessel segmentation model training method or a retinal vessel segmentation method provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a retinal vessel segmentation method program.

[0142] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11, and calls data stored in the memory 11, so as to execute various functions of the electronic device and process data.

[0143] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the retinal vascular segmentation model training method or the retinal vascular segmentation method program, but also can be used to temporarily store data that has been output or is to be output.

[0144] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0145] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0146] Fig.11 Only an electronic device with components is shown, and those skilled in the art will understand that Fig.11The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0147] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0148] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0149] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0150] The present application embodiment provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the retinal vessel segmentation model training method or retinal vessel segmentation method of the above embodiment.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0152] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A retinal blood vessel segmentation model training method, characterized in that: include: Obtain fundus images as training data sets; the training data sets include a query set and a support set; the support set is a labeled support image sample, and the query set is an unlabeled query image sample; Construct the network structure of the retinal blood vessel segmentation model; The query set and the support set are used to train the network of the retinal vascular segmentation model. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, and obtains the final retinal vascular segmentation model. The steps of processing the query set and the support set by the retinal blood vessel segmentation model include: Extracting image features of the support image to obtain support features, and extracting image features of the query image to obtain query features; The support features and query features are processed at different scales according to the cross-window attention mechanism to obtain multi-scale image features; Multiple multi-scale image features are fused to obtain retinal vascular features; Generate a retinal vascular segmentation map based on retinal vascular features; Before the supporting features and the query features are processed at different scales according to the cross-window attention mechanism to obtain a plurality of multi-scale image features, the training method further includes: Dividing the support feature into a plurality of support patches and dividing the query feature into a plurality of query patches, wherein the number of the support patches is the same as the number of the query patches; Calculate the cosine similarity between the query patch and each supporting patch respectively, and determine the supporting patch that is most similar to each query patch according to a preset rule; Reorder the support patches according to the cosine similarity between each support patch and the query patch, so that each query patch is aligned with the most similar support patch; The aligned query patch and support patch are fused according to the cross-window attention mechanism to obtain multi-scale image features.

2. The retinal blood vessel segmentation model training method according to claim 1, characterized in that: Considering a query patch and supporting patches aligned with the query patch as a set of patches; The aligned query patch and support patch are processed according to the cross-window attention mechanism to obtain multi-scale image features including: Input the query patch into the query generator to obtain the query vector Q of the query patch; Input the support patch into the KV generator to obtain the key vector K of the support patch and the value vector V of the support patch; The intra-patch similarity score is obtained by element-wise multiplication of the query vector Q of the query patch and the key vector K of the supporting patch in the same group; The similarity scores between patches are obtained by element-wise multiplication of the query vector Q of the query patch and the key vector K of the support patch of different groups; Determine the comprehensive calibration scores of the support set and the query set according to the intra-patch similarity score and the inter-patch similarity score; Generate a new query patch based on the comprehensive calibration score and the value vector V of the support patch; The query features of the new query patch and the supporting features of the supporting patches are concatenated to obtain multi-scale image features.

3. The retinal blood vessel segmentation model training method according to claim 1 or 2, characterized in that: Generating a retinal blood vessel segmentation map based on retinal blood vessel features comprises: The number of channels of the supporting features and the query features in the retinal vascular features is adjusted to obtain an adjusted retinal vascular feature map; The adjusted retinal vessel feature map is upsampled using a pixel rearrangement operation to obtain a retinal vessel segmentation map.

4. The retinal blood vessel segmentation model training method according to claim 3, characterized in that: The network structure of the retinal blood vessel segmentation model includes a feature extraction module, a cross-window module, a feature fusion module and a generation module; The feature extraction module is used to extract image features of the support image to obtain the support feature, and is used to extract image features of the query image to obtain the query feature; The cross-window module is used to process the support features and the query features at different scales according to the cross-window attention mechanism to obtain multi-scale image features; The feature fusion module is used to fuse multiple multi-scale image features to obtain retinal vascular features; The generating module is used to generate a retinal vessel segmentation map based on retinal vessel features; The cross-window modules are provided with a plurality of them, and the support features and the query features are sequentially passed through the plurality of cross-window modules to obtain multi-scale image features at different levels; The training method further comprises: The multi-scale image features at different levels are fused through jump connections to obtain retinal vascular features.

5. A retinal blood vessel segmentation model training device, characterized in that: An acquisition module, used to acquire fundus images as training data sets; the training data sets include a query set and a support set; the support set is a labeled support image sample, and the query set is an unlabeled query image sample; An input module is used to construct the network structure of the initial retinal blood vessel segmentation model; A training module is used to train the network of the retinal vascular segmentation model using the query set and the support set. In each training, the retinal vascular segmentation model extracts fundus image features of the support image and / or the query image at different scales according to the cross-window attention mechanism to determine the network parameters of the retinal vascular segmentation model, thereby obtaining a final retinal vascular segmentation model. The steps of processing the query set and the support set by the retinal blood vessel segmentation model include: Extracting image features of the support image to obtain support features, and extracting image features of the query image to obtain query features; The support features and query features are processed at different scales according to the cross-window attention mechanism to obtain multi-scale image features; Multiple multi-scale image features are fused to obtain retinal vascular features; Generate a retinal vascular segmentation map based on retinal vascular features; Before the supporting features and the query features are processed at different scales according to the cross-window attention mechanism to obtain a plurality of multi-scale image features, the training method further includes: Dividing the support feature into a plurality of support patches and dividing the query feature into a plurality of query patches, wherein the number of the support patches is the same as the number of the query patches; Calculate the cosine similarity between the query patch and each supporting patch respectively, and determine the supporting patch that is most similar to each query patch according to a preset rule; Reorder the support patches according to the cosine similarity between each support patch and the query patch, so that each query patch is aligned with the most similar support patch; The aligned query patch and support patch are fused according to the cross-window attention mechanism to obtain multi-scale image features.

6. A retinal blood vessel segmentation method, characterized in that: include: Acquiring a fundus image to be processed; The fundus image is input into a retinal vessel segmentation model trained by the retinal vessel segmentation model training method according to any one of claims 1 to 4 to obtain a retinal vessel segmentation result.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the method described in claim 1 or 2 or 3 or 4 or 6.

8. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method of claim 1 or 2 or 3 or 4 or 6.

Images