Retinal blood vessel image segmentation method, device and related equipment

Abstract

The present invention relates to the field of computer vision image processing, and discloses a retinal vascular image segmentation method and device. The method comprises: in the data processing stage, firstly obtaining a retinal vascular image to be detected; then, performing data enhancement and preprocessing on the retinal fundus image; in the training stage, firstly constructing a network model based on Transformer optimization, and then training the network model with the processed training image; in the testing stage, inputting the retinal vascular image into the trained network model for image segmentation; finally, taking a weighted average of the prediction results of multiple retinal vascular images output by the network model to obtain the classification probability of each pixel, and obtaining the final segmentation result map, and the present invention is used to improve the segmentation accuracy of the retinal vascular image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a retinal blood vessel image segmentation method, device and related equipment. Background Art

[0002] Retinal examination can provide important clinical information for the diagnosis of various retinal diseases such as diabetic retinopathy. However, manual retinal examination requires clinicians or experts with professional knowledge to screen a large number of retinas. This process is time-consuming, tedious, difficult to achieve batch processing, and prone to misdiagnosis. In order to alleviate the shortage of medical resources and reduce the burden on experts, it is necessary to develop an automatic, high-performance retinal vascular image segmentation device for pre-screening and other examinations. The automatic segmentation device for retinal vascular images can quickly and accurately segment the structural features of retinal vessels, among which branch points and bends can be used to assist in the diagnosis and analysis of cardiovascular diseases and diabetic retinopathy, while the variation characteristics of retinal vascular width can detect and analyze hypertension. Therefore, the current research on the automatic segmentation method of retinal vascular images is of great significance to the research of related retinal diseases.

[0003] The automatic segmentation technology of retinal blood vessels has been developing rapidly, and researchers have proposed a large number of retinal blood vessel segmentation methods. Some traditional methods have successfully segmented retinal blood vessel images automatically and obtained good segmentation results, but they cannot fully characterize the characteristics of the image, resulting in insufficient detection of retinal structural characteristics, resulting in unsatisfactory segmentation accuracy, and still cannot meet the needs of assisting ophthalmologists in clinical diagnosis.

[0004] Compared with traditional methods, CNN methods combine the advantages of medical image segmentation methods and semantic segmentation methods, making them achieve remarkable performance. Many excellent works have shown excellent segmentation performance in retinal vessel segmentation, proving that CNN has strong feature representation learning and recognition capabilities. However, due to the inherent locality of convolution operations, as the training data expands and the network layers increase, these methods find it difficult to learn explicit global and long-range semantic information interactions, resulting in discontinuity in the algorithm segmentation results at fine vascular bifurcations, loss of vessels with complex curvature morphology, and unclear feature distinction between the edges of retinal vessels and background areas.

[0005] As an efficient network structure, Transformer relies on self-attention to capture long-range global information and has achieved remarkable results in the field of natural language processing. Many people consider that global information is also urgently needed in computer vision tasks, and appropriate Transformer applications can help overcome the limitations of CNN, so researchers have devoted a lot of effort to exploring Transformers suitable for visual tasks. For example, Hu tried to use CNN to extract deep features, which were then fed into Transformer for processing and regression. Dosovitskiy and Cao both proposed pure Transformer networks to classify and segment images, respectively, and achieved great success. They segmented the image into multiple image blocks and treated each vectorized image block as a word / tag in NLP so that a suitable Transformer could be applied. Subsequently, based on the success of VIT, a large number of better Transformer-based architectures have emerged, and achieved better performance than CNN. However, the visual transformer still has the problem of high computational complexity and insufficient local information extraction, resulting in low accuracy.

[0006] Therefore, the existing method of segmenting retinal vascular images has the problem of low segmentation accuracy. Summary of the invention

[0007] The embodiment of the present invention provides a retinal vascular image segmentation method and device to improve the segmentation accuracy of the retinal vascular image.

[0008] In order to solve the above technical problems, the present application provides a retinal blood vessel image segmentation method, including:

[0009] Acquire a retinal vascular image, and input the retinal vascular image into a segmentation network model;

[0010] Based on the encoder-decoder module of the segmentation network model, feature extraction and fusion are performed on the retinal vascular image, and each pixel in the feature map obtained after feature extraction and fusion is classified to obtain at least one segmentation processing result;

[0011] A weighted average process is performed on all the segmentation processing results to obtain a final segmentation result of the retinal vascular image.

[0012] In order to solve the above technical problems, the embodiment of the present application further provides a retinal vascular image segmentation device based on Transformer optimization, comprising:

[0013] A retinal vascular image acquisition module, used to acquire a retinal vascular image and input the retinal vascular image into a segmentation network model;

[0014] A segmentation module, configured to extract and fuse features of the retinal vascular image based on an encoder-decoder module of the segmentation network model, and classify each pixel in the feature map obtained after feature extraction and fusion to obtain at least one segmentation processing result;

[0015] The segmentation result acquisition module is used to perform weighted average processing on all the segmentation processing results to obtain the final segmentation result of the retinal vascular image.

[0016] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above retinal vascular image segmentation method when executing the computer program.

[0017] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above retinal vascular image segmentation method are implemented.

[0018] A retinal vascular image segmentation method provided by an embodiment of the present invention obtains a retinal vascular image and inputs the retinal vascular image into a segmentation network model; based on the encoder-decoder module of the segmentation network model, features of the retinal vascular image are extracted and fused, and each pixel in the feature map obtained after feature extraction and fusion is classified to obtain at least one segmentation processing result; all the segmentation processing results are weighted averaged to obtain the final segmentation result of the retinal vascular image. The network structure based on Transformer optimization proposed for retinal vascular image segmentation is a U-shaped symmetric network based on Transformer optimization, which not only has a symmetrical encoder-decoder structure, but also adds a jump connection between the encoder part and the decoder part. While realizing end-to-end segmentation of retinal vascular images, accurate segmentation results can be obtained on a limited data set, thereby improving the segmentation accuracy of retinal vascular images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1is a flow chart of an embodiment of the retinal blood vessel image segmentation method proposed by the present invention;

[0021] Figure 2 A schematic diagram of the network model structure based on Transformer optimization provided by the present invention;

[0022] Figure 3 Detailed diagram of the optimized Transformer module in the present invention;

[0023] Figure 4 The segmentation results of blood vessel images in the DRIVE dataset;

[0024] Figure 5 The segmentation results of blood vessel images in the DRIVE dataset;

[0025] Figure 6 The segmentation results of blood vessel images in the STARE dataset;

[0026] Figure 7 is a schematic structural diagram of an embodiment of a retinal vascular image segmentation device according to the present application;

[0027] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 , Figure 1 A retinal blood vessel image segmentation method provided by an embodiment of the present invention is shown and described in detail as follows:

[0032] S201, obtaining a retinal vascular image, and inputting the retinal vascular image into a segmentation network model.

[0033] In step S201, the above-mentioned method of acquiring retinal vascular images includes but is not limited to acquiring images from public training sets, such as DRIVE, STARE and CHASE_DB1, and acquiring images based on imaging medical diagnosis technology, such as OCT imaging medical diagnosis technology.

[0034] The above segmentation network model refers to a pre-trained model for segmenting retinal vascular images. This model is a retinal vascular image segmentation network model based on Transformer optimization.

[0035] Through the segmentation network model optimized based on Transformer, end-to-end network training can be achieved without the need for complex post-processing methods. In the segmentation results, large-scale blood vessels will not appear hollow or broken, and low-contrast microvessels and edge vessels are also well preserved, which comprehensively improves the accuracy of blood vessel segmentation and the detection sensitivity of small blood vessels, thereby improving the segmentation accuracy of retinal vascular images.

[0036] In some optional implementations of this embodiment, before step 201, the retinal blood vessel image segmentation method further includes:

[0037] S101: Obtain a training image.

[0038] S102: Perform data enhancement processing on the training image to obtain an enhanced image.

[0039] S103: Use the training image and the enhanced image as training samples.

[0040] S104: Preprocess the training samples to obtain preprocessed training samples.

[0041] S105: Input the preprocessed training samples into the initial segmentation network model for training to obtain a segmentation network model.

[0042] For step S101, the training images include but are not limited to public training sets, such as the existing public retinal vascular datasets DRIVE, STARE and CHASE_DB1. For example, 20 images are randomly selected from the DRIVE dataset as training images, 16 images are randomly selected from the STARE dataset as training images, and the first 20 images are selected from the CHASE_DB1 dataset as training images.

[0043] For step S102, the above data enhancement includes but is not limited to histogram stretching, image flipping, and image rotation.

[0044] By using data augmentation method on retinal vascular images, training samples are expanded, and the training images and enhanced images obtained by data augmentation are used together as training samples of the segmentation network model.

[0045] Preferably, based on the histogram stretching method, data enhancement processing is performed on the training image to obtain an enhanced image.

[0046] By using the histogram stretching method on the retinal vascular image, an enhanced image is generated, and the training sample is doubled. The training image and the enhanced image are used together as training samples for the segmentation network model. Increasing the training data to a certain amount of data can avoid overfitting in training and improve the generalization ability of the segmentation network model.

[0047] For step S104, the above preprocessing includes but is not limited to normalization, Gaussian transformation, random rotation, random horizontal flipping and color correction processing.

[0048] Preferably, the training images are sequentially subjected to Gaussian transformation, random rotation, random horizontal flipping and color correction processing to obtain preprocessed training samples.

[0049] It should be understood that the preprocessing here can be changed accordingly according to the actual application scenario.

[0050] The following is an example to explain step S104, such as performing a Gaussian transformation with a probability of 0.5 on each color retinal vascular image, where Sigma = (0, 0.5); then performing a random rotation in the range of [0, 20°] and a random horizontal flip with a probability of 0.5 on the image; finally, applying a Gamma color correction in the range of [0.5, 2] to the image to adjust the image contrast.

[0051] By preprocessing the training images, the contrast of the training images is enhanced so as to improve the accuracy of subsequent feature extraction and other processing of the images, thereby improving the segmentation accuracy of the segmentation network model for retinal vascular images.

[0052] Through the above steps, the segmentation network model is obtained, which can realize end-to-end network training without complex post-processing methods. In the segmentation results, large-scale blood vessels will not appear hollow or broken, and low-contrast microvessels and edge vessels are also well preserved, which comprehensively improves the accuracy of blood vessel segmentation and the detection sensitivity of small blood vessels, thereby improving the segmentation accuracy of the segmentation network model for retinal blood vessel images.

[0053] S202. Based on the encoding module and the decoding module in the segmentation network model, perform feature extraction and fusion on the retinal vascular image, and classify each pixel in the feature map obtained after feature extraction and fusion to obtain at least one segmentation processing result.

[0054] In step S202, the segmentation network model is a network model for segmenting retinal vascular images, including but not limited to a segmentation network model based on Transformer optimization and a segmentation network model based on machine learning.

[0055] Preferably, a segmentation network model based on Transformer optimization is used here, such as Figure 2 As shown in the figure, the structure of the Transformer-optimized segmentation network model is divided into an encoder, a decoder, a skip connection, and a side output.

[0056] Wherein, both the encoder and the decoder include a Transformer module, and the Transformer module is the optimized Transformer proposed in the present invention.

[0057] It should be noted here that each optimized Transformer module consists of a LayerNorm normalization layer, a multi-head attention module, a residual connection and a multi-layer perception layer. The multi-head attention modules used in the two consecutive Transformer modules in the present invention are the Cross Patches Convolution Self-Attention (CPCA) module and the Inner Patches Convolution Self-Attention (IPCA). Among them, the CPCA module is used to extract the attention between the cross-patches in a feature map, and the IPCA module is used to extract and integrate the global feature information between pixels in one of the image blocks.

[0058] Two consecutive optimized Transformer modules can perform attention calculation according to the following formulas (1) to (4):

[0059]

[0060]

[0061]

[0062]

[0063] Among them, Y n-1 and Y n+1 are the input and output of the Transformer module in the present invention, and Respectively represent the output of CPCA module and IPCA module. Figure 3 As shown in the figure, LN is the LayerNorm normalization layer, MLP is the multi-layer perception layer, the image block size in the CPCA module and the IPCA module is 8×8, and the number of attention heads is 1 and 8 respectively.

[0064] The encoder includes four encoding modules, namely, a first encoding module, a second encoding module, a third encoding module and a fourth encoding module connected sequentially. The first encoding module includes an embedding module and two consecutive Transformer modules, the second encoding module and the third encoding module are both composed of a merging module and two consecutive Transformer modules, and the fourth encoding module only includes a merging module.

[0065] It should be noted here that the embedding module in the first encoding module consists of an upsampling and linear projection layer, which upsamples the 512×512 input image by 2 times to expand it to an image of 1024×1024 pixels, and then divides it into non-overlapping 256×256 image blocks, and increases the number of channels to 96. The merging module downsamples the input feature map by ×2 and increases the number of channels of the feature map. The restoration module upsamples the feature map by ×2 and reduces the number of channels of the feature map.

[0066] The decoder, which is symmetrical to the encoder, also includes four decoding modules, namely the first decoding module, the second decoding module, the third decoding module and the fourth decoding module, which are sequentially connected. Except for the first decoding module which only includes one restoration module, the other decoding modules are composed of one restoration module and two consecutive Transformer modules.

[0067] At the bottom of the segmentation network model, the Bottom Block consisting of 4 Transformer modules is used to connect the encoder and decoder.

[0068] In addition, the output of the first encoding module is successively fused with the input and output of the Transformer module in the second decoding module, the output of the second encoding module is successively fused with the input and output of the Transformer module in the third decoding module, and the output of the third encoding module is successively fused with the input and output of the Transformer module in the fourth decoding module, thereby forming a jump connection.

[0069] The side output includes four convolution modules, and the output of each convolution module is weighted averaged as the final segmentation result.

[0070] It should be noted here that the convolution module contains a 3×3 convolution, batch normalization, activation function ReLU, an upsampling and a 1×1 convolution connected sequentially. The spatial information lost in the downsampling process of each layer is restored, and the image is restored to the same size as the network input image to achieve end-to-end segmentation.

[0071] The segmentation network model is described in detail below with a specific embodiment. The network structure parameters based on Transformer optimization proposed in the present invention are shown in Table 1:

[0072] Table 1. Network structure parameters based on Transformer optimization

[0073]

[0074]

[0075] In this example, the network training is set to train the network for 250 epochs, the initial learning rate is 0.001, decays 10 times every 50 epochs, the batch size is set to 4, and the network based on Transformer optimization is trained using the Adam optimizer and the binary cross entropy loss function. The entire network is trained from scratch without using additional training data. We train the network on the DRIVE, STARE, and CHASE_DB1 training sets and evaluate them on the respective validation sets of each dataset. The experiments in this example are performed on an NVIDIA Tesla V100 32GB GPU using the PyTorch framework as the backend implementation.

[0076] In order to evaluate the performance of the model proposed in this invention, the segmentation results are compared with the expert segmentation map (Groungtruth, which takes the expert segmentation results as the standard, i.e., the label of the training data), and the comparison results of each pixel are divided into true positive (TP), false positive (FP), false negative (FN), and true negative (TN). Then, we use sensitivity (Se), specificity (Sp), accuracy (Acc), F1-Score (F1), and the area under the receiver operating characteristic (ROC) (AUC) as measurement indicators to evaluate the performance of the model, and the formula is as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] Among them, TP means that the true value is positive when a blood vessel pixel in the expert segmentation map is correctly classified in the predicted image, TN means that the true value is negative when a non-vascular pixel in the expert segmentation map is correctly classified in the predicted image, FN means that the blood vessel pixels in the expert segmentation map are incorrectly classified as non-vascular pixels in the predicted image, and FP means that the non-vascular pixels in the expert segmentation map are incorrectly marked as blood vessel pixels in the predicted image. Precision and Recall are the precision and recall rates, respectively, and the ROC curve represents the proportion of blood vessels correctly classified as blood vessel pixels and the proportion of pixels incorrectly classified as non-vascular pixels. AUC refers to the area under the ROC curve, which can be used to measure the performance of segmentation. The closer the AUC value is to 1, the closer the system performance is to perfection.

[0083] It should be noted that in the comparison of network structures of different methods, in order to benchmark the performance of the model, DRIVE, STARE and CHASE_DB1 retinal vascular segmentation datasets are used to compare different network structure systems. In a specific example, as shown in Table 2, Table 3 and Table 4, the method proposed in this application is tested with the U-Net network, R2UNet network, DFUNet network, DenseBlock-UNet network, and IterNet network on three public datasets, and the test performance is compared. It is better than other methods in AUC, ACC, and SP, which are 0.9869 / 0.9627 / 0.9902, 0.9945 / 0.9772 / 0.9903, and 0.9917 / 0.9805 / 0.9963 respectively, and the sensitivity and F1 value are also significantly improved. It can be clearly compared that the method proposed in the present invention has achieved the best performance.

[0084] Table 2: Performance comparison on DRIVE dataset

[0085]

[0086]

[0087] Table 3: Performance comparison on the STARE dataset

[0088]

[0089] Table 4: Performance comparison on the CHASE_DB dataset

[0090]

[0091] like Figure 4 , Figure 5 and Figure 6 As shown in FIG. 1 , (a) is the original retinal vascular image; (b) is the mask; (c) is the gold standard; and (d) is the segmentation result of the present invention. Figure 4 , Figure 5 and Figure 6 It can be seen that the retinal vessel segmentation result in the present invention is very close to the gold standard, which proves that the retinal vessel segmentation method based on Transformer optimization provided by the present invention has good vessel segmentation performance, improves the detection ability of low-contrast microvessels, and the segmented blood vessel trunks have good connectivity, reducing blood vessel breaks.

[0092] The present invention aims to solve the problem of few retinal vascular image samples and low detection of low-contrast microvessels, and proposes a retinal vascular image segmentation method. By constructing a symmetrical structure of the Transformer-based Encoder-Decoder and adding a convolutional layer to the multi-head attention of the Transformer module, the dependency of long-distance pixels and the local details of blood vessels can be better obtained, and the low-contrast microvessels in the segmentation results can be better preserved, which comprehensively improves the accuracy of blood vessel segmentation and the detection sensitivity of small blood vessels.

[0093] The segmentation network model optimized based on Transformer adopts a symmetrical structure of decoder-encoder. Both the decoder and encoder are built based on Transformer, and the two parts are connected by skip connection, so that the network can obtain the information lost in the downsampling process due to the reduction of the image size during the upsampling process, and at the same time can obtain sufficient contextual information and semantic information, thereby obtaining a better segmentation effect. The multi-head attention in the Transformer module adds a convolution layer. The Transformer itself has a global receptive field and can better obtain long-distance dependencies, but it is lacking in obtaining local detail information. The present invention adds a 3×3 convolution layer, which can better supplement the local detail feature information. Through the segmentation network model, the retinal vascular image is segmented and processed to improve the segmentation accuracy of the segmentation network model for the retinal vascular image.

[0094] In some optional implementations of this embodiment, in step S202, it specifically includes the following steps S2021 to S2025:

[0095] S2021. Based on the multi-layer coding blocks corresponding to the encoder of the segmentation network model, down-sample the retinal vascular image to obtain down-sampled features corresponding to each layer of coding blocks.

[0096] S2022. Use the downsampled features corresponding to the bottom-level coding block of the segmentation network model as the current feature map, and use the decoding block in the decoder of the segmentation network model corresponding to the current feature map as the current decoding block.

[0097] S2023. Input the current feature map into the current decoding block, and based on the current decoding block, perform feature restoration processing on the current feature map to obtain restored features.

[0098] S2024. Fusing the down-sampled features and the restored features in the coding block of the previous layer of the coding block corresponding to the current feature map to obtain up-sampled features, and fusing the up-sampled features with the down-sampled features in the coding block of the previous layer corresponding to the current feature map to obtain fused features.

[0099] S2025. Use the previous layer decoding block corresponding to the current decoding block as the current decoding block, use the fused features as the current feature map, return and input the current feature map into the current decoding block, and based on the current decoding block, perform feature restoration processing on the current feature map to obtain the restored feature step and continue to execute until the current decoding block is the highest layer decoding block of the segmentation network model.

[0100] For step S2021, an alternate row and alternate column replication method is used to implement image downsampling, thereby reducing the resolution of the image and increasing the number of channels.

[0101] For step S2023, the restoration module of the decoder of the segmentation network model is used for upsampling, the image size is increased by 2 times, the channel is reduced by 2 times, and upsampling is achieved through deconvolution to restore the pixel space and the details of the reconstructed target.

[0102] For step S2024, illustratively, Figure 2 The structural diagram in is used as an example for explanation, in which the structural diagram includes the bottom layer coding block, the third layer coding block, the second layer coding block and the highest layer coding block. After the feature restoration processing is performed on the current feature map to obtain the restored feature, the upper layer coding block of the bottom layer coding block, that is, the down-sampled feature of the third layer coding module, is input together with the restored feature (decoded by the bottom layer decoding block) into the Transformer in the current decoding block for fusion processing to obtain the up-sampled feature, and then the up-sampled feature is further fused with the down-sampled feature of the third coding module for detailed feature processing to obtain the fused feature, and then the upper layer decoding block (the third layer decoding block) corresponding to the current decoding block is used as the current decoding block, and the fused feature is used as the current feature map, and the current feature map is input into the current decoding block, and based on the current decoding block, the feature restoration processing is performed on the current feature map to obtain the restored feature; the down-sampled feature of the second layer coding module is input together with the restored feature into the Transformer in the current decoding block for fusion processing to obtain the up-sampled feature, and then the up-sampled feature is fused with the down-sampled feature of the second coding module for detailed feature processing to obtain the fused feature. Then, the decoding block of the previous layer (the highest layer decoding block) corresponding to the current decoding block is used as the current decoding block, and the fused features are used as the current feature map. The current feature map is input into the current decoding block, and based on the current decoding block, the current feature map is restored to obtain the restored features. The downsampled features of the first layer encoding module are input together with the restored features into the Transformer in the current decoding block for fusion processing to obtain the upsampled features, and then the upsampled features are fused with the detail features of the downsampled features of the first encoding module to obtain the fused features.

[0103] It should be noted that according to Figure 2As shown in the schematic diagram, the bottom decoding block is used as the bottom layer decoding block, the decoding block on the upper layer of the bottom layer decoding block is used as the third layer decoding block, and so on, the decoding block on the top layer is used as the highest layer decoding block. The number of decoding blocks can be selected according to actual needs and is not limited here. As a preferred method, the number of layers of decoding blocks in this embodiment is 4.

[0104] For step S2025, the above fusion processing is implemented through skip connection.

[0105] By adding skip connections in the decoder and encoder, the input of each decoding module is a fusion of the output of the previous layer and the output of the corresponding encoding module. This operation allows the upsampling process to obtain sufficient contextual information and semantic information at the same time.

[0106] By building an encoder and decoder based on Transformer and connecting the two parts with jump connections, the network can obtain information lost during downsampling due to image size reduction during upsampling, while also obtaining sufficient contextual and semantic information to improve the segmentation accuracy of retinal vascular images by the segmentation network model.

[0107] In step S203, weighted average processing is performed on all segmentation processing results to obtain a final segmentation result of the retinal vascular image.

[0108] In step S204, the retinal vascular image is input into the trained network model, and then the output result of the decoding module of each layer is passed through the convolution module to output the predicted segmentation result of each layer. The present invention performs weighted averaging on the predicted results of each layer to obtain the classification probability of each pixel, and finally obtains the segmentation result of the retinal vascular image.

[0109] When performing vascular segmentation on the retinal vascular image using the trained segmentation network model, the prediction results output by each layer of the decoding module are weighted averaged to obtain the classification probability of each pixel, which is used as the final vascular segmentation result to further improve the accuracy of retinal vascular image segmentation.

[0110] The retinal vascular image segmentation method provided by the embodiment of the present invention obtains the retinal vascular image and inputs the retinal vascular image into the segmentation network model; based on the encoder-decoder module of the segmentation network model, the retinal vascular image is feature extracted and fused, and each pixel in the feature map obtained after feature extraction and fusion is classified to obtain at least one segmentation processing result; all segmentation processing results are weighted averaged to obtain the final segmentation result of the retinal vascular image, and the network structure based on Transformer optimization proposed for retinal vascular image segmentation is a U-shaped symmetrical network built based on Transformer, which not only has a symmetrical encoder-decoder structure, but also adds a jump connection between the encoder part and the decoder part. While realizing the end-to-end segmentation of the retinal vascular image, accurate segmentation results can be obtained on a limited data set, thereby improving the segmentation accuracy of the retinal vascular image.

[0111] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0112] Figure 7 FIG. 2 is a block diagram showing a principle of a retinal vascular image segmentation device corresponding to the retinal vascular image segmentation method in the above embodiment. Figure 7 As shown, the retinal vascular image segmentation device includes a retinal vascular image acquisition module 31, a segmentation module 32 and a segmentation result acquisition module 33. The functional modules are described in detail as follows:

[0113] The retinal vascular image acquisition module 31 is used to acquire the retinal vascular image and input the retinal vascular image into the segmentation network model.

[0114] The segmentation module 32 is used to extract and fuse features of the retinal vascular image based on the encoding module and the decoding module of the segmentation network model, and classify each pixel in the feature map obtained after feature extraction and fusion to obtain at least one segmentation processing result.

[0115] The segmentation result acquisition module 33 is used to perform weighted average processing on all segmentation processing results to obtain the final segmentation result of the retinal vascular image.

[0116] In one embodiment, before the retinal vascular image acquisition module 31, the retinal vascular image segmentation device further includes:

[0117] The training image acquisition module is used to acquire training images.

[0118] The data enhancement module is used to perform data enhancement processing on the training image to obtain an enhanced image.

[0119] The training sample acquisition module is used to use the training images and enhanced images as training samples.

[0120] The preprocessing module is used to preprocess the training samples to obtain preprocessed training samples.

[0121] The model training module is used to input the preprocessed training samples into the initial segmentation network model for training to obtain the segmentation network model.

[0122] In one embodiment, the data enhancement module further comprises:

[0123] The histogram stretching unit is used to perform data enhancement processing on the training image based on the histogram stretching method to obtain an enhanced image.

[0124] In one embodiment, the pre-processing module further comprises:

[0125] The preprocessing unit is used to perform Gaussian transformation, random rotation, random horizontal flipping and color correction processing on the training image in sequence to obtain preprocessed training samples.

[0126] In one embodiment, the segmentation module 32 further comprises:

[0127] The downsampling unit is used to downsample the retinal vascular image based on the multi-layer coding blocks corresponding to the encoder of the segmentation network model to obtain the downsampled features corresponding to each layer of the coding blocks.

[0128] The current coding block acquisition unit is used to use the down-sampled features corresponding to the bottom-level coding block of the segmentation network model as the current feature map, and use the decoding block in the decoder of the segmentation network model corresponding to the current feature map as the current decoding block.

[0129] The feature restoration unit is used to input the current feature map into the current decoding block, and based on the current decoding block, perform feature restoration processing on the current feature map to obtain restored features.

[0130] The fusion unit is used to fuse the down-sampled features in the coding block of the previous layer of the coding block corresponding to the current feature map with the restored features to obtain the up-sampled features, and fuse the up-sampled features with the down-sampled features in the coding block of the previous layer corresponding to the current feature map to obtain the fused features.

[0131] The loop unit is used to take the previous layer decoding block corresponding to the current decoding block as the current decoding block, take the fused feature as the current feature map, return the current feature map to the current decoding block, and based on the current decoding block, perform feature restoration processing on the current feature map to obtain the restored feature step and continue to execute until the current decoding block is the highest layer decoding block of the segmentation network model.

[0132] In one embodiment, the downsampling unit further comprises:

[0133] The down-sampling feature acquisition unit is used to down-sample the retinal vascular image based on the merging module and the Transformer module in the multi-layer encoding block corresponding to the encoder of the segmentation network model, so as to obtain the down-sampling features corresponding to each layer of the encoding block.

[0134] The specific definition of the retinal vascular image segmentation device can be found in the definition of the retinal vascular image segmentation method above, which will not be repeated here. Each module in the above-mentioned retinal vascular image segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0135] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0136] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0137] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the retinal vascular image segmentation method as described above.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0139] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A retinal vascular image segmentation method, It is characterized in that The retinal blood vessel image segmentation method comprises: Acquire a retinal vascular image, and input the retinal vascular image into a segmentation network model; Based on the encoding module and the decoding module of the segmentation network model, feature extraction and fusion are performed on the retinal vascular image, and each pixel in the feature map obtained after feature extraction and fusion is classified to obtain at least one segmentation processing result; Performing weighted average processing on all the segmentation processing results to obtain a final segmentation result of the retinal vascular image; The encoding module and the decoding module based on the segmentation network model perform feature extraction and fusion on the retinal vascular image, including: Based on the merging modules and Transformer modules in the multi-layer coding blocks corresponding to the encoder of the segmentation network model, the retinal vascular image is downsampled to obtain downsampled features corresponding to the coding blocks of each layer, wherein the encoder includes a first coding module, a second coding module, a third coding module and a fourth coding module, the first coding module includes an embedding module and two consecutive Transformer modules, the second coding module and the third coding module are each composed of a merging module and two consecutive Transformer modules, and the fourth coding module includes a merging module; The down-sampled features corresponding to the bottom-level coding block of the segmentation network model are used as the current feature map, and the decoding block in the decoder of the segmentation network model corresponding to the current feature map is used as the current decoding block, wherein the decoder symmetrical to the encoder includes a first decoding module, a second decoding module, a third decoding module and a fourth decoding module, the first decoding module includes a restoration module, and the second decoding module, the third decoding module and the fourth decoding module are each composed of a restoration module and two consecutive Transformer modules; Inputting the current feature map into the current decoding block; Based on the current decoding block, performing feature restoration processing on the current feature map to obtain restored features; Fusing the downsampled features in the coding block of the previous layer of the coding block corresponding to the current feature map with the restored features to obtain upsampled features, and fusing the upsampled features with the downsampled features in the coding block of the previous layer corresponding to the current feature map to obtain fused features; The previous layer decoding block corresponding to the current decoding block is used as the current decoding block, the fused feature is used as the current feature map, and the step of inputting the current feature map into the current decoding block is returned to continue execution until the current decoding block is the highest layer decoding block of the segmentation network model.

2. The retinal blood vessel image segmentation method according to claim 1, It is characterized in that Before acquiring the retinal vascular image and inputting the retinal vascular image into the segmentation network model, the method includes: Get training images; Performing data enhancement processing on the training image to obtain an enhanced image; Using the training image and the enhanced image as training samples; Preprocessing the training samples to obtain preprocessed training samples; The preprocessed training samples are input into the initial segmentation network model for training to obtain a segmentation network model.

3. The method according to claim 2, It is characterized in that The step of performing data enhancement processing on the training image to obtain an enhanced image comprises: Based on the histogram stretching method, data enhancement processing is performed on the training image to obtain an enhanced image.

4. The method according to claim 2, It is characterized in that The step of preprocessing the training samples to obtain preprocessed training samples comprises: The training images are sequentially subjected to Gaussian transformation, random rotation, random horizontal flipping and color correction processing to obtain preprocessed training samples.

5. A retinal vascular image segmentation device based on Transformer optimization, It is characterized in that The retinal blood vessel image segmentation device comprises: A retinal vascular image acquisition module, used to acquire a retinal vascular image and input the retinal vascular image into a segmentation network model; A segmentation module, configured to extract and fuse features of the retinal vascular image based on the encoding module and the decoding module of the segmentation network model, and classify each pixel in the feature map obtained after the feature extraction and fusion to obtain at least one segmentation processing result; A segmentation result acquisition module, used for performing weighted average processing on all the segmentation processing results to obtain a final segmentation result of the retinal vascular image; Wherein, the segmentation module includes: Based on the merging modules and Transformer modules in the multi-layer coding blocks corresponding to the encoder of the segmentation network model, the retinal vascular image is downsampled to obtain downsampled features corresponding to the coding blocks of each layer, wherein the encoder includes a first coding module, a second coding module, a third coding module and a fourth coding module, the first coding module includes an embedding module and two consecutive Transformer modules, the second coding module and the third coding module are each composed of a merging module and two consecutive Transformer modules, and the fourth coding module includes a merging module; The down-sampled features corresponding to the bottom-level coding block of the segmentation network model are used as the current feature map, and the decoding block in the decoder of the segmentation network model corresponding to the current feature map is used as the current decoding block, wherein the decoder symmetrical to the encoder includes a first decoding module, a second decoding module, a third decoding module and a fourth decoding module, the first decoding module includes a restoration module, and the second decoding module, the third decoding module and the fourth decoding module are each composed of a restoration module and two consecutive Transformer modules; Inputting the current feature map into the current decoding block; Based on the current decoding block, performing feature restoration processing on the current feature map to obtain restored features; Fusing the downsampled features in the coding block of the previous layer of the coding block corresponding to the current feature map with the restored features to obtain upsampled features, and fusing the upsampled features with the downsampled features in the coding block of the previous layer corresponding to the current feature map to obtain fused features; The previous layer decoding block corresponding to the current decoding block is used as the current decoding block, the fused feature is used as the current feature map, and the step of inputting the current feature map into the current decoding block is returned to continue executing until the current decoding block is the highest layer decoding block of the segmentation network model.

6. The retinal blood vessel image segmentation device according to claim 5, It is characterized in that Before the segmentation result acquisition module, the device further includes: A training image acquisition module, used for acquiring training images; A data enhancement module, used to perform data enhancement processing on the training image to obtain an enhanced image; A training sample acquisition module, used to use the training image and the enhanced image as training samples; A preprocessing module, used for preprocessing the training samples to obtain preprocessed training samples; The model training module is used to input the preprocessed training samples into the initial segmentation network model for training to obtain a segmentation network model.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and running on the processor, It is characterized in that When the processor executes the computer program, the retinal vessel image segmentation method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the retinal vessel image segmentation method according to any one of claims 1 to 4 is implemented.

Images