Unsupervised retinal image segmentation method based on dual-scale fully convolutional reconstruction network

Abstract

The present invention discloses an unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network, which relates to the field of medical image processing technology; including: designing a decoder architecture, using a three-layer transposed convolutional network as a decoder; this decoder and the encoder jointly construct a seven-layer fully convolutional network; and also including: designing a dual-scale reconstruction network structure. Compared with the existing CUTS method, the present invention redesigns the backbone network and adopts a fully convolutional structure, which is more suitable for segmentation tasks; at the same time, combined with the designed backbone network, a method combining dual-scale reconstruction and local contrast learning is proposed, and global reconstruction loss is introduced into the overall loss function; on the basis of fully considering all aspects of image information, more robust feature information is provided for the next stage of segmentation tasks; this method can not only better capture the texture information inside the image, but also consider global information to improve segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network. Background Art

[0002] A late form of age-related macular degeneration (AMD) is geographic atrophy (GA), which leads to irreversible loss of visual function. The causes of geographic atrophy are complex, involving persistent oxidative stress, chronic inflammation, genetic and environmental factors, and it has become one of the main causes of vision impairment in the elderly. In the medical field, the development of artificial intelligence has enabled deep learning to achieve remarkable results in image segmentation. Deep learning uses neural networks to capture areas of interest in medical images, such as different organs, tissues, and cell types. The application of this technology in the medical field is becoming increasingly important, helping clinicians to more easily diagnose and monitor the progression of the disease and improve patient care.

[0003] Early medical image segmentation methods mainly relied on manually designed features, but have gradually been replaced by deep learning-based methods. Deep learning-based methods have superior performance and lower overhead. However, supervised neural network-based methods still have some problems in the medical field. First, the workload of labeling images for segmentation is huge, and the scarcity of medical imaging experts further increases the difficulty of labeling. Second, medical images often involve patient privacy, making data acquisition more difficult. Finally, the use of supervised networks may not be well generalized to data from different hospitals and equipment.

[0004] However, the existing CUTS method (such as the method disclosed in the paper: Amodio M, Gao F, Avesta A, et al. CUTS: A Fully Unsupervised Framework for Medical Image Segmentation [J]. arXiv preprint arXiv: 2209.11359, 2022) has some problems: First, the reconstruction network uses a simple linear network layer for patch reconstruction, which cannot reconstruct the texture and structural information of the patch well; second, the CUTS network adopts a combination of contrastive learning and local patch reconstruction, which takes into account the internal structure and texture information of the image, but does not consider the global image information well. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the prior art and propose an unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Unsupervised retinal image segmentation method based on dual-scale fully convolutional reconstruction network, including:

[0008] Design a decoder architecture, using a three-layer transposed convolutional network as the decoder; this decoder and encoder jointly construct a seven-layer fully convolutional network;

[0009] Also includes: Designing a dual-scale reconstruction network structure:

[0010] The entire image is directly reconstructed through a redesigned decoder. Local patch reconstruction and global image reconstruction are not mutually exclusive, and they complement each other to achieve better restoration results. The selected method or combination is determined by the specific restoration task and requirements. By combining global image reconstruction with local patch image reconstruction, the accuracy and completeness of image restoration are improved.

[0011] Also includes: loss function design;

[0012] The loss function combines local contrastive learning, local patch reconstruction loss, and global image reconstruction loss; the goal of the contrastive loss is to ensure that the encoder maps similar pixel-centered patches to the embedding space in an organized manner;

[0013] When designing the loss function, we use the proximity heuristic and image structure similarity metric to determine the positive and negative pairs; positive examples are local neighborhood patches whose similarity to the anchor patch is higher than a threshold, while negative examples are patches that are far from the anchor patch or fail the similarity check;

[0014] In the embedding space, contrastive learning is performed to compare the embedding vectors of each anchor patch and In this way, positive and negative pairs are generated, contrastive loss is calculated, and invariance in the embedding space is established; positive samples from different anchor patches are used as negative samples of the current anchor patch to improve the mining speed; the specific formula is as follows:

[0015]

[0016] Among them, minimizing this loss function encourages the network to convert the positive sample set Ω + Embedded to a position close to the anchor point, and the degree of closeness is determined by the negative set Ω - The temperature parameter τ controls the scaling of the distance used in the contrastive loss and is fixed to 0.5. sim(·) is the similarity function, which is set to cosine similarity in the implementation.

[0017] Preferably, in the loss function design, the local reconstruction loss and the global reconstruction loss use the same loss function, and the reconstruction loss is used to ensure that the embedding vector of each pixel-centered patch retains the information of the surrounding patches and the structure and texture information of the overall image.

[0018] Preferably: In the loss function design, for embedding The patch reconstruction loss is:

[0019] Preferably: the loss function is designed during training, using a patch reconstruction module to reconstruct the patch, which is jointly optimized with the convolutional encoder.

[0020] Preferably: the reconstruction loss of the global image is introduced, using To process the global feature map output by the encoder;

[0021]

[0022] The overall objective function is the sum of the contrast loss and the reconstruction loss, balanced by a weight coefficient λ∈[0,1] that controls the relative sizes:

[0023] loss = λ·l contrastive +(1-2·λ)·l recon +λ·l global

[0024] Here, λ is set to 0.01.

[0025] The beneficial effects of the present invention are:

[0026] 1. Compared with the existing CUTS method, this paper redesigns the backbone network and adopts a fully convolutional structure, which is more suitable for segmentation tasks. At the same time, combined with the designed backbone network, a method combining dual-scale reconstruction and local contrastive learning is proposed, and a global reconstruction loss is introduced into the redesigned overall loss function. Based on the full consideration of various aspects of image information, more robust feature information is provided for the next stage of segmentation task. This method can not only better capture the texture information within the image, but also consider global information to improve segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Example segmentation map of geographic atrophy on retinal fundus image;

[0028] Figure 2 This is the FCNCUTS network framework diagram of the unsupervised retinal image segmentation method based on the dual-scale fully convolutional reconstruction network proposed in this invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention will be further described in detail below in conjunction with specific implementation methods.

[0030] Example 1:

[0031] An unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network. The decoder converts the encoder output into the original image pixel representation and uses some known pixels or features to predict the pixel values ​​of the masked / lost area.

[0032] Designing the decoder architecture to increase complexity or use a deeper network can better extract image feature details. Specifically, a three-layer transposed convolutional network is used as the decoder. Compared with the original framework's single-layer fully connected architecture, this design has the advantages of preserving the original image's spatial structure and details, reducing parameters, and providing stronger generalization capabilities. By learning specific features, it can generate complex image reconstructions.

[0033] This decoder and encoder jointly construct a 7-layer fully convolutional network, which can achieve more refined segmentation, enrich features and support complex tasks in downstream image segmentation tasks;

[0034] In image processing, preferably, a multi-scale segmentation method is used to perform detailed segmentation processing on retinal structures at different scales.

[0035] Also includes: Designing a dual-scale reconstruction network structure:

[0036] CUTS uses local patch image reconstruction, which uses the contextual information of surrounding pixels to infer missing or damaged pixels. However, this method has limitations and shortcomings, such as being limited by context and unable to process subtle texture information and global structural information. To overcome these limitations, we propose a dual-scale reconstruction network structure that directly reconstructs the entire image through a redesigned decoder; this combination can make up for the shortcomings of the local patch reconstruction method and provide better global consistency and detail preservation. It should be noted that the local patch reconstruction and global image reconstruction methods are not mutually exclusive, and the two complement each other to achieve better restoration effects; the selected method or combination is determined according to the specific restoration task and requirements; by combining global image reconstruction with local patch image reconstruction, the accuracy and completeness of image restoration are improved. This dual reconstruction network structure has potential in practical applications and can effectively improve the quality and effect of image restoration.

[0037] Also includes: Loss function design:

[0038] The loss function combines local contrastive learning, local patch reconstruction loss, and global image reconstruction loss; the goal of the contrastive loss is to ensure that the encoder maps similar pixel-centered patches to the embedding space in an organized manner;

[0039] When designing the loss function, we use the neighbor heuristic and the image structural similarity metric (SSIM) to determine the positive and negative pairs; positive examples are local neighborhood patches whose similarity to the anchor patch is higher than a threshold, while negative examples are patches that are far away from the anchor patch or fail the similarity check;

[0040] In the embedding space, contrastive learning is performed to compare the embedding vectors of each anchor patch and In this way, positive and negative pairs are generated and contrastive loss is calculated without using supervised labels, and invariance in the embedding space is established. The negative sample sampling process is simplified, and positive samples from different anchor patches are used as negative samples of the current anchor patch to improve the mining speed. The specific formula is as follows:

[0041]

[0042] Minimizing this loss function will encourage the network to convert the positive sample set Ω + Embedded to a position close to the anchor point, and the degree of closeness is determined by the negative set Ω - and the distance between the anchor points; in this way, this loss encourages similar pairs to be embedded closer and dissimilar pairs to be embedded farther away; the temperature parameter τ controls how the distance used in the contrastive loss is scaled, and its value is fixed to 0.5; sim(·) is the similarity function, which is set to cosine similarity in the implementation.

[0043] The local reconstruction loss and the global reconstruction loss use the same loss function, and the reconstruction loss is used to ensure that the embedding vector of each pixel-centered patch retains the information of the surrounding patches and the structure and texture information of the overall image; specifically, for the embedding The patch reconstruction loss is:

[0044]

[0045] During training, the patch reconstruction module is used to reconstruct the patch, which is jointly optimized with the convolutional encoder.

[0046] In order to better capture the overall structure and texture information of the image, a global image reconstruction loss is introduced using To process the global feature map output by the encoder;

[0047]

[0048] The overall objective function is the sum of the contrast loss and the reconstruction loss, balanced by a weight coefficient λ∈[0,1] that controls the relative sizes:

[0049] loss = λ·l contrastive+(1-2·λ)·l recon +λ·l global

[0050] Here, λ is set to 0.01.

[0051] In order to solve the problems raised by the background technology, the present invention has redesigned the backbone network and adopted a fully convolutional structure, which is more suitable for segmentation tasks. At the same time, combined with the designed backbone network, a method combining dual-scale reconstruction and local contrast learning is proposed, and global reconstruction loss is introduced into the overall loss function; on the basis of fully considering all aspects of image information, more robust feature information is provided for the next stage of segmentation task. This method can not only better capture the texture information inside the image, but also consider global information and improve the segmentation accuracy. Among them, the specific implementation method of the segmentation task in the next stage can adopt the method disclosed in the paper: Amodio M, Gao F, Avesta A, et al. CUTS: A Fully Unsupervised Framework for Medical Image Segmentation [J]. arXiv preprintarXiv: 2209.11359, 2022, which will not be repeated here.

[0052] experiment:

[0053] Dataset and data preprocessing:

[0054] We validated our model based on the publicly available retinal fundus image dataset from the CUTS paper (Amodio M, Gao F, Avesta A, et al. CUTS: AFully Unsupervised Framework for Medical Image Segmentation [J]. arXiv preprint arXiv:2209.11359, 2022). Each image is 128x128 pixels and has three channels. Our dataset partitioning follows the method in the CUTS paper, which is unique. Because it uses unsupervised or self-supervised learning methods, CUTS recommends:

[0055] (1) During model training, the data is divided into a training set and a validation set in a ratio of 7:3;

[0056] (2) The entire dataset was used as the test set to evaluate the segmentation effect. This partitioning method is considered reasonable, mainly because the model does not rely on the true labels during the training phase. To further verify the rationality of this partitioning method, we retrained and tested the data in an 8:1:1 ratio and found that the results obtained by the two methods were similar.

[0057] Experimental treatment

[0058] ① Experimental environment

[0059] The model was implemented using Python 3.8 and PyTorch, making it suitable for deep learning tasks. The training and testing environment consisted of a server equipped with two DCU accelerator cards with 16GB of video memory and 40GB of RAM, providing powerful hardware support for deep learning tasks. The learning rate was set to 0.001. The batch size was set to 16, meaning that each time the model was trained, 16 samples were used to calculate the loss and perform weight updates. The model was trained for 200 epochs, each consisting of one iteration of the entire training dataset, to help the model gradually converge to optimal performance.

[0060] ②Evaluation indicators

[0061] Our evaluation metrics include the Dice coefficient, Hausdorff distance, SSIM, ERGAS, and RMSE. These metrics are used to compare different areas, such as sample similarity, distance between point sets, image structure similarity, signal processing algorithm performance, and model prediction capabilities. They have different applications and calculation methods in different scenarios and can be adjusted to meet specific needs.

[0062] ③Experimental results

[0063] In our study, we applied our proposed method (FCNCUTS) to a retinal image dataset and compared it with similar alternative methods. Figure 1 As shown in Figure 3, the multi-scale segmentation method can provide detailed segmentation results of retinal structures at different scales.

[0064] On the coarser side, our method successfully segmented the entire retina accurately from the background. Thanks to our multi-scale segmentation approach, we can visualize finer anatomical structures, such as the optic disc and the emergence of areas of geographic atrophy. This provides a powerful tool for further applications, such as automatically measuring the size, shape, and location of these structures for planning clinical interventions. Our method's ability to capture smaller structures offers potential for more sophisticated tasks, such as retinal vessel segmentation. Furthermore, to identify regions of interest, such as GA (geographic atrophy), we introduced an additional step: we binarized the segmentation labels.

[0065] The binary segmentation generated by the FCNCUTS method is able to accurately identify the atrophic regions. Despite facing many challenges in processing this dataset, including color variations, similarity of blood vessels and retinal regions, our method still successfully segmented the GA regions and was able to accurately identify the core regions of atrophy, unaffected by factors such as color variations in the dataset.

[0066] These results demonstrate the effectiveness of our method in retinal image segmentation and its ability to capture retinal structure at multiple scales. Our work provides a powerful tool for automated retinal segmentation and analysis, which is expected to be widely applicable in clinical practice.

[0067] Experimental results validate the effectiveness of our segmentation method. Compared to other unsupervised segmentation methods, the segmentation maps generated by our FCNCUTS method have a higher similarity to the ground truth data, as reflected by higher SSIM values ​​and lower ERGAS and RMSE values. Specifically, the DICE index increased by 3.4%, the Hausdorff distance decreased by 3.1%, the SSIM index increased by 4.1%, the RMSE index decreased by 3.5%, and the ERGAS index decreased by 16.7%.

[0068] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network, characterized by: include: Design the decoder architecture and use a three-layer transposed convolutional network as the decoder; This decoder and encoder jointly construct a seven-layer fully convolutional network; Also includes: Designing a dual-scale reconstruction network structure: The entire image is directly reconstructed through a redesigned decoder. Local patch reconstruction and global image reconstruction are not mutually exclusive, and they complement each other to achieve better restoration results. The selected method or combination is determined by the specific restoration task and requirements. By combining global image reconstruction with local patch image reconstruction, the accuracy and completeness of image restoration are improved. Also includes: loss function design; The loss function combines local contrastive learning, local patch reconstruction loss, and global image reconstruction loss; the goal of the contrastive loss is to ensure that the encoder maps similar pixel-centered patches to the embedding space in an organized manner; When designing the loss function, we use the proximity heuristic and image structure similarity metric to determine the positive and negative pairs; positive examples are local neighborhood patches whose similarity to the anchor patch is higher than a threshold, while negative examples are patches that are far from the anchor patch or fail the similarity check; In the embedding space, contrastive learning is performed to compare the embedding vectors of each anchor patch and In this way, positive and negative pairs are generated, contrastive loss is calculated, and invariance in the embedding space is established; positive samples from different anchor patches are used as negative samples of the current anchor patch to improve the mining speed; the specific formula is as follows: Among them, minimizing this loss function encourages the network to convert the positive sample set Ω + Embedded to a position close to the anchor point, and the degree of closeness is determined by the negative set Ω - The temperature parameter τ controls the scaling of the distance used in the contrastive loss and is fixed to 0.

5. sim(·) is the similarity function, which is set to cosine similarity in the implementation.

2. The unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network according to claim 1, characterized in that: In the loss function design, the local reconstruction loss and the global reconstruction loss use the same loss function, and the reconstruction loss is used to ensure that the embedding vector of each pixel-centered patch retains the information of the surrounding patches as well as the structure and texture information of the overall image.

3. The unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network according to claim 2, characterized in that: In the loss function design, for embedding The patch reconstruction loss is:

4. The unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network according to claim 3, characterized in that: The loss function is designed in the training process, using the patch reconstruction module f recon (·): to reconstruct the patch, which is jointly optimized with the convolutional encoder.

5. The unsupervised retinal image segmentation method based on a dual-scale fully convolutional reconstruction network according to claim 4, characterized in that: The global image reconstruction loss is introduced, using f global (·): To process the global feature map output by the encoder; The overall objective function is the sum of the contrast loss and the reconstruction loss, balanced by a weight coefficient λ∈[0,1] that controls the relative sizes: loss=λ·l contrastive +(1-2·λ)·l recon +λ·l global Here, λ is set to 0.01.

Images