Lightweight real-time retinal segmentation method, system and apparatus

Abstract

The application discloses a kind of based on lightweight retinal real-time segmentation method, system and device of double channel structure, comprising: S1, obtain retinal vascular dataset and divide into training set, verification set and test set;S2, the training set and verification set are preprocessed to obtain preprocessed dataset;S3, the training set and verification set after pre-processing are data augmented;S4, build lightweight real-time segmentation network based on double channel, use augmented training set to train segmentation network;S5, use augmented verification set to evaluate segmentation network, if performance is greater than a certain threshold, then output trained segmentation network, pre-process augmented test set, if performance is lower than a certain threshold, then improve segmentation network and re-execute S5;S6, test set is preprocessed and input to trained segmentation network for segmentation, and evaluation index is calculated after segmentation is completed.The application can realize lightweight retinal real-time segmentation based on double channel structure.

Description

Technical Field

[0001] This invention relates to the field of retinal segmentation, and in particular to a lightweight real-time retinal segmentation method, system, and apparatus. Background Technology

[0002] The retina is the light-sensitive part of the eye, responsible for converting light into visual signals. Retinal blood vessels are key structures that supply the retina with nutrients and oxygen, and their function and morphology are crucial for maintaining normal visual function. However, many eye diseases, such as diabetic retinopathy (DR), retinal vein occlusion (RVO), and age-related macular degeneration (AMD), can lead to changes in the structure and function of retinal blood vessels, thereby affecting vision.

[0003] In ophthalmological diagnosis and treatment, accurate retinal vessel segmentation helps doctors quantitatively assess lesions, thereby developing more appropriate treatment plans. Therefore, accurate segmentation and quantitative analysis of retinal vessels are of great significance. Traditional retinal vessel segmentation methods are usually based on manual feature extraction, which is computationally intensive and has limited accuracy. Automated retinal segmentation technology can also significantly save doctors' time and improve diagnostic efficiency. With the rapid development of computer technology and the groundbreaking application of deep learning in computer vision, new possibilities have been brought to retinal vessel segmentation.

[0004] Convolutional neural networks (CNNs) are a commonly used method for medical image segmentation based on deep learning. Through convolution operations, neural networks can automatically learn and extract local features in images while preserving spatial information, which is crucial for image segmentation. Furthermore, the multi-layered structure of CNNs enables hierarchical representation of image features, mapping shallow low-level features such as edges and textures to deep high-level features such as shape and semantics, achieving accurate and robust image segmentation and having a profound impact on medical diagnosis and treatment.

[0005] Prior art solutions: Objectively evaluate the advantages and disadvantages of prior art solutions (disadvantages are relative to the advantages of this invention; disadvantages that this invention cannot solve need not be written), and then write the purpose of the invention based on the problems that this invention can solve. If the prior art comes from patents, journals, books, etc., please provide at least the source.

[0006] Traditional medical imaging equipment was often bulky and heavy, requiring dedicated radiology facilities. However, with technological advancements, many new medical imaging devices have become increasingly smaller and more portable. For example, portable versions of ultrasound and X-ray equipment are now available, and some can even be operated via smartphone. Furthermore, lightweight and portable medical imaging devices allow healthcare services to better meet patient needs, provide on-site care, reduce patient mobility burdens, and improve the efficiency of healthcare services. These devices can be used not only in medical facilities but also in home care, telemedicine, and even in disaster areas and war zones.

[0007] However, current neural network research mainly focuses on improving image segmentation performance. These models often have a large number of parameters and complex structures, requiring a lot of inference time and resources. They can only be used with high-performance computing devices such as GPUs, making them difficult to adapt to the lightweight, portable, and fast medical imaging tasks of the new era. Summary of the Invention

[0008] The purpose of this invention is to provide a lightweight real-time retinal segmentation method, system, and device based on a dual-channel structure, aiming to solve the problem of lightweight real-time retinal segmentation based on a dual-channel structure.

[0009] This invention provides a lightweight real-time retinal segmentation method based on a dual-channel structure, comprising:

[0010] S1. Obtain the retinal vessel dataset and divide it into training set, validation set and test set;

[0011] S2. Preprocess the training set and validation set to obtain the preprocessed dataset;

[0012] S3. Perform data augmentation on the preprocessed training and validation sets;

[0013] S4. Use the PyTorch deep learning framework to build a lightweight real-time segmentation network based on dual channels, and use the augmented training set to train the segmentation network.

[0014] S5. Evaluate the segmentation network using the augmented validation set. If the performance is greater than a certain threshold, output the trained segmentation network. Preprocess the augmented test set. If the performance is lower than a certain threshold, improve the segmentation network and repeat S5.

[0015] S6. After preprocessing the test set, input it into the trained segmentation network for segmentation. After the segmentation is completed, calculate the evaluation index.

[0016] The present invention also provides a lightweight real-time retinal segmentation system based on a dual-channel structure, comprising:

[0017] Acquisition module: used to acquire the retinal vessel dataset and divide it into training, validation, and test sets;

[0018] Preprocessing module: Used to preprocess the training and validation sets to obtain a preprocessed dataset;

[0019] Augmentation module: Used to augment the preprocessed training and validation sets;

[0020] Training module: Used to build a lightweight real-time segmentation network based on dual channels using the PyTorch deep learning framework, and to train the segmentation network using the augmented training set;

[0021] Execution module: Used to evaluate the segmentation network using the augmented validation set. If the performance is greater than a certain threshold, the trained segmentation network is output. The augmented test set is preprocessed. If the performance is lower than a certain threshold, the segmentation network is improved and the module is executed again.

[0022] The computation module is used to input the preprocessed test set into the trained segmentation network for segmentation, and to calculate the evaluation index after the segmentation is completed.

[0023] This invention also provides a lightweight real-time retinal segmentation system based on a dual-channel structure, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above method.

[0024] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described method.

[0025] Using the embodiments of the present invention, lightweight real-time retinal segmentation can be achieved.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of the residual block of the lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the differential convolution block of the lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of a multi-scale global multilayer perceptron for a lightweight real-time retinal segmentation method based on a dual-channel structure, according to an embodiment of the present invention.

[0033] Figure 6 This is a schematic diagram of differential convolution for a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0034] Figure 7 This is a semantic aggregation module diagram of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0035] Figure 8 This is a schematic diagram of a lightweight real-time retinal segmentation system based on a dual-channel structure, according to an embodiment of the present invention.

[0036] Figure 9 This is a schematic diagram of a lightweight real-time retinal segmentation device based on a dual-channel structure according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Method Implementation Examples

[0039] According to embodiments of the present invention, a lightweight real-time retinal segmentation method based on a dual-channel structure is provided. Figure 1 This is a flowchart of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention, as shown below. Figure 1 As shown, it specifically includes:

[0040] S1. Obtain the retinal vessel dataset and divide it into training set, validation set and test set;

[0041] S2. Preprocess the training set and validation set to obtain the preprocessed dataset;

[0042] S3. Perform data augmentation on the preprocessed training and validation sets;

[0043] S4. Use the PyTorch deep learning framework to build a lightweight real-time segmentation network based on dual channels, and use the augmented training set to train the segmentation network.

[0044] S5. Evaluate the segmentation network using the augmented validation set. If the performance is greater than a certain threshold, output the trained segmentation network. Preprocess the augmented test set. If the performance is lower than a certain threshold, improve the segmentation network and repeat S5.

[0045] S6. After preprocessing the test set, input it into the trained segmentation network for segmentation. After the segmentation is completed, calculate the evaluation index.

[0046] S4 specifically includes: constructing a global channel, a local channel, and a semantic aggregation module; connecting the input port of the segmentation network to the global channel and the local channel respectively; and connecting the output ports of the global channel and the local channel to the semantic aggregation module. The global channel includes: a first residual block, a first multi-scale global multilayer perceptron, a second residual block, and a second multi-scale global multilayer perceptron connected in sequence; the input port is connected to the first residual block; and the semantic aggregation module is connected to the second multi-scale global multilayer perceptron. The local channel specifically includes: a first differential convolutional block, a second differential convolutional block, a third differential convolutional block, and a fourth differential convolutional block connected in sequence; the first differential convolutional block is connected to the input port; and the fourth differential convolutional block is connected to the semantic aggregation module. The first and second multi-scale global multilayer perceptrons are used to extract global information from the feature map, and the four differential convolutional blocks are used to obtain local contextual information. The semantic aggregation module integrates the global information and the local contextual information to achieve multi-level feature representation.

[0047] The multi-scale global multilayer perceptron consists of layer normalization, spatial linear projection, position coding, residual connectivity, and a conventional multilayer perceptron. The implementation methods of spatial linear projection and position coding are shown in the formulas:

[0048] F spa1 =Linear w-w-w (Linear h-h-h (F)) (5)

[0049] F spa2 =Linear w-2w-w (Linear h-2h-h (F)) (6)

[0050] F pos =Dwconv 5×5 (F) (7)

[0051] Among them, Dwconv 5×5 This represents a 5×5 depthwise separable convolution. The h and w in the Linear subscripts indicate that the feature map is linearly projected in the height and width directions using a fully connected layer, respectively. The 2h / 2w in the middle means that when using a fully connected layer, the size is first enlarged by a factor of 2 and then restored.

[0052] Training the segmentation network using the expanded training set specifically includes: training the segmentation network using the expanded training set, with the loss function used for training being...

[0053] Where M is the total number of pixels in the image, l i r represents the foreground prediction probability of input pixel i. i is the true label for pixel i, which is either vascular or non-vascular in the retinal vessel segmentation task.

[0054] The specific implementation method is as follows:

[0055] Step 1: Collect and organize the retinal blood vessel dataset, dividing it into training, validation, and test sets.

[0056] Step 2: Preprocessing the dataset can reduce noise and improve the contrast between blood vessel pixels and background pixels, thereby improving the model's accuracy in retinal vessel segmentation. First, the dataset images are converted to grayscale using the following formula.

[0057] Gray(x,y)=R(x,y)*0.299+G(x,y)*0.587+B(x,y)*0.114 (1)

[0058] Where R, G, and B represent the brightness values ​​of the red, green, and blue channels in the RGB image, respectively.

[0059] The image is then standardized by scaling the overall grayscale value to prevent extreme data conditions and address uneven illumination in fundus images. The standardization formula is shown in Formula 2.

[0060]

[0061] Among them l ij The pixel values ​​of the image. The image pixel values ​​are standardized, μ is the mean of the image pixel values, and δ is the standard deviation of the image pixel values. The formulas for calculating μ and δ are shown in Equations 3 and 4.

[0062]

[0063]

[0064] Where W is the number of rows of image pixels and H is the number of columns of image pixels.

[0065] The image is then subjected to adaptive histogram equalization, which stretches the grayscale histogram of the original image to a certain extent, thereby improving the image contrast and limiting the extremely dark and bright parts of the image.

[0066] Finally, the fundus image data was normalized, reducing the gray levels from 0 to 255 to between 0 and 1 to facilitate subsequent model calculations on the image data.

[0067] Step 3: After image preprocessing, due to the relatively small amount of data in the fundus image dataset, data augmentation processing is required for the training and validation sets to increase the amount of data in both sets, especially the training set, as sufficient training data is necessary to train the retinal vessel segmentation model. This invention uses four data augmentation processes: (1) random cropping, randomly cropping the image into 64×64 image blocks; (2) random horizontal flipping; (3) random vertical flipping; and (4) random rotation. After data augmentation, a large number of images are obtained for training and validation of the segmentation model.

[0068] Figure 2 This is a schematic diagram of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention.

[0069] Step 4: Use the PyTorch deep learning framework to build a lightweight real-time segmentation network based on dual channels. The structure of this network model is as follows: Figure 2 As shown, the overall structure is a dual-channel structure, consisting of a global channel at the top, a local channel at the bottom, and a semantic aggregation module on the right. The global channel consists of two residual blocks and two multi-scale global multilayer perceptrons, while the local channel consists of four differential convolutional blocks.

[0070] Figure 3 This is a schematic diagram of the residual block of the lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0071] The residual block is composed of the following:

[0072] The entry point is connected to the first 3×3 convolution and the first 1×1 convolution. The first 3×3 convolution is connected to the first batch of normalized layers. The first batch of normalized layers is followed by the first ReLU function. The first ReLU function is connected to the second 3×3 convolution, the second batch of normalized layers and the second ReLU function. The second ReLU function is connected to the 1×1 convolution at the entry point and then output.

[0073] Figure 4 This is a schematic diagram of the differential convolution block of the lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0074] The differential convolution block is composed as follows: at the entrance is a 3×3 differential convolution, followed by the third batch of normalized layers, the third ReLU function, the third 3×3 convolution, the fourth batch of normalized layers, and the fourth ReLU function.

[0075] The model employs max pooling layers as the downsampling method to construct a hierarchical structure, acquiring multi-level semantic feature maps required for image segmentation. The global channel is equipped with two multi-scale global multilayer perceptrons to extract global information and long-range dependencies from the feature maps; differential convolutions in the local channels provide rich contextual information, enabling the model to better understand the background and semantic relationships surrounding the target object. The semantic aggregation module integrates global contextual semantic information and local detail information to achieve multi-level feature representation.

[0076] Step 5: This invention designs a multi-scale global multilayer perceptron for global feature extraction. Ordinary convolutions, due to their small kernel size, can only extract local details. Mainstream network models often require stacking multiple convolutional layers to obtain global contextual semantics. However, as the number of convolutional layers increases, the number of network parameters and computational complexity gradually increase, potentially making training and inference more time-consuming and requiring more computational resources. The multi-scale global multilayer perceptron can directly extract global information and long-range dependencies from feature maps, simplifying the network structure. The multi-scale global multilayer perceptron consists of layer normalization, spatial linear projection, positional encoding, residual connections, and an ordinary multilayer perceptron. The implementation methods of spatial linear projection and positional encoding are shown in the formula:

[0077] F spa1 =Linear w-w-w (Linear h-h-h (F)) (5)

[0078] F spa2 =Linear w-2w-w (Linear h-2h-h (F)) (6)

[0079] F pos =Dwconv 5×5(F) (7)

[0080] Among them, Dwconv 5×5 This represents a 5×5 depthwise separable convolution. The h and w in the Linear subscripts indicate that the feature map is linearly projected in the height and width directions using a fully connected layer, respectively. The 2h / 2w in the middle means that when using a fully connected layer, the size is first enlarged by a factor of 2 and then restored.

[0081] Figure 5 This is a schematic diagram of a multi-scale global multilayer perceptron for a lightweight real-time retinal segmentation method based on a dual-channel structure, according to an embodiment of the present invention.

[0082] After the feature map is input, it is normalized in the first layer. Then, the normalized feature map is passed through two linear projection and relative position encoding modules with different multiples along different spatial directions. It should be noted that this is done in parallel rather than serially. Finally, these three feature maps are added and fused with the feature map after the first layer normalization, and then passed through the second layer normalization and a multilayer perceptron.

[0083] Linear projection along different spatial directions enables multi-scale global multilayer perceptrons to capture long-distance dependencies and global information, and fully consider the interaction between spatial dimensions. However, this ignores positional features to some extent, which may cause potential loss of positional information. Therefore, we added positional encoding, which is accomplished by 5×5 depthwise separable convolution. Only a few parameters are added to ensure the integrity of relative positional information, thereby improving the feature representation ability of the model. Residual connections can avoid degradation during model training.

[0084] Step 6: Local information belongs to low-level semantics, but it plays a significant role in the model's learning of local details and basic features such as edges and textures in an image. Local channels capture fine blood vessel edge and texture information through a combination of differential convolution and convolution. Compared with ordinary convolution, differential convolution can capture richer image gradient and low-level information, facilitating robust and accurate edge detection.

[0085] Figure 6 This is a schematic diagram of differential convolution for a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0086] Here we use central difference convolution, which can be expressed as the following formula:

[0087] y=w1×(x1-x2)+w2×(x2-x3)+w3×(x3-x6)+… (8)

[0088] Where w i x represents the convolution kernel weights. i Represents pixel value.

[0089] In addition, we have provided more channels for local channels to encode richer details.

[0090] Step 7: There are generally two methods for merging feature elements: addition and concatenation. However, the outputs of the two channels have different levels of feature representation. The detail channel is used for low-level semantics, while the semantic channel is used for high-level semantics. Therefore, simple addition or concatenation ignores the diversity and hierarchical inconsistency of these two types of information. Figure 7 This is a semantic aggregation module diagram of a lightweight real-time retinal segmentation method based on a dual-channel structure according to an embodiment of the present invention;

[0091] The semantic aggregation module has a first entry point consisting of a fourth 3×3 convolution, a fifth batch normalization, and a first activation function connected in sequence. The second entry point consists of a second 1×1 convolution, a first upsampling, and a sigmoid function connected in sequence. The output of the sigmoid function is multiplied by the output of the first activation function and then added to the first entry point to obtain the first aggregation information. The second entry point is multiplied by the output of the sigmoid function after the second upsampling to obtain the second aggregation information. The first and second aggregation information are then added together and input to a first 2×2 transposed convolution, a second 2×2 device convolution, a sixth batch normalization, a second activation function, a third 1×1 convolution, and a softmax function connected in sequence.

[0092] We feed the local information feature map obtained from the local channels into the lower part of the semantic guidance aggregation module. A 1×1 convolution and upsampling are used to restore the image size to match the global channel image size. Then, a Sigmoid activation function is applied to generate a local attention weight map. The initial input feature map size is adjusted to match the attention feature map, and the corresponding values ​​are multiplied to obtain the final local information feature map. The global information feature map obtained from the global channels is fed into the upper part of the semantic guidance aggregation module. After a 3×3 convolution, batch normalization, and GELU activation function, it is multiplied with the local attention weight map and added to the initial global information feature map to obtain the final global information feature map. After adding the global and local information feature maps, two 2×2 transposed convolutions are used to adjust the height and width of the feature map to match the input feature map size. Figure 1 After batch normalization, GELU activation function, and 1×1, the result is fed into the Softmax function and the final feature map is output.

[0093] We propose a semantic aggregation module to fuse complementary information from two channels, such as... Figure 7As shown, since directly extracting global information may lose some details and the accuracy of local features, the semantic aggregation module uses detailed information from local channels to guide the feature response of global channels. Through guidance at different scales, we can capture feature representations at different levels. Compared with simple combination, this guidance method makes the semantic fusion between the two channels more natural. At the same time, the initial size of the feature map restored by transposed convolution is used to output the feature map through 1×1 convolution and the Softmax function.

[0094] Step 8: Train the model using images from the training set. The loss function used during training is the cross-entropy loss function, as shown in Formula 9.

[0095]

[0096] Where M is the total number of pixels in the image, l i r represents the foreground prediction probability of input pixel i. i is the true label for pixel i, which is 1 (vessel) or 0 (non-vessel) in the retinal vessel segmentation task.

[0097] The trained model can be validated using validation set images. Based on the current model's segmentation performance on the validation set, the model's hyperparameters are tuned. Through multiple adjustments to the hyperparameters, the model can meet the requirements of real-time performance and lightweight operation while achieving optimal segmentation performance, thus completing the model training.

[0098] Step 9: After completing the training of the retinal vessel segmentation model, use test set images to perform a final performance test on the segmentation model. Evaluate the network model according to the evaluation metrics. The segmentation performance evaluation metrics that need to be calculated are: accuracy (ACC) and F1 score (F1-score). The calculation formulas for the metrics can be found in Formula 9. The evaluation metrics for model size and inference speed are the number of model parameters (Params) and the number of frames per second (FPS), respectively.

[0099]

[0100]

[0101]

[0102]

[0103] In the formula, TP represents a true positive, meaning the actual target is a positive sample, and the algorithm classifies it as a positive sample. TN represents a true negative, meaning the actual sample is a negative sample, and the algorithm classifies it as a negative sample. FP represents a false positive, meaning the actual sample is negative, but the algorithm incorrectly classifies it as a positive sample. FN represents a false negative, meaning the actual sample is positive, but the algorithm incorrectly classifies it as a negative sample. Accuracy represents the percentage of correctly classified pixels out of the total number of pixels in the image, reflecting the overall performance of the algorithm. F1-Score is a commonly used metric for evaluating the performance of binary classification models, combining the model's precision and recall to measure the model's balanced performance on positive and negative samples. All the above metrics range from 0 to 1; the higher the value, the better the segmentation performance of the algorithm. The number of model parameters refers to the number of parameters in the deep learning model. The number of parameters that need to be learned in a model is optimized and updated through the training process so that the model can adapt to the input data and produce the expected output. It can be used to measure the model's complexity, capacity, and the model's requirements for storage and computing resources. Models with fewer parameters have many advantages in terms of storage, computing, and deployment. This metric is implemented by the `thop.profile()` function in PyTorch. FPS is used to evaluate the model's inference speed. It represents the number of image frames that the model can process per unit of time. It is an important metric for measuring real-time performance. The higher the FPS value, the faster the model processes and the shorter the inference process can be completed. We uniformly measured this metric in a hardware environment with an RTX 3080 (10GB) GPU and a 12vCPU Intel(R) Xeon(R) Platinum 8255C CPU@2.50GHz.

[0104] This invention addresses the task of retinal vessel segmentation, aiming to improve segmentation speed while maintaining segmentation accuracy. Current mainstream vessel segmentation networks mostly focus on improving segmentation results, neglecting factors such as model parameter count and inference speed. This leads to medical imaging tasks relying on expensive and bulky large computing devices, making them unsuitable for mobile medical services or nursing care tasks. To address this issue, the retinal vessel segmentation method of this invention makes three innovations, as follows:

[0105] (1) A lightweight real-time segmentation network model for the retina was designed based on the dual-channel structure, which can ensure the segmentation effect of the network under the premise of few parameters and fast inference speed.

[0106] (2) A multi-scale global multilayer perceptron was designed to extract global information and long-range features, solving the problem that traditional convolutional neural networks require stacking convolutional layers to obtain global information. The structure of the multi-scale global multilayer perceptron is as follows: Figure 3 As shown, compared to ordinary convolutional modules, the multi-scale global multilayer perceptron designed in this invention can simultaneously extract spatial global correlations while retaining relative positional information, resulting in stronger feature extraction capabilities and thus improving the model's segmentation performance and generalization ability. Residual connections can avoid network degradation during model training, ensuring the positive impact of modules on model performance during training and accelerating model convergence.

[0107] (3) This invention designs a semantic aggregation module that uses multiple max pooling operations on the input image to construct an image pyramid, and splices and fuses input images and feature maps at different scales to reduce the loss of semantic information caused by downsampling, so that the model can extract richer multi-scale semantic information and improve the model segmentation effect.

[0108] System Implementation Examples

[0109] According to embodiments of the present invention, a lightweight real-time retinal segmentation system based on a dual-channel structure is provided. Figure 8 This is a schematic diagram of a lightweight real-time retinal segmentation system based on a dual-channel structure, as described in an embodiment of the present invention. Figure 8 As shown, it specifically includes:

[0110] Acquisition module: used to acquire the retinal vessel dataset and divide it into training, validation, and test sets;

[0111] Preprocessing module: Used to preprocess the training and validation sets to obtain a preprocessed dataset;

[0112] Augmentation module: Used to augment the preprocessed training and validation sets;

[0113] Training module: Used to build a lightweight real-time segmentation network based on dual channels using the PyTorch deep learning framework, and to train the segmentation network using the augmented training set;

[0114] Execution module: Used to evaluate the segmentation network using the augmented validation set. If the performance is greater than a certain threshold, the trained segmentation network is output. The augmented test set is preprocessed. If the performance is lower than a certain threshold, the segmentation network is improved and the module is executed again.

[0115] The computation module is used to input the preprocessed test set into the trained segmentation network for segmentation, and to calculate the evaluation index after the segmentation is completed.

[0116] The training module is specifically used to: build a global channel, a local channel, and a semantic aggregation module. The input port of the segmentation network is connected to the global channel and the local channel respectively, and the output ports of the global channel and the local channel are connected to the semantic aggregation module. The global channel includes: a first residual block, a first multi-scale global multilayer perceptron, a second residual block, and a second multi-scale global multilayer perceptron connected in sequence. The input port is connected to the first residual block, and the semantic aggregation module is connected to the second multi-scale global multilayer perceptron. The local channel specifically includes: a first differential convolutional block, a second differential convolutional block, a third differential convolutional block, and a fourth differential convolutional block connected in sequence. The first differential convolutional block is connected to the input port, and the fourth differential convolutional block is connected to the semantic aggregation module. The first multi-scale global multilayer perceptron and the second multi-scale global multilayer perceptron are used to extract global information from the feature map, and the four differential convolutional blocks are used to obtain local contextual information. The semantic aggregation module integrates the global information and the local contextual information to achieve multi-level feature representation.

[0117] The multi-scale global multilayer perceptron consists of layer normalization, spatial linear projection, position coding, residual connectivity, and a conventional multilayer perceptron. The implementation methods of spatial linear projection and position coding are shown in the formulas:

[0118] F spa1 =Linear w-w-w (Linear h-h-h (F)) (5)

[0119] F spa2 =Linear w-2w-w (Linear h-2h-h (F)) (6)

[0120] F pos =Dwconv 5×5 (F) (7)

[0121] Among them, Dwconv 5×5 This represents a 5×5 depthwise separable convolution. The `h` and `w` in the linear subscripts indicate linear projection of the feature map along its height and width directions using a fully connected layer, respectively. The `2h / 2w` indicates that the size is first magnified by a factor of 2 before restoring it to its original size when using a fully connected layer. Training the segmentation network using the augmented training set specifically includes: training the segmentation network using the augmented training set, with the loss function used for training being...

[0122] Where M is the total number of pixels in the image, l i r represents the foreground prediction probability of input pixel i. i is the true label for pixel i, which is either vascular or non-vascular in the retinal vessel segmentation task.

[0123] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0124] Device Example 1

[0125] This invention provides a schematic diagram of a lightweight real-time retinal segmentation system based on a dual-channel structure, as shown in the embodiment of the invention. Figure 8 As shown, it includes: a memory 80, a processor 82, and a computer program stored on the memory 80 and executable on the processor 82. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0126] Device Example 2

[0127] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 82, implements the steps described in the above method embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions to the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of the present solution.

Claims

1. A lightweight real-time retinal segmentation method based on a two-channel structure, characterized in that, The method comprises the following steps: S1, acquiring a retinal blood vessel data set and dividing it into a training set, a validation set and a test set; S2, preprocessing the training set and the validation set to obtain a preprocessed data set; S3, performing data augmentation on the preprocessed training set and the validation set; S4, using a Pytorch deep learning framework to build a lightweight real-time segmentation network based on a double channel, training the segmentation network using the augmented training set, and specifically comprising: building a global channel, a local channel and a semantic aggregation module, the input port of the segmentation network being connected with the global channel and the local channel respectively, and the output ports of the global channel and the local channel being connected with the semantic aggregation module, wherein the global channel comprises: a first residual block, a first multi-scale global multi-layer perception, a second residual block and a second multi-scale global multi-layer perception connected in sequence, the input port being connected with the first residual block, and the semantic aggregation module being connected with the second multi-scale global multi-layer perception; the local channel specifically comprises: a first difference convolution block, a second difference convolution block, a third difference convolution block and a fourth difference convolution block connected in sequence, the first difference convolution block, the second difference convolution block, the third difference convolution block and the fourth difference convolution block having the same structure, the first difference convolution block having a 3*3 difference convolution at the input port, and being sequentially connected with a third batch normalization layer, a third ReLU function, a third 3*3 convolution, a fourth batch normalization layer and a fourth ReLU function; the first difference convolution block is connected with the input port, and the fourth difference convolution block is connected with the semantic aggregation module; the first multi-scale global multi-layer perception and the second multi-scale global multi-layer perception are used to extract global information of a feature map; the semantic aggregation module has a fourth 3*3 convolution, a fifth batch normalization and a first activation function connected in sequence at the first input port, and has a second 1*1 convolution, a first upsampling and a Sigmoid function connected in sequence at the second input port; the Sigmoid function output is multiplied by the first activation function output, and then added to the first input port to obtain first aggregation information; the second input port is multiplied by the Sigmoid function output after being subjected to a second upsampling to obtain second aggregation information; the first aggregation information and the second aggregation information are added and input into a first 2*2 transpose convolution, a second 2*2 device convolution, a sixth batch normalization, a second activation function, a third 1*1 convolution and a Softmax function connected in sequence; the four difference convolution blocks are used to obtain local context information; the semantic aggregation module integrates the global information and the local context information, and realizes multi-level feature expression; The multi-scale global multi-layer perception is composed of layer normalization, spatial linear projection, position encoding, residual connection and ordinary multi-layer perception; S5, evaluating the segmentation network using the augmented validation set, outputting the trained segmentation network if the performance is greater than a certain threshold, preprocessing the augmented test set, and improving the segmentation network and then re-executing S5 if the performance is less than a certain threshold; S6, preprocessing the test set and inputting it into the trained segmentation network for segmentation, and calculating evaluation indexes after the segmentation is completed.

2. The method of claim 1, wherein, The implementation modes of the spatial linear projection and the position encoding are shown in the formula: wherein, represents a depthwise separable convolution of 5x5, h and w in the subscript respectively represent linear projection of the height and width direction of the feature map by a fully connected layer, and 2h / 2w in the middle represents that when a fully connected layer is used, the size is first expanded by 2 times and then restored. wherein, represents a depthwise separable convolution of 5x5, respectively represent linear projection in height and width direction of the feature map with a fully connected layer.

3. A lightweight real-time retinal segmentation system based on a dual-channel structure, characterized in that, The method comprises the following steps: The acquisition module is configured to acquire a retinal blood vessel dataset and divide the retinal blood vessel dataset into a training set, a validation set and a test set; The preprocessing module is configured to preprocess the training set and the validation set to obtain a preprocessed dataset; The expansion module is configured to expand the preprocessed training set and the preprocessed validation set; The training module is configured to use a Pytorch deep learning framework to build a lightweight real-time segmentation network based on a double-channel, and train the segmentation network using the expanded training set, and specifically configured to: build a global channel, a local channel and a semantic aggregation module, an input port of the segmentation network is connected with the global channel and the local channel, and an output port of the global channel and the local channel is connected with the semantic aggregation module, the global channel comprises: a first residual block, a first multi-scale global multi-layer perception, a second residual block and a second multi-scale global multi-layer perception connected in sequence, an inlet is connected with the first residual block, and the semantic aggregation module is connected with the second multi-scale global multi-layer perception, the local channel specifically comprises: a first difference convolution block, a second difference convolution block, a third difference convolution block and a fourth difference convolution block connected in sequence; the first difference convolution block, the second difference convolution block, the third difference convolution block and the fourth difference convolution block are of the same structure, the first difference convolution block has a 3*3 difference convolution at an inlet, and is sequentially connected with a third batch normalization layer, a third ReLU function, a third 3*3 convolution, a fourth batch normalization layer and a fourth ReLU function; the first difference convolution block is connected with the inlet, the fourth difference convolution block is connected with the semantic aggregation module, the first multi-scale global multi-layer perception and the second multi-scale global multi-layer perception are used to extract global information of a feature map, the semantic aggregation module has a fourth 3*3 convolution, a fifth batch normalization and a first activation function connected in sequence at a first inlet, has a second 1*1 convolution, a first upsampling and a Sigmoid function connected in sequence at a second inlet, and has a first aggregation information obtained by multiplying an outlet of the Sigmoid function with an outlet of the first activation function and then adding the first aggregation information to the first inlet, has a second aggregation information obtained by multiplying an outlet of a second upsampling with an outlet of the Sigmoid function, and has the first aggregation information and the second aggregation information added and input to a first 2*2 transposed convolution, a second 2*2 device convolution, a sixth batch normalization, a second activation function, a third 1*1 convolution and a Softmax function connected in sequence, the four difference convolution blocks are used to obtain local context information, the semantic aggregation module integrates the global information and the local context information, and realizes multi-level feature expression; The multi-scale global multi-layer perception comprises layer normalization, spatial linear projection, position encoding, residual connection and an ordinary multi-layer perception; The execution module is configured to evaluate the segmentation network using the expanded validation set, output the trained segmentation network if a performance is greater than a certain threshold, preprocess the expanded test set, and improve the segmentation network and then execute the module again if the performance is less than a certain threshold; The calculation module is configured to input the test set after preprocessing to the trained segmentation network for segmentation, and calculate evaluation indexes after the segmentation is completed.

4. The system of claim 3, wherein, The implementation manners of the spatial linear projection and the position encoding are shown in the following formula: wherein, represents a depth separable convolution of 5x5, The h and w in the subscript respectively represent linear projection of the height and width direction of the feature map by the full connection layer, and the 2h / 2w in the middle represents that when the full connection layer is used, the size is first expanded by 2 times and then restored.

5. A lightweight real-time retinal segmentation device based on a dual-channel structure, characterized in that, The implementation manners of the spatial linear projection and the position encoding are shown in the following formula: Memory, a processor, and a computer program stored on the memory and capable of running on the processor, which, when executed by the processor, implements the steps of the method for real-time segmentation of the retina based on a two-channel structure according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an information transmission implementation program, and the program, when executed by a processor, implements the steps of the method for real-time segmentation of the retina based on a two-channel structure according to any one of claims 1 to 2.

Images