A method for automatically segmenting typical lesions of retinopathy of prematurity

Abstract

The application discloses a kind of premature infant retinopathy typical lesion automatic segmentation method, comprising the following steps: S1, collect ROP image, exclude unqualified image therein, and after the qualified ROP image is labeled, it is used as ROP lesion segmentation dataset, and dataset is divided into training set, verification set and test set;S2, construct deep neural network model;S3, after the training set is preprocessed, the training set is used to train the deep neural network model, calculate loss, optimize network parameters, and obtain the optimal model after several iterations;S4, the optimal model is tested using test set, and the segmentation performance evaluation index of the model on test set is obtained.Compared with the prior art, the method can better distinguish between background and four lesion structures, reduce the misjudgment rate, achieve automatic pixel-level segmentation of ROP fundus image lesions, and ultimately improve the network segmentation performance.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer-aided medical technology, and more specifically, to a method for automatically segmenting typical lesions of retinopathy of prematurity. Background Technology

[0002] Retinopathy of prematurity (ROP) is a potentially blinding retinal disease caused by fibrovascular proliferation in the immature retina of premature or low birth weight infants. It is considered one of the leading preventable causes of childhood blindness. ROP accounts for approximately 19% of childhood blindness worldwide and is most common in premature infants born before 32 weeks of gestation and with a birth weight of less than 1500 grams. With the increasing number of high-risk children globally, ROP screening for high-risk infants has become particularly important.

[0003] In clinical practice, ophthalmologists typically diagnose Retinopathy of Premature Opticness (ROP) by observing the presence of lines, ridges, fibers, partial retinal detachment, and complete retinal detachment in fundus images. Stage 1 is the initial stage, characterized by peripheral vascular occlusion, forming a thin, flat white demarcation line between the vascularized and avascular areas of the retina. In Stage 2, the white demarcation line widens and rises, forming ridges that protrude above the retinal surface. In Stage 3, further fibrosis on the ridge manifests as bulging and widening, with neovascularization occurring within or on the ridge, giving it a pinkish appearance. Stage 4 involves partial retinal detachment, ranging from localized to peripheral retinal detachment. Stage 5 is the terminal stage, characterized by complete retinal detachment with widespread fibrosis, potentially leading to permanent blindness.

[0004] Currently, the screening and diagnosis of Retinopathy of Prematurity (ROP) are mainly performed by relatively experienced ophthalmologists. However, due to limited and unevenly distributed expert resources, unstable subjective judgments of ROP staging results, and a certain percentage of misdiagnosis, which affects standardized diagnosis and treatment, artificial intelligence-assisted diagnosis of ROP is emerging. Automatic segmentation of retinal structures in fundus images, including the optic disc, retinal vessels, and retinal lesions at different stages, is crucial for describing the characteristics of retinopathy of prematurity and can significantly improve the consistency of ROP diagnosis by ophthalmologists. Among these, the segmentation of ROP lesions is very important for identifying the ROP stage. If the lesion segmentation is clear, the thickness, color, or other properties of the lesion can help determine the ROP stage.

[0005] Currently, some studies use clustering methods to identify retinal boundaries and ridges, while others use image processing techniques to segment lesions into stages 1, 2, and 3. However, traditional machine learning algorithms use manually crafted features for segmentation, which is slow. Previous studies using deep learning methods to segment ROP fundus images have focused primarily on vessel segmentation, with fewer studies on ROP staged lesion segmentation. Existing studies either use U-Net or its architecture for ROP retinal vessel segmentation, or incorporate SE blocks and attention gates (AG) into U-Net for segmenting boundaries, ridges, and vessels in the main ROP image dataset. These studies all treat lesions in the first three stages as a single category for segmentation, and none include stage 4 lesions. According to a literature review, there are currently no studies on ROP lesion segmentation at different stages using deep learning.

[0006] In summary, current research on ROP lesion segmentation is still immature. Existing traditional methods are slow and time-consuming, while deep neural network-based methods cannot achieve efficient and accurate automatic ROP lesion segmentation because the networks are not suitable for ROP fundus images. Lesion category information is a crucial basis for ophthalmologists' clinical diagnosis and staging of ROP, but existing ROP lesion segmentation methods, which mainly rely on binary segmentation (lesion and background), cannot provide the lesion category information needed for clinical diagnosis. Currently, there has been little progress in deep learning methods for automatic lesion segmentation using ROP fundus images.

[0007] Existing technology discloses a training method and apparatus for a lesion segmentation model, and a lesion segmentation method and apparatus. This method inputs a 2D CT image containing the target lesion into a pre-trained 2D lesion segmentation model, outputting a 2D lesion segmentation result. Based on the location and size information of the target lesion contained in the 2D lesion segmentation result, the voxel unit where the target lesion is located in the 3D CT image is labeled, obtaining a target 3D CT image with target labels. The target 3D CT image is input into the 3D lesion segmentation model, using the voxel unit where the target label is located as the region of interest, outputting a 3D segmentation prediction result for the target lesion. Based on the segmentation loss between the 3D segmentation prediction result and the target label, the 3D lesion segmentation model is trained until the 3D lesion segmentation model converges. The drawback of this method is that it cannot effectively distinguish between background and lesion structures, nor can it effectively distinguish between foreground and background.

[0008] Therefore, in light of the above requirements and the shortcomings of existing technologies, this application proposes an automatic segmentation method for typical lesions of retinopathy of prematurity. Summary of the Invention

[0009] This invention provides an automatic segmentation method for typical lesions of retinopathy of prematurity, which can better distinguish the background and four types of lesion structures, reduce the misjudgment rate, achieve automatic pixel-level segmentation of lesions in fundus images of retinopathy of prematurity, and ultimately improve the network segmentation performance.

[0010] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows:

[0011] The first aspect of this invention provides an automatic segmentation method for typical lesions of retinopathy of prematurity, the method comprising the following steps:

[0012] S1. Collect ROP images, filter out unqualified images, label the qualified ROP images and use them as ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set.

[0013] S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module.

[0014] S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model using backpropagation technology based on domain knowledge, validate the current model using the validation set, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss function to obtain the optimal model.

[0015] S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

[0016] Further, step S1 specifically involves: collecting ROP images, filtering out images where ROP lesions cannot be identified due to objective conditions, labeling the test set images with pixel-level tags, and using the resulting mask image as the ROP lesion segmentation dataset. The dataset is then divided into a training set, a validation set, and a test set. Specifically, labeling the test set images with pixel-level tags involves using different colored outlines to enclose the four structures in the ROP image fundus image: the boundary line, ridge, fiber, and retinal subdetachment.

[0017] Among them, the images in which ROP lesions cannot be identified due to objective conditions include: images in which ROP lesions cannot be identified due to excessively strong or dim lighting, and images in which multiple doctors cannot identify ROP lesions.

[0018] Furthermore, in step S2, the deep neural network model includes a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. The ROP image is used as input to the deep neural network model, and the model outputs a ROP lesion segmentation map Y′, auxiliary ROP lesion segmentation maps Y1′, Y2′, and Y3′, and a feature map Y. f The distance vector between the image-level sequential encoding vector V′ and the image-level sequential encoding vector V′ is as follows: The ROP image dataset, after data augmentation, is used as input to the backbone segmentation network. The backbone segmentation network outputs high-level features F, which are then used as input to the auxiliary segmentation module. The auxiliary segmentation module outputs features F′ and feature map Y′ respectively. f The feature F′ is added to the feature F and then input into the backbone segmentation network for further segmentation. The role of the image-level sequential encoding vector learning module is to obtain the sequential encoding vector of the model output, calculate the loss with it and the real sequential encoding, and optimize the parameters of the backbone segmentation network through backpropagation.

[0019] The preprocessing step involves data augmentation, specifically: first, the image is randomly scaled; then, it is normalized; and finally, a fixed-size image is randomly cropped using a resampling method. A random number is set, and if the sampled image contains only the background and the random number is greater than 0.2, resampling is performed until the sampled image contains any type of lesion. The image label processing follows the same procedure: it is first scaled to the same scaling ratio as the image, and then cropped to the same position and size as the final sampled image. Finally, the pixel-level labels are processed to obtain image block-level labels and image-level sequential encoding vector labels.

[0020] Among them, data augmentation can effectively reduce the interference of background information.

[0021] Furthermore, the backbone segmentation network includes an encoder and a decoder. The encoder is an STDC network composed of multiple convolutional layers for feature extraction. The decoder uses bilinear interpolation to restore the high-level feature map to the resolution of the original image. The auxiliary segmentation module includes a spatial attention module, a channel attention module, convolutional layers, and a sigmoid function.

[0022] The backbone segmentation network consists of an encoder and a decoder. The encoder performs feature extraction, while the decoder's structure is built using the fusion connection mechanism of the STDC network to minimize spatial information loss, restore high-level feature maps to the original image resolution, and incorporates a block-level weighted balancing module. The segmentation network assigns a lesion class label to each pixel, ultimately obtaining a pixel-level prediction result Y′. During the training phase, it also outputs other results Y′1, Y′2, Y′3, and Y′... fAnd V′. The loss is calculated using the predicted results Y′, Y′1, Y′2, and Y′3, which are the true label Y and Y′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0023] Among them, the block-level weighted balancing module utilizes pixel-level semantic information to enhance the high-level feature representation of the smaller category and solve the class imbalance problem; the image-level sequential encoding vector learning module based on domain knowledge improves the network's ability to distinguish the background and four ROP lesion structures, strengthens the learning of five structural features, and solves the problem of high inter-class similarity.

[0024] Furthermore, in one round of training during the training phase, a ROP image I∈R is used. H×W×3 As input to the model, the outputs are ROP lesion segmentation map Y′, ROP lesion auxiliary segmentation maps Y′1, Y′2, Y′3, and feature map Y′. f The distance vector between the image-level sequential encoding vector V′ and the ground truth pixel-level label Y is processed to obtain the image block-level label Y. f The image-level sequential encoded vector label V is used, and the output obtained through the network is compared with the true label. The loss function is calculated, and the SGD optimizer is used to update and calculate the network parameters that affect the model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function. Among them, the loss is calculated with the predicted results Y′, Y′1, Y′2, Y′3 based on the true label Y and Y′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0025] Furthermore, the value of the c-th channel, i-th row, j-th column of the image block-level label is obtained. The specific mathematical expression is as follows:

[0026]

[0027]

[0028] Where H and W represent the height and width of the input image, and h and w represent the height and width of the output feature map. To round up, i = 0, 1, ..., h-1, j = 0, 1, ..., w-1. When Y contains a lesion of type c, T(Y, c) = 1, otherwise T(Y, c) = 0. c is the lesion category index, c = 0, 1, ..., C-1.

[0029] The mathematical expression for obtaining the sequential encoding vector V is as follows:

[0030] Q=U(Y,c) (8)

[0031] K=[[0,0,0,0],[1,0,0,0],[1,1,0,0],[1,1,1,0],[1,1,1,1]] (9)

[0032] V=Q×K (10)

[0033] Where Y is the pixel-level segmentation label of the image, c is the lesion category index, and the function U(Y,c) generates a 1*c vector Q, where each element of Q is either 0 or 1. Q[i] = 1 indicates the presence of the i-th type of lesion in Y, and Q[i] = 0 indicates the absence of the i-th type of lesion in Y, where i ∈ [0,c). K is the lesion sequence code. V is the image-level sequence code vector, representing the sum of the sequence codes for the lesion categories present in Y.

[0034] Furthermore, the loss function includes: backbone segmentation loss L seg Auxiliary segmentation loss auL seg1 ,auL seg2 and auL seg3 Image block-level loss L feat and the Euclidean distance loss L of the sequential encoding vector w L seg ,auL seg1 ,auL seg2 and auL seg3 It consists of ohem loss and dice loss; the mathematical expression of the total loss function is as follows:

[0035] L=αL seg +βauL seg1 +γauL seg2 +δauL seg3 +εL feat +θL w (1)

[0036] L seg =L ohem +L dice (2)

[0037] L seg1 =L ohem1 +L dice1 (3)

[0038] L seg2 =L ohem2 +L dice2 (4)

[0039] L seg3 =L ohem3 +L dice3 (5)

[0040] Among them, α=1.0, β=0.4, γ=0.4, δ=0.7, ε=1.0, and θ=1.0.

[0041] Furthermore, the image block-level loss L feat The formula is as follows:

[0042]

[0043] Among them, w i For the loss weights of category i, L bce The cross-entropy loss is used for binary classification.

[0044] Calculate the loss L of the image-level sequential encoded vectors of the true labels and the predicted results. w The formula is as follows:

[0045]

[0046] Where l is the sequential encoding length.

[0047] Furthermore, the validation process of the current model using the validation set is as follows: input all images in the validation set one by one into the currently trained model to obtain the segmentation results, calculate the evaluation metrics of the segmentation results and the true labels, calculate the AUC_PR of the current model on the validation set, and if the current AUC_PR is the highest, then save the current model as the current best model.

[0048] A second aspect of the present invention provides an automatic segmentation system for typical lesions of retinopathy of prematurity (ROP) in infants, comprising a memory and a processor. The memory includes an automatic segmentation program for typical lesions of ROP in infants, which, when executed by the processor, performs the following steps:

[0049] S1. Collect ROP images, filter out unqualified images, label the qualified ROP images and use them as ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set.

[0050] S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module.

[0051] S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model using backpropagation technology based on domain knowledge, validate the current model using the validation set, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss function to obtain the optimal model.

[0052] S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

[0053] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0054] This invention provides an automatic segmentation method for typical lesions of retinopathy of prematurity (ROP). By filtering out unqualified ROP images and performing annotation and preprocessing, the interference of background information is reduced. The constructed deep neural network model utilizes pixel-level semantic information to enhance the high-level feature representation of the smaller category, solving the class imbalance problem and improving the network's ability to distinguish between the background and the four types of ROP lesion structures. It also strengthens the learning of five structural features and solves the problem of high inter-class similarity. This method can assist ophthalmologists in completing the grading of ROP more quickly and effectively, thereby efficiently completing the diagnosis of ROP. Attached Figure Description

[0055] Figure 1 This is a flowchart of an automatic segmentation method for typical lesions of retinopathy of prematurity in infants according to the present invention.

[0056] Figure 2 This is a schematic diagram showing the marked retinopathy of prematurity lesions in one embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram showing the marked retinopathy of prematurity lesions in a stage 2 premature infant in one embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram showing the marked retinopathy of prematurity lesions in stage 3 premature infants in one embodiment of the present invention.

[0059] Figure 5 This is a schematic diagram showing the marked retinopathy of prematurity lesions in stage 4 premature infants in one embodiment of the present invention.

[0060] Figure 6 This is a schematic diagram of the framework of the deep neural network model in this invention.

[0061] Figure 7 This is a flowchart of an automatic segmentation technique for retinopathy of prematurity lesions in one embodiment of the present invention.

[0062] Figure 8 This is a flowchart illustrating the training process of a deep neural network in one embodiment of the present invention.

[0063] Figure 9 This is a schematic diagram of an automatic segmentation system for typical lesions of retinopathy of prematurity in infants according to the present invention. Detailed Implementation

[0064] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0066] Example 1

[0067] like Figure 1 As shown, this invention provides an automatic segmentation method for typical lesions of retinopathy of prematurity, which includes the following steps:

[0068] S1. Collect ROP images, filter out unqualified images, label the qualified ROP images and use them as ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set.

[0069] S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module.

[0070] S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model using backpropagation technology based on domain knowledge, validate the current model using the validation set, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss function to obtain the optimal model.

[0071] S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

[0072] Further, step S1 specifically involves: collecting ROP images, filtering out images where ROP lesions cannot be identified due to objective conditions, labeling the test set images with pixel-level tags, and using the resulting mask image as the ROP lesion segmentation dataset. The dataset is then divided into a training set, a validation set, and a test set. Specifically, labeling the test set images with pixel-level tags involves using different colored outlines to enclose the four structures in the ROP image fundus image: the boundary line, ridge, fiber, and retinal subdetachment.

[0073] Among them, the images in which ROP lesions cannot be identified due to objective conditions include: images in which ROP lesions cannot be identified due to excessively strong or dim lighting, and images in which multiple doctors cannot identify ROP lesions.

[0074] In one specific embodiment, the lesion structure in the ROP lesion fundus image includes: a boundary line, a ridge, fibers, and retinal subdetachment; wherein, the boundary line is as follows: Figure 2 As shown, ridges Figure 3 As shown, the fibers are as follows Figure 4 As shown, retinal subdetachment is as follows Figure 5 As shown.

[0075] Furthermore, in step S2, the deep neural network model includes a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. The overall network framework of the model is as follows: Figure 6 As shown, the ROP image is used as input to the deep neural network model, and the model outputs the ROP lesion segmentation map Y′, the ROP lesion auxiliary segmentation maps Y1′, Y2′, Y3′, and the feature map Y. f The image-level sequential encoding vector V′ is used as the input to the backbone segmentation network after data augmentation of the ROP image dataset. The backbone segmentation network outputs high-level features F, which are then used as the input to the auxiliary segmentation module. The auxiliary segmentation module outputs features F′ and feature map Y, respectively. f The feature F′ is added to the feature F and then input into the backbone segmentation network for further segmentation. The role of the image-level sequential encoding vector learning module is to obtain the sequential encoding vector of the model output, calculate the loss with it and the real sequential encoding, and optimize the parameters of the backbone segmentation network through backpropagation.

[0076] The preprocessing step involves data augmentation, specifically: first, the image is randomly scaled; then, it is normalized; and finally, a fixed-size image is randomly cropped using a resampling method. A random number is set, and if the sampled image contains only the background and the random number is greater than 0.2, resampling is performed until the sampled image contains any type of lesion. The image label processing follows the same procedure: it is first scaled to the same scaling ratio as the image, and then cropped to the same position and size as the final sampled image. Finally, the pixel-level labels are processed to obtain image block-level labels and image-level sequential encoding vector labels.

[0077] Among these, data augmentation can effectively reduce interference from background information.

[0078] Furthermore, the backbone segmentation network includes an encoder and a decoder. The encoder is an STDC network composed of multiple convolutional layers for feature extraction. The decoder uses bilinear interpolation to restore the high-level feature map to the resolution of the original image. The auxiliary segmentation module includes a spatial attention module, a channel attention module, convolutional layers, and a sigmoid function.

[0079] The backbone segmentation network consists of an encoder and a decoder. The encoder performs feature extraction, while the decoder's structure is built using the fusion connection mechanism of the STDC network to minimize spatial information loss, restore high-level feature maps to the original image resolution, and incorporates a block-level weighted balancing module. The segmentation network assigns a lesion class label to each pixel, ultimately obtaining a pixel-level prediction result Y′. During the training phase, it also outputs other results Y1′, Y2′, Y3′, and Y4′. f Y′ and V′. The loss is calculated using the true label Y′ and the predicted results Y′, Y1′, Y2′, Y3′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0080] Among them, the block-level weighted balancing module utilizes pixel-level semantic information to enhance the high-level feature representation of the smaller category and solve the class imbalance problem; the image-level sequential encoding vector learning module based on domain knowledge improves the network's ability to distinguish the background and four ROP lesion structures, strengthens the learning of five structural features, and solves the problem of high inter-class similarity.

[0081] Furthermore, in one round of training during the training phase, a ROP image I∈R is used. H×W×3As input to the model, the outputs are ROP lesion segmentation map Y′, ROP lesion auxiliary segmentation maps Y1′, Y2′, Y3′, and feature map Y′. f The image-level sequential encoding vector V′ is used to process the real pixel-level label Y to obtain the image block-level label Y. f The image-level sequential encoded vector label V is used, and the output obtained through the network is compared with the true label. The loss function is calculated, and the SGD optimizer is used to update and calculate the network parameters that affect the model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function. Among them, the loss is calculated with the predicted results Y′, Y′1, Y′2, Y′3 based on the true label Y and Y′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0082] Furthermore, the value of the c-th channel, i-th row, j-th column of the image block-level label is obtained. The specific mathematical expression is as follows:

[0083]

[0084]

[0085] Where H and W represent the height and width of the input image, and h and w represent the height and width of the output feature map. To round up, i = 0, 1, ..., h-1, j = 0, 1, ..., w-1. When Y contains a lesion of type c, T(Y, c) = 1, otherwise T(Y, c) = 0. c is the lesion category index, c = 0, 1, ..., C-1.

[0086] The mathematical expression for obtaining the image-level sequential encoded vector label V is as follows:

[0087] Q=U(Y,c) (8)

[0088] K=[[0,0,0,0],[1,0,0,0],[1,1,0,0],[1,1,1,0],[1,1,1,1]] (9)

[0089] V=Q×K (10)

[0090] Where Y is the pixel-level segmentation label of the image, c is the lesion category index, and the function U(Y,c) generates a 1*c vector Q, where each element of Q is either 0 or 1. Q[i] = 1 indicates the presence of the i-th type of lesion in Y, and Q[i] = 0 indicates the absence of the i-th type of lesion in Y, where i ∈ [0,c). Corresponding to Table 1, K is the lesion sequence code. V is the image-level sequence code vector, representing the sum of the sequence codes of the lesion categories present in Y.

[0091] In one specific embodiment, as shown in Table 1, K is the sequential encoding vector. V represents the sum of the sequential codes of the categories in Y.

[0092]

[0093]

[0094] Table 1. Sequential coding table of ROP lesions

[0095] Furthermore, the loss function includes: backbone segmentation loss L seg Auxiliary segmentation loss auL seg1 ,auL seg2 and auL seg3 Image block-level loss L feat and the Euclidean distance loss L of the sequential encoding vector w L seg ,auL seg1 ,auL seg2 and auL seg3 It consists of ohem loss and dice loss; the mathematical expression of the total loss function is as follows:

[0096] L=αL seg +βauL seg1 +γauL seg2 +δauL seg3 +εL feat +θL w (1)

[0097] L seg =L ohem +L dice (2)

[0098] L seg1 =L ohem1 +L dice1 (3)

[0099] L seg2 =L ohem2 +L dice2 (4)

[0100] L seg3 =Lohem3 +L dice3 (5)

[0101] Among them, α=1.0, β=0.4, γ=0.4, δ=0.7, ε=1.0, and θ=1.0.

[0102] Furthermore, the image block-level loss L feat The formula is as follows:

[0103]

[0104] Among them, w i For the loss weights of category i, L bce The cross-entropy loss is used for binary classification.

[0105] Calculate the loss L of the image-level sequential encoded vectors of the true labels and the predicted results. w The formula is as follows:

[0106]

[0107] Where l is the sequential encoding length, and in a specific embodiment, the sequential encoding length l = 4.

[0108] Based on the prediction results and true labels obtained from the above network, the total loss is calculated. The SGD optimizer is used to update and calculate the network parameters that affect model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function and finding the optimal model.

[0109] Furthermore, the validation process of the current model using the validation set is as follows: input all images in the validation set one by one into the currently trained model to obtain the segmentation results, calculate the evaluation metrics of the segmentation results and the true labels, calculate the AUC_PR of the current model on the validation set, and if the current AUC_PR is the highest, then save the current model as the current best model.

[0110] To overcome the problem of "small proportion of boundary lines and ridges in ROP fundus images, large proportion of lesions in stage 3 fiber and stage 4 retinal semi-detachment areas, and background causing ROP category imbalance," this invention proposes a resampling data preprocessing method to increase the sampling rate of ROP lesion areas, thereby reducing background interference. Additionally, a block-level weighted balancing module is proposed to balance the high-level feature representation of different categories, enhance the high-level feature representation of lesions with a very small proportion in the network, and learn domain knowledge, such as the spatial, textural, and color features of the four structures, thereby capturing more useful structural information. To overcome the problem of high lesion category similarity and difficulty in differentiation, an image-level sequential encoding vector learning method is proposed, combining the domain knowledge of the sequential nature of ROP lesions in different stages, to improve the network's ability to distinguish between the background and the four lesion structures, and strengthen the learning of the structural features of the four important lesions.

[0111] Example 2

[0112] Based on the above embodiment 1, combined with Figures 7-8 This embodiment details the training process and the functions and effects achieved by the present invention.

[0113] In a specific embodiment, such as Figure 7 As shown, the specific process of this invention is as follows:

[0114] Step (1): First, collect at least 1000 ROP images as a ROP lesion segmentation dataset;

[0115] Step (2), filter out unqualified images. Filter out images where ROP lesions cannot be identified due to objective conditions, such as excessively strong or dim lighting that makes ROP lesions unidentifiable, and images where multiple doctors cannot identify ROP lesions;

[0116] Step (3) Create image pixel-level labels. Pixel-level labels refer to using different colored lines to outline the four structures in the fundus image: "boundary line, ridge, fiber, and retinal subdetachment". Here, the labelme tool is used to complete the annotation of ROP images. Finally, 350 ROP image mask images in .png format are obtained. The final ROP dataset is divided into training set, validation set and test set.

[0117] Step (4): Establish a deep neural network model. For example... Figure 6 As shown, the model consists of a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. The backbone segmentation network comprises an encoder and a decoder. The encoder's skeleton is an STDC network, consisting of multiple convolutional layers for feature extraction. The decoder uses bilinear interpolation to restore the high-level feature map to the original image's resolution. The auxiliary segmentation module consists of a spatial attention module, a channel attention module, convolutional layers, and a sigmoid function. The high-level features F obtained from the backbone segmentation network are processed by the spatial and channel attention modules to obtain feature F′. Feature F′ is added to feature F and then fed into the backbone segmentation network for further segmentation. Simultaneously, it passes through convolutional layers and the sigmoid function to obtain the output Y of the block-level weighted feature balancing module. f The image-level sequential encoding vector learning module consists of a sequential encoding vector Euclidean distance loss. Based on domain knowledge, it utilizes backpropagation to optimize the backbone segmentation network's ability to distinguish between the background and four types of lesion structures. This invention also employs a deep-supervised loss calculation method, using segmentation images from three different layers and ground truth labels to calculate the loss and optimize the network.

[0118] Step (5), the training and validation phase. For example... Figure 8As shown in the diagram, this illustrates the entire training and validation process. The steps for optimizing the network during a single training round are as follows:

[0119] Step (a): In one round of training, a batch of images is taken for training.

[0120] Step (b) involves online data augmentation of this batch of images. First, the images are randomly scaled, then normalized, and then randomly cropped to a fixed size. A resampling method is used here: a random number is set, and if the sampled image contains only the background and the random number is greater than 0.2, it is resampled until the sampled image contains any type of lesion. The image labels are processed in the same way: first, they are scaled to the same scaling ratio as the image, and then cropped to the same position and size as the last sampled image. Finally, the pixel-level labels are processed to obtain image block-level labels and image-level sequential encoding vector labels.

[0121] Step (c) uses these images as input to the model to obtain six output results: four ROP lesion segmentation mask maps containing the boundary line, ridge, fiber, retinal subluxation and background, one image block level output and one image level sequential encoding vector for calculating the Euclidean distance loss of the sequential encoding vector.

[0122] Step (d) involves comparing the six outputs obtained from the network with the true labels to calculate the loss. The loss function consists of four sub-items: backbone segmentation loss L... seg Auxiliary segmentation loss auL seg1 ,auL seg2 and auL seg3 Image block-level loss L feat and the Euclidean distance loss L of the sequential encoding vector w The total loss is shown in formula (1);

[0123] Step (e) uses the optimizer SGD to update and compute the network parameters that affect model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function and finding the optimal model.

[0124] Step (f), validation set validation. Each image in the validation set is fed into the trained model to obtain segmentation results. The evaluation metrics for the segmentation results and the true labels are calculated. The AUC_PR of the current model on the validation set is calculated. If the current AUC_PR is the highest, the current model is saved.

[0125] Step (6) involves iteratively optimizing the invented deep neural network model through step (5), continuously minimizing the total loss function of formula (1), thereby continuously improving the segmentation performance of the network.

[0126] Step (7), Testing Phase. The current optimal network model is tested using the test set, and the segmentation performance of the model on the test set is finally obtained.

[0127] In one specific embodiment, the experimental results of the present invention are shown in Table 2.

[0128]

[0129] Table 2. Results of ROP Experiment

[0130] The specific functions and effects of this invention are as follows:

[0131] Compared to manual diagnosis, computer systems offer greater stability. This invention assists doctors in staging ROP by segmenting lesions, mitigating misdiagnosis and missed diagnosis, as well as inconsistent treatment outcomes, caused by differences in doctors' experience, workload, and physical condition. This reduces doctors' workload and improves their working conditions. Furthermore, this invention lowers the threshold for ROP screening, improves screening rates in primary hospitals and underdeveloped areas, alleviates the shortage of ROP specialists, increases the screening rate for ROP, and reduces the rate of blindness.

[0132] The segmentation network proposed in this invention can effectively segment the background and four types of lesion structures. By using an auxiliary segmentation module to enhance the high-level feature representation of the less represented categories, the network strengthens its discrimination between different categories, better distinguishing between foreground and background, and between different types of lesions. This reduces the likelihood of lesions being predicted as background, preventing missed diagnoses, and also reduces the likelihood of background being predicted as lesions, preventing misdiagnoses. Fusing the lesion feature information obtained from the auxiliary segmentation network with the feature information from the backbone segmentation network allows the backbone segmentation network to obtain more useful information to guide segmentation.

[0133] The proposed image-level sequential encoding vector learning module enhances the network's recognition of domain knowledge (color, size, and shape of four structures: boundary lines, ridges, fibers, and retinal subdetachment) and reduces misclassification between different categories. The learned domain knowledge is used to guide the backbone network in capturing background and structural features, strengthening the network's feature learning and representation capabilities and improving its ability to segment ROP lesions.

[0134] Example 3

[0135] like Figure 9 As shown, the present invention also provides an automatic segmentation system for typical lesions of retinopathy of prematurity (ROP) in infants, comprising a memory and a processor. The memory includes an automatic segmentation program for typical lesions of ROP in infants. When the processor executes the automatic segmentation program for typical lesions of ROP in infants, it performs the following steps:

[0136] S1. Collect ROP images, filter out unqualified images, label the qualified ROP images and use them as ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set.

[0137] S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module.

[0138] S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model using backpropagation technology based on domain knowledge, validate the current model using the validation set, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss function to obtain the optimal model.

[0139] S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

[0140] Further, step S1 specifically involves: collecting ROP images, filtering out images where ROP lesions cannot be identified due to objective conditions, labeling the test set images with pixel-level tags, and using the resulting mask image as the ROP lesion segmentation dataset. The dataset is then divided into a training set, a validation set, and a test set. Specifically, labeling the test set images with pixel-level tags involves using different colored outlines to enclose the four structures in the ROP image fundus image: the boundary line, ridge, fiber, and retinal subdetachment.

[0141] Among them, the images in which ROP lesions cannot be identified due to objective conditions include: images in which ROP lesions cannot be identified due to excessively strong or dim lighting, and images in which multiple doctors cannot identify ROP lesions.

[0142] Furthermore, in step S2, the deep neural network model includes a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. The ROP image is used as input to the deep neural network model, and the model outputs a ROP lesion segmentation map Y′, auxiliary ROP lesion segmentation maps Y′1, Y′2, and Y′3, and a feature map Y′. f And the image-level sequential encoding vector V′; specifically: the ROP image dataset is augmented and used as input to the backbone segmentation network. The backbone segmentation network outputs high-level features F, which are used as input to the auxiliary segmentation module. The auxiliary segmentation module outputs features F′ and feature map Y′ respectively. f The feature F′ is added to the feature F and then input into the backbone segmentation network for further segmentation. The role of the image-level sequential encoding vector learning module is to obtain the sequential encoding vector of the model output, calculate the loss with the real sequential encoding, and optimize the parameters of the backbone segmentation network through backpropagation.

[0143] The preprocessing step involves data augmentation, specifically: first, the image is randomly scaled; then, it is normalized; and finally, a fixed-size image is randomly cropped using a resampling method. A random number is set, and if the sampled image contains only the background and the random number is greater than 0.2, resampling is performed until the sampled image contains any type of lesion. The image label processing follows the same procedure: it is first scaled to the same scaling ratio as the image, and then cropped to the same position and size as the final sampled image. Finally, the pixel-level labels are processed to obtain image block-level labels and image-level sequential encoding vector labels.

[0144] Among these, data augmentation can effectively reduce interference from background information.

[0145] Furthermore, the backbone segmentation network includes an encoder and a decoder. The encoder is an STDC network composed of multiple convolutional layers for feature extraction. The decoder uses bilinear interpolation to restore the high-level feature map to the resolution of the original image. The auxiliary segmentation module includes a spatial attention module, a channel attention module, convolutional layers, and a sigmoid function.

[0146] The backbone segmentation network consists of an encoder and a decoder. The encoder performs feature extraction, while the decoder's structure is built using the fusion connection mechanism of the STDC network to minimize spatial information loss, restore high-level feature maps to the original image resolution, and incorporates a block-level weighted balancing module. The segmentation network assigns a lesion class label to each pixel, ultimately obtaining a pixel-level prediction result Y′. During the training phase, it also outputs other results Y′1, Y′2, Y′3, and Y′... f And V′. The loss is calculated using the predicted results Y′, Y′1, Y′2, and Y′3, which are the true label Y and Y′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0147] Among them, the block-level weighted balancing module utilizes pixel-level semantic information to enhance the high-level feature representation of the smaller category and solve the class imbalance problem; the image-level sequential encoding vector learning module based on domain knowledge improves the network's ability to distinguish the background and four ROP lesion structures, strengthens the learning of five structural features, and solves the problem of high inter-class similarity.

[0148] Furthermore, in one round of training during the training phase, a ROP image I∈R is used. H×W×3As input to the model, the outputs are ROP lesion segmentation map Y′, ROP lesion auxiliary segmentation maps Y′1, Y′2, Y′3, and feature map Y′. f The image-level sequential encoding vector V′ is used to process the real pixel-level label Y to obtain the image block-level label Y. f The image-level sequential encoded vector label V is used, and the output obtained through the network is compared with the true label. The loss function is calculated, and the SGD optimizer is used to update and calculate the network parameters that affect the model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function. Among them, the loss is calculated with the predicted results Y′, Y′1, Y′2, Y′3 based on the true label Y and Y′. f The loss is calculated using the label Y, which is the result of multi-hot encoding the true label. f The loss calculated by V′ is the label V that is sequentially encoded from the real label.

[0149] Furthermore, the value of the c-th channel, i-th row, j-th column of the image block-level label is obtained. The specific mathematical expression is as follows:

[0150]

[0151]

[0152] Where H and W represent the height and width of the input image, and h and w represent the height and width of the output feature map. To round up, i = 0, 1, ..., h-1, j = 0, 1, ..., w-1. When Y contains a lesion of type c, T(Y, c) = 1, otherwise T(Y, c) = 0. c is the lesion category index, c = 0, 1, ..., C-1.

[0153] The mathematical expression for obtaining the image-level sequential encoded vector label V is as follows:

[0154] Q=U(Y,c) (8)

[0155] K=[[0,0,0,0],[1,0,0,0],[1,1,0,0],[1,1,1,0],[1,1,1,1]] (9)

[0156] V=Q×K (10)

[0157] Where Y is the pixel-level segmentation label of the image, c is the lesion category index, and the function U(Y,c) generates a 1*c vector Q, where each element of Q is either 0 or 1. Q[i] = 1 indicates the presence of the i-th type of lesion in Y, and Q[i] = 0 indicates the absence of the i-th type of lesion in Y, where i ∈ [0,c). Corresponding to Table 1, K is the lesion sequence code. V is the image-level sequence code vector, representing the sum of the sequence codes of the lesion categories present in Y.

[0158] Furthermore, the loss function includes: backbone segmentation loss L seg Auxiliary segmentation loss auL seg1 ,auL seg2 and auL seg3 Image block-level loss L feat and the Euclidean distance loss L of the sequential encoding vector w L seg ,auL seg1 ,auL seg2 and auL seg3 It consists of ohem loss and dice loss; the mathematical expression of the total loss function is as follows:

[0159] L=αL seg +βauL seg1 +γauL seg2 +δauL seg3 +εL feat +θL w (1)

[0160] L seg =L ohem +L dice (2)

[0161] L seg1 =L ohem1 +L dice1 (3)

[0162] L seg2 =L ohem2 +L dice2 (4)

[0163] L seg3 =L ohem3 +L dice3 (5)

[0164] Among them, α=1.0, β=0.4, γ=0.4, δ=0.7, ε=1.0, and θ=1.0.

[0165] Furthermore, the image block-level loss L feat The formula is as follows:

[0166]

[0167] Among them, w i For the loss weights of category i, L bce The cross-entropy loss is used for binary classification.

[0168] Calculate the loss L of the image-level sequential encoded vectors of the true labels and the predicted results. w The formula is as follows:

[0169]

[0170] Where l is the sequential code length, and l = 4.

[0171] Furthermore, the validation process of the current model using the validation set is as follows: input all images in the validation set one by one into the currently trained model to obtain the segmentation results, calculate the evaluation metrics of the segmentation results and the true labels, calculate the AUC_PR of the current model on the validation set, and if the current AUC_PR is the highest, then save the current model as the current best model.

[0172] The icons in the accompanying drawings that depict the structural positional relationships are for illustrative purposes only and should not be construed as limiting this patent.

[0173] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for automatically segmenting typical lesions of retinopathy of prematurity in infants, characterized in that, Includes the following steps: S1. Collect images of retinopathy of prematurity (ROP) in premature infants, filter out unqualified images, label the qualified ROP images and use them as the ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set. S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. Use the preprocessed retinopathy of prematurity (ROP) image training set as input to the deep neural network model, and the model outputs a segmentation map of ROP lesions. Auxiliary segmentation diagram of retinopathy of prematurity lesions , , Feature map and image-level sequential encoding vector ; Specifically, the dataset of images of retinopathy of prematurity (ROP) is augmented and then used as input to the backbone segmentation network. The backbone segmentation network outputs high-level features F, which are then used as input to the auxiliary segmentation modules. The auxiliary segmentation modules output features respectively. and feature map ,feature The image-level sequential encoding vector learning module is used to obtain the sequential encoding vector of the model output, calculate the loss with the real sequential encoding, and optimize the parameters of the backbone segmentation network through backpropagation. S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model, use the validation set to validate the current model, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss to obtain the optimal model. S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

2. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 1, characterized in that, Step S1 specifically involves: collecting images of retinopathy of prematurity (ROP) in premature infants, filtering out images where ROP lesions cannot be identified due to objective conditions, labeling the remaining images with pixel-level tags, and using the resulting mask images as the ROP lesion segmentation dataset. The dataset is then divided into training, validation, and test sets. Specifically, labeling the test set images with pixel-level tags involves using different colored outlines to enclose the four structures—boundary lines, ridges, fibers, and retinal subdetachment—in the fundus images of ROP.

3. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 1, characterized in that, The backbone segmentation network includes an encoder and a decoder. The encoder is an STDC network composed of multiple convolutional layers for feature extraction. The decoder uses bilinear interpolation to restore the high-level feature map to the resolution of the original image. The auxiliary segmentation module includes a spatial attention module, a channel attention module, convolutional layers, and a sigmoid function.

4. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 3, characterized in that, During one round of training, an image of retinopathy of prematurity was used. As input to the model, the output is a segmentation map of retinopathy of prematurity lesions. Auxiliary segmentation diagram of retinopathy of prematurity lesions , , Feature map and image-level sequential encoding vector The actual pixel-level label Y is processed to obtain the image block-level label. The image-level sequentially encoded vector label V is used, and the output obtained through the network is compared with the true label. The loss function is calculated, and the SGD optimizer is used to update and calculate the network parameters that affect the model training and model output, so that they approach or reach the optimal value, thereby minimizing the loss function; where, and the prediction result , , , The loss is calculated using the true label Y, and The loss is calculated based on the labels that have undergone multy-hot encoding of the real labels. ,and The loss is calculated for the label V, which is the label sequentially encoded from the real label.

5. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 4, characterized in that, Obtain image block-level labels Specifically, this involves obtaining the value of the c-th channel, i-th row, j-th column of the image block-level label. Its mathematical expression is as follows: Where H and W represent the height and width of the input image, and h and w represent the height and width of the output feature map. H / h, , To round up, i=0,1,......,h-1, j=0,1,......,w-1. When Y contains a lesion of type c, T(Y,c)=1, otherwise T(Y,c)=0. c is the lesion category index, c=0,1,......,C-1. The mathematical expression for obtaining the image-level sequential coding vector V is as follows: Where Y is the pixel-level segmentation label of the image, c is the lesion category index, and the function U(Y, c) generates the vector Q=(q0,q2,…,q c-1 ), q i =1 (i=0,…,c-1), the vector Q represents the existence of the i-th type of lesion in Y, q i =0 (i=1,…,c-1) indicates that there is no lesion of type i in Y; V is an image-level sequential encoding vector, representing the sum of sequential encodings of lesion categories present in the pixel-level segmentation label Y.

6. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 4, characterized in that, The loss function includes: backbone segmentation loss. Auxiliary segmentation loss , and Image block-level loss and sequential encoding vector Euclidean distance loss ,in , , and It consists of ohem loss and dice loss; the mathematical expression of the total loss function is as follows: Where, α=1.0, β=0.4, γ=0.4, δ=0.7, ε=1.0, =1.

0.

7. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 6, characterized in that, Image block-level loss The formula is as follows: in, Let i be the loss weight for category i. The cross-entropy loss is used for binary classification. Calculate the loss of the image-level sequential encoded vectors of the true labels and the predicted results. The formula is as follows: (12) Where l is the sequential encoding length.

8. The method for automatically segmenting typical lesions of retinopathy of prematurity in infants according to claim 1, characterized in that, The validation process of the current model using the validation set is as follows: input all images in the validation set one by one into the currently trained model to obtain the segmentation results, calculate the evaluation metrics of the segmentation results and the true labels, calculate the AUC_PR of the current model on the validation set, and if the current AUC_PR is the highest, save the current model as the current best model.

9. An automatic segmentation system for typical lesions of retinopathy of prematurity (ROP) in premature infants, comprising a memory and a processor, wherein the memory includes an automatic segmentation program for typical lesions of ROP in premature infants, and the automatic segmentation program for typical lesions of ROP in premature infants, when executed by the processor, performs the following steps: S1. Collect images of retinopathy of prematurity (ROP) in premature infants, filter out unqualified images, label the qualified ROP images and use them as the ROP lesion segmentation dataset, and divide the dataset into training set, validation set and test set. S2. Construct a deep neural network model, including a backbone segmentation network, an auxiliary segmentation module, and an image-level sequential encoding vector learning module. Use the preprocessed retinopathy of prematurity (ROP) image training set as input to the deep neural network model, and the model outputs a segmentation map of ROP lesions. Auxiliary segmentation diagram of retinopathy of prematurity lesions , , Feature map and image-level sequential encoding vector Specifically, the dataset of images of retinopathy of prematurity (ROP) is augmented and then used as input to the backbone segmentation network. The backbone segmentation network outputs high-level features F, which are then used as input to the auxiliary segmentation modules. The auxiliary segmentation modules output features respectively. and feature map ,feature The image-level sequential encoding vector learning module is used to obtain the sequential encoding vector of the model output, calculate the loss with the real sequential encoding, and optimize the parameters of the backbone segmentation network through backpropagation. S3. After preprocessing the training set, use the training set to train the deep neural network model, calculate the loss, optimize the network parameters of the deep neural network model, use the validation set to validate the current model, and save the model with the highest evaluation index as the current optimal model. After several iterations, minimize the total loss function to obtain the optimal model. S4. Test the optimal model using the test set to obtain the segmentation performance evaluation index of the model on the test set.

Images