A retinal lesion segmentation method based on hybrid supervised learning

Abstract

The present application relates to the technical field of image processing, in particular to a retinal lesion segmentation method based on mixed supervision learning, which utilizes limited full-annotation images and more point-annotation images for retinal multi-class lesion segmentation; the mixed supervision network model adopts a parameter decoupling strategy to optimize the teacher network, and in the conventional training round, the student network updates the parameters of the teacher network based on the full-annotation images and the point-annotation images; in the specific training round, the current teacher network parameters are copied to the auxiliary optimization network, the auxiliary optimization network updates the parameters of the auxiliary optimization network based on the full-annotation images, and the updated parameters are copied back to the teacher network, so that the supervised fine-tuning of the teacher network is completed; based on the parameter decoupling strategy to optimize the teacher network, the pixel-level annotation information is fully utilized, and the strong dependency between the teacher network and the student network is relieved to a certain extent, so that the performance of the average teacher network in retinal lesion segmentation is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for retinal lesion segmentation based on hybrid supervised learning. Background Technology

[0002] Fundus lesions in the macular region of the retina can simultaneously cause multiple lesions, such as intra-retinal fluid (IRF), sub-retinal fluid (SRF), and pigment epithelial detachment (PED). These retinal lesions, as biomarkers, significantly influence the identification and treatment selection of various fundus diseases due to their shape, location, and size. With the increasing maturity of OCT imaging technology, OCT two-dimensional B-scan sections can clearly present the physiological characteristics of each retinal layer and effectively identify pathological changes such as retinal fluid accumulation. By accurately identifying retinal lesion areas in OCT images, quantitative analysis of fundus lesions, disease progression monitoring, and efficacy evaluation can be achieved, thus providing an objective basis for the development of individualized medical plans.

[0003] Traditional manual analysis methods suffer from limitations such as significant subjective differences, low efficiency, and susceptibility to misjudgments. In contrast, methods based on Convolutional Neural Networks (CNNs) can automatically learn and extract image features from the data itself, enabling automatic segmentation of retinal lesions, which is of great significance for improving clinical diagnostic efficiency. However, CNN-based lesion segmentation methods largely rely on large-scale training datasets with complete annotation information, the creation of which requires extremely high manual annotation costs. Obtaining labeled data for retinal lesions is difficult, primarily because retinal lesions typically have small areas, significant scale differences, and diverse morphological features. These characteristics further complicate the accurate identification and diagnosis of lesions by doctors, and also significantly increase the economic and time costs of retinal lesion annotation. Obtaining a sufficient amount of accurately labeled data from medical experts is a challenging task; due to the lack of sufficient pixel-level labeled data, CNN-based segmentation methods often struggle to fit the data, leading to decreased segmentation performance.

[0004] To solve this problem, a hybrid supervision method is proposed, that is, only a small amount of data in the data set is fully labeled in the form of pixel-level complete labeling information (each pixel is labeled), and the rest of the data is only labeled in the form of point labeling (that is, part of the pixel points are labeled), so as to maintain good segmentation performance under the premise of reducing the labeling workload. The hybrid supervision method is usually regarded as a special case of semi-supervised or weakly supervised, but the method needs to be adjusted to make full use of different types of labeling information. At present, the research on hybrid supervision method mainly focuses on images with bright colors in natural scenes. The target objects in these images usually occupy a large spatial area and have clear boundary contours. Compared with this, there are several challenges in retinal lesion segmentation based on hybrid supervision: 1) The difference between the lesion and the background is not obvious, the lesion area is small, the shape is irregular and the boundary is not clear, which makes it difficult to learn the discriminative lesion features and easy to misjudge; 2) The image resolution is high, and the lesion area is small, and the background occupies a large proportion, which causes the class imbalance problem, and the model tends to predict the pixel as background, further increasing the difficulty of segmentation; 3) Weakly labeled information can only provide partial information of the lesion area, and lacks complete lesion extension, shape or boundary information, which will cause the model to lack sufficient context information when learning, resulting in inaccurate prediction of the lesion boundary, or even unable to learn the complete lesion, and poor segmentation accuracy. Therefore, the current hybrid supervision method is not suitable for retinal lesion segmentation task. SUMMARY

[0005] Therefore, the technical problem to be solved by the present application is to overcome the problem that it is difficult to learn complete lesion information in the prior art, resulting in poor segmentation accuracy.

[0006] To solve the above technical problems, the present application provides a retinal lesion segmentation method based on hybrid supervision learning, applied to a hybrid supervision network model including a teacher network, a student network and an auxiliary optimization network with the same segmentation network structure, comprising:

[0007] Obtain a point-labeled image set and a fully-labeled image set of the retinal OCT image;

[0008] Input the point-labeled set and the fully-labeled set into the student network, input the fully-labeled set into the auxiliary optimization network, and input the point-labeled set into the teacher network, once a specific training round according to a predetermined number of regular training intervals, and perform hybrid supervision training on the hybrid supervision network model, comprising:

[0009] In the regular training round, the student network is allowed to perform forward propagation and gradient back propagation, update the student network model parameters, and update the teacher network model through exponential moving average;

[0010] In a specific training round, the parameters of the current teacher network model updated for a preset number of times are copied to the auxiliary optimization network, the auxiliary optimization network is subjected to forward propagation and back propagation, the parameters of the updated auxiliary optimization network are obtained, and the parameters are copied back to the teacher network model;

[0011] Until the trained student network meets a preset stop condition, a trained student network is obtained.

[0012] The trained student network is used as a target segmentation model to segment the OCT image to be segmented, and a retinal lesion segmentation result is obtained.

[0013] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0014] The retinal lesion segmentation method based on the mixed supervision learning provided in the present application uses limited full-annotation images and more point-annotation images for retinal multi-class lesion segmentation, which can effectively reduce the lesion annotation cost; the mixed supervision network model of the present application takes an average teacher network as a basic framework, and uses a parameter decoupling strategy to optimize the teacher network; under the conventional training round, the student network updates the parameters of the teacher network based on the full-annotation images and the point-annotation images; in a specific training round, the current teacher network parameters are copied to the auxiliary optimization network, and the auxiliary optimization network uses the full-annotation images to update the parameters of the auxiliary optimization network, and the updated parameters are copied back to the teacher network to complete the supervised fine-tuning of the teacher network; in a specific training round, the student network suspends the transmission of parameters to the teacher network to avoid interfering with the independent update of the teacher network. The present application optimizes the teacher network based on the parameter decoupling strategy, which can fully utilize the pixel-level annotation information and to some extent alleviate the strong dependency between the teacher network and the student network, and effectively improve the performance of the average teacher network in retinal lesion segmentation.

[0015] The local feature enhancement segmentation network of the present application adopts a U-Net architecture with VGG16 as an encoder. In order to fully utilize the supervision information provided by the sparse point annotation and improve the local feature aggregation ability, a feature neighborhood enhancement module is added to enhance the feature aggregation ability of deep features for regions of the same class, while retaining edge detail information and improving the network's ability to learn features of irregular small lesion regions under point annotation. At the same time, in view of the characteristics of retinal lesions of different sizes and irregular distribution, a multi-scale difference feature fusion module is set up at each level to perform layer-by-layer subtraction on the feature maps of adjacent levels, capture the difference information between the initial feature maps and reduce redundant information, and capture lesion distribution information from different scales, thereby improving the segmentation accuracy of retinal OCT images.

[0016] The application utilizes the superpixel pseudo label generation module and the confidence guided label correction module to generate the pseudo label and the label weight map of the point labeled image; firstly, the sparse point label is expanded into the superpixel pseudo label through the superpixel pseudo label generation module, then the confidence guided label correction module is used to identify the errors in the pseudo label through confidence learning and further refine the pseudo label to expand the supervision information, reduce the interference of the noise label on the model training, and improve the utilization efficiency of the point labeled image;

[0017] The application considers the training and supervision information difference between the point labeled image and the full labeled image from the global perspective of the mixed supervision network, and embeds the cross supervision cross attention module and the pseudo label deep supervision module in the student network; the cross supervision cross attention module can improve the feature correlation of the full supervision sample and the point supervision sample, realize the feature alignment capability between the full supervision feature and the point supervision feature, further improve the supervision capability of the point supervision sample, and alleviate the imbalance problem of the convergence speed of the point supervision branch and the full supervision branch; the pseudo label deep supervision module enhances the model stability, improves the model segmentation performance, and improves the accuracy of the predicted lesion boundary. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the drawings, in which:

[0019] Figure 1 is a step flow chart of the retinal lesion segmentation method based on mixed supervision learning of the application;

[0020] Figure 2 is a structural schematic diagram of the local enhanced feature segmentation network;

[0021] Figure 3 is a structural schematic diagram of the neighborhood feature enhancement module;

[0022] Figure 4 is a structural schematic diagram of the difference calculation unit;

[0023] Figure 5 is a structural schematic diagram of the superpixel pseudo label generation module;

[0024] Figure 6 is a structural schematic diagram of the cross supervision cross attention module;

[0025] Figure 7 is a structural schematic diagram of the mixed supervision network model;

[0026] Figure 8 is a segmentation result comparison diagram of the method of the application and other methods. DETAILED DESCRIPTION

[0027] The application will be further described below with reference to the drawings and specific examples, so that those skilled in the art can better understand the application and implement it, but the examples are not intended to limit the application.

[0028] The present application is directed to the problem of time-consuming and laborious retinal lesion annotation, and proposes a hybrid supervised network for retinal lesion segmentation using only limited labeled images. Various strategies are used to train the retinal lesion segmentation network based on fully labeled images and point-labeled images, so that the network has good segmentation performance. In the model training method, the present application proposes a consistency regularization method, which uses an average teacher model as the basic framework. And on this basis, a new parameter decoupling optimization strategy (PDOS) is proposed to alleviate the tight coupling problem. In the network structure design, in order to fully improve the network's learning ability for irregular small lesion regions under point labeling, a local feature enhancement network (LFENet) is proposed, and a neighborhood feature enhancement module (NFE) is designed to enhance the feature aggregation ability of deep features for the same class region, while retaining edge detail information. In view of the characteristics of retinal lesions of different sizes and irregular distribution, a multiscale difference feature fusion module (MDF2) is used to reduce redundant information while capturing lesion distribution information from different scales. In addition, in order to obtain more effective information from point labeling and alleviate the problem of sparse labeling, a superpixel pseudo-label generation and correction strategy is used. First, the sparse point labeling is expanded to superpixel pseudo-labels through the superpixel-guided label generation (SLG) module, and then the confident learning-guided label refinement (CL2R) module is used to further refine the pseudo-labels and improve the utilization efficiency of point-labeled images. Finally, from the global perspective of the hybrid supervised network, considering the differences in training and supervision information between point-labeled images and fully labeled images, the cross-supervised cross-attention module (CSCA) and the pseudo-label deeply supervised module (PDS) are embedded in the student network to realize the feature alignment ability between fully supervised features and weakly supervised features, and to alleviate the problem of uneven convergence speed between weakly supervised branches and fully supervised branches.

[0029] Reference Figure 1The flowchart shown illustrates the steps of the retinal lesion segmentation method based on hybrid supervised learning of the present invention. This method is applied to a hybrid supervised network model comprising a teacher network, a student network, and an auxiliary optimization network with the same segmentation network structure, and includes:

[0030] S101: Obtain the set of point-annotated images and the set of fully annotated images of retinal OCT images;

[0031] S102: Input the point-label set and the full-label set into the student network, input the full-label set into the auxiliary optimization network, and input the point-label set into the teacher network. Perform hybrid supervised training on the hybrid supervised network model according to a predetermined number of regular training intervals and one specific training epoch, including:

[0032] In regular training rounds, the student network is made to perform forward propagation and gradient backpropagation to update the parameters of the student network model, and the teacher network model is updated by exponential moving average.

[0033] In a specific training round, the parameters of the current teacher network model, which have been updated a preset number of times, are copied into the auxiliary optimization network. The auxiliary optimization network then performs forward and backward propagation to obtain the updated parameters of the auxiliary optimization network, which are then copied back into the teacher network model.

[0034] S103: Obtain the trained student network until the trained student network meets the preset cutoff condition;

[0035] S104: Using the trained student network as the target segmentation model, segment the OCT image to be segmented to obtain the segmentation results of retinal lesions.

[0036] This invention proposes for the first time a method that combines the distribution characteristics of retinal lesions in OCT images with a hybrid supervised network trained on fully labeled and point-labeled images to achieve retinal lesion segmentation in OCT images. This network can fully utilize the advantages of fully supervised, weakly supervised, and semi-supervised learning to achieve accurate segmentation of multiple types of retinal lesions. The network uses an average teacher model as its basic framework and employs a novel parameter decoupling strategy (PDOS) to optimize the teacher network, alleviating the tight coupling problem between the teacher and student networks.

[0037] After obtaining the point-labeled image set and the full-labeled image set of the retinal OCT image, the embodiment includes: respectively performing random weak enhancement and random strong enhancement on the point-labeled images and the full-labeled images in the point-labeled image set and the full-labeled image set to construct a point-labeled weak data set, a point-labeled strong data set, a full-labeled weak data set and a full-labeled strong data set; inputting the point-labeled strong data set and the full-labeled strong data set into the student network respectively, inputting the full-labeled weak data set into the auxiliary optimization network, and inputting the point-labeled weak data set into the teacher network.

[0038] Specifically, the full-labeled image refers to that each pixel in the image is labeled with a class label, and the point-labeled image refers to that only a small number of pixels in the image are labeled with a class label. In the embodiment, the strong enhancement adopted for the image includes: long-distance elastic deformation, superimposed speckle noise, singular value decomposition noise structure migration and simulation of blood vessel artifacts. In the embodiment, the weak enhancement adopted for the image includes: short-distance elastic deformation, random horizontal flip, contrast adjustment and gray scale transformation.

[0039] Specifically, the segmentation network structure of the teacher network, the student network and the auxiliary optimization network with the same segmentation network structure is a local feature enhancement segmentation network; the local feature enhancement segmentation network is a U-Net architecture with VGG16 as an encoder; as shown in FIG. 1, it is a structure diagram of the local feature enhancement segmentation network; the OCT image is input into the local feature enhancement segmentation network, which includes: Figure 2

[0040] The OCT image is input into the encoder, sequentially passes through N layers of encoding modules connected in series along the positive propagation direction, and N encoding features are obtained;

[0041] The Nth encoding feature is input into the neighborhood feature enhancement module to perform deep feature aggregation and obtain an aggregated encoding feature;

[0042] Each encoding feature and the aggregated encoding feature are subjected to channel transformation through a convolution layer to obtain N transformed features corresponding to the encoding features;

[0043] The N transformed features are input into the multi-scale differential feature fusion module to perform differential operation between the features and obtain N-1 fusion features;

[0044] The N-1 fusion features are respectively input into N-1 series-connected decoding modules to perform decoding in combination with the up-sampling features of the decoding features output by the previous decoding module to obtain N-1 decoding features; wherein the Nth decoding module decodes in combination with the Nth transformed feature;

[0045] The decoding feature output by the first decoding module is subjected to convolution to obtain a segmentation result map.

[0046] ​In the network structure design, a local feature enhancement network (LFENet) is proposed, which is based on the U-Net architecture with VGG16 as the encoder. In order to make full use of the supervision information provided by sparse annotation and improve the local feature aggregation capability, a neighborhood feature enhancement module (NFE) is proposed. At the same time, through the multi-scale differential feature fusion module (MDF2), the redundant information is reduced while the lesion distribution information is captured from different scales, and the segmentation accuracy is improved.

[0047] Referring to Figure 3 Fig. 1 is a structural schematic diagram of a neighborhood feature enhancement module; the Nth encoding feature is input into the neighborhood feature enhancement module for deep feature aggregation to obtain an aggregated encoding feature, including:

[0048] The Nth encoding feature is input into the neighborhood feature enhancement module, and 8 3x3 neighborhood convolution kernels are combined with M different scale dilated convolutions respectively to obtain 8M convolution methods, and the Nth encoding feature is convolved to obtain 8M neighborhood affinity features;

[0049] The similarity between each neighborhood affinity feature and the Nth encoding feature is calculated as the weight of each neighborhood affinity feature;

[0050] The neighborhood affinity features are weighted and summed based on the weights of the neighborhood affinity features to obtain the aggregated encoding feature;

[0051] In each neighborhood convolution kernel, only the position in a specified direction adjacent to the center pixel is fixed as 1, and the remaining positions are fixed as 0; the specified direction includes the upper, lower, left, right, and diagonal directions of the center pixel, i.e., the upper left, upper right, lower left, and lower right.

[0052] Specifically, the N transformed features are input into the multi-scale differential feature fusion module for differential operation between the features to obtain N-1 fusion features, including:

[0053] The N transformed features are input into the multi-scale differential feature fusion module, and the s th transformed feature sequentially passes through N-s difference calculation units to obtain the output of each difference calculation unit, including:

[0054] The s th transformed feature The output of the first difference calculation unit is represented as: , ;

[0055] The output of the s th difference unit of the s th transformed feature The output of the s th difference unit of the s th transformed feature is represented as: The output of the s th difference unit of the s th transformed feature ​, ;

[0056] For each transformation feature, the outputs of the Ns difference calculation units are summed and then convolved to obtain the s-th fused feature corresponding to the s-th transformation feature, expressed as: ;

[0057] in, This indicates a difference calculation operation. ; Represents the Gaussian error linear activation function. Representation group standardization, This represents a convolution with a kernel whose values ​​are all 1s. and Two features representing the input difference computation unit, This indicates the size of the filter kernel.

[0058] Reference Figure 4 The diagram shows the structure of the difference calculation unit. The first and second input features of the difference calculation unit are processed by multiple convolutional modules of different sizes to obtain convolutional features of corresponding sizes. The convolutional features of the same size as the first and second input features are subtracted element-wise to obtain the corresponding difference features. The difference features of multiple sizes are added element-wise to obtain the output of the difference calculation unit.

[0059] The local feature enhancement segmentation network in this application adopts a U-Net architecture with VGG16 as the encoder. In order to make full use of the supervision information provided by sparse point annotations and improve the local feature aggregation capability, a feature neighborhood enhancement module is added to enhance the feature aggregation capability of deep features for regions of the same category, while preserving edge detail information and improving the network's learning ability for irregular small lesion regions under point annotations. At the same time, considering the characteristics of retinal lesions of varying sizes and irregular distribution, a multi-scale differential feature fusion module is used to set multiple differential calculation units at each level to perform layer-by-layer subtraction on the feature maps of adjacent levels, capturing the difference information between the initial feature maps and reducing redundant information. At the same time, lesion distribution information is captured from different scales, thereby improving the segmentation accuracy of retinal OCT images.

[0060] To expand the supervisory information, this embodiment proposes a superpixel pseudo-label generation module (SLG) to expand sparse point annotations into superpixel pseudo-labels and assign corresponding weights to each pseudo-label. Simultaneously, a confidence-guided label correction module (CL²R) identifies errors in the pseudo-labels and further refines them to expand the supervisory information, reducing the interference of noisy labels on model training. (Refer to...) Figure 5As shown, it is a structural schematic diagram of the superpixel pseudo label generation module; in step S102, the hybrid supervised network model is trained by hybrid supervision, and the pseudo label and label weight map of the point annotation image are generated by using the superpixel pseudo label generation module and the confidence guided label correction module, including:

[0061] The point annotation image is input into the superpixel pseudo label generation module, the point annotation image is divided into a plurality of superpixel blocks by using a superpixel segmentation algorithm, the pixel label of the superpixel block with point label in the region is assigned as the class corresponding to the point label in the region, and the initial pseudo label of the superpixel block with point label in the region is obtained;

[0062] The label weight of the superpixel block with point label in the region is assigned as 1, and the label weight of the remaining superpixel block is assigned as 0.5;

[0063] Based on the similarity between the histograms of any two superpixel blocks, the initial pseudo label of the superpixel block with the initial pseudo label is propagated to the superpixel block with similarity exceeding the preset threshold, until there are no two superpixel blocks with similarity exceeding the preset threshold, and the initial pseudo label of each superpixel block is obtained;

[0064] For the superpixel block to which the initial pseudo label is propagated, the minimum distance between the superpixel block centroid of the superpixel block to which the initial pseudo label is propagated and the centroid of the superpixel block is calculated, and the label weight of the superpixel block to which the initial pseudo label is propagated is constructed; the label weight of each superpixel block is obtained, and the initial label weight map is generated;

[0065] By using the confidence learning method, a joint distribution matrix of the prediction label output by the teacher network and the initial pseudo label is constructed, the incorrectly labeled pixels in the initial pseudo label are identified, and the initial pseudo label of the incorrectly labeled pixels is corrected as the corresponding prediction label, and the label weight is corrected as 1, to obtain the target pseudo label and the target label weight map.

[0066] The present application generates the pseudo label and label weight map of the point annotation image by using the superpixel pseudo label generation module and the confidence guided label correction module; first, the sparse point annotation is expanded into a superpixel pseudo label by the superpixel pseudo label generation module, then the confidence guided label correction module is used to identify errors in the pseudo label and further refine the pseudo label to expand the supervision information, reduce the interference of noise labels on model training, and improve the utilization efficiency of the point annotation image.

[0067] The present embodiment considers the training and supervision information difference between the point annotation image and the full annotation image, embeds a cross-supervision cross-attention module (CSCA) and a pseudo label deep supervision module (PDS) in the student network to realize the feature alignment capability between full supervision features and weak supervision features, and simultaneously alleviate the speed imbalance problem.

[0068] Specifically, after obtaining the target pseudo-label and target label weight graph, the student network is subjected to pixel-level supervision using the pseudo-label deep supervision module, including:

[0069] Downsampling is performed on the target pseudo-label and target label weight maps respectively, so that the resolution of the downsampled target pseudo-label and target label weight maps is similar to that of the first-order target pseudo-label in the student network. The prediction outputs of all decoding modules are the same, and the weight map of downsampled pseudo-labels and downsampled labels is obtained.

[0070] For the first in the student network The prediction output of each decoding module is convolved and activated to obtain deep prediction labels;

[0071] The first loss is constructed based on the standard cross-entropy loss between the ground and predicted labels of the labeled image. , is represented as: ;

[0072] Based on the deep predicted labels of the point-annotated image from the (N-1)th decoding module, and the downsampled pseudo-labels and downsampled label weight map of the point-annotated image, a second loss is constructed. , is represented as: ;

[0073] The first loss and the second loss are added together to obtain the point-supervised loss, which is then used to train the student network.

[0074] in, Represents the standard cross-entropy loss. and They represent the first The true labels of the labeled images and the predicted labels of the student network; , This represents the total number of data points in the set of labeled images; The pixel-weighted cross-entropy loss function is expressed as follows: ; , and This represents a graph showing predicted labels, target pseudo-labels, and target label weights. , This represents the total number of pixels in the image; , Indicates the number of lesion categories; The target label weight graph represents the first... The label value of each pixel. Indicates the first Does the pixel belong to the ? Pseudo-tags for classes Indicates the first The pixel belongs to the first The predicted probability of a class; Indicates the first Pseudo-labels for images with point annotations. express The corresponding target label weight graph; Indicates the student network's response to the first When making predictions on a labeled image, based on the first... The deep predicted labels obtained from the output of each decoding module; express The corresponding downsampling pseudo-label; express The corresponding downsampling label weight map.

[0075] Reference Figure 6 The diagram shown illustrates the structure of the cross-supervised cross-attention module. In this embodiment, the student network is optimized using the cross-supervised cross-attention module, including:

[0076] The output features of the point-labeled image after passing through the Nth encoding module are used as point-supervised classification features; the output features of the point-labeled image after passing through the N-1th decoding module are used as point-supervised segmentation features.

[0077] The output features of the fully labeled image after passing through the Nth encoding module are used as the fully supervised classification features; the output features of the fully labeled image after passing through the N-1th decoding module are used as the fully supervised segmentation features.

[0078] Bilinear downsampling is performed on the point-supervised segmentation features and fully supervised segmentation features to make their spatial dimensions consistent with those of the classification features. Then, feature encoding and matrix transformation are performed on the point-supervised classification features, fully supervised classification features, and the downsampled fully supervised and point-supervised segmentation features, respectively, to obtain the corresponding query vectors. Key vector Value vector With the target vector ;

[0079] Based on query vector Key vector AND value vector Calculate and obtain the reference vector , represented as: ;

[0080] Calculate the reference vector With the target vector The cosine similarity between the two networks is used as the similarity loss to train the student network.

[0081] This application takes a global perspective of hybrid supervised networks, considering the differences in training and supervision information between point-labeled images and fully labeled images. It embeds a cross-supervised attention module and a pseudo-label deep supervision module into the student network. The cross-supervised attention module can improve the feature association between fully supervised samples and point-supervised samples, realize the feature alignment ability between fully supervised features and point-supervised features, further improve the supervision ability of point-supervised samples, and alleviate the problem of uneven convergence speed between point-supervised branches and fully supervised branches. The pseudo-label deep supervision module enhances model stability, improves model segmentation performance, and improves the accuracy of predicted lesion boundaries.

[0082] Specifically, this continues until the trained student network meets a preset cutoff condition, including the convergence of the total loss function of the trained student network; the construction of the total loss function includes:

[0083] Based on the cross-entropy loss function between the ground truth and predicted labels of the fully labeled image, a fully supervised loss is constructed, expressed as: ;

[0084] The first loss With the second loss Add them together to obtain the point monitoring loss. , represented as: ;

[0085] Calculate the reference vector With the target vector The cosine similarity between them is used as the similarity loss. , represented as: ;

[0086] A consistency loss is constructed based on the mean squared error of the predicted labels for point-annotated images by the teacher network and the student network. , represented as ;

[0087] The total loss function is obtained by weighted summation of the fully supervised loss, point-supervised loss, similarity loss, and consistency loss, and is expressed as: ;

[0088] in, and They represent the first The real labels on Zhang Quan's labeled images and the preset labels on the student network. , This represents the total number of data points in the fully annotated image set; This represents the total logarithm of the reference vector and the target vector. Indicates the first A vector, Represents the L2 norm; Indicates the teacher network's... Predicted labels for point-labeled images; Indicates mean square error; and These represent the similarity loss weight and the consistency loss weight, respectively.

[0089] In this embodiment, the auxiliary optimization network updates the standard cross-entropy loss between the ground truth labels and the predicted labels based on the fully labeled image; the standard cross-entropy loss between the ground truth labels and the predicted labels... , represented as: ;in, Represents the standard cross-entropy loss. and They represent the first Zhang Quan's labeled image's true label and auxiliary optimization network for the first Predicted labels for images annotated by Zhang Quan; , This represents the total number of data points in a fully labeled weak dataset.

[0090] Based on the above embodiments, in this embodiment of the invention, five encoding modules and three corresponding decoding modules are set for retinal OCT image segmentation; refer to Figure 7 The diagram shows the structure of a hybrid supervised network model. This model comprises three main branches: a student network, a teacher network, and an auxiliary optimization network (Aux). All three branches employ the same segmentation network structure: Local Feature Enhancement Network (LFENet). The student network guides the teacher network parameter updates using an exponential moving average (EMA) strategy, while the auxiliary optimization network guides the teacher network parameter updates through parameter copying.

[0091] Specifically, the parameter decoupling optimization training strategy in this embodiment involves the following steps: In regular training rounds, the student network performs forward propagation and gradient backpropagation on fully labeled and point-labeled images to update its parameters, and updates the teacher model's parameters using exponential moving average (EMA). In specific training rounds... Next, the current teacher network parameters are copied to the auxiliary optimization network, which then performs forward and backward propagation using the fully labeled image. The updated auxiliary optimization network parameters are then copied back to the teacher network, completing the supervised fine-tuning of the teacher network. During this round, the student network pauses parameter passing to the teacher model via EMA to avoid interfering with the teacher network's independent updates. It is an integer multiple of the update frequency m.

[0092] In the training process, the point-labeled images are input into the teacher network after random weak augmentation and into the student network after random strong augmentation. Meanwhile, the point-labeled images are off-line generated with initial pseudo-labels by a superpixel pseudo-label generation module (SLG). Then, combining the prediction results of the point-labeled images by the teacher network and the initial pseudo-labels, the corrected pseudo-labels and label weight maps are output by a confidence guided label correction module (CL²R). These corrected supervision information are used to supervise the student network at the pixel level by a pseudo-label deep supervision module (PDS). In addition, the point-labeled images themselves also provide point-level supervision signals for the student network. Meanwhile, consistency loss is introduced between the predictions of the student network and the teacher network for the point-labeled images. For the fully-labeled images, the randomly weak augmented images are input into the auxiliary optimization network, and the randomly strong augmented images are input into the student network, which are supervised at the pixel level. On this basis, the fully-supervised classification features and point-supervised classification features, fully-supervised segmentation features and weakly-supervised segmentation features are extracted from the student network, and cross-attention loss is calculated by a cross-supervision cross-attention module (CSCA) to further optimize the segmentation performance of the student network on the point-labeled images.

[0093] In the testing process, the OCT images are input into the trained student network to obtain the pixel-level segmentation results.

[0094] Specifically, the local feature enhancement segmentation network (LFENet) in the embodiment is used to improve the similar feature aggregation capability, and the network structure is based on the U-Net architecture with VGG16 as the encoder, including 5 encoding modules, 1 neighborhood feature enhancement module (NFE), 1 multi-scale differential feature fusion module (MDF²) and 4 decoding modules. Each encoding module is composed of multiple 3x3 convolution layers and 1 2x2 max pooling layer, and ReLU activation function is used after the convolution layer to enhance the nonlinear feature expression capability. The output features of each encoding module are 、 、 、 and , Through the NFE module, the feature after deep feature aggregation is obtained . For the feature maps 、 、 、 、 , a convolution layer is used respectively to unify the channel number, and the channel transformed features 、 、 、 and wherein the convolutional layer is composed of a convolution with a kernel size of 3x3, batch normalization and an activation function ReLU. 、 、 、 and is input into the MDF2 module, the MDF2 module connects the encoder and the decoder, performs inter-feature difference operation on the features of different stages, and aggregates the difference features of different scales of each stage to obtain the fusion features 、 、 、 .

[0095] Finally, through four decoding modules, the decoding features obtained in the last stage are added to the fusion features of this stage after upsampling (wherein the fourth stage decoding module takes the decoding features of the last stage as the last stage decoding features), and then the features are adjusted using convolution, and the decoding features 、 、 、 , are output in turn, and finally a segmentation result image with a channel number of +1 is output through a convolutional layer, each channel representing the probability of a pixel belonging to a lesion class and a background area,

[0096] wherein the NFE module extracts the neighborhood affinity features of the image, and updates the input features by weighting these features, finally enhancing the local feature representation of the image. The generation process of the neighborhood affinity features is shown in Figure 3 . First, 8 3x3 convolution kernels are initialized, each of which corresponds to up, down, left, right and diagonal directions. Only the positions adjacent to the center pixel in the specified direction in these convolution kernels are fixed as 1, and the rest are fixed as 0, which means that each convolution kernel only focuses on the neighborhood pixels in the specified direction adjacent to the center point. Then, the eight-neighborhood convolution kernel is applied to the multi-scale dilated convolution with dilation rates of 1, 3 and 5 to perform convolution calculation on the input features to obtain the neighborhood affinity feature , p represents the neighborhood affinity feature serial number under different direction and different dilation rate combinations, p=1~P, P is the total number of neighborhood affinity features, and in the present application the value of P is 24. Then the similarity between the target neighborhood center and each neighborhood affinity feature is calculated, and the calculation formula is as follows:

[0097] ;

[0098] Finally, the neighborhood affinity features are weighted and summed with the similarity as the weight to obtain the updated feature map ​The calculation formula is as follows: ;

[0099] The MDF² module incorporates multi-scale features from adjacent stages. , , , and Processing is performed through multiple Differential Computation Units (DCUs). Connect a DCU; Connect two DCUs; Connect three DCUs; Four DCUs are connected. Each DCU is used to extract cross-scale difference features between the corresponding features at the current scale and the previous scale. The calculation process is as follows:

[0100] , ;

[0101] , ;

[0102] in Let j be the j-th differential feature of the s-th stage, and DCU represent the differential computation operation.

[0103] Finally, the difference features from each stage are summarized using summation and convolution operations to obtain four fused features. The calculation formula is as follows:

[0104] ;

[0105] in, It is the first The fusion features of the stages, where Conv represents the convolution operation.

[0106] A detailed schematic diagram of the DCU unit is shown below. Figure 4 As shown, the calculation formula is as follows:

[0107] , ;

[0108] in, , These are two features of the input difference computation unit, where Conv1 represents the convolution with the all-one weighted filter. GN represents the size of the filter kernel, GN indicates group normalization, and GELU is the Gaussian error linear activation function.

[0109] Specifically, the superpixel-guided pseudo-label generation module (SLG): The process of generating superpixel pseudo-labels and label weight maps in the SLG is as follows: Figure 5 As shown, the generation process of superpixel pseudo-labels can be divided into three steps: First, the OCT image is divided into small superpixel blocks using a superpixel segmentation algorithm; then, based on the segmented superpixel blocks, label propagation is performed on the point labels. If a point label exists within a certain superpixel block, then all pixel labels in that superpixel block will be assigned the corresponding category, thus obtaining the initial pseudo-labels; finally, the histograms of each superpixel block are iteratively updated based on their similarity to obtain the final pseudo-labels. The formula for calculating the similarity between histograms is as follows:

[0110] ;

[0111] in, , Representing superpixel blocks , Histogram, Represents pixel value, , Representing superpixel blocks , The histogram at pixel values The value of .

[0112] If the similarity between two superpixel blocks exceeds a set threshold Then, the labels of the already labeled superpixel blocks are propagated to neighboring superpixel blocks, and the label propagation continues iteratively until all similarities are below the threshold. After label propagation, the final superpixel pseudo-labels are obtained.

[0113] Tag weight graph The weight map, with the same size as the pseudo-label image, represents the level of trust in each pixel location of the pseudo-label. Its value ranges from 0 to 1. During initialization, non-zero superpixel block locations in the initial superpixel pseudo-label are assigned a weight of 1, while the weights of all other pixel locations are set to 0.5. After each pseudo-label iteration update, the label weight map is also reassigned. The specific calculation process is as follows: For each superpixel block relabeled during the similarity iteration process, the minimum distance between its centroid and the centroid of the initially labeled superpixel block is calculated. . The calculation formula is as follows: ;in, and These are the distance bias parameter and the distance weight parameter, respectively.

[0114] Specifically, the Confidence-Guided Pseudo-Label Correction Module (CL²R): The CL²R module utilizes a confidence learning method to correct pseudo-labels based on the teacher model's predicted labels. and pseudo labels The joint distribution matrix identifies the mislabeled in the pseudo labels. For the identified mislabeled pixels, the label is corrected to the predicted label , and the label weight is corrected to 1; this process is iterated with each training.

[0115] Specifically, the pseudo label deep supervision module (PDS) downsamples the corrected pseudo label and its corresponding weight map to make their sizes consistent with the resolution of the decoding feature D4, and outputs a deep prediction label through convolution and softmax of the decoding feature D4. Based on the downsampled pseudo label and the deep prediction label, and weighted by the downsampled weight map, a deep supervision loss function is calculated for network optimization.

[0116] Specifically, the cross-supervision cross-attention module uses the semantic guidance of the cross-supervision branch to improve the feature learning ability of the point-labeled image by using the feature distribution of the fully-labeled image, thereby making up for the lack of label sparsity in the point-labeled image. In a batch of training, the last stage output feature of the encoder of each point-labeled image is taken as the point-supervised classification feature ; the last stage output feature of the decoder of each point-labeled image is taken as the point-supervised segmentation feature ; the last stage output feature of the encoder of each fully-labeled image is taken as the fully-supervised classification feature ; the last stage output feature of the decoder of each fully-labeled image is taken as the fully-supervised segmentation feature . The specific structure of the CSCA module is shown in Figure 6 ; first, the features and are bilinearly downsampled to make their spatial sizes consistent with and ; then, the four groups of features are sent into the feature encoding module for processing, which includes 1x1 convolution, RELU function, 1x1 convolution and matrix deformation operation in sequence. After encoding, four vectors and are generated. The reference vector is calculated as follows: ; finally, the cosine similarity loss between and is calculated for network optimization.

[0117] Based on the above embodiment, the overall loss function of the student network includes four parts: fully-supervised loss , point-supervised loss , similarity loss and consistency loss .

[0118] ①Full-supervised loss is the standard cross-entropy loss function, which is calculated on the input full-annotation images, denoted as: ; A is the number of full-annotation images. respectively represent the prediction result and the label of the student network on the i-th full-annotation image.

[0119] ②Point-supervised loss is divided into two parts, which are calculated on the input point-annotation images.

[0120] The first part is the partial cross-entropy loss function between the point label and the decoder output prediction , the formula is: ; B is the number of point-annotation images. respectively represent the prediction result and the point label of the student network on the i-th point-annotation image.

[0121] The other part is the multi-stage loss function introduced in different stages of the decoder with pseudo-labels and label weight maps, the formula is: ; respectively represent the output prediction, the down-sampled pseudo-label and the down-sampled pseudo-label weight map of the decoder on the i-th point-annotation image. respectively represent the prediction result, the pseudo-label and the pseudo-label weight map of the student network on the i-th point-annotation image.

[0122] is the cross-entropy loss function based on pixel weighting, denoted as:

[0123] ;

[0124] wherein, is the total number of pixels in the image, and C is the number of lesion categories. respectively are the prediction result, the label and the label weight map. is the value of the i-th pixel in the label weight map, is the label of whether the i-th pixel belongs to the c-th category, is the probability of the i-th pixel being predicted as the c-th category.

[0125] The final point-supervised loss is calculated as follows: ;

[0126] ③Similarity loss is the vector pair obtained by the cross-supervised cross-attention module​​​​​ Cosine similarity loss function between The specific calculation formula is as follows: ; L2 norm, the superscript l represents the lth vector, is the total number of vector pairs.

[0127] ④Consistency loss is used to measure the difference between the prediction outputs of the teacher network and the student network on the same point-labeled image, and the specific formula is: ; is the prediction result of the teacher network on the lth point-labeled image. is the prediction result of the student network on the lth point-labeled image. is the prediction result of the student network on the lth point-labeled image. MSE represents the mean square error.

[0128] ⑤The total loss function of the student network is represented as: ; respectively, the similarity loss weight and the consistency loss weight.

[0129] Specifically, the auxiliary optimization network loss function uses the standard cross-entropy loss of the full-supervised image with weak data enhancement, and is represented as: ; is the prediction result of the auxiliary optimization network on the lth full-labeled image.

[0130] Based on the above embodiment, in the embodiment of the present application, the segmentation effect of the retinal lesion segmentation method based on mixed supervision learning provided by the present application is verified, which specifically includes:

[0131] ①Data acquisition and preprocessing;

[0132] ​The dataset used in this experiment is the public dataset provided in the MICCA 2017 RETOUCH challenge, which contains 70 three-dimensional retinal OCT data collected by three different suppliers of SD-OCT equipment. The Cirrus equipment collected 24 three-dimensional data with a resolution of 1024x512x128, the Spectralis equipment collected 24 three-dimensional data with a resolution of 496x512x49, and the Topcon equipment collected 22 three-dimensional data with a resolution of 885x512x128 (T-2000 model) or 650x512x128 (T-1000 model). The dataset contains three different lesion types, namely intraretinal fluid (IRF), subretinal fluid (SRF) and pigment epithelial detachment (PED). This experiment is based on three-dimensional retinal OCT data, and the data is divided into three groups for three-fold cross-validation. The balance between the area of the three types of lesions and the collection equipment is considered during the division to ensure the fairness and reliability of the experimental results. In addition, three-dimensional retinal OCT data is used as the basic unit, and a certain proportion of samples in each fold are randomly selected as fully labeled images, providing pixel-level labeling information. The remaining samples are used as point-labeled images, and the point labels are set in combination with the connected domain and lesion area. The specific setting strategy is as follows: first, according to the area of each category, select 0.05% of the lesion area as the number of labeled points, and second, label at least one point in each connected region of the category. This labeling strategy is more comprehensive, ensuring that small lesion areas are not missed and conforming to the labeling habits of doctors when dealing with larger lesions.

[0133] In addition, since the area of the retinal lesion region in the OCT image is much smaller than the background region, and the lesions are distributed inside the retina, this embodiment takes the retina and choroid region as the region of interest, detects the internal limiting membrane (ILM) and choroid-scleral interface (CSI) as the upper and lower boundaries of the region of interest, and crops the image according to the region of interest to reduce the interference of the background on the subsequent processing. The specific implementation method is as follows: first, the Otsu method is used to quickly segment the approximate retinal region, and then the Savitsky-Golay filter is used to smooth the ILM and CSI curves. In the case of severe PED, the CSI segmentation error may occur, and the convex hull fitting strategy is used to repair the erroneous CSI segmentation. Finally, from the detected ILM and CSI curves, the points with the smallest and largest vertical coordinates are found as the upper and lower boundaries of the cropped region, and the ROI is limited to the region between ILM and CSI. Before inputting the network, the cropped image and the segmentation label are downsampled to process them into 256x256 size pictures.

[0134] ②Model training and testing;

[0135] The network was trained in an end-to-end manner, with a batch size of 8 containing 4 fully labeled images and 4 weakly labeled images during training, validation, and testing. Considering the GPU memory cost and training time cost, the OCT images and manual segmentation label images were down-sampled to 256x256 pixels before input; the cosine annealing learning rate algorithm was applied to optimize the network, and the learning rate lr was calculated as follows:

[0136] ;

[0137] where the base learning rate was set to 0.01, t was the current iteration number, was the maximum iteration number . The model was based on the public platform PyTorch and a GeForce RTX 3090 GPU with 24 GB of memory. During network training, after each iteration, the model was used to validate the validation set data, and the model parameters with the highest average Dice coefficient in the validation set were retained as the final test network model parameters. Set the superpixel similarity threshold = 0.6, the parameter in the label weight map calculation was set to 1, and the parameter was set to 0.05.

[0138] In the loss function, the similarity loss weight was set to 0.1, and the consistency loss weight was set as a function of the iteration number, as follows:

[0139] ;

[0140] where is the maximum value of the weight, which was set to 0.1; the update frequency m in the parameter decoupling optimization training strategy was selected as 20 rounds.

[0141] ③Analysis of experimental results;

[0142] To evaluate the segmentation results of IRF, SRF, and PED in OCT images, four evaluation indicators were used: Dice coefficient (DSC, Dice Similarity Coefficient), Intersection of Union (IoU), Sensitivity (Sen), and Specificity (Spe). Their definitions are as follows:

[0143] ; ; ; ;

[0144] where TP, TN, FP and FN are true positive, true negative, false positive and false negative of the segmentation respectively; the indicators of each class are calculated separately, and the total average is calculated.

[0145] All ablation experiments in this section are performed under the condition that the proportion of full annotation images and point annotation images is 20%, 80% respectively.

[0146] The results of the ablation experiment of adding LFENet segmentation network are shown in Table 1. As shown in the results of Table 1, compared with the traditional Unet network, the local feature enhancement network added with neighborhood feature enhancement module (NFE) and multi-scale differential feature fusion module (MDF2) achieves better results in IRF, SRF, PED single-class DSC, average DSC and average intersection over union, among which the average DSC and average intersection over union are improved by 0.91%, 1.07% respectively. This shows the advantage of using local feature enhancement network LFENet as the basic segmentation network in the mixed supervision task.

[0147] Table 1 Ablation experiment on adding LFENet segmentation network (%)

[0148]

[0149] The ablation experiment results of adding the parameter decoupling optimization strategy PDOS, the cross attention module CSCA and the hybrid domain deep supervision module PDS are shown in Table 2. Compared with the baseline network, the Dice coefficient is improved by 0.43% by applying the PDOS module, indicating that adding the parameter decoupling optimization strategy in the network can make full use of the pixel-level label information and to some extent alleviate the strong dependence between the teacher network and the student network, effectively improving the performance of the average teacher network in retinal lesion segmentation; the embedding of the CSCA module improves the Dice coefficient by 0.38%, because the addition of the cross attention module in the network can improve the feature association between the fully supervised samples and the weakly supervised samples, further improving the supervision ability of the weakly supervised samples. The addition of the PDS module in the network improves the Dice coefficient by 0.15%, because the addition of the pseudo-label deep supervision module PDS in the network can alleviate the imbalance problem of the convergence speed of the weakly supervised branch and the fully supervised branch, enhance the stability of the model, and improve the segmentation performance. When the PDOS, CSCA and PDS modules are combined and embedded in the baseline network, the Dice coefficient is improved compared with the embedding of each module alone, and when the PDOS, CSCA and PDS modules are used at the same time, the best segmentation result is achieved, which shows that the combination of the three modules can comprehensively exert the advantages of each module, among which the PDOS optimizes the parameter decoupling, the CSCA enhances the feature association, and the PDS improves the stability through deep supervision, so that the performance of the network reaches a balanced and stable optimal state, thereby achieving the best segmentation effect.

[0150] Table 2 Ablation experiment of different modules (%)

[0151]

[0152] Table 3 shows the results of the hybrid supervision network of the present application and the average teacher network (MT), the confidence learning assisted by the average teacher network (MTCL), the multi-label deep supervision model (MLDS) and the superpixel confidence learning point annotation network (SCLGPA-Nets) four kinds of models. The results of the full supervision annotation ratio of 20%, 30% and 50% are compared respectively. Among them, FD represents the proportion of fully supervised images, and PD represents the proportion of point supervised images. As shown in Table 3, the hybrid supervision network proposed in the present application achieves the best indicators of average DSC and average intersection over union IoU of three kinds of lesions under three different full supervision annotation ratios.

[0153] Table 3 Segmentation performance of different segmentation networks under different point annotation and full annotation image ratios (%)

[0154]

[0155] Reference Figure 8The figure shows the segmentation result comparison chart of the method of the application and other methods. Figure 8 The first row represents the retinal OCT image, the second row represents the gold standard, the third row represents the segmentation result chart of the MT model, the fourth row represents the segmentation result chart of the MTCL model, the fifth row represents the segmentation result chart of the MLDS model, the sixth row represents the segmentation result chart of the SCLGPANet model, and the seventh row represents the segmentation result chart of the application. The method of the application overcomes the difficulties of retinal lesions in limited supervised segmentation tasks, such as complex and variable lesion morphology, large or small lesion area, not obvious difference between lesions and background, and unclear lesion boundary, and provides a feasible solution for developing a low-annotation-cost and high-performance retinal lesion segmentation model; and compared with other hybrid supervised segmentation networks, the application shows lower missegmentation and missed segmentation rates. In terms of the recognition ability of micro-lesion areas, the method of the embodiment is also superior to other networks, and the overall segmentation result is closer to the manual segmentation gold standard.

[0156] The application combines the distribution characteristics of retinal lesions, and performs retinal lesion segmentation training of a hybrid supervised network based on limited full-annotation images and more point-annotation images. The network takes an average teacher model as a basic framework, and a new parameter decoupling strategy is used to optimize the teacher model on this basis to relieve the tight coupling problem between the teacher model and the student model. The segmentation network adopts a local feature enhancement network based on the U-Net architecture with VGG16 as the encoder, and embeds a neighborhood feature enhancement module and a multi-scale differential feature fusion module to fully utilize the supervision information provided by sparse annotation, improve the local feature aggregation ability, and improve the segmentation precision. In order to expand the supervision information and fully utilize the sparse annotation information provided by the point annotation, a new superpixel pseudo-label generation and correction method is proposed to expand the sparse point annotation into superpixel pseudo-labels, and assign corresponding weights to each pseudo-label. Meanwhile, confidence learning is used to identify errors in the pseudo-labels and further refine the pseudo-labels to expand the supervision information and reduce the interference of noise labels on model training. In addition, considering the training and supervision information difference between the point annotation image and the full annotation image, a cross-supervision cross-attention module and a pseudo-label deep supervision module are embedded in the student network to realize the feature alignment ability between full supervision features and weak supervision features, and to relieve the imbalance problem of convergence speed. The implementation of the method can effectively relieve the problems of medical image annotation depending on clinical experience, long time consumption, and high cost, and provides a feasible solution for developing a low-annotation-cost and high-performance retinal lesion segmentation model.

[0157] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0158] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0159] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0161] Obviously, the above-described embodiments are only examples for clarity and are not limiting on the implementation. Other different forms or variations of the above-described embodiments can be made on the basis of the above description for those skilled in the art. Here, it is not necessary or possible to exhaust all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for retinal lesion segmentation based on hybrid supervised learning, characterized in that, Applied to hybrid supervised network models that include teacher networks, student networks, and auxiliary optimization networks with the same segmentation network structure, including: Obtain the set of point-annotated images and the set of fully annotated images of retinal OCT images; The point-label set and the fully labeled set are input into the student network, the fully labeled set is input into the auxiliary optimization network, and the point-label set is input into the teacher network. The hybrid supervised network model is trained under hybrid supervision, following a predetermined number of regular training intervals followed by one specific training epoch. This includes: In the regular training rounds, the student network is made to perform forward propagation and gradient backpropagation to update the parameters of the student network model, and the teacher network model is updated by exponential moving average. In a specific training round, the parameters of the current teacher network model, which have been updated a preset number of times, are copied to the auxiliary optimization network. The auxiliary optimization network then performs forward and backward propagation to obtain the updated parameters of the auxiliary optimization network, which are then copied back to the teacher network model. The trained student network is obtained once the preset cutoff condition is met. The trained student network is used as the target segmentation model to segment the OCT image to be segmented and obtain the segmentation results of retinal lesions. Among them, the segmentation network structure of the teacher network, student network, and auxiliary optimization network with the same segmentation network structure is a local feature enhancement segmentation network; the local feature enhancement segmentation network is a U-Net architecture with VGG16 as the encoder; inputting the OCT image into the local feature enhancement segmentation network includes: The OCT image is input into the encoder and passed through N layers of encoding modules connected in series along the forward propagation direction to obtain N encoded features. The Nth encoded feature is input into the neighborhood feature enhancement module to perform deep feature aggregation and obtain the aggregated encoded feature. Each encoded feature and the aggregated encoded feature are passed through a convolutional layer for channel transformation to obtain the corresponding N transformed features; N transformed features are input into the multi-scale differential feature fusion module for feature difference operations to obtain N-1 fused features. The s-th transformed feature is sequentially processed through Ns differential calculation units, and the output of each unit is obtained, including the s-th transformed feature. The output of the first difference calculation unit , is represented as: , ; the s-th transformation feature The output of each differential unit After the first Output of each difference calculation unit , is represented as: , ; For each transformation feature, the outputs of the Ns difference calculation units are summed and then convolved to obtain the s-th fused feature corresponding to the s-th transformation feature, expressed as: ;in, This indicates a difference calculation operation. ; Represents the Gaussian error linear activation function. Representation group standardization, This represents a convolution with a kernel whose values ​​are all 1s. and Two features representing the input difference computation unit, Indicates the size of the filter kernel; The N-1 fused features are input into N-1 cascaded decoding modules respectively. The modules are then combined with the upsampled features of the decoded features output by the previous decoding module to perform decoding and obtain N-1 decoded features. The N-1th decoding module combines the Nth transform feature for decoding. The decoded features output by the first decoding module are convolved to obtain the segmentation result image.

2. The retinal lesion segmentation method based on hybrid supervised learning according to claim 1, characterized in that, After obtaining the set of annotated images and the set of fully annotated images of the retinal OCT images, the following are included: Random weak enhancement and random strong enhancement are performed on the point-annotated images and fully-annotated images in the point-annotated image set and the fully-annotated image set, respectively, to construct point-annotated weak data set, point-annotated strong data set, fully-annotated weak data set and fully-annotated strong data set; The strongly labeled point dataset and the strongly labeled full dataset are input into the student network, the weakly labeled full dataset is input into the auxiliary optimization network, and the weakly labeled point dataset is input into the teacher network.

3. The retinal lesion segmentation method based on hybrid supervised learning according to claim 1, characterized in that, The Nth encoded feature is input into the neighborhood feature enhancement module for deep feature aggregation to obtain aggregated encoded features, including: The Nth encoded feature is input into the neighborhood feature enhancement module, and eight 3×3 neighborhood convolution kernels are combined with M dilated convolutions of different scales to obtain 8M convolution methods. The Nth encoded feature is then convolved to obtain 8M neighborhood affinity features. Calculate the similarity between each neighborhood affinity feature and the Nth encoded feature, and use it as the weight of each neighborhood affinity feature; The neighborhood affinity features are weighted and summed based on their respective weights to obtain the aggregated encoding features. In each neighborhood convolution kernel, only the position in a specified direction adjacent to the center pixel is fixed as 1, and the rest are fixed as 0. The specified directions include above, below, left, and right of the center pixel, as well as the upper left, upper right, lower left, and lower right directions in the diagonal direction.

4. The retinal lesion segmentation method based on hybrid supervised learning according to claim 1, characterized in that, Hybrid supervised training of the hybrid supervised network model includes generating pseudo-labels and label weight maps for the point-annotated images using a superpixel pseudo-label generation module and a confidence-guided label correction module. The point-annotated image is input into the superpixel pseudo-label generation module. The superpixel segmentation algorithm is used to divide the point-annotated image into multiple superpixel blocks. The pixel labels of the superpixel blocks with point labels in the region are assigned the category corresponding to the point labels in the region, and the initial pseudo-labels of the superpixel blocks with point labels in the region are obtained. Assign a label weight of 1 to the superpixel blocks within the region that have point labels, and assign a label weight of 0.5 to the remaining superpixel blocks; Based on the similarity between the histograms of any two superpixel blocks, the initial pseudo-labels of the superpixel blocks with existing initial pseudo-labels are propagated to the superpixel blocks with similarity exceeding a preset threshold until there are no two superpixel blocks with similarity exceeding the preset threshold, and the initial pseudo-labels of each superpixel block are obtained. For a superpixel block that has been propagated with the initial pseudo-label, calculate the minimum distance between the centroid of the superpixel block that propagated the initial pseudo-label and the centroid of the superpixel block, and construct the label weight of the superpixel block that has been propagated with the initial pseudo-label; obtain the label weight of each superpixel block and generate the initial label weight map. Using the confidence learning method, a joint distribution matrix of the predicted labels and initial pseudo-labels output by the teacher network is constructed. The erroneously labeled pixels in the initial pseudo-labels are identified, and the initial pseudo-labels of the erroneously labeled pixels are corrected to the corresponding predicted labels. The label weights are corrected to 1, resulting in a target pseudo-label and target label weight map.

5. The retinal lesion segmentation method based on hybrid supervised learning according to claim 4, characterized in that, After obtaining the target pseudo-label and target label weight graph, the student network is subjected to pixel-level supervision using the pseudo-label deep supervision module, including: Downsampling is performed on the target pseudo-label and target label weight maps respectively, so that the resolution of the downsampled target pseudo-label and target label weight maps is similar to that of the first-order target pseudo-label in the student network. The prediction outputs of all decoding modules are the same, and the weight map of downsampled pseudo-labels and downsampled labels is obtained. For the first in the student network The output of each decoding module is convolved and activated to obtain deep prediction labels; The first loss is constructed based on the standard cross-entropy loss between the ground and predicted labels of the labeled image. , is represented as: ; Based on the deep predicted labels of the point-annotated image from the (N-1)th decoding module, and the downsampled pseudo-labels and downsampled label weight map of the point-annotated image, a second loss is constructed. , is represented as: ; The first loss and the second loss are added together to obtain the point-supervised loss, which is then used to train the student network. in, This represents the standard cross-entropy loss. and They represent the first The true labels of the labeled images and the predicted labels of the student network; , This represents the total number of data points in the set of labeled images; The pixel-weighted cross-entropy loss function is expressed as follows: ; , and This represents a graph showing predicted labels, target pseudo-labels, and target label weights. , This represents the total number of pixels in the image; , Indicates the number of lesion categories; In the target label weight graph, the first... The label value of each pixel. Indicates the first Does the pixel belong to the ? Pseudo-tags for classes Indicates the first The pixel belongs to the first The predicted probability of a class; Indicates the first Pseudo-labels for images with point annotations. express The corresponding target label weight graph; Indicates the student network's response to the first When making predictions on a labeled image, based on the first... The deep predicted labels obtained from the output of each decoding module; express The corresponding downsampling pseudo-label; express The corresponding downsampling label weight map.

6. The retinal lesion segmentation method based on hybrid supervised learning according to claim 5, characterized in that, Optimizing student networks using a cross-supervised cross-attention module includes: The output features of the point-labeled image after passing through the Nth encoding module are used as point-supervised classification features; the output features of the point-labeled image after passing through the N-1th decoding module are used as point-supervised segmentation features. The output features of the fully labeled image after passing through the Nth encoding module are used as the fully supervised classification features; the output features of the fully labeled image after passing through the N-1th decoding module are used as the fully supervised segmentation features. Bilinear downsampling is performed on the point-supervised segmentation features and fully supervised segmentation features to make their spatial dimensions consistent with those of the classification features. Then, feature encoding and matrix transformation are performed on the point-supervised classification features, fully supervised classification features, and the downsampled fully supervised segmentation features and point-supervised segmentation features, respectively, to obtain the corresponding query vectors. Key vector Value vector With the target vector ; Based on query vector Key vector AND value vector Calculate and obtain the reference vector , is represented as: ; Calculate the reference vector With the target vector The cosine similarity between the two networks is used as the similarity loss to train the student network.

7. The retinal lesion segmentation method based on hybrid supervised learning according to claim 6, characterized in that, The preset cutoff condition includes the convergence of the total loss function of the trained student network; the construction of the total loss function includes: Based on the cross-entropy loss function between the ground truth and predicted labels of the fully labeled image, a fully supervised loss is constructed, expressed as: ; The first loss With the second loss Add them together to obtain the point monitoring loss. , is represented as: ; Calculate the reference vector With the target vector The cosine similarity between them is used as the similarity loss. , is represented as: ; A consistency loss is constructed based on the mean squared error of the predicted labels for point-annotated images by the teacher network and the student network. , represented as ; The total loss function is obtained by weighted summation of the fully supervised loss, point-supervised loss, similarity loss, and consistency loss, and is expressed as: ; in, and They represent the first The real labels on Zhang Quan's labeled images and the preset labels on the student network. , This represents the total number of data points in the fully annotated image set; This represents the total logarithm of the reference vector and the target vector. Indicates the first A vector, Represents the L2 norm; Indicates the teacher network's... Predicted labels for point-labeled images; Indicates mean square error; and These represent the similarity loss weight and the consistency loss weight, respectively.

8. The retinal lesion segmentation method based on hybrid supervised learning according to claim 1, characterized in that, The auxiliary optimization network updates the standard cross-entropy loss between the ground truth labels and the predicted labels based on the fully labeled image; the standard cross-entropy loss between the ground truth labels and the predicted labels , is represented as: ； in, This represents the standard cross-entropy loss. and They represent the first Zhang Quan's labeled image's true label and auxiliary optimization network for the first Predicted labels for images annotated by Zhang Quan; , This represents the total number of data points in a fully labeled weak dataset.

Images