Subretinal Fluid Segmentation and Volume Calculation Method, System, Product and Terminal

Abstract

The present application provides a method, system, product and terminal for subretinal fluid segmentation and volume calculation. Based on a pre-constructed fundus lesion area prediction model, it automatically segments the fundus predicted lesion area image, and sums up the pixel points belonging to the subretinal fluid part in the image data of each fundus predicted lesion area to obtain the subretinal fluid volume data, which greatly shortens the film reading time of ophthalmologists, improves work efficiency. In addition, it can significantly improve the accuracy, stability and homogeneity of film reading, and can help clinicians achieve accurate and objective evaluation of central serous chorioretinopathy.

Description

Technical Field

[0001] This application relates to the field of computer vision, and particularly to a method, system, product and terminal for subretinal fluid segmentation and volume calculation. Background Art

[0002] Central serous chorioretinopathy (CSC) is a common fundus disease, especially with a relatively high incidence rate among men over 30 years old. Although acute CSC is generally considered a self-limiting disease with a good prognosis, chronic and recurrent CSC may lead to retinal pigment epithelial lesions and retinal atrophy, thereby causing irreversible vision loss. Research shows that approximately 12.8% of CSC patients progress to bilateral visual acuity below 0.1 within 10 years of follow-up, reaching the level of "legal blindness". The high incidence rate of CSC makes the accurate assessment of its condition particularly important to reduce the risk of blindness.

[0003] Optical coherence tomography (OCT) is a technique that uses near-infrared light as the incident light source and obtains reflection signals at various depths inside tissues through interference to form an image. Different retinal tissues have different light reflection characteristics. The tissue cross-sectional diagram presented by OCT generates a pseudo-color diagram or grayscale diagram according to the reflection signals of different intensities during the image processing process, which is called "optical biopsy". In the diagnosis and follow-up of CSC, OCT is widely used to detect subretinal fluid in the posterior pole of the retina. The qualitative and quantitative assessment of the fluid is of great significance for clinicians to formulate treatment strategies.

[0004] However, the current clinical assessment of CSC mainly relies on manual linear analysis, and this method has certain limitations ( Figure 1 ). On the other hand, the analysis software built into OCT cannot distinguish between the two structures of subretinal fluid and the detached area of the retina, but measures them together as a whole, and the total value obtained cannot accurately reflect the true situation of the fluid ( Figure 2 and 3 ). Therefore, both manual measurement and built-in software analysis are one-sided and ambiguous, lacking accurate volume quantification analysis of subretinal fluid in the lesion area, resulting in inaccurate assessment of the CSC condition. In addition, the process of reading fundus images is complex, requiring a large amount of time from specialist doctors, and there is also the influence of subjective factors. The judgment results of the same doctor at different times or between different doctors may be inconsistent.

[0005] Therefore, developing an automatic analysis method that can objectively, accurately and quickly calculate the volume of subretinal fluid has become an urgent need for the current clinical assessment of CSC. Summary of the Invention

[0006] In view of the deficiencies of the prior art described above, the purpose of this application is to provide a method, system, product, and terminal for segmenting and calculating the volume of subretinal fluid, which is used to solve the technical problems in the prior art such as the lack of quantitative analysis of the volume of subretinal fluid and the one-sidedness and ambiguity of manual measurement.

[0007] To achieve the above purpose and other related purposes, the first aspect of this application provides a method for segmenting and calculating the volume of subretinal fluid, including: obtaining real-time fundus image data of a patient; preprocessing the real-time fundus image data to obtain multiple real-time fundus slice image data; based on each of the real-time fundus slice image data and a pre-constructed fundus lesion area prediction model, obtaining the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data; calculating the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0008] In some embodiments of the first aspect of this application, the method for calculating the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes:

[0009] ;

[0010] where V represents the subretinal fluid volume data of the patient; represents the number of real-time fundus slice image data; represents the actual length of the real-time fundus slice image data; represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; represents the number of pixel points of the fundus predicted lesion area image data corresponding to the

[0011] In some embodiments of the first aspect of the present application, the construction method of the fundus lesion area prediction model includes: obtaining historical fundus image data of different patients, and preprocessing the historical fundus image data of different patients to construct a training data set; the training data set includes multiple historical fundus slice training image data; inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning, and at the same time inputting it into a UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model; based on a preset optimization algorithm, sequentially updating the parameters of the total loss function of branch task learning, the enhanced loss function, and the total loss function of main task learning, and iteratively training the initial fundus lesion area prediction model to obtain a finally converged fundus lesion area prediction model.

[0012] In some embodiments of the first aspect of the present application, the method of inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning, and at the same time inputting it into a UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model includes: inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning to obtain the first fundus predicted lesion area image data, and performing a softening operation on the preset main task learning label according to the first fundus predicted lesion area image data; calculating the total loss function of main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data; inputting the historical fundus slice training image data into a UNet network model for branch task learning to calculate the total loss function of branch task learning and the branch task learning label; calculating the enhanced loss function according to the branch task learning label and the softened main task learning label; updating the parameters of the UNet++ network model based on multi-semantic supervision and the UNet network model according to the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model.

[0013] In some embodiments of the first aspect of the present application, the method for inputting the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning to obtain the first fundus predicted lesion area image data includes: performing multiple downsampling operations on the historical fundus slice training image data to obtain the initial feature image data of subretinal fluid at different depth layers; performing different numbers of upsampling operations on the initial feature image data of subretinal fluid at different depth layers respectively to obtain the fused feature image data of subretinal fluid at different depth layers; based on the analogy attention enhancement module added to the UNet++ network model, performing feature enhancement and fusion operations on the initial feature image data of subretinal fluid at the last depth layer for the fused feature image data of subretinal fluid obtained after each upsampling operation respectively to generate the first subretinal fluid segmentation image data; based on the semantic constraint attention module added to the UNet++ network model, performing non-local semantic constraint operations on the initial feature image data of subretinal fluid at different depth layers for the fused feature image data of subretinal fluid obtained after the last upsampling operation respectively to generate the second subretinal fluid segmentation image data; generating the first fundus predicted lesion area image data according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

[0014] In some embodiments of the first aspect of the present application, the method for calculating the total loss function of main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data includes: calculating the output loss function of main task learning according to the preset cross-entropy loss function and the preset dice loss function; calculating the loss function of the analogy attention enhancement module and the loss function of the semantic constraint attention module according to the output loss function of main task learning; calculating the total loss function of main task learning according to the first fundus predicted lesion area image data, the output loss function of main task learning, the loss function of the analogy attention enhancement module, and the loss function of the semantic constraint attention module.

[0015] In some embodiments of the first aspect of the present application, the method for inputting the historical fundus slice training image data into the UNet network model for branch task learning, calculating the total loss function of branch task learning, and the label of branch task learning includes: inputting the historical fundus slice training image data into the UNet network model for branch task learning to obtain the level set function of the second fundus predicted lesion area image data; calculating the total loss function of branch task learning and the label of branch task learning according to the level set function of the second fundus predicted lesion area image data.

[0016] To achieve the above and other related objectives, the second aspect of the present application provides a subretinal fluid segmentation and volume calculation system, including: a data acquisition module for acquiring real-time fundus image data of a patient; a data preprocessing module for preprocessing the real-time fundus image data to obtain multiple real-time fundus slice image data; a lesion area prediction module for obtaining the predicted fundus lesion area image data corresponding to each of the real-time fundus slice image data based on each of the real-time fundus slice image data and a pre-constructed fundus lesion area prediction model; a volume calculation module for calculating the subretinal fluid volume data of the patient based on the predicted fundus lesion area image data corresponding to each of the real-time fundus slice image data.

[0017] To achieve the above and other related objectives, the third aspect of the present application provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer is enabled to implement the subretinal fluid segmentation and volume calculation method as described above.

[0018] To achieve the above and other related objectives, the fourth aspect of the present application provides an electronic terminal, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the subretinal fluid segmentation and volume calculation method as described above.

[0019] As described above, the subretinal fluid segmentation and volume calculation method, system, product, and terminal of the present application have the following beneficial effects:

[0020] (1) By introducing a level set enhancement and multiple semantic supervision mechanism, it is possible to finely segment the fluid area related to CSC.

[0021] (2) By combining the main task (segmentation) and the branch task (level set function learning), the model can simultaneously capture global semantic information and local detail information, further improving the segmentation accuracy.

[0022] (3) It realizes a fully automated process from image input to subretinal fluid volume calculation, without manual intervention, can quickly complete image segmentation and volume calculation, greatly shortening the doctor's film reading time. It can not only improve the accuracy, stability, and homogeneity of film reading, but also significantly improve the work efficiency of ophthalmologists, achieve a precise assessment of the CSC disease degree, and thus provide a more effective treatment plan for patients.

[0023] (4) By introducing a semantic constraint attention module (CAR module) and an analogical attention enhancement module (CAB module), the model shows highly stable performance on different data sets.

[0024] (5) Use a preset optimization algorithm to update the parameters of the total loss function of the branch task learning, the enhancement loss function, and the total loss function of the main task learning in sequence to ensure the rapid convergence of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It shows a sectional view of tissue imaged by OCT in the prior art, Figure 1 where A in [reference] shows a schematic diagram of manually linearly measuring the width of subretinal fluid, Figure 1 and B in [reference] shows a schematic diagram of manually linearly measuring the height of subretinal fluid.

[0026] Figure 2 It shows a schematic diagram of marking the inner limiting membrane and retinal pigment epithelium using the built-in analysis software of OCT in the prior art.

[0027] Figure 3 It shows a schematic diagram of using the built-in analysis software of OCT to display the total value of the thickness (volume) of the retina and subretinal fluid in different regions in the prior art, Figure 3 where A in [reference] is a schematic diagram of retinal partitioning, Figure 3 and B in [reference] is a schematic diagram of the total value of the thickness (volume) of the retina and subretinal fluid in different regions in the retinal partitioning shown in A.

[0028] Figure 4 It shows a schematic flowchart of the method for segmenting and calculating the volume of subretinal fluid in an embodiment of the present application.

[0029] Figure 5 It shows a schematic flowchart of constructing a fundus lesion area prediction model in an embodiment of the present application.

[0030] Figure 6 It shows historical fundus image data before preprocessing in an embodiment of the present application.

[0031] Figure 7 It shows historical fundus image data after preprocessing in an embodiment of the present application.

[0032] Figure 8 It shows a schematic flowchart of constructing an initial fundus lesion area prediction model in an embodiment of the present application.

[0033] Figure 9 It shows a schematic diagram of the backbone of the UNet++ network model based on multi-semantic supervision in an embodiment of the present application.

[0034] Figure 10 It shows a schematic diagram of the backbone of the UNet++ network model in an embodiment of the present application.

[0035] Figure 11It shows a schematic flow chart of the CAB module in an embodiment of the present application.

[0036] Figure 12 It shows a schematic flow chart of the CAR module in an embodiment of the present application.

[0037] Figure 13 It shows a schematic block diagram of a subretinal fluid segmentation and volume calculation system in an embodiment of the present application.

[0038] Figure 14 It shows a schematic structural diagram of an electronic terminal in an embodiment of the present application. Detailed implementation manners

[0039] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0040] Before further elaborating on the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are applicable to the following explanations:

[0041] <1> UNet: A classic convolutional neural network architecture designed specifically for medical image segmentation tasks. It is named after its efficient performance and unique U-shaped structure.

[0042] <2> UNet++: An improved version of UNet, aiming to solve some limitations of the original UNet in medical image segmentation. UNet++ enhances feature transfer and gradient flow by introducing dense skip connections and deep supervision, thereby improving segmentation accuracy.

[0043] <3> Relu function: Rectified linear unit, a commonly used activation function in deep learning.

[0044] <4> Softmax: A function that converts real values into a probability distribution, widely used in multi-classification tasks and attention mechanisms.

[0045] <5> sigmoid: A commonly used activation function that maps any real value to the range of (0, 1). Due to its smooth S-shaped curve and differentiability, the sigmoid function is widely used in machine learning and deep learning.

[0046] Currently, the clinical evaluation of CSC mainly relies on manual linear analysis, which has certain limitations. On the other hand, the analysis software built into OCT cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area. Instead, it measures the two together, and the total value obtained cannot accurately reflect the true situation of the fluid. Therefore, both manual measurement and built-in software analysis are one-sided and ambiguous, lacking accurate volume quantification analysis of subretinal fluid in the lesion area, resulting in inaccurate assessment of the CSC condition. In addition, the process of reading fundus images is complex, requiring a large amount of time from specialist doctors, and is affected by subjective factors. The results of image interpretation by the same doctor at different times or between different doctors may be inconsistent.

[0047] Figure 1 It shows a tissue section diagram of using OCT imaging in the prior art and a schematic diagram of manually linearly measuring the width and height of subretinal fluid. Figure 2 It shows a schematic diagram of using the analysis software built into OCT to mark the inner limiting membrane and retinal pigment epithelium in the prior art, which cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area. Figure 3 It shows a schematic diagram of using the analysis software built into OCT to display the total value of the thickness (volume) of the retina and subretinal fluid in different regions. Among them, Figure 3 retina thickness in it represents the thickness of the retina.

[0048] As Figure 1 shown, using the software built into OCT, manually measure the width and height (A and B) of subretinal fluid. Using the analysis software built into OCT, after the volume scan is completed, the inner limiting membrane and retinal pigment epithelium can be automatically marked. As Figure 2 shown, it cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area. Using the automatic zoning function of the analysis software built into OCT, as Figure 3 shown: Divide with circles centered at the fovea with diameters of 1, 3, and 6 mm, and then evenly divide them into four quadrants of superior, inferior, nasal, and temporal (a total of 9 blocks). According to the Figure 2 marking line, the software can automatically calculate the average thickness from the inner limiting membrane to the retinal pigment epithelium in each block, but cannot separately analyze the thickness of subretinal fluid and the retina in the detachment area.

[0049] Therefore, there is an urgent need for an automatic analysis method to objectively, accurately, and quickly calculate the volume of subretinal fluid, improve the accuracy, stability, and homogeneity of image reading, and while improving the clinical work efficiency of ophthalmologists for fundus diseases, achieve accurate assessment of the CSC condition.

[0050] To solve the above technical problems, the present application provides a method, system, product and terminal for subretinal fluid segmentation and volume calculation, which are used to solve the technical problems such as the lack of quantitative analysis of subretinal fluid volume and the one-sidedness and ambiguity of manual measurement in the prior art.

[0051] For the convenience of understanding the embodiments of the present application, first in combination with Figure 4 detailed description. Figure 4 FIG. shows a schematic flow chart of the method for subretinal fluid segmentation and volume calculation in an embodiment of the present invention. The method for subretinal fluid segmentation and volume calculation in this embodiment mainly includes the following steps:

[0052] S401: Obtain the real-time fundus image data of the patient.

[0053] In this embodiment, the real-time fundus image data of the patient includes optical coherence tomography (OCT) images obtained through a volume scanning mode. The OCT volume scanning mode can generate a series of continuous fundus cross-sectional images. According to the specific lesion range of the patient, adjust the scanning parameters, such as the scanning line length, scanning range and the number of cross-sections, to meet the diagnostic requirements.

[0054] S402: Preprocess the real-time fundus image data to obtain multiple real-time fundus slice image data.

[0055] In this embodiment, the preprocessing methods include but are not limited to: cropping preprocessing, image augmentation preprocessing, normalization preprocessing, regularization preprocessing, etc. First, perform cropping preprocessing on the real-time fundus image data to retain multiple real-time OCT fundus slice region images.

[0056] In this embodiment, the image augmentation preprocessing is to randomly magnify the cropped real-time OCT fundus slice region image by 1 to 1.2 times, then randomly crop it with the original size, and flip the image horizontally with a probability of 50% and vertically with a probability of 50%. The normalization preprocessing is to normalize the gray value of the real-time OCT fundus slice region image after image augmentation preprocessing from normalize to and calculate the mean and variance of the data gray value. The regularization preprocessing is to regularize the image using the calculated mean and variance.

[0057] S403: According to each of the real-time fundus slice image data, and based on a pre-constructed fundus lesion area prediction model, obtain the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0058] In this embodiment, as Figure 5As shown in the figure, it shows a schematic flowchart of constructing a fundus lesion area prediction model in an embodiment of the present invention. Among them, the construction method of the fundus lesion area prediction model includes:

[0059] S4031: Obtain the historical fundus image data of different patients, and preprocess the historical fundus image data of different patients to construct a training data set; the training data set includes multiple historical fundus slice training image data.

[0060] In this embodiment, as Figure 6 shown, it shows an OCT historical fundus slice area image before preprocessing in an embodiment of the present invention. As Figure 7 shown, it shows an OCT historical fundus slice area image after preprocessing in an embodiment of the present invention. Crop the historical fundus image data of different patients, and retain multiple OCT historical fundus slice area images. Perform image augmentation preprocessing, normalization preprocessing, and regularization preprocessing on a part of the cropped OCT historical fundus slice area images to construct a training data set. Perform normalization preprocessing and regularization preprocessing on the remaining cropped OCT historical fundus slice area images to construct a test data set and a validation data set.

[0061] In this embodiment, the method of the regularization preprocessing includes:

[0062] ; Formula (1)

[0063] Among them, represents the original pixel value of each image in the data set; represents the mean value of the gray values of all pixel points of all images in the data set; represents the image data after regularization; represents the variance of the gray values of all pixel points in all images in the data set.

[0064] S4032: Input the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning, and at the same time input it into the UNet network model for branch task learning, and then calculate the total loss function of the main task learning, the total loss function of the branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model.

[0065] In this embodiment, the main task learning is used for fundus lesion area segmentation. The branch task learning is used for fundus lesion area boundary detection.

[0066] In this embodiment, as Figure 8As shown, it shows a schematic flowchart of constructing an initial fundus lesion area prediction model in an embodiment of the present invention. Inputting the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning, and simultaneously inputting it into the UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function. The ways to construct the initial fundus lesion area prediction model include:

[0067] S4032a: Input the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning, obtain the first fundus predicted lesion area image data, and perform a softening operation on the preset main task learning label according to the first fundus predicted lesion area image data.

[0068] In this embodiment, as Figure 9 shown, it shows a schematic diagram of the skeleton of the UNet++ network model based on multi-semantic supervision in an embodiment of the present invention. As Figure 10 shown, it shows a schematic diagram of the skeleton of the UNet++ network model in an embodiment of the present invention. Figure 10 Backbone in it represents the model skeleton, Down-sampling represents the down-sampling operation, Up-sampling represents the up-sampling operation, Skip connection represents the skip connection, and Convolution represents the feature image. For main task learning, the UNet++ network model based on multi-semantic supervision is used. Input each historical fundus slice training image data in the training dataset into the UNet++ network model based on multi-semantic supervision for forward propagation to obtain a UNet++ network model based on multi-semantic supervision.

[0069] In this embodiment, the ways to input the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision to obtain the first fundus predicted lesion area image data include:

[0070] (1) Perform multiple down-sampling operations on the historical fundus slice training image data to obtain the initial feature image data of subretinal fluid at different depth layers.

[0071] In this embodiment, the UNet++ network model of the model skeleton acts on the input historical fundus slice training image data with a convolution module to obtain the initial feature image data of subretinal fluid at the first depth layer ( (), the convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Then repeat the convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Four downsampling operations are performed on the initial subretinal fluid feature image data of the first depth layer to obtain the initial subretinal fluid feature image data of different depth layers . Among them, represents the initial subretinal fluid feature image data of the first depth layer, represents the initial subretinal fluid feature image data of the second depth layer, represents the initial subretinal fluid feature image data of the third depth layer, represents the initial subretinal fluid feature image data of the fourth depth layer, represents the initial subretinal fluid feature image data of the fifth depth layer. The network is divided into five depth layers, and the resolution and number of channels of the images obtained in each depth layer are the same. The image resolution and number of channels obtained from the first depth layer to the fifth depth layer are , , , , .

[0072] In this embodiment, each downsampling operation is first a pooling operation with a kernel size of 2*2 and a stride of 2, followed by a convolutional module. The convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels of the next depth layer, and no bias, followed by batch normalization once and the Relu function once. Then repeat the convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels of the next depth layer, and no bias, followed by batch normalization once and the Relu function once.

[0073] (2) Upsampling operations are performed on the initial subretinal fluid feature image data of different depth layers for different numbers of times to obtain the fused subretinal fluid feature image data of different depth layers.

[0074] In this embodiment, upsampling operations are performed on the initial subretinal fluid feature image data of the second to fifth depth layers for different numbers of times to extract the subretinal fluid feature information of different depth layers to obtain the fused subretinal fluid feature image data of different depth layers. Specifically, perform 1 upsampling operation on to obtain . Perform 2 upsampling operations on to obtain , Perform three upsampling operations to obtain , , Perform four upsampling operations to obtain , , , . Among them, represents the first subretinal fluid fusion feature image data of the first depth layer, represents the second subretinal fluid fusion feature image data of the first depth layer, represents the third subretinal fluid fusion feature image data of the first depth layer, represents the fourth subretinal fluid fusion feature image data of the first depth layer. represents the first subretinal fluid fusion feature image data of the second depth layer, represents the second subretinal fluid fusion feature image data of the second depth layer, represents the third subretinal fluid fusion feature image data of the second depth layer. represents the first subretinal fluid fusion feature image data of the third depth layer, represents the second subretinal fluid fusion feature image data of the third depth layer. represents the first subretinal fluid fusion feature image data of the fourth depth layer.

[0075] In this embodiment, specifically, each upsampling operation first uses the bilinear interpolation method with an upsampling rate of 2 to double the resolution of the input image (height width number of channels) to obtain the interpolated image . Then, the interpolated image is concatenated with the image of the same resolution obtained in the same depth layer, and then a convolution module is applied. The convolution module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is half of the number of channels of the input image , without using bias, followed by one batch normalization and one application of the Relu function. Then, repeat a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is half of the number of channels of the input image , without using bias, followed by one batch normalization and one application of the Relu function. Finally, the output layer uses a convolution kernel with a kernel size of 1 and the number of channels equal to the number of classes to change the number of output channels to the number of classes, so as to obtain the subretinal fluid fusion feature image data of different depth layers through the upsampling operation.

[0076] (3) Based on the analog attention enhancement module added to the UNet++ network model, feature enhancement and fusion operations are respectively performed on the subretinal fluid fusion feature image data obtained after each upsampling operation on the initial feature image data of the subretinal fluid in the last depth layer to generate the first subretinal fluid segmentation image data.

[0077] In this embodiment, as Figure 11 shown, the flowchart of the CAB module in the embodiment of the present invention is shown. The working process of the analog attention enhancement module (CAB module) is described as follows:

[0078] First, category attention is calculated, and a normalization exponential mapping (softmax mapping) is performed on the third dimension of the rough segmentation map to obtain a probability map , where represents the height of the image, represents the width of the image, and is the number of categories. Then, a Mask Split operation (mask segmentation operation) is performed, that is, each channel of the probability map is separated to obtain spatial weights corresponding to each category, and then a Mul operation (weighted multiplication operation) is performed, that is, multiplying with the corresponding positions of another fine feature map to achieve the category attention mechanism, and feature maps are respectively obtained. The C feature maps are the feature maps such as Class1, Class2, Class3 shown in the figure. Class1 represents the first weighted fine feature map, Class2 represents the second weighted fine feature map, and Class3 represents the third weighted fine feature map, where is the number of channels. A concatenation operation (Concat) is performed on these feature maps to obtain a feature image with a dimension of . A convolution module is applied to the feature image for feature fusion. The convolution module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels being halved of the input channels, without using bias, followed by a batch normalization and a Relu function application. Then, it is repeated with a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels being 2 (the number of categories), without using bias, followed by a batch normalization and a Relu function application, to obtain an image with a size of . The obtained image is added to the rough segmentation map to obtain the output image .

[0079] In this embodiment, the initial characteristic image data of the subretinal fluid in the last depth layer represents the initial characteristic image data of the subretinal fluid in the fifth depth layer. , perform 4 upsampling operations on , perform Add a CAB module, whose function is to use rough segmentation to accelerate the transmission of semantic information to fine segmentation for feature enhancement and fusion operations. Exemplarily, take as an example. First, apply a convolutional kernel with a channel number of 2 and a kernel size of 1 to change the channel number to 2 channels, obtaining a rough feature map of the subretinal fluid region , perform upsampling on the rough feature map of the subretinal fluid region to double the resolution and obtain a high-resolution feature map of the fluid region . Then and are respectively used as and in the above steps and input into the CAB module, and output the first fine segmentation feature map of the subretinal fluid region , where serves as the rough segmentation map, and serves as the fine feature map.

[0080] In this embodiment, the method of adding a CAB module to the UNet++ network model is as follows. Since it is the first upsampling feature map, the CAB module is not used. For , first apply a convolutional layer with a kernel size of 1 to to obtain a rough feature map of the subretinal fluid region with the same resolution and 2 channels . Then enter the CAB module together with , output the first fine segmentation feature map of the subretinal fluid region , and then enter the next depth layer to obtain the second fine segmentation feature map of the subretinal fluid region . and are input into the CAB module, and finally obtain the first subretinal fluid segmentation image data , which is the final output of the CAB module.

[0081] (4) Based on the semantic constraint attention module added to the UNet++ network model, perform non-local semantic constraint operations on the initial characteristic image data of the subretinal fluid in different depth layers for the subretinal fluid fusion feature image data obtained after the last upsampling operation, respectively, to generate the second subretinal fluid segmentation image data.

[0082] In this embodiment, as Figure 12 shown, a schematic flowchart of the CAR module in an embodiment of the present invention is presented. The working process of the Semantic Constraint Attention Module (CAR module, i.e., the CAR module) is described as follows:

[0083] First, a normalized exponential mapping is performed on the third dimension (i.e., the dimension of the number of channels C) of the coarse segmentation map to obtain a probability map , that is, on an image with a resolution of HW, the i-th pixel corresponds to a probability vector , the j-th pixel corresponds to a probability vector , represents the probability vector of the HW-th pixel, and HW represents the total number of pixels. Then and the input with a dimension of enter the Category Attention Module (Efficient Category attention block, abbreviated as Efficient CA block, i.e., the CA module). P is used as prior information to perform non-local semantic constraints on the input Figure X , and the specific steps are as follows: Figure X Step 1: Use P to construct a spatial attention map:

[0084] ; Equation (2)

[0085] ; Equation (3)

[0086] ; Equation (3)

[0087] where A represents the spatial attention map; represents the semantic correlation between pixel i and pixel j; represents the probability vector that the i-th pixel predicted by the model belongs to the segmentation region; represents the probability vector that the j-th pixel predicted by the model belongs to the segmentation region; T represents the transpose operation; represents the transpose operation on the probability vector .

[0088] Step 2: Calculate the normalization matrix:

[0089] ; Equation (4)

[0090] where N represents the normalization matrix; diag represents the diagonal matrix; represents the probability vector that the i-th pixel predicted by the model belongs to the segmentation region; It represents the probability vector that the j-th pixel predicted by the model belongs to the segmentation region; T represents the transpose operation; It represents the probability vector for which the transpose operation is performed.

[0091] The third step: By performing input space attention constraint and normalization on the input Figure X the output of the CA module is obtained:

[0092] ; Equation (5)

[0093] where, Y represents the output of the CA module; X represents the input image; N represents the normalization matrix; T represents the transpose operation; It represents the transpose operation on the probability map ; It represents the probability map; A represents the spatial attention map.

[0094] In the specific calculation process, we use the associative law of matrix multiplication, first calculate , and then calculate . The complexity of these two steps of calculation is and , while the complexity of calculating and is and respectively. Using the former calculation is more efficient. At the same time, since the CA module aggregates global information, it may cause confusion of context information. Therefore, the output of the CA module enters the balance module (Balance block) to balance local information and global information. The specific calculation is to introduce classification adaptive weights:

[0095] ; Equation (6)

[0096] ; Equation (7)

[0097] where, represents the classification adaptive weight matrix; diag represents the diagonal matrix; T represents the transpose operation; It represents the transpose operation on the probability vector ; represents the classification adaptive weight matrix of the control parameter; represents the learnable parameter; represents the activation function.

[0098] Using the classification adaptive weight Obtain the output of the BCA module (Balanced Category attention block, abbreviated as Balance block, that is, the channel-level adaptive balance module):

[0099] ; Formula (VIII)

[0100] Among them, represents the output of the BCA module; represents the classification adaptive weight matrix; Y represents the output of the CA module; represents the identity matrix; X represents the input image.

[0101] The output by the BCA module and the input Figure X are concatenated to obtain , and a conv operation (convolution operation) is performed, that is, a convolution kernel with a kernel size of 1 and the number of channels equal to the number of categories is applied to change the number of channels. The obtained result is added to the rough segmentation map , and the output is the fine segmentation map .

[0102] In this embodiment, non-local semantic constraint operations are respectively performed on the initial characteristic image data of subretinal fluid at different depth layers (that is, ) of the subretinal fluid fusion characteristic image data obtained after the last upsampling operation, that is, the CAR module is used to perform non-local semantic constraint operations on respectively, and the rough segmentation result is used to perform non-local semantic constraint on the fine segmentation result to generate the second subretinal fluid segmentation image data .

[0103] In this embodiment, the method of adding the CAR module to the UNet++ network model is as follows, After passing through a convolutional layer with a kernel size of 1, a rough segmentation feature map with the same resolution and 2 channels is obtained. Regard as a rough segmentation of the input image, that is, in the above steps, and enter the CAR module together with , and successively obtain the first fine segmentation feature map , the second fine segmentation feature map . And so on, the final output of the CAR module is the second subretinal fluid segmentation image data .

[0104] (5) Generate the first fundus predicted lesion area image data according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

[0105] In this embodiment, the first subretinal fluid segmentation image data output by the CAB module and the second subretinal fluid segmentation image data output by the CAR module are added and divided by 2, and then passed through a convolutional layer with a kernel size of 1 and a channel number of 2 as the final output, that is, the first fundus predicted lesion area image data .

[0106] S4032b: Calculate the total loss function of the main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data;

[0107] In this embodiment, the method for calculating the total loss function of the main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data includes:

[0108] (1) Calculate the output loss function of the main task learning according to the preset cross-entropy loss function and the preset dice loss function.

[0109] In this embodiment, the formula of the cross-entropy loss function is:

[0110] ; Formula (IX)

[0111] Where represents the cross-entropy loss function; represents the predicted probability vector that the i-th pixel point predicted by the model belongs to the segmentation area; represents the original image; represents the true label of the i-th pixel point; represents the number of pixel points in the entire original image.

[0112] In this embodiment, the formula of the dice loss function is:

[0113] ; Formula (X)

[0114] Where represents the dice loss function; represents the true segmentation area; represents the segmentation area predicted by the model; represents the number of pixel points included in the true segmentation area; represents the number of pixel points included in the segmentation area predicted by the model; represents and the number of pixel points in the intersection.

[0115] In this embodiment, the method for calculating the output loss function of the main task learning includes:

[0116] ; Equation (XI)

[0117] Wherein, represents the output loss function of the main task learning; represents the cross-entropy loss function; represents the dice loss function.

[0118] (2) According to the output loss function of the main task learning, the loss functions of the analog attention enhancement module and the semantic constraint attention module are calculated.

[0119] In this embodiment, for the initial characteristic image data of subretinal fluid in the fifth deep layer perform 4 upsampling operations, and successively obtain , and bilinearly interpolate the four outputs of the upsampling path to the original image size respectively, and generate the corresponding , and add them to the output loss function with weighted summation, and the weights are successively: , the method for calculating the loss function of the analog attention enhancement module (CAB module) includes:

[0120] ; Equation (XII)

[0121] Wherein, represents the loss function of the analog attention enhancement module; represents the output loss function of the main task learning; represents the rough feature map of the subretinal fluid area after bilinear interpolation; represents the fine segmentation feature map of the first subretinal fluid area after bilinear interpolation; represents the fine segmentation feature map of the second subretinal fluid area after bilinear interpolation; represents the first subretinal fluid segmentation image data after bilinear interpolation.

[0122] In this embodiment, the rough segmentation feature map , the first fine segmentation feature map , the second fine segmentation feature map , the second subretinal fluid segmentation image data , and add them to the output loss function with weighted summation, and the weights are successively , the method for calculating the loss function of the semantic constraint attention module (CAR module) includes:

[0123] ; Equation (XIII)

[0124] Among them, represents the loss function of the semantic constraint attention module; represents the output loss function of the main task learning; represents the rough segmentation feature map; represents the first fine segmentation feature map; represents the second fine segmentation feature map; represents the second subretinal fluid segmentation image data.

[0125] (3) According to the first fundus prediction lesion area image data, the output loss function of the main task learning, the loss function of the analog attention enhancement module, and the loss function of the semantic constraint attention module, calculate the total loss function of the main task learning.

[0126] In this embodiment, the method for calculating the total loss function of the main task learning includes:

[0127] ; Formula (XIV)

[0128] ; Formula (XV)

[0129] Among them, represents the total loss function of the main task learning; represents the loss function of the semantic constraint attention module; represents the loss function of the analog attention enhancement module; represents the output loss function of the main task learning; represents the final output loss function of the main task learning; represents the first fundus prediction lesion area image data.

[0130] S4032c: Input the historical fundus slice training image data into the UNet network model for branch task learning, and calculate the total loss function of the branch task learning and the branch task learning label.

[0131] In this embodiment, the method for inputting the historical fundus slice training image data into the UNet network model for branch task learning, and calculating the total loss function of the branch task learning and the branch task learning label includes:

[0132] (1) Input the historical fundus slice training image data into the UNet network model for branch task learning to obtain the level set function of the second fundus prediction lesion area image data.

[0133] (2) According to the level set function of the second fundus prediction lesion area image data, calculate the total loss function of the branch task learning and the branch task learning label.

[0134] In this embodiment, the branch task learning is to learn the level set regression task. The training data set is input into the neural network UNet for forward propagation. The UNet network first applies a convolutional module to the input image. The convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, a channel number of 32, and no bias, followed by batch normalization and a Relu function. Then, it repeats a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, a channel number of 32, no bias, batch normalization, and a Relu function. Subsequently, it is also divided into two processes of downsampling and upsampling. The downsampling process is divided into four downsampling modules in total, and each downsampling module is the same as the downsampling module of the previous UNet++. The upsampling process is also divided into four upsampling modules, and the upsampling module is the same as the upsampling module of the previous UNet++. Finally, a convolutional kernel with a kernel size of 3*3, a stride of 1, a padding of 1, and a channel number of 1 is applied to obtain the level set function of the second fundus predicted lesion area image data. is the level set function predicted by the model, and the true level set function calculated by the model using the label Supervised learning, The expression is as follows:

[0135] ; Formula (XVI)

[0136] where, represents the level set function of the second fundus predicted lesion area image data; represents the true level set function; represents the shortest distance from the pixel point x to the boundary of the lesion area; represents the lesion area; represents the boundary of the lesion area; represents the complement of Ω.

[0137] In this embodiment, the branch task learning adopts the L1 norm (i.e., absolute error) as the loss function. The methods for calculating the total loss function of the branch task learning include:

[0138] ; Formula (XVII)

[0139] where, represents the total loss function of the branch task learning; represents the level set function of the second fundus predicted lesion area image data; represents the true level set function; represents the number of pixel points in the entire original image; x represents the pixel point.

[0140] In this embodiment, the total loss function of the branch task learning is used to measure the difference between the level set function predicted by the model and the true level set function, enabling the model to learn how to more accurately predict the fundus lesion area.

[0141] In this embodiment, the method for calculating the branch task learning label by performing an inverse transformation on the level set function obtained from the branch task learning includes:

[0142] ; Formula (XVIII)

[0143] where represents the branch task learning label; x represents the pixel point.

[0144] S4032d: Calculate the enhanced loss function according to the branch task learning label and the main task learning label after the softening operation;

[0145] In this embodiment, according to the first fundus predicted lesion area image data , the method for performing a softening operation on the preset main task learning label includes:

[0146] ; Formula (XIX)

[0147] where represents the main task learning label after the softening operation; is the sigmoid function; represents the preset main task learning label.

[0148] In this embodiment, the method for calculating the enhanced loss function by taking the difference between the label obtained from the branch task learning and the softened label obtained from the main task learning and then applying the L1 norm as the loss function includes:

[0149] ; Formula (XX)

[0150] where represents the enhanced loss function; represents the branch task learning label; x represents the pixel point; represents the main task learning label after the softening operation; represents the original image.

[0151] S4032e: Update the parameters of the UNet++ network model and the UNet network model based on multi-semantic supervision according to the total loss function of the main task learning, the total loss function of the branch task learning, and the enhanced loss function, so as to construct an initial fundus lesion area prediction model.

[0152] In this embodiment, by introducing branch tasks and enhancing the loss function, the learning of the main task can be effectively assisted, the performance of the UNet++ network model based on multi-semantic supervision can be improved, and a high-performance initial fundus lesion area prediction model can be constructed, making full use of the advantages of multi-task learning and significantly enhancing the generalization ability and segmentation accuracy of the model.

[0153] S4033: Based on a preset optimization algorithm, sequentially update the parameters of the total loss function of the branch task learning, the enhanced loss function, and the total loss function of the main task learning, and perform iterative training on the initial fundus lesion area prediction model to obtain a finally converged fundus lesion area prediction model.

[0154] In this embodiment, batch stochastic gradient descent is used as the optimization algorithm to update the model parameters through backpropagation. More specifically, first, for the total loss function of the branch task learning Backpropagation to update the parameters helps the model learn global features or boundary information related to the lesions. Then, for the enhanced loss function Backpropagation to update the parameters helps introduce additional prior knowledge or constraints and improve the robustness of the model. Finally, for the total loss function of the main task learning Update the parameters, which helps the model focus on learning the precise segmentation of the lesion area. The initial learning rate is 0.01, and the momentum method is used to accelerate the gradient update. The weight is set to 0.9 for learning rate decay, and the weight is set to 0.0004.

[0155] In this embodiment, the validation dataset is input into the model, and the first fundus predicted lesion area image data output by the model is used as the fundus predicted lesion area validation image data. According to the fundus predicted lesion area validation image data and the real lesion area image data, the validation effect of the model is obtained. Repeat the model training and input of the validation dataset for a total of 50 batches, and select the model with the best performance on the validation dataset to obtain a finally converged fundus lesion area prediction model. Input the test dataset into this model, and the first fundus predicted lesion area image data output by the model is used as the fundus predicted lesion area test image data. According to the fundus predicted lesion area test image data and the real lesion area image data, the test effect of the finally converged fundus lesion area prediction model is obtained.

[0156] In this embodiment, as shown in Table 1, it is the test effect accuracy table:

[0157] Table 1

[0158]

[0159] To test the effectiveness of the model of this application and compare it with other different models, in this embodiment, CAB UNet is a UNet model with a CAB module added, CAB UNet++ is a UNet++ model with a CAB module added, CAR UNet++ is a UNet++ model with a CAR module added, and LSEM-SC UNet++ represents a UNet++ model with level set enhanced multi-semantic supervision (Level Set Enhanced Muti-Semantic Constrained UNet++, LSEM-SC UNet++), that is, the final converged fundus lesion area prediction model. More specifically, the main task is a UNet++ with a CAR module and a CAB module added, and the branch task is a model for learning the level set function. In terms of the indicators PPV and DSC, the model of this application achieves the best results. The test indicators are TPVF (True Positive Volume Fraction), DSC (Dice Similarity Coefficient), and PPV (Positive Predicative Value) respectively. The definitions are as follows:

[0160] ; Formula (XXI)

[0161] ; Formula (XXII)

[0162] ; Formula (XXIII)

[0163] Among them, TPVF represents the recall rate, which measures the proportion of positive samples correctly predicted by the model among the actual positive samples; DSC represents the overlap rate, which measures the overlap degree between the prediction result and the true result; PPV represents the accuracy rate, which measures the proportion of samples predicted as positive by the model that are actually positive; represents the number of samples predicted as 1 and actually 1; is the number of samples actually 1; is the number of samples predicted as 1; is the number of samples predicted as 1 and actually 0.

[0164] In this embodiment, by iteratively training the initial fundus lesion area prediction model, repeating the input of the model training and validation data sets for a total of 50 batches, and selecting the model with the highest DSC index on the validation data set, the final converged fundus lesion area prediction model is obtained.

[0165] S404: Calculate the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0166] In this embodiment, is the number of real-time fundus slice image data, . The method for calculating the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes:

[0167] ; Formula (XXIV)

[0168] where V represents the subretinal fluid volume data of the patient; represents the number of real-time fundus slice image data; L represents the actual length of the real-time fundus slice image data; represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; represents the th number of pixel points of the fundus predicted lesion area image data corresponding to the real-time fundus slice image data.

[0169] In this embodiment, based on the pre-constructed fundus lesion area prediction model, the fundus predicted lesion area image is automatically segmented, and the pixel points belonging to the subretinal fluid part in each fundus predicted lesion area image data are summed to obtain the subretinal fluid volume data, which greatly shortens the reading time of fundus doctors, improves work efficiency, and can also significantly improve the accuracy, stability and homogeneity of reading, and can help clinicians achieve accurate and objective evaluation of CSC.

[0170] In this embodiment, the OCT volume scanning mode can generate a series of continuous fundus slice image data. Assume that a is the scanning line length, in degrees, and the actual length of the real-time fundus slice image data is L, in mm. When the OCT scanning range width is , the calculation method of the actual length of the real-time fundus slice image data includes:

[0171] ; Formula (XXV)

[0172] where L represents the actual length of the real-time fundus slice image data; a is the scanning line length.

[0173] In this embodiment, assume that b is the original width of the scanning area, in degrees, and W is the actual width of the scanning area, in mm. When the OCT scanning range width is , the calculation method of the actual width of the scanning area includes:

[0174] ; Formula (XXVI)

[0175] Among them, W represents the actual width of the scanning area; b represents the original width of the scanning area.

[0176] In this embodiment, the calculation method of the interval between each real-time fundus slice image data includes:

[0177] Formula (XXVII)

[0178] Among them, h represents the interval between each real-time fundus slice image data; W represents the actual width of the scanning area; represents the number of real-time fundus slice image data.

[0179] It should be noted that according to Formula (XXIV) to Formula (XXVII), as long as the length and width of the input scanning area, as well as the number of real-time fundus slice image data are input, after importing the image, the volume data of the subretinal fluid can be automatically calculated.

[0180] It should be noted that the method for segmenting and calculating the volume of subretinal fluid of the present invention has the following advantages:

[0181] (1) By introducing the level set enhancement and multiple semantic supervision mechanisms, it is possible to finely segment the fluid accumulation area related to CSC.

[0182] (2) By combining the main task (segmentation) and the branch task (level set function learning), the model can simultaneously capture global semantic information and local detail information, further improving the segmentation accuracy.

[0183] (3) It realizes a fully automated process from image input to subretinal fluid volume calculation, without manual intervention, can quickly complete image segmentation and volume calculation, greatly shortening the doctor's film reading time, not only improving the accuracy, stability and homogeneity of film reading, but also significantly improving the work efficiency of fundus doctors, realizing a precise assessment of the CSC lesion degree, so as to provide a more effective treatment plan for patients.

[0184] (4) By introducing the semantic constraint attention module (CAR module) and the analogy attention enhancement module (CAB module), the model shows highly stable performance on different data sets.

[0185] (5) Using a preset optimization algorithm, the parameters of the total loss function of the branch task learning, the enhancement loss function, and the total loss function of the main task learning are updated in sequence to ensure the rapid convergence of the model.

[0186] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects. For example, the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data are only used to distinguish different segmentation image data, and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily limit being different.

[0187] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0188] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b or c can represent: a, b, c, a - b, a - c, b - c or a - b - c, where a, b, c can be single or multiple.

[0189] Figure 13 is a schematic block diagram of a subretinal fluid differential segmentation and volume calculation system provided by the embodiments of the present application. As Figure 13 shown, the subretinal fluid differential segmentation and volume calculation system 1300 includes:

[0190] A data acquisition module 1301, configured to acquire real-time fundus image data of a patient.

[0191] A data preprocessing module 1302, configured to preprocess the real-time fundus image data to obtain multiple real-time fundus slice image data.

[0192] A lesion area prediction module 1303, configured to obtain fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data according to each of the real-time fundus slice image data and based on a pre-constructed fundus lesion area prediction model.

[0193] A volume calculation module 1304 is configured to calculate the subretinal fluid volume data of a patient based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0194] It should be understood that the specific processes for each module to execute the corresponding steps have been described in detail in the above method embodiments. For the sake of brevity, they will not be repeated here.

[0195] It should also be understood that the division of modules in the embodiments of the present application is illustrative, merely a logical function division. In actual implementation, there may be other division methods. Additionally, in each embodiment of the present application, each functional module may be integrated in a processor, may exist alone physically, or two or more modules may be integrated in one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0196] Figure 14 is a schematic block diagram of an electronic terminal provided by an embodiment of the present application. The electronic terminal includes a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the subretinal fluid segmentation and volume calculation method as described above. As Figure 14 shown, the electronic terminal 1400 includes: at least one processor 1401, a memory 1402, at least one network interface 1403, and a user interface 1405. Each component in the device is coupled together through a bus system 1404. It can be understood that the bus system 1404 is used to realize the connection and communication between these components. In addition to including a data bus, the bus system 1404 further includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 14 all kinds of buses are labeled as the bus system.

[0197] Among them, the user interface 1405 may include a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touchpad, or a touch screen, etc.

[0198] It can be understood that the memory 1402 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiments of the present invention is intended to include but not limited to these and any other suitable categories of memories.

[0199] The memory 1402 in the embodiments of the present invention is used to store various categories of data to support the operation of the electronic terminal 1400. Examples of these data include: any executable programs for operating on the electronic terminal 1400, such as the operating system 14021 and application programs 14022; the operating system 14021 contains various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application programs 14022 can include various application programs, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. Implementing the subretinal fluid segmentation and volume calculation method provided by the embodiments of the present invention can be included in the application programs 14022.

[0200] The method disclosed in the embodiments of the present invention above can be applied to or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 1401 or the instructions in the form of software. The above-mentioned processor 1401 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1401 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 1401 may be a microprocessor or any conventional processor, etc. Combining the steps of the accessory optimization method provided in the embodiments of the present invention can be directly embodied as being executed and completed by the hardware decoding processor, or by the combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, and this storage medium is located in the memory. The processor reads the information in the memory and combines its hardware to complete the steps of the foregoing method.

[0201] In an exemplary embodiment, the electronic terminal 1400 may be one or more application-specific integrated circuits (ASICs, Application Specific Integrated Circuit), DSPs, programmable logic devices (PLDs, Programmable Logic Device), complex programmable logic devices (CPLDs, Complex Programmable Logic Device) for executing the foregoing method.

[0202] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code runs on a computer, it causes the computer to execute Figures 4 to 12 the method of any one of the embodiments shown.

[0203] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores program code, when the program code runs on a computer, it causes the computer to execute Figures 4 to 12 the method of any one of the embodiments shown.

[0204] As used in this specification, the terms "component", "module", "system", etc. are used to represent computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be components. One or more components can reside in a process and / or an execution thread, and a component can be located on one computer and / or distributed between two or more computers. In addition, these components can execute from various computer-readable media storing various data structures. A component can communicate, for example, through local and / or remote processes via a signal having one or more data packets (e.g., data from two components interacting with another component in a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via a signal).

[0205] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether such functions are implemented in hardware or software depends upon the particular application and design constraints of the technical solution. Skilled artisans may implement the described functions in different ways for each particular application, but such implementation should not be considered to exceed the scope of this application.

[0206] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0208] The unit described as a separation component may or may not be physically separated, and the component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, each functional unit in various embodiments of this application can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0210] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the processes or functions according to the embodiments of this application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that contains one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a high-density digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD), etc.).

[0211] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0212] As described above, this is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the technical field of this application can easily think of changes or substitutions within the technical scope disclosed by this application, and all should be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

[0213] In summary, this application provides a method, system, product, and terminal for subretinal fluid segmentation and volume calculation. Based on a pre-constructed fundus lesion area prediction model, it automatically segments the fundus predicted lesion area image, and by summing the pixel points belonging to the subretinal fluid part in the data of each fundus predicted lesion area image, the subretinal fluid volume data is obtained, greatly shortening the reading time of fundus disease doctors and improving work efficiency. In addition, it can significantly improve the accuracy, stability, and homogeneity of reading, and can help clinicians achieve accurate and objective evaluation of CSC. Therefore, this application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0214] The above embodiments are only illustrative of the principles and effects of this application and are not used to limit this application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by this application should still be covered by the claims of this application.

Claims

1. A method for segmenting and calculating the volume of subretinal fluid, characterized in that: include: Acquire real-time fundus image data of patients; Preprocessing the real-time fundus image data to obtain a plurality of real-time fundus slice image data; According to each of the real-time fundus slice image data and based on a pre-constructed fundus lesion area prediction model, the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data is obtained; The method for constructing the fundus lesion area prediction model includes: Acquiring historical fundus image data of different patients and preprocessing the historical fundus image data of the different patients to construct a training data set; the training data set includes a plurality of historical fundus slice training image data; The historical fundus slice training image data is input into the UNet++ network model based on multiple semantic supervisions for main-line task learning, and is simultaneously input into the UNet network model for branch-line task learning, and then the total loss function of the main-line task learning, the total loss function of the branch-line task learning and the enhanced loss function are calculated to construct an initial fundus lesion area prediction model; the method includes: inputting the historical fundus slice training image data into the UNet++ network model based on multiple semantic supervisions for main-line task learning, obtaining the first fundus predicted lesion area image data, and performing a softening operation on the preset main-line task learning label according to the first fundus predicted lesion area image data; according to the preset The cross entropy loss function, the preset dice loss function and the first fundus prediction lesion area image data are used to calculate the total loss function of the main task learning; the historical fundus slice training image data is input into the UNet network model for branch task learning, and the total loss function of the branch task learning and the branch task learning label are calculated; according to the branch task learning label and the main task learning label after the softening operation, the enhancement loss function is calculated; according to the total loss function of the main task learning, the total loss function of the branch task learning and the enhancement loss function, the parameters of the UNet++ network model and the UNet network model based on multiple semantic supervision are updated to construct an initial fundus lesion area prediction model; Based on a preset optimization algorithm, the total loss function of the branch task learning, the enhancement loss function, and the total loss function of the main task learning are sequentially updated, and the initial fundus lesion area prediction model is iteratively trained to obtain a final converged fundus lesion area prediction model; The patient's subretinal fluid volume data is calculated based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

2. The method for subretinal fluid segmentation and volume calculation according to claim 1, wherein: The method of calculating the patient's subretinal fluid volume data based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes: Wherein, V represents the volume data of the patient's subretinal fluid; N' represents the number of real-time fundus slice image data; L represents the actual length of the real-time fundus slice image data; D' represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; M i’ It represents the number of pixels of the fundus predicted lesion area image data corresponding to the i'th real-time fundus slice image data.

3. The method for subretinal fluid segmentation and volume calculation according to claim 1, wherein: Inputting the historical fundus slice training image data into a UNet++ network model based on multiple semantic supervision to perform mainline task learning, and obtaining the first fundus predicted lesion area image data includes: Performing multiple downsampling operations on the historical fundus slice training image data to obtain initial feature image data of subretinal fluid at different depth layers; Performing upsampling operations on the initial feature image data of subretinal fluid at different depth layers for different times to obtain fused feature image data of subretinal fluid at different depth layers; Based on the analog attention improvement module added to the UNet++ network model, feature enhancement and fusion operations are performed on the subretinal fluid initial feature image data of the last depth layer and the subretinal fluid fusion feature image data obtained after each upsampling operation to generate the first subretinal fluid segmentation image data; Based on the semantic constraint attention module added to the UNet++ network model, non-local semantic constraint operations are performed on the subretinal fluid fusion feature image data obtained after the last upsampling operation of the initial feature image data of subretinal fluid at different depth layers to generate the second subretinal fluid segmentation image data; First fundus predicted lesion area image data is generated according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

4. The method for subretinal fluid segmentation and volume calculation according to claim 3, wherein: Methods for calculating the total loss function for the main task learning include: According to the preset cross entropy loss function and the preset dice loss function, the output loss function of the main task learning is calculated; Based on the output loss function learned from the main task, the loss function of the analog attention enhancement module and the loss function of the semantic constraint attention module are calculated; Based on the first fundus predicted lesion area image data, the output loss function of the main task learning, the loss function of the analogy attention enhancement module, and the loss function of the semantic constraint attention module, the total loss function of the main task learning is calculated.

5. The method for subretinal fluid segmentation and volume calculation according to claim 1, wherein: Inputting the historical fundus slice training image data into the UNet network model for branch task learning, and calculating the total loss function of the branch task learning and the branch task learning label include: Inputting the historical fundus slice training image data into the UNet network model to perform branch task learning, and obtaining the level set function of the second fundus prediction lesion area image data; According to the level set function of the second fundus predicted lesion area image data, the total loss function of the branch task learning and the branch task learning label are calculated.

6. A subretinal fluid area segmentation and volume calculation system, characterized in that: include: A data acquisition module, used to acquire real-time fundus image data of the patient; A data preprocessing module, configured to preprocess the real-time fundus image data to obtain a plurality of real-time fundus slice image data; a lesion area prediction module, configured to obtain, based on each of the real-time fundus slice image data and a pre-built fundus lesion area prediction model, fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data; The method for constructing the fundus lesion area prediction model includes: Acquiring historical fundus image data of different patients and preprocessing the historical fundus image data of the different patients to construct a training data set; the training data set includes a plurality of historical fundus slice training image data; The historical fundus slice training image data is input into the UNet++ network model based on multiple semantic supervisions for main-line task learning, and is simultaneously input into the UNet network model for branch-line task learning, and then the total loss function of the main-line task learning, the total loss function of the branch-line task learning and the enhanced loss function are calculated to construct an initial fundus lesion area prediction model; the method includes: inputting the historical fundus slice training image data into the UNet++ network model based on multiple semantic supervisions for main-line task learning, obtaining the first fundus predicted lesion area image data, and performing a softening operation on the preset main-line task learning label according to the first fundus predicted lesion area image data; according to the preset The cross entropy loss function, the preset dice loss function and the first fundus prediction lesion area image data are used to calculate the total loss function of the main task learning; the historical fundus slice training image data is input into the UNet network model for branch task learning, and the total loss function of the branch task learning and the branch task learning label are calculated; according to the branch task learning label and the main task learning label after the softening operation, the enhancement loss function is calculated; according to the total loss function of the main task learning, the total loss function of the branch task learning and the enhancement loss function, the parameters of the UNet++ network model and the UNet network model based on multiple semantic supervision are updated to construct an initial fundus lesion area prediction model; Based on a preset optimization algorithm, the total loss function of the branch task learning, the enhancement loss function, and the total loss function of the main task learning are sequentially updated, and the initial fundus lesion area prediction model is iteratively trained to obtain a final converged fundus lesion area prediction model; The volume calculation module is used to calculate the patient's subretinal fluid volume data based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

7. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are run on a computer, the computer is enabled to implement the method for segmenting and calculating the subretinal fluid area according to any one of claims 1 to 5.

8. An electronic terminal comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the subretinal fluid area segmentation and volume calculation method according to any one of claims 1 to 5.

Images