Subretinal effusion region segmentation and volume calculation method, system, product and terminal

Abstract

The invention provides a subretinal effusion region segmentation and volume calculation method and system, a product and a terminal, and the method comprises the steps: automatically segmenting a fundus prediction lesion region image based on a pre-constructed fundus lesion region prediction model, and carrying out the summation of pixel points, belonging to a subretinal effusion part, in the data of each fundus prediction lesion region image. According to the method, the volume data of the subretinal fluid is obtained, the film reading time of a fundus disease doctor is greatly shortened, the working efficiency is improved, in addition, the film reading accuracy, stability and homogeneity can be remarkably improved, and clinical doctors can be helped to accurately and objectively evaluate the central serous chorioretinopathy.

Description

Technical Field

[0001] This application relates to the field of computer vision, and particularly to a method, system, product, and terminal for subretinal fluid segmentation and volume calculation. Background Art

[0002] Central Serous Chorioretinopathy (CSC) is a common fundus disease, especially with a relatively high incidence rate among men over 30 years old. Although acute CSC is generally considered a self-limiting disease with a good prognosis, chronic and recurrent CSC may lead to retinal pigment epithelial lesions and retinal atrophy, thus causing irreversible vision loss. Research shows that about 12.8% of CSC patients progress to bilateral visual acuity below 0.1 within 10 years of follow-up, reaching the level of "legal blindness". The high incidence rate of CSC makes the accurate assessment of its condition particularly important to reduce the risk of blindness.

[0003] Optical Coherence Tomography (OCT) is a technology that uses near-infrared light as the incident light source and obtains reflection signals at various depths inside tissues through interference to form images. Different retinal tissues have different light reflection characteristics. The tissue cross-sectional images presented by OCT are generated into pseudo-color images or grayscale images according to the reflection signals of different intensities during the image processing process, and are called "optical biopsies". In the diagnosis and follow-up of CSC, OCT is widely used to detect subretinal fluid in the posterior pole. The qualitative and quantitative evaluation of the fluid is of great significance for clinicians to formulate treatment strategies.

[0004] However, the current clinical evaluation of CSC mainly relies on manual linear analysis, and this method has certain limitations ( Figure 1 ). On the other hand, the analysis software built into OCT cannot distinguish between the two layers of subretinal fluid and the detached area of the retina, but measures them together, and the obtained total value cannot accurately reflect the true situation of the fluid ( Figure 2 and 3 ). Therefore, both manual measurement and built-in software analysis are one-sided and ambiguous, lacking accurate volume quantification analysis of subretinal fluid in the lesion area, resulting in inaccurate assessment of the CSC condition. In addition, the process of reading fundus images is complex, requiring a large amount of time from specialist doctors, and there is also the influence of subjective factors. The judgment results of the same doctor at different times or between different doctors may be inconsistent.

[0005] Therefore, developing an automatic analysis method that can objectively, accurately, and quickly calculate the volume of subretinal fluid has become an urgent need for the current clinical evaluation of CSC. Summary of the Invention

[0006] In view of the deficiencies of the prior art described above, the purpose of this application is to provide a method, system, product and terminal for segmenting and calculating the volume of subretinal fluid, which is used to solve the technical problems such as the lack of quantitative analysis of the volume of subretinal fluid and the one-sidedness and ambiguity of manual measurement in the prior art.

[0007] To achieve the above purpose and other related purposes, the first aspect of this application provides a method for segmenting and calculating the volume of subretinal fluid, including: obtaining real-time fundus image data of a patient; preprocessing the real-time fundus image data to obtain multiple real-time fundus slice image data; according to each of the real-time fundus slice image data, and based on a pre-constructed fundus lesion area prediction model, obtaining the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data; calculating the volume data of the subretinal fluid of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0008] In some embodiments of the first aspect of this application, the method of calculating the volume data of the subretinal fluid of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes:

[0009] ;

[0010] where V represents the volume data of the subretinal fluid of the patient; represents the number of real-time fundus slice image data; represents the actual length of the real-time fundus slice image data; represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; represents the number of pixel points of the fundus predicted lesion area image data corresponding to the

[0011] In some embodiments of the first aspect of the present application, the construction method of the fundus lesion area prediction model includes: obtaining historical fundus image data of different patients, and preprocessing the historical fundus image data of different patients to construct a training data set; the training data set includes a plurality of historical fundus slice training image data; inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning, and simultaneously inputting it into a UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model; based on a preset optimization algorithm, sequentially updating the parameters of the total loss function of branch task learning, the enhanced loss function, and the total loss function of main task learning, and iteratively training the initial fundus lesion area prediction model to obtain a finally converged fundus lesion area prediction model.

[0012] In some embodiments of the first aspect of the present application, the method of inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning, and simultaneously inputting it into a UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model includes: inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning to obtain first fundus predicted lesion area image data, and performing a softening operation on a preset main task learning label according to the first fundus predicted lesion area image data; calculating the total loss function of main task learning according to a preset cross-entropy loss function, a preset dice loss function, and the first fundus predicted lesion area image data; inputting the historical fundus slice training image data into a UNet network model for branch task learning, and calculating the total loss function of branch task learning and the branch task learning label; calculating the enhanced loss function according to the branch task learning label and the softened main task learning label; updating the parameters of the UNet++ network model and the UNet network model based on multi-semantic supervision according to the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model.

[0013] In some embodiments of the first aspect of the present application, the method of inputting the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning to obtain the first fundus predicted lesion area image data includes: performing multiple downsampling operations on the historical fundus slice training image data to obtain subretinal fluid initial feature image data of different depth layers; performing upsampling operations with different numbers of times on the subretinal fluid initial feature image data of different depth layers to obtain subretinal fluid fusion feature image data of different depth layers; based on the analog attention enhancement module added to the UNet++ network model, performing feature enhancement and fusion operations on the subretinal fluid fusion feature image data obtained after each upsampling operation on the subretinal fluid initial feature image data of the last depth layer to generate the first subretinal fluid segmentation image data; based on the semantic constraint attention module added to the UNet++ network model, performing non-local semantic constraint operations on the subretinal fluid fusion feature image data obtained after the last upsampling operation on the subretinal fluid initial feature image data of different depth layers to generate the second subretinal fluid segmentation image data; generating the first fundus predicted lesion area image data according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

[0014] In some embodiments of the first aspect of the present application, the method of calculating the total loss function of the main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data includes: calculating the output loss function of the main task learning according to the preset cross-entropy loss function and the preset dice loss function; calculating the loss function of the analog attention enhancement module and the loss function of the semantic constraint attention module according to the output loss function of the main task learning; calculating the total loss function of the main task learning according to the first fundus predicted lesion area image data, the output loss function of the main task learning, the loss function of the analog attention enhancement module, and the loss function of the semantic constraint attention module.

[0015] In some embodiments of the first aspect of the present application, the method of inputting the historical fundus slice training image data into the UNet network model for branch task learning, calculating the total loss function of the branch task learning, and the branch task learning label includes: inputting the historical fundus slice training image data into the UNet network model for branch task learning to obtain the level set function of the second fundus predicted lesion area image data; calculating the total loss function of the branch task learning and the branch task learning label according to the level set function of the second fundus predicted lesion area image data.

[0016] To achieve the above object and other related objects, the second aspect of the present application provides a subretinal fluid segmentation and volume calculation system, including: a data acquisition module for acquiring real-time fundus image data of a patient; a data preprocessing module for preprocessing the real-time fundus image data to obtain a plurality of real-time fundus slice image data; a lesion area prediction module for obtaining the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data according to each of the real-time fundus slice image data and based on a pre-constructed fundus lesion area prediction model; a volume calculation module for calculating the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0017] To achieve the above object and other related objects, the third aspect of the present application provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer implements the subretinal fluid segmentation and volume calculation method as described above.

[0018] To achieve the above object and other related objects, the fourth aspect of the present application provides an electronic terminal, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the subretinal fluid segmentation and volume calculation method as described above.

[0019] As described above, the subretinal fluid segmentation and volume calculation method, system, product, and terminal of the present application have the following beneficial effects:

[0020] (1) By introducing a level set enhancement and multiple semantic supervision mechanism, it is possible to finely segment the fluid area related to CSC.

[0021] (2) Combining the main task (segmentation) and the branch task (level set function learning), the model can simultaneously capture global semantic information and local detail information, further improving the segmentation accuracy.

[0022] (3) An automated process from image input to subretinal fluid volume calculation is realized, without manual intervention. It can quickly complete image segmentation and volume calculation, greatly shortening the doctor's film reading time. It can not only improve the accuracy, stability, and homogeneity of film reading, but also significantly improve the work efficiency of ophthalmologists, realize the accurate assessment of the CSC disease degree, and thus provide a more effective treatment plan for patients.

[0023] (4) By introducing a semantic constraint attention module (CAR module) and an analog attention enhancement module (CAB module), the model shows highly stable performance on different data sets.

[0024] (5) Use a preset optimization algorithm to sequentially update the parameters of the total loss function for the branch task learning, the enhancement loss function, and the total loss function for the main task learning, ensuring the rapid convergence of the model. Description of the Drawings

[0025] Figure 1 It is shown as a sectional view of tissue imaged by OCT in the prior art, Figure 1 where A in [reference] is a schematic diagram of manually linearly measuring the width of subretinal fluid, Figure 1 and B in [reference] is a schematic diagram of manually linearly measuring the height of subretinal fluid.

[0026] Figure 2 It is shown as a schematic diagram of marking the inner limiting membrane and retinal pigment epithelium using the built-in analysis software of OCT in the prior art.

[0027] Figure 3 It is shown as a schematic diagram of showing the total value of the thickness (volume) of the retina and subretinal fluid in different regions using the built-in analysis software of OCT in the prior art, Figure 3 where A in [reference] is a schematic diagram of retinal partitioning, Figure 3 and B in [reference] is a schematic diagram of the total value of the thickness (volume) of the retina and subretinal fluid in different regions in the retinal partitioning shown in A.

[0028] Figure 4 It is shown as a schematic diagram of the process of subretinal fluid segmentation and volume calculation method in an embodiment of the present application.

[0029] Figure 5 It is shown as a schematic diagram of the process of constructing a fundus lesion area prediction model in an embodiment of the present application.

[0030] Figure 6 It is shown as historical fundus image data before preprocessing in an embodiment of the present application.

[0031] Figure 7 It is shown as historical fundus image data after preprocessing in an embodiment of the present application.

[0032] Figure 8 It is shown as a schematic diagram of the process of constructing an initial fundus lesion area prediction model in an embodiment of the present application.

[0033] Figure 9 It is shown as a schematic diagram of the backbone of the UNet++ network model based on multi-semantic supervision in an embodiment of the present application.

[0034] Figure 10 It is shown as a schematic diagram of the backbone of the UNet++ network model in an embodiment of the present application.

[0035] Figure 11It shows a schematic flowchart of the CAB module in an embodiment of the present application.

[0036] Figure 12 It shows a schematic flowchart of the CAR module in an embodiment of the present application.

[0037] Figure 13 It shows a schematic block diagram of a subretinal fluid segmentation and volume calculation system in an embodiment of the present application.

[0038] Figure 14 It shows a schematic structural diagram of an electronic terminal in an embodiment of the present application. Detailed implementation manners

[0039] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0040] Before further elaborating on the present invention, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are applicable to the following explanations:

[0041] <1> UNet: A classic convolutional neural network architecture designed specifically for medical image segmentation tasks, named after its efficient performance and unique U-shaped structure.

[0042] <2> UNet++: An improved version of UNet, aiming to address some limitations of the original UNet in medical image segmentation. UNet++ enhances feature transfer and gradient flow by introducing dense skip connections and deep supervision, thereby improving segmentation accuracy.

[0043] <3> Relu function: Rectified linear unit, a commonly used activation function in deep learning.

[0044] <4> Softmax: A function that converts real values into a probability distribution, widely used in multi-classification tasks and attention mechanisms.

[0045] <5> sigmoid: A commonly used activation function that maps any real value to the range of (0, 1). Due to its smooth S-shaped curve and differentiability, the sigmoid function is widely used in machine learning and deep learning.

[0046] Currently, the clinical evaluation of CSC mainly relies on manual linear analysis, which has certain limitations. On the other hand, the analysis software built into OCT cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area, but measures them together, and the total value obtained cannot accurately reflect the true situation of the fluid. Therefore, both manual measurement and built-in software analysis are one-sided and ambiguous, lacking accurate volume quantification analysis of subretinal fluid in the lesion area, resulting in inaccurate assessment of the CSC condition. In addition, the process of reading fundus images is complex, requiring a large amount of time from specialist doctors, and being affected by subjective factors. The results of image interpretation by the same doctor at different times or between different doctors may be inconsistent.

[0047] Figure 1 It shows a tissue section diagram using OCT imaging in the prior art and a schematic diagram of manually linearly measuring the width and height of subretinal fluid. Figure 2 It shows a schematic diagram of using the built-in analysis software of OCT to mark the inner limiting membrane and retinal pigment epithelium in the prior art, which cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area. Figure 3 It shows a schematic diagram of using the built-in analysis software of OCT to display the total value of the thickness (volume) of the retina and subretinal fluid in different regions. Among them, Figure 3 retina thickness in it represents the retinal thickness.

[0048] As Figure 1 shown, using the built-in software of OCT, manually measure the width and height (A and B) of subretinal fluid. Using the built-in analysis software of OCT, after the volume scan is completed, the inner limiting membrane and retinal pigment epithelium can be automatically marked. As Figure 2 shown, it cannot distinguish between the two layers of subretinal fluid and the retina in the detachment area. Using the automatic zoning function of the built-in analysis software of OCT, as Figure 3 shown: Divide with a circular range with the fovea as the center and diameters of 1, 3, and 6 mm, and then evenly divide it into four quadrants of superior, inferior, nasal, and temporal (a total of 9 blocks). According to Figure 2 the marked lines, the software can automatically calculate the average thickness from the inner limiting membrane to the retinal pigment epithelium in each block, but cannot separately analyze the thickness of subretinal fluid and the retina in the detachment area.

[0049] Therefore, there is an urgent need for an automatic analysis method to objectively, accurately, and quickly calculate the volume of subretinal fluid, improve the accuracy, stability, and homogeneity of image reading, and while improving the clinical work efficiency of ophthalmologists for fundus diseases, achieve accurate assessment of the degree of CSC condition.

[0050] To solve the above technical problems, the present application provides a method, system, product and terminal for subretinal fluid segmentation and volume calculation, which are used to solve the technical problems such as the lack of quantitative analysis of subretinal fluid volume and the one-sidedness and ambiguity of manual measurement in the prior art.

[0051] For the convenience of understanding the embodiments of the present application, first, in combination with Figure 4 detailed description is given. Figure 4 FIG. shows a schematic flowchart of the method for subretinal fluid segmentation and volume calculation in the embodiments of the present invention. The method for subretinal fluid segmentation and volume calculation in this embodiment mainly includes the following steps:

[0052] S401: Obtain the real-time fundus image data of the patient.

[0053] In this embodiment, the real-time fundus image data of the patient includes optical coherence tomography (OCT) images obtained through a volume scanning mode. The OCT volume scanning mode can generate a series of continuous fundus cross-sectional images. According to the specific lesion range of the patient, the scanning parameters, such as the scanning line length, scanning range and the number of cross-sections, are adjusted to meet the diagnostic requirements.

[0054] S402: Preprocess the real-time fundus image data to obtain a plurality of real-time fundus slice image data.

[0055] In this embodiment, the preprocessing methods include but are not limited to: cropping preprocessing, image augmentation preprocessing, normalization preprocessing, regularization preprocessing, etc. First, perform cropping preprocessing on the real-time fundus image data to retain a plurality of real-time OCT fundus slice region images.

[0056] In this embodiment, the image augmentation preprocessing is to randomly magnify the cropped real-time OCT fundus slice region image by 1 to 1.2 times, then randomly crop it with the original size, and horizontally flip the image with a probability of 50% and vertically flip the image with a probability of 50%. The normalization preprocessing is to normalize the gray value of the real-time OCT fundus slice region image after the image augmentation preprocessing from normalize to and calculate the mean and variance of the data gray value. The regularization preprocessing is to regularize the image using the calculated mean and variance.

[0057] S403: According to each of the real-time fundus slice image data, and based on a pre-constructed fundus lesion area prediction model, obtain the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0058] In this embodiment, as Figure 5As shown in the figure, it shows a schematic flowchart of constructing a fundus lesion area prediction model in an embodiment of the present invention. Among them, the construction method of the fundus lesion area prediction model includes:

[0059] S4031: Obtain historical fundus image data of different patients, and preprocess the historical fundus image data of different patients to construct a training data set; the training data set includes multiple historical fundus slice training image data.

[0060] In this embodiment, as Figure 6 shown, it shows an OCT historical fundus slice area image before preprocessing in an embodiment of the present invention. As Figure 7 shown, it shows an OCT historical fundus slice area image after preprocessing in an embodiment of the present invention. Crop the historical fundus image data of different patients, and retain multiple OCT historical fundus slice area images. Perform image augmentation preprocessing, normalization preprocessing, and regularization preprocessing on a part of the cropped OCT historical fundus slice area images to construct a training data set. Perform normalization preprocessing and regularization preprocessing on the remaining cropped OCT historical fundus slice area images to construct a test data set and a validation data set.

[0061] In this embodiment, the method of regularization preprocessing includes:

[0062] ; Formula (1)

[0063] Among them, represents the original pixel value of each image in the data set; represents the mean value of the gray values of all pixel points of all images in the data set; represents the image data after regularization; represents the variance of the gray values of all pixel points in all images in the data set.

[0064] S4032: Input the historical fundus slice training image data into the UNet++ network model based on multi-semantic supervision for main task learning, and at the same time input it into the UNet network model for branch task learning, and then calculate the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function to construct an initial fundus lesion area prediction model.

[0065] In this embodiment, the main task learning is used for fundus lesion area segmentation. The branch task learning is used for fundus lesion area boundary detection.

[0066] In this embodiment, as Figure 8As shown in the figure, it shows the schematic flow chart of constructing an initial fundus lesion area prediction model in an embodiment of the present invention. Inputting the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning, and at the same time inputting it into a UNet network model for branch task learning, and then calculating the total loss function of main task learning, the total loss function of branch task learning, and the enhanced loss function. The ways to construct the initial fundus lesion area prediction model include:

[0067] S4032a: Input the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision for main task learning to obtain the first fundus predicted lesion area image data, and perform a softening operation on the preset main task learning label according to the first fundus predicted lesion area image data.

[0068] In this embodiment, as Figure 9 shown, it shows the schematic diagram of the UNet++ network model framework based on multi-semantic supervision in an embodiment of the present invention. As Figure 10 shown, it shows the schematic diagram of the framework of the UNet++ network model in an embodiment of the present invention. Figure 10 Backbone in it represents the model framework, Down-sampling represents the down-sampling operation, Up-sampling represents the up-sampling operation, Skip connection represents the skip connection, and Convolution represents the feature image. For main task learning, a UNet++ network model based on multi-semantic supervision is used. Input each historical fundus slice training image data in the training dataset into the UNet++ network model based on multi-semantic supervision for forward propagation to obtain a UNet++ network model based on multi-semantic supervision.

[0069] In this embodiment, the ways to input the historical fundus slice training image data into a UNet++ network model based on multi-semantic supervision to obtain the first fundus predicted lesion area image data include:

[0070] (1) Perform multiple down-sampling operations on the historical fundus slice training image data to obtain the initial feature image data of subretinal fluid at different depth layers.

[0071] In this embodiment, the UNet++ network model of the model framework acts on the input historical fundus slice training image data with a convolution module to obtain the initial feature image data of subretinal fluid at the first depth layer ( ), the convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Then repeat the convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Four downsampling operations are performed on the initial subretinal fluid feature image data of the first depth layer to obtain the initial subretinal fluid feature image data of different depth layers in turn. . Among them, represents the initial subretinal fluid feature image data of the first depth layer, represents the initial subretinal fluid feature image data of the second depth layer, represents the initial subretinal fluid feature image data of the third depth layer, represents the initial subretinal fluid feature image data of the fourth depth layer, represents the initial subretinal fluid feature image data of the fifth depth layer. The network is divided into five depth layers, and the resolution and number of channels of the images obtained in each depth layer are the same. The image resolutions and numbers of channels obtained from the first depth layer to the fifth depth layer are , , , , .

[0072] In this embodiment, each downsampling operation is first a pooling operation with a kernel size of 2*2 and a stride of 2, followed by a convolutional module. The convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels of the next depth layer, and no bias, followed by batch normalization once and the Relu function once. Then repeat the convolution with a kernel size of 3*3, a stride of 1, a padding of 1, the number of channels of the next depth layer, and no bias, followed by batch normalization once and the Relu function once.

[0073] (2) Perform upsampling operations with different numbers of times on the initial subretinal fluid feature image data of different depth layers to obtain the fused subretinal fluid feature image data of different depth layers.

[0074] In this embodiment, upsampling operations with different numbers of times are performed on the initial subretinal fluid feature image data of the second to fifth depth layers respectively to extract the subretinal fluid feature information of different depth layers to obtain the fused subretinal fluid feature image data of different depth layers. Specifically, perform 1 upsampling operation on to obtain . Perform 2 upsampling operations on to obtain , Perform 3 upsampling operations on to obtain , , Perform 4 upsampling operations on to obtain , , , Among them, represents the first subretinal fluid fusion feature image data of the first depth layer, represents the second subretinal fluid fusion feature image data of the first depth layer, represents the third subretinal fluid fusion feature image data of the first depth layer, represents the fourth subretinal fluid fusion feature image data of the first depth layer. represents the first subretinal fluid fusion feature image data of the second depth layer, represents the second subretinal fluid fusion feature image data of the second depth layer, represents the third subretinal fluid fusion feature image data of the second depth layer. represents the first subretinal fluid fusion feature image data of the third depth layer, represents the second subretinal fluid fusion feature image data of the third depth layer. represents the first subretinal fluid fusion feature image data of the fourth depth layer.

[0075] In this embodiment, specifically, each upsampling operation first uses the bilinear interpolation method with an upsampling rate of 2 to double the resolution of the input image (height width number of channels) to obtain the interpolated image . Then, the interpolated image is concatenated with the image of the same resolution obtained in the same depth layer, and then a convolution module is applied. The convolution module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is half of the number of channels of the input image , without using bias, followed by one batch normalization and one application of the Relu function. Then repeat a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is half of the number of channels of the input image , without using bias, followed by one batch normalization and one application of the Relu function. Finally, the output layer uses a convolution kernel with a kernel size of 1 and the number of channels equal to the number of classes to change the number of output channels to the number of classes, so as to obtain the subretinal fluid fusion feature image data of different depth layers through the upsampling operation.

[0076] (3) Based on the analog attention enhancement module added to the UNet++ network model, feature enhancement and fusion operations are respectively performed on the subretinal fluid fusion feature image data obtained after each upsampling operation on the initial feature image data of the subretinal fluid in the last depth layer to generate the first subretinal fluid segmentation image data.

[0077] In this embodiment, as Figure 11 shown, a schematic flow diagram of the CAB module in the embodiment of the present invention is shown. The following is an explanation of the working process of the analog attention enhancement module (CAB module):

[0078] First, category attention is calculated, and a normalized exponential mapping (softmax mapping) is performed on the third dimension of the rough segmentation map to obtain a probability map , where represents the height of the image, represents the width of the image, and is the number of categories. Then, a Mask Split operation (mask segmentation operation) is performed, that is, each channel of the probability map is separated to obtain spatial weights corresponding to each category, and then a Mul operation (weighted multiplication operation) is performed, that is, multiplying with another fine feature map at the corresponding position to achieve the category attention mechanism, and feature maps are respectively obtained. The C feature maps are the feature maps such as Class1, Class2, Class3, etc. shown in the figure. Class1 represents the first weighted fine feature map, Class2 represents the second weighted fine feature map, and Class3 represents the third weighted fine feature map, where is the number of channels. A concatenation operation (Concat) is performed on these feature maps to obtain a feature image with a dimension of . A convolution module is applied to the feature image for feature fusion. The convolution module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is halved of the input channels, without using bias, followed by a batch normalization and a Relu function application. Then, a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, and the number of channels is 2 (the number of categories), without using bias, followed by a batch normalization and a Relu function application, to obtain an image with a size of . The obtained image is added to the rough segmentation map to obtain the output image .

[0079] In this embodiment, the initial characteristic image data of the subretinal fluid in the last depth layer represents the initial characteristic image data of the subretinal fluid in the fifth depth layer. , Perform 4 upsampling operations on , Add a CAB module, which is used to accelerate the transmission of semantic information to fine segmentation using rough segmentation for feature enhancement and fusion operations. Exemplarily, taking as an example, first apply a convolutional kernel with 2 channels and a kernel size of 1 to change the number of channels to 2 channels to obtain a rough feature map of the subretinal fluid region , Perform upsampling on the rough feature map of the subretinal fluid region to double the resolution to obtain a high-resolution feature map of the fluid region. and are respectively used as and in the above steps and input into the CAB module to output the first fine segmentation feature map of the subretinal fluid region , where is used as the rough segmentation map, is used as the fine feature map.

[0080] In this embodiment, the method of adding the CAB module to the UNet++ network model is as follows. Since it is the first upsampling feature map, the CAB module is not used. For , first apply a convolutional layer with a kernel size of 1 to to obtain a rough feature map of the subretinal fluid region with the same resolution and 2 channels . Then enter the CAB module together with to output the first fine segmentation feature map of the subretinal fluid region , and then enter the next depth layer to obtain the second fine segmentation feature map of the subretinal fluid region . and are input into the CAB module, and finally the first subretinal fluid segmentation image data is obtained, which is the final output of the CAB module.

[0081] (4) Based on the semantic constraint attention module added to the UNet++ network model, perform non-local semantic constraint operations on the initial characteristic image data of the subretinal fluid in different depth layers on the subretinal fluid fusion characteristic image data obtained after the last upsampling operation to generate the second subretinal fluid segmentation image data.

[0082] In this embodiment, as Figure 12 shown, a schematic flowchart of the CAR module in the embodiment of the present invention is presented. The working process of the Semantic Constraint Attention Module (CAR module, i.e., the CAR module) is described as follows:

[0083] First, a normalization exponential mapping is performed on the third dimension (i.e., the dimension of the number of channels C) of the rough segmentation map to obtain a probability map , that is, on an image with a resolution of HW, the i-th pixel corresponds to a probability vector , the j-th pixel corresponds to a probability vector , represents the probability vector of the HW-th pixel, and HW represents the total number of pixels. Then and the input with a dimension of Figure X enter the Category Attention Module (Efficient Category attention block, abbreviated as Efficient CA block, i.e., the CA module). P is used as prior information to perform non-local semantic constraints on the input Figure X , and the specific steps are as follows:

[0084] The first step: Construct a spatial attention map using P:

[0085] ; Formula (2)

[0086] ; Formula (3)

[0087] where A represents the spatial attention map; represents the semantic correlation between pixel i and pixel j; represents the probability vector that the i-th pixel predicted by the model belongs to the segmentation region; represents the probability vector that the j-th pixel predicted by the model belongs to the segmentation region; T represents the transpose operation; represents the transpose operation on the probability vector .

[0088] The second step: Calculate the normalization matrix:

[0089] ; Formula (4)

[0090] where N represents the normalization matrix; diag represents the diagonal matrix; represents the probability vector that the i-th pixel predicted by the model belongs to the segmentation region; It represents the probability vector that the j-th pixel predicted by the model belongs to the segmentation region; T represents the transpose operation; It represents the probability vector to perform the transpose operation.

[0091] The third step: By applying input Figure X perform input space attention constraint and normalization to obtain the output of the CA module:

[0092] ; Formula (V)

[0093] Among them, Y represents the output of the CA module; X represents the input image; N represents the normalization matrix; T represents the transpose operation; It represents the probability map to perform the transpose operation; It represents the probability map; A represents the spatial attention map.

[0094] In the specific calculation process, we use the associative law of matrix multiplication. First, calculate , and then calculate . The complexity of these two steps of calculation is and , while the complexity of calculating and is and respectively. Using the former calculation is more efficient. At the same time, since the CA module aggregates global information, it may cause context information confusion. Therefore, the output of the CA module enters the balance module (Balance block) to balance local information and global information. The specific calculation is to introduce classification adaptive weights:

[0095] ; Formula (VI)

[0096] ; Formula (VII)

[0097] Among them, represents the classification adaptive weight matrix; diag represents the diagonal matrix; T represents the transpose operation; It represents the probability vector to perform the transpose operation; represents the classification adaptive weight matrix of the control parameter; represents the learnable parameter; represents the activation function.

[0098] Using the classification adaptive weight Obtain the output of the BCA module (Balanced Category attention block, abbreviated as Balance block, i.e., the channel-level adaptive balance module):

[0099] ; Formula (VIII)

[0100] Wherein, represents the output of the BCA module; represents the classification adaptive weight matrix; Y represents the output of the CA module; represents the identity matrix; X represents the input image.

[0101] The output by the BCA module and the input Figure X are concatenated to obtain , and a conv operation (convolution operation) is performed, that is, a convolution kernel with a kernel size of 1 and the number of channels equal to the number of categories is used to change the number of channels. The obtained result is added to the rough segmentation map to output the fine segmentation map .

[0102] In this embodiment, non-local semantic constraint operations are respectively performed on the subretinal fluid initial feature image data of different depth layers (i.e., ) in the subretinal fluid fusion feature image data obtained after the last upsampling operation, that is, the CAR module is used to perform non-local semantic constraint operations on respectively, and the rough segmentation result is used to perform non-local semantic constraint on the fine segmentation result to generate the second subretinal fluid segmentation image data .

[0103] In this embodiment, the method of adding the CAR module to the UNet++ network model is as follows, A rough segmentation feature map with the same resolution and 2 channels is obtained through a convolution layer with a convolution kernel size of 1. is regarded as a rough segmentation of the input image, that is, in the above steps, and enters the CAR module together with to sequentially obtain the first fine segmentation feature map , the second fine segmentation feature map . And so on, the final output of the CAR module is the second subretinal fluid segmentation image data .

[0104] (5) Generate the first fundus predicted lesion area image data according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

[0105] In this embodiment, the first subretinal fluid segmentation image data output by the CAB module and the second subretinal fluid segmentation image data output by the CAR module are added and divided by 2, and then passed through a convolutional layer with a kernel size of 1 and 2 channels as the final output, that is, the first fundus predicted lesion area image data .

[0106] S4032b: Calculate the total loss function of the main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data;

[0107] In this embodiment, the method for calculating the total loss function of the main task learning according to the preset cross-entropy loss function, the preset dice loss function, and the first fundus predicted lesion area image data includes:

[0108] (1) Calculate the output loss function of the main task learning according to the preset cross-entropy loss function and the preset dice loss function.

[0109] In this embodiment, the formula of the cross-entropy loss function is:

[0110] ; Formula (IX)

[0111] Wherein, represents the cross-entropy loss function; represents the predicted probability vector that the i-th pixel point predicted by the model belongs to the segmentation region; represents the original image; represents the true label of the i-th pixel point; represents the number of pixel points in the entire original image.

[0112] In this embodiment, the formula of the dice loss function is:

[0113] ; Formula (X)

[0114] Wherein, represents the dice loss function; represents the true segmentation region; represents the segmentation region predicted by the model; represents the number of pixel points included in the true segmentation region; represents the number of pixel points included in the segmentation region predicted by the model; represents and the number of pixel points in the intersection.

[0115] In this embodiment, the method for calculating the output loss function of the main task learning includes:

[0116] ; Formula (XI)

[0117] Wherein, represents the output loss function of the main task learning; represents the cross-entropy loss function; represents the dice loss function.

[0118] (2) According to the output loss function of the main task learning, the loss function of the analog attention enhancement module and the loss function of the semantic constraint attention module are calculated.

[0119] In this embodiment, the initial characteristic image data of the subretinal fluid of the fifth depth layer is subjected to 4 upsampling operations to successively obtain , and the four outputs of the upsampling path are bilinearly interpolated to the original image size respectively to generate corresponding , and then weighted and added to the output loss function, and the weights are successively: , the method for calculating the loss function of the analog attention enhancement module (CAB module) includes:

[0120] ; Formula (XII)

[0121] Wherein, represents the loss function of the analog attention enhancement module; represents the output loss function of the main task learning; represents the rough feature map of the subretinal fluid region after bilinear interpolation; represents the fine segmentation feature map of the first subretinal fluid region after bilinear interpolation; represents the fine segmentation feature map of the second subretinal fluid region after bilinear interpolation; represents the first subretinal fluid segmentation image data after bilinear interpolation.

[0122] In this embodiment, the rough segmentation feature map , the first fine segmentation feature map , the second fine segmentation feature map , the second subretinal fluid segmentation image data , and the output loss function are weighted and added again, and the weights are successively , the method for calculating the loss function of the semantic constraint attention module (CAR module) includes:

[0123] ; Formula (XIII)

[0124] Among them, represents the loss function of the semantic constraint attention module; represents the output loss function of the main task learning; represents the rough segmentation feature map; represents the first fine segmentation feature map; represents the second fine segmentation feature map; represents the second subretinal fluid segmentation image data.

[0125] (3) According to the first fundus prediction lesion area image data, the output loss function of the main task learning, the loss function of the analogical attention enhancement module, and the loss function of the semantic constraint attention module, calculate the total loss function of the main task learning.

[0126] In this embodiment, the method for calculating the total loss function of the main task learning includes:

[0127] ; Formula (XIV)

[0128] ; Formula (XV)

[0129] Among them, represents the total loss function of the main task learning; represents the loss function of the semantic constraint attention module; represents the loss function of the analogical attention enhancement module; represents the output loss function of the main task learning; represents the final output loss function of the main task learning; represents the first fundus prediction lesion area image data.

[0130] S4032c: Input the historical fundus slice training image data into the UNet network model for branch task learning, and calculate the total loss function of the branch task learning and the branch task learning label.

[0131] In this embodiment, the method for inputting the historical fundus slice training image data into the UNet network model for branch task learning, and calculating the total loss function of the branch task learning and the branch task learning label includes:

[0132] (1) Input the historical fundus slice training image data into the UNet network model for branch task learning to obtain the level set function of the second fundus prediction lesion area image data.

[0133] (2) According to the level set function of the second fundus prediction lesion area image data, calculate the total loss function of the branch task learning and the branch task learning label.

[0134] In this embodiment, the branch task learning is to learn the level set regression task. The training data set is input into the neural network UNet for forward propagation. The UNet network first applies a convolutional module to the input image. The convolutional module is a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Then, it repeats a convolution with a kernel size of 3*3, a stride of 1, a padding of 1, 32 channels, and no bias, followed by batch normalization once and the Relu function once. Subsequently, it is also divided into two processes: downsampling and upsampling. The downsampling process is divided into four downsampling modules in total, and each downsampling module is the same as the downsampling module of the previous UNet++. The upsampling process is also divided into four upsampling modules, and the upsampling module is the same as the upsampling module of the previous UNet++. Finally, a convolutional kernel with a kernel size of 3*3, a stride of 1, a padding of 1, and 1 channel is applied to obtain the level set function of the second fundus predicted lesion area image data. is the level set function predicted by the model, and the true level set function calculated by the model using the label Supervised learning, The expression is as follows:

[0135] ; Formula (XVI)

[0136] Among them, represents the level set function of the second fundus predicted lesion area image data; represents the true level set function; represents the shortest distance from the pixel point x to the boundary of the lesion area; represents the lesion area; represents the boundary of the lesion area; represents the complement of Ω.

[0137] In this embodiment, the branch task learning adopts the L1 norm (i.e., the absolute error) as the loss function. The methods for calculating the total loss function of the branch task learning include:

[0138] ; Formula (XVII)

[0139] Among them, represents the total loss function of the branch task learning; represents the level set function of the second fundus predicted lesion area image data; represents the true level set function; represents the number of pixel points of the entire original image; x represents the pixel point.

[0140] In this embodiment, the total loss function of the sub-task learning is used to measure the difference between the level set function predicted by the model and the true level set function, so that the model can learn how to more accurately predict the fundus lesion area.

[0141] In this embodiment, the method for calculating the sub-task learning label by performing an inverse transformation on the level set function obtained by the sub-task learning includes:

[0142] ; Formula (XVIII)

[0143] where represents the sub-task learning label; x represents the pixel point.

[0144] S4032d: Calculate the enhanced loss function according to the sub-task learning label and the main-task learning label after the softening operation;

[0145] In this embodiment, according to the first fundus predicted lesion area image data , the method for performing a softening operation on the preset main-task learning label includes:

[0146] ; Formula (XIX)

[0147] where represents the main-task learning label after the softening operation; is the sigmoid function; represents the preset main-task learning label.

[0148] In this embodiment, the method for calculating the enhanced loss function by using the difference between the label obtained by the sub-task learning and the softened label obtained by the main-task learning and then applying the L1 norm as the loss function includes:

[0149] ; Formula (XX)

[0150] where represents the enhanced loss function; represents the sub-task learning label; x represents the pixel point; represents the main-task learning label after the softening operation; represents the original image.

[0151] S4032e: Update the parameters of the UNet++ network model and the UNet network model based on multi-semantic supervision according to the total loss function of the main-task learning, the total loss function of the sub-task learning, and the enhanced loss function, so as to construct an initial fundus lesion area prediction model.

[0152] In this embodiment, by introducing branch tasks and enhancing the loss function, the learning of the main task can be effectively assisted, the performance of the UNet++ network model based on multi-semantic supervision can be improved, and a high-performance initial fundus lesion area prediction model can be constructed, making full use of the advantages of multi-task learning and significantly improving the generalization ability and segmentation accuracy of the model.

[0153] S4033: Based on a preset optimization algorithm, sequentially update the parameters of the total loss function of the branch task learning, the enhanced loss function, and the total loss function of the main task learning, and perform iterative training on the initial fundus lesion area prediction model to obtain a finally converged fundus lesion area prediction model.

[0154] In this embodiment, batch stochastic gradient descent is used as the optimization algorithm to update the model parameters through backpropagation. More specifically, first, the total loss function of the branch task learning Updating the parameters through backpropagation helps the model learn global features or boundary information related to the lesion. Then, for the enhanced loss function Updating the parameters through backpropagation helps introduce additional prior knowledge or constraints and improve the robustness of the model. Finally, for the total loss function of the main task learning Updating the parameters helps the model focus on learning the accurate segmentation of the lesion area. The initial learning rate is 0.01, the momentum method is used to accelerate the gradient update, the weight is set to 0.9 for learning rate decay, and the weight is set to 0.0004.

[0155] In this embodiment, the validation dataset is input into the model, and the first fundus predicted lesion area image data output by the model is used as the fundus predicted lesion area validation image data. According to the fundus predicted lesion area validation image data and the real lesion area image data, the validation effect of the model is obtained. Repeat the model training and input of the validation dataset for a total of 50 batches, select the model with the best performance on the validation dataset, and obtain a finally converged fundus lesion area prediction model. Input the test dataset into this model, and the first fundus predicted lesion area image data output by the model is used as the fundus predicted lesion area test image data. According to the fundus predicted lesion area test image data and the real lesion area image data, the test effect of the finally converged fundus lesion area prediction model is obtained.

[0156] In this embodiment, as shown in Table 1, it is the test effect accuracy table:

[0157] Table 1

[0158] To test the effectiveness of the model of the present application and compare it with other different models, in this embodiment, CAB UNet is a UNet model with a CAB module added, CAB UNet++ is a UNet++ model with a CAB module added, CAR UNet++ is a UNet++ model with a CAR module added, and LSEM-SC UNet++ represents a UNet++ model with level set enhanced multi-semantic supervision (Level Set Enhanced Muti-Semantic Constrained UNet++, LSEM-SC UNet++), that is, the final converged fundus lesion area prediction model. More specifically, the main task is a UNet++ with a CAR module and a CAB module, and the branch task is a model for learning the level set function. In terms of the indicators PPV and DSC, the model of the present application achieves the best results. The test indicators are TPVF (True Positive Volume Fraction), DSC (Dice Similarity Coefficient), and PPV (Positive Predicative Value). The definitions are as follows:

[0159] ; Formula (XXI)

[0160] ; Formula (XXII)

[0161] ; Formula (XXIII)

[0162] Among them, TPVF represents the recall rate, which measures the proportion of positive samples correctly predicted by the model among the actual positive samples; DSC represents the overlap rate, which measures the degree of overlap between the prediction result and the true result; PPV represents the accuracy rate, which measures the proportion of samples predicted as positive by the model that are actually positive; represents the number of samples predicted as 1 and actually 1; is the number of samples actually 1; is the number of samples predicted as 1; is the number of samples predicted as 1 and actually 0.

[0163] In this embodiment, by iteratively training the initial fundus lesion area prediction model, the model training and the input of the validation data set are repeated a total of 50 batches, and the model with the highest DSC index on the validation data set is selected to obtain the final converged fundus lesion area prediction model.

[0164] S404: Calculate the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0165] In this embodiment, is the number of real-time fundus slice image data, . The method for calculating the subretinal fluid volume data of the patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes:

[0166] ; Formula (XXIV)

[0167] where V represents the subretinal fluid volume data of the patient; represents the number of real-time fundus slice image data; L represents the actual length of the real-time fundus slice image data; represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; represents the th number of pixel points of the fundus predicted lesion area image data corresponding to the real-time fundus slice image data.

[0168] In this embodiment, based on the pre-constructed fundus lesion area prediction model, the fundus predicted lesion area image is automatically segmented, and the pixel points belonging to the subretinal fluid part in each fundus predicted lesion area image data are summed to obtain the subretinal fluid volume data, which greatly shortens the reading time of fundus doctors, improves work efficiency, and can also significantly improve the accuracy, stability and homogeneity of reading, and can help clinicians achieve accurate and objective evaluation of CSC.

[0169] In this embodiment, the OCT volume scanning mode can be used to generate a series of continuous fundus slice image data. Assuming that a is the scanning line length, in degrees, and the actual length of the real-time fundus slice image data is L, in mm. When the OCT scanning range width is the calculation method of the actual length of the real-time fundus slice image data includes:

[0170] ; Formula (XXV)

[0171] where L represents the actual length of the real-time fundus slice image data; a is the scanning line length.

[0172] In this embodiment, assuming that b is the original width of the scanning area, in degrees, and W is the actual width of the scanning area, in mm. When the OCT scanning range width is the calculation method of the actual width of the scanning area includes:

[0173] ; Formula (XXVI)

[0174] Among them, W represents the actual width of the scanning area; b represents the original width of the scanning area.

[0175] In this embodiment, the calculation method of the interval between each real-time fundus slice image data includes:

[0176] Formula (XXVII)

[0177] Among them, h represents the interval between each real-time fundus slice image data; W represents the actual width of the scanning area; represents the number of real-time fundus slice image data.

[0178] It should be noted that according to Formulas (XXIV) to (XXVII), as long as the length and width of the input scanning area, as well as the number of real-time fundus slice image data are input, after importing the image, the volume data of the subretinal fluid can be automatically calculated.

[0179] It should be noted that the method for segmenting and calculating the volume of subretinal fluid of the present invention has the following advantages:

[0180] (1) By introducing a level set enhancement and multi-semantic supervision mechanism, it is possible to finely segment the fluid accumulation area related to CSC.

[0181] (2) By combining the main task (segmentation) and the branch task (level set function learning), the model can simultaneously capture global semantic information and local detail information, further improving the segmentation accuracy.

[0182] (3) It realizes a fully automated process from image input to subretinal fluid volume calculation, without manual intervention, can quickly complete image segmentation and volume calculation, greatly shortening the doctor's film reading time, not only improving the accuracy, stability and homogeneity of film reading, but also significantly improving the work efficiency of fundus doctors, realizing a precise assessment of the CSC lesion degree, so as to provide a more effective treatment plan for patients.

[0183] (4) By introducing a semantic constraint attention module (CAR module) and an analog attention enhancement module (CAB module), the model shows highly stable performance on different data sets.

[0184] (5) Using a preset optimization algorithm, the parameters of the total loss function of the branch task learning, the enhancement loss function, and the total loss function of the main task learning are updated in sequence to ensure the rapid convergence of the model.

[0185] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects. For example, the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data are only used to distinguish different segmentation image data, and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0186] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0187] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item)" or similar expressions thereof refer to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b or c can mean: a, b, c, a - b, a - c, b - c or a - b - c, where a, b, c can be single or multiple.

[0188] Figure 13 is a schematic block diagram of a subretinal fluid differential segmentation and volume calculation system provided by the embodiments of the present application. As Figure 13 shown, the subretinal fluid differential segmentation and volume calculation system 1300 includes:

[0189] A data acquisition module 1301, configured to acquire real-time fundus image data of a patient.

[0190] A data preprocessing module 1302, configured to preprocess the real-time fundus image data to obtain a plurality of real-time fundus slice image data.

[0191] A lesion area prediction module 1303, configured to obtain fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data according to each of the real-time fundus slice image data and based on a pre-constructed fundus lesion area prediction model.

[0192] A volume calculation module 1304 is configured to calculate the subretinal fluid volume data of a patient according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

[0193] It should be understood that the specific processes for each module to execute the corresponding steps have been described in detail in the above method embodiments. For the sake of brevity, they will not be repeated here.

[0194] It should also be understood that the division of modules in the embodiments of the present application is illustrative, merely a logical function division. In actual implementation, there may be other division methods. Additionally, in each embodiment of the present application, the functional modules can be integrated in one processor, can exist independently physically, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software function modules.

[0195] Figure 14 is a schematic block diagram of an electronic terminal provided by an embodiment of the present application. The electronic terminal includes a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the subretinal fluid segmentation and volume calculation method as described above. As Figure 14 shown, the electronic terminal 1400 includes: at least one processor 1401, a memory 1402, at least one network interface 1403, and a user interface 1405. Each component in the device is coupled together through a bus system 1404. It can be understood that the bus system 1404 is used to realize the connection and communication between these components. The bus system 1404 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 14 all the various buses are labeled as the bus system.

[0196] Among them, the user interface 1405 may include a display, a keyboard, a mouse, a trackball, a click gun, a button, a touchpad, or a touch screen, etc.

[0197] It can be understood that the memory 1402 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiments of the present invention is intended to include but not limited to these and any other suitable categories of memory.

[0198] The memory 1402 in the embodiments of the present invention is used to store various categories of data to support the operation of the electronic terminal 1400. Examples of these data include: any executable program for operating on the electronic terminal 1400, such as the operating system 14021 and the application program 14022; the operating system 14021 contains various system programs, such as the framework layer, the core library layer, the driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 14022 can include various application programs, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. Implementing the subretinal fluid segmentation and volume calculation method provided by the embodiments of the present invention can be included in the application program 14022.

[0199] The method disclosed in the embodiments of the present invention above can be applied to the processor 1401 or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 1401 or instructions in software form. The above-mentioned processor 1401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1401 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 1401 may be a microprocessor or any conventional processor, etc. Combining the steps of the accessory optimization method provided in the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and this storage medium is located in the memory. The processor reads the information in the memory and combines its hardware to complete the steps of the foregoing method.

[0200] In an exemplary embodiment, the electronic terminal 1400 may be an application-specific integrated circuit (ASIC), a DSP, a programmable logic device (PLD), or a complex programmable logic device (CPLD) for executing the foregoing method.

[0201] According to the method provided in the embodiments of the present application, the present application further provides a computer program product, which includes: computer program code, and when the computer program code runs on a computer, it causes the computer to execute Figures 4 to 12 the method of any one of the embodiments shown.

[0202] According to the method provided in the embodiments of the present application, the present application further provides a computer-readable storage medium, which stores program code, and when the program code runs on a computer, it causes the computer to execute Figures 4 to 12 the method of any one of the embodiments shown.

[0203] As used in this specification, the terms "component", "module", "system", etc. are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be components. One or more components can reside in a process and / or thread of execution, and a component can be located on one computer and / or distributed between two or more computers. In addition, these components can execute from various computer-readable media having various data structures stored thereon. A component can communicate, for example, by signals according to one or more data packets (e.g., data from two components interacting with another component in a local system, a distributed system, and / or a network, such as via the Internet interacting with other systems through signals) through local and / or remote processes.

[0204] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether such functions are implemented in hardware or software depends upon the particular application and design constraints of the technical solution. Skilled artisans may implement the described functions in different ways for each particular application, but such implementation should not be considered to exceed the scope of this application.

[0205] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0206] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0209] In the above embodiments, the functions of the functional units can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a high-definition digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD), etc.).

[0210] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0211] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0212] In summary, this application provides a method, system, product, and terminal for subretinal fluid segmentation and volume calculation. Based on a pre-constructed fundus lesion area prediction model, it automatically segments the fundus predicted lesion area image. By summing the pixel points belonging to the subretinal fluid part in the data of each fundus predicted lesion area image, the subretinal fluid volume data is obtained, greatly shortening the reading time of fundus disease doctors and improving work efficiency. In addition, it can significantly improve the accuracy, stability, and homogeneity of reading, and can help clinicians achieve accurate and objective evaluation of CSC. Therefore, this application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0213] The above embodiments are only illustrative of the principles and effects of this application, rather than limiting this application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by this application should still be covered by the claims of this application.

Claims

1. A method for segmenting and calculating the volume of subretinal fluid, characterized in that: include: Acquire real-time fundus image data of patients; Preprocessing the real-time fundus image data to obtain a plurality of real-time fundus slice image data; According to each of the real-time fundus slice image data and based on a pre-constructed fundus lesion area prediction model, the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data is obtained; The patient's subretinal fluid volume data is calculated based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

2. The method for segmenting and calculating the volume of subretinal fluid according to claim 1, characterized in that: The method of calculating the patient's subretinal fluid volume data according to the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data includes: ； Among them, V represents the patient's subretinal fluid volume data; Indicates the number of real-time fundus slice image data; Indicates the actual length of real-time fundus slice image data; represents the fixed depth of the real-time fundus slice image data; h represents the interval between each real-time fundus slice image data; Indicates The number of pixels of the fundus predicted lesion area image data corresponding to the real-time fundus slice image data.

3. The method for segmenting and calculating the volume of subretinal fluid according to claim 1, characterized in that: The method for constructing the fundus lesion area prediction model includes: Acquire historical fundus image data of different patients, and pre-process the historical fundus image data of different patients to construct a training data set; the training data set includes a plurality of historical fundus slice training image data; The historical fundus slice training image data is input into the UNet++ network model based on multiple semantic supervision for main task learning, and is simultaneously input into the UNet network model for branch task learning, and then the total loss function of the main task learning, the total loss function of the branch task learning and the enhanced loss function are calculated to construct an initial fundus lesion area prediction model; Based on a preset optimization algorithm, the parameters of the total loss function of the branch task learning, the enhanced loss function and the total loss function of the main task learning are updated in turn, and the initial fundus lesion area prediction model is iteratively trained to obtain a final converged fundus lesion area prediction model.

4. The method for segmenting and calculating the volume of subretinal fluid according to claim 3, characterized in that: The historical fundus slice training image data is input into a UNet++ network model based on multiple semantic supervision for main task learning, and is simultaneously input into a UNet network model for branch task learning, and then a total loss function for main task learning, a total loss function for branch task learning, and an enhanced loss function are calculated to construct an initial fundus lesion area prediction model, including: Inputting the historical fundus slice training image data into the UNet++ network model based on multiple semantic supervision to perform main line task learning, obtaining first fundus predicted lesion area image data, and performing a softening operation on the preset main line task learning label according to the first fundus predicted lesion area image data; The total loss function of the main task learning is calculated according to the preset cross entropy loss function, the preset dice loss function and the first fundus predicted lesion area image data; Inputting the historical fundus slice training image data into the UNet network model to perform branch task learning, and calculating the total loss function of the branch task learning and the branch task learning label; An enhanced loss function is calculated based on the branch task learning label and the main task learning label after the softening operation; According to the total loss function of the main task learning, the total loss function of the branch task learning and the enhanced loss function, the parameters of the UNet++ network model and the UNet network model based on multiple semantic supervision are updated to construct an initial fundus lesion area prediction model.

5. The method for segmenting and calculating the volume of subretinal fluid according to claim 4, characterized in that: The method of inputting the historical fundus slice training image data into the UNet++ network model based on multiple semantic supervision to perform main line task learning to obtain the first fundus predicted lesion area image data includes: Performing multiple downsampling operations on the historical fundus slice training image data to obtain initial feature image data of subretinal fluid at different depth layers; Performing upsampling operations for different times on the initial feature image data of subretinal fluid at different depth layers, respectively, to obtain fused feature image data of subretinal fluid at different depth layers; Based on the analog attention enhancement module added to the UNet++ network model, the subretinal fluid initial feature image data of the last depth layer is subjected to feature enhancement and fusion operations on the subretinal fluid fusion feature image data obtained after each upsampling operation to generate the first subretinal fluid segmentation image data; Based on the semantic constraint attention module added to the UNet++ network model, non-local semantic constraint operations are performed on the subretinal fluid fusion feature image data obtained after the last upsampling operation of the initial feature image data of subretinal fluid at different depth layers to generate the second subretinal fluid segmentation image data; First fundus predicted lesion area image data is generated according to the first subretinal fluid segmentation image data and the second subretinal fluid segmentation image data.

6. The method for segmenting and calculating the volume of subretinal fluid according to claim 5, characterized in that: According to the preset cross entropy loss function, the preset dice loss function and the first fundus predicted lesion area image data, the method of calculating the total loss function of the main line task learning includes: According to the preset cross entropy loss function and the preset dice loss function, the output loss function of the main task learning is calculated; According to the output loss function of the main task learning, the loss function of the analog attention enhancement module and the loss function of the semantic constraint attention module are calculated; According to the first fundus predicted lesion area image data, the output loss function of the main task learning, the loss function of the analogy attention enhancement module and the loss function of the semantic constraint attention module, the total loss function of the main task learning is calculated.

7. The method for segmenting and calculating the volume of subretinal fluid according to claim 4, characterized in that: The historical fundus slice training image data is input into the UNet network model for branch task learning, and the method of calculating the total loss function of the branch task learning and the branch task learning label includes: Inputting the historical fundus slice training image data into the UNet network model to perform branch task learning, and obtaining the level set function of the second fundus prediction lesion area image data; According to the level set function of the second fundus prediction lesion area image data, the total loss function of the branch task learning and the branch task learning label are calculated.

8. A subretinal fluid area segmentation and volume calculation system, characterized in that: include: A data acquisition module, used to acquire real-time fundus image data of the patient; A data preprocessing module, used for preprocessing the real-time fundus image data to obtain a plurality of real-time fundus slice image data; A lesion area prediction module, used to obtain fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data based on each of the real-time fundus slice image data and a pre-constructed fundus lesion area prediction model; The volume calculation module is used to calculate the patient's subretinal fluid volume data based on the fundus predicted lesion area image data corresponding to each of the real-time fundus slice image data.

9. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the subretinal fluid area segmentation and volume calculation method according to any one of claims 1 to 7.

10. An electronic terminal comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the subretinal fluid area segmentation and volume calculation method according to any one of claims 1 to 7.

Images