Method and apparatus for extracting fundus exudative lesions

Abstract

This disclosure provides a method and apparatus for extracting fundus exudative lesions, comprising: detecting a target eye image using a pre-trained fundus exudative lesion detection model to obtain an initial image of the fundus exudative lesion region; segmenting the initial image using a pre-trained lesion segmentation model to obtain a first image of the fundus exudative lesion; extracting the fundus exudative lesion region from the target eye image based on a preset image processing algorithm to obtain a second image of the fundus exudative lesion region; and performing intersection processing on the initial image, the first image, and the second image to obtain a segmentation result corresponding to the fundus exudative lesion region. By performing intersection processing on the images obtained from the lesion segmentation model and the image processing algorithm respectively, accurate segmentation results of fundus exudative lesions can be obtained, and this accurate segmentation result helps to quickly determine the type of exudative lesion.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for extracting fundus exudative lesions. Background Technology

[0002] Diabetic retinopathy (DR) is the most common eye complication of diabetes and a leading cause of new types of blindness in adults aged 20-65. It has become a key focus of blindness prevention and treatment in the 21st century. Timely and regular DR screenings are crucial for preventing and reducing its progression. Early detection of treatable DR during the non-proliferative phase (before irreversible damage occurs) is key to developing appropriate treatment plans. Therefore, accurately screening diabetic patients without significant visual impairment for DR not only provides an advantage for early diagnosis and treatment but also saves significant social medical resources.

[0003] Diabetic retinopathy (DR) nonproliferative type includes three stages: Stage I presents with microaneurysms; Stage II presents with hard exudates; Stage III presents with cotton wool spots, also known as soft exudates; the nonproliferative stage is accompanied by hemorrhage symptoms. Because both hard exudates and cotton wool spots appear as pale yellow or white patchy or punctate protrusions in color fundus images, and their shape, size, and location are diverse and highly random, they are often collectively referred to as diabetic retinopathy white lesions. In clinical diagnosis, the detection results of hard and soft exudates in the fundus are important references in the diagnosis, monitoring, and treatment of diabetic retinopathy. Currently, the identification and classification of hard and soft exudate lesions usually rely on manual judgment by doctors and other professionals. This not only leads to a heavy workload for doctors but also presents a high difficulty in classifying and identifying lesions, making it prone to misidentification or misclassification due to human factors.

[0004] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a method and apparatus for extracting fundus exudative lesions.

[0006] A first aspect of this disclosure provides a method for extracting fundus exudative lesions, comprising: detecting a target eye image using a pre-trained fundus exudative lesion detection model to obtain an initial image of a fundus exudative lesion region; segmenting the initial image using a pre-trained lesion segmentation model to obtain a first image of the fundus exudative lesion; extracting the fundus exudative lesion region from the target eye image based on a preset image processing algorithm to obtain a second image of the fundus exudative lesion region; and performing intersection processing on the initial image, the first image, and the second image to obtain a segmentation result of the fundus exudative lesion corresponding to the target eye image.

[0007] According to an embodiment of this disclosure, the method further includes: before detecting the target eye image, the method further includes: removing the background region of the eye image to be processed to obtain a first eye image; and normalizing the first eye image to obtain the target eye image.

[0008] According to an embodiment of this disclosure, the initial image is segmented using a pre-trained lesion segmentation model to obtain a first image of fundus exudative lesions, including: expanding each region selected by a detection box in the initial image to obtain expanded regions; cropping image blocks corresponding to the expanded regions; and inputting each image block into the pre-trained lesion segmentation model to obtain the first image of the fundus exudative lesion corresponding to the image block.

[0009] According to embodiments of this disclosure, the fundus exudative lesion area is extracted from the target eye image to obtain a second image of the fundus exudative lesion area, including:

[0010] Segment the target region from the target eye image, ensuring that its brightness, roundness, or edge sharpness meets a preset threshold.

[0011] Remove the target area located within the visual disc area to obtain the target candidate exudation area;

[0012] Based on the different positional relationships between the target candidate exudation area and the vascular area, different preset algorithms are executed to determine the second image of the fundus exudation lesion.

[0013] According to embodiments of this disclosure, determining a second image of a fundus exudative lesion based on different preset algorithms according to the different positional relationships between the target candidate exudative region and the vascular region includes:

[0014] Determine the grayscale difference and color difference between the target candidate exudation area and the surrounding background, as well as the grayscale difference and color difference between the viewing disc area and the surrounding background;

[0015] If the target candidate exudation area and the vascular area meet a preset positional relationship, then based on the magnitude relationship between the grayscale difference and color difference between the target candidate exudation area and the surrounding background, and between the target candidate exudation area and the grayscale difference and color difference between the optic disc area and the surrounding background, a second image of the fundus exudation lesion area is determined; or

[0016] Determine the gradient changes between the target candidate exudation region and its surrounding background, as well as the gradient changes between the viewing disc region and its surrounding background;

[0017] If the target candidate exudation area and the vascular area meet the preset positional relationship, then based on the magnitude relationship between the gradient change of the target candidate exudation area and the surrounding background, and the gradient change of the optic disc area and the surrounding background, a second image of the fundus exudation lesion area is determined.

[0018] According to an embodiment of this disclosure, before segmenting the initial image using a pre-trained lesion segmentation model, the method further trains the lesion segmentation model, including: acquiring a first sample set, wherein the samples in the first sample set are sample images of fundus exudative lesions; determining sample label images corresponding to the samples in the first sample set by label annotation; and inputting the sample images and the sample label images corresponding to the samples into the pre-established lesion segmentation model to train the pre-established lesion segmentation model.

[0019] According to an embodiment of this disclosure, the method further includes: extracting the boundary of the fundus exudative lesion segmentation result to obtain the boundary of the fundus exudative lesion; and overlaying the boundary onto the target eye image to obtain an overlaid fundus image of the fundus exudative lesion.

[0020] A second aspect of this disclosure provides a device for extracting fundus exudative lesions, comprising: a detection module configured to detect a target eye image using a pre-trained fundus exudative lesion detection model to obtain an initial image of a fundus exudative lesion region; a first segmentation module configured to segment the initial image using a pre-trained lesion segmentation model to obtain a first image of the fundus exudative lesion; a second segmentation module configured to extract the fundus exudative lesion region from the target eye image based on a preset image processing algorithm to obtain a second image of the fundus exudative lesion region; and a calculation module configured to perform intersection processing on the first image and the second image to obtain a segmentation result corresponding to the fundus exudative lesion region.

[0021] According to an embodiment of this disclosure, the apparatus further includes: a processing module configured to remove the background region of the eye image to be processed to obtain a first eye image; and to perform normalization processing on the first eye image to obtain the target eye image.

[0022] A third aspect of this disclosure provides a device for extracting fundus exudative lesions, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

[0023] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0024] The method and apparatus for extracting fundus exudative lesions in this embodiment include: detecting a target eye image using a pre-trained fundus exudative lesion detection model to obtain an initial image of the fundus exudative lesion region; segmenting the initial image using a pre-trained lesion segmentation model to obtain a first image of the fundus exudative lesion; extracting the fundus exudative lesion region from the target eye image based on a preset image processing algorithm to obtain a second image of the fundus exudative lesion region; and performing intersection processing on the initial image, the first image, and the second image to obtain the segmentation result corresponding to the fundus exudative lesion region. By performing intersection processing on the images processed by the lesion segmentation model and the image processing algorithm respectively, accurate segmentation results of fundus exudative lesions can be obtained, and this accurate segmentation result helps to quickly determine the type of exudative lesion. This solves the problem that in related technologies, the identification and classification of hard and soft exudative lesions usually rely on manual judgment by doctors and other professionals. This not only leads to a heavy workload for doctors, but also makes the classification and identification of lesions difficult and prone to misidentification or misclassification due to human factors. Attached Figure Description

[0025] Figure 1 A schematic flowchart of a method for extracting fundus exudative lesions according to an embodiment of the present disclosure is shown as an example;

[0026] Figure 2 A schematic diagram of the structure of the YOLOv5 detection network according to an embodiment of this disclosure is shown as an example;

[0027] Figure 3 A schematic diagram of a fundus exudative lesion detection frame according to an embodiment of the present disclosure is shown as an example;

[0028] Figure 4 A schematic diagram of an eye image exemplarily illustrating an embodiment of the present disclosure is shown;

[0029] Figure 5 A schematic diagram of image blocks and label images of an embodiment of this disclosure is shown as an example;

[0030] Figure 6An exemplary diagram of the lesion segmentation model according to an embodiment of this disclosure is shown.

[0031] Figure 7 A schematic diagram illustrating the division boundary of an embodiment of this disclosure is shown as an example;

[0032] Figure 8 A schematic diagram illustrating the segmentation result of an embodiment of this disclosure is shown as an example;

[0033] Figure 9 A block diagram of an illustrative embodiment of the fundus exudative lesion extraction device is shown;

[0034] Figure 10 This is a block diagram illustrating a device for extracting fundus exudative lesions according to an exemplary embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0037] It should be understood that in the various embodiments of this disclosure, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0038] It should be understood that in this disclosure, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0039] It should be understood that in this disclosure, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "and / or B" can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0040] It should be understood that in this disclosure, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0041] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0042] The technical solutions of this disclosure will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0043] Figure 1 An exemplary flowchart of a method for extracting fundus exudative lesions according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the method includes:

[0044] Step S101: The target eye image is detected by a pre-trained fundus exudative lesion detection model to obtain an initial image of the fundus exudative lesion area.

[0045] In this embodiment, fundus exudation is an ocular lesion. Specifically, fundus exudation includes hard exudation and soft exudation. Hard exudation is common in diabetic retinopathy. When the disease progresses to the mid-non-proliferative stage of diabetic retinopathy, i.e., stage II, hemorrhage, exudation, and punctate yellow substances will appear in the fundus. This is mainly due to the destruction of the vascular barrier by the toxic effects of high blood sugar, followed by the leakage of lipid and protein components from the blood vessels into the retina, leaving yellow punctate exudates. Soft exudation is common in hypertensive retinopathy, diabetic retinopathy, and other conditions. Soft exudation is not an exudate of fluid, but rather a result of retinal circulatory disorders. Cotton wool spots, like cotton or cotton wool adhering to the retina, can be seen in the fundus. That is, hard exudation and soft exudation have different characteristics and boundaries. Hard exudation lesions have sharper boundaries, while soft exudation has a cotton wool-like shape and less sharp boundaries.

[0046] For example, during a medical examination of the eye, an image of the eye to be processed can be acquired. For instance, the image of the eye to be processed can be obtained as the target eye image by means of fundus camera, fundus fluorescein angiography, or optical coherence tomography.

[0047] Furthermore, the pre-trained fundus exudative lesion detection model is used to analyze the target eye image and detect the location of fundus exudative lesions. For each fundus exudative lesion, a fundus lesion detection box with a fundus exudative lesion type label and capable of selecting the fundus exudative lesion area is generated. The initial image in this embodiment is obtained by superimposing the fundus lesion detection boxes onto the target eye image; each fundus lesion detection box includes the type information of the fundus exudative lesion it selects. The fundus lesion detection box can be the smallest bounding rectangle of the exudative lesion, or it can be a detection box of other shapes, such as a circle, a triangle, etc. The initial image can include the fundus exudative lesion area selected by the detection box. Each detection box area in the initial image can also correspond to the type of fundus exudative lesion, which can include hard exudation or soft exudation. The type of fundus exudative lesion can be displayed when the mouse pointer moves over the detection box. The detection model for fundus exudative lesions can be the YOLOv5 detection network. This network can be used to extract the detection boxes for fundus exudative lesions and the lesion label type corresponding to each detection box.

[0048] For example, refer to Figure 2The diagram shows the structure of the YOLOv5 detection network in this embodiment. The CBL module consists of Conv+BN+Leaky_relu activation functions; Res-unit: borrowing from the residual structure in ResNet, used to construct deep networks, with CBM being a sub-module within the residual module; CSP1_X: borrowing from the CSPNet network structure, consisting of CBL, Res-unit, convolutional layers, and Concaten; CSP2_X: borrowing from the CSPNet network structure, consisting of convolutional layers multiplied by X Res-unit modules and then Concatenated; FOCUS: concatenating multiple slice results and feeding them into the CBL module; SPP: using 1x1, 5x5, 9x9, and 13x13 max pooling methods for multi-scale fusion.

[0049] The improvements to the network structure in this embodiment include:

[0050] (1) Input end: During the model training phase, Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling are used;

[0051] (2) Baseline network: FOCUS and CSP structures were used;

[0052] (3) Neck network: An FPN_PAN structure is inserted between the Backbone and the final Head output layer;

[0053] (4) Head output layer: loss function GIOU_Loss during training, DIOU_nns for predicting the filter box;

[0054] CutMix: Combines two images.

[0055] Mosaic: An improvement on CutMix, it uses four images and stitches them together using random scaling, random cropping, and random arrangement.

[0056] Advantages: Combining several images into one not only enriches the dataset and greatly improves the network training speed, but also reduces model memory usage; furthermore, the input image is a 416*416 three-channel color fundus image, and the label file is in XML format.

[0057] Furthermore, the YOLOv5 detection network in this embodiment can obtain detection boxes for many candidate regions. Non-maximum suppression can be used to remove redundant detection boxes, and finally the initial image of the fundus exudative lesion region can be obtained.

[0058] refer to Figure 3 The diagram illustrates, exemplarily, a detection frame for fundus exudative lesions according to an embodiment of the present disclosure. Figure 3As shown, each detection box is a detection box for fundus exudative lesions, used to select fundus exudative lesions.

[0059] As an optional implementation of this embodiment, before detecting the target eye image, the method further includes: removing the background region of the eye image to be processed to obtain a first eye image; and normalizing the first eye image to obtain the target eye image.

[0060] In this optional implementation, Figure 4 A schematic diagram of an eye image, exemplarily illustrating an embodiment of the present disclosure, is shown below. Figure 4 As shown, due to factors such as the shape of the lens, a portion of the background area may exist in the eye image to be processed, for example, Figure 4 The black background area in the image lacks information about the eye and may interfere with model calculations. Furthermore, due to the varying resolutions of lenses, the size and resolution of the eye images to be processed may also differ. All of these factors can negatively impact the processing of eye images and the identification and classification of lesions.

[0061] To mitigate the aforementioned adverse effects, the background region of the eye image to be processed can be removed, and the first eye image after background removal can be scaled to a preset size to obtain the target eye image. This processing reduces interference from the background region and ensures that the target eye images processed by the model are of consistent size, thus improving the model's robustness in processing, as well as its robustness in lesion identification and classification.

[0062] In the example, a ROI (region of interest), i.e., the non-background region, can be determined in the eye image to be processed. This non-background region is then cropped to obtain the first eye image. The first eye image can then be normalized to obtain a target eye image of uniform size. Normalization can include, but is not limited to, translation, rotation, and scaling, such as scaling to a size of 512×512.

[0063] Step S102: The initial image is segmented using a pre-trained lesion segmentation model to obtain the first image of the fundus exudative lesion.

[0064] In this embodiment, a lesion segmentation model can be pre-established and pre-trained. Based on the trained lesion segmentation model, when the initial image containing step S101 is input, the trained lesion segmentation model can segment the fundus exudative lesion area and output the fundus exudative lesion area in each detection frame, i.e., the first image. The first image includes a first segmentation boundary and the area enclosed by the first segmentation boundary.

[0065] As an optional implementation of this embodiment, the initial image is segmented using a pre-trained lesion segmentation model to obtain a first image of fundus exudative lesions. This includes: expanding each region selected by the detection box in the initial image to obtain expanded regions; cropping image blocks corresponding to the expanded regions; and inputting the image blocks into the pre-trained lesion segmentation model to obtain the first image of the fundus exudative lesions corresponding to each image block.

[0066] In this optional implementation, the detection box contains limited information, and its edges may overlap with the edges of fundus exudative lesions, making direct segmentation difficult. Therefore, the detection box can be expanded, for example, by expanding it upwards, downwards, leftwards, and rightwards to obtain an expanded area. This enlarges the detection box, increases the amount of information, and reduces the possibility of overlap with lesion edges. After expanding the area selected by the detection box, image blocks of fundus exudative lesions within the expanded area can be extracted, and each image block can be further segmented using a lesion segmentation model. By pre-extracting image blocks and then accurately segmenting each image block using a lesion segmentation model, the accuracy of fundus exudative lesion segmentation can be improved. After segmentation by the lesion segmentation model, the segmentation boundaries in each image block and the areas enclosed by the segmentation boundaries can be obtained. The set of segmentation boundaries in each image block constitutes the first segmentation boundary.

[0067] The above processing can improve the segmentation accuracy of the lesion segmentation model for fundus exudative lesions, thereby obtaining a first segmentation boundary with higher accuracy.

[0068] For example, refer to Figure 5 , Figure 5 The leftmost image in the text is an image block. Figure 5 The rightmost image in the document is the labeled image corresponding to the fundus exudative lesion area shown in the image patch. (Reference) Figure 6 , Figure 6 This is the first image output after processing by the lesion segmentation model.

[0069] Step S103: Based on a preset image processing algorithm, extract the fundus exudative lesion area from the target eye image to obtain a second image of the fundus exudative lesion area.

[0070] In this embodiment, by Figure 6It can be intuitively seen that the lesion segmentation model, such as Unet, extracts a relatively coarse segmentation result boundary. Therefore, in order to make the boundary of the extracted fundus exudative lesion more refined, the following method can be used to achieve fine segmentation: perform boundary detection processing on the target eye image through other computer vision methods to obtain a second image, which includes a second segmentation boundary and the area enclosed by the second segmentation boundary; and in step S104, based on the second image obtained by boundary detection and the first image obtained by the lesion segmentation model, determine the final segmentation result, that is, the final fundus exudative lesion segmentation boundary and the area enclosed by the fundus exudative lesion segmentation boundary.

[0071] As an optional implementation of this embodiment, based on a preset image processing algorithm, the fundus exudative lesion area is extracted from the target eye image to obtain a second image of the fundus exudative lesion area. This includes: segmenting a target area from the target eye image whose brightness, roundness, or edge sharpness meets a preset threshold; removing the optic disc area within the target area to obtain a target candidate exudative area; and executing different preset algorithms to determine the second image of the fundus exudative lesion based on the different positional relationships between the target candidate exudative area and the vascular area.

[0072] In this optional implementation, a Region of Interest (ROI) image can be extracted from the target eye image. Histogram equalization is then applied to the ROI image to enhance its grayscale range, expanding it to 0-255. Dynamic thresholding is performed on the green channel image (or other channels can be used for dynamic thresholding as needed; using the green channel image for dynamic thresholding provides more accurate segmentation compared to other channels), resulting in the brighter areas of the image, i.e., the target region. The dynamic threshold setting can depend on the image's sharpness. Then, the target region located within the optic disc region is removed to obtain the target candidate exudative region. It should be understood that this is only one way to determine the second image of the fundus exudative lesion; it can also be achieved by first removing the optic disc region and then segmenting the target region whose brightness, roundness, or edge sharpness meets a preset threshold.

[0073] Furthermore, after determining the target candidate exudation area, the vascular region can be extracted from the target eye image. After extracting the vascular region, the main vascular arch region can be further obtained. Based on the different positional relationships between the target candidate exudation area and the main vascular arch region, different preset algorithms are executed.

[0074] As an optional implementation of this embodiment, determining the second image of the fundus exudative lesion by executing different preset algorithms based on the different positional relationships between the target candidate exudative region and the vascular region includes: determining the second image of the fundus exudative lesion by executing different preset algorithms based on the judgment result includes:

[0075] Determine the grayscale difference and color difference between the target candidate exudation area and the surrounding background, as well as the grayscale difference and color difference between the viewing disc area and the surrounding background;

[0076] If the target candidate exudation area and the vascular area meet a preset positional relationship, then a second image of the fundus exudation lesion area is determined based on the magnitude relationship between the grayscale difference and color difference between the target candidate exudation area and the surrounding background, and between the grayscale difference and color difference between the target candidate exudation area and the surrounding background, and between the target candidate exudation area and the surrounding background; or the gradient change between the target candidate exudation area and the surrounding background, and the gradient change between the target candidate exudation area and the surrounding background; if the target candidate exudation area and the vascular area meet a preset positional relationship, then a second image of the fundus exudation lesion area is determined based on the magnitude relationship between the gradient change between the target candidate exudation area and the surrounding background, and between the target candidate exudation area and the surrounding background, and between the target candidate exudation area and the surrounding background.

[0077] In this optional implementation, the situation where the target candidate exudation region and the vascular region meet the preset positional relationship may include the following:

[0078] a) The target candidate exudation area is located within the main vascular arch area and is within a first preset distance from the blood vessel (i.e., relatively close to the blood vessel).

[0079] Once a) is satisfied, the second image of the fundus exudative lesion area is determined based on the first preset method.

[0080] b) The target candidate exudation area is within the main vascular arch area and the distance from the blood vessel is not within the first preset distance. It can be between the first preset distance and the second preset distance, where the second preset distance is greater than the first preset distance (i.e., farther from the blood vessel).

[0081] Once b) is satisfied, the second image of the fundus exudative lesion area is determined based on the second preset method.

[0082] c) The target candidate exudation area is not within the main vascular arch area, and the distance between the target candidate exudation area and the blood vessel is within the first preset distance range (i.e., relatively close to the blood vessel).

[0083] Once c) is satisfied, a second image of the fundus exudative lesion area can be determined based on the third preset method.

[0084] d) The target candidate exudation area is not within the main vascular arch area, and the distance between the target candidate exudation area and the blood vessel is not within the first preset distance. It can be between the first preset distance and the second preset distance, where the second preset distance is greater than the first preset distance (i.e., farther from the blood vessel).

[0085] Once d) is satisfied, a second image of the fundus exudative lesion area can be determined based on the fourth preset method.

[0086] Furthermore, the first, second, third, or fourth preset methods are all set based on the magnitude relationship between the grayscale difference and color difference between the target candidate exudation region and the surrounding background, and between the grayscale difference and color difference between the optic disc region and the surrounding background. It can be understood that when determining the grayscale difference and color difference between the optic disc region and the surrounding background, a Region of Interest (ROI) image can be extracted from the target eye image, and the optic disc, blood vessels, and main vascular arch regions can be extracted from the ROI image. Then, the grayscale difference and color difference between the optic disc region and the surrounding background are calculated. Similarly, when determining the grayscale difference and color difference between the target candidate exudation region and the surrounding background, it can be obtained by calculating the grayscale difference and color difference between the target candidate exudation region and the surrounding background after obtaining the target candidate exudation region.

[0087] For example, the first preset method may be to determine that the grayscale difference and color difference between the target candidate exudation area and the surrounding background are greater than a preset multiple, such as 2 / 3, of the grayscale difference and color difference between the optic disc area and the surrounding background, and then take the target candidate exudation area that meets the condition as the fundus exudation area to obtain the second image of the fundus exudation lesion.

[0088] The second preset method can be to determine that the grayscale difference and color difference between the target candidate exudation area and the surrounding background are greater than a preset multiple, such as 1 / 2, of the grayscale difference and color difference between the optic disc area and the surrounding background. The target candidate exudation area that meets this condition is taken as the fundus exudation area, and a second image of the fundus exudation lesion is obtained.

[0089] The third preset method can be to determine that the grayscale difference and color difference between the target candidate exudation area and the surrounding background are greater than a preset multiple, such as 1 / 2, than the grayscale difference and color difference between the optic disc area and the surrounding background. The target candidate exudation area that meets this condition is taken as the fundus exudation area, and a second image of the fundus exudation lesion is obtained.

[0090] The fourth preset method can be to determine that the grayscale difference and color difference between the target candidate exudation area and the surrounding background are greater than a preset multiple, such as 1 / 3, of the grayscale difference and color difference between the optic disc area and the surrounding background. The target candidate exudation area that meets this condition is taken as the fundus exudation area, and a second image of the fundus exudation lesion is obtained.

[0091] For example, the first preset method may be to determine that the gray gradient change between the target candidate exudation area and the surrounding background is greater than the gradient change between the optic disc area and the surrounding background by a preset multiple, such as 2 / 3, and then take the target candidate exudation area that meets the condition as the fundus exudation area to obtain the second image of the fundus exudation lesion.

[0092] The second preset method can be to determine that the gradient change between the target candidate exudation area and the surrounding background is greater than a preset multiple, such as 1 / 2, than the gradient change between the optic disc area and the surrounding background. The target candidate exudation area that meets this condition is taken as the fundus exudation area, and a second image of the fundus exudation lesion is obtained.

[0093] The third preset method can be to determine that the gradient change between the target candidate exudation area and the surrounding background is greater than a preset multiple, such as 1 / 2, of the gradient change between the optic disc area and the surrounding background, and then take the target candidate exudation area that meets this condition as the fundus exudation area to obtain the second image of the fundus exudation lesion.

[0094] The fourth preset method can be to determine that the gradient change between the target candidate exudation area and the surrounding background is greater than a preset multiple, such as 1 / 3, of the gradient change between the optic disc area and the surrounding background, and then take the target candidate exudation area that meets this condition as the fundus exudation area to obtain the second image of the fundus exudation lesion.

[0095] Furthermore, after determining the second image of the fundus exudative lesion through the third and fourth preset methods, the type of exudative lesion can be further determined. For example, after obtaining the fundus exudative lesion, the distance between the area and the already determined exudative area in the image can be determined. If it is relatively close (a certain value can be set for judgment), it is an aggregated exudate.

[0096] Using the above methods, the second segmentation boundary (the area enclosed by the second segmentation boundary and the second segmentation boundary is the second image) and the first segmentation boundary obtained by the fundus exudative lesion segmentation model are obtained.

[0097] Step S104: Perform intersection processing on the initial image, the first image, and the second image to obtain the segmentation result corresponding to the fundus exudative lesion area.

[0098] In the embodiments of this disclosure, the first image may have segmentation errors, so the first image can be corrected using the second image. The segmentation result is obtained by intersecting the second image (including the second segmentation boundary and the area enclosed by the second segmentation boundary) obtained based on a preset image processing algorithm, and the first image (i.e., the image containing the aforementioned first segmentation boundary and the area enclosed by the first segmentation boundary) obtained through the fundus exudative lesion segmentation model. Further, the first image segmented by the lesion segmentation model is determined based on the area selected by the expanded detection box. To ensure that the final segmentation result is the fundus exudative lesion within the area selected by the detection box in the initial image, the initial image and the first image can be intersected to obtain a segmented image of the area selected by the detection box in the initial image. Then, this segmented image is intersected with the second image to obtain the final accurate segmentation result. Alternatively, the first and second images can be intersected before intersecting with the initial image. The order of the intersection operations is not limited here; the ultimate goal is to achieve intersection processing among the three images. By taking the intersection of the three, we can obtain the final segmentation boundary of the fundus exudative lesion and the image of the region enclosed by the final segmentation boundary of the fundus exudative lesion.

[0099] By taking the intersection processing step, the error-prone part in the first segmentation boundary of the lesion segmentation model can be automatically removed, thereby reducing the error of the first segmentation boundary and obtaining a more accurate segmentation boundary for fundus exudative lesions.

[0100] As an optional implementation of this embodiment, the method further includes: extracting the boundary of the fundus exudative lesion segmentation result to obtain the boundary of the fundus exudative lesion; and overlaying the boundary onto the target eye image to obtain an overlaid fundus image of the fundus exudative lesion.

[0101] In this optional implementation, since the segmentation result of fundus exudative lesions is a black and white binary image, in order to facilitate observation, this optional implementation can process the binary image into a line drawing with only the lesion boundary, and then superimpose the lesion boundary onto the target eye image to obtain the final result. Finally, the result can be output to a display device for display.

[0102] As an optional implementation of this embodiment, before segmenting the initial image using a pre-trained lesion segmentation model, the method further trains the lesion segmentation model, including: acquiring a first sample set, wherein the samples in the first sample set are sample images of fundus exudative lesions; determining the sample label images corresponding to the samples in the first sample set by label annotation; and inputting the sample images and the sample label images corresponding to the samples into the pre-established lesion segmentation model to train the pre-established lesion segmentation model.

[0103] In this optional implementation, the image of the fundus exudative lesion can be used as the sample image, and its corresponding label image (such as a binary image containing the background area and the fundus exudative lesion area) can be determined by label annotation; the two are then input into the lesion segmentation model to complete the training of the model.

[0104] Furthermore, since there is an error between the segmentation result of the lesion segmentation model and the labeled image, the loss function of the fundus exudative lesion segmentation model is determined by the error between the lesion segmentation result and the labeled image during the training of the segmentation model. For example, the loss function can be calculated by the positional error between the segmentation boundaries. The loss function can then be backpropagated to adjust the parameters of the fundus exudative lesion segmentation model based on gradient descent, thereby reducing the loss function. This training process is then iterated multiple times until training conditions are met, such as loss function convergence; or reaching a set number of training iterations; or the accuracy on the validation set meeting the precision requirements. This disclosure does not impose any restrictions on the training conditions. After the training conditions are met, the trained fundus exudative lesion segmentation model can be obtained.

[0105] As an optional implementation method in this embodiment, the pre-established lesion segmentation model is a u-net network structure.

[0106] This embodiment pre-detects fundus exudative lesions and then segments only these lesions, improving segmentation accuracy. A second image obtained through computer vision methods is used to correct the first image obtained from the fundus exudative lesion segmentation model, further enhancing the accuracy of the segmentation results. Furthermore, during the training process of the lesion segmentation model, boundary detection is used to assist annotators, reducing their workload and improving annotation quality, thereby increasing the accuracy of the fundus exudative lesion segmentation model and ultimately improving the accuracy of the segmentation results.

[0107] Figure 7 A schematic diagram illustrating the division boundary of an embodiment of this disclosure is shown as an example. Figure 7 As shown, the boundary line surrounding each fundus exudative lesion is the segmentation boundary.

[0108] According to embodiments of this disclosure, the segmentation result of the fundus exudative lesion can be obtained through the segmentation boundary. In the example, the segmentation result can be a binary image corresponding to the target eye image, for example, a binary image with the same size as the target eye image, and a pixel value of 1 in the area where the fundus exudative lesion is located (i.e., the area within the segmentation boundary), and a pixel value of 0 in other areas. This disclosure does not limit the form of the segmentation result.

[0109] Figure 8A schematic diagram illustrating the segmentation result of an embodiment of this disclosure is shown as an example, such as... Figure 8 As shown, the segmentation result can be a binary image where the pixel value of the area where the fundus exudative lesion is located is 1, and the pixel value of other areas is 0.

[0110] In this embodiment, the fundus exudative lesion detection model can also be trained. The training samples can be images containing fundus exudative lesions. Before training, the types of fundus exudative lesions can be manually labeled in advance. However, manually labeling the boundaries of fundus exudative lesions in eye image samples is labor-intensive and has a high labeling error rate.

[0111] In the training process of the fundus exudative lesion detection model, to avoid the problems of high workload and high annotation error rate caused by directly annotating the eye image samples, the following steps can be adopted: Boundary detection processing is performed on the eye image samples to obtain the third segmentation boundary of the eye image samples; selection processing of the third segmentation boundary is received to obtain the annotation information corresponding to the third segmentation boundary. After boundary detection, manual selection of the boundary of fundus exudative lesions from the known third boundary is assisted for annotation, thereby avoiding the problems of high workload and high annotation error rate caused by directly annotating the eye image samples, and improving the accuracy of annotation.

[0112] During the training of the fundus exudative lesion detection model, boundary detection is used to assist the annotators, reducing the workload of annotation, improving the annotation quality, and thus improving the accuracy of the fundus exudative lesion segmentation model, which can further improve the accuracy of the segmentation results.

[0113] Furthermore, the boundary detection method can be similar to the above-mentioned brightness normalization and / or color normalization and threshold segmentation processing, and this disclosure does not limit it.

[0114] Figure 9 A block diagram of an illustrative embodiment of a fundus exudative lesion extraction device is shown, the device comprising:

[0115] The detection module is configured to detect the target eye image using a pre-trained fundus exudative lesion detection model to obtain an initial image of the fundus exudative lesion region.

[0116] The first segmentation module is configured to segment the initial image using a pre-trained lesion segmentation model to obtain a first image of fundus exudative lesions.

[0117] The second segmentation module is configured to extract the fundus exudative lesion area from the target eye image based on a preset image processing algorithm, and obtain a second image of the fundus exudative lesion area.

[0118] The calculation module is configured to perform intersection processing on the initial image, the first image, and the second image to obtain the segmentation result corresponding to the fundus exudative lesion area.

[0119] As an optional implementation of this embodiment, the device further includes:

[0120] The processing module is configured to remove the background region of the eye image to be processed to obtain a first eye image; and to perform normalization processing on the first eye image to obtain the target eye image.

[0121] As an optional implementation of this embodiment, the initial image is segmented using a pre-trained lesion segmentation model to obtain a first image of the fundus exudative lesion. This includes: expanding each region selected by the detection box in the initial image to obtain expanded regions; cropping the image block corresponding to the expanded region; and inputting each image block into the pre-trained lesion segmentation model to obtain the first image of the fundus exudative lesion corresponding to the image block.

[0122] As an optional implementation of this embodiment, the target eye image is extracted from the fundus exudative lesion area based on a preset image processing algorithm to obtain a second image of the fundus exudative lesion area. This includes: performing multi-scale filtering, brightness normalization and / or color normalization on the target eye image in sequence to obtain an enhanced image; and performing threshold segmentation on the enhanced image to obtain the second image.

[0123] As an optional implementation of this embodiment, before segmenting the initial image using a pre-trained lesion segmentation model, the method further trains the lesion segmentation model, including: acquiring a first sample set, wherein the samples in the first sample set are sample images of fundus exudative lesions; determining the sample label images corresponding to the samples in the first sample set by label annotation; and inputting the sample images and the sample label images corresponding to the samples into the pre-established lesion segmentation model to train the pre-established lesion segmentation model.

[0124] As an optional implementation method in this embodiment, the pre-established lesion segmentation model is a u-net network structure.

[0125] Figure 10This is a block diagram illustrating a fundus exudative lesion extraction device according to an exemplary embodiment. For example, the device 1600 may be provided as a terminal or server. The device 1600 includes a processing component 1602 and memory resources represented by a memory 1603 for storing instructions executable by the processing component 1602, such as application programs. The application programs stored in the memory 1603 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1602 is configured to execute instructions to perform the described method.

[0126] Device 1600 may also include a power supply component 1606 configured to perform power management of device 1600, a wired or wireless network interface 1605 configured to connect device 1600 to a network, and an input / output (I / O) interface 1608. Device 1600 can operate on an operating system stored in memory 1603, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0127] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0131] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] Note that, unless otherwise explicitly stated, all features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features for achieving the same, equivalent, or similar purpose. Therefore, unless explicitly stated otherwise, each disclosed feature is merely one example of a set of equivalent or similar features. Where used, "further," "preferably," "even further," and "more preferably" are simple starting points for describing another embodiment based on the foregoing embodiments, the combination of which with the foregoing embodiments constitutes the complete configuration of another embodiment. Any combination of several "further," "preferably," "even further," or "more preferably" settings following the same embodiment constitutes yet another embodiment.

[0136] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. An extraction method of an exudation lesion of an ocular fundus, characterized by, The method comprises the following steps: detecting the target eye image by using a pre-trained exudation lesion detection model to obtain an initial image of the exudation lesion region of the fundus; segmenting the initial image by using a pre-trained lesion segmentation model to obtain a first image of the exudation lesion of the fundus, which comprises the following steps: expanding each region in the initial image framed by the detection frame to obtain an expanded region, intercepting an image block corresponding to the expanded region, and inputting each image block into the pre-trained lesion segmentation model to obtain a first image of the exudation lesion of the fundus corresponding to the image block; extracting the exudation lesion region of the fundus from the target eye image based on a preset image processing algorithm to obtain a second image of the exudation lesion of the fundus; performing intersection processing on the initial image, the first image and the second image to obtain the segmentation result of the exudation lesion of the fundus corresponding to the target eye image.

2. The method according to claim 1, wherein Before detecting the target eye image, the method further comprises the following steps: removing the background region of the eye image to be processed to obtain a first eye image; performing normalization processing on the first eye image to obtain the target eye image.

3. The method according to claim 1, wherein extracting the exudation lesion region of the fundus from the target eye image based on a preset image processing algorithm to obtain a second image of the exudation lesion region of the fundus, which comprises the following steps: segmenting a target region with brightness, circularity or edge sharpness meeting a preset threshold from the target eye image; removing the target region located in the optic disc region to obtain a target candidate exudation region; based on different positional relationships between the target candidate exudation region and the blood vessel region, executing different preset algorithms to determine the second image of the exudation lesion of the fundus.

4. The method according to claim 3, wherein based on different positional relationships between the target candidate exudation region and the blood vessel region, executing different preset algorithms to determine the second image of the exudation lesion of the fundus, which comprises the following steps: determining the gray scale difference and color difference between the target candidate exudation region and the surrounding background of the target candidate exudation region, and the gray scale difference and color difference between the optic disc region and the surrounding background of the optic disc region; if the target candidate exudation region meets the preset positional relationship with the blood vessel region, determining the second image of the exudation lesion region of the fundus based on the size relationship between the gray scale difference and color difference between the target candidate exudation region and the surrounding background, and the gray scale difference and color difference between the optic disc region and the surrounding background of the optic disc region; or determining the gradient change of the target candidate exudation region and the surrounding background of the target candidate exudation region, and the gradient change of the optic disc region and the surrounding background of the optic disc region; if the target candidate exudation region meets the preset positional relationship with the blood vessel region, determining the second image of the exudation lesion region of the fundus based on the size relationship between the gradient change of the target candidate exudation region and the surrounding background, and the gradient change of the optic disc region and the surrounding background.

5. The method according to claim 1, wherein Before segmenting the initial image by using the pre-trained lesion segmentation model, the method further trains the lesion segmentation model, which comprises the following steps: obtaining a first sample set, wherein the samples in the first sample set are sample images of the exudation lesion of the fundus; Determine a sample label image corresponding to a sample in the first sample set by a label annotation manner; Input the sample image and the sample label image corresponding to the sample into a pre-established lesion segmentation model to train the pre-established lesion segmentation model.

6. The method according to claim 1, wherein The method further comprises: Perform boundary extraction on the exudation lesion segmentation result to obtain a boundary of the exudation lesion of the fundus; Superimpose the boundary on the target eye image to obtain an exudation lesion superimposed fundus image of the fundus.

7. An ocular exudation lesion extraction device characterized by comprising: Comprise: A detection module configured to detect a target eye image by a pre-trained exudation lesion detection model of the fundus to obtain an initial image of an exudation lesion region of the fundus; A first segmentation module configured to segment the initial image by a pre-trained lesion segmentation model to obtain a first image of the exudation lesion of the fundus, comprising: Respectively, expand each region framed by a detection frame in the initial image to obtain an expanded region; intercept an image block corresponding to the expanded region; input each image block into the pre-trained lesion segmentation model to obtain a first image of the exudation lesion of the fundus corresponding to the image block; A second segmentation module configured to extract an exudation lesion region of the fundus from the target eye image based on a pre-set image processing algorithm to obtain a second image of the exudation lesion region of the fundus; An operation module configured to perform an intersection processing on the initial image, the first image and the second image to obtain a segmentation result corresponding to the exudation lesion region of the fundus.

8. An ocular fundus exudation lesion extraction apparatus characterized by comprising: Comprise: A processor; A memory for storing processor-executable instructions; Wherein the processor is configured to call the instructions stored in the memory to execute the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 6. The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 6.

Images