Eye fundus image segmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a fundus image segmentation method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a sample fundus image and a sample plaque atrophy focus area label; wherein the sample plaque atrophy focus region label indicates a plaque atrophy focus region in the sample fundus image; performing image preprocessing on the sample eye fundus image to obtain a selected eye fundus image; performing optic disc covering on the selected eye fundus image to obtain a covered optic disc image; according to the covered optic disc image and the sample plaque atrophic lesion area label, training a preset lesion segmentation model to obtain a target lesion segmentation model; and obtaining a target fundus image, and performing focus region segmentation on the target fundus image based on the target focus segmentation model to obtain a target plaque atrophic focus region. According to the embodiment of the invention, the eye fundus image segmentation accuracy can be improved, and then the plaque atrophy focus area in the eye fundus image can be accurately obtained.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method and device for fundus image segmentation, an electronic device, and a storage medium. Background Art

[0002] Image segmentation is an image processing technology that divides an image into multiple regions or objects with similar features, and each region represents a specific part or object in the image. For example, in medical imaging, image segmentation can be used to segment fundus images to obtain the patchy atrophy lesion regions in the fundus images. Usually, when segmenting a fundus image to obtain the patchy atrophy lesion region, due to the high brightness and complex morphological features of the optic disc region in the fundus image, the accuracy of fundus image segmentation is not high, and thus the patchy atrophy lesion region in the fundus image cannot be accurately obtained. Therefore, how to improve the accuracy of fundus image segmentation and thus accurately obtain the patchy atrophy lesion region in the fundus image has become an urgent problem to be solved. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a method and device for fundus image segmentation, an electronic device, and a storage medium, aiming to improve the accuracy of fundus image segmentation and thus accurately obtain the patchy atrophy lesion region in the fundus image.

[0004] To achieve the above object, a first aspect of the embodiments of this application proposes a method for fundus image segmentation, and the method includes:

[0005] Obtain a sample fundus image and a sample patchy atrophy lesion region label; wherein, the sample patchy atrophy lesion region label indicates the region of the patchy atrophy lesion in the sample fundus image;

[0006] Perform image preprocessing on the sample fundus image to obtain a selected fundus image;

[0007] Perform optic disc masking on the selected fundus image to obtain a masked optic disc image;

[0008] Train a preset lesion segmentation model according to the masked optic disc image and the sample patchy atrophy lesion region label to obtain a target lesion segmentation model;

[0009] Obtain a target fundus image, and perform lesion region segmentation on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophy lesion region.

[0010] In some embodiments, the training of a preset lesion segmentation model according to the masked optic disc image and the sample patchy atrophy lesion region label to obtain a target lesion segmentation model includes:

[0011] Performing image segmentation on the occluded optic disc image based on the preset lesion segmentation model to obtain a predicted patchy atrophy lesion area label;

[0012] Calculating a training loss value based on the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label;

[0013] Optimizing the parameters of the preset lesion segmentation model according to the training loss value to obtain the target lesion segmentation model.

[0014] In some embodiments, the preset lesion segmentation model includes an encoder and a decoder. The encoder includes an overlapping patch embedding layer, a multi-head self-attention layer, and a feature extraction layer. Performing image segmentation on the occluded optic disc image based on the preset lesion segmentation model to obtain a predicted patchy atrophy lesion area label includes:

[0015] Performing convolutional feature extraction on the occluded optic disc image based on the overlapping patch embedding layer to obtain convolutional image features;

[0016] Performing self-attention feature extraction on the convolutional image features based on the multi-head self-attention layer to obtain self-attention image features;

[0017] Performing residual feature extraction on the self-attention image features based on the feature extraction layer to obtain residual image features;

[0018] Performing decoding segmentation on the residual image features based on the decoder to obtain the predicted patchy atrophy lesion area label.

[0019] In some embodiments, performing self-attention feature extraction on the convolutional image features based on the multi-head self-attention layer to obtain self-attention image features includes:

[0020] Performing linear feature extraction on the convolutional image features based on the multi-head self-attention layer to obtain a query matrix, a key matrix, and a value matrix;

[0021] Performing matrix dimensionality reduction on the key matrix based on the multi-head self-attention layer and a preset reduction ratio coefficient to obtain a reduced-dimensional matrix;

[0022] Performing self-attention calculation on the query matrix, the reduced-dimensional matrix, and the value matrix based on the multi-head self-attention layer to obtain the self-attention image features.

[0023] In some embodiments, performing residual feature extraction on the self-attention image features based on the feature extraction layer to obtain residual image features includes:

[0024] Performing non-linear feature extraction on the self-attention image features based on the feature extraction layer to obtain non-linear image features;

[0025] Performing convolutional feature extraction on the non-linear image features based on the feature extraction layer to obtain selected convolutional features;

[0026] Performing Gaussian activation on the selected convolutional features based on the feature extraction layer to obtain activated image features;

[0027] Performing non-linear feature extraction on the activated image features based on the feature extraction layer to obtain selected non-linear features;

[0028] Performing summation processing on the selected non-linear features and the self-attention image features to obtain the residual image features.

[0029] In some embodiments, the covering the optic disc of the selected fundus image to obtain a covered optic disc image includes:

[0030] Performing optic disc mask prediction on the selected fundus image based on a preset optic disc mask model to obtain optic disc mask data;

[0031] Performing mask covering on the selected fundus image according to the optic disc mask data to obtain the covered optic disc image.

[0032] In some embodiments, the preprocessing the sample fundus image to obtain a selected fundus image includes:

[0033] Performing image normalization on the sample fundus image to obtain a standard fundus image;

[0034] Performing image enhancement on the standard fundus image to obtain an enhanced fundus image.

[0035] To achieve the above object, a second aspect of the embodiments of the present application proposes a fundus image segmentation device, the device includes:

[0036] A data acquisition module, configured to acquire a sample fundus image and a sample patchy atrophy lesion area label; wherein, the sample patchy atrophy lesion area label indicates the area of the patchy atrophy lesion in the sample fundus image;

[0037] An image processing module, configured to preprocess the sample fundus image to obtain a selected fundus image;

[0038] An optic disc covering module, configured to cover the optic disc of the selected fundus image to obtain a covered optic disc image;

[0039] A model training module, configured to train a preset lesion segmentation model according to the covered optic disc image and the sample patchy atrophy lesion area label to obtain a target lesion segmentation model;

[0040] An image segmentation module, configured to obtain a target fundus image, and perform lesion area segmentation on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophy lesion area.

[0041] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0042] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0043] A fundus image segmentation method, device, electronic device and storage medium provided by the present application obtain a sample fundus image and a sample patchy atrophy lesion area label indicating the area of the patchy atrophy lesion in the sample fundus image. Then, image preprocessing is performed on the sample fundus image to transform the image into a standard selected fundus image, and then the optic disc of the selected fundus image is covered, so as to solve the influence on the fundus segmentation image caused by the high brightness and complex morphological features of the optic disc in the fundus image, improve the segmentation accuracy of subsequent fundus image segmentation, and at the same time decouple the optic disc covering and lesion segmentation, reduce the difficulty of subsequent image segmentation, and thus improve the segmentation accuracy of subsequent fundus image segmentation; further, a preset lesion segmentation model is trained according to the covered optic disc image and the sample patchy atrophy lesion area label to obtain a target lesion segmentation model that can accurately segment the patchy atrophy lesion area from the covered optic disc image, and finally obtain a target fundus image, and perform lesion area segmentation on the target fundus image based on the target lesion segmentation model to accurately obtain the target patchy atrophy lesion area in the fundus image. Description of the Drawings

[0044] Figure 1 is a flowchart of the fundus image segmentation method provided by the embodiments of the present application;

[0045] Figure 2 is Figure 1 a flowchart of step S102 in

[0046] Figure 3 is Figure 1 a flowchart of step S103 in

[0047] Figure 4 is Figure 1 the flowchart of step S104 in

[0048] Figure 5 is Figure 4 the flowchart of step S401 in

[0049] Figure 6 is Figure 5 the flowchart of step S502 in

[0050] Figure 7 is Figure 5 the flowchart of step S503 in

[0051] Figure 8 is the structural schematic diagram of the fundus image segmentation device provided by the embodiments of the present application;

[0052] Figure 9 is the hardware structural schematic diagram of the electronic device provided by the embodiments of the present application;

[0053] Figure 10 is the model schematic diagram of the target lesion segmentation model provided by the embodiments of the present application;

[0054] Figure 11 is the model accuracy comparison chart of the target lesion segmentation model provided by the embodiments of the present application

[0055] Figure 12 is the schematic diagram of the fundus image segmentation process provided by the embodiments of the present application. Detailed implementation manners

[0056] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the description and claims and the above accompanying drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, several terms involved in this application are parsed as follows:

[0060] Image Segmentation: Image segmentation is a key technology in computer vision and image processing, which is used to decompose digital images into multiple regions or objects, and each region contains pixels with similar attributes. This process is extremely important for analyzing and interpreting image content, enabling computers to identify, locate, and track objects in images. The applications of image segmentation are very extensive, including medical image analysis, environmental perception of autonomous vehicles, satellite image processing, video surveillance, and robot vision, etc. In the field of artificial intelligence, image segmentation technology usually combines machine learning and deep learning algorithms to improve the accuracy and efficiency of segmentation. For example, using convolutional neural networks (CNNs) for image segmentation can effectively process complex image features, extract key information, enabling artificial intelligence systems to better understand visual data and make intelligent responses. Through this technology, artificial intelligence not only simulates the visual cognitive process of humans but also expands its capabilities in automated and intelligent applications, thus achieving precise perception and intelligent interaction with the environment.

[0061] Lesion Area in Fundus Images: In fundus images, the lesion area refers to the part of the fundus tissue that shows abnormal or pathological features, and these abnormalities are usually related to various ophthalmic diseases, such as diabetic retinopathy, glaucoma, and macular degeneration, etc. The identification and analysis of the lesion area in fundus images are crucial for diagnosing and monitoring the eye health status. In modern ophthalmic medical practice, with the help of high-resolution imaging technologies, such as optical coherence tomography (OCT) and color fundus photography, clear fundus images can be obtained.

[0062] Optic Disc: The optic disc is a key anatomical structure in fundus images, located on the retina, where optic nerve fibers converge and exit the eyeball. In fundus examinations, the optic disc usually appears as a round or oval bright white area. The health status of the optic disc is of great significance for diagnosing various ophthalmic diseases, such as glaucoma, optic neuritis, and retinal diseases, etc. In medical imaging, by observing the size, shape, edge clarity of the optic disc, and its relationship with the surrounding retinal blood vessels, the fundus health status can be determined. With the development of medical imaging technologies, including optical coherence tomography (OCT) and color fundus photography, etc., the detailed observation of the optic disc has become more precise, thus improving the accuracy and efficiency of ophthalmic disease diagnosis.

[0063] Dice Loss Function: The Dice loss function is a statistical tool used to evaluate the performance of a model and is commonly used in medical image processing in the field of image segmentation. This loss function calculates the similarity between two samples based on the Dice coefficient. The Dice loss function is particularly suitable for situations where the data class is imbalanced. It optimizes the model by measuring the overlapping area between the predicted result and the actual label. The application of the Dice loss function is not limited to medical images and can also be used for any task that requires precise segmentation, such as satellite image processing, biological image analysis, etc. By emphasizing the reduction of false positives and false negatives, this loss function helps the model improve the accuracy of its segmentation while retaining image details.

[0064] Image segmentation is an image processing technique that divides an image into multiple regions or objects with similar characteristics, where each region represents a specific part or object in the image. For example, in medical imaging, image segmentation can be used to segment fundus images to obtain the patchy atrophy lesion regions in the fundus images. Usually, when segmenting fundus images to obtain the patchy atrophy lesion regions, due to the high brightness and complex morphological features of the optic disc region in the fundus images, the accuracy of fundus image segmentation is not high, and thus it is impossible to accurately obtain the patchy atrophy lesion regions in the fundus images. Therefore, how to improve the accuracy of fundus image segmentation and thus accurately obtain the patchy atrophy lesion regions in the fundus images has become an urgent problem to be solved.

[0065] Based on this, the embodiments of the present application provide a fundus image segmentation method, device, electronic device, and storage medium, aiming to improve the accuracy of fundus image segmentation and thus accurately obtain the patchy atrophy lesion regions in the fundus images.

[0066] The fundus image segmentation method, device, electronic device, and storage medium provided by the embodiments of the present application are specifically described through the following embodiments. First, the fundus image segmentation method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0068] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The fundus image segmentation method provided by the embodiments of this application relates to the field of image processing technology. The fundus image segmentation method provided by the embodiments of this application can be applied to a terminal, or to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the fundus image segmentation method, etc., but is not limited to the above forms.

[0070] This application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific embodiment of the present application, when it comes to performing relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0072] Figure 1 is an optional flowchart of the fundus image segmentation method provided by the embodiments of the present application. Figure 1 The method in may include but is not limited to steps S101 to S105.

[0073] Step S101, obtain a sample fundus image and a sample patchy atrophy lesion area label; wherein, the sample patchy atrophy lesion area label indicates the area of the patchy atrophy lesion in the sample fundus image;

[0074] Step S102, perform image preprocessing on the sample fundus image to obtain a selected fundus image;

[0075] Step S103, perform optic disc occlusion on the selected fundus image to obtain an occluded optic disc image;

[0076] Step S104, train a preset lesion segmentation model according to the occluded optic disc image and the sample patchy atrophy lesion area label to obtain a target lesion segmentation model;

[0077] Step S105, obtain a target fundus image, and perform lesion area segmentation on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophy lesion area.

[0078] Steps S101 to S105 illustrated in the embodiments of the present application, by obtaining a sample fundus image and a sample patchy atrophy lesion area label indicating the area of the patchy atrophy lesion in the sample fundus image, and then performing image preprocessing on the sample fundus image to transform the image into a standard selected fundus image, and then performing optic disc occlusion on the selected fundus image, thus solving the influence on the fundus segmentation image caused by the high brightness and complex morphological features of the optic disc in the fundus image, achieving an improvement in the segmentation accuracy of subsequent fundus image segmentation, and at the same time decoupling the optic disc occlusion from the lesion segmentation, reducing the difficulty of subsequent image segmentation, thereby improving the segmentation accuracy of subsequent fundus image segmentation; further, training a preset lesion segmentation model according to the occluded optic disc image and the sample patchy atrophy lesion area label to obtain a target lesion segmentation model capable of accurately segmenting the patchy atrophy lesion area from the occluded optic disc image, finally obtaining a target fundus image, and performing lesion area segmentation on the target fundus image based on the target lesion segmentation model to accurately obtain the target patchy atrophy lesion area in the fundus image.

[0079] In step S101 of some embodiments, the sample fundus image refers to a high-resolution image of the posterior retina, optic nerve head and related structures of the human eye obtained by a fundus camera or other imaging device. The sample patchy atrophy lesion area label refers to the marker information used to label the specific position and range of the patchy atrophy lesion in the sample fundus image. This label exists in the form of a binary mask to distinguish the lesion area from the normal area. For example, on a fundus image, the lesion area of patchy atrophy is marked as a white area, while the rest is black.

[0080] Please refer to Figure 2 , in some embodiments, step S102 may include but is not limited to steps S201 to S202:

[0081] Step S201, performing image standardization on the sample fundus image to obtain a standard fundus image;

[0082] Step S202, performing image enhancement on the standard fundus image to obtain an enhanced fundus image.

[0083] Steps S201 to S202 illustrated in the embodiments of the present application, by performing image standardization on the sample fundus image to obtain a standard fundus image, and then performing image enhancement on the standard fundus image to obtain an enhanced fundus image, can improve the image quality and the usability of information, and further improve the segmentation accuracy of subsequent lesion segmentation.

[0084] In step S201 of some embodiments, image normalization refers to processing the sample fundus image so that the sample fundus image conforms to a unified standard processing flow. In one embodiment, image normalization includes normalizing the sample fundus image, normalizing the pixels in the sample fundus image to the interval [0, 1], and standardizing it to a distribution with a mean of 0.5 and a standard deviation of 0.5. Then, the image is adjusted to a resolution of 512×512 to obtain a standard fundus image.

[0085] In step S202 of some embodiments, image enhancement refers to enhancing the dataset of the standard fundus image to enhance the diversity and robustness of the dataset and improve the quality of the dataset. In one embodiment, image enhancement includes, but is not limited to, randomly flipping, randomly rotating, and adjusting the brightness of the standard fundus image. The image obtained after image enhancement of the standard fundus image is the enhanced fundus image.

[0086] Please refer to Figure 3 , in some embodiments, step S103 may include, but is not limited to, steps S301 to S302:

[0087] Step S301, performing optic disc mask prediction on the selected fundus image based on a preset optic disc mask model to obtain optic disc mask data;

[0088] Step S302, performing mask covering on the selected fundus image according to the optic disc mask data to obtain a covered optic disc image.

[0089] Steps S301 to S302 illustrated in the embodiments of the present application perform optic disc mask prediction on the selected fundus image based on a preset optic disc mask model to obtain optic disc mask data, and then perform mask covering on the selected fundus image according to the optic disc mask data to obtain a covered optic disc image, thereby solving the influence on the fundus segmentation image caused by the high brightness and complex morphological features of the optic disc in the fundus image, achieving improved segmentation accuracy of subsequent fundus image segmentation, and at the same time decoupling optic disc covering and lesion segmentation, reducing the difficulty of subsequent image segmentation, and thus improving the segmentation accuracy of subsequent fundus image segmentation.

[0090] In step S301 of some embodiments, the preset optic disc mask model is a pre-trained neural network model used to predict the optic disc mask for a selected fundus image, obtaining the optic disc mask of the selected fundus image. The optic disc mask prediction is used to predict the pixel coordinates of the area where the optic disc is located in the fundus image, and outputs the pixel coordinates of the predicted optic disc area through binary data. In one embodiment, the preset optic disc mask model is a neural network model based on the UNet structure model. The optic disc mask data is a type of binary data composed of 0 or 1, used to indicate the pixel coordinate area where the optic disc is located. For example, 1 represents the area where the optic disc is located, and 0 represents the area where the optic disc is not located. It should be noted that this application does not specifically limit the specific values of the binary data and the corresponding areas, and can be set to other values in other embodiments.

[0091] In step S302 of some embodiments, the mask covering is a processing flow that changes the pixel values of the selected fundus image according to the optic disc mask data and changes the pixel values to black. For example, if the optic disc mask data is [0, 0, 1, 1], and the selected fundus image is [[1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]], the 1 in the optic disc mask data indicates that this point in the selected fundus image is the optic disc area. After mask covering, it becomes [[1, 1, 1], [2, 2, 2], [0, 0, 0], [0, 0, 0]]. The image after mask covering is the covered optic disc image.

[0092] Please refer to Figure 4 , in some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0093] Step S401, perform image segmentation on the covered optic disc image based on the preset lesion segmentation model to obtain the predicted patchy atrophy lesion area label;

[0094] Step S402, calculate the loss value according to the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label to obtain the training loss value;

[0095] Step S403, optimize the parameters of the preset lesion segmentation model according to the training loss value to obtain the target lesion segmentation model.

[0096] Steps S401 to S403 illustrated in the embodiments of this application perform image segmentation on the covered optic disc image based on the preset lesion segmentation model to obtain the predicted patchy atrophy lesion area label, then calculate the loss value according to the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label to obtain the training loss value, and finally optimize the parameters of the preset lesion segmentation model according to the training loss value, so as to obtain the target lesion segmentation model that can accurately segment the patchy atrophy lesion area from the covered optic disc image.

[0097] Please refer to Figure 5 In some embodiments, the preset lesion segmentation model includes an encoder and a decoder. The encoder includes an overlapping patch embedding layer, a multi-head self-attention layer, and a feature extraction layer. Step S401 includes but is not limited to steps S501 to S504:

[0098] Step S501, perform convolutional feature extraction on the covered optic disc image based on the overlapping patch embedding layer to obtain convolutional image features;

[0099] Step S502, perform self-attention feature extraction on the convolutional image features based on the multi-head self-attention layer to obtain self-attention image features;

[0100] Step S503, perform residual feature extraction on the self-attention image features based on the feature extraction layer to obtain residual image features;

[0101] Step S504, perform decoding segmentation on the residual image features based on the decoder to obtain the predicted patchy atrophy lesion area label.

[0102] Steps S501 to S504 illustrated in the embodiments of the present application perform convolutional feature extraction on the covered optic disc image based on the overlapping patch embedding layer to obtain convolutional image features, then perform self-attention feature extraction on the convolutional image features based on the multi-head self-attention layer to obtain self-attention image features, then perform residual feature extraction on the self-attention image features based on the feature extraction layer to obtain residual image features, and finally perform decoding segmentation on the residual image features based on the decoder to obtain the predicted patchy atrophy lesion area label, thereby realizing efficient segmentation and accurate recognition of the patchy atrophy lesion area in the covered optic disc image, and improving the accuracy of lesion area extraction.

[0103] In step S501 of some embodiments, the overlapping patch embedding layer is a neural network layer composed of convolutional kernels. The convolutional dimension of the convolutional kernel is greater than the stride of the convolutional kernel, so as to convolve the covered optic disc image into at least one patch, and there is overlap between the patches. For example, the convolutional dimension of the convolutional kernel is 7 and the stride is 4. Performing convolutional feature extraction on the covered optic disc image based on the overlapping patch embedding layer ensures that there is overlap between the patches. While extracting local features, it also enhances the coherence of the features by maintaining the overlapping area, effectively captures the local information of the image, and at the same time retains a certain global perception ability, thereby improving the segmentation accuracy of subsequent lesion segmentation.

[0104] Please refer to Figure 6 In some embodiments, step S502 includes but is not limited to steps S601 to S603:

[0105] Step S601: Based on the multi-head self-attention layer, perform linear feature extraction on the convolutional image features to obtain a query matrix, a key matrix, and a value matrix;

[0106] Step S602: Based on the multi-head self-attention layer and a preset reduction ratio coefficient, perform matrix dimensionality reduction on the key matrix to obtain a reduced-dimensional matrix;

[0107] Step S603: Based on the multi-head self-attention layer, perform self-attention calculation on the query matrix, the reduced-dimensional matrix, and the value matrix to obtain self-attention image features.

[0108] In steps S601 to S603 illustrated in the embodiments of the present application, based on the multi-head self-attention layer, perform linear feature extraction on the convolutional image features to obtain a query matrix, a key matrix, and a value matrix, then based on the multi-head self-attention layer and a preset reduction ratio coefficient, perform matrix dimensionality reduction on the key matrix to obtain a reduced-dimensional matrix, and finally based on the multi-head self-attention layer, perform self-attention calculation on the query matrix, the reduced-dimensional matrix, and the value matrix to obtain self-attention image features, achieving efficient extraction and information compression of image features, improving the accuracy and computational efficiency of feature representation, and enhancing the ability to process long sequences.

[0109] In step S601 of some embodiments, the multi-head self-attention layer includes three linear layers, namely a query layer, a key layer, and a value layer. The convolutional image features are respectively input into the query layer, the key layer, and the value layer to obtain a query matrix, a key matrix, and a value matrix.

[0110] In step S602 of some embodiments, the reduction ratio coefficient is a value between [0, 1], which is used to reduce the sequence length of the features. It should be noted that in the traditional self-attention mechanism, the computational complexity is relatively high when calculating long sequences, resulting in a slow speed for lesion segmentation of fundus images. By performing matrix dimensionality reduction on the key matrix based on the multi-head self-attention layer and a preset reduction ratio coefficient, the computational complexity is reduced, and thus the segmentation speed for lesion segmentation of fundus images is improved.

[0111] Performing matrix dimensionality reduction on the key matrix based on the multi-head self-attention layer and a preset reduction ratio coefficient is as shown in Equation (1):

[0112]

[0113] where K is the key matrix, R is the reduction ratio coefficient, C is the number of channels of the convolutional image features, H is the length of the optic disc-covered image, W is the width of the optic disc-covered image, P is the convolutional dimension of the convolutional kernel in the overlapping patch embedding layer, that is, the patch size, Reshape() is the rearrangement operation, Linear() is the linear transformation operation, and K 2 is the reduced-dimensional matrix.

[0114] In step S603 of some embodiments, the self-attention calculation is an aggregation calculation based on a query matrix, a dimensionality reduction matrix, and a value matrix. The aggregation calculation specifically includes calculating an attention weight matrix based on the query matrix, the dimensionality reduction matrix, and the value matrix to obtain a self-attention matrix, then performing multi-head self-attention splicing based on the self-attention matrix to obtain a target self-attention matrix, and then sequentially passing through a residual connection, normalization, two-layer linear transformation, and an activation function to obtain a matrix in numerical form, that is, obtaining self-attention image features.

[0115] Please refer to Figure 7 , in some embodiments, step S503 may include but is not limited to steps S701 to S705:

[0116] Step S701, performing non-linear feature extraction on the self-attention image features based on a feature extraction layer to obtain non-linear image features;

[0117] Step S702, performing convolutional feature extraction on the non-linear image features based on the feature extraction layer to obtain selected convolutional features;

[0118] Step S703, performing Gaussian activation on the selected convolutional features based on the feature extraction layer to obtain activated image features;

[0119] Step S704, performing non-linear feature extraction on the activated image features based on the feature extraction layer to obtain selected non-linear features;

[0120] Step S705, performing a summation process based on the selected non-linear features and the self-attention image features to obtain residual image features.

[0121] Steps S701 to S705 shown in the embodiments of the present application perform non-linear feature extraction on the self-attention image features based on a feature extraction layer to obtain non-linear image features, then perform convolutional feature extraction on the non-linear image features based on the feature extraction layer to obtain selected convolutional features, then perform Gaussian activation on the selected convolutional features based on the feature extraction layer to obtain activated image features, then perform non-linear feature extraction on the activated image features based on the feature extraction layer to obtain selected non-linear features, and finally perform a summation process based on the selected non-linear features and the self-attention image features to obtain residual image features, thereby realizing the fusion and enhancement of multi-level features, improving the representation ability and detail expression ability of image features, improving the segmentation accuracy of the target lesion segmentation model for fundus images, and then accurately obtaining the patchy atrophy lesion area in the fundus image.

[0122] In steps S701 to S705 of some embodiments, the feature extraction layer is a neural network layer formed by sequentially connecting a first multi-layer perceptron, a convolutional layer, a Gaussian function activation layer, a second multi-layer perceptron, and an output layer. The self-attention image features are subjected to non-linear feature extraction through the first multi-layer perceptron to obtain non-linear image features. The non-linear image features are subjected to convolutional feature extraction through the convolutional layer to obtain selected convolutional features. The selected convolutional features are subjected to Gaussian activation through the Gaussian function activation layer to obtain activated image features. The activated image features are subjected to non-linear feature extraction through the second multi-layer perceptron. The selected non-linear features and the self-attention image features are subjected to summation processing through the output layer to obtain residual image features, as shown in Equation (2):

[0123] X out = Sum(MLP 2 (GELU(Conv 3×3 (MLP 1 (x in )))),x iu ) (2),

[0124] where x in is the self-attention image feature, MLP 1 is the first multi-layer perceptron, Conv 3×3 is the convolutional layer, GELU is the Gaussian activation function layer, MLP 2 is the second multi-layer perceptron, and Sum is the output layer.

[0125] It should be noted that a 3×3 convolutional layer is used between the first multi-layer perceptron and the second multi-layer perceptron to improve the ability of local feature extraction. At the same time, the position information is implicitly retained through convolution, and finally, while retaining the local feature information, the segmentation accuracy of the target lesion segmentation model is improved.

[0126] In step S504 of some embodiments, the decoder is a neural network model used to perform decoding segmentation based on the residual image features to obtain the prediction label of the patchy atrophy lesion area in the fundus image. In one embodiment, the decoder is composed of UNet. The predicted patchy atrophy lesion area is a binary matrix used to represent whether a certain pixel point in the fundus image is in the patchy atrophy lesion area or not.

[0127] Please refer to Figure 10, in one embodiment, the target lesion segmentation model consists of an encoder and a decoder. Among them, the decoder is a Mit-b2 encoder, and the decoder is composed of a U-Net decoder. The encoder includes an overlapping patch embedding layer, a multi-head attention layer, and a feature extraction layer. Among them, the overlapping patch embedding layer is composed of Over Patch Embeddings, the multi-head self-attention layer is composed of EFFICIENT Self-Attn in the Transformer Block, and the feature extraction layer is composed of Mix-FFN.

[0128] In step S402 of some embodiments, the loss value is calculated as the difference between the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label. In one embodiment, the loss value is calculated as shown in Equation (3):

[0129] Focal_DiceLoss = Focal Loss + λ × Dice Loss (3),

[0130] where Focal Loss is the Focal loss value between the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label, Dice Loss is the Dice loss value between the predicted patchy atrophy lesion area label and the sample patchy atrophy lesion area label, and Focal_DiceLoss is the training loss value.

[0131] Please refer to Figure 11 , in one embodiment, when using Mit-b2 as the decoder and UNet as the encoder, and the loss value using Focal_DiceLoss, the segmentation accuracy of the target lesion segmentation model is the highest.

[0132] In step S403 of some embodiments, parameter optimization refers to performing stochastic gradient optimization on the preset lesion segmentation model according to the training loss value, adjusting the model parameters of the preset lesion segmentation model, and obtaining a target lesion segmentation model that can accurately perform image segmentation.

[0133] Please refer to Figure 12 , in step S105 of some embodiments, the target fundus image is obtained, and then the target fundus image is preprocessed to obtain a target preprocessed image. Then, the optic disc is judged and segmented on the target preprocessed image to generate an optic disc mask. Then, the target preprocessed image is covered by the optic disc mask to obtain a target optic disc-covered image. Finally, the target optic disc-covered image is input into the target lesion segmentation model to obtain a target patchy atrophy lesion area.

[0134] Please refer to Figure 8, an embodiment of the present application further provides a fundus image segmentation device, which can implement the above fundus image segmentation method. The device includes:

[0135] An acquisition data module 801, configured to acquire a sample fundus image and a label of the sample patchy atrophy lesion area; wherein, the label of the sample patchy atrophy lesion area indicates the area of the patchy atrophy lesion in the sample fundus image;

[0136] An image processing module 802, configured to perform image preprocessing on the sample fundus image to obtain a selected fundus image;

[0137] An optic disc covering module 803, configured to cover the optic disc of the selected fundus image to obtain a covered optic disc image;

[0138] A model training module 804, configured to train a preset lesion segmentation model according to the covered optic disc image and the label of the sample patchy atrophy lesion area to obtain a target lesion segmentation model;

[0139] An image segmentation module 805, configured to acquire a target fundus image, and perform lesion area segmentation on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophy lesion area.

[0140] The specific implementation manner of the fundus image segmentation device is basically the same as that of the specific embodiment of the above fundus image segmentation method, and will not be elaborated here.

[0141] An embodiment of the present application further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above fundus image segmentation method is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0142] Please refer to Figure 9 , Figure 9 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0143] A processor 901, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0144] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 is used to call and execute the fundus image segmentation method of the embodiments of this application;

[0145] The input / output interface 903 is used to implement information input and output;

[0146] The communication interface 904 is used to implement communication and interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.), or can also achieve communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0147] The bus 905 transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0148] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 achieve communication connections with each other inside the device through the bus 905.

[0149] The embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned fundus image segmentation method.

[0150] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and their combinations.

[0151] The fundus image segmentation method, fundus image segmentation device, electronic device and storage medium provided by the embodiments of the present application first obtain a sample fundus image and a sample patchy atrophy lesion area label indicating the area of the patchy atrophy lesion in the sample fundus image. Then, the sample fundus image is preprocessed to transform the image into a standard selected fundus image. Then, the optic disc in the selected fundus image is covered, so as to solve the influence on the fundus segmentation image caused by the high brightness and complex morphological features of the optic disc in the fundus image, improve the segmentation accuracy of subsequent fundus image segmentation, and at the same time decouple the optic disc covering and lesion segmentation, reduce the difficulty of subsequent image segmentation, and thus improve the segmentation accuracy of subsequent fundus image segmentation. Further, the preset lesion segmentation model is trained according to the covered optic disc image and the sample patchy atrophy lesion area label to obtain a target lesion segmentation model that can accurately segment the patchy atrophy lesion area from the covered optic disc image. Finally, a target fundus image is obtained, and the lesion area of the target fundus image is segmented based on the target lesion segmentation model to accurately obtain the target patchy atrophy lesion area in the fundus image.

[0152] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0153] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware and their appropriate combinations.

[0156] In the description of this application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0157] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0158] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, in each embodiment of the present application, the functional units may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0162] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. However, this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall fall within the scope of the rights of the embodiments of the present application.

Claims

1. A fundus image segmentation method, characterized in that: The method comprises: Acquire a sample fundus image and a sample patchy atrophy lesion region label; wherein the sample patchy atrophy lesion region label indicates the region of the patchy atrophy lesion in the sample fundus image; Performing image preprocessing on the sample fundus image to obtain a selected fundus image; Performing optic disc masking on the selected fundus image to obtain a masked optic disc image; Training a preset lesion segmentation model according to the masked optic disc image and the sample patchy atrophy lesion region label to obtain a target lesion segmentation model; A target fundus image is acquired, and lesion regions are segmented on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophic lesion region.

2. The method according to claim 1, characterized in that: The method of training a preset lesion segmentation model according to the masked optic disc image and the sample patchy atrophy lesion region label to obtain a target lesion segmentation model includes: Performing image segmentation on the covered optic disc image based on the preset lesion segmentation model to obtain a predicted patchy atrophy lesion area label; Calculating a loss value based on the predicted patchy atrophy lesion region label and the sample patchy atrophy lesion region label to obtain a training loss value; The preset lesion segmentation model is optimized in parameters according to the training loss value to obtain the target lesion segmentation model.

3. The method according to claim 2, characterized in that The preset lesion segmentation model includes an encoder and a decoder, the encoder includes an overlapping patch embedding layer, a multi-head self-attention layer and a feature extraction layer, and the image segmentation of the covered optic disc image based on the preset lesion segmentation model to obtain a predicted patchy atrophy lesion area label includes: Performing convolution feature extraction on the covered optic disc image based on the overlapping patch embedding layer to obtain convolution image features; Based on the multi-head self-attention layer, self-attention feature extraction is performed on the convolution image feature to obtain a self-attention image feature; Performing residual feature extraction on the self-attention image feature based on the feature extraction layer to obtain a residual image feature; The residual image features are decoded and segmented based on the decoder to obtain the predicted patchy atrophy lesion area label.

4. The method according to claim 3, characterized in that The step of performing self-attention feature extraction on the convolution image features based on the multi-head self-attention layer to obtain self-attention image features includes: Performing linear feature extraction on the convolution image features based on the multi-head self-attention layer to obtain a query matrix, a key matrix and a value matrix; Performing matrix dimensionality reduction on the key matrix based on the multi-head self-attention layer and a preset reduction ratio coefficient to obtain a reduced-dimensional matrix; Based on the multi-head self-attention layer, self-attention calculation is performed on the query matrix, dimensionality reduction matrix and value matrix to obtain the self-attention image features.

5. The method according to claim 3, characterized in that: The step of performing residual feature extraction on the self-attention image feature based on the feature extraction layer to obtain the residual image feature includes: Based on the feature extraction layer, nonlinear feature extraction is performed on the self-attention image feature to obtain a nonlinear image feature; Performing convolution feature extraction on the nonlinear image features based on the feature extraction layer to obtain selected convolution features; Based on the feature extraction layer, Gaussian activation is performed on the selected convolution feature to obtain an activated image feature; Performing nonlinear feature extraction on the activated image features based on the feature extraction layer to obtain selected nonlinear features; The residual image feature is obtained by performing a summation process based on the selected nonlinear feature and the self-attention image feature.

6. The method according to any one of claims 1 to 5, characterized in that: The step of performing optic disc masking on the selected fundus image to obtain a masked optic disc image comprises: Predicting an optic disc mask for the selected fundus image based on a preset optic disc mask model to obtain optic disc mask data; The selected fundus image is masked according to the optic disc mask data to obtain the masked optic disc image.

7. The method according to any one of claims 1 to 5, characterized in that: The performing image preprocessing on the sample fundus image to obtain a selected fundus image includes: Performing image standardization on the sample fundus image to obtain a standard fundus image; The standard fundus image is enhanced to obtain an enhanced fundus image.

8. A fundus image segmentation device, characterized in that: The device comprises: A data acquisition module is used to acquire a sample fundus image and a sample patchy atrophy lesion area label; wherein the sample patchy atrophy lesion area label indicates the area of ​​the patchy atrophy lesion in the sample fundus image; An image processing module, used for performing image preprocessing on the sample fundus image to obtain a selected fundus image; An optic disc masking module, used for masking the optic disc of the selected fundus image to obtain a masked optic disc image; A model training module, used for training a preset lesion segmentation model according to the masked optic disc image and the sample patchy atrophy lesion region label to obtain a target lesion segmentation model; The image segmentation module is used to obtain a target fundus image, and perform lesion area segmentation on the target fundus image based on the target lesion segmentation model to obtain a target patchy atrophic lesion area.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the fundus image segmentation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the fundus image segmentation method according to any one of claims 1 to 7 is implemented.

Images