Eye fundus image segmentation method and device, equipment and storage medium

Abstract

The invention relates to the field of image processing, and discloses an eye fundus image segmentation method, device and equipment and a storage medium, and the method comprises the steps: collecting an eye fundus image of a patient, and carrying out the data preprocessing of the eye fundus image of the patient, and obtaining original image data and evaluation data; performing physical enhancement on the original image data to obtain enhanced image data; establishing a convolutional neural network, and performing feature extraction on the enhanced image data by using the convolutional neural network to obtain feature data; performing adaptive weighting processing on the feature data to obtain weighted data; performing morphological operation on the weighted data to obtain to-be-evaluated data; performing data evaluation on the to-be-evaluated data by using the evaluation data to obtain an evaluation result, and obtaining output data according to the evaluation result; feature extraction is performed by using the convolutional neural network, and feature weighting processing is performed on the fundus image by using adaptive weighting processing, so that the ability of analyzing the multi-feature fundus image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a fundus image segmentation method, device, equipment and storage medium. Background Art

[0002] Fundus image segmentation is crucial for the diagnosis and treatment of eye diseases. However, fundus image segmentation currently faces many challenges. The structure of fundus images is complex and diverse. The shapes, sizes, and colors of structures such as blood vessels, nerve fibers, and retinal lesions vary greatly among individuals, which greatly increases the difficulty of designing a general high-precision segmentation algorithm. Image acquisition is easily affected by lighting conditions, shooting equipment, and pathological changes, resulting in blurred boundaries and noise, which seriously interferes with the accurate recognition of edges and subtle structures by segmentation algorithms.

[0003] Traditional deep learning-based retinal vessel segmentation methods, such as UNet, FCN, DeepLab v3+, etc., have achieved certain results, but they still have limitations. For example, the U-shaped network based on the MobileNetV2 pre-trained component proposed by Laibacher et al. has made progress in real-time operation of high-resolution images, but its analysis ability is limited when facing complex image structures and lacks the ability to process multi-feature images. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a fundus image segmentation method, device, equipment and storage medium, which utilizes a convolutional neural network to extract features from complex image information, suppresses and removes irrelevant features, and then performs adaptive weighted processing on the feature data, dynamically adjusts the weights of key features according to the overall feature distribution of the image, enhances key feature information, ensures that key feature information can be accurately extracted and utilized in various complex situations, improves the adaptability of the model to multi-feature images, and thereby improves the accuracy and reliability of the fundus image segmentation results.

[0005] The first aspect of the present invention provides a fundus image segmentation method, comprising: collecting a patient's fundus image, and performing data preprocessing on the patient's fundus image to obtain original image data and evaluation data; physically enhancing the original image data to obtain enhanced image data; establishing a convolutional neural network, and using the convolutional neural network to extract features from the enhanced image data to obtain feature data; performing adaptive weighted processing on the feature data to obtain weighted data; performing morphological operations on the weighted data to obtain data to be evaluated; using the evaluation data to perform data evaluation on the data to be evaluated to obtain evaluation results, and obtaining output data based on the evaluation results.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the method of acquiring a fundus image of a patient and performing data preprocessing on the fundus image of the patient to obtain original image data and evaluation data includes: acquiring a fundus image of the patient; performing image format conversion on the fundus image to obtain sorted data; performing denoising on the sorted data to obtain denoised data; performing normalization on the denoised data to obtain normalized data; and performing data extraction on the normalized data to obtain original image data and evaluation data.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the physical enhancement of the original image data to obtain enhanced image data includes: simulating illumination changes on the original image data to obtain illumination simulation data; simulating geometric transformations on the original image data to obtain geometric simulation data; simulating optical distortions on the original image data to obtain distortion simulation data; and merging illumination simulation data, geometric simulation data and distortion simulation data to obtain enhanced image data.

[0008] Optionally, in a third implementation manner of the first aspect of the present invention, the establishing of a convolutional neural network and using the convolutional neural network to perform feature extraction on the enhanced image data to obtain feature data includes: establishing a convolutional neural network, the convolutional neural network including a shallow convolutional layer and a deep convolutional layer; using the shallow convolutional layer to perform preliminary feature extraction on the enhanced image data to obtain initial feature data; using the deep convolutional layer to perform deep feature extraction on the initial feature data to obtain deep feature data; and performing data fusion on the initial feature data and the deep feature data to obtain feature data.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the data fusion of the initial feature data and the deep feature data to obtain the feature data includes: establishing a jump connection channel and an attention gate circuit between the shallow convolution layer and the deep convolution layer; using the jump connection channel to splice the initial feature data with the deep feature data to obtain spliced ​​data; and using the attention gate circuit to perform local feature analysis on the spliced ​​data to obtain the feature data.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the adaptive weighted processing of the feature data to obtain weighted data includes: performing global average pooling processing on the feature data to obtain pooled data; performing coefficient learning on the pooled data to obtain weighting coefficients; and using the weighting coefficients to adaptively weight the feature data to obtain weighted data.

[0011] Optionally, in a sixth implementation method of the first aspect of the present invention, the use of the evaluation data to perform data evaluation on the data to be evaluated, obtain an evaluation result, and obtain output data based on the evaluation result, includes: using the evaluation data to perform data evaluation on the data to be evaluated, obtain an evaluation result; if the evaluation result is qualified, marking the data to be evaluated as output data; if the evaluation result is unqualified, generating adjustment data based on the evaluation data and the data to be evaluated, using the adjustment data to optimize the generation process of the data to be evaluated, so as to regenerate the data to be evaluated and evaluate it again, and repeating this process until the evaluation result is qualified, and marking the data to be evaluated at this time as output data.

[0012] The second aspect of the present invention provides a fundus image segmentation device, including: an image acquisition module, which is used to acquire a patient's fundus image and perform data preprocessing on the patient's fundus image to obtain original image data and evaluation data; an image enhancement module, which is used to physically enhance the original image data to obtain enhanced image data; a feature extraction module, which is used to establish a convolutional neural network and use the convolutional neural network to extract features from the enhanced image data to obtain feature data; a feature weighting module, which is used to perform adaptive weighted processing on the feature data to obtain weighted data; a data optimization module, which is used to perform morphological operations on the weighted data to obtain data to be evaluated; and a result evaluation module, which is used to perform data evaluation on the data to be evaluated using the evaluation data to obtain evaluation results, and obtain output data based on the evaluation results.

[0013] The third aspect of the present invention provides a fundus image segmentation device, which includes: a memory and at least one processor, wherein instructions are stored in the memory; at least one of the processors calls the instructions in the memory so that the fundus image segmentation device performs each step of the fundus image segmentation method described in any one of the above items.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the various steps of any of the fundus image segmentation methods described above are implemented.

[0015] In the technical solution of the present invention, firstly, the fundus image of the patient is collected and data preprocessing is performed to obtain original image data and evaluation data; then, the original image data is physically enhanced to simulate the changes of various actual imaging factors to enrich the diversity of the image and obtain enhanced image data; then, a convolutional neural network is established to extract features, and skip connections and attention gate circuit technology are used to suppress irrelevant or redundant features and highlight key features, thereby obtaining feature data; then, the extracted feature data is adaptively weighted to improve the adaptability of the model to different feature images and obtain weighted data; then, morphological operations are performed on the weighted data to further optimize the image quality and obtain data to be evaluated; finally, the evaluation data is used to evaluate the data to be evaluated, and output data is obtained according to the evaluation results; the present invention uses a convolutional neural network to extract features from complex image information, suppress and remove irrelevant features, and then performs adaptive weighted processing on the feature data, dynamically adjusts the weights of key features according to the overall feature distribution of the image, enhances key feature information, ensures that key feature information can be accurately extracted and used in various complex situations, improves the adaptability of the model to multi-feature images, and thereby improves the accuracy and reliability of fundus image segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0017] Figure 1 A first flow chart of a fundus image segmentation method provided by an embodiment of the present invention;

[0018] Figure 2 A second flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0019] Figure 3 A third flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0020] Figure 4 A fourth flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0021] Figure 5 A fifth flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0022] Figure 6 A sixth flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0023] Figure 7 A seventh flow chart of the fundus image segmentation method provided by an embodiment of the present invention;

[0024] Figure 8A schematic diagram of the structure of a fundus image segmentation device provided by an embodiment of the present invention;

[0025] Fig. 9 A schematic diagram of the structure of a fundus image segmentation device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The invention provides a fundus image segmentation method, device, equipment and storage medium. The invention first collects a patient's fundus image and performs data preprocessing to obtain original image data and evaluation data; then physically enhances the original image data, simulates the changes of multiple actual imaging factors to enrich the diversity of the image, and obtains enhanced image data; then establishes a convolutional neural network to extract features, and uses skip connection and attention gate circuit technology to suppress irrelevant or redundant features and highlight key features, thereby obtaining feature data; then adaptively weighted processing is performed on the extracted feature data to improve the adaptability of the model to different feature images, thereby obtaining weighted data; then morphological operations are performed on the weighted data to further optimize the image quality, thereby obtaining data to be evaluated; finally, the evaluation data is used to evaluate the data to be evaluated, and output data is obtained according to the evaluation result; the invention uses a convolutional neural network to extract features from complex image information, suppresses and removes irrelevant features, and then adaptively weighted processing is performed on the feature data, the weights of key features are dynamically adjusted according to the overall feature distribution of the image, and key feature information is enhanced, thereby ensuring that key feature information can be accurately extracted and used in various complex situations, thereby improving the adaptability of the model to multi-feature images, and thereby improving the accuracy and reliability of fundus image segmentation results.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the fundus image segmentation method in the embodiment of the present invention includes:

[0029] 101. Collect fundus images of patients, and perform data preprocessing on the fundus images of patients to obtain original image data and evaluation data;

[0030] In this embodiment, a fundus imaging device is first used to collect multi-view and multi-mode fundus images, and the collected images in different formats are uniformly converted into a format suitable for processing. Then, preliminary data processing is performed on the fundus images. For example, Gaussian filtering and mean filtering techniques are used to reduce noise interference, and normalization or standardization operations are used to enhance data consistency. Finally, the data is divided into original image data and evaluation data in proportion; the original image data is used for model training, and the evaluation data is used for performance evaluation.

[0031] 102. Physically enhance the original image data to obtain enhanced image data;

[0032] In this embodiment, the image is enhanced by introducing a physics-based model (such as an optical, reflection or object morphology model). The physical enhancement method includes adjusting its brightness, morphological size and degree of optical distortion. Through physical enhancement, the original data can be enriched, so that the neural network can obtain more image data for training, thereby improving its generalization ability and adaptability to different imaging conditions.

[0033] 103. Establish a convolutional neural network, and use the convolutional neural network to extract features of the enhanced image data to obtain feature data;

[0034] In this embodiment, a convolutional neural network is established and the enhanced image data is input into the convolutional neural network. After the convolutional neural network analyzes the multi-scale features in the enhanced image data, the redundant information is weakened and the key information is enhanced, the important structure is highlighted in the complex image, the segmentation accuracy is improved, and finally the feature data is output.

[0035] 104. Perform adaptive weighting processing on the feature data to obtain weighted data;

[0036] In this embodiment, global average pooling is performed on the feature data, and its key information is deeply learned to obtain pooled data. The pooled data is used to adaptively weight the feature data to further highlight its key features, suppress irrelevant features, reduce background noise interference, and improve the ability to identify and segment important structures to obtain weighted data; adaptive weighting can strengthen or weaken data information of different features, improve the expressiveness of the data, and provide more accurate data for fundus image segmentation.

[0037] 105. Perform morphological operations on the weighted data to obtain data to be evaluated;

[0038] In this embodiment, morphological operations are performed on the weighted fundus image data, and the fundus image data is further optimized through dilation and erosion operations to make structures such as blood vessels more coherent and complete, with clear and accurate boundaries, thereby enhancing recognizability, improving the model structure recognition and division capabilities, and enhancing the interpretability of the segmentation results.

[0039] 106. Perform data evaluation on the data to be evaluated using the evaluation data to obtain evaluation results, and obtain output data according to the evaluation results;

[0040] In this embodiment, the Dice coefficient, IoU and other evaluation indicators are used to compare the data to be evaluated with the evaluation data, and the performance of the data to be evaluated is judged. If it is qualified, the data to be evaluated is marked as output data; if it is unqualified, the difference between the two is analyzed to generate adjustment data, and the adjustment data is used to optimize the generation process of the data to be evaluated, generate new data to be evaluated, and perform data evaluation until the evaluation result is qualified and the output data is obtained. By setting reasonable evaluation indicators, deficiencies can be discovered in time, and the accuracy and reliability of fundus image segmentation can be improved.

[0041] In an embodiment of the present invention, firstly, fundus images of patients are collected and data preprocessed to obtain original image data and evaluation data; then, physical enhancement is performed on the original image data to simulate the changes of various actual imaging factors to enrich the diversity of the image and obtain enhanced image data; then, a convolutional neural network is established to extract features, and skip connections and attention gate circuit technology are used to suppress irrelevant or redundant features and highlight key features, thereby obtaining feature data; then, adaptive weighted processing is performed on the extracted feature data to improve the adaptability of the model to different feature images and obtain weighted data; then, morphological operations are performed on the weighted data to further optimize the image quality and obtain data to be evaluated; finally, the evaluation data is used to evaluate the data to be evaluated, and output data is obtained according to the evaluation results; the present invention uses a convolutional neural network to extract features from complex image information, suppress and remove irrelevant features, and then performs adaptive weighted processing on the feature data, dynamically adjusts the weights of key features according to the overall feature distribution of the image, enhances key feature information, ensures that key feature information can be accurately extracted and used in various complex situations, improves the adaptability of the model to multi-feature images, and thereby improves the accuracy and reliability of fundus image segmentation results.

[0042] See also Figure 2 , a second embodiment of the fundus image segmentation method in the embodiment of the present invention includes:

[0043] 201. Collect fundus images of the patient;

[0044] In this embodiment, a fundus imaging device (such as a fundus camera) is used to collect fundus images of the patient; specifically, in order to obtain more comprehensive fundus information, the fundus of the patient needs to be photographed from different angles, such as different angles in the horizontal direction and tilt angles in the vertical direction, etc.; in addition, a variety of imaging modalities can be used, such as color imaging, fluorescence angiography imaging, etc. Different modalities can highlight different structures or pathological characteristics of the fundus, providing a rich source of data for subsequent comprehensive analysis.

[0045] 202. Convert the image format of the fundus image to obtain sorted data;

[0046] In this embodiment, the various formats of the acquired fundus images are converted into a format suitable for subsequent processing and analysis, such as TIFF format; and the image files are classified and organized according to a predetermined directory structure to facilitate subsequent data management and use.

[0047] 203. Perform denoising on the sorted data to obtain denoised data;

[0048] In this embodiment, the noise type of the image in the collated data is first analyzed. Common noise types include Gaussian noise, salt and pepper noise, etc. For different types of noise, a suitable denoising algorithm is selected, and the data is denoised to obtain denoised data. In fundus images, noise may obscure the detailed features of important structures such as blood vessels and lesions. Denoising can highlight these key features.

[0049] 204. Normalize the denoised data to obtain normalized data;

[0050] In this embodiment, the entire batch of image data after denoising is normalized to unify the image pixel values ​​into a specific range or distribution, so that the data between different images are comparable. In subsequent feature extraction, convolutional neural network training and other processes, the convolutional neural network can process data more stably, avoiding problems such as training instability or overfitting caused by large differences in data ranges.

[0051] 205. Perform data extraction on the normalized data to obtain original image data and evaluation data;

[0052] In this embodiment, the normalized data is divided into original image data and evaluation data by random sampling. An appropriate division ratio is set, for example, 80% is used as original image data for model training and feature extraction, and 20% is used as evaluation data for model performance evaluation and verification. In the sampling process, randomness and independence are ensured.

[0053] See also Figure 3 , a third embodiment of the fundus image segmentation method in the embodiment of the present invention includes:

[0054] 301. Simulate illumination changes on the original image data to obtain illumination simulation data;

[0055] In this embodiment, the range of brightness adjustment is first determined, and then the grayscale value of each pixel in the original image data is randomly increased or decreased within the range of brightness adjustment; then the range of contrast adjustment is set, and each pixel in the original image data is subjected to contrast stretching processing, thereby changing the contrast of the image; the original image data is subjected to illumination change simulation to make the difference between different structures in the image (such as blood vessels, background, etc.) more obvious or soft, and obtain illumination simulation data. In actual segmentation tasks, fundus image acquisition may be affected by different lighting conditions, such as lighting differences in hospital examination rooms, illumination intensity settings of different equipment, etc. Illumination change simulation helps convolutional neural networks understand the influence of illumination on fundus image features, so that in actual segmentation tasks, even if images with different illumination conditions from the training data are encountered, accurate segmentation can be performed based on the learned knowledge of illumination feature changes.

[0056] 302. Perform geometric transformation simulation on the original image data to obtain geometric simulation data;

[0057] In this embodiment, the original image data is simulated by geometric transformations such as rotation, translation, and scaling to obtain geometric simulation data, thereby expanding the geometric change form of the original image data and obtaining the characteristic performance of the fundus image at different shooting angles, positions, and focal lengths; in the actual fundus image acquisition process, due to factors such as the patient's eye movement and camera operation differences, the image may appear to be rotated, translated, and scaled to varying degrees. By simulating these geometric transformations, the convolutional neural network can learn the invariance characteristics of the fundus structure in geometric changes, thereby enhancing the generalization ability of the convolutional neural network.

[0058] 303. Perform optical distortion simulation on the original image data to obtain distortion simulation data;

[0059] In this embodiment, firstly, a suitable distortion model is selected, such as barrel distortion and pincushion distortion models, and then suitable distortion parameters are preset; each pixel in the original image is traversed, and its coordinate position after distortion is calculated according to the distortion model and distortion parameters, and distortion simulation data is generated. During the processing, attention is paid to the processing of the edge area of ​​the image. Since the distortion may cause a large displacement of the edge pixels, the edge filling or image boundary extension method can be used to ensure that the generated distortion simulation image is complete and can accurately reflect the distortion effect. During the fundus image acquisition process, optical devices such as camera lenses may produce certain optical distortions, such as barrel distortion or pincushion distortion. By simulating optical distortion, the convolutional neural network learns the characteristic change law of the fundus image under distortion, which enhances the generalization ability of the convolutional neural network.

[0060] 304. Merging the illumination simulation data, the geometry simulation data, and the distortion simulation data to obtain enhanced image data;

[0061] In this embodiment, the illumination simulation data, geometric simulation data and distortion simulation data are merged to obtain enhanced image data. The enhanced image data expands more content than the original image data, including samples of fundus images with changes in illumination, geometry, distortion, etc., providing a wider range of learning resources for the convolutional neural network.

[0062] See also Figure 4 , a fourth embodiment of the fundus image segmentation method in the embodiments of the present invention includes:

[0063] 401. Establish a convolutional neural network, wherein the convolutional neural network includes a shallow convolutional layer and a deep convolutional layer;

[0064] In this embodiment, a convolutional neural network is first established, and multiple shallow convolutional layers and multiple deep convolutional layers can be set; the shallow convolutional layer can use a small convolution kernel of 3×3 with a step size of 1; the deep convolutional layer can appropriately increase the number of convolution kernels or use a larger convolution kernel (such as 5×5); the network architecture of the shallow and deep convolutional layers provides a powerful framework for fundus image feature extraction. The shallow convolutional layer focuses on capturing the local details of the image, such as the edges of blood vessels, the texture of nerve fibers and other initial features. These detail features are crucial for accurately identifying the subtle structures of the fundus. The deep convolutional layer can further learn more abstract features based on the shallow features, such as the overall direction of the blood vessels, the spatial relationship between different structures, etc., thereby realizing the feature extraction process from local to global, from details to semantics.

[0065] 402. Perform preliminary feature extraction on the enhanced image data using a shallow convolutional layer to obtain initial feature data;

[0066] In this embodiment, the convolution kernel of the shallow convolution layer slides on the enhanced image data to perform convolution calculations to capture local detail features in the enhanced image data, such as the edges of small blood vessels and the subtle texture of nerve fibers. Specifically, for the extraction of fundus blood vessel features, a convolution kernel (1*3) with a slender shape is designed so that the direction of the convolution kernel is consistent with the possible direction of the blood vessel. When the convolution kernel slides on the image, it can better capture the linear features of the blood vessels, and enhance the contrast of the blood vessel edges through convolution calculations, making the blood vessels more obvious in the feature map; for small blood vessel branches in fundus images, a smaller and directional convolution kernel can be used to more accurately extract their local features; for the extraction of fundus nerve fiber features, a convolution kernel with texture perception is used, such as a convolution kernel based on the Gabor filter feature. The convolution kernel designed by the method uses the sensitive response ability of the Gabor filter to texture features to extract features of nerve fibers of different thicknesses and directions; for the lesion areas (such as hemorrhages or spots) in the fundus image, multi-scale circular or elliptical convolution kernels are designed to adapt to the lesion features of different sizes and shapes. At the same time, combined with the color features of the lesion area (such as hemorrhages are usually dark red), a convolution kernel sensitive to a specific color channel (dark red) is added to the convolution layer, and a weighted convolution operation is performed on the dark red channel to enhance the feature extraction capability of lesion areas such as hemorrhages. In summary, since the convolution kernel slides on the image, it can extract features from each local area, thereby representing these local features in the form of feature maps; the initial feature data provides the basis for subsequent deep feature extraction.

[0067] 403. Using a deep convolutional layer to extract deep features from the initial feature data to obtain deep feature data;

[0068] In this embodiment, the initial feature data obtained by the shallow convolution layer is passed to the deep convolution layer, and the deep convolution layer also performs convolution operation, so as to learn more abstract and semantically stronger deep feature data; specifically, because the deep convolution layer pays more attention to feature analysis and extraction from a macroscopic perspective, a larger convolution kernel (such as 5×5 or 7×7) is used for convolution operation, so that macroscopic features such as the overall shape, spatial distribution and relationship with surrounding structures of the optic disc and macular area can be identified; for example, when processing the optic disc area, a large convolution kernel can cover most of the optic disc area at one time, extract its boundary contour, connection mode with blood vessels and other feature information, so as to better understand the position of the optic disc in the entire fundus image. and function; for fine structures such as small blood vessels and nerve fibers in fundus images, smaller convolution kernels are still retained in the deep convolution layer for feature extraction. Although these features have been preliminarily extracted in the shallow convolution layer, the deep convolution layer pays more attention to its inherent abstract features, such as the bending pattern of blood vessels and the interweaving pattern of nerve fibers; the deep convolution layer also learns the semantic information in the fundus image. By stacking the convolution results of multiple deep convolution layers, the deep convolution layer can learn the spatial relationships such as the intersection and parallelism between blood vessels and nerve fibers, as well as the relative position relationship between the lesion area and the surrounding normal tissue, so as to correctly understand the nature and cause of fundus lesions and provide assistance for fundus image recognition and diagnosis. In the convolution process, dilated convolution is used instead of traditional convolution to expand the receptive field of the convolution layer without increasing the size of the convolution kernel and the amount of calculation. Dilated convolution controls the interval between the sampling points of the convolution kernel by setting different dilation rates, skips some pixels for convolution calculation, and obtains feature information in a wider range while maintaining the resolution of the feature map. For example, for a 3×3 dilated convolution kernel with a dilation rate of 2, its actual receptive field is equivalent to a 5×5 ordinary convolution kernel, but the amount of calculation is greatly reduced. The dilated convolution kernel helps to discover the potential connections and feature change laws between different regions, and improves the ability to grasp the overall features of the fundus image. The deep feature data reflects the macroscopic relationship and overall features between different structures in the fundus image, such as the overall architecture of the vascular system, the spatial association between the lesion area and the surrounding normal tissue, etc.

[0069] 404. Perform data fusion on the initial feature data and the deep feature data to obtain feature data;

[0070] In this embodiment, a jump connection is used to directly connect the initial feature data of the shallow convolution layer to the input or intermediate layer of the deep convolution layer for feature fusion, and feature information of different scales is combined to obtain feature data. The initial feature data of the shallow layer contains rich detail information, while the deep feature data of the deep layer has more abstract semantic information. After the two are fused, the network can simultaneously utilize local details and global semantic features to better understand the complex structure of the fundus image. Specifically, when establishing a jump connection channel between the shallow convolution layer and the deep convolution layer, since the vascular features have an important position in both the shallow and deep feature data, it is necessary to give a higher weight to the vascular-related feature channel during data fusion, so that the vascular features in the fused feature data are more prominent; at the same time, considering the importance differences of different areas of the fundus image, for example, the optic disc and macular area are usually the key areas for the diagnosis of fundus diseases. When performing data fusion, these key areas need to be processed in a targeted manner, and these areas are fused first, and then other areas are spliced, so as to retain the feature information of the key areas. It should be noted that because deep feature extraction involves multiple pooling operations, the resolution of the image will be reduced, causing the image to lose some information. By fusing the initial feature data and the deep feature data, the edge details of the blood vessels in the shallow features can be retained, and the overall direction and distribution information of the blood vessels in the deep features can be combined, thereby improving the recognition and segmentation accuracy of the vascular structure.

[0071] See also Figure 5 , a fifth embodiment of the fundus image segmentation method in the embodiments of the present invention includes:

[0072] 501. Establish a skip connection channel and an attention gate circuit between the shallow convolution layer and the deep convolution layer;

[0073] In this embodiment, a jump connection channel is established. First, the specific positions of the shallow convolution layer and the deep convolution layer where the jump connection needs to be established are determined; for example, the earlier second layer in the shallow convolution layer is selected to establish a jump connection with the deeper fourth layer, so that the initial feature data output by the second layer can bypass the third layer and directly establish a connection with the fourth layer, and the initial feature data output by the second layer and the deep feature data output by the fourth layer are spliced ​​or added. Fusion operations such as splicing or adding. The jump connection can realize the direct transmission of feature information and avoid the problem of loss of detailed features that may occur during the multi-layer convolution transmission process.

[0074] In this embodiment, an attention gate circuit is further constructed on the jump connection channel, and the attention gate circuit is used to further enhance the fused initial feature data and deep feature data; specifically, the attention gate circuit includes multiple 1×1 convolution layers, and the first 1×1 convolution layer is used to perform dimensionality reduction processing on the spliced ​​feature data, reduce the number of feature channels, and extract the main feature information; the second 1×1 convolution layer restores the reduced-dimensional feature data to the original number of feature channels; in this process, the attention gate circuit dynamically adjusts the weight of the convolution kernel to learn the importance of each channel and each position feature; through this adaptive learning, the attention gate circuit can perform refined processing on the fused feature data, so that the convolutional neural network can focus on the key areas in the data and give higher attention weights to these key areas, so that in subsequent feature processing and segmentation tasks, the convolutional neural network can highlight the key features in the key areas.

[0075] 502. Use a skip connection channel to concatenate the initial feature data with the deep feature data to obtain concatenated data;

[0076] In this embodiment, after establishing the jump connection channel, the initial feature data and the deep feature data can complete data splicing in the channel through the jump connection technology; specifically, before data splicing, it is necessary to adjust the size of the initial feature data and the deep feature data so that they meet the splicing requirements in terms of width, height, and number of channels; then perform the splicing operation according to the channel dimension, integrate the shallow and deep feature information, and form a spliced ​​data with richer data. The initial feature data has rich local detail information, which is crucial for accurately capturing the subtle structural features such as blood vessel edges and nerve fiber textures in fundus images, while the deep depth feature data contains more abstract and semantic global features. The spliced ​​data can carry these multi-level feature information at the same time, so that the subsequent network layers can fully utilize these features. This integration method avoids the problem of isolation or information loss of feature information at different levels in traditional networks. When facing fundus images with multiple complex features, a more complete feature image can be generated, so that the convolutional neural network has more feature basis to distinguish different structures and regions when facing complex fundus images, thereby improving the accuracy and reliability of segmentation.

[0077] 503. Perform local feature analysis on the spliced ​​data using an attention gate circuit to obtain feature data;

[0078] In this embodiment, the convolution module in the attention gate circuit is used to perform a convolution operation on the spliced ​​data to extract the local feature information of each position; the neighborhood features of each pixel position are aggregated and transformed in the spatial dimension through the convolution layer to obtain the feature descriptor corresponding to each position. This process is equivalent to re-encoding the features of each position so that it can better reflect the degree of feature importance of the position in the entire image; then, these feature descriptors are converted into attention weights between 0 and 1 through an activation function (such as a sigmoid function), and the original features in the spliced ​​data are weighted according to the attention weights to achieve differentiated emphasis on features at different positions; through such weighted operations, the discriminating ability of the convolutional neural network for feature data is improved, so that the convolutional neural network can automatically highlight the key areas in the spliced ​​data, suppress the features of non-key areas such as the background, and output more accurate feature data.

[0079] See also Figure 6 , a sixth embodiment of the fundus image segmentation method in the embodiments of the present invention includes:

[0080] 601. Perform global average pooling processing on the feature data to obtain pooled data;

[0081] In this embodiment, the feature data generally exists in the form of a feature map, whose dimension is expressed as W*H*C (where W represents the width of the feature map, H represents the height, and C represents the number of channels). Then, for each channel C, a global average pooling operation is performed respectively, and all pixel values ​​in the channel are summed up and then divided by the total number of pixels to obtain a value representing the overall feature strength of the channel. Through such an operation, the original feature data (feature map) is compressed into a 1*1*C vector, which integrates the global feature information of the entire feature map in each channel, avoids excessive reliance on local features, and enables the importance of features to be considered from a more macro perspective in the future.

[0082] 602. Perform coefficient learning on the pooled data to obtain a weighted coefficient;

[0083] In this embodiment, the pooled data vector obtained by global average pooling is used as input and connected to a learnable module; the learnable module is usually implemented using a fully connected layer in a convolutional neural network; based on the overall feature distribution of the input fundus image data and the requirements of the segmentation task, the importance of each channel feature is adaptively evaluated to generate a weighting coefficient; the weighting coefficient is used to further amplify the key features in the fundus image.

[0084] 603. Adaptively weight the feature data using the weighting coefficient to obtain weighted data;

[0085] In this embodiment, the weighting coefficient is applied to the feature data for weighted processing, and each pixel value of each channel in the original feature map is multiplied by the corresponding weighting coefficient. The features of different channels are correspondingly enhanced or weakened according to the weighting coefficient, so that the key features are highlighted and the non-key features are suppressed; after the above-mentioned weighted calculation, the feature values ​​of all positions and all channels are updated to form a new feature map, namely, weighted data; the weighted data integrates the feature information after adaptive adjustment, and its overall feature representation is more in line with the requirements of the fundus image segmentation task.

[0086] See also Figure 7 , a seventh embodiment of the fundus image segmentation method in the embodiments of the present invention includes:

[0087] 701. Perform data evaluation on the data to be evaluated using the evaluation data to obtain an evaluation result;

[0088] In this embodiment, appropriate evaluation indicators are selected to measure the difference and similarity between the data to be evaluated and the evaluation data; specifically, for fundus vascular features, the vascular segmentation accuracy (VSA) is used as the main evaluation indicator, and the vascular segmentation results in the data to be evaluated are compared pixel by pixel with the real vascular annotations in the evaluation data to evaluate the eligibility of the vascular features; the vascular connectivity index can also be evaluated by calculating the number of vascular breakpoints or the integrity of vascular branches; for fundus nerve fiber features, the nerve fiber texture similarity index (NFTSI) is used to measure, by calculating the distance or similarity metric between the texture feature vectors of the two, calculating the similarity of the nerve fiber texture features in the data to be evaluated and the evaluation data, thereby evaluating the similarity of the nerve fiber texture. In addition to the above-mentioned specific indicators for blood vessels and nerve fibers, the overall structural similarity index (SSIM) is also used to comprehensively evaluate the similarity of the data to be evaluated and the evaluation data in the overall structure of the fundus. SSIM can simultaneously consider the brightness, contrast and structural information of the image, and comprehensively measure the degree of difference between the two images; for each image to be evaluated and the corresponding evaluation image, its SSIM value is calculated, and then the average value is taken as the overall SSIM evaluation result. Then, the evaluation result is judged to be qualified according to the preset evaluation threshold, for example, the threshold of the blood vessel segmentation accuracy is set to 0.9, the threshold of the nerve fiber texture similarity index is set to 0.85, and the threshold of the overall structure similarity index is set to 0.8; if the data to be evaluated reaches or exceeds the corresponding thresholds in these three indicators, the evaluation result is judged to be qualified; otherwise, it is judged to be unqualified; the threshold-based evaluation result judgment method provides a clear standard for the application of the model. In practical applications, when fundus image segmentation is used for clinical diagnosis assistance, higher accuracy may be required, and the qualified labeling of the evaluation results is set more strictly accordingly; in some preliminary screening or large-scale data processing scenarios, the qualified labeling of the evaluation results can be appropriately relaxed.

[0089] 702. If the evaluation result is qualified, the data to be evaluated is marked as output data;

[0090] In this embodiment, when the evaluation result meets the qualification standard, it means that the current data to be evaluated has met the accuracy requirements and can be directly applied to actual scenarios such as diagnosis of ophthalmic diseases and condition monitoring, and the data to be evaluated is marked as output data.

[0091] 703. If the evaluation result is unqualified, adjustment data is generated according to the evaluation data and the data to be evaluated, and the generation process of the data to be evaluated is optimized by using the adjustment data to regenerate the data to be evaluated and evaluate again. This process is repeated until the evaluation result is qualified, and the data to be evaluated at this time is marked as output data.

[0092] In this embodiment, when the evaluation result is unqualified, the difference between the evaluation data and the data to be evaluated is deeply analyzed, and adjustment data is generated; the adjustment data is passed to each step of the process of generating the data to be evaluated through the back propagation algorithm to generate a more prioritized convolutional neural network, or to provide a more accurate basis for adaptive weighted processing, thereby generating more accurate data to be evaluated, and then the evaluation data is used again to evaluate the new data to be evaluated; the above process is repeated until the evaluation result is qualified, and the data to be evaluated at this time is marked as output data. By continuously optimizing in a cycle until the evaluation result is qualified, the output data is guaranteed to have high accuracy and reliability.

[0093] The above describes the fundus image segmentation method in the embodiment of the present invention. The following describes the fundus image segmentation device in the embodiment of the present invention. Figure 8 , an embodiment of the fundus image segmentation device in the embodiment of the present invention includes:

[0094] The image acquisition module 801 is used to acquire the patient's fundus image and perform data preprocessing on the patient's fundus image to obtain original image data and evaluation data;

[0095] An image enhancement module 802 is used to physically enhance the original image data to obtain enhanced image data;

[0096] The feature extraction module 803 is used to establish a convolutional neural network and use the convolutional neural network to extract features from the enhanced image data to obtain feature data;

[0097] The feature weighting module 804 is used to perform adaptive weighting processing on the feature data to obtain weighted data;

[0098] The data optimization module 805 is used to perform morphological operations on the weighted data to obtain data to be evaluated;

[0099] The result evaluation module 806 is used to perform data evaluation on the data to be evaluated using the evaluation data, obtain evaluation results, and obtain output data according to the evaluation results.

[0100] In this embodiment, the image acquisition module 801 is first used to acquire the fundus image of the patient and perform data preprocessing to obtain original image data and evaluation data; then the image enhancement module 802 is used to perform physical enhancement on the original image data, simulating the changes of various actual imaging factors to enrich the diversity of the image, and obtain enhanced image data; then the feature extraction module 803 is used to establish a convolutional neural network for feature extraction, and skip connection and attention gate circuit technology are used to suppress irrelevant or redundant features and highlight key features, thereby obtaining feature data; the feature weighting module 804 then performs adaptive weighting processing on the extracted feature data to improve the adaptability of the model to different feature images, and obtain weighted data; The data optimization module 805 performs morphological operations on the weighted data to further optimize the image quality and obtain the data to be evaluated; finally, the result evaluation module 806 uses the evaluation data to evaluate the data to be evaluated and obtains output data according to the evaluation result; the present invention uses a convolutional neural network to extract features of complex image information, suppresses and removes irrelevant features, and then performs adaptive weighted processing on the feature data, dynamically adjusts the weights of key features according to the overall feature distribution of the image, enhances key feature information, ensures that key feature information can be accurately extracted and utilized in various complex situations, improves the adaptability of the model to multi-feature images, and thereby improves the accuracy and reliability of fundus image segmentation results.

[0101] above Figure 8 The fundus image segmentation device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The fundus image segmentation device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0102] Fig. 9: is a schematic diagram of the structure of a fundus image segmentation device provided by an embodiment of the present invention. The fundus image segmentation device 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 910 (for example, one or more processors) and a memory 920, and one or more storage media 930 (for example, one or more massive storage devices) storing application programs 933 or data 932. Among them, the memory 920 and the storage medium 930 may be temporary storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the fundus image segmentation device 900. Furthermore, the processor 910 may be configured to communicate with the storage medium 930, and execute a series of instruction operations in the storage medium 930 on the fundus image segmentation device 900 to implement the steps of the fundus image segmentation method provided by the above-mentioned method embodiments.

[0103] The fundus image segmentation device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input and output interfaces 960, and / or one or more operating systems 931, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Fig. 9 The structure of the fundus image segmentation device shown does not constitute a limitation on the fundus image segmentation device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0104] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the fundus image segmentation method.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device or unit described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0107] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fundus image segmentation method, characterized in that: include: Collecting fundus images of patients, and performing data preprocessing on the fundus images of patients to obtain original image data and evaluation data; Physically enhancing the original image data to obtain enhanced image data; Establish a convolutional neural network, and use the convolutional neural network to extract features from enhanced image data to obtain feature data; Adaptively weighting the feature data to obtain weighted data; Perform morphological operations on the weighted data to obtain the data to be evaluated; The evaluation data is used to perform data evaluation on the data to be evaluated, to obtain evaluation results, and to obtain output data based on the evaluation results.

2. The fundus image segmentation method according to claim 1, characterized in that: The collecting of the patient's fundus image and performing data preprocessing on the patient's fundus image to obtain original image data and evaluation data include: Acquire fundus images of the patient; Perform image format conversion on fundus images to obtain collated data; De-noising the sorted data to obtain de-noised data; Normalizing the denoised data to obtain normalized data; Data extraction is performed on the normalized data to obtain original image data and evaluation data.

3. The fundus image segmentation method according to claim 1, characterized in that: The physically enhancing the original image data to obtain enhanced image data includes: Simulating illumination changes on the original image data to obtain illumination simulation data; Performing geometric transformation simulation on the original image data to obtain geometric simulation data; Performing optical distortion simulation on original image data to obtain distortion simulation data; The illumination simulation data, the geometric simulation data and the distortion simulation data are merged to obtain enhanced image data.

4. The fundus image segmentation method according to claim 1, characterized in that: The convolutional neural network is established, and the convolutional neural network is used to extract features from the enhanced image data to obtain feature data, including: Establishing a convolutional neural network, wherein the convolutional neural network includes a shallow convolutional layer and a deep convolutional layer; performing preliminary feature extraction on the enhanced image data using the shallow convolutional layer to obtain initial feature data; Use the deep convolution layer to extract deep features from the initial feature data to obtain deep feature data; The initial feature data and the deep feature data are fused to obtain feature data.

5. The fundus image segmentation method according to claim 4, characterized in that: The method of fusing the initial feature data and the deep feature data to obtain the feature data includes: establishing a skip connection channel and an attention gate circuit between the shallow convolution layer and the deep convolution layer; using the skip connection channel to perform data splicing on the initial feature data and the deep feature data to obtain spliced ​​data; The attention gate circuit is used to perform local feature analysis on the spliced ​​data to obtain feature data.

6. The fundus image segmentation method according to claim 1, characterized in that: The step of performing adaptive weighted processing on the feature data to obtain weighted data includes: Perform global average pooling on the feature data to obtain pooled data; Perform coefficient learning on the pooled data to obtain the weighted coefficient; The characteristic data is adaptively weighted using the weighting coefficient to obtain weighted data.

7. A fundus image segmentation device, characterized in that The fundus image segmentation method according to claim 1 is characterized in that the step of using the evaluation data to perform data evaluation on the data to be evaluated to obtain an evaluation result, and obtaining output data according to the evaluation result comprises: Using the evaluation data to evaluate the data to be evaluated, and obtaining the evaluation results; If the evaluation result is qualified, the data to be evaluated is marked as output data; If the evaluation result is unqualified, adjustment data will be generated based on the evaluation data and the data to be evaluated. The adjustment data will be used to optimize the generation process of the data to be evaluated, so as to regenerate the data to be evaluated and evaluate it again. This process will be repeated until the evaluation result is qualified, and the data to be evaluated at this time will be marked as output data.

8. A fundus image segmentation device, characterized in that: include: An image acquisition module is used to acquire fundus images of patients and perform data preprocessing on the fundus images of patients to obtain original image data and evaluation data; An image enhancement module, used for physically enhancing the original image data to obtain enhanced image data; A feature extraction module is used to establish a convolutional neural network and use the convolutional neural network to extract features from the enhanced image data to obtain feature data; The feature weighting module is used to perform adaptive weighting processing on the feature data to obtain weighted data; the data optimization module is used to perform morphological operations on the weighted data to obtain the data to be evaluated; the result evaluation module is used to use the evaluation data to perform data evaluation on the data to be evaluated to obtain the evaluation result, and obtain the output data according to the evaluation result.

9. A fundus image segmentation device, characterized in that: The fundus image segmentation device comprises: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory to enable the fundus image segmentation device to perform each step of the fundus image segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the various steps of the fundus image segmentation method according to any one of claims 1 to 7 are implemented.

Images