Fundus image optimization method and equipment for auxiliary diagnosis of glaucoma
Through the combination of multi-feature extraction module and attention module, the fundus image is optimized using a feature migration compensation mechanism, which solves the problem of poor retinal vascular image segmentation effect in the prior art and significantly improves diagnostic efficiency.
Patent Information
- Application Number
- CN202411897877.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
AI Technical Summary
Among the existing glaucoma diagnostic techniques, the retinal vascular image segmentation effect is poor, resulting in insufficiency of diagnosis.
The multi-feature extraction module is used to connect in series and combine the attention module to optimize the fundus image through the feature migration compensation mechanism, and adaptive images are generated to improve the accuracy of retinal vascular image segmentation.
The recognition accuracy of the retinal terminal blood vessels is significantly improved, and the terminal blood vessels are more obvious in the optimized image, greatly improving the effectiveness of image segmentation work.
Smart Images
Figure CN119833085A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of auxiliary diagnosis of glaucoma, and in particular relates to a fundus image optimization method and equipment for auxiliary diagnosis of glaucoma. Background Art
[0002] The permanent visual impairment caused by glaucoma will seriously affect the patient's labor income, greatly increase the burden on the medical system, and the loss of life quality will also cause a heavy socioeconomic burden; visual defects not only endanger the quality of the contemporary population, but also affect the quality of my country's future population. Glaucoma is an optic neuropathy with increased intraocular pressure as the main risk factor and apoptosis of retinal ganglion cells (RGCs) as the main pathological feature. Although lowering intraocular pressure is currently recognized as the only effective treatment for glaucoma, it cannot completely prevent the progression of optic nerve damage in some glaucoma patients. The ultimate common pathway of optic nerve damage in glaucomatous eyes is the death of retinal ganglion cells. Existing treatments for glaucoma can delay the apoptosis of retinal ganglion cells to a certain extent, but they cannot restore dead optic ganglion cells.
[0003] Glaucoma is the second leading cause of blindness in the world, with a huge patient base. Many institutions or individuals around the world have conducted related research. Whether in clinical diagnosis and treatment or scientific research, professionals are required to frequently diagnose glaucoma, which takes a lot of time and energy. In order to solve this problem, an intelligent diagnostic solution combining machine learning with fundus images has been developed. Segmenting retinal vascular images from fundus images is an important step in diagnosis using machine learning algorithms. However, due to the complex visual appearance of retinal blood vessels and the presence of many unclear areas, the segmentation effect of existing algorithms is still relatively poor. Summary of the invention
[0004] In view of this, the present invention provides a fundus image optimization method and device for auxiliary diagnosis of glaucoma, which performs special adaptive optimization on fundus images for retinal vascular image segmentation tasks to overcome the above-mentioned shortcomings of the prior art.
[0005] The technical scheme is as follows: A fundus image optimization method for auxiliary diagnosis of glaucoma, comprising the following steps:
[0006] Multiple feature extraction modules are connected in series in sequence, and multiple feature extraction modules are used to perform feature extraction operations on the fundus image step by step. In the specific calculation process, the first feature extraction module takes the fundus image as input, outputs data at its tail after calculation and hands it over to the next feature extraction module for processing. Therefore, each subsequent feature extraction module receives the data output at the tail of the previous feature extraction module connected to it as input.
[0007] Furthermore, an attention module is provided in the feature extraction module, and the attention module is used to generate an attention map.
[0008] Performing an optimization operation on feature data generated by at least one previous feature extraction module based on at least one attention map generated in at least one subsequent feature extraction module to generate at least one optimized feature data;
[0009] An adapted image is constructed based on at least one optimized characteristic data and at least one characteristic data generated by a subsequent feature extraction module.
[0010] In fundus images, retinal blood vessels show a gradually divergent distribution characteristic, and the target image information and background noise are arranged in a mesh-like cross pattern. In order to enable the feature extraction module to better cope with the above phenomenon, the feature extraction module preferably adopts the following operation process to form a multi-head feature confusion extraction mechanism. Specifically, the internal operation process of the feature extraction module includes:
[0011] After calculating the head characteristic data input to the feature extraction module through a first convolution operation unit, a second convolution operation unit and a deformable convolution operation unit arranged in parallel, first process characteristic data, second process characteristic data and third process characteristic data are generated respectively;
[0012] multiplying the corresponding elements of the second process characteristic data and the third process characteristic data and activating them to generate fourth process characteristic data;
[0013] Adding the fourth process characteristic data to the corresponding elements of the first process characteristic data to generate fifth process characteristic data;
[0014] After the fifth process characteristic data is calculated by a third convolution operation unit, sixth process characteristic data is generated;
[0015] After the fourth process characteristic data is processed by the attention module, the attention map is generated;
[0016] After adjusting the sixth process characteristic data via the attention map, tail characteristic data output by the feature extraction module is generated.
[0017] Furthermore, the first convolution operation unit includes a first convolution layer and a first activation layer arranged in sequence, and the convolution kernel size of the first convolution layer is 1×1; the second convolution operation unit includes a second convolution layer and a second activation layer arranged in sequence, and the convolution kernel size of the second convolution layer is 3×3; the third convolution operation unit includes a third convolution layer and a third activation layer arranged in sequence, and the convolution kernel size of the third convolution layer is 3×3; the deformable convolution operation unit includes a deformable convolution layer and a fourth activation layer arranged in sequence, and the convolution kernel size of the deformable convolution layer is 3×3.
[0018] Furthermore, based on the attention maps generated in the subsequent feature extraction modules, the tail feature data output by the multiple previous feature extraction modules are optimized to generate at least one optimized feature data.
[0019] Furthermore, M feature extraction modules are used to perform feature extraction operations on the fundus image step by step. When M is an even number greater than 2, the characteristic data generated by the first M / 2 feature extraction modules are optimized based on the attention map generated in the latter M / 2 feature extraction modules; when M is an odd number greater than 2, the characteristic data generated by the first (M-1) / 2 feature extraction modules are optimized based on the attention map generated in the latter (M-1) / 2 feature extraction modules.
[0020] Furthermore, the attention module is a spatial attention module, and the attention map is a spatial attention map.
[0021] Further, the process of performing an optimization operation on the tail feature data output by the plurality of preceding feature extraction modules based on the attention maps generated in the plurality of succeeding feature extraction modules to generate at least one optimized feature data comprises:
[0022] Adding corresponding elements of tail characteristic data output by a plurality of previous feature extraction modules to generate first-order optimal data;
[0023] splicing together a plurality of attention maps generated in the subsequent feature extraction modules to generate first adjustment data, and adding corresponding elements of the attention maps generated in the subsequent feature extraction modules to generate second adjustment data;
[0024] Performing a first convolution process and a first activation process on the first adjusted data in sequence to generate third adjusted data, wherein the channel size of the third adjusted data is 1; multiplying the third adjusted data by the first optimized data to generate second optimized data;
[0025] Performing an expansion process on the second adjusted data to generate fourth adjusted data, wherein the fourth adjusted data is a one-dimensional vector; performing a second convolution process and a second activation process on the fourth adjusted data in sequence to generate fifth adjusted data, wherein the length of the fifth adjusted data is equal to the size of the second optimal data channel;
[0026] The fifth adjustment data is multiplied by the second optimal data to generate the optimized characteristic data.
[0027] The attention map generated in the subsequent feature extraction module uses a recursive serial control optimization adjustment scheme to optimize the tail feature data output by the previous feature extraction module, so that the optimized feature data and the tail feature data output by the subsequent feature extraction module have a more unified style and a better continuous image feature distribution, so that the generated adapted image has good coherence on a small scale, and the images at each position are more closely connected, effectively avoiding the phenomenon of disordered hierarchical boundaries in areas with dense high-frequency information.
[0028] Furthermore, an adapted image is constructed based on the optimized characteristic data and tail characteristic data output by a plurality of subsequent feature extraction modules.
[0029] Furthermore, the process of constructing an adapted image based on the optimized characteristic data and tail characteristic data output by a plurality of subsequent feature extraction modules includes:
[0030] splicing the optimized characteristic data with the tail characteristic data output by the subsequent feature extraction modules to generate adapted characteristic data;
[0031] The adapted characteristic data is sequentially subjected to a third convolution process and a third activation process to generate the adapted image.
[0032] The above process of generating an adapted image from the original fundus image through a series of processing or calculation constitutes a convolutional neural network. As is well known, the convolutional neural network usually needs to be trained before the above processing or calculation. When constructing a training data set for a convolutional neural network, the unclear retinal blood vessels in the collected sample fundus image are artificially enhanced to obtain an enhanced image. The enhanced image and the sample fundus image before enhancement constitute an image pair for training the neural network.
[0033] The present invention also provides a fundus image optimization device for auxiliary diagnosis of glaucoma, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the method as described above by loading the computer program.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The study found that as the depth of the neural network increases, the subsequent feature extraction module can well learn the more obvious and larger blood vessel image features, while the previous feature extraction module relatively learns more terminal blood vessel image features with a smaller range. Since the image features of the terminal blood vessels are not obvious, have irregular shapes, lack a unified feature paradigm, etc., the current algorithms have poor recognition effects on the terminal retinal blood vessels. The present invention creatively designs a feature migration compensation mechanism, which migrates the retinal blood vessel pattern learned by the subsequent feature extraction module to the previous feature extraction module, improves the recognition accuracy of the terminal blood vessels through the front and back coordination, and enhances the terminal blood vessels in the optimized image, making the visual performance more obvious, greatly improving the effectiveness of subsequent image segmentation work. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a schematic diagram of the flow chart of the fundus image optimization method of the present invention. DETAILED DESCRIPTION
[0037] The present invention will be further described in detail below in conjunction with the accompanying drawings.
[0038] This embodiment provides a fundus image optimization method for auxiliary diagnosis of glaucoma. Figure 1 As shown, the following steps are included:
[0039] M feature extraction modules are used to perform feature extraction operations on the fundus image step by step, where M is usually a positive integer greater than or equal to 3. An attention module is provided in the feature extraction module, and the attention module is used to generate an attention map. Based on the attention map generated in at least one subsequent feature extraction module, the characteristic data generated by at least one previous feature extraction module is optimized to generate at least one optimized characteristic data; based on the at least one optimized characteristic data and the characteristic data generated by at least one subsequent feature extraction module, an adapted image is constructed, the adapted image has 3 channels and the same resolution as the fundus image.
[0040] That is to say, the attention map generated in one or more subsequent feature extraction modules can be arbitrarily selected to optimize the feature data generated by any one or more previous feature extraction modules. For example, the attention map generated in the last feature extraction module can be selected to optimize the feature data generated by the first feature extraction module, or the attention maps generated in the last two feature extraction modules can be selected to optimize the feature data generated by the first feature extraction module, or the attention maps generated in the third and fifth feature extraction modules can be selected to optimize the feature data generated by the first feature extraction module, or the attention map generated in the fifth feature extraction module can be selected to optimize the feature data generated by the first and third feature extraction modules, or the attention maps generated in the fourth and sixth feature extraction modules can be selected to optimize the feature data generated by the first and second feature extraction modules.
[0041] Similarly, one or more optimized characteristic data and one or more characteristic data generated by subsequent feature extraction modules can be arbitrarily selected to construct an adapted image. For example, the first optimized characteristic data and the characteristic data generated in the last feature extraction module can be selected to construct an adapted image, or the first optimized characteristic data and the characteristic data generated in the last two feature extraction modules can be selected to construct an adapted image, or the second optimized characteristic data and the characteristic data generated in the third, fifth, and sixth feature extraction modules can be selected to construct an adapted image, or the second and third optimized characteristic data and the characteristic data generated in the sixth feature extraction module can be selected to construct an adapted image, or the second and fourth optimized characteristic data and the characteristic data generated in the fifth and sixth feature extraction modules can be selected to construct an adapted image.
[0042] As one of the implementation modes, when M is an even number greater than 2, the characteristic data generated by the previous M / 2 feature extraction modules are optimized based on the attention map generated in the subsequent M / 2 feature extraction modules; when M is an odd number greater than 2, the characteristic data generated by the previous (M-1) / 2 feature extraction modules are optimized based on the attention map generated in the subsequent (M-1) / 2 feature extraction modules.
[0043] As an example, this embodiment uses six serially connected feature extraction modules to perform feature extraction operations on the fundus image step by step, optimizes the characteristic data generated by the first three feature extraction modules based on the attention map generated in the last three feature extraction modules, generates an optimized characteristic data, and then constructs an adapted image based on the optimized characteristic data and the characteristic data generated by the last three feature extraction modules.
[0044] The feature extraction module in the present invention can be implemented using many existing algorithms, such as VGG module (VGGBlock), residual module (Residual Block) or densely connected module (Densely-Connected Block). As a preferred implementation, the internal operation process of the feature extraction module in this embodiment includes:
[0045] After calculating the head characteristic data of the input feature extraction module through the first convolution operation unit, the second convolution operation unit and the deformable convolution operation unit arranged in parallel, first process characteristic data, second process characteristic data and third process characteristic data of equal size are generated respectively;
[0046] The second process characteristic data is multiplied by the corresponding elements of the third process characteristic data and then activated (ReLU or tanh activation may be used) to generate fourth process characteristic data;
[0047] Adding the fourth process characteristic data to the corresponding elements of the first process characteristic data to generate fifth process characteristic data;
[0048] After the fifth process characteristic data is calculated by the third convolution operation unit, sixth process characteristic data is generated;
[0049] After processing the fourth process characteristic data through the attention module, an attention map is generated;
[0050] After adjusting the sixth process characteristic data through the attention map, the tail characteristic data output by the feature extraction module is generated.
[0051] The first convolution operation unit includes a first convolution layer and a first activation layer arranged in sequence, and the convolution kernel size of the first convolution layer is 1×1; the second convolution operation unit includes a second convolution layer and a second activation layer arranged in sequence, and the convolution kernel size of the second convolution layer is 3×3; the third convolution operation unit includes a third convolution layer and a third activation layer arranged in sequence, and the convolution kernel size of the third convolution layer is 3×3; the deformable convolution operation unit includes a deformable convolution layer and a fourth activation layer arranged in sequence, and the convolution kernel size of the deformable convolution layer is 3×3. The strides of the first convolution layer, the second convolution layer, the third convolution layer and the deformable convolution layer can all be set to 1.
[0052] The attention module can be directly implemented using many existing public algorithms. For example, in this embodiment, the attention module is a spatial attention module. The internal calculation process of the attention module includes: using the Average Pooling layer (also called the global average pooling layer) and the activation layer to process the fourth process characteristic data in sequence, that is, to obtain an attention map, wherein the Average Pooling layer processes the fourth process characteristic data along the channel dimension to generate a two-dimensional matrix attention map, and the activation layer can use a sigmoid function. The height and width of the attention map are equal to the height and width of the sixth process characteristic data, and the attention map is broadcast along the channel direction. The size of the broadcasted attention map is equal to the size of the sixth process characteristic data. The broadcasted attention map is multiplied by the corresponding elements of the sixth process characteristic data to generate tail characteristic data.
[0053] The attention map generated in the subsequent feature extraction module can optimize the characteristic data generated at various positions in the middle or at the tail of the previous feature extraction module to generate optimized characteristic data. For example, at least one of the first process characteristic data, the second process characteristic data, the third process characteristic data, the fourth process characteristic data, the fifth process characteristic data, the sixth process characteristic data and the tail characteristic data can be optimized to generate optimized characteristic data. Exemplarily, this embodiment optimizes the tail characteristic data output by the first three feature extraction modules based on the attention map generated in the latter three feature extraction modules to generate an optimized characteristic data, and the specific operation process includes:
[0054] The corresponding elements of the tail feature data output by the first three feature extraction modules are added together to generate the first optimization data;
[0055] The attention maps generated in the following three feature extraction modules are concatenated to generate the first adjustment data, and the corresponding elements of the attention maps generated in the following three feature extraction modules are added to generate the second adjustment data;
[0056] The first adjusted data is sequentially subjected to the first convolution processing (convolution kernel size 1×1, stride 1) and the first activation processing (sigmoid function) to generate the third adjusted data, the channel size of the third adjusted data is 1, and the height and width are equal to the height and width of the first optimal data; the third adjusted data is broadcasted along the channel direction, the size of the third adjusted data after broadcasting is equal to the size of the first optimal data, and then the third adjusted data after broadcasting is multiplied by the corresponding elements of the first optimal data to generate the second optimal data;
[0057] The second adjusted data is expanded (the expansion process can be implemented by the existing flatten layer) to generate fourth adjusted data, which is a one-dimensional vector; the fourth adjusted data is sequentially subjected to the second convolution process (the convolution kernel size is 1×1, and the stride is 1) and the second activation process (sigmoid function) to generate fifth adjusted data, and the length of the fifth adjusted data is equal to the channel size of the second optimized data;
[0058] The fifth adjusted data is broadcasted along the spatial direction, and the size of the fifth adjusted data after broadcasting is equal to the size of the second optimal data. The fifth adjusted data after broadcasting is multiplied by corresponding elements of the second optimal data to generate optimized characteristic data.
[0059] An adapted image may be constructed based on the optimized characteristic data and the characteristic data generated at various positions or at the tail of the subsequent feature extraction module. For example, an adapted image may be constructed based on the optimized characteristic data and at least one of the first process characteristic data, the second process characteristic data, the third process characteristic data, the fourth process characteristic data, the fifth process characteristic data, the sixth process characteristic data, and the tail characteristic data generated in the subsequent feature extraction module. Exemplarily, this embodiment constructs an adapted image based on the optimized characteristic data and the tail characteristic data generated in the following three feature extraction modules, and the specific process includes:
[0060] The optimized feature data is spliced with the tail feature data output by the following three feature extraction modules to generate the adapted feature data;
[0061] The adaptation characteristic data is subjected to the third convolution processing (convolution kernel size 3×3, stride 1) and the third activation processing (ReLU function) in sequence to generate an adapted image.
[0062] Finally, it should be noted that the above description is only a preferred embodiment of the present invention. Under the guidance of the present invention, ordinary technicians in this field can make various similar expressions without violating the purpose and claims of the present invention, and such changes all fall within the scope of protection of the present invention.
Claims
1. A fundus image optimization method for auxiliary diagnosis of glaucoma, characterized in that: The following steps are involved: Using multiple feature extraction modules to perform feature extraction operations on the fundus image step by step, wherein the feature extraction module is provided with an attention module, and the attention module is used to generate an attention map; Performing an optimization operation on feature data generated by at least one previous feature extraction module based on at least one attention map generated in at least one subsequent feature extraction module to generate at least one optimized feature data; An adapted image is constructed based on at least one optimized characteristic data and at least one characteristic data generated by a subsequent feature extraction module.
2. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 1, characterized in that: The internal operation process of the feature extraction module includes: After calculating the head characteristic data input to the feature extraction module through a first convolution operation unit, a second convolution operation unit and a deformable convolution operation unit arranged in parallel, first process characteristic data, second process characteristic data and third process characteristic data are generated respectively; multiplying the corresponding elements of the second process characteristic data and the third process characteristic data and activating them to generate fourth process characteristic data; Adding the fourth process characteristic data to the corresponding elements of the first process characteristic data to generate fifth process characteristic data; After the fifth process characteristic data is calculated by a third convolution operation unit, sixth process characteristic data is generated; After the fourth process characteristic data is processed by the attention module, the attention map is generated; After adjusting the sixth process characteristic data via the attention map, tail characteristic data output by the feature extraction module is generated.
3. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 2, characterized in that: The first convolution operation unit includes a first convolution layer and a first activation layer arranged in sequence, and the convolution kernel size of the first convolution layer is 1×1; the second convolution operation unit includes a second convolution layer and a second activation layer arranged in sequence, and the convolution kernel size of the second convolution layer is 3×3; the third convolution operation unit includes a third convolution layer and a third activation layer arranged in sequence, and the convolution kernel size of the third convolution layer is 3×3; the deformable convolution operation unit includes a deformable convolution layer and a fourth activation layer arranged in sequence, and the convolution kernel size of the deformable convolution layer is 3×3.
4. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 2, characterized in that: Based on the attention maps generated in the subsequent feature extraction modules, the tail feature data output by the multiple previous feature extraction modules are optimized to generate at least one optimized feature data.
5. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 1 or 4, characterized in that: M feature extraction modules are used to perform feature extraction operations on the fundus image step by step. When M is an even number greater than 2, the characteristic data generated by the first M / 2 feature extraction modules are optimized based on the attention map generated in the latter M / 2 feature extraction modules; when M is an odd number greater than 2, the characteristic data generated by the first (M-1) / 2 feature extraction modules are optimized based on the attention map generated in the latter (M-1) / 2 feature extraction modules.
6. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 4, characterized in that: The attention module is a spatial attention module, and the attention map is a spatial attention map.
7. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 6, characterized in that: The process of performing an optimization operation on tail feature data output by a plurality of preceding feature extraction modules based on attention maps generated in a plurality of succeeding feature extraction modules to generate at least one optimized feature data comprises: Adding corresponding elements of tail characteristic data output by a plurality of previous feature extraction modules to generate first-order optimal data; splicing together a plurality of attention maps generated in the subsequent feature extraction modules to generate first adjustment data, and adding corresponding elements of the attention maps generated in the subsequent feature extraction modules to generate second adjustment data; Performing a first convolution process and a first activation process on the first adjusted data in sequence to generate third adjusted data, wherein the channel size of the third adjusted data is 1; multiplying the third adjusted data by the first optimized data to generate second optimized data; Performing an expansion process on the second adjusted data to generate fourth adjusted data, wherein the fourth adjusted data is a one-dimensional vector; performing a second convolution process and a second activation process on the fourth adjusted data in sequence to generate fifth adjusted data, wherein the length of the fifth adjusted data is equal to the size of the second optimal data channel; The fifth adjustment data is multiplied by the second optimal data to generate the optimized characteristic data.
8. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 2 or 7, characterized in that: An adapted image is constructed based on the optimized characteristic data and tail characteristic data output by a plurality of subsequent feature extraction modules.
9. The fundus image optimization method for auxiliary diagnosis of glaucoma according to claim 8, characterized in that: The process of constructing an adapted image based on the optimized characteristic data and tail characteristic data output by a plurality of subsequent feature extraction modules includes: splicing the optimized characteristic data with the tail characteristic data output by the subsequent feature extraction modules to generate adapted characteristic data; The adapted characteristic data is sequentially subjected to a third convolution process and a third activation process to generate the adapted image.
10. A fundus image optimization device for auxiliary diagnosis of glaucoma, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the method according to any one of claims 1 to 9 by loading the computer program.