Multi-modal fundus image fusion method and device, computer equipment and storage medium
Patent Information
- Application Number
- CN202411795522.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-08-27
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-08
Smart Images
Figure CN119784582A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image fusion method, and more specifically to a multimodal fundus image fusion method, device, computer equipment and storage medium. Background Art
[0002] In clinical screening for fundus diseases, fundus images collected by patients through various methods and devices belong to different modalities. Doctors need to compare these images of different modalities to find fundus abnormalities and lesions, so as to make an accurate diagnosis; however, due to the different shooting fields of different modal images, the position of the same fundus structure in each modality is different, which increases the difficulty of doctors' positioning and judgment and affects the efficiency of diagnosis. Therefore, it is necessary to align and fuse these fundus images of different modalities to improve the efficiency and quality of diagnosis.
[0003] Traditional image fusion methods usually only support two or specific modalities of fundus images. However, in clinical practice, doctors will take images of certain modalities in a targeted manner based on the patient's medical history and specific condition, which limits the applicability of these methods. In addition, image fusion includes two stages: registration and fusion, and most of the existing methods are based on traditional computer vision technology. Although these technologies have good effects on overall registration, the accuracy of local details still needs to be improved, especially for the retention of important structures such as fundus blood vessels. The consistency problem of existing methods in the fusion process has not been completely solved.
[0004] Therefore, it is necessary to design a new method to solve the limitations of the registration and fusion of fundus images of different modalities in the clinical screening of fundus diseases, the low accuracy of local details, and the consistency problem of preserving important structures such as fundus blood vessels. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a multimodal fundus image fusion method, device, computer equipment and storage medium.
[0006] To achieve the above object, the present invention adopts the following technical solution: a multimodal fundus image fusion method, comprising:
[0007] Acquire multimodal fundus images to be fused;
[0008] Determining a reference modality image and a registration modality image from the multi-modality fundus images to be fused;
[0009] Calculating a registration order according to the reference modality image and the registration modality image;
[0010] Performing coarse registration on the reference modality image and the registration modality image to obtain a preliminary registration result;
[0011] Performing blood vessel prior segmentation on the reference modality image and the registration modality image to obtain blood vessel prior information;
[0012] Performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result;
[0013] The fusion result is output.
[0014] A further technical solution is: calculating the registration sequence according to the reference modality image and the registration modality image comprises:
[0015] Calculating the similarity between the reference modality image and the registration modality image using Chamfer distance;
[0016] The Kruskal algorithm is used to generate the minimum spanning tree of the registration order, and the order of the minimum spanning tree is used as the registration order.
[0017] A further technical solution is: the coarse registration of the reference modality image and the registration modality image to obtain a preliminary registration result includes:
[0018] Using a SURF detector in sequence according to the registration order to detect feature points of the reference modality image and the registration modality image;
[0019] Extracting and describing feature points of the reference modality image and the registration modality image using the PIIFD algorithm;
[0020] The RANSAC algorithm is used to perform pattern matching and delete unmatched points, and the matched images are deformed to obtain preliminary registration results.
[0021] A further technical solution is: performing a priori segmentation of blood vessels on the reference modality image and the registration modality image to obtain a priori information of blood vessels includes:
[0022] A vascular priori segmentation network is used to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information.
[0023] A further technical solution is: using a vascular prior segmentation network to perform vascular prior segmentation on the reference modality image and the registration modality image to obtain vascular prior information, including:
[0024] Using the blood vessel priori segmentation network to perform feature encoding on the reference modality image and the registration modality image to obtain an encoding result;
[0025] The encoding result is decoded by a decoder to obtain blood vessel prior information.
[0026] A further technical solution is: performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, including:
[0027] The reference modality image, the preliminary registration result and the blood vessel prior information are input into a blood vessel prior fusion network to perform image fusion to obtain a fusion result.
[0028] A further technical solution is: after performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, it also includes:
[0029] The generated fusion image is segmented using a blood vessel prior segmentation network to form a fusion image segmentation result;
[0030] Comparing the fused image segmentation result with the blood vessel priori segmentation result of the reference modality image to calculate the loss value;
[0031] The loss value is used to optimize the parameters of the blood vessel priori segmentation network.
[0032] The present invention also provides a multimodal fundus image fusion device, comprising:
[0033] An image acquisition unit, used for acquiring multimodal fundus images to be fused;
[0034] A determination unit, used to determine a reference modality image and a registration modality image from the multi-modality fundus images to be fused;
[0035] A registration order calculation unit, used for calculating a registration order according to the reference modality image and the registration modality image;
[0036] A coarse registration unit, used for coarsely registering the reference modality image and the registration modality image to obtain a preliminary registration result;
[0037] a segmentation unit, configured to perform a priori blood vessel segmentation on the reference modality image and the registration modality image to obtain a priori blood vessel information;
[0038] A fusion unit, configured to perform vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result;
[0039] An output unit is used to output the fusion result.
[0040] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.
[0041] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0042] The beneficial effects of the present invention compared with the prior art are as follows: the present invention acquires multimodal fundus images, determines reference and registration images, performs coarse registration and vascular prior segmentation, and finally fuses the images to output the results, thereby improving the contrast and details of the images, thereby enhancing the accuracy and comprehensiveness of the diagnosis; it can solve the problems of limitations in the registration and fusion of fundus images of different modalities in clinical screening of fundus diseases, low accuracy of local details, and consistency issues in retaining important structures such as fundus blood vessels.
[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0045] Figure 1 A schematic diagram of an application scenario of the multimodal fundus image fusion method provided by an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of a flow chart of a multimodal fundus image fusion method provided by an embodiment of the present invention;
[0047] Figure 3 A schematic block diagram of a multimodal fundus image fusion device provided by an embodiment of the present invention;
[0048] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0051] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0052] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0053] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the multimodal fundus image fusion method provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a multimodal fundus image fusion method provided by an embodiment of the present invention. The multimodal fundus image fusion method is applied to a server. The server interacts with a terminal, and inputs a multimodal fundus image to be fused through the terminal. The two images, the reference and the registration, are first determined, and the registration modality image and the reference modality image are sequentially input into a neural network according to the calculated registration order to obtain a directly fused image. At the same time, in order to ensure the consistency of the vascular structure before and after image fusion, the reference image and the fused image are respectively segmented and constrained to be consistent, so as to ensure that the important structural information remains unchanged; the fusion process of this embodiment first performs a rough registration, and then performs a fine registration, and the input of the rough registration stage is a fundus image with modality differences; in addition to the fundus image, the fine registration stage adds vascular segmentation. In addition, the fine registration stage does not use vascular segmentation as the registration basis, but uses vascular segmentation to retain important image information; it can solve the problem of limitations in the registration and fusion of fundus images of different modalities in the clinical screening of fundus diseases, low accuracy of local details, and consistency of important structures such as fundus blood vessels.
[0054] Figure 2 FIG. 1 is a flow chart of a multimodal fundus image fusion method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S170.
[0055] S110: Acquire multimodal fundus images to be fused.
[0056] In this embodiment, the multimodal fundus images to be fused refer to fundus images from different modalities, which may be acquired through different imaging technologies or different spectral ranges, in order to obtain more comprehensive and multi-angle fundus structure information. Generally, these fundus images of different modalities may include:
[0057] Color Fundus Photograph: This is the most common fundus imaging modality and is used for general eye examination and documentation.
[0058] Infrared fundus images: Infrared imaging can see deeper structures of the fundus, such as the deep retina and blood vessels.
[0059] Fluorescent angiography images: By injecting fluorescent dye and using specific imaging equipment, the morphology and distribution of blood vessels in the fundus can be shown more clearly.
[0060] Optical coherence tomography (OCT) images: OCT provides high-resolution images of the retinal structure, including details of the retinal nerve fiber layer and the macular region.
[0061] Ultrasound fundus imaging (US): Ultrasound imaging can penetrate the eyeball and provide information about the structure behind the retina, and has good diagnostic value for lesions such as posterior retinal detachment.
[0062] Other specific fundus functional imaging: for example, automated perimetry (AFI) and vascular detection (VDI), etc., for specific lesions or functional assessment.
[0063] These different fundus imaging modalities each have specific advantages and scopes of application. By integrating them together, they can provide doctors with more comprehensive and accurate information on fundus structure and lesions, facilitating more precise diagnosis and treatment planning.
[0064] S120 , determining a reference modality image and a registration modality image from the multi-modality fundus images to be fused.
[0065] In this embodiment, the reference modality image is an image selected as a reference or main basis in the multimodality image. Usually, the reference modality image is selected as the modality with the highest resolution, the most extensive structural information or the most commonly used modality. For example, in fundus images, the commonly used reference modality can be a color fundus photograph or a high-resolution OCT image. The selection of the reference modality image depends on the specific application and fusion target.
[0066] Registered modality images refer to other modality images that need to be aligned with the reference modality images through registration technology. These images may come from different imaging devices or use different imaging parameters, so their spatial positions and scales may be different. The purpose of registration is to align the information of these modality images to the spatial coordinate system of the reference modality for subsequent fusion and analysis.
[0067] Specifically, one modality is first determined as the reference modality, and the other modalities are used as registration modalities. In the process of fundus image fusion, it is necessary to first select an appropriate reference modality, and then register all other modality images to ensure that they are aligned in the same spatial coordinate system. This can integrate information from different modalities and improve the comprehensiveness and accuracy of fundus structure and lesion analysis.
[0068] S130, calculating a registration order according to the reference modality image and the registration modality image.
[0069] In this embodiment, the registration order refers to the order in which multiple modality images (such as the reference modality image and the registration modality image) are aligned to the same spatial coordinate system.
[0070] In one embodiment, the above-mentioned step S130 may include steps S131 - S132 .
[0071] S131. Calculate the similarity between the reference modality image and the registration modality image using Chamfer distance.
[0072] In this embodiment, the Chamfer distance (or Chamfer matching) is used to calculate the similarity between the reference modality image and each registered modality image. The Chamfer distance is a commonly used distance measurement method for comparing the similarity between two shapes. In the context of fundus images, the Chamfer distance can help determine the spatial offset and rotation angle between each registered modality image and the reference modality image.
[0073] Chamfer distance can provide a more accurate shape matching metric, which helps to accurately evaluate the similarity between images of different modalities. Compared with other complex matching algorithms, Chamfer distance calculation is relatively efficient and suitable for large-scale image data processing.
[0074] S132. Generate a minimum spanning tree of the registration order using the Kruskal algorithm, and use the order of the minimum spanning tree as the registration order.
[0075] In this embodiment, the Kruskal algorithm is used to generate a minimum spanning tree of the registration order. The minimum spanning tree is a tree that contains all vertices but does not form a loop and has the smallest sum of edge weights. Here, the vertices can be regarded as the modal images to be registered, that is, the registered modal images, and the edge weights can be regarded as the similarity calculated based on the Chamfer distance. In the subsequent registration process, the registered modal image is fused with the reference modal image in turn according to the minimum spanning tree.
[0076] By generating a minimum spanning tree, an optimized registration order can be determined to ensure that each step of the registration operation is based on the minimum transformation amount, thereby improving the accuracy and stability of the overall registration; the Kruskal algorithm has relatively low computational complexity and can effectively handle large-scale registration problems while ensuring the optimality of the results.
[0077] Taken together, these steps, combined with the use of Chamfer distance and Kruskal algorithm, can effectively determine the registration order of multimodal fundus images, thereby achieving accurate image fusion and analysis, and providing reliable support for medical diagnosis and research.
[0078] S140: Roughly register the reference modality image and the registration modality image to obtain a preliminary registration result.
[0079] In this embodiment, in the context of image registration, the preliminary registration result refers to the preliminary aligned image obtained through the coarse registration step.
[0080] In one embodiment, the above-mentioned step S140 may include steps S141 - S143 .
[0081] S141 . Using a SURF detector in sequence according to a registration order to detect feature points of the reference modality image and the registration modality image.
[0082] In this embodiment, a SURF (Speeded Up Robust Features) detector is used to detect feature points in the reference modality image and the registration modality image in sequence. SURF is an algorithm based on local features and is used to detect key points in an image. These key points usually correspond to significant structures and regions in the image.
[0083] Due to its efficient design, the SURF algorithm can quickly and accurately detect feature points in images and is suitable for real-time or large-scale image data processing. SURF has good robustness to image rotation, scaling and partial occlusion, and can effectively find similar feature points between different modal images.
[0084] S142: Extract and describe feature points of the reference modality image and the registration modality image using a PIIFD algorithm.
[0085] In this embodiment, the PIIFD (Pose Invariant Image Feature Descriptor) algorithm is used to extract and describe each detected feature point. The feature descriptor generated by the PIIFD algorithm is invariant and can effectively represent the image information around the feature point without being affected by posture changes.
[0086] The feature descriptor generated by the PIIFD algorithm can more accurately describe the local structure and characteristics of each feature point, which is helpful for subsequent accurate matching and registration; the PIIFD algorithm has a certain posture invariance and can cope with some rotations and transformations in the image, improving the accuracy and stability of matching.
[0087] S143, using the RANSAC algorithm to perform pattern matching and delete unmatched points, and deform the matched images to obtain a preliminary registration result.
[0088] In this embodiment, the RANSAC (Random Sample Consensus) algorithm is used to perform pattern matching on the extracted and described feature points, identify and delete unmatched points, and then use the matched feature points to calculate a transformation model (such as an affine transformation or a perspective transformation) to deform the registration modality image to align with the reference modality image.
[0089] The RANSAC algorithm can effectively estimate the transformation model in the presence of outliers (mismatched points), thereby improving the robustness and accuracy of the registration. By deleting mismatched points and applying an accurate transformation model, a more accurate preliminary registration result can be obtained, ensuring the optimal spatial alignment of the two modal images.
[0090] In summary, these steps, combined with the use of the SURF detector, the PIIFD algorithm, and the RANSAC algorithm, can achieve preliminary image registration. At this stage, preliminary registration operations are performed on images of different modalities to align and keep the image sizes, fields of view, etc. of different modalities consistent as a whole; this lays the foundation for subsequent more sophisticated registration steps to ensure the reliability and accuracy of image data in medical image analysis and diagnosis.
[0091] S150 , performing blood vessel priori segmentation on the reference modality image and the registration modality image to obtain blood vessel priori information.
[0092] In this embodiment, the vascular prior information refers to the information of the vascular structure known in advance in the image, which is usually extracted from the image through a specific image processing method or a deep learning model. This information includes the location, shape, distribution and other characteristics of the blood vessels, which can be used as prior knowledge for subsequent image processing tasks, such as registration, segmentation, analysis, etc.
[0093] In medical image processing, fundus blood vessels are one of the important bases for ophthalmologists to diagnose and observe eye diseases. Therefore, extracting the vascular structure in fundus images and using it as prior information can help doctors better understand and analyze fundus images and improve the accuracy of diagnosis.
[0094] Vascular prior information can be obtained through various methods, including traditional vascular segmentation algorithms based on image processing technology, or by learning from a large amount of labeled fundus image data through deep learning neural networks. Regardless of the acquisition method, vascular prior information can play an important role in the subsequent image processing process, improving the accuracy and efficiency of processing.
[0095] Specifically, a vascular priori segmentation network is used to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information.
[0096] In one embodiment, the above-mentioned step S150 may include steps S151 - S152 .
[0097] S151, using the blood vessel priori segmentation network to perform feature encoding on the reference modality image and the registration modality image to obtain an encoding result;
[0098] S152: Decode the encoding result using a decoder to obtain blood vessel priori information.
[0099] For the vascular prior segmentation network, obtain the reference modality image and the registration modality image. These images are usually fundus images, including the structure and blood vessels of the retina; prepare the labeled data for training the deep learning model, including the segmentation label, that is, whether each pixel belongs to a blood vessel; after the fundus image is input, the image features are first extracted through the encoder. The encoder is usually composed of multiple convolutional layers, pooling layers, etc. to gradually extract high-level semantic features; the encoded features are passed to the decoder, which maps the features back to the original image space and outputs the predicted segmentation results. The decoder also consists of multiple layers of convolution and upsampling layers to restore the spatial resolution and generate a vascular segmentation map.
[0100] The segmentation error is calculated using the segmentation label (real blood vessel segmentation) and the predicted segmentation result output by the model. Common loss functions include cross entropy loss, Dice coefficient loss, etc. The model parameters are updated through the back propagation algorithm and optimizer (such as Adam, SGD) to minimize the loss function and improve the segmentation accuracy.
[0101] After the training is completed, the trained segmentation network is used to perform vascular segmentation on the reference modality image and the registration modality image respectively to obtain the vascular prior information of the two images.
[0102] The vascular prior information obtained by segmentation is used as input for further precise registration. Advanced registration methods based on feature point matching and deformation models (such as B-spline transformation) can be used to ensure that the vascular structure is perfectly aligned in the two images.
[0103] Vascular structure is a very important and significant feature in fundus images. By accurately segmenting the blood vessels through a deep learning segmentation network, the accuracy of registration can be significantly improved, especially the alignment effect on fine structures; initial coarse registration often aligns images on a global scale, but the errors in details may be large. Using vascular prior information for refined registration helps to reduce the accumulation of these errors and make the final fusion result more accurate; the deep learning segmentation network can adapt to a variety of image quality and perspective changes, provide robust vascular segmentation results, and reduce the sensitivity of the registration process to image noise and other interference factors; the deep learning segmentation network makes the vascular segmentation process highly automated, reducing the need for human intervention and manual annotation, and improving processing efficiency and consistency; this method is not only applicable to fundus images, but can also be extended to other medical imaging fields that require fine structure registration, such as cerebral blood vessels, pulmonary blood vessels, etc., to improve the accuracy and reliability of various medical image analyses.
[0104] By combining the vascular prior segmentation network, the image registration and fusion process becomes more accurate and efficient, providing more reliable imaging data support for medical diagnosis and treatment.
[0105] S160 , performing blood vessel prior fusion according to the reference modality image, the preliminary registration result, and the blood vessel prior information to obtain a fusion result.
[0106] In this embodiment, the fusion result refers to the final image obtained by inputting the reference modality image, the preliminary registration result and the vascular prior information into the vascular prior fusion network for processing. This fusion result is the output of the image fusion process guided by the vascular prior information.
[0107] Specifically, the reference modality image, the preliminary registration result and the vascular prior information are input into a vascular prior fusion network to perform image fusion to obtain a fusion result.
[0108] The vascular prior fusion network combines the reference modality images and the preliminary registration results, and uses the vascular prior information to guide the image fusion process. The network uses a convolutional neural network or other deep learning model to learn effective feature representations based on the input information and generate fusion results.
[0109] The fusion result usually contains the information of two or more input images. After registration and vascular prior guidance, a more accurate, clearer, and more vascular structurally consistent image is obtained. The fusion result can be used for further medical image analysis, diagnosis, disease detection and other applications, helping to improve doctors' understanding of images and decision-making capabilities.
[0110] In summary, the fusion result is obtained by inputting the reference modality image, preliminary registration result and vascular prior information into the vascular prior fusion network. The final output image obtained after image fusion processing has better image quality and vascular structure consistency.
[0111] In one embodiment, after the above step S160, the method further includes:
[0112] The generated fusion image is segmented using a blood vessel prior segmentation network to form a fusion image segmentation result;
[0113] Comparing the fused image segmentation result with the blood vessel priori segmentation result of the reference modality image to calculate the loss value;
[0114] The loss value is used to optimize the parameters of the blood vessel priori segmentation network.
[0115] Specifically, in order to ensure the consistency of the vascular structure before and after fusion, the previously obtained fusion image is input into the vascular prior segmentation network for feature extraction to capture the important features in the image; the output obtained is the vascular segmentation result of the fusion image, which can also be called the fusion image segmentation mask; the vascular prior segmentation result of the reference modality image is usually a real segmentation mask that is pre-labeled or obtained through another high-precision model; commonly used loss functions include cross entropy loss, Dice loss, IoU (Intersection over Union) loss, etc. These loss functions are used to measure the difference between the predicted segmentation result and the real segmentation result; the fusion image segmentation result (predicted mask) and the vascular prior segmentation result (real mask) of the reference modality image are input into the loss function to calculate the loss value. This loss value reflects the degree of difference between the two; using the calculated loss value, the gradient of the loss value relative to each parameter of the vascular prior segmentation network is calculated through the backpropagation algorithm; the gradient descent algorithm or its variants (such as the Adam optimizer) are applied to update the parameters of the network. Specifically, the weights and biases of the network are adjusted so that the loss value is reduced during the next prediction, thereby improving the segmentation performance. The above steps are repeated, and the network is trained iteratively for multiple times to gradually optimize the parameters of the vascular prior segmentation network, so that it can perform vascular segmentation more accurately.
[0116] In summary, these steps describe in detail the entire process from fusion image segmentation to loss value calculation, and then to optimizing the segmentation network parameters using the loss value. Through continuous optimization, the segmentation accuracy of the vascular prior segmentation network will gradually improve, thereby providing more accurate vascular segmentation results in practical applications.
[0117] S170: Output the fusion result.
[0118] The fusion result is output for the doctor to review.
[0119] The method of this embodiment is not limited to fundus images of a specific modality, and can fuse fundus images of multiple modalities, and has broad clinical application value and versatility; the multi-modal fundus images are preliminarily aligned through coarse registration technology, and then a neural network is used to optimize the local registration effect, thereby significantly improving the precision and accuracy of subsequent image fusion; image fusion is performed in combination with vascular prior information to ensure the consistency of vascular structure before and after fusion, while increasing the interpretability of the results.
[0120] By detecting feature points in fundus images, matching feature points and deep learning neural networks, multiple fundus images of different modalities and fields of view are fused into one fundus image, allowing doctors to find abnormalities and make diagnoses by comparing fundus images of different modalities. At the same time, in order to ensure the structural consistency of the fused image, this method segments the vascular structure in the fundus image and constrains the consistency of the vascular structure before and after fusion.
[0121] The above-mentioned multimodal fundus image fusion method obtains multimodal fundus images, determines the benchmark and registration images, performs coarse registration and vascular prior segmentation, and finally outputs the results after fusing the images, thereby improving the contrast and details of the images, thereby enhancing the accuracy and comprehensiveness of the diagnosis; it can solve the limitations of the registration and fusion of fundus images of different modalities in the clinical screening of fundus diseases, the low accuracy of local details, and the consistency problem of retaining important structures such as fundus blood vessels.
[0122] Figure 3 is a schematic block diagram of a multimodal fundus image fusion device 300 provided in an embodiment of the present invention. Figure 3 As shown, corresponding to the above multimodal fundus image fusion method, the present invention also provides a multimodal fundus image fusion device 300. The multimodal fundus image fusion device 300 includes a unit for executing the above multimodal fundus image fusion method, and the device can be configured in a server. Figure 3 The multimodal fundus image fusion device 300 includes an image acquisition unit 301, a determination unit 302, a registration order calculation unit 303, a coarse registration unit 304, a segmentation unit 305, a fusion unit 306 and an output unit 307.
[0123] An image acquisition unit 301 is used to acquire multimodal fundus images to be fused; a determination unit 302 is used to determine a reference modality image and a registration modality image from the multimodal fundus images to be fused; a registration order calculation unit 303 is used to calculate a registration order based on the reference modality image and the registration modality image; a coarse registration unit 304 is used to perform coarse registration on the reference modality image and the registration modality image to obtain a preliminary registration result; a segmentation unit 305 is used to perform vascular prior segmentation on the reference modality image and the registration modality image to obtain vascular prior information; a fusion unit 306 is used to perform vascular prior fusion based on the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result; and an output unit 307 is used to output the fusion result.
[0124] In one embodiment, the registration order calculation unit 303 includes:
[0125] The similarity calculation subunit is used to calculate the similarity between the reference modality image and the registration modality image using the Chamfer distance; the order determination subunit is used to generate a minimum spanning tree of the registration order using the Kruskal algorithm, and use the order of the minimum spanning tree as the registration order.
[0126] In one embodiment, the coarse registration unit 304 includes:
[0127] The feature point detection subunit is used to detect the feature points of the reference modality image and the registration modality image using the SURF detector in sequence according to the registration order; the description subunit is used to extract and describe the feature points of the reference modality image and the registration modality image using the PIIFD algorithm; the matching subunit is used to use the RANSAC algorithm to perform pattern matching and delete unmatched points, and deform the matched images to obtain preliminary registration results.
[0128] In one embodiment, the segmentation unit 305 is used to perform vascular prior segmentation on the reference modality image and the registration modality image using a vascular prior segmentation network to obtain vascular prior information.
[0129] In one embodiment, the segmentation unit 305 includes:
[0130] The encoding subunit is used to perform feature encoding on the reference modality image and the registration modality image using the vascular prior segmentation network to obtain an encoding result; the decoding subunit is used to decode the encoding result using a decoder to obtain vascular prior information.
[0131] In one embodiment, the fusion unit 306 is used to input the reference modality image, the preliminary registration result and the vascular prior information into a vascular prior fusion network to perform image fusion to obtain a fusion result.
[0132] In one embodiment, the above-mentioned device further includes:
[0133] The blood vessel segmentation subunit is used to segment the generated fused image using a blood vessel priori segmentation network to form a fused image segmentation result; the comparison subunit is used to compare the fused image segmentation result with the blood vessel priori segmentation result of the reference modality image to calculate a loss value; the optimization subunit is used to optimize the parameters of the blood vessel priori segmentation network using the loss value.
[0134] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned multimodal fundus image fusion device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.
[0135] The multimodal fundus image fusion device 300 can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.
[0136] See also Figure 4 , Figure 4 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.
[0137] See also Figure 4 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0138] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a multi-modal fundus image fusion method.
[0139] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0140] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multi-modal fundus image fusion method.
[0141] The network interface 505 is used to communicate with other devices over the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0142] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:
[0143] Acquire a multimodal fundus image to be fused; determine a reference modality image and a registration modality image from the multimodal fundus image to be fused; calculate a registration order according to the reference modality image and the registration modality image; perform coarse registration on the reference modality image and the registration modality image to obtain a preliminary registration result; perform vascular prior segmentation on the reference modality image and the registration modality image to obtain vascular prior information; perform vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result; and output the fusion result.
[0144] In one embodiment, when the processor 502 implements the step of calculating the registration sequence according to the reference modality image and the registration modality image, the processor 502 specifically implements the following steps:
[0145] The Chamfer distance is used to calculate the similarity between the reference modality image and the registration modality image; the minimum spanning tree of the registration order is generated using the Kruskal algorithm, and the order of the minimum spanning tree is used as the registration order.
[0146] In one embodiment, when the processor 502 implements the step of roughly registering the reference modality image and the registration modality image to obtain a preliminary registration result, the processor 502 specifically implements the following steps:
[0147] The SURF detector is used in sequence to detect the feature points of the reference modality image and the registration modality image in the registration order; the PIIFD algorithm is used to extract and describe the feature points of the reference modality image and the registration modality image; the RANSAC algorithm is used to perform pattern matching and delete unmatched points, and the matched images are deformed to obtain a preliminary registration result.
[0148] In one embodiment, when the processor 502 implements the step of performing a priori segmentation of blood vessels on the reference modality image and the registration modality image to obtain a priori information of blood vessels, the processor 502 specifically implements the following steps:
[0149] A vascular priori segmentation network is used to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information.
[0150] In one embodiment, when the processor 502 implements the step of performing vascular prior segmentation on the reference modality image and the registration modality image using the vascular prior segmentation network to obtain vascular prior information, the processor 502 specifically implements the following steps:
[0151] The reference modality image and the registration modality image are feature encoded using the vascular prior segmentation network to obtain an encoding result; the encoding result is decoded using a decoder to obtain vascular prior information.
[0152] In one embodiment, when the processor 502 implements the step of performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, the processor 502 specifically implements the following steps:
[0153] The reference modality image, the preliminary registration result and the blood vessel prior information are input into a blood vessel prior fusion network to perform image fusion to obtain a fusion result.
[0154] In one embodiment, after implementing the step of performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, the processor 502 further implements the following steps:
[0155] The generated fusion image is segmented using a vascular priori segmentation network to form a fusion image segmentation result; the fusion image segmentation result is compared with the vascular priori segmentation result of the reference modality image to calculate a loss value; and the loss value is used to optimize the parameters of the vascular priori segmentation network.
[0156] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0157] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0158] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:
[0159] Acquire a multimodal fundus image to be fused; determine a reference modality image and a registration modality image from the multimodal fundus image to be fused; calculate a registration order according to the reference modality image and the registration modality image; perform coarse registration on the reference modality image and the registration modality image to obtain a preliminary registration result; perform vascular prior segmentation on the reference modality image and the registration modality image to obtain vascular prior information; perform vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result; and output the fusion result.
[0160] In one embodiment, when the processor executes the computer program to implement the step of calculating the registration sequence according to the reference modality image and the registration modality image, the processor specifically implements the following steps:
[0161] The Chamfer distance is used to calculate the similarity between the reference modality image and the registration modality image; the minimum spanning tree of the registration order is generated using the Kruskal algorithm, and the order of the minimum spanning tree is used as the registration order.
[0162] In one embodiment, when the processor executes the computer program to implement the step of roughly registering the reference modality image and the registration modality image to obtain a preliminary registration result, the processor specifically implements the following steps:
[0163] The SURF detector is used in sequence to detect the feature points of the reference modality image and the registration modality image in the registration order; the PIIFD algorithm is used to extract and describe the feature points of the reference modality image and the registration modality image; the RANSAC algorithm is used to perform pattern matching and delete unmatched points, and the matched images are deformed to obtain a preliminary registration result.
[0164] In one embodiment, when the processor executes the computer program to implement the step of performing a priori segmentation of blood vessels on the reference modality image and the registration modality image to obtain a priori information of blood vessels, the processor specifically implements the following steps:
[0165] A vascular priori segmentation network is used to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information.
[0166] In one embodiment, when the processor executes the computer program to implement the step of using a vascular priori segmentation network to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information, the processor specifically implements the following steps:
[0167] The reference modality image and the registration modality image are feature encoded using the vascular prior segmentation network to obtain an encoding result; the encoding result is decoded using a decoder to obtain vascular prior information.
[0168] In one embodiment, when the processor executes the computer program to implement the step of performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, the processor specifically implements the following steps:
[0169] The reference modality image, the preliminary registration result and the blood vessel prior information are input into a blood vessel prior fusion network to perform image fusion to obtain a fusion result.
[0170] In one embodiment, after the processor executes the computer program to implement the step of performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result, the processor further implements the following steps:
[0171] The generated fusion image is segmented using a vascular priori segmentation network to form a fusion image segmentation result; the fusion image segmentation result is compared with the vascular priori segmentation result of the reference modality image to calculate a loss value; and the loss value is used to optimize the parameters of the vascular priori segmentation network.
[0172] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0174] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0175] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0177] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A multimodal fundus image fusion method, characterized in that: include: Acquire multimodal fundus images to be fused; Determining a reference modality image and a registration modality image from the multi-modality fundus images to be fused; Calculating a registration order according to the reference modality image and the registration modality image; Performing coarse registration on the reference modality image and the registration modality image to obtain a preliminary registration result; Performing blood vessel prior segmentation on the reference modality image and the registration modality image to obtain blood vessel prior information; Performing vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result; The fusion result is output.
2. The multimodal fundus image fusion method according to claim 1, characterized in that: The calculating the registration order according to the reference modality image and the registration modality image comprises: Calculating the similarity between the reference modality image and the registration modality image using Chamfer distance; The Kruskal algorithm is used to generate the minimum spanning tree of the registration order, and the order of the minimum spanning tree is used as the registration order.
3. The multimodal fundus image fusion method according to claim 1, characterized in that: The coarsely registering the reference modality image and the registration modality image to obtain a preliminary registration result includes: Using a SURF detector in sequence according to the registration order to detect feature points of the reference modality image and the registration modality image; Extracting and describing feature points of the reference modality image and the registration modality image using the PIIFD algorithm; The RANSAC algorithm is used to perform pattern matching and delete unmatched points, and the matched images are deformed to obtain preliminary registration results.
4. The multimodal fundus image fusion method according to claim 1, characterized in that: The performing a priori segmentation of blood vessels on the reference modality image and the registration modality image to obtain a priori information of blood vessels includes: A vascular priori segmentation network is used to perform vascular priori segmentation on the reference modality image and the registration modality image to obtain vascular priori information.
5. The multimodal fundus image fusion method according to claim 4, characterized in that: The using a vascular prior segmentation network to perform vascular prior segmentation on the reference modality image and the registration modality image to obtain vascular prior information includes: Using the blood vessel priori segmentation network to perform feature encoding on the reference modality image and the registration modality image to obtain an encoding result; The encoding result is decoded by a decoder to obtain blood vessel prior information.
6. The multimodal fundus image fusion method according to claim 1, characterized in that: The performing blood vessel prior fusion according to the reference modality image, the preliminary registration result and the blood vessel prior information to obtain a fusion result includes: The reference modality image, the preliminary registration result and the blood vessel prior information are input into a blood vessel prior fusion network to perform image fusion to obtain a fusion result.
7. The multimodal fundus image fusion method according to claim 1, characterized in that: After performing blood vessel prior fusion according to the reference modality image, the preliminary registration result and the blood vessel prior information to obtain a fusion result, the method further includes: The generated fusion image is segmented using a blood vessel prior segmentation network to form a fusion image segmentation result; Comparing the fused image segmentation result with the blood vessel priori segmentation result of the reference modality image to calculate the loss value; The loss value is used to optimize the parameters of the blood vessel priori segmentation network.
8. A multimodal fundus image fusion device, characterized in that: include: An image acquisition unit, used for acquiring multimodal fundus images to be fused; A determination unit, used to determine a reference modality image and a registration modality image from the multi-modality fundus images to be fused; A registration order calculation unit, used for calculating a registration order according to the reference modality image and the registration modality image; A coarse registration unit, used for coarsely registering the reference modality image and the registration modality image to obtain a preliminary registration result; a segmentation unit, configured to perform a priori blood vessel segmentation on the reference modality image and the registration modality image to obtain a priori blood vessel information; A fusion unit, configured to perform vascular prior fusion according to the reference modality image, the preliminary registration result and the vascular prior information to obtain a fusion result; An output unit is used to output the fusion result.
9. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.